You just doubled the RAM on your database server to handle a climb in p95 latency. You expect the extra memory to absorb your growing dataset and bring those 45ms spikes back down to 8ms. Instead, the dashboard shows minimal improvement. Write latency remains high, and query response times stay variable.
The problem isn’t that you added too little RAM. It’s that you gave most of it to the wrong layer.
PostgreSQL and your operating system both cache data independently. When you over-allocate memory to Postgres, the OS loses the RAM it needs to do its own caching. Both layers end up storing identical data blocks simultaneously, a condition known as double buffering, while your system spends CPU cycles shuffling data between two pools instead of serving queries. At scale, this pattern becomes a vicious cycle: you add resources, the database absorbs them, performance recovers briefly, and then degrades again as the dataset grows.
This guide explains the double buffering mechanism, gives you the tuning rule that breaks the cycle, and shows you how to diagnose whether your current configuration is already caught in it. By the end, you will know how to calculate the correct shared_buffers value for your server, run a query to identify which tables are crowding out your buffer cache, and interpret the results to decide what to do next.
The Two Layers of Database Memory
To manage memory effectively, you need to understand the differences between the two independent caches that operate simultaneously on every Postgres server.
The internal buffer cache is defined by the shared_buffers configuration parameter. When a query needs a data block, Postgres checks here first. Ideally, it finds the data block so it can avoid a system call entirely. This cache is where your hot data lives.
The OS page cache lives in whatever RAM the operating system has not allocated elsewhere. When Postgres requests a block that is not in shared_buffers, it issues a file system call. If the OS has that block in its page cache, it serves the data immediately. If not, the OS falls through to a physical disk read.
It’s important to note that Postgres does not manage the OS page cache at all. Instead, the kernel manages the cache on its own, including allocating space and moving data into and out of the cache. Regardless, the OS page cache is a required part of Postgres, and not just a backup option for the internal buffer cache.
The Double Buffering Problem
Double buffering happens because neither cache knows what the other holds. Postgres does not inspect the OS page cache before storing a block in shared_buffers. The OS does not inspect shared_buffers before caching a file page. Both layers frequently hold copies of the same data at the same time.
This is wasteful at any size, but at scale it becomes actively harmful.
When shared_buffers is set too high (e.g. 80% of total RAM), the OS page cache is confined to the remaining 20%. Under a write-heavy workload, the OS needs that headroom to manage checkpoint writes, background writer activity, and WAL file flushes that grow proportionally with data volume. When the OS cache is too small, the kernel is forced to evict useful data pages to make room for incoming writes. Postgres then misses in both caches and falls through to disk, even if you have plenty of RAM.
This creates a vicious cycle. Adding more RAM to shared_buffers temporarily absorbs the working set, but as the dataset grows the same pressure returns. Each tuning cycle buys less time than the one before it.
Using The 25% Rule
The standard recommendation for Postgres is to set shared_buffers to 25% of total system RAM. By leaving 75% of memory to the OS, you give the kernel the headroom it needs to cache active data files, manage writes, and handle I/O bursts without evicting pages that Postgres will immediately need again.
To apply this, open postgresql.conf and update the parameter:
# For a server with 64GB RAM: 25% = 16GB
shared_buffers = '16GB'
This parameter requires a full server restart. A configuration reload is not sufficient.
Large Memory Servers
On systems with 512GB or more of RAM, 25% works out to 128GB. Beyond this point, the overhead of managing the internal buffer mapping can decrease performance rather than improve it. For very large memory systems, many teams cap shared_buffers at 128GB to 256GB and let the OS page cache handle the rest. Treat 128GB as your starting ceiling and benchmark from there.
Additional Settings
Changing shared_buffers in isolation can produce misleading results if these settings are not also configured correctly:
-
effective_cache_size: Tells the query planner how much total cache (shared_buffersplus OS page cache combined) it can expect to use. Set this to 50-75% of total RAM. It does not allocate memory, but rather informs planning decisions and affects whether the planner chooses index scans over sequential scans. -
work_mem: Controls per-operation memory for sorts and hash joins. Too high, and concurrent queries can exhaust available RAM; too low, and sort operations spill to disk. A conservative starting point is total RAM divided by (max_connectionsx 2). On a 64GB server with 200max_connections, that works out to roughly 163MB per operation, a reasonable baseline to start from and adjust under load. -
checkpoint_completion_target: Set to 0.9 to spread checkpoint writes across a longer window, reducing the I/O spikes that compete with the OS page cache during heavy write periods.
Diagnosing Your Current Configuration
Once you apply the 25% rule, the pg_buffercache extension shows you exactly which tables and indexes are occupying your buffer cache right now.
SELECT
c.relname AS table_name,
count(*) AS buffered_pages,
pg_size_pretty(count(*) * 8192) AS buffer_size,
round(100.0 * count(*) /
(SELECT setting FROM pg_settings WHERE name = 'shared_buffers')::integer, 2
) AS percent_of_cache
FROM pg_buffercache b
INNER JOIN pg_class c ON b.relid = c.oid
INNER JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')
GROUP BY c.relname
ORDER BY buffered_pages DESC
LIMIT 10;
Interpreting Your Results
A healthy result shows no single object above 15-20% of the cache.
If any single table or index exceeds 30% of the cache, treat it as a signal that one object is crowding out everything else. Do not respond by increasing shared_buffers. If the object is already larger than your current allocation, giving Postgres more memory will only delay the problem until the table grows again. Instead, ask yourself the following questions:
- Can the table be partitioned by time or key range so that queries touch only a recent, smaller slice of the data?
- Can the queries driving the cache pressure be rewritten to use more selective indexes rather than scanning large portions of the table?
Addressing Index Bloat
A separate but related problem is index bloat. When index entries dominate the output over table entries, your indexes have likely grown faster than your access patterns have changed. Use this query to identify indexes that are consuming cache but receiving no scans:
SELECT
schemaname,
tablename,
indexname,
idx_scan AS scans,
pg_size_pretty(pg_relation_size(indexrelid)) AS index_size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;
Any index returned here is a candidate for removal. Dropping unused indexes directly reduces buffer pressure and frees cache space for objects that are actually serving queries.
Re-run the pg_buffercache query after any significant data volume increase or schema change to catch concentration drift before it affects query performance.
When Tuning Reaches Its Limit
The 25% rule and the diagnostics above will recover significant performance for most Postgres deployments. But when your working dataset is larger than the memory you can reasonably allocate to either cache layer, buffer management stops being the constraint. Instead, the data volume itself is the problem.
You can see this in pg_buffercache directly. If your largest table is 60GB and shared_buffers is 16GB, the table will never be fully cached regardless of how the allocation is tuned. percent_of_cache for that object will always approach 100% as the query workload pulls it in, leaving nothing for everything else:
At this point, adding more RAM extends the runway but does not change the slope. The next doubling of your dataset will return you to this same result. Columnar storage changes the equation by compressing data aggressively before it ever reaches the cache, reducing the volume that needs to be buffered in the first place.
You can test whether your workload would benefit from this approach by running the same pg_buffercache checks on a Tiger Data instance. Start a free trial today to optimize your database and internal buffer cache without affecting production.
