Ardent Performance Computing

Latest Post

Why Postgres Breaks Kubernetes container_memory_working_set_bytes Metric

Kubernetes metric container_memory_working_set_bytes is used for evicting/killing pods with too much memory use, especially if memory request < limit (don’t do this with Postgres). The metric is calculated from cgroups v2 memory.stat as current-inactive_file [source].

You’d assume it’s a good metric for memory usage in kubernetes. But with Postgres, this metric is very inaccurate for memory utilization and doesn’t tell you at all if you’re going to OOM crash your database.

After having the same conversation so many times about Postgres on Kubernetes, I need to write it down so I can just send people here to read it.

I will show better metrics to watch.

We start with fundamentals.

Note: scripts to reproduce all tests and graphs are at https://github.com/ardentperf/cgroup-postgres-memtest

Kubernetes Node E2E Tests

This is ground-zero for what Kubernetes promises to be true. AI research is telling me make test-e2e-node has several memory-pressure eviction tests:

• MemoryAllocatableEviction [source]
• MemoryAllocatableEvictionPodLevelResources [source]
• PriorityMemoryEvictionOrdering [source]
• PriorityMemoryEvictionOrderingPodLevelResources [source]

I believe these tests all use a test kit called agnhost [source]. Lets fire it up in docker and grab a few cgroup v2 metrics

docker run --name graph-repro-run_1-1821100 \
    --memory 512m --memory-swap 512m --detach \
    registry.k8s.io/e2e-test-images/agnhost:2.47 \
    stress --mem-alloc-size 25Mi --mem-alloc-sleep 5s --mem-total 1Gi

container_memory_working_set_bytes is the yellow line: current-inactive_file. It tells current memory usage, excluding linux page cache contents on the “active” file LRUs. The blue line is my own metric, where I’ve excluded all file LRUs (both active and inactive) – basically I’m saying “memory usage not including the page cache”.

Looking at the graph:

Anonymous memory ramp-up. As expected, OOM when memory usage hits the cgroup max (aka Pod Memory Limit). If you’re taking notes, remember that OOM will be a full database crash and restart for Postgres.

Simple. No shmem in the test, no active page cache in the test.

Postgres Simple Sort (ORDER BY)

Now Postgres.

docker run --name graph-repro-run_2-1821100 \
    --memory 512m --memory-swap 512m --detach \
    --env POSTGRES_PASSWORD=graphrepro \
    postgres:18 \
    -c shared_buffers=128MB

In a loop, let’s run a SQL query that sorts rows in memory. Add a half million rows each time until we OOM.

By default, Postgres limits itself to 4MB of working memory for sorts, and spills to temp files on disk after that. Tell Postgres to use more working memory. (Usually you’d decrease working memory if there are lots of concurrent connections all needing memory…)

SET work_mem = '1GB'; 
SET max_parallel_workers_per_gather = 0; 

SELECT count(*) FROM (
  SELECT md5(n::text) AS sort_key 
  FROM generate_series(1, $rows) AS input(n) 
  ORDER BY sort_key
) AS sorted_values;

Tracks pretty closely with Kubernetes agnhost. So far, so good. Postgres uses kernel anon memory to perform sorts. It can sort 4.5 million rows, but sorting 5 million rows crashes the database with OOM.

No active page cache.

Postgres Shared Buffers (database cache) are allocated as shmem by the Linux kernel. In this test, Postgres config has 128MB of memory for cache (cf. green line) but the memory has not been allocated by the kernel. This is because we didn’t create any tables.

Postgres with a Small Workload

Enter pgbench – the Postgres hackers best friend. Lets run it in the background while we test ORDER BY statements.

We’ll run the select-only workload and drop the PK from accounts to force full table scans on the accounts table (dropping the PK will also drop the index). We’re going for memory pressure, not TPS.

pgbench --initialize --scale=4

psql -c "ALTER TABLE pgbench_accounts DROP CONSTRAINT pgbench_accounts_pkey"

pgbench --select-only --client=2 --jobs=2

Scale 4 is about 70 MB.