Optimizing m and ef_construction Parameters

Up: HNSW & IVFFlat Index Creation & Tuning

The Hierarchical Navigable Small World (HNSW) index in pgvector is only as good as the two numbers you pass at build time: m and ef_construction. Get them wrong and the failure is quiet — the index builds cleanly, queries return results, and nothing errors — but recall silently plateaus below your target, build windows stretch from minutes into hours, or the graph’s neighbor lists blow past your memory budget and push the working set out of shared_buffers. Unlike ef_search, which you can retune per query, m and ef_construction are frozen into the graph the moment CREATE INDEX finishes; the only way to change them is a full rebuild. This page is the calibration reference for choosing those values against a real workload: how each knob reshapes the graph, the diagnostic SQL that tells you which one to move, a repeatable sweep procedure, and the monitoring hooks that catch topology drift before users feel it.

Architectural Divergence & Trade-offs #

Before tuning HNSW at all, confirm HNSW is the right structure — that decision belongs to the HNSW vs IVFFlat algorithm selection framework, not to this page. HNSW targets sub-millisecond latency at high recall and accepts a large, RAM-resident graph to get there; IVFFlat trades a little latency for a predictable memory ceiling and fast rebuilds. If your SLA is a hard memory cap or you can tolerate recall in the 0.85–0.90 band, tuning IVFFlat lists for high-throughput similarity search may make the entire m/ef_construction exercise moot. Everything below assumes you have already committed to HNSW.

Within HNSW there are effectively three levers, and the art is understanding which failure each one fixes:

m — graph density. m sets the maximum number of bidirectional edges a node keeps on each layer (the ground layer actually allows up to 2 * m). It defines the navigational skeleton: more edges mean more shortcuts for greedy descent to follow, which raises the recall ceiling the graph can ever reach, but every edge is stored forever. m costs memory linearly and build time superlinearly, and it is the parameter you cannot compensate for at query time — a sparse graph traps greedy search in local minima no matter how high you push ef_search.

ef_construction — build-time search effort. ef_construction is the size of the dynamic candidate list explored while inserting each new node. A larger list means each node evaluates more potential neighbors before committing its edges, producing a graph with fewer dead ends and better-placed long-range links. It costs build time but almost no query-time memory (the candidate list is transient) — its entire footprint is baked into edge quality, not edge count.

ef_search — query-time search effort. Not a construction parameter, but the reason the other two matter. ef_search widens the candidate list at query time and is the only one of the three you can tune live. It can recover recall lost to a thin search, but it cannot manufacture edges that were never built. The practical corollary drives the whole diagnostic workflow below: if turning ef_search up stops helping, the problem is structural and lives in m or ef_construction.

The trade-off surface, then, is not one-dimensional. Two indexes with identical m can differ sharply in recall if one was built with ef_construction = 64 and the other with 256, because the second placed better edges within the same edge budget. Conversely, past a point, more ef_construction cannot help a graph that is simply too sparse — you have to raise m. The tuning job is deciding which wall you are hitting.

Parameter Space & Diagnostic Workflow #

pgvector ships conservative defaults — m = 16, ef_construction = 64 — that are tuned for a clean first build on modest data, not for production recall. The table below pairs each with the ranges that hold up under real embedding workloads.

The memory math is what makes m the dangerous knob. For N vectors the HNSW graph structure alone costs roughly 4 * m * N * 1.1 bytes for the edge lists (independent of the vector payload itself), so at m = 32 and 50M rows the neighbor lists approach ~7 GB before you count the vectors. That is why m > 32 so often becomes the primary bottleneck on large corpora, forcing vertical scaling or sharding. Combine that with the per-row vector cost from the pgvector storage overhead analysis to get the true resident footprint before you commit to a value.

Start every tuning session with the same diagnostic: read what is actually built. pgvector exposes the WITH options through the standard catalog.

-- Inspect the m / ef_construction an existing HNSW index was built with
SELECT
    i.relname                              AS index_name,
    t.relname                              AS table_name,
    am.amname                              AS index_type,
    c.reloptions                           AS build_options
FROM pg_class i
JOIN pg_index x  ON x.indexrelid = i.oid
JOIN pg_class t  ON t.oid = x.indrelid
JOIN pg_am am    ON am.oid = i.relam
JOIN pg_class c  ON c.oid = i.oid
WHERE am.amname = 'hnsw';

If build_options is NULL, the index was built on defaults (m=16, ef_construction=64) — the single most common root cause of “our recall is stuck at 0.88.” Confirm the on-disk size so you know what a rebuild will cost in memory:

SELECT
    relname                                   AS index_name,
    pg_size_pretty(pg_relation_size(oid))     AS index_size
FROM pg_class
WHERE relkind = 'i' AND relname LIKE '%hnsw%';

The workflow from here is strictly ordered, because the knobs are not independent:

1. Fix ef_search at a high value (e.g. 200) and measure recall. This tells you the recall ceiling the current graph can reach. If it clears your target, you have a query-time tuning problem, not a construction problem — stop and lower ef_search to the cheapest value that still hits target.
2. If the ceiling is below target, the graph is the limit. Sweep ef_search from 40 to 256. If recall keeps climbing, ef_construction is the constraint — rebuild with a higher value. If recall flattens early, m is the constraint — rebuild denser.
3. Only change one construction parameter per rebuild. Because build time and memory both move with m and ef_construction, changing both at once makes it impossible to attribute the result.

Step-by-Step Implementation #

The safe pattern is: stage on an unindexed (or lightly loaded) table, build with explicit parameters, then promote without blocking traffic. Index construction in pgvector runs single-threaded per connection for the serial portions of the build, so ef_construction maps almost directly to wall-clock time — orchestrate it off the live write path. The full non-blocking build machinery lives in asynchronous index build strategies; the ordered end-to-end procedure is in step-by-step HNSW index creation for production workloads.

Step 1 — Prepare the build session. Raise maintenance_work_mem so the graph builds in RAM instead of spilling, and give HNSW parallel workers. These are session-local and do not touch global config.

SET maintenance_work_mem = '4GB';
SET max_parallel_maintenance_workers = 6;

Step 2 — Build with explicit parameters. Never rely on defaults for a production index. Match the operator class to the distance metric the query actually uses — a vector_cosine_ops index cannot serve an L2 query and the planner will silently fall back to a sequential scan.

CREATE INDEX CONCURRENTLY idx_chunks_hnsw
    ON document_chunks
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 32, ef_construction = 128);

Step 3 — Refresh statistics. After the index is valid, re-run ANALYZE so the planner has accurate estimates; a stale row estimate can push the planner back toward a sequential scan even with a healthy index present.

ANALYZE document_chunks;

Step 4 — Set a workload-appropriate ef_search. Pin it at the session or role level for the querying service so recall is reproducible.

SET hnsw.ef_search = 100;   -- start here, then tune down against recall

For a systematic sweep rather than a single guess, drive the whole matrix from Python so each (m, ef_construction) pair is built, measured, and torn down under identical conditions:

import psycopg
import numpy as np
from pgvector.psycopg import register_vector


def benchmark_hnsw_params(conn, probe_vectors, ground_truth, m_vals, ef_c_vals,
                          ef_search=100, k=10):
    """Build each HNSW config, measure recall@k against exhaustive ground truth."""
    register_vector(conn)
    results = []
    for m in m_vals:
        for ef_c in ef_c_vals:
            with conn.cursor() as cur:
                cur.execute("DROP INDEX IF EXISTS test_hnsw")
                cur.execute(
                    "CREATE INDEX test_hnsw ON embeddings "
                    "USING hnsw (vec vector_cosine_ops) "
                    "WITH (m = %s, ef_construction = %s)",
                    (m, ef_c),
                )
                cur.execute("SET hnsw.ef_search = %s", (ef_search,))
                hits = 0
                for q, truth in zip(probe_vectors, ground_truth):
                    cur.execute(
                        "SELECT id FROM embeddings ORDER BY vec <=> %s LIMIT %s",
                        (q, k),
                    )
                    approx = {row[0] for row in cur.fetchall()}
                    hits += len(approx & set(truth[:k]))
            recall = hits / (len(probe_vectors) * k)
            results.append({"m": m, "ef_construction": ef_c, "recall": round(recall, 4)})
    return results

Validation & Recall Testing #

Parameter tuning without a recall measurement is guessing. Recall must be measured against a ground truth produced by exhaustive (exact) search on the same probe set, then compared to the approximate index results. Never validate on your development laptop against 10k rows — cluster coherence and the curse of dimensionality both change behavior at scale, so validate on a representative slice or a shadow copy of production.

Establish ground truth with an exact scan. Force the exact path by computing distances without the index (or on a table with no HNSW index), take the top-k per probe, and store those IDs.

-- Exact top-10 neighbors for one probe vector (ground truth)
SELECT id
FROM   embeddings
ORDER  BY vec <=> :probe_vector
LIMIT  10;

Confirm the index is actually used. The single most common silent failure is the planner choosing a sequential scan — often because of an operator/opclass mismatch or a stale ANALYZE. EXPLAIN ANALYZE is the arbiter:

EXPLAIN (ANALYZE, BUFFERS)
SELECT id
FROM   embeddings
ORDER  BY vec <=> :probe_vector
LIMIT  10;

Look for an Index Scan using ... hnsw node. A Seq Scan here means every recall number you collect is measuring exact search, not your index — fix the scan before trusting any metric. The same EXPLAIN (ANALYZE, BUFFERS) discipline used to read Heap Fetches and buffer hits in the tuning IVFFlat lists for high-throughput similarity search matrix applies unchanged to an HNSW m × ef_search sweep.

Interpret the recall curve. Plot recall against ef_search for each (m, ef_construction) build:

• A curve that keeps rising through ef_search = 256 → the graph has more recall to give; increase ef_construction and rebuild.
• A curve that flattens by ef_search = 100 well below target → the graph is too sparse; increase m and rebuild.
• A curve that hits target at low ef_search → done; lock the smallest ef_search that clears target to keep latency down.

Beyond about ef_construction = m * 8, expect recall gains under 0.5% while build time keeps climbing — that is the signal to stop raising it. When a rebuild produces an index whose recall or validity is off, classify the outcome with index validation error categorization rather than blindly rebuilding again.

Failure Modes & Gotchas #

• Silent sequential-scan fallback. An operator-class mismatch (cosine index, L2 query), an index left invalid by a failed concurrent build, or a missing ANALYZE all send the planner to a Seq Scan. Queries still return correct results, so nothing alerts — only EXPLAIN ANALYZE reveals it. Always check the plan after any rebuild.
• Recall plateau mistaken for a ef_search problem. Teams crank ef_search to 500 chasing recall that a sparse graph can never deliver, paying latency for nothing. If two doublings of ef_search add <2% recall, stop tuning the query and rebuild with higher m.
• Build OOM and swap thrash. When the graph working set exceeds maintenance_work_mem, the build spills and can drive shared_buffers eviction and OS-level swapping, stretching a 20-minute build into hours. Watch for it in pg_stat_activity; the fix is more maintenance_work_mem or a lower m, not patience.
• WAL pressure from concurrent rebuilds. CREATE INDEX CONCURRENTLY on a large HNSW graph generates substantial WAL and runs a two-phase build; on a busy primary this can pressure replication lag and archive throughput. Stage large rebuilds on a replica or off-peak, and see resolving pgvector index build timeout errors when a build stalls.
• High-dimensional distance concentration. Above ~1024 dimensions, distances converge (the curse of dimensionality) and extra m buys little recall while costing full memory. Prefer m = 16–24 with ef_construction = 96–128 and lean on ef_search, rather than densifying a graph that cannot separate neighbors anyway.
• Distribution drift after model migration. A new embedding model shifts the vector distribution, and a graph tuned for the old distribution degrades even though nothing in the index changed. Re-validate recall after any model swap and rebuild on a cadence (quarterly is a reasonable default for stable corpora).

Monitoring & Alerting Hooks #

Topology problems are invisible to error logs — you have to instrument for them. Three signals catch nearly every regression.

Watch builds in progress. pg_stat_progress_create_index reports live phase and block counts so you can estimate completion and detect a stalled build early instead of discovering it at timeout.

SELECT
    p.phase,
    p.blocks_done,
    p.blocks_total,
    round(100.0 * p.blocks_done / NULLIF(p.blocks_total, 0), 1) AS pct_done,
    a.query
FROM pg_stat_progress_create_index p
JOIN pg_stat_activity a ON a.pid = p.pid;

Alert on unused or invalid indexes. An HNSW index with a flat idx_scan count is being bypassed by the planner — the sequential-scan trap, caught after the fact. Scrape this for a Prometheus-compatible gauge and alert when scans stay at zero on a table that is being queried.

SELECT
    indexrelid::regclass       AS index_name,
    relid::regclass            AS table_name,
    idx_scan                   AS index_scans,
    idx_tup_read               AS tuples_read
FROM pg_stat_user_indexes
WHERE indexrelname LIKE '%hnsw%';

Guard recall in CI. The only reliable defense against slow topology drift is a scheduled recall check against a fixed probe set with known ground truth, run in CI or a cron job. Emit the number as a metric and alert when it drops below your floor.

def recall_gauge(conn, probes, ground_truth, k=10, ef_search=100):
    """Return recall@k for a fixed probe set — export to Prometheus / fail CI on drop."""
    with conn.cursor() as cur:
        cur.execute("SET hnsw.ef_search = %s", (ef_search,))
        hits = 0
        for q, truth in zip(probes, ground_truth):
            cur.execute(
                "SELECT id FROM embeddings ORDER BY vec <=> %s LIMIT %s", (q, k)
            )
            approx = {row[0] for row in cur.fetchall()}
            hits += len(approx & set(truth[:k]))
    return hits / (len(probes) * k)

Pair these with a build-time budget so a regression in ef_construction (or an accidental default rebuild) shows up as a build-duration alert, not a user complaint.

Operational Tuning Checklist #

• Never tune m and ef_construction in isolation — always pair a rebuild with an ef_search sweep to separate construction from traversal effects.
• Change one construction parameter per rebuild so results are attributable.
• Automate the parameter sweep in CI against a representative slice; hard-coding defaults guarantees suboptimal recall.
• Rebuild after embedding-model migrations and on a cadence — distribution shift degrades a fixed topology over time.

FAQ #

Related #

• HNSW vs IVFFlat algorithm selection — decide whether HNSW tuning even applies before touching m
• Tuning IVFFlat lists for high-throughput similarity search — the lists/probes companion to these knobs
• Step-by-step HNSW index creation for production workloads — the ordered build procedure end to end
• Asynchronous index build strategies — build these parameters into a live table without downtime
• Index validation & error categorization — classify a rebuild that comes back invalid or degraded
• pgvector storage overhead analysis — size the resident footprint before committing to an m

Up: HNSW & IVFFlat Index Creation & Tuning