pgvector Index Management Embedding Pipeline Optimization

Automating recall@k Benchmarks in GitHub Actions

This page shows how to run a pgvector recall@k benchmark on every pull request in GitHub Actions, so an index or parameter change that quietly degrades search quality fails the build instead of shipping. It scopes the problem narrowly: spin up a postgres+pgvector service container, load a fixture corpus, build an HNSW index, compute exact ground truth by forcing a sequential scan, and assert measured recall clears a threshold.

Up: Recall Regression Testing in CI

Recall is the one property of an approximate index you cannot observe in production without extra work: a query always returns some neighbours, so a drop from 0.98 to 0.85 recall is invisible until users notice worse results. The defence is a deterministic benchmark in CI that compares the index’s answers against brute-force ground truth on a fixed corpus, and fails the pull request on regression. The subtle part is generating trustworthy ground truth — you must force the planner off the very index under test — which is why the corpus and golden neighbours are treated as a versioned artifact in building a ground-truth recall test set.

Prerequisites

  • A GitHub repository with Actions enabled.
  • A small fixture corpus of embeddings (a few thousand vectors is enough for a stable signal) checked into the repo or fetched in the workflow.
  • Python 3.11+ with psycopg, numpy, and pytest.
  • A decided target metric and index type. This example uses HNSW with the inner-product opclass; whether HNSW or IVFFlat suits your workload is the subject of HNSW vs IVFFlat algorithm selection.
  • A recall threshold agreed with stakeholders (for example, recall@10 ≥ 0.95). The benchmark enforces it; it does not choose it.
The recall@k benchmark job as a CI sequence Five sequential stages joined by arrows. First a postgres pgvector service container starts. Then the fixture corpus loads. Then the HNSW index builds. Then brute-force ground truth is computed with sequential scan forced on. Finally recall at k is asserted against the threshold, failing the build on regression. One benchmark job, five ordered stages STEP 1 postgres+pgvector service container STEP 2 load fixture corpus STEP 3 build HNSW index STEP 4 ground truth (Seq Scan forced) STEP 5 assert recall@k ≥ threshold
Ground truth (step 4) must run with the index disabled, or it validates the index against itself.

Step-by-step

1. Define the workflow with a pgvector service container

Use the official pgvector/pgvector image as a service so Postgres is up before your job runs. The health check gates the steps until the server accepts connections.

YAML
name: recall-benchmark
on: [pull_request]

jobs:
  recall:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: pgvector/pgvector:pg16
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: bench
        ports: ['5432:5432']
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s --health-timeout 5s --health-retries 10
    strategy:
      matrix:
        params:
          - { m: 16, ef_construction: 64,  ef_search: 40 }
          - { m: 32, ef_construction: 128, ef_search: 80 }
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install psycopg[binary] numpy pytest
      - name: Run recall benchmark
        env:
          PGURL: postgresql://postgres:postgres@localhost:5432/bench
          HNSW_M: $
          HNSW_EFC: $
          HNSW_EFS: $
          RECALL_THRESHOLD: '0.95'
        run: pytest -q tests/test_recall.py

2. Load the fixture corpus and build the index

Inside the test’s setup, create the extension, load the fixtures, and build the HNSW index with the matrix parameters. Doing it per-run keeps the benchmark hermetic.

PYTHON
import os, numpy as np, psycopg

def setup_corpus(conn, vectors: np.ndarray):
    with conn.cursor() as cur:
        cur.execute("CREATE EXTENSION IF NOT EXISTS vector")
        cur.execute("DROP TABLE IF EXISTS items")
        cur.execute(f"CREATE TABLE items (id int PRIMARY KEY, embedding vector({vectors.shape[1]}))")
        with cur.copy("COPY items (id, embedding) FROM STDIN") as cp:
            for i, v in enumerate(vectors):
                cp.write_row((i, "[" + ",".join(f"{x:.6f}" for x in v) + "]"))
        cur.execute(
            f"CREATE INDEX ON items USING hnsw (embedding vector_ip_ops) "
            f"WITH (m = {int(os.environ['HNSW_M'])}, "
            f"ef_construction = {int(os.environ['HNSW_EFC'])})"
        )
    conn.commit()

3. Compute exact ground truth with a forced sequential scan

This is the load-bearing step. To get true nearest neighbours you must stop the planner from using the HNSW index — otherwise you are grading the index against itself and recall is trivially 1.0. Disable index and bitmap scans for the session so the exact ORDER BY ... <#> runs as a brute-force Seq Scan.

PYTHON
def ground_truth(conn, probe: np.ndarray, k: int) -> list[int]:
    lit = "[" + ",".join(f"{x:.6f}" for x in probe) + "]"
    with conn.cursor() as cur:
        cur.execute("SET LOCAL enable_indexscan = off")
        cur.execute("SET LOCAL enable_bitmapscan = off")
        cur.execute(
            "SELECT id FROM items ORDER BY embedding <#> %s::vector LIMIT %s",
            (lit, k),
        )
        return [r[0] for r in cur.fetchall()]

4. Run the ANN query and compute recall@k

Set hnsw.ef_search from the matrix, run the same query with the index enabled, and compare the two ID sets. Recall@k is the mean overlap fraction across a seeded query sample.

PYTHON
def ann_neighbours(conn, probe, k):
    lit = "[" + ",".join(f"{x:.6f}" for x in probe) + "]"
    with conn.cursor() as cur:
        cur.execute(f"SET LOCAL hnsw.ef_search = {int(os.environ['HNSW_EFS'])}")
        cur.execute(
            "SELECT id FROM items ORDER BY embedding <#> %s::vector LIMIT %s",
            (lit, k),
        )
        return [r[0] for r in cur.fetchall()]

def recall_at_k(conn, queries, k=10):
    scores = []
    for q in queries:
        truth = set(ground_truth(conn, q, k))
        approx = set(ann_neighbours(conn, q, k))
        scores.append(len(truth & approx) / k)
    return float(np.mean(scores))

5. Assert the threshold in pytest

The test fails the build when recall drops below the agreed floor. Seed the query sample so the number is reproducible run-to-run.

PYTHON
def test_recall_regression():
    rng = np.random.default_rng(42)
    corpus = np.load("tests/fixtures/corpus.npy").astype(np.float32)
    corpus /= np.linalg.norm(corpus, axis=1, keepdims=True)
    queries = corpus[rng.choice(len(corpus), size=200, replace=False)]

    with psycopg.connect(os.environ["PGURL"]) as conn:
        setup_corpus(conn, corpus)
        recall = recall_at_k(conn, queries, k=10)

    threshold = float(os.environ["RECALL_THRESHOLD"])
    assert recall >= threshold, f"recall@10={recall:.4f} < {threshold}"

Parameter reference

Parameter Type Default Production recommendation Notes
RECALL_THRESHOLD float 0.95 Set from a baseline minus margin The floor the PR must clear; derive from a measured baseline, not a guess.
k int 10 Match production LIMIT Recall@k is sensitive to k; benchmark the k your app actually requests.
query sample size int 200 200–1000 Larger samples tighten the estimate but lengthen CI; 200 is usually stable.
RNG seed int 42 Any fixed value Fixing the seed makes recall reproducible so a diff of ±0.01 is real, not sampling noise.
hnsw.ef_search GUC 40 Sweep in the matrix The recall/latency dial at query time; test the values you will run in production.
matrix m / ef_construction build params 16 / 64 Sweep 2–3 pairs Higher values raise recall and build cost; the matrix shows the trade-off per PR.

Verification

Confirm the ground truth truly ran without the index before trusting any recall number. Wrap the ground-truth query in EXPLAIN inside the test setup and assert the plan is a Seq Scan.

PYTHON
def assert_seq_scan(conn, probe, k):
    lit = "[" + ",".join(f"{x:.6f}" for x in probe) + "]"
    with conn.cursor() as cur:
        cur.execute("SET LOCAL enable_indexscan = off")
        cur.execute("SET LOCAL enable_bitmapscan = off")
        cur.execute(
            "EXPLAIN (FORMAT JSON) SELECT id FROM items "
            "ORDER BY embedding <#> %s::vector LIMIT %s", (lit, k))
        plan = cur.fetchone()[0][0]["Plan"]
        assert "Seq Scan" in str(plan), "ground truth used an index!"

A green run whose logs show recall printed and the Seq Scan assertion passing means the benchmark is measuring what you think. Publish the recall value as a job artifact or push it to Prometheus so the dashboard gauge and the CI gate share one number.

Troubleshooting

  • Recall is exactly 1.0 on every run. The ground truth used the HNSW index, so it agrees with itself. Confirm SET LOCAL enable_indexscan = off and enable_bitmapscan = off are issued on the same connection and session as the ground-truth query, and add the EXPLAIN Seq Scan assertion above.
  • Recall jitters ±0.03 between identical runs. The query sample is unseeded or too small. Fix the RNG seed and raise the sample to 200–1000; both make the estimate reproducible so a real regression stands out from noise.
  • The benchmark job takes many minutes. Brute-force ground truth is O(n·q·d) and dominates at large corpus sizes. Keep the CI fixture to a few thousand vectors, or precompute and cache the golden neighbours as an artifact instead of recomputing every run.
  • connection refused at the first query. The job started before Postgres was ready. Ensure the service --health-cmd and retries are set; steps only wait on the health check when it is defined.
  • Recall passes in CI but production feels worse. The fixture corpus is not representative, or hnsw.ef_search differs between CI and production. Match the CI parameters and query distribution to production, and sweep ef_search in the matrix to see the real recall/latency curve.