Prometheus Metrics for pgvector Search

A vector search endpoint fails silently in a way row-count dashboards never catch: recall drifts downward as data grows past the lists you originally tuned for, hnsw.ef_search gets bumped in a config reload and p95 latency doubles, or an autovacuum storm evicts the index from cache and every query starts hitting disk. None of this shows up in request counts, and — critically — the metric that matters most, the client-perceived latency of an ORDER BY embedding <=> $1 query, is invisible to PostgreSQL’s own statistics views because they measure server-side execution, not the round trip your users feel. Robust observability for pgvector therefore needs two complementary metric sources stitched into Prometheus: a database-side scrape that exposes index usage, size, and cache behavior from the pg_stat_* catalog, and application-side instrumentation that times the real query and records measured recall. This page builds both, shows the queries.yaml and prometheus_client code, and gives the PromQL and alert rules that turn the raw series into pages that fire before users notice.

Up: Monitoring, Observability & Operations

Architectural Divergence & Trade-offs #

There is no single metric endpoint that tells you whether vector search is healthy. Three sources each see a different slice of the truth, and the common mistake is to wire up only the database exporter — which cannot measure the one number your SLA is written against.

Database-side scrape via postgres_exporter. The exporter connects as a monitoring role, runs SQL against the statistics catalog on each scrape, and republishes rows as Prometheus series. A custom queries.yaml maps pg_stat_user_indexes (scan counts, tuples read/fetched), pg_statio_user_indexes (buffer hits vs disk reads), and pg_stat_user_tables (sequential vs index scans, dead-tuple counts) onto gauges and counters. This is the only view of index internals — is the HNSW index even being used, how large has it grown, is it resident in shared_buffers. What it fundamentally cannot see is the latency the client experiences: these are cumulative counters sampled at the scrape interval, not per-query timings, and they stop at the server boundary.

Application-side instrumentation with prometheus_client. Wrapping the actual search call — the ORDER BY embedding <=> $1 LIMIT k round trip — in a Histogram captures end-to-end ANN latency including connection acquisition, network, planning, execution, and result serialization. This is the number that matches a user-facing p95 SLA. The same in-process instrumentation is the only place you can expose a recall Gauge, because recall is a measured property of results (see recall regression testing in CI) that the database has no concept of. The cost is that you must instrument every code path that queries vectors, and you must keep label cardinality disciplined.

Per-statement timing via pg_stat_statements. Loaded through shared_preload_libraries, this extension aggregates execution statistics per normalized query, giving calls, mean_exec_time, stddev_exec_time, min_exec_time, and max_exec_time for the distance query specifically. It is the sharpest tool for attributing server-side cost to one statement and separating planning from execution — but note it exposes mean, standard deviation, min, and max, not a true percentile, so it complements rather than replaces the application histogram for p95 work.

The production answer is to run all three: the exporter and pg_stat_statements explain why the server is slow (index not used, cache cold, plan changed), while the application histogram tells you that the client is slow and by how much. Latency you cannot correlate to an index-usage counter is guesswork, and index counters with no latency series next to them cannot page you on an SLA breach.

Parameter Space & Diagnostic Workflow #

The knobs that govern signal quality live in three places: PostgreSQL configuration, the exporter’s collector definition, and the histogram bucket layout in your application. Get any of them wrong and the metrics are present but misleading — a histogram whose buckets top out at 500 ms is useless once p99 crosses a second, and a label set that includes the query text explodes into millions of series.

Before writing any collector, confirm what the planner is doing and which indexes are actually earning their keep. Start at pg_stat_user_indexes filtered to the vector access methods:

SELECT s.relname,
       s.indexrelname,
       s.idx_scan,
       s.idx_tup_read,
       s.idx_tup_fetch,
       pg_size_pretty(pg_relation_size(s.indexrelid)) AS index_size
FROM pg_stat_user_indexes s
JOIN pg_class c ON c.oid = s.indexrelid
JOIN pg_am    am ON am.oid = c.relam
WHERE am.amname IN ('hnsw', 'ivfflat')
ORDER BY s.idx_scan DESC;

An idx_scan of zero on a vector index that should be serving traffic is the loudest signal there is: the planner is choosing a sequential scan, usually because hnsw.ef_search combined with a restrictive filter made the index path look expensive, or because the index was rebuilt and ANALYZE never ran. Cross-check cache residency from the I/O view, since a resident index and a disk-bound one differ by an order of magnitude in latency:

SELECT indexrelname,
       idx_blks_hit,
       idx_blks_read,
       round(idx_blks_hit::numeric
             / nullif(idx_blks_hit + idx_blks_read, 0), 4) AS hit_ratio
FROM pg_statio_user_indexes
WHERE indexrelname LIKE '%embedding%'
ORDER BY hit_ratio ASC;

Then attribute server cost to the distance query itself. Filter pg_stat_statements to statements containing a pgvector distance operator:

SELECT calls,
       round(mean_exec_time::numeric, 2)  AS mean_ms,
       round(stddev_exec_time::numeric, 2) AS stddev_ms,
       round(max_exec_time::numeric, 2)   AS max_ms,
       round((100 * total_exec_time
              / sum(total_exec_time) OVER ())::numeric, 1) AS pct_total
FROM pg_stat_statements
WHERE query LIKE '%<=>%'
   OR query LIKE '%<->%'
   OR query LIKE '%<#>%'
ORDER BY mean_exec_time DESC
LIMIT 10;

A high stddev_ms relative to mean_ms here is the fingerprint of cache thrash or lock contention — the same query is sometimes fast and sometimes not — and is exactly the case where the aggregate mean lies and you need the application histogram’s percentiles to see the tail.

Step-by-Step Implementation #

The build has two halves that meet at Prometheus: expose the database view through the exporter, and instrument the search path in the service. Both must land in the same Prometheus so the series can be correlated in one query.

Step 1 — Load pg_stat_statements and confirm it is active. Add it to shared_preload_libraries in postgresql.conf, restart, then create the extension in the target database.

-- postgresql.conf:  shared_preload_libraries = 'pg_stat_statements'
-- postgresql.conf:  track_io_timing = on
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Verify the library actually loaded (empty result => restart was skipped)
SELECT count(*) FROM pg_stat_statements;

If the CREATE EXTENSION succeeds but the view stays empty, the library was never preloaded — the single most common reason per-statement metrics silently go missing.

Step 2 — Define the exporter collectors in queries.yaml. This maps the diagnostic queries above onto Prometheus metric types. Keep labels to the schema, table, and index names; expose scan counts as counters and size as a gauge.

pg_vector_index_usage:
  query: |
    SELECT s.schemaname, s.relname, s.indexrelname,
           s.idx_scan, s.idx_tup_read, s.idx_tup_fetch,
           pg_relation_size(s.indexrelid) AS index_size_bytes
    FROM pg_stat_user_indexes s
    JOIN pg_class c ON c.oid = s.indexrelid
    JOIN pg_am    am ON am.oid = c.relam
    WHERE am.amname IN ('hnsw', 'ivfflat')
  master: true
  cache_seconds: 30
  metrics:
    - schemaname:   {usage: "LABEL",   description: "Schema"}
    - relname:      {usage: "LABEL",   description: "Table"}
    - indexrelname: {usage: "LABEL",   description: "Vector index"}
    - idx_scan:     {usage: "COUNTER", description: "Index scans initiated"}
    - idx_tup_read: {usage: "COUNTER", description: "Index entries returned"}
    - index_size_bytes: {usage: "GAUGE", description: "On-disk index size"}

pg_vector_index_io:
  query: |
    SELECT relname, indexrelname, idx_blks_hit, idx_blks_read
    FROM pg_statio_user_indexes
    WHERE indexrelname LIKE '%embedding%'
  cache_seconds: 30
  metrics:
    - relname:       {usage: "LABEL"}
    - indexrelname:  {usage: "LABEL"}
    - idx_blks_hit:  {usage: "COUNTER", description: "Buffer hits"}
    - idx_blks_read: {usage: "COUNTER", description: "Disk block reads"}

Point the exporter at this file (--extend.query-path=/etc/postgres_exporter/queries.yaml) and it publishes series named after the collector, e.g. pg_vector_index_usage_idx_scan{indexrelname="items_embedding_hnsw"}.

Step 3 — Instrument the search path in the service. Wrap only the query round trip in the Histogram, and keep the label set to the index name and k. Fitting the buckets to your SLA is the load-bearing decision here.

from prometheus_client import Histogram, Gauge, start_http_server
import asyncpg

SEARCH_LATENCY = Histogram(
    "pgvector_search_seconds",
    "End-to-end ANN search latency, client-perceived",
    labelnames=("index", "k"),
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)
RECALL_AT_10 = Gauge(
    "pgvector_recall_at_10",
    "Most recent measured recall@10 versus brute-force ground truth",
    labelnames=("index",),
)

SEARCH_SQL = "SELECT id FROM items ORDER BY embedding <=> $1 LIMIT $2"

async def search(pool: asyncpg.Pool, qvec: str, k: int = 10) -> list[int]:
    # .time() records the full acquire+execute+fetch round trip.
    with SEARCH_LATENCY.labels(index="items_embedding_hnsw", k=str(k)).time():
        async with pool.acquire() as conn:
            rows = await conn.fetch(SEARCH_SQL, qvec, k)
    return [r["id"] for r in rows]

The label values (index, k) are drawn from a small fixed set. Never add the query id, tenant id, or the vector literal as a label — each distinct value creates a new time series and Prometheus memory grows linearly with cardinality.

Step 4 — Feed the recall Gauge from a periodic job. Recall is expensive to measure, so update it on a schedule rather than per request, reusing the ground-truth machinery detailed under recall regression testing in CI.

async def refresh_recall(pool, probe_vectors, truth_ids, k=10):
    hits = total = 0
    for qvec, truth in zip(probe_vectors, truth_ids):
        ann = await search(pool, qvec, k)
        hits  += len(set(ann) & set(truth[:k]))
        total += k
    RECALL_AT_10.labels(index="items_embedding_hnsw").set(hits / total)

Step 5 — Expose and scrape. Call start_http_server(9109) in the service so prometheus_client serves /metrics, and add both the app and the exporter as scrape targets. A 15-second interval matches the exporter’s cache_seconds and keeps latency SLOs responsive:

scrape_configs:
  - job_name: pgvector_app
    scrape_interval: 15s
    static_configs: [{targets: ["search-svc:9109"]}]
  - job_name: postgres_exporter
    scrape_interval: 15s
    static_configs: [{targets: ["pg-exporter:9187"]}]

Validation & Recall Testing #

Metrics are only useful once you have proven they move when the underlying reality moves. Validate each source against a known-good query before trusting a dashboard built on it.

Prove the index counter increments. Confirm the search actually uses the HNSW index — not a sequential scan that would make idx_scan flat forever — with EXPLAIN ANALYZE, then check the counter advanced:

EXPLAIN (ANALYZE, BUFFERS)
SELECT id FROM items
ORDER BY embedding <=> '[0.12, -0.03, ...]'::vector
LIMIT 10;
-- Expect: "Index Scan using items_embedding_hnsw ...", not "Seq Scan"

If the plan shows Index Scan using items_embedding_hnsw, the next scrape must show a higher pg_vector_index_usage_idx_scan for that index. A plan that falls back to Seq Scan explains a permanently flat counter and points you toward statistics or hnsw.ef_search tuning covered in optimizing m and ef_construction parameters.

Prove the histogram reflects true latency. Issue a known-slow query (cold cache, or a deliberately high k) and confirm it lands in the expected bucket. A _count that rises without the matching _sum moving means the timer scope is wrong — a frequent bug when the .time() context is placed around the wrong call.

Prove the recall Gauge tracks reality. Deliberately degrade recall by lowering hnsw.ef_search and confirm pgvector_recall_at_10 drops on the next refresh. A recall Gauge that never moves is worse than no Gauge — it implies safety that is not being measured. The full ground-truth methodology, including forcing an exact brute-force scan for the reference set, lives in the sibling page on recall regression testing in CI.

Finally, cross-validate the exporter’s cache-hit ratio against pg_statio directly during a load test; if the exporter reports 0.99 while a cold benchmark shows disk-bound latency, the collector is reading a warmed connection’s stats rather than the workload’s.

Failure Modes & Gotchas #

• High-cardinality label explosion. Adding a per-query, per-user, or per-vector label to SEARCH_LATENCY mints a fresh series for every distinct value. Prometheus holds all active series in memory, so an unbounded label OOM-kills the scrape target’s exposition and then the Prometheus server. Constrain labels to a bounded set — index name and k — and validate with count({__name__="pgvector_search_seconds_count"}).
• Scrape interval versus counter resets. idx_scan and idx_blks_hit are cumulative counters that reset to zero on a PostgreSQL restart or an explicit pg_stat_reset(). Always graph them through rate() or increase(), which are reset-aware; a raw counter difference across a restart produces a huge negative spike or a fake plunge to zero.
• Missing shared_preload_libraries='pg_stat_statements'. CREATE EXTENSION will succeed against a running server, but the view stays empty because the shared-memory hooks are only installed at startup. Per-statement metrics then silently return nothing. Confirm with SHOW shared_preload_libraries; and restart if it is absent.
• The database view cannot measure client-perceived latency. pg_stat_* and pg_stat_statements both stop at the server boundary — they exclude connection acquisition, network, and serialization. If your only latency source is the exporter, you will report healthy while users time out. The application histogram is not optional for an SLA.
• Buckets that miss the tail. A histogram whose largest finite bucket is below your real p99 collapses everything above it into +Inf, and histogram_quantile extrapolates garbage. Size the top bucket above your worst realistic latency, and revisit it whenever ef_search or corpus size changes.
• Exporter connection overhead. Heavy collectors run on every scrape; against a busy primary this adds load and can itself perturb the metrics. Use cache_seconds to throttle expensive catalog scans and prefer scraping a replica for pg_stat_user_* where the workload permits.

Monitoring & Alerting Hooks #

The series above become actionable through a handful of PromQL expressions and the alert rules built on them. The p95 latency query uses the histogram’s _bucket series and must aggregate by le before applying the quantile:

# p95 client-perceived ANN latency, per index
histogram_quantile(
  0.95,
  sum(rate(pgvector_search_seconds_bucket[5m])) by (le, index)
)

# Index-scan rate — falling to zero means the planner stopped using the index
rate(pg_vector_index_usage_idx_scan{indexrelname="items_embedding_hnsw"}[5m])

# Index cache-hit ratio from the I/O counters
sum(rate(pg_vector_index_io_idx_blks_hit[5m]))
/
(sum(rate(pg_vector_index_io_idx_blks_hit[5m]))
 + sum(rate(pg_vector_index_io_idx_blks_read[5m])))

Wire these into Alertmanager rules that fire before users complain. The three that catch the most incidents are an SLA breach on p95, an index that has gone cold (scan rate at zero while traffic continues), and a cache-hit ratio collapse that presages disk-bound latency:

groups:
  - name: pgvector_search
    rules:
      - alert: PgvectorSearchLatencyHigh
        expr: >
          histogram_quantile(0.95,
            sum(rate(pgvector_search_seconds_bucket[5m])) by (le)) > 0.15
        for: 10m
        labels: {severity: page}
        annotations:
          summary: "pgvector p95 search latency above 150ms SLA"

      - alert: PgvectorIndexUnused
        expr: >
          rate(pg_vector_index_usage_idx_scan{indexrelname=~".+hnsw.+"}[15m]) == 0
        for: 15m
        labels: {severity: warning}
        annotations:
          summary: "HNSW index scan rate is zero — planner likely fell back to Seq Scan"

      - alert: PgvectorIndexCacheHitLow
        expr: >
          sum(rate(pg_vector_index_io_idx_blks_hit[10m]))
          / (sum(rate(pg_vector_index_io_idx_blks_hit[10m]))
             + sum(rate(pg_vector_index_io_idx_blks_read[10m]))) < 0.95
        for: 20m
        labels: {severity: warning}
        annotations:
          summary: "Vector index cache-hit ratio below 95% — index no longer resident"

      - alert: PgvectorRecallRegressed
        expr: pgvector_recall_at_10 < 0.95
        for: 5m
        labels: {severity: page}
        annotations:
          summary: "Measured recall@10 fell below 0.95 threshold"

The for: clauses matter: a single slow scrape or a momentary cache miss should not page. Pair the recall alert here with an offline gate so regressions are caught before deploy, not just after — the CI approach in recall regression testing in CI blocks the merge, while this rule catches drift that only appears under production data growth. For raw index-usage and bloat signals feeding the same Grafana boards, the sibling page on pg_stat monitoring for vector indexes covers the catalog views in depth.

FAQ #

Related #

• pg_stat monitoring for vector indexes — the catalog views behind the exporter’s index-usage and bloat metrics
• Recall regression testing in CI — how the recall Gauge is measured against brute-force ground truth
• Optimizing m and ef_construction parameters — the tuning that latency and recall metrics steer
• HNSW vs IVFFlat algorithm selection — index choice that determines which access-method counters to scrape
• Up: Monitoring, Observability & Operations