Handling Metadata Drift During Vector Ingestion

This page shows how to stop upstream schema, type, and semantic changes from silently corrupting the metadata that rides alongside your embeddings in pgvector. It walks a concrete ingestion-boundary defense — versioned contracts, pre-embedding validation, deterministic coercion, per-revision indexing, and distribution monitoring — so drift is caught and quarantined before it degrades hybrid-filter recall.

Up: Metadata Mapping & Schema Design

Metadata drift appears when a source system changes a field’s type, structure, or meaning without a synchronized change to the downstream embedding ingestion pipeline. The vectors keep flowing and inserts keep succeeding, so nothing errors — but hybrid queries that filter on metadata->>'category' quietly stop matching, partial indexes fall out of use, and recall decays in a way that only surfaces in a post-mortem. The fix is to treat metadata as a versioned contract enforced at the exact point where a record crosses into the pipeline, before it consumes embedding compute or reaches an ANN index.

Prerequisites #

• PostgreSQL 15+ with the pgvector 0.5+ extension installed (CREATE EXTENSION vector;)
• A vector table whose auxiliary attributes live in a jsonb column plus a small set of typed columns (the hybrid layout described in metadata mapping and schema design)
• A metadata_schema_version (or schema_revision) integer column on that table to route records by contract version
• Pydantic v2 (or jsonschema) available in the ingestion worker for synchronous validation
• A dead-letter queue (DLQ) or quarantine table for records that fail validation
• An observability sink (OpenTelemetry, Prometheus, or Datadog) to receive structured drift events

Step-by-Step Procedure #

1. Classify the drift before changing the pipeline #

Drift falls into three classes, and each demands a different response, so name the class first. Type coercion — an upstream service silently sends "42" where it once sent 42, or 1 where it sent true; PostgreSQL’s jsonb preserves the distinction, but lossy Python casting downstream changes filter results. Structural mutation — keys are added, removed, or nested, so deterministic field extraction hits fallback defaults. Semantic shift — a field is repurposed without aliasing, e.g. product_category moves from a single string to an array of tags, and equality filters silently miss every row while vector similarity looks fine.

Audit the live table shape against what the contract expects before trusting any incoming batch:

SELECT column_name,
       data_type,
       is_nullable,
       ordinal_position
FROM   information_schema.columns
WHERE  table_name = 'document_vectors'
ORDER  BY ordinal_position;

For fields buried in jsonb, sample the observed key types so a semantic shift shows up as a mixed-type histogram rather than a surprise at query time:

SELECT jsonb_typeof(metadata -> 'product_category') AS observed_type,
       count(*)                                      AS n
FROM   document_vectors
GROUP  BY 1
ORDER  BY n DESC;

2. Establish a versioned metadata contract #

Define an explicit schema for every metadata field and pin it to an integer revision stored beside the row. Adopt backward-compatible evolution rules: additive-only changes within a revision, a deprecation window when semantics change (for example v2 accepts v1 aliases for 90 days), and a mandatory alias whenever a field is repurposed. Persist the revision so ingestion and queries can route deterministically instead of silently falling back.

ALTER TABLE document_vectors
    ADD COLUMN IF NOT EXISTS metadata_schema_version integer NOT NULL DEFAULT 1;

-- Reject rows that arrive without a known contract version
ALTER TABLE document_vectors
    ADD CONSTRAINT chk_schema_version
    CHECK (metadata_schema_version BETWEEN 1 AND 3);

3. Validate at the ingestion boundary before spending embedding compute #

Run a synchronous validator inside the ingestion worker so a non-conforming payload is rejected before it consumes an embedding API call or GPU forward pass. This is the same defensive posture used across the pipeline in async processing with Python asyncio, where validation belongs in the earliest stage of the event loop rather than after fan-out.

from pydantic import BaseModel, ValidationError, field_validator
from typing import Optional

class VectorMetadataV2(BaseModel):
    doc_id: str
    category: str
    tags: list[str] = []
    confidence: float = 0.0
    schema_version: int = 2

    @field_validator("confidence")
    @classmethod
    def clamp_confidence(cls, v: float) -> float:
        return max(0.0, min(1.0, v))

def validate_payload(raw: dict) -> Optional[VectorMetadataV2]:
    try:
        return VectorMetadataV2.model_validate(raw)
    except ValidationError as exc:
        route_to_dlq(raw, errors=exc.errors())  # structured, replayable
        return None

4. Coerce recoverable drift deterministically, quarantine the rest #

Recoverable drift (a stringified number, a missing optional key) should be coerced to an explicit, documented sentinel; unrecoverable drift (a semantic shift, a required field gone) goes to the DLQ. The rule that matters: never let coercion silently alter the values used in hybrid scoring, because a shifted numeric distribution changes ANN ranking without any error. Where a field feeds a distance-weighted score, normalize it on the same terms you use when normalizing embeddings before pgvector insertion.

def coerce_v1_to_v2(raw: dict) -> dict:
    out = dict(raw)
    # string -> single-element tag array (semantic shift, but aliasable)
    cat = out.get("product_category")
    if isinstance(cat, str):
        out["tags"] = [cat]
    out.setdefault("tags", [])
    out.setdefault("confidence", 0.0)
    out["schema_version"] = 2
    log_drift_event(field_path="product_category",
                    expected_type="array", observed_type="string",
                    severity="coerced")
    return out

Emit each drift event as structured telemetry (source_version, field_path, expected_type, observed_type, severity) so a rising coercion rate becomes an alert rather than a mystery.

5. Version the table schema and build a per-revision partial index #

When a new contract ships, keep the query planner efficient by giving the latest revision its own partial HNSW index. Routing reads to the current revision means the planner scans a tighter graph and does not mix incompatible metadata semantics across versions.

CREATE INDEX CONCURRENTLY idx_vectors_hnsw_v2
    ON document_vectors
    USING hnsw (embedding vector_cosine_ops)
    WHERE metadata_schema_version = 2;

Match m and ef_construction on that index to your recall target using optimizing m and ef_construction parameters; a per-revision index is where a mistuned build shows up first.

6. Add an expression or GIN index for fields that changed shape #

A field that drifted from string to array needs a containment index or the filter falls back to a sequential scan. Index the normalized path so hybrid queries stay index-driven:

CREATE INDEX CONCURRENTLY idx_vectors_tags_gin
    ON document_vectors
    USING gin ((metadata -> 'tags') jsonb_path_ops);

Use ->> consistently for text extraction and -> for JSON, and never mix them in application code — a stray -> where the planner expected ->> is enough to skip the index. See PostgreSQL’s JSON and JSONB data types documentation for the operator and casting semantics that preserve precision here.

7. Monitor distribution drift continuously #

Structural validation catches shape changes; it does not catch a field whose values slowly shift. Compute the Population Stability Index (PSI) for categorical fields and Kullback–Leibler (KL) divergence for numeric ones against a stored baseline, and alert on threshold breaches before the drift reaches production queries.

import numpy as np

def population_stability_index(baseline: np.ndarray, current: np.ndarray) -> float:
    eps = 1e-6
    b = np.clip(baseline / baseline.sum(), eps, None)
    c = np.clip(current / current.sum(), eps, None)
    return float(np.sum((c - b) * np.log(c / b)))

# Alert when PSI > 0.25 (categorical) or KL > 0.10 (numeric)

Parameter Reference #

Verification #

Confirm the contract is enforced, the partial index is chosen, and no drifted rows slipped through unversioned:

-- 1. No row escaped without a known contract version
SELECT count(*) AS unversioned
FROM   document_vectors
WHERE  metadata_schema_version IS NULL
   OR  metadata_schema_version NOT IN (1, 2, 3);   -- expect 0

-- 2. The per-revision partial index is actually used, not a seq scan
EXPLAIN (ANALYZE, BUFFERS)
SELECT id
FROM   document_vectors
WHERE  metadata_schema_version = 2
  AND  metadata @> '{"tags": ["billing"]}'
ORDER  BY embedding <=> '[...]'::vector
LIMIT  10;   -- expect a Bitmap Index Scan feeding an Index Scan, no Seq Scan

# 3. Drift monitor is emitting events (should be non-empty during a real batch)
assert drift_events_last_hour() >= 0        # sink reachable
assert coercion_rate() < 0.05               # < 5% coerced, else investigate source

Troubleshooting #

• Filters return zero rows but similarity still ranks results. Classic semantic shift — a field moved from scalar to array. Confirm with the jsonb_typeof histogram from step 1, add the jsonb_path_ops GIN index from step 6, and rewrite the filter to containment (metadata @> '{"tags": ["x"]}') instead of equality.
• Query falls back to a sequential scan after a schema change. The partial index predicate no longer matches the query’s metadata_schema_version, or statistics are stale. Run ANALYZE document_vectors;, then re-check EXPLAIN (ANALYZE) for an Index Scan; recreate the partial index for the new revision if the predicate drifted.
• invalid input syntax for type vector or NaN on insert during reprocessing. A coerced numeric field overflowed or a zero-magnitude vector reached the write path. Clamp with the epsilon guard used when normalizing, and validate metadata types before the embedding step rather than after.
• Recall silently degrades with no errors. Value drift the structural validator can’t see. Check the PSI/KL monitor from step 7 (PSI > 0.25 or KL > 0.10), snapshot the metadata distribution per batch, and compare against the stored baseline to locate the batch where it moved.
• DLQ fills faster than it drains. Coercion rules lag the source. Inspect the structured field_path/observed_type on quarantined records, extend coerce_v1_to_v2 to alias the new shape, bump the contract revision, and replay the DLQ through the updated validator.

FAQ #

Related #

• Normalizing Embeddings Before pgvector Insertion — deterministic value handling on the write path that drift monitoring depends on.
• Building a Resilient Python Embedding Pipeline with Celery — the dead-letter and reconciliation path that replays quarantined, drifted payloads.
• Metadata Mapping & Schema Design — the hybrid typed-plus-jsonb contract these versioned checks enforce.

Up: Metadata Mapping & Schema Design