BigQuery proves your data is fresh, complete, unique, and structurally valid. None of that catches the pipeline that silently drifts in meaning. Semantic Drift Sentinel treats semantic quality as a continuous infrastructure metric, with a circuit breaker at the gate.
Here is the same support-ticket table after two deployments of the upstream pipeline. An "improved" prompt in v2 silently regressed: the resolution summaries stopped actually addressing the tickets. Both versions pass identically.
"I was charged twice for October."
→ "Confirmed the duplicate charge and issued a refund; posts in 3-5 days."
"I was charged twice for October."
→ "Thanks for reaching out. You can view invoices in Account Settings."
The whole point: at scale the model call is the easy part. The hard part is the engineering boundary around it, concurrency throttling, retries, idempotency, and absolute cost control. Keep the expensive semantic check for only the rows that need it.
Run the cheap, deterministic checks first (freshness, completeness, schema, uniqueness). Route only high-risk or sampled rows out of the warehouse, never pay to semantically score everything.
An asynchronous Python service manages batching, exponential-backoff retries, idempotency, and a strict concurrency cap, the real engineering so a spike never melts the budget or trips rate limits.
Batched rows go to the evaluation layer for model inference and strict structural JSON scoring. Every answer validates against a typed contract (Noul / Choice / Score with calibrated probabilities), no free text to parse.
Apply thresholds at the gate. If semantic quality drops too far versus the baseline, or below an absolute floor, instantly trip: fire a downstream alert and route anomalies to a human review queue.
Downstream pipeline alert (Slack / PagerDuty)
Worst rows → human review queue
Continuously write versioned quality metrics back into BigQuery to trace and track long-term semantic drift by deployment, turning a one-off check into a monitorable time series.
Full diagram: docs/architecture.png
Real output from make demo-tickets on the support-ticket dataset. v1 is the healthy baseline; v2 is the silently-drifted deployment that passed all deterministic checks.
Reason: semantic drift 0.59 > 0.15; mean quality 0.37 < 0.70 floor.
The core question, answered automatically on every deployment:
Did this deployment preserve the meaning of our data, and can we prove it at scale?
─── 1 · BigQuery · deterministic SQL checks ─── v1 completeness PASS schema PASS uniqueness PASS freshness PASS v2 completeness PASS schema PASS uniqueness PASS freshness PASS ← all green ─── 2-3 · Cloud Run async adapter → Jev typed eval ─── → routing 13 v2 rows (backend=jev, concurrency=8) TCK-001 addresses_ticket 0.02 resolution 1.0/5 quality 0.19 TCK-004 addresses_ticket 0.98 resolution 5.0/5 quality 0.97 ─── 4 · data circuit breaker ─── 🚨 CIRCUIT BREAKER TRIPPED (OPEN) mean quality 0.374 drift vs base +0.591 actions fired pipeline alert · routed 3 rows to human review ─── operational metrics ─── mean quality · v1 0.966 mean quality · v2 0.374 semantic drift (v1→v2) +0.591 human vs evaluator disagreement 12%
The pipeline is generic over its data (datasets.py); pick one with DATASET=tickets|goods. In both, every deterministic check stays green and only the semantic evaluator catches the regression.
v2 resolution summaries silently stop addressing the ticket.
v1 "Confirmed the duplicate charge and issued a refund."
v2 "You can view invoices in Account Settings."
make demo-tickets
v2 silently miscategorises products into a different valid category; the string is fine, the meaning is wrong.
iPhone 15 Pro
Mobile → Home Appliances
make demo-goods
─── 1 · BigQuery · deterministic SQL checks ─── v1 completeness PASS schema PASS uniqueness PASS freshness PASS v2 completeness PASS schema PASS uniqueness PASS freshness PASS ← Home Appliances is a valid category ─── 2-3 · Cloud Run async adapter → Jev typed eval ─── → routing 12 v2 rows (backend=jev, concurrency=8) SKU-000 iPhone 15 Pro category_ok 0.01 catalog 1.0/5 quality 0.19 SKU-009 Dyson V15 Vacuum category_ok 0.02 catalog 1.0/5 quality 0.19 ─── 4 · data circuit breaker ─── 🚨 CIRCUIT BREAKER TRIPPED (OPEN) mean quality 0.223 drift vs base +0.698 actions fired pipeline alert · routed 3 rows to human review ─── operational metrics ─── mean quality · v1 0.921 mean quality · v2 0.223 semantic drift (v1→v2) +0.698
Jev returns answers that validate against a typed schema, so the pipeline never parses free text. The boundary in schemas.py is deliberately backend-agnostic, so the evaluator can be swapped without the rest of the pipeline moving.
The real TypeSafe AI "System One" model, the production path in the diagram. Purpose-built typed decisions with calibrated probabilities.
The headline signal, quality per deployment, over time.
Is the judge itself trustworthy? Measure it.
The real cost of catching drift, p50 / p95.
Proven, not assumed, tracked per row and per run.
# clone git clone https://github.com/\ Tanishque99/semantic-drift-sentinel cd semantic-drift-sentinel # install deps make install # set your key (or put it in .env) export TYPESAFE_API_KEY=ts-... # run the full loop (Jev backend) make demo
Run the adapter as a service and deploy the real topology:
# the Jev backend, explicitly export TYPESAFE_API_KEY=ts-... make jev # Cloud Run adapter, locally make serve # FastAPI on :8080
Deploying the real topology (Cloud Run + BigQuery) is in deploy/README.md; the SQL in bigquery/ maps 1:1 onto the local DuckDB version.