Practice · 2024–now · AI Present
Eval theater
LLM evals as slideware — metrics that look scientific and never gate a deploy. The foil to real eval pipelines in CI.
Eval theater failed because measurement without enforcement is theater. Real LLM evals stuck when they lived in CI with owners. The fad was buying an eval product to decorate a demo; the stuck practice is regression tests that can fail the build.
Cost of the fad
Dashboards of vibe-check scores that never blocked a release. Green charts for leadership; prod still hallucinated.
Patterns
Context
Demos consolidate; evals remain
AI pair programming changed how code is typed faster than how it is reviewed. Agent frameworks and standalone vector stores sorted into demos versus durable plumbing; mid-market RAG folded back into Postgres. The permanent layer is familiar: evals that gate deploys, model routing for cost, tool protocols instead of plugin snowflakes, and humans who own production. Autopilot rewrites and vibe-shipped auth middleware are still big-bang migrations with better slides — and a longer on-call.
Compare with
Related
Practice · 2023–now
LLM eval pipelines
Regression tests for nondeterministic models that actually fail the build. The unglamorous CI that separates demos from products.
Practice · 2023–now
Agent ops / LLM observability
Tracing, cost caps, and prompt versioning for production LLM features — mostly constrained tool loops, not autonomous agents. Datadog for tokens.
Practice · 2023–now
RAG as default architecture
Retrieval-augmented generation as the answer to every knowledge problem — then "just stuff the window" as the counter-fad. Often right; often Postgres with pgvector would suffice.
$ Teams stood up vector pipelines, chunking strategies, and rerankers before asking if fine-tuning or a SQL query would answer the question. Retrieval infra became the product.