Practice · 2024–now · AI Present
Eval theater
LLM evals as slideware — metrics that look scientific and never gate a deploy. The foil to real eval pipelines in CI.
Why Eval theater became a costly fad
Eval theater failed because measurement without enforcement is theater. Real LLM evals stuck when they lived in CI with owners. The fad was buying an eval product to decorate a demo; the stuck practice is regression tests that can fail the build.
Cost of the fad
What Eval theater cost
Dashboards of vibe-check scores that never blocked a release. Green charts for leadership; prod still hallucinated.
Patterns
Compare with
Related
Practice · 2023–now
LLM eval pipelines
Regression tests for nondeterministic models that actually fail the build. The unglamorous CI that separates demos from products.
Practice · 2023–now
Agent ops / LLM observability
Tracing, cost caps, and prompt versioning for production LLM features — mostly constrained tool loops, not autonomous agents. Datadog for tokens.
Practice · 2023–now
RAG as default architecture
Retrieval-augmented generation as the answer to every knowledge problem — then "just stuff the window" as the counter-fad. Often right; often Postgres with pgvector would suffice.
$ Teams stood up vector pipelines, chunking strategies, and rerankers before asking if fine-tuning or a SQL query would answer the question. Retrieval infra became the product.