Practice · 2023–now · AI Present
LLM eval pipelines
Regression tests for nondeterministic models that actually fail the build. The unglamorous CI that separates demos from products.
LLM evals stuck because shipping prompts without measurement is shipping bugs with confidence intervals. Early adopters treated eyeball checks as QA; by 2026 enforcement in CI is the difference from eval theater. What remains is boring gates for model behavior — and that is the point.
Context
Demos consolidate; evals remain
AI pair programming changed how code is typed faster than how it is reviewed. Agent frameworks and standalone vector stores sorted into demos versus durable plumbing; mid-market RAG folded back into Postgres. The permanent layer is familiar: evals that gate deploys, model routing for cost, tool protocols instead of plugin snowflakes, and humans who own production. Autopilot rewrites and vibe-shipped auth middleware are still big-bang migrations with better slides — and a longer on-call.
Compare with
Related
Practice · 2023–now
RAG as default architecture
Retrieval-augmented generation as the answer to every knowledge problem — then "just stuff the window" as the counter-fad. Often right; often Postgres with pgvector would suffice.
$ Teams stood up vector pipelines, chunking strategies, and rerankers before asking if fine-tuning or a SQL query would answer the question. Retrieval infra became the product.
Framework · 2023–now
LLM app frameworks
LangChain-class glue gave way to vendor SDKs and thin wrappers. The durable pieces are boring: evals, retrieval, and product UX.
Practice · 2023–now
Agent ops / LLM observability
Tracing, cost caps, and prompt versioning for production LLM features — mostly constrained tool loops, not autonomous agents. Datadog for tokens.
Practice · 2024–now
Eval theater
LLM evals as slideware — metrics that look scientific and never gate a deploy. The foil to real eval pipelines in CI.
$ Dashboards of vibe-check scores that never blocked a release. Green charts for leadership; prod still hallucinated.