Architecture · 2024–now · AI Present
Model routing / tiered inference
Cheap model drafts; frontier model escalates. The boring cost-control layer that survived the agent hype.
Model routing stuck because token economics and latency budgets are product constraints, not afterthoughts. It failed when slides replaced product design — routing diagrams without quality gates. What remains is tiered inference with evals; what faded is one-model-fits-all keynotes.
Patterns
Context
Demos consolidate; evals remain
AI pair programming changed how code is typed faster than how it is reviewed. Agent frameworks and standalone vector stores sorted into demos versus durable plumbing; mid-market RAG folded back into Postgres. The permanent layer is familiar: evals that gate deploys, model routing for cost, tool protocols instead of plugin snowflakes, and humans who own production. Autopilot rewrites and vibe-shipped auth middleware are still big-bang migrations with better slides — and a longer on-call.
Compare with
Related
Practice · 2023–now
Agent ops / LLM observability
Tracing, cost caps, and prompt versioning for production LLM features — mostly constrained tool loops, not autonomous agents. Datadog for tokens.
Practice · 2023–now
LLM eval pipelines
Regression tests for nondeterministic models that actually fail the build. The unglamorous CI that separates demos from products.
Practice · 2024–now
Reasoning models for grep tasks
o1-class deliberation for tickets that needed a filter. Resume-driven inference.
$ Thinking tokens burned on problems a SQL query and a unit test would settle. Latency and invoices grew; correctness did not.
Framework · 2023–now
LLM app frameworks
LangChain-class glue gave way to vendor SDKs and thin wrappers. The durable pieces are boring: evals, retrieval, and product UX.