Practice · 2024–now · AI Present
AI-generated test theater
Coverage theater with a chat box. Tests that mirror the bug and never catch it.
AI test-generation theater failed because coverage without behavioral contracts is decoration. Generated tests stick when humans specify invariants and review assertions. The fad was green dashboards from mirrored implementation; the stuck practice is property tests and parity checks owned by people.
Cost of the fad
LLMs wrote tests that boosted coverage and asserted nothing. Green CI, false confidence — eval theater's cousin in the test suite.
Case studies
Patterns
Context
Demos consolidate; evals remain
AI pair programming changed how code is typed faster than how it is reviewed. Agent frameworks and standalone vector stores sorted into demos versus durable plumbing; mid-market RAG folded back into Postgres. The permanent layer is familiar: evals that gate deploys, model routing for cost, tool protocols instead of plugin snowflakes, and humans who own production. Autopilot rewrites and vibe-shipped auth middleware are still big-bang migrations with better slides — and a longer on-call.
Compare with
Related
Practice · 2024–now
AI review bottleneck
Copilot ships PRs faster than teams can review. Generation outran ownership and security review — the real 2026 invoice.
Practice · 2024–now
Vibe coding
Accept AI output on feel, skip the diff, hope tests exist. Fast demos; CVEs and nobody owns the middleware.
$ Ship-to-prod without reading the diff became a flex until incidents revealed nobody knew which prompt wrote the auth middleware. Review debt arrived in one weekend demo.
Practice · 2024–now
Eval theater
LLM evals as slideware — metrics that look scientific and never gate a deploy. The foil to real eval pipelines in CI.
$ Dashboards of vibe-check scores that never blocked a release. Green charts for leadership; prod still hallucinated.
Practice · 2021–now
AI pair programming
Autocomplete that understands the file — now table stakes. Generation outran review capacity; ownership is the bottleneck.