When cheap measurement overturned 2.5 years of beliefs about an AI product

Background

TrendVesting started as a scheduled equity trader and pivoted through options, social trading, a multi-model “room of experts,” and prediction-market fair value. For most of its life I was the only engineer, and commit messages like “fix it” were the norm.

What changed

In mid-2026 I started pairing with Claude Code on most of the work. Velocity went up, but the bigger change was that measurement got cheap. An agent will happily write the reconciliation query, the calibration script, and a commit message explaining the result.

What the measurements said

• Confidence was decorative. Across 573 resolved signals, the win rate stayed flat at about 34% no matter what confidence threshold you filtered on. Stated confidence ran about 55 points above reality.
• The ledger was polluted. A reconciler bug had created 9,655 phantom closed option trades in the tracking data, swamping the 510 real ones. Separately, 114 losing exits had been recorded as target hits.
• The “edge” was a stale feed. The fair-value model lost to the market on log-loss (0.343 for the market vs 0.493 for the model), and an earlier apparent edge came from a frozen spot-price feed.
• Flags aren’t facts. A weather-market screen trusted a flag that was only ever set on one pod, so zero training pairs were collected.

Some hit rates I’d written about earlier didn’t survive this. I’d rather publish the correction than keep the flattering number.

Cost controls that held up

The cheapest model for re-evaluation, LLM calls only when a cheap deterministic check trips, a daily throttle that survives restarts, caches that show their age, and turning off a paid data refresh (about $17 a month) that wasn’t earning its cost.

The consulting lesson

Most teams’ AI quality and cost numbers are unverified in exactly this way. The first engagement is usually just making the numbers true.