TrendVesting: validating 2.5 years of production AI

TrendVesting started on February 14, 2024 as a cron job that bought stocks through Alpaca. Today it is a multi-mode Go backend, a Next.js app, an Expo mobile app, and an MCP server with 48 tools, running on DigitalOcean Kubernetes. It has been an options bot, an AI content generator, a social trading platform, and, this month, a scanner for prediction markets and weather contracts.

I own it. I pay the invoice, I approve what ships, and when a dashboard says a strategy is working, I am the person who is supposed to know whether that is true. For most of its life, I didn’t. This post is about how I changed that: what I decided to check, how I review changes now, and what the checks found. Several findings overturned numbers I had been quoting, including on this site.

Three ways I have managed the same repo

The git log covers 31 months and 1,859 non-merge commits. The useful way to split it is by how the work was checked.

• Feb 2024 to May 2025: shipping on instinct. 1,046 commits. 4.9% touched a test file and not one had a commit body. If it compiled and the screen looked right, it shipped, often under the message fix it.
• Jun 2025 to Jun 2026: reviewing pull requests. 502 commits. Work started arriving as pull requests, including from early coding agents like Codex and Copilot, and my job shifted toward reading diffs. The share of changes with a test barely moved, to 8.4%. A diff tells you what changed. It does not tell you whether the result is right.
• Jul 2026 to now: measure, then change. 311 commits. Earlier in 2026 I started pairing with Claude Code, and over the summer I settled on one rule for every change, whoever or whatever typed it: the commit has to explain itself with evidence. Since July, 43% of commits change a test, and the Go suite has gone from 99 test functions to 540.

What I require a change to carry

Every commit body now answers four questions, and I read it the way I would read a short design doc from someone on my team:

1. What is the root cause? Not the symptom.
2. What do the production numbers say? A count, a duration, a row total. If a claim has no number, I ask for one.
3. What does this deliberately not do? Scope is where risky changes hide.
4. How was it verified? Which test, which query, which dry run.

A typical one is 01871ae Stop the options reconciler duplicating closed trades, and defend the read path : about 30 lines, ending with “Verified: go build passes, analytics tests pass including new coverage built from the real TLT and USO rows.”

Beyond whether the code reads well, I look for three things. A number that came from production, not a guess. A test that pins the boundary, not the constant: ccf56f5 raise the confidence bar from 70 to 75 changes one number, but its test checks that 75 passes and 74.9 does not, because “an off-by-one there would silently re-admit the band this change exists to exclude.” And a hard line between fixing code and changing data.

The six code fixes in this post from September 9 to 12 added 40 test functions between them. Here is what the checks found.

Check 1: reconcile the ledger against the broker

The data-integrity week started with a small question: had one options agent recorded a May 26 exit? The answer, from the commit: “It is recorded about 170 times.”

The reconciler creates a trade row when the broker reports a position it cannot match. Its duplicate guard only looked for status = "open" trades, while the comment above it said “even closed ones.” Once a trade closed, the position looked orphaned again, and the reconciler minted another closed row, roughly once a minute. Across the options agents: 9,655 closed rows for 312 signals, summing to −$4,836,144 in realized P&L.

The first commit fixed the writer and added a read-side repair. It deliberately deleted nothing: “Deleting trade records is destructive and belongs in a separate, explicitly approved step.”

That step came the same day, as dc39bf1 migration removing the phantom duplicate option trades , and it is the best argument I have for dry runs. The first version kept one row per agent, signal and symbol. Run read-only, it would have deleted 71 rows with a real exit order: legitimate partial exits and re-entries. The applied rule keys on the order id instead. Every deleted row was archived first, so a restore is one INSERT ... SELECT, and a post-check confirmed no group kept more than one order-less closed row. Closed option rows went from 9,655 to 561.

Check 2: make labels unable to change numbers

Two days later: 685b5bf stop recording losing option exits as target hits .

Two code paths close an options trade. One refused to label a losing close a “target” hit. The other, used when a position vanishes and the exit order is looked up after the fact, labelled any filled exit order a target hit. A stop-loss is also an exit order, and in practice it was usually the one that filled.

Result: 114 rows and −$60,155 of losses recorded as target hits, from April 27 to August 13, 2026. The commit states the stakes: “A strategy that reports its stop-outs as target hits cannot be evaluated: win rate, average win, payoff ratio and expectancy are all wrong, and wrong in the flattering direction.” Its tests use the real rows, plus guards against over-correcting in the other direction.

The data fix again came separately ( a8cb276 backfill option exits mislabelled as target hits ): 114 rows corrected, zero profitable rows touched, original labels preserved in their own column so the correction is auditable and reversible.

A follow-up, 56d1888 stop deriving a trade’s exit price from its exit label , removed code that would have valued a “target” trade at its target price whenever the real fill was missing. It never fired. It was “one missing fill away from firing.” It is now an invariant test: for the same prices, every exit reason must produce the same exit price.

Check 3: join stated confidence to outcomes

A signal gets created when the model’s stated confidence clears a bar. On September 10, I approved raising that bar from 70 to 75 because “the model now routinely scores in the 80s.”

The next day we checked whether confidence meant anything. Both facts had been stored for months, the confidence in one table and the eventual win or stop in another. Nothing had ever joined them. 47f21e0 check whether model confidence predicts a win did, against 573 resolved signals in production. It separates two questions that are easy to blur: discrimination (do higher scores win more, which a threshold needs) and calibration (does 85% mean 85%, which position sizing needs).

It failed both. Win rate is flat at every bar, and stated confidence runs 54.7 points above the realized rate. The bar I had approved raising the day before was a knob connected to nothing.

The first version of the tool failed in an instructive way. It split signals at the median confidence, but 62% of them sit in a single 90 to 95 bucket, so the split degenerated and the tool reported “not enough data to say” on 573 signals. As the commit says: “It read as a sample-size problem and was a logic bug.” Had I accepted the first output, I would have decided to wait for more data.

Check 4: score forecasts against a baseline

Version 1 of a fair-value model for Kalshi crypto contracts looked like it had an edge. The edge came from “one bull afternoon plus a stale spot feed”: BTC spot came from a candles endpoint that sat frozen. In a 24-hour paper sample, contracts it bought YES on resolved YES only about 18% of the time (33 of 187).

After fixing the feed, the deciding check was a proper scoring rule. Log-loss punishes confident wrong forecasts, and lower is better. Over about 4,200 settled observations, the market scored 0.343 and the model 0.493. The market was the better forecaster, and the strategy doc was rewritten to lead with that ( 4539be0 market-is-efficient finding ). This is not trading advice. “The model has an edge” and “the model is reading a frozen number” look identical on a dashboard until someone scores it against the obvious baseline.

Check 5: derive status from data, not flags

The weather work produced my favorite commit title of the year: 31e6253 collect the settlement side, and stop reporting a flag as a fact .

The admin screen said weather collection was disabled. In fact the archive was growing normally: 294 forecasts, 127 readings, last write minutes old. The status endpoint ran in two processes and read an environment flag from whichever one answered, and the API pod never had the flag.

The second fault was worse. We archived forecasts but never the observed highs, so usable training pairs “numbered zero and would have stayed zero however many days accrued.” Nothing alerted, because “every check watched for calls that failed, not for a series nobody asked for.”

Status is now derived from the archive. Readiness reports the minimum across gates, because “300 forecasts and no actuals is 0% complete, and an average would have rendered that as half done.” And all 48 weather rules stay quarantined on a station-id mismatch rather than loosening the gate “on our own reading of the contract text.” Readiness flags it for a human.

The numbers I published without validating

Until this week, this site quoted TrendVesting at ”~$1.50–2.00 fully loaded per signal.” That number comes from a cost ledger I drafted in July 2026. Here is how it was built, and what survives this month.

The monthly lines came from invoices and dashboards, plus GitHub and Copilot at about $74 and managed Postgres at about $15. One 30-day OpenAI window showed $59.17 for 117.9M tokens across 43,470 requests.

What survives: the shape of the bill. The cluster alone costs more than the tokens. The expensive part of production AI is the production part.

What does not:

1. That same draft said monthly hit rates were 73% in April, 63% in May and 66% in June. I cannot reproduce those from anything in the repo. The first real join of signals to outcomes found about 34% across 573 resolved signals. Definitions could account for some of the gap. Either way, I published a quality number I had never validated the way I now require.
2. The per-signal math divided the bill by “a few hundred tracked signals per month.” That was an estimate, not a count.
3. Every P&L-based view of quality in that period sat on a table with 9,655 duplicate rows in it.

So $1.50 to $2.00 is, at best, a rough cost per signal generated. It says nothing about cost per signal worth acting on. As arithmetic, not a measurement: if about a third of signals win, the cost per winning signal is roughly three times higher, before anyone asks whether the wins pay for the losses.

The cost fixes held up better, because each had to show its number before I approved it. The LLM re-evaluator now runs only when a cheap price, sentiment or time check trips, on gpt-5-nano instead of gpt-5-mini ( fd6f4f8 signal-watcher re-evaluation mode ). A “once a day” throttle that reset on every deploy now lives in the database ( c875f3a throttle tax-estimation LLM to once/day ). A paid refresh button used about 340 times a month, roughly $17, is off by default ( 2f6b54c disable paid SnapTrade manual refresh ). A circuit breaker ended a 54.7-second wait on timeouts during a market-data outage ( 3fa974d skip Alpaca after repeated failures ).

The gates between AI and money

More checking means more tooling with access to production, so the gates matter more than the prompts:

• Trading gates in layers. For Kalshi: no credentials means disabled, ARB_TRADING_ENABLED must be set, the environment defaults to the demo sandbox, orders are capped server-side, and MCP order tools only preview unless called with confirm=true.
• Humans on anything live. An AI reviewer can auto-confirm market pairs for paper scanning only. Human confirmation remains the bar for live trading.
• Data changes are their own approved step. Deletes and backfills get a separate commit, a dry run, and an archive or preserved original.
• Risk only moves one way. The model may tighten a stop, never widen it.

What I would ask of any team running AI in production

I have spent my career on some version of this problem. At Roku it was attributing AWS spend. At Apple it is financial attribution across a very large server fleet. Nobody owns a cost they can’t see, and nobody owns a quality number they have never joined against an outcome.

The lesson from TrendVesting is not about any one bug. Validation has to be a requirement of the review, not a favor someone does when they have time. The checklist I now run:

1. Make every change carry its evidence. Review the record, not just the diff.
2. Separate fixing code from changing data. Dry-run it, archive first, approve it on its own.
3. Join stated confidence to outcomes. If you gate on a score, check that higher scores win more.
4. Score forecasts against a baseline with a proper scoring rule.
5. Make labels unable to change numbers. Write the invariant test.
6. Derive status from data, not flags, and alert on the series nobody asked for.
7. Give cost per unit a counted denominator and a measured outcome.

Most AI cost and quality numbers I see have never been checked this way. Mine hadn’t either. If you want someone to run this audit on your system, that is what I do.

Previously in Human in the Loop: TendForm: managing a solo SaaS build to production quality. Next: TigerMill: quality gates for an AI content pipeline.