The evaluator forgot the session

Update, November 13, 2026: The PR author reproduced the finding and addressed it in commit a13095d. A separate independent check of that author commit passed all 32 tests, including the three resumed-session cases. PR 3 remains open and unmerged. Saved-state schema validation is still outside the checker’s scope. The account below preserves the original reproduction at ac16a1f and our local candidate; its original test result is not a test of the later author commit.

A checker that assumed every session starts at zero

Evaluators carry assumptions about the system they grade, and the ones that cause trouble are about time. A rule that is correct for the first message of a fresh session can be wrong for the first message of a resumed one. Unit tests built around fresh sessions won’t notice the difference.

This week’s case was small and concrete. A transcript checker for a tutoring skill flagged a valid resumed session as a violation, even though the skill explicitly allows that path. This article covers what was reproduced, what the candidate fix does and doesn’t do, and why I think “test the evaluator’s lifecycle assumptions” is a better takeaway than “build another eval framework.”

What the skill promises about a resumed session

The skill is socratic-tutor in Jane Street’s public agent_skills repository. Its session-start rules are strict. Break the topic into subskills, then:

The fresh-start path asks for 3 to 5 diagnostic questions before the tutor presents an initial map of the learner’s subskills. Each subskill has a level from 0 to 5.

The same file provides a save command and a resume command. The saved block preserves the topic, levels and notes about coverage and misconceptions. A learner can paste it into another session after resume from: to restore the prior state and continue.

So the skill supports two ways to start a session. A fresh session places the learner with a diagnostic. A resumed session restores a placement that already happened. Requiring a complete fresh diagnostic on every restore would defeat that documented continuation path.

The checker in PR 3 deserves credit first

PR 3 was opened by an existing contributor, and it adds evals/check_transcript.py, which checks observable transcript structure. It is careful work. It defines eleven rules, each tied to a quoted sentence from SKILL.md: one question per message, ending on the question, diagnostic length, map format, levels rising only after two answered checks, the “show progress” and “just tell me” commands, flattery phrases, the one-off-question guard, and save-state fidelity.

It is also clear about its own limits. The report prints a LIMITATIONS string on every run, which says the checker cannot judge strict grading, withholding answers or spaced review timing, and explicitly warns that passing does not establish tutoring quality. The test suite pins each rule to an isolated failure fixture and checks that every finding’s source text appears in the skill file.

The finding below is a narrow regression against that work. It isn’t a criticism of the approach. Our local candidate was not pushed to PR 3; the dated update above records the author’s subsequent change.

Two messages that made diagnostic-length fire

The reproduction used a synthetic transcript with two messages:

1. User: resume from: followed by a valid four-row skill map for the topic and a notes line.
2. Assistant: a line saying the state was restored, the same map, and the next question.

At ac16a1f, the checker reported a diagnostic-length violation at message 1:

Found 0 diagnostic question marks before the first map; expected 3–5.

The cause shows in check_diagnostic. It finds the first recognized map, counts question marks in assistant messages before that map, and requires 3 to 5. It never looks at what the user sent before the map. The resumed fixture has no diagnostic before its restored map, so it fails this rule.

The violation is not cosmetic. The checker exits 1 when any finding has violation severity, so a pipeline that gates on exit codes would reject a transcript in which the tutor did exactly what the skill says.

One rule knew about resume from: and another didn’t

Reading the rest of the file made this more interesting. The checker already knows the resume command exists. In answered_questions, which supports the rule that levels must rise only after two answered checks, a user reply doesn’t count as an answer if it is a known command or starts with resume from::

if text and text not in COMMANDS and not text.startswith("resume from:"):

An existing test, test_level_checks_require_nonempty_noncommand_answers, includes "resume from: old notes" as a reply that must not count as an answer. The save-state rule also checks that a saved block keeps the latest topic and levels.

So the checker modeled saving, and it modeled resume as a non-answer. It did not model resume as a different way to start a session. Each rule was written against its own slice of the skill, and the diagnostic rule’s slice was the “Session start” section, which describes only the fresh path. The commands section further down the same file changes what “start” means, and that rule never read it.

I take a general lesson from this. A rule-by-rule evaluator can be correct about every rule and still be wrong about the state machine those rules share.

The candidate fix: one guard clause and three tests

The local candidate adds a guard to check_diagnostic, before the count:

# A session resumed from a saved state block restores placement instead of re-running it.
# A bare "resume from:" carries no state, so it does not exempt the diagnostic.
if any(m["role"] == "user" and re.fullmatch(r"resume from:\s*\S.*", command(m["content"]))
       for m in messages[:first.index]):
    return

command() is the checker’s existing normalizer. It collapses whitespace, casefolds and strips trailing punctuation, so a multi-line saved block becomes one line and Resume from: matches as well. The regex requires at least one non-space character after the colon.

The new test, test_resume_from_saved_state_skips_diagnostic, covers three cases:

• Valid resume: a saved map and notes, then a restored map and one question. The test expects zero findings from all rules, not only the diagnostic rule.
• Resume after a short diagnostic: a fresh session runs only 2 diagnostic questions and shows a map, and then the user resumes. The test expects diagnostic-length to still fire, because the exemption only applies to a resume before the first map.
• Bare prefix: "resume from:", "Resume from: " and "resume from:\n\n" each go before a map, and each must still be flagged, because nothing was restored.

With the change, an independent rerun passed all 32 tests. Without it, the valid synthetic fixture produces the false positive described above.

What the exemption does not check

The exemption has limits a reviewer should see before considering it.

Any nonempty text qualifies. resume from: hello exempts the diagnostic just as a well-formed state block does. The guard checks that something follows the prefix. It does not check that it is a saved state.

It skips the rule entirely. A resumed session whose tutor then asks seven placement questions before the first map would also pass this rule, because the count is never computed. Whether that should be a finding is a design question for the maintainers. The candidate patch takes no position on it.

It doesn’t check fidelity. Nothing compares the restored map to the saved one. A tutor could accept a block with “Comparing 3/5” and render “Comparing 0/5”, and this rule would stay silent. The existing save-state rule checks the save direction. No rule checks the restore direction.

It doesn’t validate semantics. Topic, levels, notes and coverage go unchecked. It is a prefix guard with a nonempty requirement, and I wouldn’t describe it as anything more.

The public PR comment explicitly identifies semantic validation and map fidelity as unchecked.

Test the evaluator’s lifecycle, not only its rules

The broader point is about evaluators, not tutors. An evaluator is a program with its own model of the system’s states. When that model has only “fresh start,” every legitimate transition into the middle of a session looks like a violation.

Yesterday’s article, The rollback that made the new column lie, made a related point in a different domain. It used a synthetic, serial four-row test on PostgreSQL 16.15, not concurrency evidence. The parallel is only that tests built from an initial state miss what happens across a transition. That is where the comparison ends.

None of this calls for a new framework. Promptfoo and Vercel already publish agent eval tooling, and the checker in PR 3 is a reasonable shape for a skill-specific evaluator. The cheaper move is to add small regression fixtures upstream wherever the skill defines a way to enter or leave a session.

For this checker, I’d propose two follow-ups. Neither exists yet:

1. A roundtrip test. Take an assistant’s save state output, feed it back as resume from:, and assert that the first restored map has the same topic and levels. The checker’s parse_maps function could recognize the block, which would replace “nonempty” with “contains a recognizable map.”
2. A fresh-versus-restored matrix. For each of the eleven rules, write one fixture that starts fresh and one that starts from a resume, and record the expected result for each. Most cells would be “same result,” but the table forces the question for each rule instead of leaving it to whoever finds the next false positive.

A checklist for evaluators that read transcripts

1. List every way the system under test can enter a session: fresh start, resume, handoff, retry. Give each one a fixture.
2. For each rule, ask whether it holds on every entry path or only on a fresh start. Write that answer into a test.
3. Search the evaluator for every command it already recognizes. If one rule handles a command, check whether the others should too.
4. Make exemptions as narrow as the evidence allows, and add a test showing that the near-miss input (a bare prefix, a resume after the first map) is still flagged.
5. Say what the exemption does not validate, in the code comment and in the PR comment.
6. Pair every one-directional check, such as save-state fidelity, with its inverse, or record that the inverse is unchecked.
7. Confirm that the test environment can run the fixtures before attributing a failed run to the product.
8. Prefer a two-message fixture upstream to a new tool downstream.

What these checks did not establish

• Tutor behavior. No tutoring model was evaluated. The transcripts are synthetic, so this shows nothing about whether any tutor restores state correctly.
• Learning or tutoring quality. The checker’s own limitations string says a clean report doesn’t establish this. The patch doesn’t change that.
• Production use. No real sessions were examined.
• Other rules on resumed sessions. Only diagnostic-length was reproduced as a false positive. The other ten rules were not tested systematically against resumed transcripts. The matrix above is a proposal.
• Maintainer intent. The maintainers may prefer to require a short re-check after a resume, or a different exemption. The comment reports a finding. It doesn’t decide the design.
• Upstream status. Our original local candidate was not pushed. The author subsequently addressed the finding in a13095d, as recorded in the dated update. PR 3 remains the original contributor’s work and was still open and unmerged when this update was checked.

Sources and reproduction evidence

• PR 3, the transcript checker
• The finding, posted as a PR comment
• SKILL.md at ac16a1f
• check_transcript.py at ac16a1f

If you test evaluators against more than the fresh-start path, I’d like to hear how you structure it: join the newsletter or get in touch.

Disclosure: Claude Code helped draft the local patch, the regression test and this article.

Previously in Human in the Loop: The rollback that made the new column lie.