Repository navigation
Make the evals more realistic and improve the skills accordingly - #39217
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Required comparison evidence is omitted and fixture provenance hashes are incorrect.
4 open findings
What changed in this PR
Updates the evaluation harness for more realistic fixtures, configurable judge effort, and richer comparison evidence.
Changes:
- Adds comparison-input preparation and completed-trial handling.
- Upgrades evaluation models and refreshes scenarios/fixtures.
- Expands repository skills and associated harness coverage.
| File | Description |
|---|---|
eng/harness-evaluation/test/harness.test.mjs |
Tests validation, judge effort, and completion gates. |
eng/harness-evaluation/test/eval-text-graders.test.mjs |
Empty test file in the reviewed snapshot. |
eng/harness-evaluation/test/comparison.test.mjs |
Tests comparison evidence preparation and safety. |
eng/harness-evaluation/src/harness.mjs |
Adds completed-variant detection and fixture validation. |
eng/harness-evaluation/src/comparison.mjs |
Builds bounded comparison evidence from workspace patches. |
eng/harness-evaluation/src/cli.mjs |
Forwards judge effort and prepares comparisons. |
eng/harness-evaluation/skills/update-pipeline/fixtures/shared-identity/source-excerpts.md |
Documents fixture context and provenance. |
eng/harness-evaluation/skills/update-pipeline/fixtures/shared-identity/proposed/NonSharedModelUpdatesTestBase.cs |
Adds proposed update-pipeline test snapshot. |
eng/harness-evaluation/skills/update-pipeline/fixtures/shared-identity/proposed/ColumnAccessorsFactory.cs |
Adds proposed accessor snapshot. |
eng/harness-evaluation/skills/update-pipeline/fixtures/shared-identity/before/NonSharedModelUpdatesTestBase.cs |
Adds baseline test snapshot. |
eng/harness-evaluation/skills/update-pipeline/fixtures/shared-identity/before/ColumnAccessorsFactory.cs |
Adds baseline accessor snapshot. |
eng/harness-evaluation/skills/update-pipeline/eval.yaml |
Reworks shared-identity evaluation. |
eng/harness-evaluation/skills/triage/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/skills/tooling/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/skills/sqlite-adonet/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/skills/servicing-pr/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/skills/scaffolding/eval.yaml |
Corrects schema-filter evaluation semantics. |
eng/harness-evaluation/skills/run-apichief/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/skills/query-pipeline/fixtures/query-pipeline/split-enumerables.cs |
Adds split-query review fixture. |
eng/harness-evaluation/skills/query-pipeline/fixtures/query-pipeline/projection-retry.cs |
Adds projection retry fixture. |
eng/harness-evaluation/skills/query-pipeline/fixtures/query-pipeline/fallback-and-tests.cs |
Adds fallback and concurrency fixture. |
eng/harness-evaluation/skills/query-pipeline/fixtures/provider-test-placement/source-context.md |
Supplies provider-test hierarchy context. |
eng/harness-evaluation/skills/query-pipeline/fixtures/provider-test-placement/proposed/NorthwindMiscellaneousQuerySqlServerTest.cs |
Adds proposed provider-only test snapshot. |
eng/harness-evaluation/skills/query-pipeline/fixtures/provider-test-placement/before/NorthwindMiscellaneousQueryTestBase.cs |
Adds specification baseline snapshot. |
eng/harness-evaluation/skills/query-pipeline/fixtures/provider-test-placement/before/NorthwindMiscellaneousQuerySqlServerTest.cs |
Adds SQL Server baseline snapshot. |
eng/harness-evaluation/skills/query-pipeline/fixtures/provider-test-placement/before/NorthwindMiscellaneousQueryRelationalTestBase.cs |
Adds relational baseline snapshot. |
eng/harness-evaluation/skills/query-pipeline/eval.yaml |
Introduces query-pipeline skill evaluation. |
eng/harness-evaluation/skills/model-building/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/skills/migrations/fixtures/upgrade-report.txt |
Expands the upgrade regression report. |
eng/harness-evaluation/skills/migrations/fixtures/upgrade-only/source-context.md |
Documents migration-diff code paths. |
eng/harness-evaluation/skills/migrations/fixtures/upgrade-only/SnapshotModelProcessor.cs |
Adds reduced processor snapshot. |
eng/harness-evaluation/skills/migrations/fixtures/upgrade-only/CustomerContextModelSnapshot.cs |
Adds historical model snapshot. |
eng/harness-evaluation/skills/migrations/fixtures/upgrade-only/CustomerContext.cs |
Adds current model fixture. |
eng/harness-evaluation/skills/migrations/eval.yaml |
Reworks upgrade compatibility evaluation. |
eng/harness-evaluation/skills/make-skill/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/skills/make-instructions/eval.yaml |
Focuses evaluation on proposal review. |
eng/harness-evaluation/skills/make-github-actions-workflow/eval.yaml |
Focuses evaluation on external PR policy. |
eng/harness-evaluation/skills/make-custom-agent/eval.yaml |
Focuses evaluation on agent boundaries. |
eng/harness-evaluation/skills/cosmos-provider/fixtures/CosmosTransactionalBatchTest.cs |
Adds Cosmos transactional-batch fixture. |
eng/harness-evaluation/skills/cosmos-provider/eval.yaml |
Replaces analysis with offline test generation. |
eng/harness-evaluation/skills/code-review/fixtures/dbset-sequence/DbSetConsumer.cs |
Adds binary compatibility consumer fixture. |
eng/harness-evaluation/skills/code-review/fixtures/concurrency-detector/proposed/ConcurrencyDetectorTest.cs |
Adds proposed concurrency tests. |
eng/harness-evaluation/skills/code-review/fixtures/concurrency-detector/proposed/ConcurrencyDetector.cs |
Adds proposed detector implementation. |
eng/harness-evaluation/skills/code-review/fixtures/concurrency-detector/before/ConcurrencyDetectorTest.cs |
Adds baseline concurrency tests. |
eng/harness-evaluation/skills/code-review/fixtures/concurrency-detector/before/ConcurrencyDetector.cs |
Adds baseline detector implementation. |
eng/harness-evaluation/skills/code-review/eval.yaml |
Reworks compatibility and concurrency reviews. |
eng/harness-evaluation/skills/change-tracking/eval.yaml |
Updates model defaults. |
eng/harness-evaluation/README.md |
Documents snapshots, comparison evidence, and judge controls. |
eng/harness-evaluation/instructions/copilot-instructions/fixtures/query-pipeline/projection-retry.cs |
Expands instruction fixture behavior. |
eng/harness-evaluation/instructions/copilot-instructions/fixtures/activation-proposal.cs |
Adds API implementation proposal. |
eng/harness-evaluation/instructions/copilot-instructions/eval.yaml |
Replaces planning with implementation evaluation. |
.github/copilot-instructions.md |
Refines shared test-infrastructure guidance. |
.agents/skills/update-pipeline/SKILL.md |
Updates shared-row value-access guidance. |
.agents/skills/scaffolding/SKILL.md |
Corrects anti-semi-join guidance. |
.agents/skills/query-pipeline/SKILL.md |
Adds query-pipeline domain guidance. |
.agents/skills/migrations/SKILL.md |
Clarifies historical metadata normalization. |
.agents/skills/make-instructions/SKILL.md |
Adds initial-authoring validation order. |
.agents/skills/make-github-actions-workflow/SKILL.md |
Tightens bot-comment ownership checks. |
.agents/skills/make-custom-agent/SKILL.md |
Adds proposal-review boundary guidance. |
🧠 Review effort: Balanced
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Code coverage overviewLanguages: C# C# / code-coverage
View coverage of the 10 most impacted files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. Updated |
AndriySvyryd
force-pushed
the
ImproveSkills
branch
from
October 14, 2026 20:35
c2b520d to
66e1010
Compare
AndriySvyryd
marked this pull request as ready for review
October 14, 2026 20:56
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
Contributor
Agent evaluation:
|
AndriySvyryd
force-pushed
the
ImproveSkills
branch
from
October 15, 2026 02:34
66e1010 to
57cead9
Compare
Update the models used for evals Allow to specify the judge effort level and timeout
AndriySvyryd
force-pushed
the
ImproveSkills
branch
from
October 15, 2026 21:46
57cead9 to
04dfedf
Compare
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Runtime rewriting of Vally and broad grading changes need end-to-end validation with the pinned dependencies before approval.
0 open findings
1 resolved since last review
Files not reviewed (1)
- eng/harness-evaluation/package-lock.json: Generated file
🧠 Review effort: Balanced
cincuranet
approved these changes
Oct 16, 2026
rogerchappel
added a commit
to rogerchappel/launchclip
that referenced
this pull request
Oct 27, 2026
The PR's facts line now reads "Pull request dotnet/efcore#39217 (pull request 39217) by …", so a script that cites the PR by number can be checked against the PR itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rogerchappel
added a commit
to rogerchappel/launchclip
that referenced
this pull request
Oct 27, 2026
* feat(capabilities): one params-builder file per module family; planned means no builder
Six capability-builders-*.mjs files (social, code, particles, objects, explain,
channel) each return {builders, additions} from the shared helpers, so the
families can fill their modules in parallel without touching one table.
isPlanned(id) replaces the hand-kept planned flag: a capability is planned
while it has no builder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(beat-director): matched evidence carries its module-ready data
matchEvidence passes capture.data (else claim.data) on as evidence.data, the
input the evidence builders map into post, PR, commit and chart params.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(grammar): re-pin the minimal golden for its opening scene starting at rest
#367 made HyperFrames start an editorial opening scene at rest on frame 0, as
Remotion does. minimal opens on one, so its compiled HTML changes in exactly
two places (#s1 and #s1-loop enter from opacity 1). Its parity fix was right;
this pin was missed because grammar.test.js was not rerun after it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(grammar): re-pin data-story-9x16 too; it also opens on an editorial scene
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(capabilities): family files bring their own evidence sample beats
An evidence-only builder (paper-stack, social-post, repo-pr-card, …) returns
null on the shared sample beats, which carry no evidence.data, so the
'every params builder resolves' test could not check it. Each family file now
returns samples (beats with the evidence.data its builders need), exported as
capabilitySamples(); the test tries those after BEAT and SERIES. The neutral-bag
check uses a module id with no builder rather than one a family will fill.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(director): params builders for the social cards
social-post, social-stack, social-wall and social-sort get their params
from beat.evidence.data (one post or {posts}), so the director uses them
instead of falling back to quote-card and feed-frame. Real posts only: no
data or no source.url (a demo account aside) builds nothing, and the
clause never becomes a post. The clause picks which phrase the highlight
sweeps (the longest run of words its quote, else the clause, shares with
the post) and which post leads. Long posts become excerpts ending in …;
the sort keeps three posts so 9:16 fits. The resolve check gets a sample
beat with demo posts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(director): the social builders' honesty rules and their output
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore(skills): sync the social builders
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(director): the social family's sample beats build every social module
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(director): params builders for the code-evidence modules
repo-pr-card, commit-card, diff-view, evidence-chain, date-pinned-timeline
and terminal-session now take a clause when its evidence carries module-ready
data, so the director stops falling back to window-chrome, excerpt or
timeline for them.
- PR and commit cards pass src/evidence/github.js output through ({module,
params} or the bare params) and box the token the clause names: a spoken
word widened to the identifier it sits in ("terra" -> gpt-6.1-terra), the
mapped token first. highlightAt follows where the clause says it, inside
the module's window; otherwise the module's own timing stands.
- diff-view takes a diff, PR or commit mapping; the focus steps to the change,
then to the token's row.
- evidence-chain and date-pinned-timeline trim each source to the rows round
the shared token (two in a chain of three or four, so 9:16 stacks fit) and
drop sources without it; the timeline turns into the why-now summary when
the clause asks why now.
- terminal-session replays recorded steps and boxes the named token.
No evidence.data, no params: nothing is typed from narration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(eval): module-ready data on the fixture's PR and commit captures
sdk-pr-1842 and sdk-fix-commit now carry data in the shape
src/evidence/github.js maps ({module, params}), drawn from the fixture's own
evidence ledger (northlight/sdk#1842 "default to halcyon-2", 2026-03-10;
commit 9f3c2e1 "fix: respect format setting behind legacy_format flag",
2026-03-14), so the dry run exercises the code-evidence builders.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(eval): regenerate the 16-minute beat-map fixture with the code builders
node eval/beat-map-dry-run.mjs --fresh. Both PR clauses now land on
repo-pr-card and the fix clause on commit-card (they were concept-card);
the report lists no fallbacks. Cadence still passes; variety rises to 25
modules. The terminal clause stays on window-chrome and the two chain
clauses on concept-card: the fixture has no recorded session and no
multi-source data for them, and those builders return null without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(builders): logo-sting from the story brand
The channel family's builder: the sting takes beat.brand's name (or a
text wordmark), its logo when it is a project image (logo or mark), and
its tagline. No brand, or a brand with no name, gives null, so the
director falls back to brand-lockup and then the title, as before. A
branded sting beat is the family's sample, so the shared capability test
resolves the builder without a brand on its own sample beats.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(director): long-form storyboards wear plates, glass and the channel
beatMapToStory, on a long-form map only (Shorts and other formats are
untouched):
- each chapter's plate on its chapter (on by default; plates: false turns
it off), so the backdrop changes at chapter boundaries and never cuts
inside one; a window inside one chapter takes it as the story's plate.
The director already cycles plate kinds per chapter.
- style "glass-explainer": the story names the style and its rim-glass
panel.
- with a channel ({name, logo?}, default map.channel): a subscribe-cta pill
on track 1 over the scene before each call to action, which holds through
it in place of the cta card, and a channel-watermark on track 2 across
the whole window.
directLongForm keeps a named stingBrand as map.channel, and puts it in the
chunk cache key only when set, so maps planned without one keep their keys.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(dry-run): the fixture's channel, so its stings are logo-stings
The dry run passes a channel (a placeholder name of our own, "Signal Lab",
--channel to change it, --no-channel to drop it) as the director's sting
brand. The open and the close now plan as logo-stings instead of falling
back to a concept card, and the map carries the channel for the story's
subscribe pills and watermark. Regenerated BEATMAP.json, DIRECTION.json and
CADENCE.md; the cadence gate still passes (157 events, p50 5.75 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(director): plan the fixture with its channel, as the dry run does
The committed-fixture check plans with the dry run's structure, which now
includes its channel, so the stings are logo-stings named for it. The
no-brand fallback (the sting is the video's title) is checked in
test/beat_director_look.test.js.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(transitions): a rack-focus lands as an event
The rack ran on gentle eases over the whole transition and both cards
faded on their own under it, so a scene detector saw almost nothing (peak
scene score 0.008 on the long-form sample, against the reference's 0.08
event threshold). Now:
- both cards move at once and most of the move lands in the first frames
(expo.out), the incoming card from deeper (blur 22, scale 1.08) and the
outgoing one further back (blur 26, scale 0.9);
- the arriving card carries a short exposure lift (brightness 1.45 → 1)
that settles with its blur: one filter prop, rackIn, so GSAP never drops
one of the two;
- a card a rack-focus or card-to-card carries rests whole at both ends
(CARRY_TRANSITIONS, modules.mjs carriedByTransition): the transition
brings it in and takes it away instead of the card fading under it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(states): a bloom marks each state, and its moves land first
Each state with a target (focus, glow, else what it reveals, else a band
along its guide) gets one bloom: a soft fill in the focus colour behind it
that lands in about two frames and decays over 0.85 s. Param morphs, the
ring's travel, guides and dims now ease expo.out (dims in 0.25 s), so the
change happens in its first frames instead of spreading over 18. On the
shot-states sample the per-state peak scene score goes from about 0.005 to
a median near 0.1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(beat-director): builds inside a shot rack focus instead of fading
A shot's states become scenes, often of different modules, and a 0.35 s
fade between two near-identical cards changed almost no pixels: the
long-form sample's in-shot changes peaked at a 0.009 scene score. They now
rack focus (0.4 s), which carries the cards and lands in its first frames.
A new shot still cuts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(eval): state events, and a gate that every state change reads
stateMarks(story) lists every state change (scene.states and the in-shot
swaps that arrive by a transition; a hard cut is not a state), and
stateEvents(scores, story) takes each one's peak ffmpeg scene score over
[t - 0.1, t + 0.8] and their median, against the reference's event
threshold (0.08). The scorecard gains a state_events check and a
per-change table in its report.
scripts/state-events.mjs measures it without a full render: it compiles
the storyboard to HyperFrames, seeks the timeline frame by frame in
Chromium around each change, encodes each window and reads the scene
score. The media test runs it on the shot-states sample and fails when
the median is under 0.08.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(transitions): the racked-in card arrives within a few frames
The incoming card's opacity now runs over a quarter of the rack instead of
most of it: it is there, blurred and lifted, within a few frames, then pulls
sharp while the old card falls back under it. On the long-form sample all
eight in-shot swaps now read as events (median peak scene score 0.115,
none under 0.08; it was 0.009 with the fade).
Docs: rack-focus, the state bloom, carried cards and the state-events
measure in KIT_CONTRACTS. Skill copies synced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(states): the sample racks into its table in 0.5 s
The beat director's racks run 0.4–0.5 s; the sample's 0.8 s rack spread
its change too thin to read as an event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(research): module-ready evidence data for the capability builders
attachEvidenceData() fills the data matchEvidence() hands the evidence
builders, to the contracts in the capability-builders-* headers: PR, commit
and diff captures get github.js params from its cached API JSON; post
captures get the author, handle and text their sidecar recorded; verified
claims get series and panels from the table row they quote, people (a photo
only from a licence-clear portrait), the tokenizer split of a quote, TeX with
an SVG when mathjax-full is installed, and an evidence chain when two or more
GitHub sources back them. Anything that cannot be built is reported with its
reason, never invented.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(research): the ledger carries module-ready data; table ids move to claim.tables
claim.data was the list of data-file ids, but matchEvidence() passes it to
the builders as module-ready data. The ids are now claim.tables, and
runResearch attaches evidence data before writing. The fixture is rebuilt
from its recording (research-fixture.mjs --replay): the quote claim gets its
tokens and both benchmark claims their series.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(research): launchclip research attach, once the clips exist
Post data needs the capture's sidecar, which only exists after launchclip
evidence capture, so attach reruns the evidence-data pass over a research
folder and writes the data into the ledger and capture requests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(eval): the beat-map dry run plans with the evidence data attached
loadFixture() reads the research folder through loadResearchEvidence(), so
the director's matched evidence carries GitHub cards, posts, series, people
and tokens from the folder's captures and caches. The committed 16-minute
fixture is unchanged (its captures have no cached sources).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(research): a PR snapshot names its repo and number
The PR's facts line now reads "Pull request dotnet/efcore#39217 (pull request
39217) by …", so a script that cites the PR by number can be checked
against the PR itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(research): four false verifications found on the GPT-6.1 Terra run
- A passage must name what the claim names: every benchmark, and all other
entities but one when there are three or more (one-word fragments aside).
OpenRouter's "released on October 26" no longer verifies a GitHub Copilot
launch date.
- A multiple or currency needs the same unit in the source: "9x" is not the
9 in "July 9". A percentage may still match a bare table figure.
- Versions inside model and benchmark names ("Claude Opus 5.5",
"Terminal-Bench 4.0") are not quantities, and a capitalised run no longer
swallows the start of a model name ("HumanEval Nimbus-").
- openrouter.ai is not OpenAI's own domain: an org's name with its suffix
stripped must keep five letters to claim a host.
The DeepSeek fixture replays unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(research): the entity-coverage and unit rules
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(research): highlight targets the page can find
Found capturing the GPT-6.1 Terra sources: five of eleven requests asked for
text the rendered page does not hold. Page text now joins inline tags as the
page shows them ("GPT-6.1 <a>Sol</a>." is "GPT-6.1 Sol.", not "Sol .") and
decodes named entities (’); spans are at most eight words and stay in
one table cell. A script-rendered page (an X post) keeps its og:description
as text. The DeepSeek fixture's one long span is rebuilt from its recording.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(research): post data from the fetched page, card data for a lone PR
X refuses headless browsers, so a post often has no clip or sidecar; its
data then comes from the page the research stage fetched and hashed (author
and handle from the title, text from the verifying passage), marked
source.via: "page metadata" with no screenshot. A claim backed by one PR or
commit and no capture gets that card's {module, params}, so a diff too long
to capture still draws from the API.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(research): prices are not TeX, and a versioned name is not a person
Found on the Terra run: "$1 per million … $5" was read as an equation, and
"Claude Opus" (before 5.5) as a person because the sentence said "posted".
TeX now opens on a non-digit and needs a TeX mark; a name followed by a
version is never a person.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(research): cut a table row to its cell before stripping markup
plain() removed the pipes the cell split needs, so a flattened row's
highlight ran across cells ("Agents' Last Exam, medium effort 15 points
above") and the page could not find it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(eval): regenerate the 16-minute beat-map fixture with every family's builders
No capability falls back any more; the plan uses 27 modules (24 before) and
shows 55 of 85 claims with a sourced visual.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.