Skip to content

Bump the npm group in /eng/harness-evaluation with 2 updates - #39115

Merged
AndriySvyryd merged 3 commits into
mainfrom
dependabot/npm_and_yarn/eng/harness-evaluation/npm-19a50c1d78
Oct 5, 2026
Merged

AndriySvyryd merged 3 commits into
mainfrom
dependabot/npm_and_yarn/eng/harness-evaluation/npm-19a50c1d78

Conversation

@dependabot

@dependabot dependabot Bot commented on behalf of github Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Bumps the npm group in /eng/harness-evaluation with 2 updates: @microsoft/vally and @microsoft/vally-cli.

Updates @microsoft/vally from 0.16.0 to 0.17.0

Updates @microsoft/vally-cli from 0.16.0 to 0.17.0

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
  • @dependabot ignore <dependency name> major version will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
  • @dependabot ignore <dependency name> minor version will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
  • @dependabot ignore <dependency name> will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
  • @dependabot unignore <dependency name> will remove all of the ignore conditions of the specified dependency
  • @dependabot unignore <dependency name> <ignore condition> will remove the ignore condition of the specified dependency and ignore conditions

Bumps the npm group in /eng/harness-evaluation with 2 updates: @microsoft/vally and @microsoft/vally-cli.


Updates `@microsoft/vally` from 0.16.0 to 0.17.0

Updates `@microsoft/vally-cli` from 0.16.0 to 0.17.0

---
updated-dependencies:
- dependency-name: "@microsoft/vally"
  dependency-version: 0.17.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm
- dependency-name: "@microsoft/vally-cli"
  dependency-version: 0.17.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
@github-code-quality

github-code-quality Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Code Coverage Overview

Languages: C#

C# / code-coverage

The overall line coverage in commit d08c1ba in the dependabot/npm_and_y... branch remains at 76%, unchanged from commit ac7beba in the main branch.

Show a line coverage summary of the most impacted files.
File main ac7beba dependabot/npm_and_y... d08c1ba +/-
src/EFCore.Rela...delValidator.cs 96% 96% 0%
src/EFCore.Sqli...SqlGenerator.cs 97% 97% 0%
src/EFCore.Rela...ationalModel.cs 94% 96% +2%
src/EFCore.Rela...nalJsonIndex.cs 60% 62% +2%
src/EFCore.Rela...CommandBatch.cs 90% 94% +4%
src/EFCore.Rela...redProcedure.cs 78% 88% +10%
src/EFCore.SqlS...SqlGenerator.cs 84% 96% +12%
src/EFCore.Rela...ationCommand.cs 85% 98% +13%
src/EFCore.Rela...CommandBatch.cs 65% 89% +24%
src/EFCore.Rela...edureBuilder.cs 62% 88% +26%

Updated October 02, 2026 02:11 UTC

@AndriySvyryd
AndriySvyryd force-pushed the dependabot/npm_and_yarn/eng/harness-evaluation/npm-19a50c1d78 branch from 9460e9f to d08c1ba Compare October 2, 2026 01:41
@AndriySvyryd
AndriySvyryd enabled auto-merge (squash) October 5, 2026 20:35
@AndriySvyryd
AndriySvyryd merged commit 57a3cbe into main Oct 5, 2026
24 checks passed
@AndriySvyryd
AndriySvyryd deleted the dependabot/npm_and_yarn/eng/harness-evaluation/npm-19a50c1d78 branch October 5, 2026 20:56
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: change-tracking

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation change-tracking: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.16}}$ (3 wins, 1 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
diagnose-temporary-value-snapshot-transition 5/5 1.00 $\color{red}{\mathbf{+64{,}224\ tokens}}$ file-matches analysis.md /(InternalEnti…/: 5/5
prompt: 5/5
token-budget max 2000000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: copilot-instructions

Status: ❌ failed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ❌ failed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.05}}$ (4 wins, 9 ties, 2 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
add-a-query-test-through-provider-hierarchies 5/5 0.44 $\color{red}{\mathbf{+62{,}053\ tokens}}$ file-matches test-plan.md /(?i)(?=[\s\S]…/: 5/5
prompt: 1/5
token-budget max 750000: 5/5
design-a-nonshared-provider-query-regression 5/5 0.86 $\color{red}{\mathbf{+24{,}828\ tokens}}$ file-matches regression-pl… /(?i)(?=[\s\S]…/: 5/5
prompt: 4/5
token-budget max 1200000: 5/5
plan-a-provider-independent-async-api 5/5 1.00 $\color{red}{\mathbf{+9{,}485\ tokens}}$ file-matches implementatio… /(?i)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: cosmos-provider

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation cosmos-provider: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.24}}$ (3 wins, 2 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
explain-readitem-discriminator-selection 5/5 1.00 $\color{red}{\mathbf{+30{,}565\ tokens}}$ file-matches analysis.md /(?=[\s\S]*Cos…/: 5/5
prompt: 5/5
token-budget max 2000000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-custom-agent

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation make-custom-agent: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.08}}$ (2 wins, 2 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
create-an-efcore-api-review-agent 5/5 1.00 $\color{green}{\mathbf{-45{,}007\ tokens}}$ file-matches .github/agent… /(?m)^---[\s\S…/: 5/5
file-matches eng/harness-e… /(?m)^name:\s*…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-github-actions-workflow

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation make-github-actions-workflow: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): 0.00 (0 wins, 5 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
extend-the-release-target-validation-workflow 5/5 1.00 $\color{red}{\mathbf{+22{,}533\ tokens}}$ file-matches .github/workf… /(?=[\s\S]*act…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-instructions

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation make-instructions: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): 0.00 (1 wins, 3 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
create-query-pipeline-file-instructions 5/5 1.00 $\color{red}{\mathbf{+27{,}246\ tokens}}$ file-matches .github/instr… /(?m)^---[\s\S…/: 5/5
file-matches eng/harness-e… /(?m)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: make-skill

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation make-skill: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.24}}$ (3 wins, 2 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
create-context-pool-diagnostics-skill 5/5 0.95 $\color{red}{\mathbf{+1{,}859{,}574\ tokens}}$ file-matches .agents/skill… /(?m)^---(?=[\…/: 5/5
file-matches eng/harness-e… /(?m)(?=[\s\S]…/: 5/5
prompt: 5/5
token-budget max 1500000: 0/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: migrations

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation migrations: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.08}}$ (2 wins, 2 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
diagnose-an-upgrade-only-spurious-diff 5/5 1.00 $\color{red}{\mathbf{+20{,}098\ tokens}}$ file-matches analysis.md /(SnapshotMode…/: 5/5
prompt: 5/5
token-budget max 2000000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: model-building

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation model-building: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.16}}$ (2 wins, 3 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
place-a-property-discovery-convention-change 5/5 1.00 $\color{red}{\mathbf{+44{,}132\ tokens}}$ file-matches analysis.md /(ConventionSe…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: query-pipeline

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation query-pipeline: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.24}}$ (3 wins, 2 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
place-and-validate-a-where-translation-change 5/5 0.89 $\color{red}{\mathbf{+30{,}014\ tokens}}$ file-matches analysis.md /(?=[\s\S]*Sel…/: 3/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: run-apichief

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation run-apichief: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.20}}$ (1 wins, 4 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
plan-an-efcore-baseline-refresh 5/5 0.87 $\color{green}{\mathbf{-15{,}882\ tokens}}$ file-matches apichief-plan… /(?=[\s\S]*res…/: 5/5
prompt: 4/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: scaffolding

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation scaffolding: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.08}}$ (1 wins, 4 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
place-a-database-model-shape-change 5/5 1.00 $\color{green}{\mathbf{-30{,}057\ tokens}}$ file-matches analysis.md /IDatabaseMode…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: servicing-pr

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation servicing-pr: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.16}}$ (3 wins, 1 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
draft-a-change-tracking-servicing-pr 5/5 1.00 $\color{red}{\mathbf{+7{,}394\ tokens}}$ file-matches servicing-pr.… /[release/10\…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: sqlite-adonet

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation sqlite-adonet: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.16}}$ (3 wins, 1 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
diagnose-reused-command-state 5/5 0.87 $\color{green}{\mathbf{-17{,}458\ tokens}}$ file-matches analysis.md /Sqlite(Connec…/: 5/5
prompt: 4/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: tooling

Status: ❌ failed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ❌ failed
Required skill activation tooling: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.08}}$ (2 wins, 2 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
thread-a-migrations-add-option 5/5 0.61 $\color{red}{\mathbf{+62{,}167\ tokens}}$ file-matches tooling-plan.… /suppress-summ…/: 5/5
prompt: 2/5
token-budget max 750000: 3/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: triage

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation triage: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.52}}$ (7 wins, 3 ties, 0 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
stop-on-a-security-sensitive-relational-report 5/5 1.00 $\color{green}{\mathbf{-70{,}936\ tokens}}$ file-not-exists triage-report.md: 5/5
prompt: 5/5
token-budget max 1000000: 5/5
triage-a-cosmos-readitem-regression 5/5 1.00 $\color{green}{\mathbf{-32{,}878\ tokens}}$ file-matches triage-report… /AI Triage[\s\…/: 5/5
prompt: 5/5
token-budget max 1000000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Agent evaluation: update-pipeline

Status: ✅ passed for 57a3cbe4be0bc5e5e7f44195abf70be3d505c6c3
Treatment quality threshold: ✅ passed
Required skill activation update-pipeline: ✅ passed

Model: gpt-5.6-sol

Quality score (treatment relative to control): $\color{green}{\mathbf{0.24}}$ (4 wins, 0 ties, 1 losses).

Treatment grades by stimulus

Stimulus Successful trials Mean score Token delta Grader passes
diagnose-fk-ordering-and-concurrency 5/5 1.00 $\color{green}{\mathbf{-187{,}555\ tokens}}$ file-matches analysis.md /CommandBatchP…/: 5/5
prompt: 5/5
token-budget max 750000: 5/5

Previous result from PR #39087 provides the baseline.

Download Vally reports · View workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants