Repository navigation
Bump the npm group in /eng/harness-evaluation with 2 updates - #39115
AndriySvyryd merged 3 commits into
Conversation
Bumps the npm group in /eng/harness-evaluation with 2 updates: @microsoft/vally and @microsoft/vally-cli. Updates `@microsoft/vally` from 0.16.0 to 0.17.0 Updates `@microsoft/vally-cli` from 0.16.0 to 0.17.0 --- updated-dependencies: - dependency-name: "@microsoft/vally" dependency-version: 0.17.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: npm - dependency-name: "@microsoft/vally-cli" dependency-version: 0.17.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: npm ... Signed-off-by: dependabot[bot] <support@github.com>
Code Coverage OverviewLanguages: C# C# / code-coverageThe overall line coverage in commit d08c1ba in the Show a line coverage summary of the most impacted files.
Updated |
…on/npm-19a50c1d78
9460e9f to
d08c1ba
Compare
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| diagnose-temporary-value-snapshot-transition | 5/5 | 1.00 | file-matches analysis.md /(InternalEnti…/: 5/5 prompt: 5/5 token-budget max 2000000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| add-a-query-test-through-provider-hierarchies | 5/5 | 0.44 | file-matches test-plan.md /(?i)(?=[\s\S]…/: 5/5 prompt: 1/5 token-budget max 750000: 5/5 |
|
| design-a-nonshared-provider-query-regression | 5/5 | 0.86 | file-matches regression-pl… /(?i)(?=[\s\S]…/: 5/5 prompt: 4/5 token-budget max 1200000: 5/5 |
|
| plan-a-provider-independent-async-api | 5/5 | 1.00 | file-matches implementatio… /(?i)(?=[\s\S]…/: 5/5 prompt: 5/5 token-budget max 1000000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| explain-readitem-discriminator-selection | 5/5 | 1.00 | file-matches analysis.md /(?=[\s\S]*Cos…/: 5/5 prompt: 5/5 token-budget max 2000000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| create-an-efcore-api-review-agent | 5/5 | 1.00 | file-matches .github/agent… /(?m)^---[\s\S…/: 5/5 file-matches eng/harness-e… /(?m)^name:\s*…/: 5/5 prompt: 5/5 token-budget max 1000000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| extend-the-release-target-validation-workflow | 5/5 | 1.00 | file-matches .github/workf… /(?=[\s\S]*act…/: 5/5 prompt: 5/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| create-query-pipeline-file-instructions | 5/5 | 1.00 | file-matches .github/instr… /(?m)^---[\s\S…/: 5/5 file-matches eng/harness-e… /(?m)(?=[\s\S]…/: 5/5 prompt: 5/5 token-budget max 1000000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| create-context-pool-diagnostics-skill | 5/5 | 0.95 | file-matches .agents/skill… /(?m)^---(?=[\…/: 5/5 file-matches eng/harness-e… /(?m)(?=[\s\S]…/: 5/5 prompt: 5/5 token-budget max 1500000: 0/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| diagnose-an-upgrade-only-spurious-diff | 5/5 | 1.00 | file-matches analysis.md /(SnapshotMode…/: 5/5 prompt: 5/5 token-budget max 2000000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| place-a-property-discovery-convention-change | 5/5 | 1.00 | file-matches analysis.md /(ConventionSe…/: 5/5 prompt: 5/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| place-and-validate-a-where-translation-change | 5/5 | 0.89 | file-matches analysis.md /(?=[\s\S]*Sel…/: 3/5 prompt: 5/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| plan-an-efcore-baseline-refresh | 5/5 | 0.87 | file-matches apichief-plan… /(?=[\s\S]*res…/: 5/5 prompt: 4/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| place-a-database-model-shape-change | 5/5 | 1.00 | file-matches analysis.md /IDatabaseMode…/: 5/5 prompt: 5/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| draft-a-change-tracking-servicing-pr | 5/5 | 1.00 | file-matches servicing-pr.… /[release/10\…/: 5/5 prompt: 5/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| diagnose-reused-command-state | 5/5 | 0.87 | file-matches analysis.md /Sqlite(Connec…/: 5/5 prompt: 4/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| thread-a-migrations-add-option | 5/5 | 0.61 | file-matches tooling-plan.… /suppress-summ…/: 5/5 prompt: 2/5 token-budget max 750000: 3/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| stop-on-a-security-sensitive-relational-report | 5/5 | 1.00 | file-not-exists triage-report.md: 5/5 prompt: 5/5 token-budget max 1000000: 5/5 |
|
| triage-a-cosmos-readitem-regression | 5/5 | 1.00 | file-matches triage-report… /AI Triage[\s\…/: 5/5 prompt: 5/5 token-budget max 1000000: 5/5 |
Previous result from PR #39087 provides the baseline.
Agent evaluation:
|
| Stimulus | Successful trials | Mean score | Token delta | Grader passes |
|---|---|---|---|---|
| diagnose-fk-ordering-and-concurrency | 5/5 | 1.00 | file-matches analysis.md /CommandBatchP…/: 5/5 prompt: 5/5 token-budget max 750000: 5/5 |
Previous result from PR #39087 provides the baseline.
Bumps the npm group in /eng/harness-evaluation with 2 updates: @microsoft/vally and @microsoft/vally-cli.
Updates
@microsoft/vallyfrom 0.16.0 to 0.17.0Updates
@microsoft/vally-clifrom 0.16.0 to 0.17.0Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore <dependency name> major versionwill close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)@dependabot ignore <dependency name> minor versionwill close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)@dependabot ignore <dependency name>will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)@dependabot unignore <dependency name>will remove all of the ignore conditions of the specified dependency@dependabot unignore <dependency name> <ignore condition>will remove the ignore condition of the specified dependency and ignore conditions