Skip to content

[Perf] Linux/x64: 4 Regressions on 6/29/2026 9:51:56 AM +00:00 #130041

New issue
New issue
Closed
Bug
#131993
Closed
[Perf] Linux/x64: 4 Regressions on 6/29/2026 9:51:56 AM +00:00#130041
Bug
#131993
Assignees
iremyux
Labels
arch-x64area-System.Formats.Taros-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark
Milestone
11.0.0

Description

@performanceautofiler
performanceautofiler
bot
opened on Jun 30, 2026

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue ViperUbuntu
Baseline 0d3df07c6048d8cbe4f544227462cae62040e587
Compare 93ec56ee6c38a470c6fed16b59d3f130587b1a06
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Formats.Tar.Tests.Perf_TarWriter

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
191.81 ns 269.80 ns 1.41 0.09 False
187.02 ns 248.65 ns 1.33 0.14 False
205.86 ns 273.10 ns 1.33 0.11 False
1.36 μs 1.53 μs 1.13 0.02 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Formats.Tar.Tests.Perf_TarWriter*'
Details

System.Formats.Tar.Tests.Perf_TarWriter.UstarTarEntry_WriteEntry

ETL Files

Histogram

JIT Disasms

System.Formats.Tar.Tests.Perf_TarWriter.V7TarEntry_WriteEntry

ETL Files

Histogram

JIT Disasms

System.Formats.Tar.Tests.Perf_TarWriter.GnuTarEntry_WriteEntry

ETL Files

Histogram

JIT Disasms

System.Formats.Tar.Tests.Perf_TarWriter.PaxTarEntry_WriteEntry

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    added
    arch-x64
    os-linuxLinux OS (any supported distro)
    PGO
    runtime-coreclrspecific to the CoreCLR runtime
    untriagedNew issue has not been triaged by the area owner
    on Jun 30, 2026

  2. LoopedBard3 commented on Jun 30, 2026

    @LoopedBard3
    LoopedBard3
    on Jun 30, 2026
    Member

    🔍 Automated Triage Analysis

    Summary: Verified regression in UstarTarEntry_WriteEntry — caused by commit ff08290 ("Refactor sync/async implementations by using Adapter pattern (#129282)"). 4 tests bisected to related culprit commits.

    Finding Confidence: 5/5 (analysis accuracy)
    Regression Confidence: 5/5 (likelihood of true regression)

    Likely Cause: ff08290b777d78822cc4505bf7308d33b9735705

    🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 4 dry-run (rule R4_known_regression_new — see Full Analysis for the table). · 🦆 Re-analysis: ✅ concurs

    📊 Full Analysis (click to expand)

    Summary: Four System.Formats.Tar TarWriter benchmarks regressed +13–41% as clear step changes (z-scores 8.0–17.4) attributed to commit ff08290 ("Refactor sync/async implementations by using Adapter pattern"). Finding confidence 4/5 (excellent historical data, 298 runs, low variance), regression confidence 5/5 (surgical change directly to TarWriter/TarHeader code, exact temporal match, all four entry types affected simultaneously). Runtime stack: Desktop + CoreCLR.

    Issue Overview

    • Issue: [Perf] Linux/x64: 4 Regressions on 6/29/2026 9:51:56 AM +00:00 #130041 - [Perf] Linux/x64: 4 Regressions on 6/29/2026 9:51:56 AM +00:00
    • Date: 2026-06-29 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux/x64
    • Commits: 0d3df07c6048 → 93ec56ee6c38 (10 commits)
    • Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro, PGO, compilationmode-tiered, runkind-micro
    • Historical Data: Available (298–299 runs per test, 1225 total runs examined)

    Test Analysis

    Test Name Baseline Compare Δ% μ σ n z Thresh% Noise? Assessment
    UstarTarEntry_WriteEntry 191.81 ns 269.80 ns +40.66% 196.45 4.21 298 17.43 5.00% N Step change, very stable
    V7TarEntry_WriteEntry 187.02 ns 248.65 ns +32.95% 185.50 4.66 298 13.56 5.02% N Step change, stable
    GnuTarEntry_WriteEntry 205.86 ns 273.10 ns +32.66% 207.16 7.75 299 8.51 7.48% N Step change, pre-existing downward drift
    PaxTarEntry_WriteEntry 1357.80 ns 1531.90 ns +12.82% 1362.07 21.26 298 7.99 5.00% N Step change, very stable

    Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

    Finding Confidence: 4/5 - Excellent historical data (298–299 runs per test), low coefficient of variation (1.6–3.7%), complete commit information. All four tests show unambiguous step changes with z-scores far above the significance threshold.

    Candidate Stack Relevance

    Commit Touched Paths (summary) Relevance Notes
    ff08290b777d src/libraries/System.Formats.Tar/ (TarWriter, TarHeader, TarReader, TarHelpers, GnuSparseStream), src/libraries/Common/ (ReadWriteAdapter) Shared BCL library refactor; directly touches the benchmarked code paths
    664063fb61eb src/coreclr/tools/, src/coreclr/vm/jitinterface.cpp Relevant CoreCLR devirtualization change; could affect virtual dispatch but unlikely to cause +30% regression in specific library
    e137b2c01822 src/coreclr/jit/scev.cpp Relevant JIT SCEV fix; narrow scope, unlikely to impact Tar benchmarks
    c366a5387287 src/coreclr/vm/interpexec.cpp, src/coreclr/vm/reflectioninvocation.cpp Relevant Wasm interpreter fix; not exercised in this config

    Confirmed Regressions

    System.Formats.Tar.Tests.Perf_TarWriter.UstarTarEntry_WriteEntry (+40.66%)

    • Delta: 191.81 ns → 269.80 ns (+40.66%)
    • Historical Context: μ=196.45 (σ=4.21, n=298); z-score=17.43; adaptive threshold=5.00%
    • Assessment: Unmistakable step change. Before series tightly clustered ~186–206 ns (CV=2.1%); all after values cluster ~263–276 ns, entirely disjoint from historical distribution.
    • Finding Confidence: 4/5 - Excellent data quality with 298 historical runs and very low variance (CV=2.1%)
    • Regression Confidence: 5/5 - Surgical change directly to TarWriter code paths, extreme z-score of 17.4, exact temporal alignment with commit
    • Likely Cause: ff08290b777d - "Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)"
      • Affected Areas: System.Formats.Tar / TarWriter / TarHeader.Write / ReadWriteAdapter
      • Stack Relevance: Shared
      • Rationale: This commit is a large refactor (251 additions, 1,019 deletions across 14 files) that restructures TarWriter.cs (139 changes), TarHeader.Write.cs (310 changes), TarHeader.Read.cs (222 changes), TarReader.cs (299 changes), and introduces a new ReadWriteAdapter pattern. The sync/async adapter pattern likely introduces additional virtual dispatch, delegate allocation, or adapter overhead in the hot write path. The commit timestamp (2026-06-29T09:51:56) exactly matches the regression date, and all four TarWriter entry type benchmarks regressed simultaneously — consistent with changes to the shared write infrastructure.
      • Candidates Considered:
        • ff08290b777d - Direct hit: refactors TarWriter sync/async code paths (primary suspect)
        • 664063fb61eb - Devirtualization change in CoreCLR; could theoretically affect virtual method dispatch but too broad to explain Tar-specific regression
    • Verification Status: Not verified — high confidence from static analysis; verification via RunWithBuildAtHash recommended to confirm

    System.Formats.Tar.Tests.Perf_TarWriter.V7TarEntry_WriteEntry (+32.95%)

    • Delta: 187.02 ns → 248.65 ns (+32.95%)
    • Historical Context: μ=185.50 (σ=4.66, n=298); z-score=13.56; adaptive threshold=5.02%
    • Assessment: Clear step change. Before series stable ~177–198 ns (CV=2.5%); all after values cluster ~249–265 ns, no overlap with historical distribution.
    • Finding Confidence: 4/5 - Excellent data quality
    • Regression Confidence: 5/5 - Same rationale as UstarTarEntry; corroborates shared write path regression
    • Likely Cause: ff08290b777d - "Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)"
      • Affected Areas: System.Formats.Tar / TarWriter / TarHeader.Write / ReadWriteAdapter
      • Stack Relevance: Shared
      • Rationale: Same shared TarWriter infrastructure change. V7 entry type uses the same WriteEntry code path that was refactored.
    • Verification Status: Not verified

    System.Formats.Tar.Tests.Perf_TarWriter.GnuTarEntry_WriteEntry (+32.66%)

    • Delta: 205.86 ns → 273.10 ns (+32.66%)
    • Historical Context: μ=207.16 (σ=7.75, n=299); z-score=8.51; adaptive threshold=7.48%
    • Assessment: Step change. Before series had a pre-existing downward drift (~218→201 ns), but all after values (~271–296 ns) are dramatically above even the highest historical values. One outlier at 262 ns in history does not materially affect classification.
    • Finding Confidence: 4/5 - Good data quality; slight downward drift noted but does not affect classification
    • Regression Confidence: 5/5 - Same rationale; GnuSparseStream.cs was also directly modified (163 changes)
    • Likely Cause: ff08290b777d - "Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)"
      • Affected Areas: System.Formats.Tar / TarWriter / TarHeader.Write / GnuSparseStream / ReadWriteAdapter
      • Stack Relevance: Shared
      • Rationale: Same shared write path refactor. Additionally, GnuSparseStream.cs was directly modified (44 additions, 119 deletions) in this commit.
    • Verification Status: Not verified

    System.Formats.Tar.Tests.Perf_TarWriter.PaxTarEntry_WriteEntry (+12.82%)

    • Delta: 1357.80 ns → 1531.90 ns (+12.82%)
    • Historical Context: μ=1362.07 (σ=21.26, n=298); z-score=7.99; adaptive threshold=5.00%
    • Assessment: Clear step change. Before series very stable ~1320–1430 ns (CV=1.6%); all after values cluster ~1518–1562 ns, entirely disjoint. Smaller relative delta than the other three tests likely because PaxTarEntry_WriteEntry involves additional PAX extended header work that dilutes the relative impact of the adapter overhead.
    • Finding Confidence: 4/5 - Excellent data quality with very low variance
    • Regression Confidence: 5/5 - Same rationale as other tests; consistent with shared write path regression
    • Likely Cause: ff08290b777d - "Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)"
      • Affected Areas: System.Formats.Tar / TarWriter / TarHeader.Write / ReadWriteAdapter
      • Stack Relevance: Shared
      • Rationale: Same shared write path refactor. PAX entries do more work per WriteEntry call (extended headers), explaining the smaller relative regression (+12.8% vs +33–41% for other types).
    • Verification Status: Not verified

    Open Questions

    • GnuTarEntry Pre-existing Drift: Before series shows a downward drift (~218→201 ns) prior to the regression; this is unrelated to the current step change and may warrant separate follow-up.
    • Adapter Pattern Overhead: The ReadWriteAdapter refactor likely introduces virtual dispatch or delegate indirection in the sync write path. Source review of the adapter implementation would clarify the exact overhead mechanism.
    • Verification Needed: While static analysis gives very high confidence, verification via RunWithBuildAtHash on the baseline commit (c366a53) vs the compare commit (ff08290) would provide definitive confirmation.

    Recommended Actions

    1. Source review: Manual code review of ff08290b777d focusing on the ReadWriteAdapter pattern and its impact on the synchronous TarWriter.WriteEntry hot path — look for added virtual dispatch, delegate allocation, or unnecessary abstraction layers in the sync code path.
    2. Verification: Verify regression with RunWithBuildAtHash on baseline c366a5387287 vs suspected commit ff08290b777d.
    3. Profiling: Collect hardware counters (branch mispredictions, instructions retired) with collectHardwareCounters=true to quantify the overhead introduced by the adapter pattern.
    4. Follow-up: File issue against PR Refactor sync/async implementations by using Adapter pattern #129282 to address performance regression in synchronous TarWriter paths.

    Bisection Results

    Test Status Culprit Commit Relevance Reported Diff (issue) Bisected Diff (at culprit) Duration
    UstarTarEntry_WriteEntry ✅ Success ff08290b777d ✅ RELATED 191.8→269.8 ns (+41%) 199.2→266.6 ns (+34%) 3896s
    V7TarEntry_WriteEntry ✅ Success ff08290b777d ✅ RELATED 187.0→248.6 ns (+33%) 184.2→252.0 ns (+37%) 662s
    GnuTarEntry_WriteEntry ✅ Success ff08290b777d ✅ RELATED 205.9→273.1 μs (+33%) 199.6→302.7 μs (+52%) 1002s
    PaxTarEntry_WriteEntry ✅ Success ff08290b777d ✅ RELATED 1.4→1.5 μs (+13%) 1.3→1.5 μs (+13%) 670s

    Total bisection time: 6231s
    Success rate: 4/4 tests

    System.Formats.Tar.Tests.Perf_TarWriter.UstarTarEntry_WriteEntry

    • Culprit: ff08290b777d
    • Commit Message: Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)
    • Relevance: RELATED
    • Explanation: The commit directly modifies TarWriter.cs, TarHeader.Write.cs, TarReader.cs, and other files in the System.Formats.Tar namespace. The benchmark tests UstarTarEntry_WriteEntry which exercises TarWriter write paths — exactly the code refactored by this PR. The adapter pattern refactoring likely introduced overhead in the sync write path.

    Helix jobs

    Job ID Submitted by bisection Last status
    c6020ce6-5441-4c5d-a10f-a45a8c40445c ✅ submitted
    88a2af83-f320-4b6b-94c4-db99c6de925b ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    0dca32569f45 mid 269.231 ns
    17be26d18c72 quarter 191.436 ns
    c366a5387287 last-good 199.189 ns
    ff08290b777d culprit 266.562 ns
    Bisection details

    …[earlier output omitted]

    │ import json, sys
    │ data = json.load(sys.stdin)
    │ report = json.loads(data['report'])
    │ benchmarks = report.get('Benchmarks', [])…
    └ 15 lines…

    Narrowed down:

    • c366a53: 199.19 → ✓ baseline (within 2σ of 191.81)
    • ff08290: 266.56 → ✗ regressed

    Culprit is ff08290. Let me examine it.

    ● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "ff08290b…
    └ {"index":null,"sha":"ff08290b777d78822cc4505bf7308d33b9735705","abbrevSha":"f...

    This is clearly RELATED. The commit "Refactor sync/async implementations by using Adapter pattern (#129282)" directly modifies System.Formats.Tar files including TarWriter.cs, TarHeader.Write.cs, and more — the exact code path exercised by the UstarTarEntry_WriteEntry benchmark.

    BISECTION RESULT:
- Culprit Commit: ff08290b777d78822cc4505bf7308d33b9735705
- Commit Message: Refactor sync/async implementations by using Adapter pattern (#129282)
- Commits Tested: 4
- Confidence: 4
- Commit Relevance: RELATED
- Relevance Explanation: The commit directly modifies TarWriter.cs, TarHeader.Write.cs, TarReader.cs, and other files in the System.Formats.Tar namespace. The benchmark tests UstarTarEntry_WriteEntry which exercises TarWriter write paths — exactly the code refactored by this PR. The adapter pattern refactoring likely introduced overhead in the sync write path.
- Baseline Value: 199.189 ns
- Culprit Value: 266.562 ns
- Summary: Bisected 10 commits in 2 Helix jobs. The regression was introduced at ff08290b which refactors System.Formats.Tar sync/async implementations using the Adapter pattern, directly impacting the TarWriter write path measured by this benchmark.

HELIX JOBS:
- c6020ce6-5441-4c5d-a10f-a45a8c40445c
- 88a2af83-f320-4b6b-94c4-db99c6de925b

MEASUREMENTS:
- 17be26d18c72 (quarter): 191.436 ns
- 0dca32569f45 (mid): 269.231 ns
- c366a5387287 (last-good): 199.189 ns
- ff08290b777d (culprit): 266.562 ns

    System.Formats.Tar.Tests.Perf_TarWriter.V7TarEntry_WriteEntry

    • Culprit: ff08290b777d
    • Commit Message: Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)
    • Relevance: RELATED
    • Explanation: The commit directly modifies System.Formats.Tar source files including TarWriter.cs, TarHeader.Write.cs, TarReader.cs, and TarHelpers.cs, refactoring sync/async implementations using the Adapter pattern. The regressed test V7TarEntry_WriteEntry exercises TarWriter, which was significantly changed (38 additions, 101 deletions). The refactoring introduced an abstraction layer (ReadWriteAdapter) that likely adds overhead to the hot write path.

    Helix jobs

    Job ID Submitted by bisection Last status
    ff4e3113-43c2-4894-b8b8-ecd208b4d7ab ✅ submitted
    61e5ca7b-a995-4e28-b4f3-3d0bb85bd2a1 ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    0dca32569f45 mid 250.170 ns
    17be26d18c72 quarter 179.630 ns
    c366a5387287 last-good 184.230 ns
    ff08290b777d culprit 251.970 ns
    Bisection details

    …[earlier output omitted]

    • ff08290 (index 4): 251.97 ns ✗ (regression present)

    The culprit is ff08290b777d78822cc4505bf7308d33b9735705. Let me examine it.

    ● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "ff08290b…
    └ {"index":null,"sha":"ff08290b777d78822cc4505bf7308d33b9735705","abbrevSha":"f...

    This is a clear match. The commit ff08290b — "Refactor sync/async implementations by using Adapter pattern (#129282)" — directly modifies System.Formats.Tar source files including TarWriter.cs, TarHeader.Write.cs, and others. The test V7TarEntry_WriteEntry exercises TarWriter, so this commit is directly related.

    The step change is large and unambiguous: 184.23 → 251.97 ns (~37%), well above the 2σ noise floor of 9.32 ns.

    BISECTION RESULT:
- Culprit Commit: ff08290b777d78822cc4505bf7308d33b9735705
- Commit Message: Refactor sync/async implementations by using Adapter pattern (#129282)
- Commits Tested: 4
- Confidence: 5
- Commit Relevance: RELATED
- Relevance Explanation: The commit directly modifies System.Formats.Tar source files including TarWriter.cs, TarHeader.Write.cs, TarReader.cs, and TarHelpers.cs, refactoring sync/async implementations using the Adapter pattern. The regressed test V7TarEntry_WriteEntry exercises TarWriter, which was significantly changed (38 additions, 101 deletions). The refactoring introduced an abstraction layer (ReadWriteAdapter) that likely adds overhead to the hot write path.
- Baseline Value: 184.23 ns
- Culprit Value: 251.97 ns
- Summary: Bisected 10 commits in 2 Helix jobs. Clear step change from 184.23 ns to 251.97 ns (~37% regression) at commit ff08290b which refactored TarWriter sync/async code using the Adapter pattern.

HELIX JOBS:
- ff4e3113-43c2-4894-b8b8-ecd208b4d7ab
- 61e5ca7b-a995-4e28-b4f3-3d0bb85bd2a1

MEASUREMENTS:
- 17be26d18c72 (quarter): 179.63 ns
- 0dca32569f45 (mid): 250.17 ns
- c366a5387287 (last-good): 184.23 ns
- ff08290b777d (culprit): 251.97 ns

    System.Formats.Tar.Tests.Perf_TarWriter.GnuTarEntry_WriteEntry

    • Culprit: ff08290b777d
    • Commit Message: Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)
    • Relevance: RELATED
    • Explanation: This commit directly modifies System.Formats.Tar code including TarWriter.cs, TarHeader.Write.cs, TarHeader.Read.cs, TarReader.cs, and GnuSparseStream.cs. It refactors sync/async implementations using an Adapter pattern, which changes the exact code path exercised by the GnuTarEntry_WriteEntry benchmark. The large refactoring (251 additions, 1019 deletions across 14 files) introduces an abstraction layer that likely adds overhead to the synchronous write path.

    Helix jobs

    Job ID Submitted by bisection Last status
    13febc8a-a6bb-4327-803a-1d0d17bc0021 ✅ submitted
    2c22e185-ad23-4dcb-9993-70d301678552 ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    0d3df07c6048 baseline 204.410 μs
    17be26d18c72 q1 200.000 μs
    0dca32569f45 mid 272.680 μs
    93ec56ee6c38 head 272.220 μs
    c366a5387287 last-good 199.570 μs
    ff08290b777d culprit 302.690 μs
    Bisection details

    …[earlier output omitted]

    └ {"index":null,"sha":"ff08290b777d78822cc4505bf7308d33b9735705","abbrevSha":"f...

    This is a very clear match. The commit "Refactor sync/async implementations by using Adapter pattern (#129282)" directly modifies System.Formats.Tar code including TarWriter.cs, TarHeader.Write.cs, and other Tar-related files. This is exactly the code path exercised by the GnuTarEntry_WriteEntry benchmark.

    All measurements are well beyond the noise floor (delta ~100μs vs 2σ ~15.5μs), so this is a confident bisection.

    Here are my findings:

    BISECTION RESULT:
- Culprit Commit: ff08290b777d78822cc4505bf7308d33b9735705
- Commit Message: Refactor sync/async implementations by using Adapter pattern (#129282)
- Commits Tested: 7
- Confidence: 5
- Commit Relevance: RELATED
- Relevance Explanation: This commit directly modifies System.Formats.Tar code including TarWriter.cs, TarHeader.Write.cs, TarHeader.Read.cs, TarReader.cs, and GnuSparseStream.cs. It refactors sync/async implementations using an Adapter pattern, which changes the exact code path exercised by the GnuTarEntry_WriteEntry benchmark. The large refactoring (251 additions, 1019 deletions across 14 files) introduces an abstraction layer that likely adds overhead to the synchronous write path.
- Baseline Value: 199.57 μs
- Culprit Value: 302.69 μs
- Summary: Bisected 10 commits between 0d3df07 and 93ec56e. Double bisection in round 1 narrowed the regression to commits c366a53–0dca325, and round 2 pinpointed ff08290 as the culprit. The regression (~50% increase) is well above the noise floor and directly caused by a Tar library refactoring.

HELIX JOBS:
- 13febc8a-a6bb-4327-803a-1d0d17bc0021
- 2c22e185-ad23-4dcb-9993-70d301678552

MEASUREMENTS:
- 0d3df07c6048 (baseline): 204.41 μs
- 17be26d18c72 (Q1): 200.00 μs
- 0dca32569f45 (mid): 272.68 μs
- 93ec56ee6c38 (head): 272.22 μs
- c366a53872872 (last-good): 199.57 μs
- ff08290b777d (culprit): 302.69 μs

    System.Formats.Tar.Tests.Perf_TarWriter.PaxTarEntry_WriteEntry

    • Culprit: ff08290b777d
    • Commit Message: Refactor sync/async implementations by using Adapter pattern (Refactor sync/async implementations by using Adapter pattern #129282)
    • Relevance: RELATED
    • Explanation: The commit directly modifies TarWriter.cs, TarHeader.Write.cs, TarReader.cs, and other System.Formats.Tar source files, refactoring sync/async implementations using an Adapter pattern. This is exactly the code path exercised by the PaxTarEntry_WriteEntry benchmark. The refactoring changed 310 lines in TarHeader.Write.cs alone, making it highly plausible as the cause.

    Helix jobs

    Job ID Submitted by bisection Last status
    1211437d-b01a-47d0-b56a-761026594355 ✅ submitted
    896c2e72-85f7-4935-9808-d5cc69bd89a7 ✅ submitted

    Per-commit measurements

    Commit Position Measurements
    c366a5387287 index 3 1.350 μs
    ff08290b777d index 4 1.528 μs
    0dca32569f45 mid 1.543 μs
    17be26d18c72 quarter 1.358 μs
    Bisection details

    …[earlier output omitted]

    ● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "ff08290b…
    └ {"index":null,"sha":"ff08290b777d78822cc4505bf7308d33b9735705","abbrevSha":"f...

    This is a very clear match. The commit "Refactor sync/async implementations by using Adapter pattern (#129282)" directly modifies System.Formats.Tar code including TarWriter.cs, TarHeader.Write.cs, and related files — exactly the code path exercised by the PaxTarEntry_WriteEntry benchmark.

    Now let me compile the final result. Here's a summary of all measurements:

    BISECTION RESULT:
- Culprit Commit: ff08290b777d78822cc4505bf7308d33b9735705
- Commit Message: Refactor sync/async implementations by using Adapter pattern (#129282)
- Commits Tested: 4
- Confidence: 3
- Commit Relevance: RELATED
- Relevance Explanation: The commit directly modifies TarWriter.cs, TarHeader.Write.cs, TarReader.cs, and other System.Formats.Tar source files, refactoring sync/async implementations using an Adapter pattern. This is exactly the code path exercised by the PaxTarEntry_WriteEntry benchmark. The refactoring changed 310 lines in TarHeader.Write.cs alone, making it highly plausible as the cause.
- Baseline Value: 1349.99 ns   (measured at c366a53872872cbcafd5145a23038659ac3815c3)
- Culprit Value: 1528.09 ns    (measured at ff08290b777d78822cc4505bf7308d33b9735705)
- Summary: Bisected a 10-commit range using double bisection. The regression (+13.2%) was introduced by a refactoring of sync/async implementations in System.Formats.Tar using the Adapter pattern. Confidence capped at 3/5 due to noisy single-sample measurements (historical CV=1.56%, 2σ noise floor ~42.5 ns vs observed delta ~178 ns, but the delta well exceeds noise).

HELIX JOBS:
- 1211437d-b01a-47d0-b56a-761026594355
- 896c2e72-85f7-4935-9808-d5cc69bd89a7

MEASUREMENTS:
- 17be26d18c72 (quarter): 1358.48 ns
- 0dca32569f45 (mid): 1542.65 ns
- c366a5387287 (index 3): 1349.99 ns
- ff08290b777d (index 4): 1528.09 ns

    🤖 Proposed Next Actions

    Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

    Verified regression attributed to commit ff08290 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

    # Action Status Details
    1 Back-reference cause PR ⏭️ Skipped (dry-run) on culprit PR for ff08290b777d
    2 Transfer issue ⏭️ Skipped (dry-run) → dotnet/runtime
    3 Ping owners ⏭️ Skipped (dry-run) no owners resolved
    4 Remove label ⏭️ Skipped (dry-run) untriaged

    🦆 Pre-bisection Re-analysis (advisory)

    ⚠️ Disputes the pre-bisection plan — advisory only, not applied

    Flags: wasted-bisection-slot

    Rationale: All four findings share the same likelyCommit (ff08290), the same stack relevance (Shared), and identical confidence scores, meaning they almost certainly regressed together from the same code change. Bisecting all four is redundant — two representative tests (e.g. one older format like V7 and one extended format like Pax) are sufficient to confirm the commit, saving ~120 Helix minutes.

    Suggested drop from bisection: System.Formats.Tar.Tests.Perf_TarWriter.UstarTarEntry_WriteEntry, System.Formats.Tar.Tests.Perf_TarWriter.GnuTarEntry_WriteEntry (advisory — not applied)

    This critique is advisory in the current configuration and did not change which tests were bisected.

    🦆 Re-analysis (advisory)

    ✅ Agrees with the analysis — advisory only, not applied

    Rationale: All four bisections converged successfully on the same culprit commit ff08290 with no alignment warnings, covering four related TarWriter benchmarks with consistent RELATED relevance and shared stack relevance. The evidence strongly supports both the 5/5 change confidence and the 5/5 finding confidence.

    This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

    Generated by PerfTriageAgent on 2026-06-30 16:25 UTC

  3. LoopedBard3
    transferred this issue fromdotnet/perf-autofiling-issueson Jun 30, 2026

  4. LoopedBard3 commented on Jun 30, 2026

    @LoopedBard3
    LoopedBard3
    on Jun 30, 2026
    Member

    Likely due to #129282. FYI @alinpahontu2912.

    Related regressions:

  5. github-actions
    added
    area-System.Formats.Tar
    on Jun 30, 2026

  6. dotnet-policy-service commented on Jun 30, 2026

    @dotnet-policy-service
    dotnet-policy-serviceBot
    on Jun 30, 2026 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @dotnet/area-system-formats-tar
    See info in area-owners.md if you want to be subscribed.

  7. LoopedBard3
    added
    tenet-performancePerformance related issue
    tenet-performance-benchmarksIssue from performance benchmark
    and removed
    PGO
    on Jun 30, 2026
  8. rosebyte
    added this to the 11.0.0 milestone on Jul 2, 2026
  9. rosebyte
    removed
    untriagedNew issue has not been triaged by the area owner
    on Jul 2, 2026

  10. MihaZupan commented on Jul 2, 2026

    @MihaZupan
    MihaZupan
    on Jul 2, 2026
    Member

    @MihuBot benchmark Perf_TarWriter 0d3df07...93ec56e -medium

  11. MihuBot commented on Jul 2, 2026

    @MihuBot
    MihuBot
    on Jul 2, 2026
    System.Formats.Tar.Tests.Perf_TarWriter 
    BenchmarkDotNet v0.16.0-nightly.20260518.1249, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
AMD EPYC 9V74 3.52GHz, 1 CPU, 4 logical and 2 physical cores
Memory: 15.61 GB Total, 6.99 GB Available
MediumRun : .NET 11.0.0 (11.0.0-dev, 42.42.42.42424), X64 RyuJIT x86-64-v4
Job=MediumRun  OutlierMode=Default  IterationCount=15
LaunchCount=2  MemoryRandomization=Default  WarmupCount=10
    Method Toolchain Mean Error Ratio Allocated Alloc Ratio
    V7TarEntry_WriteEntry 664063fb61 186.3 ns 1.54 ns 1.00 464 B 1.00
    V7TarEntry_WriteEntry 17be26d18c 188.9 ns 2.32 ns 1.01 464 B 1.00
    V7TarEntry_WriteEntry c366a53872 195.2 ns 4.86 ns 1.05 464 B 1.00
    V7TarEntry_WriteEntry ff08290b77 264.8 ns 1.41 ns 1.42 464 B 1.00
    V7TarEntry_WriteEntry 0dca32569f 271.6 ns 5.25 ns 1.46 464 B 1.00
    V7TarEntry_WriteEntry 185a5f29ab 264.4 ns 2.86 ns 1.42 464 B 1.00
    V7TarEntry_WriteEntry 8d10d8c604 297.7 ns 24.23 ns 1.60 464 B 1.00
    V7TarEntry_WriteEntry e137b2c018 265.2 ns 2.51 ns 1.42 464 B 1.00
    V7TarEntry_WriteEntry 93ec56ee6c 267.3 ns 2.19 ns 1.43 464 B 1.00
    V7TarEntry_WriteEntry_Async 664063fb61 285.8 ns 1.78 ns 1.00 464 B 1.00
    V7TarEntry_WriteEntry_Async 17be26d18c 290.6 ns 6.89 ns 1.02 464 B 1.00
    V7TarEntry_WriteEntry_Async c366a53872 282.7 ns 1.42 ns 0.99 464 B 1.00
    V7TarEntry_WriteEntry_Async ff08290b77 286.8 ns 1.83 ns 1.00 464 B 1.00
    V7TarEntry_WriteEntry_Async 0dca32569f 284.7 ns 0.84 ns 1.00 464 B 1.00
    V7TarEntry_WriteEntry_Async 185a5f29ab 286.7 ns 1.30 ns 1.00 464 B 1.00
    V7TarEntry_WriteEntry_Async 8d10d8c604 288.9 ns 2.40 ns 1.01 464 B 1.00
    V7TarEntry_WriteEntry_Async e137b2c018 295.4 ns 8.64 ns 1.03 464 B 1.00
    V7TarEntry_WriteEntry_Async 93ec56ee6c 292.0 ns 3.99 ns 1.02 464 B 1.00
    UstarTarEntry_WriteEntry 664063fb61 190.7 ns 0.65 ns 1.00 464 B 1.00
    UstarTarEntry_WriteEntry 17be26d18c 194.9 ns 2.37 ns 1.02 464 B 1.00
    UstarTarEntry_WriteEntry c366a53872 193.9 ns 2.58 ns 1.02 464 B 1.00
    UstarTarEntry_WriteEntry ff08290b77 274.9 ns 2.74 ns 1.44 464 B 1.00
    UstarTarEntry_WriteEntry 0dca32569f 269.0 ns 4.93 ns 1.41 464 B 1.00
    UstarTarEntry_WriteEntry 185a5f29ab 271.1 ns 5.41 ns 1.42 464 B 1.00
    UstarTarEntry_WriteEntry 8d10d8c604 269.8 ns 3.58 ns 1.42 464 B 1.00
    UstarTarEntry_WriteEntry e137b2c018 267.9 ns 3.28 ns 1.41 464 B 1.00
    UstarTarEntry_WriteEntry 93ec56ee6c 277.9 ns 9.17 ns 1.46 464 B 1.00
    UstarTarEntry_WriteEntry_Async 664063fb61 291.8 ns 1.78 ns 1.00 464 B 1.00
    UstarTarEntry_WriteEntry_Async 17be26d18c 298.8 ns 6.02 ns 1.02 464 B 1.00
    UstarTarEntry_WriteEntry_Async c366a53872 292.0 ns 1.13 ns 1.00 464 B 1.00
    UstarTarEntry_WriteEntry_Async ff08290b77 296.9 ns 2.13 ns 1.02 464 B 1.00
    UstarTarEntry_WriteEntry_Async 0dca32569f 309.3 ns 12.81 ns 1.06 464 B 1.00
    UstarTarEntry_WriteEntry_Async 185a5f29ab 311.4 ns 5.97 ns 1.07 464 B 1.00
    UstarTarEntry_WriteEntry_Async 8d10d8c604 305.8 ns 2.54 ns 1.05 464 B 1.00
    UstarTarEntry_WriteEntry_Async e137b2c018 298.5 ns 0.75 ns 1.02 464 B 1.00
    UstarTarEntry_WriteEntry_Async 93ec56ee6c 298.0 ns 1.00 ns 1.02 464 B 1.00
    PaxTarEntry_WriteEntry 664063fb61 1,394.8 ns 18.96 ns 1.00 2208 B 1.00
    PaxTarEntry_WriteEntry 17be26d18c 1,418.4 ns 9.89 ns 1.02 2208 B 1.00
    PaxTarEntry_WriteEntry c366a53872 1,403.7 ns 5.29 ns 1.01 2208 B 1.00
    PaxTarEntry_WriteEntry ff08290b77 1,562.8 ns 17.39 ns 1.12 2208 B 1.00
    PaxTarEntry_WriteEntry 0dca32569f 1,608.4 ns 22.24 ns 1.15 2208 B 1.00
    PaxTarEntry_WriteEntry 185a5f29ab 1,602.2 ns 24.91 ns 1.15 2208 B 1.00
    PaxTarEntry_WriteEntry 8d10d8c604 1,578.2 ns 16.55 ns 1.13 2208 B 1.00
    PaxTarEntry_WriteEntry e137b2c018 1,589.6 ns 12.65 ns 1.14 2208 B 1.00
    PaxTarEntry_WriteEntry 93ec56ee6c 1,560.0 ns 14.52 ns 1.12 2208 B 1.00
    PaxTarEntry_WriteEntry_Async 664063fb61 1,679.7 ns 15.27 ns 1.00 2208 B 1.00
    PaxTarEntry_WriteEntry_Async 17be26d18c 1,651.9 ns 5.92 ns 0.98 2208 B 1.00
    PaxTarEntry_WriteEntry_Async c366a53872 1,673.7 ns 10.00 ns 1.00 2208 B 1.00
    PaxTarEntry_WriteEntry_Async ff08290b77 1,655.8 ns 8.72 ns 0.99 2208 B 1.00
    PaxTarEntry_WriteEntry_Async 0dca32569f 1,648.7 ns 6.66 ns 0.98 2208 B 1.00
    PaxTarEntry_WriteEntry_Async 185a5f29ab 1,648.0 ns 7.88 ns 0.98 2208 B 1.00
    PaxTarEntry_WriteEntry_Async 8d10d8c604 1,666.4 ns 10.70 ns 0.99 2208 B 1.00
    PaxTarEntry_WriteEntry_Async e137b2c018 1,650.9 ns 7.99 ns 0.98 2208 B 1.00
    PaxTarEntry_WriteEntry_Async 93ec56ee6c 1,678.0 ns 12.30 ns 1.00 2208 B 1.00
    GnuTarEntry_WriteEntry 664063fb61 201.0 ns 0.93 ns 1.00 464 B 1.00
    GnuTarEntry_WriteEntry 17be26d18c 206.5 ns 2.72 ns 1.03 464 B 1.00
    GnuTarEntry_WriteEntry c366a53872 204.0 ns 1.59 ns 1.01 464 B 1.00
    GnuTarEntry_WriteEntry ff08290b77 273.7 ns 2.19 ns 1.36 464 B 1.00
    GnuTarEntry_WriteEntry 0dca32569f 278.9 ns 1.19 ns 1.39 464 B 1.00
    GnuTarEntry_WriteEntry 185a5f29ab 290.3 ns 10.23 ns 1.44 464 B 1.00
    GnuTarEntry_WriteEntry 8d10d8c604 274.6 ns 2.15 ns 1.37 464 B 1.00
    GnuTarEntry_WriteEntry e137b2c018 276.8 ns 2.53 ns 1.38 464 B 1.00
    GnuTarEntry_WriteEntry 93ec56ee6c 274.8 ns 1.42 ns 1.37 464 B 1.00
    GnuTarEntry_WriteEntry_Async 664063fb61 294.3 ns 1.11 ns 1.00 464 B 1.00
    GnuTarEntry_WriteEntry_Async 17be26d18c 300.7 ns 3.85 ns 1.02 464 B 1.00
    GnuTarEntry_WriteEntry_Async c366a53872 293.7 ns 1.33 ns 1.00 464 B 1.00
    GnuTarEntry_WriteEntry_Async ff08290b77 301.0 ns 1.08 ns 1.02 464 B 1.00
    GnuTarEntry_WriteEntry_Async 0dca32569f 301.9 ns 1.20 ns 1.03 464 B 1.00
    GnuTarEntry_WriteEntry_Async 185a5f29ab 301.7 ns 0.90 ns 1.03 464 B 1.00
    GnuTarEntry_WriteEntry_Async 8d10d8c604 304.7 ns 2.10 ns 1.04 464 B 1.00
    GnuTarEntry_WriteEntry_Async e137b2c018 301.0 ns 2.31 ns 1.02 464 B 1.00
    GnuTarEntry_WriteEntry_Async 93ec56ee6c 303.1 ns 1.59 ns 1.03 464 B 1.00
  12. rzikm
    added the
    Bug
    issue type on Jul 7, 2026
  13. rosebyte
    assigned
    iremyux
    on Jul 30, 2026
  14. iremyux
    closed this as completedin #131993on Aug 10, 2026
  15. jkoritzinsky
    added a commit that references this issue on Aug 10, 2026

    Restore synchronous TarWriter fast path (dotnet#131993)

    19ea048
  16. jtschuster
    added a commit that references this issue on Aug 11, 2026

    Restore synchronous TarWriter fast path (dotnet#131993)

    9304c03
  17. github-actions
    locked and limited conversation to collaborators on Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-x64area-System.Formats.Taros-linuxLinux OS (any supported distro)runtime-coreclrspecific to the CoreCLR runtimetenet-performancePerformance related issuetenet-performance-benchmarksIssue from performance benchmark

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions