Single file start time regression

With SDK builds taken between 4/8 and 4/10 we noticed a regression in startup time for applications published as single files.

Environment:

This was reproduced only on Windows (Linux is not affected) and can still be consistently. Same machine, Windows Defender disabled on the folder starting the app. Same parent folder between runs, though any AV should happen the same way on both builds.
Not specific to a single app
Here showing ASP.NET application without DI or middleware (Platform benchmark)

SDK versions:
10.0.100-preview.4.25207.27
10.0.100-preview.4.25209.44

black: single file
blue: trimmed
yellow: aot
pink: self-contained

I have the executables built for the two SDKs. Contact me on Teams if you want them.

Crank command lines for reference:

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml --scenario json --profile aspnet-citrine-win --application.buildArguments "/p:PublishReadyToRun=true /p:PublishSingleFile=true" --application.framework net10.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 10.0.0-preview.4.25181.4 --application.runtimeVersion 10.0.0-preview.4.25174.9 --application.sdkVersion 10.0.100-preview.4.25207.27 --profile short --application.options.fetch true

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/platform.benchmarks.yml --scenario json --profile aspnet-citrine-win --application.buildArguments "/p:PublishReadyToRun=true /p:PublishSingleFile=true" --application.framework net10.0 --application.options.collectCounters true --load.options.reuseBuild true --application.aspNetCoreVersion 10.0.0-preview.4.25208.1 --application.runtimeVersion 10.0.0-preview.4.25209.9 --application.sdkVersion 10.0.100-preview.4.25209.44 --profile short --application.options.fetch true

Presumably that's x64, right? (ie. not x86, not arm64)

@filipnavara

@filipnavara In this case INTEL X64 Windows.

@agocke

Tagging subscribers to this area: @agocke, @vitek-karas, @VSadov
See info in area-owners.md if you want to be subscribed.

Slightly narrowed SDK versions:

10.0.100-preview.4.25208.4: 186ms
10.0.100-preview.4.25208.7: 512ms

singlefilehost is grabbed from the SDK known apphost packs, so that seems to be the version that matters here. Unfortunately, that is still a big range for runtime: 10.0.0-preview.4.25174.9 to 10.0.0-preview.4.25207.7 - 659321f...9559b2a

I haven't figured out how to get a repro that isn't the full build and application run through crank though - which presumably I'd need to try any sort of fix.

When I unzipped the downloaded applications (from application.options.fetch) and then kickoff jobs to just run the executables, I see the same start time for both versions: ~500ms

This is my config:

# crank --config crank-config.yml --scenario json-good --profile aspnet-citrine-win --profile short
# crank --config crank-config.yml --scenario json-bad --profile aspnet-citrine-win --profile short

# readyStateText, arguments and environmentVariables copied from platformbenchmarks
platformbenchmarks-good:
    source:
      localFolder: C:\issues\perf\good-25208.4
    executable: PlatformBenchmarks.exe

platformbenchmarks-bad:
    source:
      localFolder: C:\issues\perf\bad-25208.7
    executable: PlatformBenchmarks.exe

scenarios:
  json-good:
    application:
      job: platformbenchmarks-good

  json-bad:
    application:
      job: platformbenchmarks-bad

I also tried to do a local version of crank's time to readyStateText measurement. I also saw the same elapsed times for both the 'good' and 'bad' versions.

using var process = new Process { StartInfo = startInfo };
var stopwatch = new Stopwatch();
bool found = false;

string readyText = "Application started.";
process.OutputDataReceived += (sender, e) =>
{
    if (e.Data != null && e.Data.StartsWith(readyText))
    {
        stopwatch.Stop();
        found = true;
    }
};

Console.WriteLine($"{startInfo.FileName}");
stopwatch.Start();
process.Start();
process.BeginOutputReadLine();
while(!found)
{
    Thread.Sleep(100);
}
process.Kill();
Console.WriteLine($"  Start time (ms): {stopwatch.ElapsedMilliseconds}");

I passed --application.collect true --application.collectStartup true to get traces (for the full build / run jobs, where this repros). Unfortunately, >70% of the stacks are reported as broken for both traces.

When I unzipped the downloaded applications (from application.options.fetch) and then kickoff jobs to just run the executables, I see the same start time for both versions: ~500ms

Another thing to note is that the 500ms from just running the output from the jobs from the base and regressed versions is on par with the reported regressed time when the job is to build/publish and run. It almost seems like a first run thing (single-file does pass assemblies to AMSI), but I'm not sure how that would happen.

Locally, if I try a similar build+run of the PlatformBenchmarks project and measure two executions of the output as in #116318 (comment), the first run is ~550ms and the second ~200ms. But I see the same thing for both 25208.4 and 25208.7

@sebastienros

@sebastienros Is there any way to make crank use/upload a local SDK or patch a file in the one it downloads?

I haven't been able to figure out a local repro or get traces that aren't largely broken stacks (they show a difference in blocked time, but the stacks are too broken to see from what), but I think I may have narrowed it down a bit via trying different jobs with crank.

PlatformBenchmarks project with a commit that allows updating the KnownAppHostPack 10.0 version (corresponds to the singlefilehost that is used)
Ran jobs for the 10.0.0-preview.4.25207.7 runtime libraries and singlefilehost with the 10.0.100-preview.4.25208.4 and 10.0.100-preview.4.25208.7 SDKs - verified that both were using the 10.0.0-preview.4.25207.7 runtime libraries and singlefilehost
Saw the start time regression with just the SDK version difference.

That leaves the SDK itself as the only difference - dotnet/sdk@e378cb0...d4bbb19 - the dotnet/runtime update looks like the only interesting change. The only thing I could see that was changed there was Microsoft.NET.HostModel in edb0a06. That changed to writing out a temporary file for the bundle and then copying over the existing file (instead of updating it in place). I have no idea why that would change the start time on Windows, but it is the one thing I see that is shipped via the SDK and only affect single-file. That behaviour is also going to be changing with #116659 (cc @jtschuster)

If there's some way to kick off a crank job with a local/patched SDK, we could see if Microsoft.NET.HostModel is actually the difference in those runs.

Based on the above, it sounds like this regression may be specific to the startup of a just created app. That is, the first run after the app has just been built. This isn't as high a priority as the startup time of a shipping application and they don't necessarily correspond due to differences in signing/virus scanning.

@agocke

Tagging subscribers to this area: @agocke, @jeffschwMSFT, @elinor-fung
See info in area-owners.md if you want to be subscribed.

I confirmed that this scenario is only first-run of a just-created app. It turns out this is from Windows Code Integrity checks. The perf lab machines appear to have that enabled, which would explain why I could not repro this locally as I do not. There is no effect on the startup of a deployed application (not just created on the machine). The cold start of a just-created app slows down, but the warm run of that is much improved. Closing this - real deployed apps are not affected and this is actually an improvement for just-created apps after the first run.

More details:

With Windows Code Integrity enabled, the entire image is hashed at startup (as part of CreateProcess). In order to do this, any dirty/modified pages must be flushed. The bundler was switched to writing the final single-file app via a file stream write rather than a memory-mapped write. The file stream write meant that much of the file is left dirty and must be reconciled on startup - with the memory-mapped write this was not necessary. However, the switch to the file stream write also meant that the results of the Code Integrity were cached - after the first run, the cost of the check goes away. With the memory-mapped write, the check was not cached and re-ran every app run. With copying over the app (as would be in shipped apps), this difference disappears - the check occurs on first-run and is cached.

Runs were all on aspnet-gold-win using the json scenario.

Running a just-created app:

Version	Cold (1st run)	Warm (2nd+ run)
10.0.100-preview.4.25207.27	2378 ms	2374 ms
10.0.100-preview.4.25209.44	2720 ms	164 ms
11.0.100-preview.6.26359.118	2710 ms	145 ms

With tracing on (ci.dll is Windows Code Integrity):

Version	Run	start-time	ci.dll samples
10.0.100-preview.4.25207.27	cold	3167 ms	1760
10.0.100-preview.4.25207.27	warm	3147 ms	1789
10.0.100-preview.4.25209.44	cold	3479 ms	1793
10.0.100-preview.4.25209.44	warm	266 ms	0
11.0.100-preview.6.26359.118	cold	3485 ms	1851
11.0.100-preview.6.26359.118	warm	269 ms	0

Running a copied app shows no difference - real deployment scenarios:

Version	Cold	Warm
11.0.100-preview.6.26359.118	2694 ms	144 ms
10.0.100-preview.4.25207.27	2712 ms	145 ms

Navigation Menu

Repository navigation

Single file start time regression #116318

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Single file start time regression #116318

Description

Activity

filipnavara commented on Jun 4, 2025

sebastienros commented on Jun 4, 2025

dotnet-policy-service commented on Jun 4, 2025

elinor-fung commented on Jun 20, 2025

elinor-fung commented on Jun 20, 2025

elinor-fung commented on Jun 23, 2025

elinor-fung commented on Jun 24, 2025

agocke commented on Jul 17, 2025

dotnet-policy-service commented on Apr 8, 2026

elinor-fung commented on Aug 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions