dotnet CLI commands no longer work in .NET 7/8 with detached console

Describe the bug

We have some Azure pipelines that run some .NET commands through Wine on Linux agents. Upon trying to upgrade from .NET 6 to .NET 8, we can no longer run any dotnet command from Wine (for example, /opt/wine-stable/bin/wine64 $HOME/win-dotnet/dotnet.exe --version) on our pipelines. Instead we get an exception:

Unhandled exception. System.IO.IOException: Invalid access.
   at System.ConsolePal.SetConsoleOutputEncoding(Encoding enc)
   at System.Console.set_OutputEncoding(Encoding value)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

I'm using dotnet --version as an example in this bug, but it happens for dotnet *anything*.

Running /path/to/wine64 /path/to/dotnet.exe --version from the Azure agent pipeline job works on .NET 5/6
Running /path/to/wine64 /path/to/dotnet.exe --version in a regular Terminal on the same agent works on .NET 5/6/7/8
Native Linux .NET 8 commands work from the pipeline, if dotnet-sdk is installed

In Wine, we get the following warning when running from an Azure agent:
0024:warn:console:console_ioctl Incorrect access to Shell-no-window console (ioctl=50403c)

Which possibly was not a problem until hitting this code that was added: dotnet/sdk@697d62c

To Reproduce

Prereq: an Azure agent that runs on Linux

Install the latest version of wine-stable on the agent
Download and unzip the latest version of .NET 7 or .NET 8 for Windows somewhere on the agent, for example $HOME/win-dotnet
In a pipeline script targeting the agent, run /opt/wine-stable/bin/wine64 $HOME/win-dotnet/dotnet.exe --version
Run pipeline -> hit exception.

Example pipeline file: https://gist.github.com/Wedgetable/e75b08ae1eb3b2fb10cc2a53a9953745

Exceptions (if any)

Unhandled exception. System.IO.IOException: Invalid access.
   at System.ConsolePal.SetConsoleOutputEncoding(Encoding enc)
   at System.Console.set_OutputEncoding(Encoding value)
   at Microsoft.DotNet.Cli.Program.Main(String[] args)

Further technical details

Agent version: 3.232.1
OS: Ubuntu 20.04.6 LTS 64-bit
wine64 version: wine-9.0
.NET SDK for Windows version(s): 7.0.405, 8.0.101

Not sure if this is a .NET bug, a Wine bug, or an Azure pipelines agent bug. We are currently getting technical support from CodeWeavers on this issue as well.

The bug is actually in .NET Runtime. Don't call SetConsoleCP if you don't have any console handles. In this case, all handles are redirected so there's nothing for SetConsoleCP to act on.

It's like trying to call SetConsoleCP from a Win32 GUI application; it doesn't work. Win32 GUI applications with attached standard input and output are real things; and Console.WriteLine() will not function.

In fact the bug's more than that. It's not just trying to call SetConsoleCP when there's no console. It's trying to call SetConsoleCP when the console is not its to play with. The following command on Windows shouldn't call SetConsoleCP but does: dotnet --version > NUL 2>NUL

Unfortunately, .NET is not supported on Wine so we are unable to help here. Customers can source build .NET with a patch for the issue on Wine but we won't be taking changes for this at this time.

@marcpopMSFT

@marcpopMSFT - Fundamentally, this doesn't have anything to do with Wine, and the fix seems trivial. The OutputEncoding property is documented to throw exceptions in some situations. The commit linked above sets the property without any kind of exception handling. It doesn't seem appropriate to prevent usage of the cli just because the output encoding can't be set? Is there opposition to a PR that catches exceptions raised when setting that property?

@nick-beer

@nick-beer we'd gladly accept such a PR - @marcpopMSFT was mainly getting at that the .NET team doesn't directly support Wine, so we would prefer to use our team-time on things that are related to the platforms we do support.

@baronfel

@baronfel : The wine tracer is convenient for looking for bogus API calls. I can probably build a Wine free repro of trashing console output encoding on the console despite all handles redirected now, but I never would have figured it out otherwise.

@jhudsoncedaron

@jhudsoncedaron I'm not familiar with the 'wine tracer' - do you have a link?

https://wiki.winehq.org/Debug_Channels

It's kind of a pill to use but I've been able to find on occasion exactly which API call the process started going wrong at. The output of WINEDEBUG=+relay wine dotnet is going to be enormous; the trick is to arrange for the fault to be near the bottom.

I can probably build a Wine free repro of trashing console output encoding on the console despite all handles redirected now,

Yes, it would help to show how you can run dotnet on Windows and make it crash in the code in question.

I think that the root cause of the problem is that Wine does not emulate Windows console APIs faithfully.

As I said, the bug is in the runtime. This should not throw with redirected standard output, but does:

NativeMethods.FreeConsole();
Console.OutputEncoding = System.Text.Encoding.UTF8;
Console.WriteLine("Output CP = " + NativeMethods.GetConsoleOutputCP());

I stuck NativeMethods.FreeConsole(); on the top because the reproduction is difficult otherwise, but it is possible. The wine setup is giving you something more like ssh hostname dotnet --version instead of dotnet --version. Your console process doesn't have a console yet.

Run the attached project with dotnet run > log 2> log2 and the exception appears in log2.

r38613.zip

Calling SetConsoleOutputCP() for a redirected handle is a bug even when it doesn't crash. It will mess up the caller's console by changing the glyph mappings. Call isatty first to check for redirected standard output/error.

I stuck NativeMethods.FreeConsole(); on the top because the reproduction is difficult otherwise, but it is possible.

dotnet does not free its own console like this.

How is it possible to hit the problem with a console app like dotnet on Windows?

By passing DETACHED_PROCESS to CreateProcessW(). https://learn.microsoft.com/en-us/windows/win32/procthread/process-creation-flags

If you think about it, it makes perfect sense to pass DETACHED_PROCESS when you redirect standard input, output, and error as a GUI launching process; that way it doesn't show up.

@adamsitnik

If we want to make console apps and Console.OutputEncoding API work well when the process is launched with DETACHED_PROCESS, does it need to be better fixed in System.Console?

On Windows, Console.OutputEncoding API does two things:

sets the encoding using SetConsoleOutputCP
sets the encoding to use by the System.Console to encode Unicode to set.

It is still valid to do the second part for redirected output.

If we go with the proposed fix to catch exceptions around Console.OutputEncoding, the second step is not going to happen, and the redirected output will be encoded using some random encoding instead of UTF-8 that was explicitly set.

I think the correct fix should be in System.Console. System.Console should not call SetConsoleOutputCP when the output is redirected or ignore the invalid handle error returned by SetConsoleOutputCP .

cc @adamsitnik

@nick-beer

@nick-beer Are you interested in submitting a PR with proposed fix?

@jkotas

@jkotas - yes, I'll try to get something posted soon.

isatty() doesn't work too good because you could be redirected to a printer. This is one ugly edge case. Too bad SetConsoleOutputCP doesn't take a file handle.

I've worked out an actual solution involving CreateFile("CONOUT$") and CompareObjects. If nothing else appears I might be able to do something about this next week.

On the other hand, calling GetFileType() (isatty() is implemented via GetFileType()) may well be a good enough choice and quite a bit faster.

@jkotas

@jkotas - thinking about this over the last few hours, I wonder if the complexity/risk to reward tradeoff is right here? We were previously talking about handling an extremely rare exception in the SDK. The impact of handling the exception is the same as if we hadn't set the property at all - exposing the previous bug (but not crashing). However, we're now talking about making changes to the console APIs that could (?) change behavior for an unkown number of applications, to fix an issue that hasn't been reported (presumably). Thoughts?

If we take the workaround in the SDK, we are saying that all uses of Console.OutputEncoding setter should better have try/catch around it. It does not make sense to me. For example, are you going to hit the problem for similar call in Roslyn next?

I think that the fix in the console APIs can be very low-risk and simple. I would just ignore the "Invalid Handle" error code from SetConsoleOutputCP API.

@jkotas

@jkotas - the error returned when the process is run with DETACHED_PROCESS is ERROR_INVALID_HANDLE (0x00000006). The error returned in the case we (@Wedgetable and I) are encountering is ERROR_INVALID_ACCESS (0x0000000C). I'll submit the PR to ignore INVALID_HANDLE, but do you think it's inappropriate to also ignore INVALID_ACCESS?

ignore INVALID_ACCESS?

It would be ok with me.

Ignore invalid handle/access when setting console cp

PR created

#98441 - Ignore invalid handle/access when setting console cp (#98641)

Navigation Menu

Repository navigation

dotnet CLI commands no longer work in .NET 7/8 with detached console #98441

Describe the bug

To Reproduce

Exceptions (if any)

Further technical details

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

dotnet CLI commands no longer work in .NET 7/8 with detached console #98441

Description

Describe the bug

To Reproduce

Exceptions (if any)

Further technical details

Activity

jhudsoncedaron commented on Feb 8, 2024

marcpopMSFT commented on Feb 14, 2024

nick-beer commented on Feb 14, 2024

baronfel commented on Feb 14, 2024

jhudsoncedaron commented on Feb 14, 2024

baronfel commented on Feb 14, 2024

jhudsoncedaron commented on Feb 14, 2024

jkotas commented on Feb 14, 2024

jhudsoncedaron commented on Feb 14, 2024

jkotas commented on Feb 14, 2024

jhudsoncedaron commented on Feb 14, 2024

jkotas commented on Feb 14, 2024

6 remaining items

jkotas commented on Feb 14, 2024

nick-beer commented on Feb 14, 2024

joshudson commented on Feb 15, 2024

nick-beer commented on Feb 15, 2024

jkotas commented on Feb 15, 2024

nick-beer commented on Feb 18, 2024

jkotas commented on Feb 18, 2024

nick-beer commented on Feb 18, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions