Windows subsystem support

[Diggsey]

Rendered

[mbrubeck: edited to link to final rendered version]

[sfackler]

cc @retep998

[ahicks92]

I haven't tried to do this myself, but the RFC is implying that it requires special compiler flags. If this is a use case we want to support, then perhaps Cargo should also gain something for it.

There might be a similar concern with mac. I'm not sure what the difference between a console app and a nonconsole app is over there, or even what the proper names are. But it might be worth covering them both at the same time and with the same Cargo option. If this is also a concern, designing it to hit both with as few changes as possible could be helpful. Even if we need a separate RFC to finish it.

One of the things I want to do with Rust is Windows desktop apps. I had assumed this was trivially possible and that I could just kill the console at any time, but I guess I should have checked.

[retep998]

Right now you can change the subsystem by passing-Clink-args="/SUBSYSTEM:WINDOWS" either via cargo rustc or RUSTFLAGS. It's not ideal, but to have a proper way to configure that would need to also support choosing the target NT version as well.

[Diggsey]

@camlorn AFAIK, there's no equivalent for OSX or linux, all executables are the same there, the only difference between a GUI app and a console app are code differences.

[ahicks92]

@Diggsey
Just verified. The only difference is whether or not it's in an app bundle. Source.

@retep998
Perhaps subsystems should be handled by letting us specify an oldest Windows version and having Rustc or Cargo automatically determine what it should be. I've looked into the subsystem stuff before and came away vowing to never touch it because of poor documentation. It's possible being able to configure the subsystem has uses I don't know about, but it feels like an option that would exist just to let me get it wrong.

I would personally vote for defaulting such an option to Windows 7, with Vista and XP requiring you to explicitly set it.

[Diggsey]

@camlorn Right, but the executable does not need to be linked specially - it is sufficient to simply place it in the correct location in the application bundle. I see this as being a problem for one of the various packaging plugins for cargo to solve.

[alexcrichton]

Thanks for the PR @Diggsey! I've got a few questions, but mostly along the lines of clarification.

Could you provide a detailed explanation of what WinMain would look like and what the compiler generates (the compiler generates main, so I assume it'd also generate WinMain). The signature looks like it has some information like hInstance and nCmdShow which we'd probably want to make sure are accessible through at least some means, right?

Also, this would in theory be a breaking change for anyone who explicitly has a WinMain function in their crate today I believe, so this may want to explicitly mention that and perhaps try to find a few crates that either do this or confirm that it's quite rare.

Being relatively unfamiliar with this space as well, I wouldn't mind expanding the alternatives a bit more in detail. For example I'm not sure what a #[cfg] attribute would look like here. Also is it true that object files need to be compiled with a particular subsystem in mind? If that's true then we need to consider how we ship the standard library, but I thought that this only had to do with linker flags at the very end. Along those lines, it's not clear to me at least why dependencies would need to be recompiled.

Also, how would a subsystem function help? Doesn't compile-time generation need to be different as opposed to something at runtime? It seems like libraries could discover whether they're part of an application or not, but it's also not clear to me how it'd solve the problem at hand.

[retep998]

Could you provide a detailed explanation of what WinMain would look like and what the compiler generates

extern "system" WinMain(_: HINSTANCE, _: HINSTANCE, _: LPSTR, _: c_int) -> c_int {
    // call Rust's std::rt entry point
    return exit_code;
}

None of the arguments matter and can all be obtained via other means.

perhaps try to find a few crates that either do this or confirm that it's quite rare.

https://github.com/search?q=WinMain+language%3Arust&type=Code&utf8=%E2%9C%93

Also is it true that object files need to be compiled with a particular subsystem in mind

Nope, the only thing that matters is choosing the subsystem at link time. Since any non-console app can still be run from within a console, it is up to the application to determine whether stdout/stderr are valid at runtime and redirect logging appropriately.

[Diggsey]

@alexcrichton Hopefully these changes address your comments.

Doesn't compile-time generation need to be different as opposed to something at runtime?

No, the aim is for the entire compilation process to be completely identical up until link time. If libraries want to have different code-gen, they should do so via the existing feature or cfg systems.

It's not all that different from other linker options such as what icon to use for the produced executable: it greatly simplifies things if it's independent of the compilation process.

[alexcrichton]

To clarify here, the compiler generates a main symbol as the very last thing when an executable is generated. This main symbol is then wired up to libstd's #[start] function. In that sense, I don't think that this would require multiple versions of libstd?

Also, are you thinking that this alternative is basically rustc --cfg subsystem=windows foo.rs or something like that?

[Diggsey]

This is considering the hypothetical that the main symbol generation is moved out of the compiler and into libstd - libstd would then conditionally export a main or WinMain function depending on cfg options.

I've seen similar approaches in C++ libraries which attempt to abstract over platform differences.

[alexcrichton]

Right now that's not actually possible with the compiler, something like a --cfg flag can't conditionally export different symbols when we're shipping a precompiled artifact, so maybe that's why I'm finding this confusing? Are you assuming here that we're compiling std from source on demand or something like that?

[DemiMarie]

One major difference is that console programs need to use WriteConsole and UTF-16 to do console I/O correctly.

[retep998]

@DemiMarie It's trivial to figure out whether the handle you get via GetStdHandle is a console handle or a pipe (libstd already does this). It's also easy to determine whether writing to that handle results in an error, so you can fallback to logging to a file. There's simply no need to know which subsystem you were linked for.

[tomaka]

Naive question: why does passing -Clink-args="/SUBSYSTEM:WINDOWS" work today even if you don't have any WinMain function?

[CryZe]

I guess without the WinMain you can't get access to the hInstance which is probably why the Window with glutin is always so disconnected from the Main Process in the Task Manager?!

Update: Nvm, that seems to be the case for every program.

[tomaka]

@CryZe That problem is probably due to the fact that you have the console subsystem.

One has to pass the hInstance when creating a window, however this parameter mostly exists to cover corner cases where DLLs want to register window classes. Glutin passes null for this parameter, which means "the main app".

[Diggsey]

@tomaka It doesn't, either something is exporting WinMain (maybe one of your dependencies, it would be useful to know if that's the case!) or the linker arguments are not getting through to the linker.

[retep998]

I guess without the WinMain you can't get access to the hInstance which is probably why the Window with glutin is always so disconnected from the Main Process in the Task Manager?!

GetModuleHandle(NULL). There's your hinstance.

All the startup information like nCmdShow can be obtained via GetStartupInfo.

[Diggsey]

I've removed some of the "hypothetical" alternatives at @alexcrichton's suggestion, and I've fleshed out a possible design for the "command-line configurable" alternative.

I initially went with the "dual symbol" solution as the main proposal because I thought it would be a quick way to get this support into rustc, with room for later improvement. That may still be the case, but if there's no objection to the specific design I went with for the alternative, it may be better to jump straight to that.

The main problem I see is that it adds extra command-line surface for everyone, but it's only going to be relevant when targeting windows, and I can imagine there may be some objection to that.

[Diggsey]

@tomaka Ah, you may have been using the GNU toolchain. This is more of a problem for MSVC. I've updated the RFC accordingly.

[retep998]

@Diggsey The /SUBSYSTEM flag can also be used to specify the minimum NT version the binary should run on. How would you specify that with this RFC, or how is there room left for this in a future RFC?

[Diggsey]

The duplicate symbol solution which this RFC proposes does not affect linker flags at all, and the version numbers specified for the subsystem do not affect which entry point is used, so I don't think there's any overlap at all there.

The alternative which adds a subsystem config to rustc and cargo does have some overlap: as proposed, rustc would not understand a version number, and so would not pass it through to the linker, but it would still be possible to manually pass in the option as we do now. I don't believe it restricts us in future: we could backwards compatibly extend the set of possible values for subsystem to include a version number, or we could add a separate option controlling the version.

[3442853561]

并没有理解如何隐藏后台窗体。
Help!There is something wrong in my program.

[3442853561]

Help！How can I hide the DOS form?
请问，我应该如何隐藏后台窗体？

[ticki]

@3442853561 This repository is for RFCs, proposals for the Rust programmer language. This is not the right place to ask questions.

Secondly, your question is vague and cannot be answer as is. Please be specific, and ask it at either https://reddit.com/r/rust or https://stackoverflow.com.

[joshtriplett]

This wouldn't involve libstd support, but for the benefit of applications using libcore, could this support the "NATIVE" subsystem as well?

[retep998]

@joshtriplett the NATIVE subsystem is not for simple no_std applications. It is for kernel mode drivers which is an area that nobody has even attempted with Rust yet, so there is basically no information on what the right way forward is yet. There's nothing stopping the proposed attribute from eventually supporting that, but it'll have to be part of a larger effort to figure out how to write Windows kernel drivers in Rust.

[joshtriplett]

@retep998 I wasn't suggesting that the RFC needed to define how that would work. I just wondered if a Rust crate producing a Windows driver could use this mechanism to declare the correct target subsystem, assuming it doesn't use std and otherwise meets the assumptions of a Windows driver, or if such a crate would still have to use linker options directly.

[retep998]

@joshtriplett At the moment this RFC doesn't specify any subsystems other than windows or console. To make a driver you have to pass other linker args anyway such as /DRIVER, so it shouldn't be too hard to pass along /SUBSYSTEM:NATIVE manually as well.

[alexcrichton]

@rfcbot fcp merge

Discussion I think has reached a close at this point on the RFC, and having discussed it in a tools team meeting I believe our concerns are addressed as well. I propose we merge!

[rfcbot]

Team member alexcrichton has proposed to merge this. The next step is review by the rest of the tagged teams:

• @alexcrichton
• @brson
• @michaelwoerister
• @nrc
• @vadimcn
• @wycats

No concerns currently listed.
Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[michaelwoerister]

@rfcbot reviewed

[vadimcn]

@rfcbot reviewed

[alexcrichton]

🔔 This RFC is now entering its week-long final comment period for merging 🔔

[rfcbot]

All relevant subteam members have reviewed. No concerns remain.

[rfcbot]

The final comment period is now complete.

[alexcrichton]

Ok, looks like nothing new came up during FCP, so merging! Thanks again for the RFC @Diggsey!

Tracking issue: rust-lang/rust#37499

[pravic]

I find it strange that panic! behavior isn't mentioned in this RFC. Currently it just exits with code 3 and prints nothing. I understand that in general you can't expect any console output in GUI executable, but the fact that the panic behavior isn't specified in this case is odd.

What would be the next steps to address such an issue? Issue in the https://github.com/rust-lang/rust repository?

[rpjohnst]

@pravic Panic behavior is probably better handled with custom panic hooks.