Moving WebAssembly inline assembly forward

@daxpedda

I'm opening this issue to consolidate some scattered discussion about what is needed to make inline assembly for Wasm work. This is tracked along with the other architectures at #93335. That thread notes:

It must have clobber_abi.

It must be possible to clobber every register that is normally clobbered by a function call.

Generally review that the exposed register classes make sense.

So, for Wasm specifically:

Syntax seems to be the biggest question (see below discussion), WAT vs. LLVM's format
There are not any registers to be clobbered. Should there be some way to indicate the asm clobbers the top N elements on the stack?
Should we be restricting dir specs (in, out, lateout, inout, inlateout) or options (pure, nomem, readonly, preserves_flags, noreturn, nostack, raw)? We should probably reject preserves_flags. I'm not sure if lateout and inlateout make sense.
Is the LLVM side generally considered stable? It appears unchanging, but I don't see any of its constraints documented in langref https://llvm.org/docs/LangRef.html#inline-asm-constraint-string

Cc @daxpedda @hanna-kruppe @alexcrichton @hoodmane @solomatov, I think you have all been involved in the wasm-inline-asm discussion in different places.

My impression is that LLVM's support for wasm inline assembly is not comparable to support for other architecture:

There's no GCC equivalent that Clang would presumably try and be interoperable with, so much less inertia against changing how exactly it works. I can't even find any information anywhere how it's supposed to work.
The support that does exist in LLVM is barely tested, and only one a single test even tries to look like legit code instead of a made-up smoke test. It seems to be a side effect of enabling support for non-inline assembly syntax (https://reviews.llvm.org/D52914 - I've not seen any work since then that's specific to inline asm).
Searching for usage in some repositories where I'd expect to see usage of it reveals close to zero usage and a maintainer saying "should probably be considered experimental" in 2021 and expressing a similar sentiment again in 2023. Instead, most of what you'd expect a libc to use inline asm for is either done with wasm-specific intrinsics or written in separate non-inline assembly.
Insofar the inline asm support is stable, that seems to be due to not getting any ongoing development. For example, trying to pass simd128 vectors to inline asm just crashes.

So I would not recommend moving forward with stabilizing Wasm inline assembly without first getting a vibe check from the LLVM backend maintainers. Even if Rust people wanted to put in the work of fixing the bugs that affect Rust users, is there even enough upstream interest and maintainer bandwidth to review and merge that work?

There's no GCC equivalent that Clang would presumably try and be interoperable with, so much less inertia against changing how exactly it works. I can't even find any information anywhere how it's supposed to work.

It's even worse than there being no GCC equivalent. There is a standard text format specified for wasm, but LLVM created an entirely different format for whatever reason which no other tool in existence works with.

Sounds like this is nowhere close then. Our flags could probably still be improved but I guess that is about it.

It's even worse than there being no GCC equivalent. There is a standard text format specified for wasm, but LLVM created an entirely different format for whatever reason which no other tool in existence works with.

Does LLVM provide equivalents for control flow or is it just a strict subset of WAT without S-expressions?

We could very nearly handle wasm text->binary ourselves at least as well as LLVM could, if that wouldn't also require us doing the validation...

@sbc100

Well the one advantage of the llvm format is generating relocations. I think the webassembly text format is missing some information needed to emit relocations and that's why llvm invented its own format. It seems like it could have been kept more similar to wat though. Cc @sbc100 @kripken.

If both are worth having then in theory there could be a watdialect flag (which we could then enable by default), similar to inteldialect.

The LLVM text format was chosen to match the existing assembly format support in LLVM. It reuses a lot of the existing infrastructure. It doesn't support any kind of nesting and looks more like the unfolded/linear form of wat, but that is largely because that is what llvm expects from an asm format. Doing this differently would be very hard I think.

I don't think there are any plans to replace this format in LLVM and its use is fairly widespread now, admittedly a lot of it is in form of non-inline separate .s files. While I would support added the ability for wat to express things like relocations, I don't think it would make sense to convert LLVM to use some new format like that, and I don't think it makes much sense to have different format for inline and non-inline asm, or another/different format for inline asm in rust vs C/C++. In other words, I think the current inline asm support is worth improving, but I not changing radically.

Is there a formal definition of the text format LLVM uses? There is for the official wasm text format.

How hard would it be for rustc to convert the official text format (+ annotations for things like relocations) to the format LLVM uses? If we get another backend which supports wasm, I think it is unlikely that this other backend will exactly mirror LLVM's text format. Using the official text format is more likely and even if it doesn't, it should be a lot easier to convert a text format with a robust definition and an existing parser written in rust (wasmparser) to whichever format said backend uses than it would be to parse LLVM's custom text format. And if the other backend uses LLVM's custom text format after all it can reuse the same conversion code as we use for the LLVM backend.

Is there a formal definition of the text format LLVM uses?

No, there is no formal definition that I know of. However, I don't think there are formal definitions of other assembly languages either. For example from the x86_64 asm docs https://docs.oracle.com/cd/E19253-01/817-5477/817-5477.pdf: "There is no standard assembly language for the x86 architecture."

Unlike for x86 there is actually a standard text format for wasm, so there is much less of an excuse to use a poorly specified format. And I believe for arm at least the format for individual instructions is specified by Arm Limited, but I'm not sure if assembler directives are also specified by them.

@sbc100

I think the important part is that the syntax including important directives realistically won't change in backwards incompatible ways. @sbc100 has said that there's no plans to replace this format and it has a fair amount of usage in separate .s files, and I believe that. But I also believe that a change in that syntax is still somewhat more likely than for any inline assembly dialect currently exposed in stable Rust: those probably have several orders of magnitude more existing code, split between far more stakeholders, than LLVM's wasm syntax. I'm not sure if WAT is much better in this regard, especially if it has to be extended in some yet-undecided form to properly support relocations. But at least, as @bjorn3 said, there's high quality WAT-as-of-today parsers written in Rust, so it might be easier for rustc to support the format indefinitely and translate it to whatever a future backend expects.

Rollup merge of #147225 - daxpedda:wasm-u-u-atomics-threads, r=alexcr…

Rollup merge of rust-lang#147225 - daxpedda:wasm-u-u-atomics-threads,…

Rollup merge of #147225 - daxpedda:wasm-u-u-atomics-threads, r=alexcr…

In the next experimental version of wasm-bindgen we have switched from LLVM's .s format to the WAT format proposed in WebAssembly/tool-conventions#258. This has been quite successful, but there are still a lot of open questions:

The text format specifies that everything has to be explicitly marked with @sym and @reloc. I believe going with a global relocatable flag ((module (@relocatable))) would probably be a big improvement here.
Because LLVM has been built with the .s format and it own llvm-mc compiler in mind, it explicitly built for its relocation output. There are bound to be some bugs (e.g. [LLD][Wasm] Import table without WASM_SYM_EXPLICIT_NAME can't be found llvm/llvm-project#198509).
The .s format supports declaring/defining global values regardless of scope (e.g. asm!/naked_asm!). WAT doesn't. E.g. you can not declare that a function exists inside a function.
```
asm!(
    ".functype foo () -> (i32)",
    "call foo",
    "local.set {}",
    out(local) output,
);
```
```
asm!(
    "call foo",
    "local.set {}",
    out(local) output,
);
```
In the WAT version there is just no way to declare foo(). This is problematic on many levels:
- LLVM object files are just Wasm, which is not valid unless you declare the type of every function.
- WAT has no way to do this and adding any workaround via custom descriptors can't solve that the WAT itself would still be invalid and can't produce valid Wasm.
- Unless we add a workaround specifically in Rust somehow, e.g. allowing global WAT next to the function's WAT somehow, we will have to re-create the type during generation. The same limitation applies to LLVM, if it ever becomes interested in supporting WAT.
Until/if LLVM implements the text format, Rust will have to internally compile the WAT to a relocatable object file and pass it to the linker. This works great for global_asm! and naked_asm!, but inline_asm! will have some performance overhead because it is a separate function. It is unclear if something can be done about that from our side or if this would require some sort of explicit support from LLVM.

For now I believe the best thing we can do is provide feedback to WebAssembly/tool-conventions#258. I will also figure out if we could get an experimental implementation of this into Rust, so we can gather more user feedback.

Perhaps we could have a (@hoist ...) directive which hoists the argument out of the enclosing function into the module context?

This works great for global_asm! and naked_asm!, but inline_asm! will have some performance overhead because it is a separate function.

wasm-opt would be able to inline it again after linking, right?

Perhaps we could have a (@hoist ...) directive which hoists the argument out of the enclosing function into the module context?

One weird trick! WAT parsers hate him!
(is my initial gut reaction, I'm sure Alex can clarify)

This works great for global_asm! and naked_asm!, but inline_asm! will have some performance overhead because it is a separate function.

wasm-opt would be able to inline it again after linking, right?

Yes, but it is still unexpected that asm! would behave so differently between targets that it requires an external tool to achieve parity. I don't want to argue that its undesirable to require an external tool for performance in general, because the reality today is that the gap between the LLVM and wasm-opt output is big enough that it is often already considered a requirement for performance.

@bjorn3

In the WAT version there is just no way to declare foo().

To dig into this a bit, I'm wondering if this is true or a hard constraint. In some sense there's a fair bit of freedom with inline assembly since, by definition, the assembly doesn't necessarily fully define the context in which it's compiled. For example you can use i32.load without explicitly defining a memory. Given that what I'd expect for call $foo is to be something like:

call (@sym ...) 0

or some @foo annotation or something like that. Basically the relocations/etc would be required to tell the inline assembly parser for what's where and what needs to be in the outer context. To handle things like type definitions I think it'd be reasonable to have more (@foo ...) like what @bjorn3 mentions. Not so much to define a $foo name but to define something like "hey this linking-symbol is defined somewhere else and it'll have this type" or something like that.

Or, put another way, I don't fully understand the constraints that lead you to this conclusion. I don't doubt that anything wasm-related in this space is going to be a lot of work since LLVM doesn't implement this natively, though. How to square that I'm not entirely sure...

The problem is compilation from WAT to Wasm. In Wasm we can't say call 0 if 0 is not the index to an actual function. Indeed, we could do call (@sym @hoist (func (result i32))) 0 (kind of mashing together the two suggestions). The WAT parser then has to add a (import "..." "..." (func (@sym) (result i32))) to the top of the Wasm file. If we go down that route we would have to add that to WebAssembly/tool-conventions#258. I was just pointing out that this is a missing piece right now.

So why is all this necessary? Even if we pretend that LLVM supports WAT, even LLVM has to somehow generate valid object files. I will give an example how this currently works:

global_asm! {
    ".globl bar",
    "bar:",
    "  .functype bar () -> (i32)",
    "  i32.const 42",
    "	end_function",
}

#[unsafe(no_mangle)]
fn foo() {
    unsafe { asm!("call bar") }
}

This works perfectly fine. But if bar is defined in a separate module, it will fail. That is because we now have two separate object files and bar is not declared in the object file where foo is defined. The .s format supports this through the .functype statement, which works anywhere, even inside a function definition.

The same issue doesn't apply to e.g. x86, where its perfectly fine to write call bar and LLVM is able to generate valid object files, even if bar is defined in a different object file. I should point out here why the example is setup so strangely. Usually one wouldn't write call bar but instead "call {}", sym bar. But this is often not possible in Wasm when dealing with e.g. externref, which is not representable in Rust.

I hope this explains the issue. I believe that not being able to write global WAT sections in asm! is a fundamental constraint of WAT that precludes parity with other targets (or the current .s format in Wasm). Sure, we can solve the whole function type declaration with a bunch of workarounds in the relocation text format spec. But asm! is able to do something global_asm! is not: access generics. So I believe long-term we need a way to actually write global sections into the WAT of (naked_)asm!. I don't have a proposal on how to do this right now, but I will keep pondering on it.

@daxpedda

@daxpedda Is there any library which provides this global_asm! for wasm? Or is it just an example?

Works on current nightly with #![feature(asm_experimental_arch)].

@daxpedda

@daxpedda oh right yeah that all makes sense, and w.r.t.

I was just pointing out that this is a missing piece right now.

I agree with everything you've said, and the general thrust is what I'm thinking. Exact syntax and bikeshedding TBD, but basically if the official wasm text format is used for inline assembly then I think there's no route other than having some sort of syntax that introduces typed symbols which is then processed by the inline assembly parser. For example LLVM, I assume, already handles this with .functype directives and such where they're processed into imports/declarations as appropriate to be able to assemble the final object file.

The same issue doesn't apply to e.g. x86, where its perfectly fine to write call bar and LLVM is able to generate valid object files

Personally I think it's reasonable to chalk this up to architecture differences. As inline assemble I think it's pretty reasonable to have pretty different implementations/assumptions per-platform. In this situation for example I view this as a consequence of symbols effectively having a type in WebAssembly, where on no other platform do they have a type. That means that idioms from other platforms won't apply and WebAssembly will be a bit different, but personally I think it would still be tractable.

Something like @hoist or similar is what I'd expect in this case. Effectively whenever a symbol is introduced in an assembly block I think it'd be reasonable for Rust/LLVM to require that the type of the symbol is written down within the block. For example (@sym ...) would live somewhere or maybe (@type ...) as a hint for inline assembly.

Navigation Menu

Repository navigation

Moving WebAssembly inline assembly forward #136382

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Moving WebAssembly inline assembly forward #136382

Description

Activity

hanna-kruppe commented on Feb 1, 2025

bjorn3 commented on Feb 1, 2025

tgross35 commented on Feb 1, 2025

hoodmane commented on Feb 1, 2025

tgross35 commented on Feb 1, 2025

sbc100 commented on Feb 1, 2025

bjorn3 commented on Feb 1, 2025

sbc100 commented on Feb 1, 2025

bjorn3 commented on Feb 1, 2025

hanna-kruppe commented on Feb 1, 2025

12 remaining items

daxpedda commented on May 25, 2026

bjorn3 commented on May 25, 2026

daxpedda commented on May 25, 2026

alexcrichton commented on May 26, 2026

daxpedda commented on May 27, 2026

solomatov commented on May 27, 2026

daxpedda commented on May 27, 2026

alexcrichton commented on May 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions