Repository navigation
Implement all x86 vendor intrinsics #40
Description
This is intended to be a tracking issue for implementing all vendor intrinsics in this repository.
This issue is also intended to be a guide for documenting the process of adding new vendor intrinsics to this crate.
If you decide to implement a set of vendor intrinsics, please check the list below to make sure somebody else isn't already working on them. If it's not checked off or has a name next to it, feel free to comment that you'd like to implement it!
At a high level, each vendor intrinsic should correspond to a single exported Rust function with an appropriate target_feature attribute. Here's an example for _mm_adds_epi16:
/// Add packed 16-bit integers in `a` and `b` using saturation.
#[inline]
#[target_feature(enable = "sse2")]
#[cfg_attr(test, assert_instr(paddsw))]
pub unsafe fn _mm_adds_epi16(a: __m128i, b: __m128i) -> __m128i {
unsafe { paddsw(a, b) }
}Let's break this down:
- The
#[inline]is added because vendor intrinsic functions generally should always be inlined because the intent of a vendor intrinsic is to correspond to a single particular CPU instruction. A vendor intrinsic that is compiled into an actual function call could be quite disastrous for performance. - The
#[target_feature(enable = "sse2")]attribute intructs the compiler to generate code with thesse2target feature enabled, regardless of the target platform. That is, even if you're compiling for a platform that doesn't supportsse2, the compiler will still generate code for_mm_adds_epi16as ifsse2support existed. Without this attribute, the compiler might not generate the intended CPU instruction. - The
#[cfg_attr(test, assert_instr(paddsw))]attribute indicates that when we're testing the crate we'll assert that thepaddswinstruction is generated inside this function, ensuring that the SIMD intrinsic truly is an intrinsic for the instruction! - The types of the vectors given to the intrinsic should match exactly the types as provided in the vendor interface. (with things like
int64_ttranslated toi64in Rust) - The implementation of the vendor intrinsic is generally very simple. Remember, the goal is to compile a call to
_mm_adds_epi16down to a single particular CPU instruction. As such, the implementation typically defers to a compiler intrinsic (in this case,paddsw) when one is available. More on this below as well. - The intrinsic itself is
unsafedue to the usage of#[target_feature]
Once a function has been added, you should also add at least one test for basic functionality. Here's an example for _mm_adds_epi16:
#[simd_test = "sse2"]
unsafe fn test_mm_adds_epi16() {
let a = _mm_set_epi16(0, 1, 2, 3, 4, 5, 6, 7);
let b = _mm_set_epi16(8, 9, 10, 11, 12, 13, 14, 15);
let r = _mm_adds_epi16(a, b);
let e = _mm_set_epi16(8, 10, 12, 14, 16, 18, 20, 22);
assert_eq_m128i(r, e);
}Note that #[simd_test] is the same as #[test], it's just a custom macro to enable the target feature in the test and generate a wrapper for ensuring the feature is available on the local cpu as well.
Finally, once that's done, send a PR!
Writing the implementation
An implementation of an intrinsic (so far) generally has one of three shapes:
- The vendor intrinsic does not have any corresponding compiler intrinsic, so you must write the implementation in such a way that the compiler will recognize it and produce the desired codegen. For example, the
_mm_add_epi16intrinsic (note the missingsinadd) is implemented viasimd_add(a, b), which compiles down to LLVM's cross platform SIMD vector API. - The vendor intrinsic does have a corresponding compiler intrinsic, so you must write an
externblock to bring that intrinsic into scope and then call it. The example above (_mm_adds_epi16) uses this approach. - The vendor intrinsic has a parameter that must be a constant value when given to the CPU instruction, where that constant is often a parameter that impacts the operation of the intrinsic. This means the implementation of the vendor intrinsic must guarantee that a particular parameter be a constant. This is tricky because Rust doesn't (yet) have a stable way of doing this, so we have to do it ourselves. How you do it can vary, but one particularly gnarly example is
_mm_cmpestri(make sure to look at theconstify_imm8!macro).
References
All intel intrinsics can be found here: https://software.intel.com/sites/landingpage/IntrinsicsGuide/#expand=5236
The compiler intrinsics available to us through LLVM can be found here: https://gist.github.com/anonymous/a25d3e3b4c14ee68d63bd1dcb0e1223c
The Intel vendor intrinsic API can be found here: https://gist.github.com/anonymous/25d752fda8521d29699a826b980218fc
The Clang header files for vendor intrinsics can also be incredibly useful. When in doubt, Do What Clang Does:
https://github.com/llvm-mirror/clang/tree/master/lib/Headers
TODO
["MMX"]
-
_mm_srli_pi16 -
_mm_srl_pi16 -
_mm_mullo_pi16 -
_mm_slli_si64 -
_mm_mulhi_pi16 -
_mm_srai_pi16 -
_mm_srli_si64 -
_mm_and_si64 -
_mm_cvtsi32_si64 -
_mm_cvtm64_si64 -
_mm_andnot_si64 -
_mm_packs_pu16 -
_mm_madd_pi16 -
_mm_cvtsi64_m64 -
_mm_cmpeq_pi16 -
_mm_sra_pi32 -
_mm_cvtsi64_si32 -
_mm_cmpeq_pi8 -
_mm_srai_pi32 -
_mm_sll_pi16 -
_mm_srli_pi32 -
_mm_slli_pi16 -
_mm_srl_si64 -
_mm_empty -
_mm_srl_pi32 -
_mm_slli_pi32 -
_mm_or_si64 -
_mm_sll_si64 -
_mm_sra_pi16 -
_mm_sll_pi32 -
_mm_xor_si64 -
_mm_cmpeq_pi32
Activity
alexcrichton commentedon Sep 25, 2017
cc @BurntSushi @gnzlbg, I've opened this up and moved
TODO.mdout here, I figure it may be easier to collaborate here to ensure we can attach names everywhere!mattico commentedon Sep 25, 2017
Could you edit the guide to suggest unsafe functions for the intrinsics? #21
alexcrichton commentedon Sep 26, 2017
@mattico makes sense yeah! Although we may want to wait until #21 is closed out to avoid inconsistencies
AdamNiederer commentedon Sep 26, 2017
For those wishing to implement intrinsics above SSE2, make sure you're running your tests with
RUSTFLAGS="-C target-cpu=native" cargo teston something which supports that instruction set extension. It looks lilke it's only running the SSE2 tests otherwise.gnzlbg commentedon Sep 26, 2017
You can use `RUSTFLAGS="-C target-feature=+avx2" to enable a particular extension. Note however that a CPU that does support the extension is needed for running the tests. To develop tests for a different architecture (e.g. develop for ARM from x86) you can use cross-compilation. To run the tests... travis is an option. I don't know if there is a better option though.
AdamNiederer commentedon Sep 26, 2017
It looks like travis only runs SSE2 and below with our current config. I wonder if their machines support AVX...
alexcrichton commentedon Sep 26, 2017
@AdamNiederer oh that's actually a bug! I think I see what's going on though, I'll submit a fix.
gnzlbg commentedon Sep 26, 2017
@alexcrichton https://github.com/rust-lang-nursery/stdsimd/blob/master/ci/run.sh probably needs to set
RUSTFLAGS="-C target-cpu=native"to run most tests. @AdamNiederer makes a point though, what instruction sets does travis support? If it doesn't support AVX2, those will never be tested (I am pretty sure travis does not support AVX512, so we'll need a different solution for that).AdamNiederer commentedon Sep 26, 2017
Added in #45. Let's see what Travis has to say about it.
EDIT: The build is failing, but those same 20 tests were failing for me on my Ivy Bridge box last night. I think LLVM might be spitting out wider version of 128 or 64-wide instructions on CPUs which support them. It also looks like travis supports AVX2 🎉
alexcrichton commentedon Sep 26, 2017
@gnzlbg oh I'm going to add
cfg_feature_enabled!to all tests and enable them all unconditionally all the time, that way whatever your cpu supports we'll be testing everything (without any required interaction)@AdamNiederer thanks! I'll look into the failures and see if I can fix them.
dlrobertson commentedon Sep 28, 2017
Interested in helping out with this. Figured I'd start super small with
cvtps2dq#65vbarrielle commentedon Sep 29, 2017
Hello, I've given a try at
__mm256_div_psand its double counterpart, see #73.dlrobertson commentedon Sep 30, 2017
Post #81 SSE 4.2 should be covered.
32 remaining items
Add _mm_loadu_si64
Add _mm_loadu_si64
Add _mm_loadu_si64
et-tommythorn commentedon Sep 17, 2020
Is this the right place to mention that core::arch is missing RISC-V support or should I open a tracking bug? (I'm specifically interested in adding support for the equivalent of rdtsc).
Amanieu commentedon Sep 17, 2020
We generally try to stick to vendor-specified intrinsics, e.g. SSE intrinsics and ARM NEON intrinsics. AFAIK RISC-V doesn't have any target-specific intrinsics defined in GCC or Clang.
tommythorn commentedon Sep 17, 2020
Ough. Thanks. I can see your reasoning, but that raises the bar by orders of magnitude and pushes the problem to all clients of core::arch :(
Amanieu commentedon Sep 17, 2020
You can always just use inline assembly if you really want a specific instruction...
tommythorn commentedon Sep 17, 2020
That's literally what "pushes the problem to all clients" means.
Lokathor commentedon Sep 18, 2020
Probably best to just open a new issue where it can get eyes and discussion. The tail end of a long-closed issue isn't a good way to bring your problem to light.
jack-pappas commentedon Sep 18, 2020
@Amanieu It doesn't look like there are any RISC-V intrinsics in llvm/clang yet, but there is some recent work in that area: https://www.sifive.com/blog/risc-v-vector-extension-intrinsic-support
Amanieu commentedon Sep 18, 2020
Those are actually much trickier than it seems since they involve scalable vectors with a size not known at compile-time. This requires special support in the compiler. The same issue applies to the ARM SVE intrinsics.
mjptree commentedon Dec 9, 2020
Out of interest and because it has recently become relevant: ["VMX"] would be helpful.
newpavlov commentedon May 14, 2021
Maybe it's worth to open separate issue for each target feature? For example, I wanted to use
_mm_stream_load_si128and was quite surprised thatstd::arch::x86_64does not have it.Is there a reason why streaming load intrinsics were omitted?
Amanieu commentedon May 14, 2021
Please open a new issue if there are any missing intrinsic.