Support native FFI calls via libffi #11
Description
It would be nice to use libffi to allow for external calls.
- Passing non-pointer scalars across the FFI barrier: Adding FFI support to Miri, by emarteca #2365
- Letting C read from Miri-allocated memory: Enable Miri to pass pointers through FFI rust#129684
- Letting C write to Miri-allocated memory
- C code can initialize previously initialized memory
- C code can store pointers in Miri-allocated memory
- Letting Miri access memory that was allocated in C
- Passing non-scalar types across the FFI barrier
Reacted by Elichai Turkel, jyn, Tomas Tauber, Johan Bjäreholt, David Craven, Techcable, Jay Oster, Matthew Maurer, Kornel, Cherry and 3 more
Reacted by Cherry, Juhan Oskar Hennoste and Alex
Reacted by Cherry, ErikWDev, Juhan Oskar Hennoste and Alex
Reacted by Cherry, Juhan Oskar Hennoste and Alex
Activity
eddyb commentedon Apr 30, 2016
Previously suggested on IRC based on @cmr's mentions in a different context.
The closure example presents what I think is one of the most valuable features of this library: the ability to JIT a C function which hardcodes extra owned values, a whole closure masquerading in a function pointer.
solson commentedon Apr 30, 2016
@drbo We've definitely been considering that.
For a bit of context, there are two versions of Miri we'll have in the future. If the Rust teams decide to use a MIR-based constant evaluator, some version of Miri will get incorporated into rustc, and that version will never use libffi because you want to ban arbitrary FFI calls in constant evaluation.
But it would be a lot of fun to add libffi to this external version of Miri. I would probably wait until after any Miri → rustc merging happens, though.
BatmanAoD commentedon May 3, 2018
Now that Miri is indeed in rustc (wooooo!), any further thoughts on this?
If I understand correctly, FFI would be necessary for us to use Miri as the basis for a Jupyter kernel.
eddyb commentedon May 4, 2018
@BatmanAoD I'd advise against using miri for such a purpose - have you looked at webassembly?
BatmanAoD commentedon May 7, 2018
@eddyb I don't actually know what the requirements are for a Jupyter kernel; I was basing my understanding on an old discussion about using
rustiinstead of Miri for the basis of the kernel.I can't actually find documentation on how the kernel API works, though (I probably just haven't spent enough time looking). It may be that communication between JupyterLab and the kernel is entirely over HTTP.
In any case: I'm not sure how WebAssembly would solve the problem. The Jupyter kernel is a REPL for a given language; the goal of a Rust kernel would be to permit people to create Jupyter notebooks that would use the Rust language instead of IPython (or any of the other languages that are already supported via kernels). So the need is for a Rust REPL that supports whatever interface JupyterLab uses.
eddyb commentedon May 7, 2018
@BatmanAoD Oops, I misunderstood "kernel" as "OS kernel", not an IPython-like thing.
Then miri is likely the way to go, and I agree libffi would be needed.
BatmanAoD commentedon May 7, 2018
@eddyb Yeah, I think the terminology is a bit confusing!
oli-obk commentedon May 7, 2018
This project is almost ready (as in compiling and testing on Travis) again, I have a PR against rustc ready that should fix all the tests. After that we can accept new PRs. So if anyone wants to hack up a prototype using libffi to execute some c functions that we're currently having horrible fake support for (like syscall), that would be totally awesome. Things like
mallocare a little harder though, but with a new allocation kind totally doable without touching rustc.eddyb commentedon May 11, 2018
@oli-obk I think pointers in general are hard. Various syscalls take pointers (e.g.
readandwrite) while others return them (e.g.mmap) - somallocis not unique in this difficulty.It's easier to detect miri pointers being passed to FFI, and do something about it, than it is to handle the other way - I'm thinking maybe
miricould be self-aware (e.g. via/proc/self/maps) and allow access via integer pointers to memory regions that would surely not segfault (that would, amusingly, allow corrupting miri itself - which C code could do anyway, so it's not introducing a new issue).BatmanAoD commentedon May 11, 2018
@eddyb If it were possible to guarantee that improper writes would segfault, wouldn't that be preferable to corrupting the Miri process without terminating it?
eddyb commentedon May 11, 2018
@BatmanAoD My point was that integer pointers could be used to corrupt miri itself - regardless of how you handle segfaults, the values for which corruption could happen are generally hard to detect, and pointless anyway, because you're also calling into C, which can do anything anyway.
oli-obk commentedon May 11, 2018
Yea.... I don't think we should care too much about users doing horrible things. If you do care, don't enable the ffi feature ^^
elichai commentedon Aug 15, 2019
Any news on this?
Would love to use miri on some projects that have a lot of FFI calls.
Ericson2314 commentedon Aug 15, 2019
libffi is due for a new release. (GHC and probably other projects have had to vendor it which is a pain.) If you do end up using it, please lobby upstream so we can all get a working release!
[#38, #40] Fix lifetime errors in `sub_self_call`
emarteca commentedon Jul 12, 2022
Hey! I've started working on this issue -- here is a design doc of our proposed plan so far: link here. We'd love your feedback/comments on the proposed approach!
@oli-obk @RalfJung
Thanks! :)
RalfJung commentedon Jul 13, 2022
@emarteca that's a very welcome surprise. :-)
Let's make a new issue to specifically discuss your project: #2365
Auto merge of #2363 - emarteca:int-function-args-returns, r=oli-obk
Auto merge of #118488 - andjo403:remove_deadlock, r=SparrowLii
[-]Consider using libffi to allow for external calls[/-][+]Support native FFI calls via libffi[/+]ci: run the test suite under AddressSanitizer and LeakSanitizer (#275)