rlua is looking for maintainers!

Copied from a reddit discussion here:

I don't actually use Lua for anything currently, and I also don't have much interest in returning to Lua in the future. As such, my Lua related crates rlua (and also luster, but that's much less important) are not receiving much attention currently.

That's a real shame, and I'd like to help at least rlua not become abandoned. It needs some work done to it, none of which is terribly complex but I just don't have the time or proper motivation to be the one to do it. Also, I don't actually even think I should be the one to do it, as it's very hard to make good APIs when you don't use your own API.

I'd be happy to guide people through my ideas for rlua and help especially with reviewing PRs for unsoundness, but there is quite a lot of stuff that needs doing in the pretty near term. The internals of rlua are pretty hairy, which is why I don't expect anybody who wants to take up the mantle to go it completely alone, I'll happily be around to help with the hard parts. What it needs though is better leadership and organization skills than I have or at least have to spare currently.

(Also copied from the same reddit discussion)

For anybody who's interested in working on this, here's a quick wishlist of things that I think rlua needs in the near future that I would work on if I had the time:

rlua::Lua needs a compile-time switch to either require Send or to not require Send. Not-requiring Send seems like the obvious choice, but it makes it very hard to integrate into game engines that use threaded systems or perhaps into things like web servers.
rlua needs to either not use the lua_State "extradata" region or have a way to turn it off so it doesn't conflict with things trying to use it to create stand-alone Lua modules. The "extradata" region is unfortunately the fastest way to store arbitrary data alongside the main Lua state, which is why this has not been addressed yet.
rlua needs the ability to turn off the pcall / xpcall wrappers or any other provided function wrappers to be used in module position. They exist for a reason, but in module position it's very rude to muck with the global Lua state like this.
rlua needs to at least support Lua 5.4 when it is released, and ideally it would also support Lua 5.1 (and LuaJIT) and 5.2. This is potentially hard to make truly sound, but rlua has at least already done the hard work to protect against memory allocation and GC errors, so this is not completely infeasible.
rlua needs to give up on trying to patch out the UB in PUC-Rio Lua itself, and simply admit this fact by making the Lua constructor unsafe. This is COMPLETELY different than giving up on bindings soundness, I still consider bindings soundness to be extremely important, but this would give up on things like trying to protect rlua usage from things like the debug module or its many other ways to cause UB like directly loading dlls or even just through loading bytecode. Maybe some form of truly sandboxed Lua could be provided as a safe interface, but I don't think it should be advertised as the default because it removes too much of Lua itself.
rlua needs some ability to use async functions as coroutines and vice versa, but it also needs a way to in general more ergonomically return via lua_yield, possibly anywhere that a callback returns.
rlua contains within its guts a dirty lie, and this lie permeates the entire crate and makes proving the soundness of rlua very very hard. The signature of this type alias SHOULD be:

pub(crate) type Callback<'a> =
    Box<for<'lua> dyn Fn(Context<'lua>, MultiValue<'lua>) -> Result<MultiValue<'lua>> + 'a>;

Making this change would immediately remove much of the very delicate and honestly sort of sketchy logic in the callback creation process and the entirety of the "scope" system and make it more obviously sound. However, since I screwed this up when I initially created rlua and wrote the type signature wrong, I "accidentally" found a way to make it so that callback creation did not require macros. This has prevented me from fixing this because doing so almost certainly makes the API even less convenient by requiring macro wrappers around callbacks. There might be a way to fix this but I've tried very hard and haven't found any solution, it almost certainly requires GATs to fix properly. Edit: This is the biggest blocker in my mind to a 1.0 level API, but maybe being able to generally yield in callbacks also deserves to be a 1.0 blocker.

To continue the reddit discussion, it seems like there are a few of us that would be interested in taking on maintainer-ship by committee if that is an acceptable solution.

I'm a fan of both Rust and Lua, though I don't have the time to commit to rlua. That said, I'll take a look around when I'm bored and thanks for the great bindings.

@Grokmoo

🗣️ to @Grokmoo of https://github.com/Grokmoo/sulis

And if someone knows how to get in touch with @khvzak of mlua that'd be great! (edit: found their email on crates, sent a ping.)

@kyren

Thanks @kyren for looking into my work on the rlua fork.
Initially my primary goal was to bring Rust to Lua 5.3 and LuaJIT (the versions I work with) by providing a way to write high-level Lua modules in Rust. That's way I chose rlua 0.15 as a starting point, since it has more suitable API for this (I believe).
Therefore, standalone mode to bring Lua to Rust is also important and in focus!

My long-term goals are partially the same:

Provide Send / non-Send API or compile time switch.
Add Lua 5.4 support (the release candidate sources are already in the lua-src crate and (probably) MoonJIT (the project looks promising to be a luajit succeeder).
Out of the box cross compilation support to all major non x86 platforms (ARM/MIPS/etc) with CI (using qemu?)
Much more tests
More "batteries", like serde support (by new crates).
Improve documentation and create more examples

Regarding your comment in the reddit discussion:

The first mlua async implementation was built using lua_yieldk call. You can see this code. But later I postponed this idea as it does not work with Lua 5.1 (+ LuaJIT) and switched to more universal solution. Also, I'm thinking about providing support to define a custom yield function, as recently found that necessity and currently overriding the global coroutine.yield.
I'm going to return (optional?) custom allocator to the Lua constructor to provide more resilience/soundness.

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I'd be happy to get in touch and discuss about unsoundness problem that mlua have or maybe future of both projects.

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I think this might be the most important point to discuss. I think there is a more restricted form of soundness that is actually what people want that is still very useful to have, and I think the main Rust <-> Lua bindings system, whatever it is, should work very hard to maintain it.

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe. That by itself is somewhat of a useless property if as I'm proposing rlua::Lua::new becomes an unsafe function, by that definition anything is at that point sound because you must write unsafe. What I really mean by "soundness" here is more subtle, that nothing you can do directly with the bindings system can cause UB. In this definition, things like the debug table and loading C libraries inside Lua are "sound" because they are simply out of scope. This sounds like a cop-out but I really don't think it is, for example is opening a file and writing to /proc/self/mem unsound? Certainly not, despite the fact that it causes data races, it's just not within the scope for our definition, and the existence of /proc/self/mem doesn't make safety guarantees of rust any less useful.

So, for this library, "soundness" would mean bindings soundness, but as a way to make sure that people understand this, Lua::new() should just simply be unsafe. If at some point there is a good enough sanctioned sandboxing technique or a different implementation of Lua, then maybe a different method could be marked as safe, but as it stands I think this is similar to problems like what the memmap crate has. Whether or not a memory map is "safe" in rust is not just a function of the API, it is a function of the entire system as a whole, entirely different processes can cause data races with certain uses of memory maps, so the memmap crate just acknowledges this with an unsafe constructor and then proceeds to cause no extra harm.

This is (AFAIK currently) the state of the rlua crate, that it provides "bindings soundness". This is a really subtle point ofc and kind of hard to talk about, so maybe I just misunderstood what your project README said, perhaps mlua is already "bindings sound"? I was also worried because the 0.16 and 0.17 releases of rlua definitely also contain soundness fixes, however maybe the 0.16 API change is not a problem since you have also removed Lua::scope, but removing the scope system is somewhat unfortunate.

What I was worried about was there being an obviously best Lua bindings crate that does not attempt bindings soundness at all. You could make the case that since Lua is so unsafe, that there's no point anymore, but I definitely don't think that's true. I think that the kinds of things you can do in Lua that are unsafe are somewhat obvious: load C libraries, the debug table, bugs in the interpreter etc, but the kinds of things you can do that are unsafe in the Lua C API are anything but obvious, because the C API is so freakishly hard to safely use.

It's possible that your goal with mlua is already bindings soundness, and if so that's great! If that's true I don't see why there needs to be a second Lua bindings system really, both crates would have more or less the same goals in mind. My next question would be how we can work together instead of splitting effort, especially since you seem to be much more actively interested in this than I am currently.

There's some interesting perspectives: https://docs.rs/dtolnay/0.0.9/dtolnay/macro._03__soundness_bugs.html

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe.

I don't think that's a useful definition of soundness. The link above doesn't define it precisely, but my view is that the most useful definition of soundness is that UB is impossible when all code is either safe or uses unsafe in a way that adheres to the relevant documentation.

One of the examples for making frob sound is marking it as unsafe with documentation about it's precondition.

@khvzak

I don't think that's a useful definition of soundness. The link above doesn't define it precisely, but my view is that the most useful definition of soundness is that UB is impossible when all code is either safe or uses unsafe in a way that adheres to the relevant documentation.

Yeah, that's a better definition obviously but in this case that's still not it, becasue there is no contract with Lua::new you can fulfill because every possible method would have the same contract: don't cause Lua code to execute that can cause unsafe. Marking every method as unsafe is not helpful though, and you can limit the rules you have to follow to some categories of Lua code not to run rather than anything in the bindings layer like lifetime unsoundness. I'm saying we should allow the first but disallow the second, and that is the current situation with rlua itself.

You're correct in saying that simply writing unsafe at all does not give someone carte blanche to cause UB, that's not really what I meant to say, I was just skipping a lot of the detail. If the contract of Lua::new is "you cannot misuse the API", then by a certain definition it's "sound" but it's not useful, an example of this is real life could be "by calling Lua::new you agree not to overflow the Lua stack" or "by calling Lua::new you agree not to to throw errors in a __gc metamethod" or "by calling Lua::new you agree that a certain part of API is in unsafe if you use LuaJIT".

I promise I already understand what soundness is. Obviously an API that has an unsafe function with a clear cut guarantee is not sound if you call the unsafe function correctly and it causes UB, it's just that with Lua specifically the strawman I was describing is that the contract for Lua::new is "it's Lua, anything goes". The context here is this:

However, it's super hard (or even impossible) to guarantee safety in module mode where lua_State is something you cannot control and the environment can already have loaded debug along with other unsafe code.

I'm just arguing for "bindings soundness" even though the contract must technically be that any method at all can cause UB, I just want to limit how that UB may occur.

Also just to be clear I'm not saying @khvzak is suggesting this strawman definition either, I'm just trying to make sure our definitions are aligned. I think that even though someone can pass a lua_State and muck with the stack themselves, we should still aggressively strive for completely safe Lua C API usage, and advertise that fact.

I think the mmap analogy works the best. Creating an mmap or lua_State is unsafe and has a huge set of caveats, some easy to understand and some not. Documenting all of the caveats may not even be possible, in either case. But either way, the rest of the api can be sound in the face of a correctly created mmap or lua_State and that's what matters.

FWIW (I've separately wrapped Lua in a Rust crate but haven't got around to publishing it), my aims for safety/soundness sounds in line with to the above:

Bindings can be written in safe Rust
Ordinary Lua code can't cause Rust UB but I consider loading C libraries, using the debug library, and interpreter bugs (though I'd want to fix any I knew about) as out of scope.

My use cases are Lua as configuration - the user isn't malicious but may make mistakes and may be handling malicious data (e.g. from the web or e-mails).

I think rlua is doing a better job than my own bindings and is likely to be better maintained and has had more thought put into safety, so I may look into switching over. I'm interested in being involved but do not have time to be a full maintainer.

Usually when people ask if an API is sound they ofc mean that no UB is possible without writing unsafe.
...
What I really mean by "soundness" here is more subtle, that nothing you can do directly with the bindings system can cause UB. In this definition, things like the debug table and loading C libraries inside Lua are "sound" because they are simply out of scope.

Am I correctly understanding, that the following 100% rust safe code using rlua, which 1) modify RO variable 2) causes UB; is sound?

fn main() -> LuaResult<()> {
    let lua = Lua::new();
    lua.context(|ctx| {
        let memcpy: LuaFunction = ctx.load(r#"
            local ffi = require("ffi")
            ffi.cdef[[
            void * memcpy(void *, const void *, int);
            ]]
            return ffi.C.memcpy
        "#).eval()?;

        let a: i32 = 0;
        let b: i32 = 255;

        // Set "a" to "b"
        memcpy.call((&a as &i32 as *const i32 as i64, &b as &i32 as *const i32 as i64, 4))?;
        println!("{}", a); // Prints 255

        memcpy.call((&a as &i32 as *const i32 as i64, LuaNil, 4))?; // UB; Segfault

        Ok(())
    })
}

ffi is installed from this repo.

I was also worried because the 0.16 and 0.17 releases of rlua definitely also contain soundness fixes, however maybe the 0.16 API change is not a problem since you have also removed Lua::scope, but removing the scope system is somewhat unfortunate.

Could you please provide an example(s) of unsoundness in rlua 0.15, for better understanding ? The Scope module was removed in 0.3 since my higher priority was releasing async/await support which does not play nice with Scope system. I'm planning to return Scope support in the next release, but without create_nonstatic_userdata which is pain.

It's possible that your goal with mlua is already bindings soundness, and if so that's great!

I definitely not against bindings soundness, if it's possible and does not affect API usability. Soundness is a great advantage!
However, the example above shows the Rust "safety" rule violation even in rlua
From https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html

The need for all of this separation boils down a single fundamental property of Safe Rust, the soundness property:
No matter what, Safe Rust can't cause Undefined Behavior.

Am I correctly understanding, that the following 100% rust safe code using rlua, which 1) modify RO variable 2) causes UB; is sound?

I'm suggesting that Lua::new be marked as unsafe, but yes it is "bindings sound". The UB comes from Lua violating one of the rules I laid out, that it loads a dll to run C (asm?) code.

However, the example above shows the Rust "safety" rule violation even in rlua
From https://doc.rust-lang.org/nomicon/safe-unsafe-meaning.html

That's not bindings unsoundness, that's the trap I don't want to fall into, not to let perfect be the enemy of good. Since nothing in Lua can be perfectly sound without a complete sandboxing solution which is just not how people use Lua, it's not useful to mark every method that can trigger Lua code to run (which is nearly every method) as unsafe. Just mark Lua::new as unsafe and explain the situation.

Could you please provide an example(s) of unsoundness in rlua 0.15, for better understanding ? The Scope module was removed in 0.3 since my higher priority was releasing async/await support which does not play nice with Scope system. I'm planning to return Scope support in the next release, but without create_nonstatic_userdata which is pain.

Yeah the scope system was maybe the main one, but also there were other bugfixes, and I re-designed some of the lifetimes at some point after better understanding the mess I had gotten myself into with the dirty lie. If you didn't include the code that removes the need for the gc_guard hack, then there's probably soundness issues there too. Sorry I can't give you a more concrete answer I don't know when you forked and I don't have the energy right now to go digging that closely, 0.15.3 was released Oct 13, 2018.

Like I said it's possible that mlua already intends to be "bindings sound" and I just misunderstood the README changes.

@erlend-sh

I appreciate the shout out @erlend-sh. I will make more of an effort to understand the internals of rlua and see if that takes me to where I could make a useful contribution.

I'm also available to help out with more administrative or devops type stuff if the need arises.

I'm suggesting that Lua::new be marked as unsafe, but yes it is "bindings sound". The UB comes from Lua violating one of the rules I laid out, that it loads a dll to run C (asm?) code.

Maybe to mark Chunk::exec() as unsafe would be enough, but unsafe Lua::new seems better. I doubt that is possible to disable functionality like package.loadlib in Lua :(

Yeah the scope system was maybe the main one, but also there were other bugfixes, and I re-designed some of the lifetimes at some point after better understanding the mess I had gotten myself into with the dirty lie. If you didn't include the code that removes the need for the gc_guard hack, then there's probably soundness issues there too.

After forking rlua 0.15 I spent many time reading the code and changing it. I didn't include gc_guard and other (now legacy) code. The Scope subsystem in mlua 0.2 came with new lifetimes. I also refactored the internals (and found couple of bugs in rlua, but forgot the details), changed error handling, added metatables cache instead of static global u8 and did some other changes.

Personally I don't use Scope subsystem and doubt a bit about use cases. All the tables/strings/objects/etc created inside executed scoped functions will be alive. Only scope.create_(function|userdata) will be destroyed with tails remaining in Lua. Also it was quite hard for me to understand Scope lifetime design and why they were chosen as they are.
However I finished writing async_scope() variant of it to allow execution of scoped asynchronous code to return Scope in next releases.

I also doubt a bit the appropriateness of adding Context and radically changes in the API.
As I understood the key point was to catch at compile time cross-usage of multiple Lua states? (where object in Lua stateA pushed to a stateB). This check is performed at runtime in mlua (as in 0.15).
The Context API seems making modules creation more harder.

Like I said it's possible that mlua already intends to be "bindings sound" and I just misunderstood the README changes.

I hope so :) I believe there is no unsoundness at the moment (with the exception of UB caused by loading C code). I need to review the README.

@kyren

@kyren I was just thinking about doing a minor release (0.17.1 with #211) and realised I probably need your help. First to check if you have a release checklist/process (though I'm sure I can figure something reasonable out), but mainly because presumably it currently needs your key to publish to crates.io. Are you able to help sort something out for that?
Thanks!

@jugglerchris

@jugglerchris, I can't help with the release checklist, but Kyren also made me an owner of the rlua pacakge on crates and I can help with getting a release up.

@azdle

@azdle Thanks! So when I think things are ready, I can get you to run cargo publish on the right branch?

Definitely. You can @ me on the PR (if you do one) or send me an email (in my GH profile).

…

On Thu, Sep 2, 2021, at 16:06, Chris Emerson wrote: @azdle Thanks! So when I think things are ready, I can get you to run `cargo publish` on the right branch? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #172 (comment)

@khvzak

@khvzak have the unsound parts of mlua addressed here been fixed? If not could someone summarize what's unsound about mlua ?

@makspll

@makspll it's fixed. I don't have any open issues about mlua unsoundness. mlua moved far way forward since beginning of this discussion.

@jugglerchris

Hmm that's fantastic, in this case I would propose that rlua be retired to focus community efforts on a single package. Are there any benefits rlua has over mlua at this point in time ? @jugglerchris

@makspll

@makspll I don't know enough about mlua to answer that - I'm afraid I haven't been following it. From scanning https://crates.io/crates/mlua/ it sounds like lots of good stuff has been done (and it's getting more downloads). There are obviously some big design differences, for example no sign of rlua's Context in the examples - I guess I'd like to understand how that's different and why it's still safe.

@jugglerchris

@jugglerchris that's very fair. @khvzak would you mind elaborating on the safety without Context?

mlua uses bounded lifetimes for functions and a single coroutine-aware Lua instance.

Neither mlua nor rlua can guarantee absolute safely and Context is not a silver bullet here. If you discover any unsoundness in mlua I would appreciate a bug report.

@khvzak

@khvzak I am not saying mlua is unsound but rather wanting to spark discourse around the possibility of unifying the community behind one package. If you don't believe it possible or don't think it to be right direction I totally understand.

It just seems that there items in the roadmap for both packages (for example wasm support) which overlap, but I feel like development time would be more effectively spent if the workload was reduced and collaborators increased!

Do let me know if either of you disagree though!

@makspll

@makspll I agree that unifying and expanding a community around one package would be beneficial, but I don't see a way how this can done.
Unfortunately I don't share some rlua decisions, for example Context-based APIs (by the way, it was a reason to make a fork). Also our internals are very different.
Plus, I don't want mlua to be a part of any existing organization apart from a new one created for this project only.

@makspll

rlua only joined the amethyst organization because they had experience with managing open source projects and it only happened because rlua lost its original maintainer. So, I don't see a reason for mlua to join the amethyst organization.

the context API was (as far as I know) made to solve the problems that @makspll referred to. With mlua having solved those same problems in others ways I don't see why mlua should get the same context based api.

Now, the question if one should be deprecated over the other is a totally different question which I don't really have an opinion about. I also doubt that this issue is even the correct place to discuss that. Maybe it is best to move that to something dedicated to that to increase the chance of people seeing it and chime in compared to having that happen in a long closed issue?

Agreed that this isn't the best place to discuss it. I think probably the answer is that the two projects have slightly different aims and are complementary. I'd like to learn more about how mlua differs, though, in any case. Is there a good starting point to learn about the big differences?

Navigation Menu

Repository navigation

rlua is looking for maintainers! #172

18 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

rlua is looking for maintainers! #172

Description

Activity

kyren commented on Apr 21, 2020

rustysec commented on Apr 21, 2020

ahmedcharles commented on Apr 23, 2020

erlend-sh commented on Apr 23, 2020

khvzak commented on Apr 24, 2020

kyren commented on Apr 24, 2020

ahmedcharles commented on Apr 24, 2020

kyren commented on Apr 24, 2020

ahmedcharles commented on Apr 24, 2020

jugglerchris commented on Apr 24, 2020

khvzak commented on Apr 25, 2020

kyren commented on Apr 25, 2020

Grokmoo commented on Apr 27, 2020

khvzak commented on Apr 28, 2020

18 remaining items

jugglerchris commented on Sep 2, 2021

azdle commented on Sep 2, 2021

jugglerchris commented on Sep 2, 2021

azdle commented on Sep 2, 2021

makspll commented on Jun 27, 2022

khvzak commented on Jun 27, 2022

makspll commented on Jun 27, 2022

jugglerchris commented on Jun 29, 2022

makspll commented on Jun 29, 2022

khvzak commented on Jun 30, 2022

makspll commented on Jun 30, 2022

khvzak commented on Jun 30, 2022

lenscas commented on Jun 30, 2022

jugglerchris commented on Jul 2, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions