Pikelet driver/loader API #175

Activity

1. brendanzab

   added this to the v0.1.0 milestone on Nov 5, 2018
2. brendanzab commented on Nov 5, 2018

   

   brendanzab

   on Nov 5, 2018

   MemberAuthor

   More actions

   Might be interesting to look at the Amethyst scripting API proposal for another perspective on what a client embedding Pikelet might want: amethyst/rfcs#1
3. brendanzab commented on Nov 5, 2018

   

   brendanzab

   on Nov 5, 2018

   MemberAuthor

   More actions

   Datafrog has a nice approach to evaluation, in that it leaves stepping the runtime up client of the library. This could be handy for embedding.
4. brendanzab commented on Nov 5, 2018

   

   brendanzab

   on Nov 5, 2018

   MemberAuthor

   More actions

   Also kind of interesting to have a look at the The Lua-C API for inspiration... 🤔
5. PaulBone commented on Nov 9, 2018

   

   PaulBone

   on Nov 9, 2018

   More actions

   So I havn't read up on this stuff all that much. But the kind of thing I'd imagine for such a loader API/language server interface is much simplier:

   Assert new info:

   • Import pre-existing library (eg, load a .so/.dll/bytecode)
   • Know this declaration (like an old C compiler seeing a forward declaration)
   • Load/compile this definition (give it some code and maybe some other meta-info like filename & line number of the code, Ask it to compile)
   • 'forget" this symbol. it forgets the name->declaration/definition mappings but a GC may cause the declaration/definition to hang around. You can now load a new definition (maybe also declaration)
     gine.

   A definition can be:

   • A type
   • A function
   • etc.

   Query info:

   • Tell me about a symbol's type
   • Execute a function with some arguments
   • anything else you can ima

   You can think of a tool supporting this API as calling parts of the compiler as a library call. The compiler itself may have a different interface (so it can do more optimisations / compile multiple definitions at once). Things like "typecheck this/compile this" are the know definition call. And things such as rename symbol and autoformat are not actually handled here, they're handled by whatever is making these calls such as an editor. Eg: rename symbol might do a forget then a know. Editors still work with text files which is the canonical version of the program, it just updates this online version to support editor functions/a repl, but can be thrown away at any time.

   Maybe you could add +reload definition+ and handle that by also re-compiling anything transitively that referred to that symbol.
6. brendanzab commented on Nov 10, 2018

   

   brendanzab

   on Nov 10, 2018

   MemberAuthor

   More actions

   Thanks for sharing your thoughts @PaulBone! Some nice food for thought!
7. brendanzab commented on Nov 10, 2018

   

   brendanzab

   on Nov 10, 2018

   Last edited by brendanzab

   MemberAuthor

   More actions

   jonathandturner/rhai looks like it has a nice API for this stuff!

   One nice thing is how you can register functions:

   extern crate rhai;
   use rhai::{Engine, RegisterFn};
   
   fn add(x: i64, y: i64) -> i64 {
       x + y
   }
   
   fn main() {
       let mut engine = Engine::new();
   
       engine.register_fn("add", add);
   
       if let Ok(result) = engine.eval::<i64>("add(40, 2)") {
          println!("Answer: {}", result);  // prints 42
       }
   }

   And types as well:

   extern crate rhai;
   use rhai::{Engine, RegisterFn};
   
   #[derive(Clone)]
   struct TestStruct {
       x: i64
   }
   
   impl TestStruct {
       fn update(&mut self) {
           self.x += 1000;
       }
   
       fn new() -> TestStruct {
           TestStruct { x: 1 }
       }
   }
   
   fn main() {
       let mut engine = Engine::new();
   
       engine.register_type::<TestStruct>();
   
       engine.register_fn("update", TestStruct::update);
       engine.register_fn("new_ts", TestStruct::new);
   
       if let Ok(result) = engine.eval::<TestStruct>("let x = new_ts(); x.update(); x") {
           println!("result: {}", result.x); // prints 1001
       }
   }

   This could help reduce the current mess we have in pikelet_elablorate::context.
8. brendanzab commented on Nov 14, 2018

   

   brendanzab

   on Nov 14, 2018

   MemberAuthor

   More actions

   Gluon also has nice embnedding and marshalling APIs that might be worth drawing inspiration from.
9. brendanzab commented on Nov 14, 2018

   

   brendanzab

   on Nov 14, 2018

   MemberAuthor

   More actions

   One issue that comes to mind is how closures might be handled, passing from Rust into Pikelet. Currently we only have an interpreter, so we can actually call closures during normalization. But eventually we'll want to have a JIT, and a code generator, so that might not be possible. This would most likely limit our ability to support things like rhai's Engine::register_fn API.
10. brendanzab commented on Nov 15, 2018

    

    brendanzab

    on Nov 15, 2018

    MemberAuthor

    More actions

    Here's a nice list of embeddable languages that we might be able to get inspiration from (thanks @photex!).

    I'm kind of feeling that you might have disjoint concerns when embedding with a JIT vs compiling to native code -in the former case you want to register Rust types and data/closures with the VM, and in the latter there you might want to statically/dynamically link to a Rust or a C library. It's kind of tricky to support both. 🤔
11. brendanzab commented on Nov 15, 2018

    

    brendanzab

    on Nov 15, 2018

    Last edited by brendanzab

    MemberAuthor

    More actions

    Chatted to the peeps on the Cranelift Gitter, and got some nice responses! @sunfishcode says:

    For passing closures into JIT'd code, the first version of this will look like: at the machine code level, you pass a pointer to the function in, and call it indirectly, passing in pointers to its data. Pretty low-tech to start with. But there are people working on building Cranelift-based Rust backends, which should open up more options in the future.

    They might be able to do some work to make this easier though, which would be neat!
12. Marwes commented on Dec 3, 2018

    

    Marwes

    on Dec 3, 2018

    More actions

    Thought I'd point out https://github.com/gluon-lang/gluon/blob/master/src/compiler_pipeline.rs . It is not perfect by any stretch but it has worked out quite well in gluon.

    High level overview is that It defines a trait for each compile step.

    MacroExpandable
    Renameable
    MetadataExtractable
    InfixReparseable
    ...
    

    Each trait takes Self as input and outputs it's own type on success MacroValue, Renamed, ... then I add two implementations for the trait, one on the previous steps output (impl Renamed for MacroValue and one as a blanket implementation on the previous trait Renameable for T where T: MacroExpandable which just calls the previous step (yielding a MacroValue in this example) and then calls the current step on that output.

    All in all, this makes it quite easy to run only the compile steps up to step X so a formatter may only need to parse, while a language server needs to run up to typechecking but no further). It also makes it possible to inject logic between steps and then continue the compilation without worrying about a step being omitted.

    👍React with 👍2Reacted by Brendan Zabarauskas and Kevin Leimkuhler
13. brendanzab commented on Dec 3, 2018

    

    brendanzab

    on Dec 3, 2018

    MemberAuthor

    More actions

    Oh nice! I like this! This is cool too:

    pub type SalvageResult<T> = Result<T, (Option<T>, Error)>;