A macro to generating AST with python grammar #55

Activity

1. MichaReiser commented on May 18, 2023

   

   MichaReiser

   on May 18, 2023

   Contributor

   More actions

   I'm struggling to understand what's happening in

   &python_ast! {
       Vars { names },
       "(*names)"
   }
   

   even with the example. I'm also vary of macros because most Rust tooling breaks down (formatting, autocompletion) and can be difficult to understand if you haven't used the macro before.

   Have you considered alternative mutation APIs? E.g. Rome has two APIs:

   • SyntaxFactory::new_if_stmt(condition, body) for the creation of nodes
   • Node::with_X(x) -> Node methods for returning a new copy with X set to a specific value.

   This allows, in combination, to write SyntaxFactory::new_if_stmt(condition, body).with_orelse(orelse).
2. youknowone commented on May 18, 2023

   

   youknowone

   on May 18, 2023

   MemberAuthor

   More actions

   It turns a python snippet to python ast with given values. It has very different user experience to factories.

   Starting from simpler example

   &python_ast! {
       "def new_function(a, b, c): pass"
   }

   will be parsed to

   FunctionDef(
         name='new_function',
         args=arguments(
           posonlyargs=[],
           args=[
             arg(arg='a'),
             arg(arg='b'),
             arg(arg='c')],
           kwonlyargs=[],
           kw_defaults=[],
           defaults=[]),
         body=[
           Pass()],
         decorator_list=[])
   

   The factory will be somewhere between it, but it will be more close to the latter.

   Simple value example

   let new_function_name = Identifier::new("overriding_name");
   &python_ast! {
       Vars { new_function_name },
       "def new_function_name(a, b, c): pass"
   }

   will turned into

   FunctionDef(
         name='name',
         args=arguments(
           posonlyargs=[],
           args=[
             arg(arg='a'),
             arg(arg='b'),
             arg(arg='c')],
           kwonlyargs=[],
           kw_defaults=[],
           defaults=[]),
         body=[
           Pass()],
         decorator_list=[])
   

   By looking in every node and checking its identifier name new_function_name is same as the given Vars list.

   More complex one, similar to the original example.

   let args: Vec<_> = ["a", "b", "c"].iter().map(|name| Identifier::new(name));
   let values = HashMap::new();
   values.insert("a", 10);
   values.insert("b", 20);
   values.insert("c", 30);
   
   &python_ast! {
       Vars { args, values },
       r#"
       def new_function(*args):
           return { **values }
       #"
   }

   will be originally parsed to:

       FunctionDef(
         name='new_function',
         args=arguments(
           posonlyargs=[],
           args=[],
           vararg=arg(arg='args'),
           kwonlyargs=[],
           kw_defaults=[],
           defaults=[]),
         body=[
           Return(
             value=Dict(
               keys=[
                 None],
               values=[
                 Name(id='values', ctx=Load())]))],
         decorator_list=[])],
   

   and then folded to

       FunctionDef(
         name='new_function',
         args=arguments(
           posonlyargs=[
             arg(arg='a'),
             arg(arg='b'),
             arg(arg='c')],
           args=[],
           kwonlyargs=[],
           kw_defaults=[],
           defaults=[]),
         body=[
           Return(
             value=Dict(
               keys=[
                 Constant(value='a'),
                 Constant(value='b'),
                 Constant(value='c')],
               values=[
                 Constant(value=10),
                 Constant(value=10),
                 Constant(value=10)]))],
         decorator_list=[])],
   

   More with comprehension

   let args: Vec<_> = ["a", "b", "c"].iter().map(str::to_owned).collect();
   
   &python_ast! {
       Vars { args },
       r#"
       def new_function(*args):
           return [f"prefixed_{arg}" for arg in args]
       #"
   }

   originally turns into

       FunctionDef(
         name='new_function',
         args=arguments(
           posonlyargs=[],
           args=[],
           vararg=arg(arg='args'),
           kwonlyargs=[],
           kw_defaults=[],
           defaults=[]),
         body=[
           Return(
             value=ListComp(
               elt=JoinedStr(
                 values=[
                   Constant(value='prefixed_'),
                   FormattedValue(
                     value=Name(id='arg', ctx=Load()),
                     conversion=-1)]),
               generators=[
                 comprehension(
                   target=Name(id='arg', ctx=Store()),
                   iter=Name(id='args', ctx=Load()),
                   ifs=[],
                   is_async=0)]))],
         decorator_list=[])],
   

   and then folded to

       FunctionDef(
         name='new_function',
         args=arguments(
           posonlyargs=[
             arg(arg='a'),
             arg(arg='b'),
             arg(arg='c')],
           args=[],
           kwonlyargs=[],
           kw_defaults=[],
           defaults=[]),
         body=[
           Return(
             value=List(
               elts=[
                 Constant(value='prefixed_a'),
                 Constant(value='prefixed_b'),
                 Constant(value='prefixed_c')],
               ctx=Load()))],
         decorator_list=[])],
   
3. youknowone commented on May 18, 2023

   

   youknowone

   on May 18, 2023

   Last edited by youknowone

   MemberAuthor

   More actions

   I'm also vary of macros because most Rust tooling breaks down (formatting, autocompletion) and can be difficult to understand if you haven't used the macro before.

   Unlike other macros, the linked smart_quote! doesn't break tools due to the similar restriction - it must be always a valid rust code.

   One good thing about it is python_ast! doesn't include any rust code inside. It only requires python code and binding list of variables.