Add support for repetition to proc_macro::quote

@dtolnay

Part 1: complete in #141608

I'm opening a standalone issue to have something with "help wanted" labels in case anyone is interested in picking this up.

Our proc_macro::quote does not support repetition, unlike quote from the quote crate. As mentioned many times on the tracking issue, this is something we should support or at least account for before proc_macro::quote can be stabilized.

This should use the syntax:

proc_macro::quote! {
    $( CONTENTS )SEP*
}

Where CONTENTS is the thing to be repeated and SEP is an optional single-character separator. Expansion should work for anything that implements IntoIterator. This matches the quote crate's logic (except quote::quote uses # rather than $).

It's probably easiest to just copy quote's logic here, which uses an extension trait to facilitate this.

quote crate source repo: https://github.com/dtolnay/quote.
Link to source for our quote that needs to be updated: https://github.com/rust-lang/rust/blob/fa58ce343ad498196d799a7381869e79938e952a/library/proc_macro/src/quote.rs
Link to tests that should be updated: https://github.com/rust-lang/rust/blob/c02a4f0852e6665cf3df3867982021383f5615df/tests/ui/proc-macro/quote/auxiliary/basic.rs (plus any new tests in tests/ui/proc-macro/quote for things we should reject)
Tracking issue: Tracking issue for the quote! macro in proc_macro #54722

Part 2: incomplete

From @dtolnay's comment at #141608 (comment), there are some more minor issues to be resolved before stabilization:

#![feature(proc_macro_quote)]

use proc_macro::TokenStream;

macro_rules! decl {
    ($($iter:tt)*) => {
        stringify!($($iter) << *)
    };
}

#[proc_macro]
pub fn repro(input: TokenStream) -> TokenStream {
    // macro_rules macro
    let tokens = decl!(a b c);
    eprintln!("{}", tokens);

    // quote crate
    let input2 = proc_macro2::TokenStream::from(input.clone());
    let iter2 = input2.into_iter();
    let tokens = quote::quote!(#(#iter2) << *);
    eprintln!("{}", tokens);

    // libproc_macro
    let iter = input.into_iter();
    let tokens = proc_macro::quote!($($iter) << *);
    eprintln!("{}", tokens);

    TokenStream::new()
}

Macro_rules macro: a << b << c
Quote crate: a << b << c
Libproc_macro:

error: proc macro panicked
  --> src/lib.rs:25:18
   |
25 |     let tokens = proc_macro::quote!($($iter) << *);
   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: message: `$(...)` must be followed by `*` in `quote!`

Another example:

error[E0425]: cannot find value `j` in this scope
  --> src/lib.rs:25:47
   |
25 |     let tokens = proc_macro::quote!($$ j $($$ j $iter)*);
   |                                               ^ not found in this scope

The parsing logic will need some more scrutiny in followup PRs before the macro can be stabilized.

@dtolnay

Cc @dtolnay in case you have any notes here

@SpriteOvO

Also fyi @SpriteOvO if you might be interested in this, since you have done a lot of other work with quote!

I recommend copying from the quote crate. It is tricky to get something that has all the behaviors that people expect are "obvious". If you see $($thing)*,—

You cannot turn it into for x in $thing {...} because this is incompatible with $($thing)* $($thing)* where thing is something non-copyable like a Vec, a ubiquitous use case.
You cannot turn it into for x in &$thing {...} because this is incompatible with thing being an Iterator, a ubiquitous use case.
You cannot turn it into for x in &mut $thing {...} because this requires let mut thing in the caller's code which is highly unexpected.
On top of all the above, in $($thing1 $thing2)* the correct behavior depends on which one of these is an iterator/collection and which one is a single token (Ident) or TokenStream. For a TokenStream, even though TokenStream has an IntoIterator impl, using that here produces an extremely unexpected result.

Hi there!

Excuse me for interrupting, but I’d like to try working on this PR when I have time — as long as it’s not an urgent task. (I’m not very familiar with Rust macros, but I have solid experience with meta-programming in OCaml and Scala, so I believe I can handle it given enough time.)

The main task seems to be porting the logic from declarative macros to procedural macros:
https://github.com/dtolnay/quote/blob/ab1e92c27a492e6077e203a6f85496bb4b1522e8/src/lib.rs#L888-L937

The most challenging part for me is likely the parsing.
In the declarative macro logic, parsing is done using a sliding window with 3-token(-tree) lookahead and lookback. To replicate this in procedural macros, I'll need to implement the logic in a stateful way.
However, I don’t have much experience with hand-written stateful parsers, so it may take me some time. (More importantly, I’ll first need to fully understand the overall BNF.)

@moatom

@moatom I don't see why you have to replicate the parsing logic of the crate. I'm sure it won't interfere with the quoting behavior. The code for the quote crate I feel is unnecessarily complex by virtue of it being a declarative macro, with one of the comments also saying that it is written that way to avoid using a muncher. I think recursive descent (or a simpler method) would suffice.

As for the grammar, I think I have got it here, though I'm not 100% sure I got everything correct. It is quite simple:

<content> ::= <tokentree> | <tokentree> <quote> <content> | ε
<quote> ::= "$" <ident> | "$" <repetition>
<repetition> ::= "(" <content> ")" <quantifier>
<quantifier> ::= "*" | <punct> "*"

Most of the logic here is landing in #141608. There are a few follow up items David mentioned that I added to the top post.

Rollup merge of rust-lang#141608 - moatom:proc_macro-140238, r=dtolnay

Unrolled build for #141608

Rollup merge of #141608 - moatom:proc_macro-140238, r=dtolnay

Rollup merge of rust-lang#141608 - moatom:proc_macro-140238, r=dtolnay

@rustbot

@rustbot claim

@tgross35

~~@tgross35 @dtolnay Hi!, May I ask about the first error:~~

25 |     let tokens = proc_macro::quote!($($iter) << *);
   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

~~I'm considering preparing a test that checks all the operators defined in compiler/rustc_ast/src/ast.rs (specifically BinOpKind, AssignOpKind, and UnOp, as far as I understand).~~
~~Would that be sufficient?~~

[Need investigation]
It may not be an error related to operator support in minimal_quote_ts, but rather an issue with parsing this part:

let sep_opt: Option<Punct> = match (iter.next(), iter.peek()) {
                        (Some(TokenTree::Punct(sep)), Some(TokenTree::Punct(star)))
                            if sep.spacing() == Spacing::Joint && star.as_char() == '*' =>
                        {
                            iter.next();
                            Some(sep)
                        }
...

I see that #148209 is no longer being worked on. Can I work on this or should I hold off?

@kohsine

@kohsine
I haven't received any further feedback in #148209 for a while. I suspect this is because the reviewer was concerned about the compatibility of my parsing logic, so I was planning to verify the compatibility and explain it when I have time.

However, it may take me until around the end of the year to get to it. So if you are able to take this issue on in the meantime, please feel free to do so!

Navigation Menu

Repository navigation

Add support for repetition to `proc_macro::quote` #140238

Part 1: complete in #141608

Part 2: incomplete

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add support for repetition to proc_macro::quote #140238

Description

Part 1: complete in #141608

Part 2: incomplete

Activity

tgross35 commented on Apr 24, 2025

tgross35 commented on Apr 24, 2025

dtolnay commented on Apr 24, 2025

moatom commented on May 15, 2025

ora-0 commented on May 26, 2025

tgross35 commented on Jun 16, 2025

moatom commented on Oct 21, 2025

moatom commented on Oct 21, 2025

kohsine commented on Sep 10, 2026

moatom commented on Sep 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Add support for repetition to `proc_macro::quote` #140238