Repository navigation
Parallelize parsing. #25
Description
This can be an interesting idea to keep in mind for the future. I have already experimented with single-consumer-single-producer queues in order to have a token-producing and a token-consuming queue, but the performance is lack-luster, even when using state-of-the-art data structures and algorithms.
A possible approach here would be to have a 2-layered parsing approach: one would divide the input with a really simple grammar, and the other would parallelly parse these divisions.
Activity
[-]Parallelise parsing & processing.[/-][+]Parallelize parsing & processing.[/+][-]Parallelize parsing & processing.[/-][+]Parallelize parsing.[/+]gzz2000 commentedon Sep 27, 2022
+1 on this. Not only does this allow parallelization, but it can also improve performance without actually opening new threads. This is because we will no longer need to parse in two passes (first get token tree in pest, then interpret the result into custom data structure). This provides better memory locality with very large inputs.