Skip to content

re: no ASCII fast path for \w in Unicode mode, unlike \s #159251

Open
Open
re: no ASCII fast path for \w in Unicode mode, unlike \s#159251
Labels
performancePerformance or resource usagetopic-regextype-featureA feature request or enhancement

Description

@maurycy
maurycy
opened on Oct 12, 2026

Feature or enhancement

Use the existing ASCII table to speed up word matching for ASCII characters, while keeping Unicode by default.

For example:

>>> import re
>>> re.findall(r"\w+", "hello café 123")
['hello', 'café', '123']

Most characters in this example are ASCII, but the re still uses Unicode classification for each character that \w examines.

Idea

The idea is for chars less than 128, use SRE_IS_WORD(). For the rest, keep the existing Unicode predicate:

#define SRE_UNI_IS_WORD(ch)\
    ((ch) < 128 ? SRE_IS_WORD(ch) : (SRE_UNI_IS_ALNUM(ch) || (ch) == '_'))

The same predicate is used by \W, \b and \B so it'd speed them up, too.

We're doing it for \s:

cpython/Include/cpython/unicodeobject.h

Lines 753 to 758 in 9028df3

static inline int Py_UNICODE_ISSPACE(Py_UCS4 ch) {
if (ch < 128) {
return _Py_ascii_whitespace[ch];
}
return _PyUnicode_IsWhitespace(ch);
}

Microbenchmark

Benchmark main patched
\w+ findall, ASCII prose 295 us 235 us (1.25x faster)
\w+|[^\w\s]+ (NLTK WordPunct), ASCII prose 417 us 327 us (1.28x faster)
\W+ findall, ASCII prose 321 us 276 us (1.16x faster)
\b\w findall, ASCII prose 347 us 223 us (1.55x faster)
\w+ findall, ASCII punctuation only 132 us 81.8 us (1.61x faster)
\w+ findall, Latin-1 prose 326 us 270 us (1.21x faster)
\w+ findall, CJK text 71.8 us 77.4 us (1.08x slower)
\w+ findall, mixed scripts 206 us 172 us (1.20x faster)
\w+ findall with re.ASCII (control) 229 us not significant
\d+ findall, ASCII prose (control) 154 us 165 us (1.07x slower)

Linux (AMD EPYC 4484PX, Ubuntu 24.04, gcc 13), PGO+LTO builds of main at 9028df3

perf stat -r 5 on the same machine:

Case main instructions patched instructions main cycles patched cycles
\w+ ASCII prose 14.85G 12.14G (-18%) 3.20G 2.61G (-18%)
NLTK WordPunct, ASCII prose 15.64G 12.84G (-18%) 3.32G 2.68G (-19%)
\w+ CJK text 13.63G 13.50G (-1%) 2.97G 3.27G (+10%)
\w+ with re.ASCII (control) 12.08G 12.09G (0%) 2.57G 2.55G (-1%)
\d+ ASCII prose (control) 14.85G 14.85G (0%) 3.03G 3.09G (+2%)

So the \d+ is likely just noise between PGO+LTO builds, but 100% CJK seems to be real cost. Are we ok with it?

pyperf microbenchmark 
import re

import pyperf

ascii_text = ("The quick brown fox jumps over the lazy dog, said Alice; "
              "it was 42 degrees outside and the kettle_was on. ") * 200
latin_text = ("Zażółć gęślą jaźń, señor; la niña está aquí: café, "
              "naïve, crème brûlée. ") * 300
cjk_text = "漢字仮名交じり文で書かれた文章です。これはテストの文章です。" * 300
mixed_text = (ascii_text[:2000] + latin_text[:1000] + cjk_text[:500]) * 4
punct_text = "!!! ??? ... --- *** ((( ))) ;;; ::: ,,, " * 300

word_run = re.compile(r"\w+")
nltk_wordpunct = re.compile(r"\w+|[^\w\s]+")
nonword_split = re.compile(r"\W+")
boundary = re.compile(r"\b\w")
ascii_flag = re.compile(r"\w+", re.ASCII)
digit_run = re.compile(r"\d+")


def findall(pattern, text):
    return pattern.findall(text)


runner = pyperf.Runner()
runner.bench_func(r"\w+ findall, ASCII prose", findall, word_run, ascii_text)
runner.bench_func(r"\w+|[^\w\s]+ (NLTK WordPunct), ASCII prose", findall, nltk_wordpunct, ascii_text)
runner.bench_func(r"\W+ findall, ASCII prose", findall, nonword_split, ascii_text)
runner.bench_func(r"\b\w findall, ASCII prose", findall, boundary, ascii_text)
runner.bench_func(r"\w+ findall, ASCII punctuation only", findall, word_run, punct_text)
runner.bench_func(r"\w+ findall, Latin-1 prose", findall, word_run, latin_text)
runner.bench_func(r"\w+ findall, CJK text", findall, word_run, cjk_text)
runner.bench_func(r"\w+ findall, mixed scripts", findall, word_run, mixed_text)
runner.bench_func(r"\w+ findall with re.ASCII (control)", findall, ascii_flag, ascii_text)
runner.bench_func(r"\d+ findall, ASCII prose (control)", findall, digit_run, ascii_text)

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

Activity

  1. maurycy
    added
    type-featureA feature request or enhancement
    on Oct 12, 2026

  2. maurycy commented on Oct 12, 2026

    @maurycy
    maurycy
    on Oct 12, 2026
    ContributorAuthor

    cc @serhiy-storchaka if we're ok with the trade-off (slowdown for no ASCII, like pure CJK)? If so, perhaps we could think about Py_UNICODE_ISALNUM?

  3. terryjreedy commented on Oct 12, 2026

    @terryjreedy
    terryjreedy
    on Oct 12, 2026
    Member

    Preliminary question: would it make sense to expand ascii table(s?) to latin1 table(s?) and/or add -N flag (N for latiN since L is taken)?
    Assuming that re has access to the internal CPython string kind (1/2/4 bytes per char), I should think it would.

    I also expect the slowdown for any all-non-latin1 string. Though it would make me sad to completely give up the speedup for the ascii/latin1 world, I don't think the tradeoff is fair. Let's think harder about how to reduce the tradeoff here and maybe elsewhere. The ascii-else-unicode expression is only needed for long-enough mixed ascii/non-ascii strings with enough ascii to be a net benefit. Perhaps it could be made opt-in.

    Note: I am not sure what happens when one compiles a mixed string with the .A flag.

  4. picnixz
    added
    performancePerformance or resource usage
    extension-modulesC modules in the Modules dir
    topic-regex
    and removed
    extension-modulesC modules in the Modules dir
    on Oct 13, 2026

  5. picnixz commented on Oct 13, 2026

    @picnixz
    picnixz
    on Oct 13, 2026
    Member

    For CJK texts: can we have a adaptive threshold that enables the optimization according to the amount of text being parsed instead? usually regexp are for finding things in text of your own language. So if the input is CJK text, obvioulsy, hitting \w would be less frequent.

    That being said, it also makes me wonder whether we can have optimization flags, e.g., thehe user says "this regex will be used for English text" and thus we can enable specific optimizations in general (not just the re.ASCII one). Or disable it if necessayr. I don't know how hard it is tfor that though.

    cc @serhiy-storchaka

  6. maurycy commented on Oct 13, 2026

    @maurycy
    maurycy
    on Oct 13, 2026
    ContributorAuthor

    Thank you for the comments.

    I reran perf stat -r 5 more rigorously (only CJK; two builds per branch, two interleaved runs):

    Counter main a1 main a2 patched b1 patched b2 A/A B/B patched vs main
    instructions 33.96G 33.97G 33.39G 33.40G +0.0% +0.0% -1.7%
    cycles 7.25G 7.04G 7.39G 8.12G -2.9% +9.9% +8.5%
    branch-misses 2.33M 2.30M 2.38M 2.45M -1.5% +2.6% +4.2%
    stalled-cycles-frontend 216.90M 114.67M 240.82M 657.97M -47.1% +173.2% +171.1%
    op-cache misses 14.83M 13.71M 14.05M 136.29M -7.5% +870.0% +426.7%
    op-cache miss ratio 0.25% 0.24% 0.23% 2.17% -5.1% +834.2% +386.8%
    L1-icache-load-misses 91.8K 96.5K 101.3K 93.9K +5.1% -7.3% +3.6%
    iTLB-load-misses 13.7K 13.8K 13.7K 14.6K +0.6% +6.0% +2.7%

    Also, perf record shows more samples in sre_category (a1: 972, b1: 1136, b2: 1129), and cmp $0x7f,%esi is the hottest instr there. The profile also changes elsewhere, particularly in sre_ucs2_count. (I can share but rechecking would be valuable, too.) Specifically, PGO+LTO places the ch >= 128 path about 2KB away from the compare, with a jump back.

    Hypothesis: PGO training is mostly ASCII, so CJK is a regression but its size depends on where it lands in the op cache. That'd explain why b1 is 3% and b2 is 11%.

    @terryjreedy Thinking harder how to reduce the trade-off:

    diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h
index 54defdb..7339a55 100644
--- a/Include/cpython/unicodeobject.h
+++ b/Include/cpython/unicodeobject.h
@@ -777,11 +777,10 @@ static inline int Py_UNICODE_ISSPACE(Py_UCS4 ch) {
 
 #define Py_UNICODE_ISALPHA(ch) _PyUnicode_IsAlpha(ch)
 
+PyAPI_FUNC(int) _PyUnicode_IsAlnum(Py_UCS4 ch);
+
 static inline int Py_UNICODE_ISALNUM(Py_UCS4 ch) {
-   return (Py_UNICODE_ISALPHA(ch)
-           || Py_UNICODE_ISDECIMAL(ch)
-           || Py_UNICODE_ISDIGIT(ch)
-           || Py_UNICODE_ISNUMERIC(ch));
+   return _PyUnicode_IsAlnum(ch);
 }
 
 
diff --git a/Modules/_sre/sre.c b/Modules/_sre/sre.c
index 47c2c4a..da816a2 100644
--- a/Modules/_sre/sre.c
+++ b/Modules/_sre/sre.c
@@ -170,7 +170,8 @@ static unsigned int sre_upper_locale(unsigned int ch)
 #define SRE_UNI_IS_SPACE(ch) Py_UNICODE_ISSPACE(ch)
 #define SRE_UNI_IS_LINEBREAK(ch) Py_UNICODE_ISLINEBREAK(ch)
 #define SRE_UNI_IS_ALNUM(ch) Py_UNICODE_ISALNUM(ch)
-#define SRE_UNI_IS_WORD(ch) (SRE_UNI_IS_ALNUM(ch) || (ch) == '_')
+#define SRE_UNI_IS_WORD(ch)\
+    ((ch) < 128 ? SRE_IS_WORD(ch) : (SRE_UNI_IS_ALNUM(ch) || (ch) == '_'))
 #define SRE_UNI_IS_ALPHA(ch) Py_UNICODE_ISALPHA(ch)
 #define SRE_UNI_IS_LOWER(ch) Py_UNICODE_ISLOWER(ch)
 #define SRE_UNI_IS_UPPER(ch) Py_UNICODE_ISUPPER(ch)
diff --git a/Objects/unicodectype.c b/Objects/unicodectype.c
index fdd3801..0a0f369 100644
--- a/Objects/unicodectype.c
+++ b/Objects/unicodectype.c
@@ -279,6 +279,15 @@ int _PyUnicode_IsCaseIgnorable(Py_UCS4 ch)
 /* Returns 1 for Unicode characters having the category 'Ll', 'Lu', 'Lt',
    'Lo' or 'Lm',  0 otherwise. */
 
+int
+_PyUnicode_IsAlnum(Py_UCS4 ch)
+{
+    const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
+
+    return (ctype->flags
+            & (ALPHA_MASK | DECIMAL_MASK | DIGIT_MASK | NUMERIC_MASK)) != 0;
+}
+
 int _PyUnicode_IsAlpha(Py_UCS4 ch)
 {
     const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);

    (already updated in #159252 as 6c5f44f)

    As above, two builds per variant:

    Benchmark main this PR PR + single mask
    \w+ ASCII prose 249.6 / 251.1 us 192.0 / 190.5 us (-23.6%) 194.3 / 190.2 us (-23.2%)
    \w+|[^\w\s]+ (NLTK WordPunct), ASCII prose 381.6 / 395.4 us 283.2 / 285.7 us (-26.8%) 279.1 / 285.5 us (-27.3%)
    \W+ ASCII prose 265.4 / 281.3 us 227.6 / 225.0 us (-17.2%) 224.5 / 219.6 us (-18.8%)
    \b\w ASCII prose 297.8 / 313.4 us 185.9 / 202.8 us (-36.4%) 185.1 / 185.2 us (-39.4%)
    \w+ ASCII punctuation only 116.5 / 121.9 us 65.3 / 65.4 us (-45.2%) 66.4 / 65.4 us (-44.7%)
    \w+ Latin-1 prose 274.6 / 277.4 us 219.5 / 229.5 us (-18.7%) 215.9 / 219.1 us (-21.2%)
    \w+ CJK text 62.8 / 63.3 us 64.5 / 68.6 us (+5.6%) 52.5 / 53.3 us (-16.1%)
    \w+ mixed scripts 169.6 / 172.3 us 140.5 / 142.9 us (-17.1%) 136.5 / 137.7 us (-19.8%)
    \w+ with re.ASCII (control) 190.9 / 190.0 us 190.9 / 191.9 us (+0.5%) 192.0 / 190.6 us (+0.5%)
    \d+ ASCII prose (control) 126.7 / 129.3 us 127.9 / 131.0 us (+1.2%) 133.4 / 129.1 us (+2.6%)

    perf stat -r 5 (only CJK):

    Counter main this PR PR + single mask
    instructions 33.96G / 33.97G 33.39G / 33.40G (-1.7%) 29.07G / 29.07G (-14.4%)
    cycles 7.25G / 7.04G 7.39G / 8.12G (+8.5%) 6.10G / 6.13G (-14.4%)
    branch-misses 2.33M / 2.30M 2.38M / 2.45M (+4.2%) 2.20M / 2.19M (-5.2%)
    stalled-cycles-frontend 216.90M / 114.67M 240.82M / 657.97M (+171.1%) 105.85M / 115.13M (-33.4%)
    op-cache misses 14.83M / 13.71M 14.05M / 136.29M (+426.7%) 12.69M / 12.81M (-10.7%)
    op-cache miss ratio 0.25% / 0.24% 0.23% / 2.17% (+386.8%) 0.25% / 0.24% (+0.5%)
    L1-icache-load-misses 91.8K / 96.5K 101.3K / 93.9K (+3.6%) 90.9K / 94.3K (-1.7%)
    iTLB-load-misses 13.7K / 13.8K 13.7K / 14.6K (+2.7%) 12.5K / 12.9K (-7.7%)

    str.isalnum() microbenchmark:

    Benchmark main PR + single mask
    str.isalnum(), ASCII word 8000 chars 21.4 / 18.6 us 10.2 / 10.2 us (-49.0%)
    str.isalnum(), ASCII prose (fails at char 4) 42.7 / 44.5 ns 36.9 / 38.4 ns (-13.7%)
    str.isalnum(), ASCII word + trailing '!' 19.8 / 18.6 us 10.2 / 10.2 us (-46.9%)
    str.isalnum(), Latin-1 word 7500 chars 20.0 / 16.8 us 10.9 / 11.2 us (-39.9%)
    str.isalnum(), CJK word 8500 chars 20.2 / 19.2 us 12.4 / 12.5 us (-36.9%)
    str.isalnum(), CJK prose (fails at char 17) 77.0 / 76.0 ns 54.9 / 58.4 ns (-25.9%)
    str.isalnum(), 1200 short words 20.7 / 21.3 us 16.2 / 16.9 us (-21.2%)
    str.isalnum(), 3600 prose tokens 83.8 / 81.9 us 63.1 / 60.8 us (-25.2%)
    str.isalnum() microbenchmark 
    import pyperf

ascii_word = "kettle42" * 1000
ascii_prose = ("The quick brown fox jumps over the lazy dog, said Alice; "
                "it was 42 degrees outside and the kettle_was on. ") * 200
latin_word = "Zażółćgęśląjaźń" * 500
cjk_word = "漢字仮名交じり文で書かれた文章です" * 500
cjk_prose = "漢字仮名交じり文で書かれた文章です。これはテストの文章です。" * 300
tail_punct = "kettle42" * 1000 + "!"
short_words = ["kettle", "42", "café", "漢字", "a", "x1"] * 200
mixed_tokens = ("The quick brown fox jumps over the lazy dog, said Alice; "
                "it was 42 degrees outside and the kettle_was on. " * 200).split()


def isalnum(s):
    return s.isalnum()


def isalnum_many(words):
    return [w.isalnum() for w in words]


runner = pyperf.Runner()
runner.bench_func("isalnum, ASCII word 8000 chars", isalnum, ascii_word)
runner.bench_func("isalnum, ASCII prose (fails at char 4)", isalnum, ascii_prose)
runner.bench_func("isalnum, ASCII word + trailing '!'", isalnum, tail_punct)
runner.bench_func("isalnum, Latin-1 word 7500 chars", isalnum, latin_word)
runner.bench_func("isalnum, CJK word 8500 chars", isalnum, cjk_word)
runner.bench_func("isalnum, CJK prose (fails at char 17)", isalnum, cjk_prose)
runner.bench_func("isalnum, 1200 short words", isalnum_many, short_words)
runner.bench_func("isalnum, 3600 prose tokens", isalnum_many, mixed_tokens)

    I note

    cpython/Lib/test/test_capi/test_unicode.py

    Lines 2600 to 2614 in 48ddda1

    # TODO: Add tests to the following character functions:
    # - Py_UNICODE_ISALNUM
    # - Py_UNICODE_ISALPHA
    # - Py_UNICODE_ISDECIMAL
    # - Py_UNICODE_ISDIGIT
    # - Py_UNICODE_ISLINEBREAK
    # - Py_UNICODE_ISLOWER
    # - Py_UNICODE_ISNUMERIC
    # - Py_UNICODE_ISPRINTABLE
    # - Py_UNICODE_ISSPACE
    # - Py_UNICODE_ISTITLE
    # - Py_UNICODE_ISUPPER
    # - Py_UNICODE_TODECIMAL
    # - Py_UNICODE_TODIGIT
    # - Py_UNICODE_TOLOWER

  7. maurycy
    added a commit that references this issue on Oct 13, 2026

    http://github.com/python/cpython/issues/159251#issuecomment-6140434277

    6c5f44f

  8. maurycy commented on Oct 13, 2026

    @maurycy
    maurycy
    on Oct 13, 2026
    ContributorAuthor

    I think that (ALPHA_MASK | DECIMAL_MASK | DIGIT_MASK | NUMERIC_MASK)) != 0 might be the answer here.

    It could be also in str.islower() ((flags & (UPPER_MASK | TITLE_MASK)) != 0), str.isupper() ((flags & (LOWER_MASK | TITLE_MASK)) != 0), str.istitle() ((flags & (TITLE_MASK | UPPER_MASK)) != 0). Then, we could go with Lu as (flags & (UPPER_MASK | ALPHA_MASK)) == (UPPER_MASK | ALPHA_MASK), Cf as (flags & (CASE_IGNORABLE_MASK | PRINTABLE_MASK)) == CASE_IGNORABLE_MASK etc.

    What do you think?

    I can extend the PR to all masks and add missing test coverage, but would prefer to make sure we're on the same boat. This is a bit more than the initial ch < 128.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagetopic-regextype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions