You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Move lexer punctuation confusables list to unicode-security crate and sync it with newest Unicode version. #70002
It's a bit tough because that list isn't automatically generated, or it has been at a point in the past but since then it has been manually edited and changed. Those amendments are valuable and should not be lost by an automatic regeneration. One could think about having two lists though, one automatically generated, the other with the rust-specific edits/improvements.
As i've recently got a crate that has confusables.txt data in it. I made a small analysis on existing items in fore-mentioned list. There're 178 items that are already covered by confusables.txt, 85 items not covered, and one duplicate item (Canadian Syllabics Final Middle Dot).
Yes, and rust syntax here means most of ASCII punctuation characters.
I saw one major difference between this list and the UTS39 one is that most of the items here are just unintentional, and not for security reasons. For example,
('，', "Fullwidth Comma", ','),
I'm using an IME daily, and a single shift key press toggles between Latin mode and Han mode, where the comma key corresponds to these two characters in each mode. And it's really easy to unintentionally get them wrong. So while this item is not in UTS39 list, it is practically quite useful.
changed the title [-]Move lexer punctuation confusables list to `unicode-security` crate.[/-][+]Move lexer punctuation confusables list to `unicode-security` crate and sync it with newest Unicode version.[/+]on Mar 26, 2021
Activity
jonas-schievink commentedon Mar 14, 2020
#69929
est31 commentedon Mar 14, 2020
@jonas-schievink That PR is missing https://github.com/rust-lang/rust/blob/9e6fb538f9254884ca9f958fdce413d6c3f2016c/src/librustc_parse/lexer/unicode_chars.rs
newest list is https://www.unicode.org/Public/security/13.0.0/confusables.txt
See also #51080
Mark-Simulacrum commentedon Mar 14, 2020
cc @estebank @rust-lang/wg-diagnostics re:confusables list
est31 commentedon Mar 14, 2020
It's a bit tough because that list isn't automatically generated, or it has been at a point in the past but since then it has been manually edited and changed. Those amendments are valuable and should not be lost by an automatic regeneration. One could think about having two lists though, one automatically generated, the other with the rust-specific edits/improvements.
crlf0710 commentedon Apr 30, 2020
As i've recently got a crate that has
confusables.txtdata in it. I made a small analysis on existing items in fore-mentioned list. There're 178 items that are already covered byconfusables.txt, 85 items not covered, and one duplicate item (Canadian Syllabics Final Middle Dot).crlf0710 commentedon Apr 30, 2020
The former two lists are here. Also cc @Manishearth to see whether the second list is of some value to further development of UTS39.
Manishearth commentedon Apr 30, 2020
The in-tree list is specifically about things which are confusable with rust syntax
crlf0710 commentedon May 1, 2020
Yes, and rust syntax here means most of ASCII punctuation characters.
I saw one major difference between this list and the UTS39 one is that most of the items here are just unintentional, and not for security reasons. For example,
I'm using an IME daily, and a single shift key press toggles between Latin mode and Han mode, where the comma key corresponds to these two characters in each mode. And it's really easy to unintentionally get them wrong. So while this item is not in UTS39 list, it is practically quite useful.
[-]Update to Unicode 13.[/-][+]Move lexer punctuation confusables list to `unicode-security` crate.[/+][-]Move lexer punctuation confusables list to `unicode-security` crate.[/-][+]Move lexer punctuation confusables list to `unicode-security` crate and sync it with newest Unicode version.[/+]crlf0710 commentedon Mar 26, 2021
Updated the issue title to reflect the actual discussion topic here - the libs part is already done in #69929.