Repository navigation
Relax bare key restrictions to allow additional unicode letters and numbers #687
Description
Activity
- changed the title
[-]Relax bare key character restrictions to allow some additional 'letter-like'[/-][+]Relax bare key character restrictions to allow some additional 'letter-likes'[/+]on Dec 10, 2019 I wholeheartedly support this. Identifiers in non-English languages should not be discriminated against in TOML (and having to put them in quotes is a form of discrimination). [Note: the following refers to the original form of the proposal, which since then has been considerably extended.] Admittedly, with this proposal this would still only be the case for languages using the Latin script, but not for Russian, Arabic, Chinese etc. But it would still be a step in the right direction – and I see that there might be issues with allowing, say, Cyrillic letters, because they might be used to spoof a key that looks like ASCII but actually isn't. With Latin diacritics, this risk is much lower.
For completeness, I'd suggest to also support Latin Extended-B (Pan-Nigerian alphabet, Pinyin, Romanian), Latin Extended Additional (Vietnamese), and Latin Extended-C (Shona, a Bantu language).
Reacted by Mark Gillard, Clar Fon and Eric ArellanoSo there would be no simple rule for human writer to decide if quotes are necessary.
Even good old C++ supports unicode characters in identifiers on modern compilers.
Do "modern C++ compilers" support most of Unicode, or only chosen subset of Latin Extended?
@ChristianSi Certainly there's lots of additional characters we could add. My list of suggestions was in no way meant to be exhaustive and I'm hoping that a more useful set of ranges is borne out of discussion. Being an Australian who only speaks English does limit my perspective a bit here!
So there would be no simple rule for human writer to decide if quotes are necessary.
Technically-speaking the rule could be: quotes if you need whitespace, an escape code, or a TOML-reserved character, otherwise anything goes. My feeling is that requiring users to think about this at all is getting away from the design goals of TOML and drifting too far into Think-Like-A-Programmer territory, which risks defeating the purpose of a simple config file format that intends to "just work" the way people would expect in the layman case.
Do "modern C++ compilers" support most of Unicode, or only chosen subset of Latin Extended?
Absolutely no idea. Here's a live demo of some Unicode on Clang, GCC and MSVC; I encourage you to experiment. I'm certain if we needed a more definitive answer we could read the compiler source code (Clang or GCC) or ask the relevant developers (MSVC).
Reacted by Christian SiefkesI'm on board.
I like the way Python handles alphabetical characters:
Alphabetic characters are those characters defined in the Unicode character database as “Letter”
That should be do-able for us, but I wonder how could complicate TOML implementations which are currently just doing a simple ASCII-value-range check. This also means we'd have to add a lot to TOML's ABNF.
Reacted by Christian Siefkes and Dave Ostroske@pradyunsg: I too would be fine with saying "Bare keys may contain arbitrary Unicode letters as well as ASCII digits, underscores, and dashes". But in effect this would likely mean that implementations would have to depend on some kind of Unicode library – easy in Python, where the
isalpha()check is built in, probably not so easy in some other languages.An advantage of @marzer 's original proposal, or my somewhat modified one, is that it would be easy to enumerate the affected ranges manually, in the ABNF or in code. With arbitrary Unicode letters this likely becomes effectively impossible – especially since the ranges would have to be extended with each new version of the Unicode standard.
But, of course, we might also decide to allow letters in arbitrary languages and scripts, not just Latin-based ones, and accept the Unicode dependency.
Reacted by Mark GillardHere's a listing of all the Unicode Character Categories and all the characters that belong to each one. Unsurprisingly, it's quite long!
@ChristianSi Given that TOML is supposed to be UTF-8 I'm inclined to think that requiring implementations use unicode machinery, hand-rolled or otherwise, isn't really a big deal, regardless of the direction this proposal takes.
@pradyunsg I too like the python approach, and I don't think it would be too hard to implement. I'm currently writing a TOML library of my own and I'd be happy to build it into my utf-8 decoder as a proof-of-concept, if that's useful.
Reacted by Christian Siefkes, Eric Arellano and TR Lim@ChristianSi I wrote a script to scrape letter characters from the website you linked, sort them and list them as ranges. If you omit the letter categories
LmandLothe set of character ranges seems totally manageable:removed, see below
@pradyunsg I don't know much about ABNF's but if characters can be expressed as ranges this wouldn't be much work.
@marzer Interesting. But the problem is that nearly all Unicode letters are in the "Letter, other" (Lo) category – 97 percent according to Wikipedia. Ignoring them, you only get the letters in alphabets that distinguish between upper case and lower case forms – Latin, Cyrillic, Greek, and a few others. But most writing systems don't – e.g. those used to write Chinese, Arabic, Hebrew, Korean, and certain Indian languages such as Tamil and Telugu know no such distinction. Hence their letters go into the "other" category.
What happens when you consider all letters? I suppose ranges become a bit unwieldy?
@ChristianSi Surprisingly it's not that unwieldy:
removed, see below
@ChristianSi Ah, good find. I'll update the script later tonight and see how it looks.
23 remaining items
I'm coming to this as someone who is incorporating TOML into a project with keys that will often contain symbols/punctuation. I've read through this thread and I have not seen anyone propose that keys allow any valid unicode except the symbols needed by the TOML parser itself. That would:
- maximize permissiveness
- create an easy and small set of rules for when people need to quote keys (if the key contains space, dot, brackets, quotes, etc, then it must be quoted)
- not strictly require a unicode library for a parser
- Likely be faster than other options to parse
I'm not currently arguing this is the best approach but it seemed worth adding to the set of options in the discussion space.
Reacted by Michaël DefferrardIf anyone is interested in playing around with a parser that supports this tentative feature (as specified in the OP, anyways), my C++ TOML library is now in a publishable state: https://marzer.github.io/tomlplusplus/
@thoughtafter It seems as though your suggestion is very much in-line with @abelbraaksma's (which from my reading, advocates including everything except syntactically-relevant/ambiguous characters).
Reacted by Christian Siefkes, Abel Braaksma and TR LimIn my opinion, we don't program in any language, including English. What we are coding is symbol. ASCII in programming is safe symbols, not English.
In high level languages, identifier could be defined as any charactor, because here is IDE and highlight.
But TOML is designed for ini file, usually no any extra support when editing.
That's also why (and the mainly reason why TOML exists) TOML is better than YAML—because we can't indent/deindent easily, for nested values or multiline strings.
In any other language, indent is better than non-indent design, we all know that.So I think that's really dangerous to allow bare keys include non-ASCII charactor.
- If we support complex charactor range under sematic of Unicode, like JS (also HTML) does, then all languages users need bear the mental burden of all other languages which they not use and unfamiliar. It's not minimal.
- If we support all non-ASCII as identifier, like CSS does, then there could be a lot of invisible characters maybe written in key, which is not obvious. As ini files are usally used in low level symtem program config, it's so dangerous.
But, I think spec allow implementations to support user specified language bare keys support is good. What languages you are fimilar, you use that. For example,
/[\u4e00-\u9fa5]/is Chinese, so it can be easily supported and then easy to write bare key, and, safe. But who you know and care? But as specific language user, I know, I can pass range argument as an options value to parser, preserving highly controlled at the same time.I think simplicity and nationality could be no conflict, not must one or the other—otherwise, absolute "fairness" will lead to widespread inefficiency.
@LongTengDao it's a config file format, not a database specification or real-time streaming format; I don't think 'efficiency' is all that relevant (if you mean the computational complexity of parsing, that is).
Unless you mean the efficiency of the actual implementing of the new functionality? As in, it will be a bit complex for implementers and maintainers to get this working in their parsers, thus being inefficient for them? If so, that's not even true. It's pretty easy to implement. I've done it myself, and provide relevant information in the original post.
I'm not sure what other sorts of efficiencies you could mean. It wouldn't make TOML any less efficient to write (if anything it would get simpler and easier to use as a result of this proposal).
Reacted by Christian Siefkes and Abel Braaksma@marzer I've never considered the difficulty of writing a parser is a hindrance, and it's not worth considering in the face of a perfectly formatted file design task. If anyone objects to this, I will be on your side.
I only mean the efficiency of writing and checking. Introducing special characters too broadly will make the process of reading and writing a file stressful again. Remember, Unicode doesn't just include characters in common languages like the ones you and me use (1en or 1em width). Instead, there are many combinations of display and invisible even right-to-left characters that are in character category, rather than punctuation or whitespace category. It's a nightmare, if you've ever developed an typesetting software like Office Word. After that, I have been frightened by words like "all valid Unicode".
But anyway, it doesn't affect my use. If spec said ASCII only, I will support user options to support any Unicode character range. If spec said any Unicode character is valid as your suggestion, I will support user options to limit ASCII only. I think this right belongs to user.
Reacted by Christian SiefkesReacted by Abel BraaksmaIf spec said any Unicode character is valid as your suggestion
@LongTengDao to be clear, my proposal isn't to support "any Unicode character", as you seem to think. It's to support a subset (letters, numbers, and some combining marks).
Yeah there might be characters in those categories that are effectively garbage for our purposes but they can probably just be ignored; if it's not a character on a keyboard then someone has gone to effort to put it in their config, and if that breaks stuff then that's the life they chose. Parser library users can trivially add additional sanity-checking if they feel the need.
Reacted by Christian Siefkes and Abel BraaksmaInstead, there are many combinations of display and invisible even right-to-left characters that are in character category, rather than punctuation or whitespace category.
@LongTengDao I believe the opposite to be true. Limiting users that are accustomed to right-to-left writing means limiting over 1.7 billion people worldwide to a system that is not native to them. What is perhaps perceived by you as "a nightmare" is perceived by others as a nightmare if it isn't allowed. Not everyone speaks English or can write in their native tongue using only ASCII characters (in fact, it is a relative small share of the world population).
Inclusion of other cultures, languages and writing systems is a good thing, and although TOML is not a programming language, many well-known programming language embrace inclusion more than exclusion: C#, VB, F# (allows any character), Java (they allow a broader set than defined here), Ruby, Perl, XML/HTML tag names, CSS classes/id and there are many more.
Unicode even has a specific TR that describes the recommended way for allowing Unicode characters in identifiers: https://unicode.org/reports/tr31/.
Differences between languages will always exist, but the closer a language (or a spec like TOML) gets to TR31, the better it is for the worldwide community of thousands of languages that can write in their native tongue.
If any company or individual wishes to limit the allowed set of characters in identifiers, or in coding in general, they are of course free to do so, just like coding styles exist for many programming languages, you could limit your style to "only ASCII" or whatever you prefer.
And as already has been said, the proposal here is a safe subset of the Unicode language.
Reacted by Christian Siefkes, Dave Ostroske, Ariel and Mark GillardRegarding ranges full of marks, you mention that "they are not allowed as start character in XML and we should probably disallow it as well." Maybe you could investigated that further? It would be interesting to see which ranges are prohibited in NameStartChar, but allowed in NameChar – and why. If they are full of non-ASCII digits, we might want to allow them even at the start, but if they are full of marks, we certainly don't.
@ChristianSi, apologies for the wait, I forgot about your question here.
The precise definition of
NameCharis that ofNameStartCharwith a few additions. These additions are therefore not allowed as a starting character:NameChar ::= NameStartChar | "-" | "." | [0-9] | #xB7 | [#x0300-#x036F] | [#x203F-#x2040]Let's split that up. According to this, the following are not allowed as a starting character (and I believe this mostly follows current practice in TOML as well):
- The minus sign
-and the period. - Any ASCII digit
0-9 - Middle dot
xB7, or· x300 - x36F, these are the combining diacritical marks, which means you cannot start with a combiningacuteorcircumflexfor instance, see https://en.wikipedia.org/wiki/Combining_Diacritical_Marks. I think this makes sense.- The "tie",
x203Fandx2040, or‿(under tie)⁀(character tie), https://en.wikipedia.org/wiki/Tie_(typography)
I'm not sure why the "tie" is forbidden as starting character in XML names (it is not a combining tie, it is spaced), but the other ones seem sensible.
I can write up a TOML spec proposal for this set, and/or extend it to TR31 if that somehow makes sense, but I think it is easier for people in general to use the XML specification (without reference to XML, of course, as it is otherwise unrelated), since they already did the necessary research, it's concise, and it's trivial to implement. TR31 is quite hard to read and probably raises new questions again.
Reacted by Ariel- The minus sign
This looks like we're missing a PR for doing this. If someone wants to pick this up, and file a PR expanding the allowed bare keys syntax to include letters from the broader unicode spec, that'd be welcome!
@pradyunsg, done, I've created a PR in #891. I tried to be both as inclusive as possible, while maintaining simplicity for parsers. Basically the rule is now: "Any Unicode letter, letterlike character or digit, except dot", as discussed above.
This has now been merged. Thanks everyone for their support and insights!
Reacted by Dave Ostroske and Christian SiefkesReacted by Christian Siefkes- added 2 commits that reference this issue
on Apr 14, 2025
Issue
TOML's "bare key" syntax is too restrictive. People who regularly use characters from languages other than English should be able to do so in TOML keys without additional gymnastics.
I know there's already been a lot of discussion about this but much of it was from when TOML was less established and I think it warrants revisiting.
Proposed change
Expand the set of accepted characters allowed in bare keys to include letters and numbers from the entire Unicode space, similar to how identifiers are handled in other Unicode-compliant contexts (e.g. python, javascript, etc.). Specifically:
Rationale
After reading much of the existing discussion on the issue, I've identified the points below as being the main objections. I've written a counterpoint for each.
"ASCII-only is easy to understand"
Allowing Unicode letters and numbers wouldn't change the understandability of the written word in "mostly-ASCII" contexts, excepting maybe people from English-centric countries encountering characters they otherwise rarely see and being unsure how to pronounce them. I'm one of those people and my brain seems to consume them just fine. And it's almost certainly going to improve the understandability of bare keys to people for whom an ASCII environment is not their regular one.
It also wouldn't change the semantic/syntatic understandability of the language; I'm only advocating relaxing the spec to allow letter and number characters, not anything that might be confused for a language construct (no math symbols, for instance).
"Guides users to choose simple key names"
See above. I'd argue that the keys would be no less simple with this change. I live and work in a European country and a number of my friends and colleagues have non-ASCII letters in their name (e.g.
ä). I doubt they consider their names to be complex; I certainly don't. If anything, by forcing people to jump through hoops just to type in their language, we're actually making the key names more complex w.r.t. cognitive load."Eliminate any weirdness that could come from having to deal with undelimited Unicode"
The TOML spec dictates UTF-8, not UTF-8-ish. UTF-8 is a solved problem at this point. If a parser doesn't correctly detect and handle malformed UTF-8, I'd argue that the parser needs fixing, not that we should bend over to accommodate users who are using crap tools and libraries. It's such a solved problem that you can even portably consume it using a state machine and validate it using vector intrinsics.
"Keys should be identifier-like"
Despite the fact that the concept of an "identifier" isn't a thing in TOML, I'll concede that in some situations this might be a concern. A reasonable example is using TOML in code generation contexts; if you used TOML keys to inform variable names historically you'd run into issues in many languages with non-ASCII characters, though this is no longer true. Even good old C++ supports unicode characters in identifiers on modern compilers.
...all of which is rendered moot by the fact that TOML supports hyphens in bare keys which are often invalid in identifier contexts, so this objection is a non-starter anyway.
"It complicates implementation"
It really doesn't. Many implementations will be able to leverage built-in helper functions or libraries for working with Unicode. For those that can't, I've put my money where my mouth is and implemented this as a proof-of-concept in my own TOML parser and I'm happy for my code to be used as a starting point:
is_unicode_XXXXX()codepoint identity functions (generated by a script)Of course you might argue that simply accepting UTF-8 bytes from a TOML implementation is not an option for everyone, and you'd be right; there will always be situations where only ASCII makes sense (e.g. legacy codebases). I'd respond by pointing out that detecting non-ASCII characters in a character stream is laughably trivial. Applications requiring ASCII-only can easily enforce this themselves.