Repository navigation
dump's parameter "ensure_ascii" creates too long sequences #656
Description
I detected a problem in the code of PR #654:
The code seems to created too long \uxxxx sequences. Take the € sign for instance. It is U+20AC and should be encoded as string "\u20ac". The current code encodes it as "\u00e2\u0082\u00ac". This is incorrect, as this does not roundtrip.
Example:
#include <iostream>
#include <fstream>
#include "json.hpp"
using json = nlohmann::json;
int main() {
json j1 = u8"€";
std::cout << j1.dump(0, ' ', false) << std::endl;
std::cout << j1.dump(0, ' ', true) << std::endl;
json j2 = json::parse("\"\\u20ac\"");
std::cout << j2.dump(0, ' ', false) << std::endl;
std::cout << j2.dump(0, ' ', true) << std::endl;
json j3 = json::parse(j1.dump(0, ' ', true));
std::cout << j3.dump(0, ' ', false) << std::endl;
std::cout << j3.dump(0, ' ', true) << std::endl;
}Output:
"€"
"\u00e2\u0082\u00ac"
"€"
"\u00e2\u0082\u00ac"
"�"
"\u00c3\u00a2\u00c2\u0082\u00c2\u00ac"
Expected output:
"€"
"\u20ac"
"€"
"\u20ac"
"€"
"\u20ac"
Sorry for not detecting this earlier. The provided test case was correct as it coped with Emojis which created longer sequences anyway.
Activity
nlohmann commentedon Jul 12, 2017
Example in Python:
Output:
nlohmann commentedon Jul 12, 2017
We basically need a conversion from UTF-8 encoded chars to the Unicode codepoint and then use the existing escaping if the codepoint is 0..127 and UTF-16 hex(es) otherwise.
ryanjmulder commentedon Jul 12, 2017
yup, I agree. Sorry about the bug, my mistake.
nlohmann commentedon Jul 12, 2017
No worries. I used the all_unicode.json file, created a serialization in Python with the
ensure_asciiparameter and compared the output with the library'sdumpoutput.Any help appreciated - I'm going to bed now ;)
🐛 fix for #656
nlohmann commentedon Jul 17, 2017
I rewrote the escaping code. @ryanjmulder - if you can find the time, I would be happy if you could have a look at the diff.