OS X 10.11: system calls return non-normalized unicode strings

@evanlucas

test/sequential/test-chdir.js persistently fails on OS X 10.11 because the output of process.cwd() doesn't match the path we're doing process.chdir() on before.

Here's a reduced test case (npm i mkdirp hexy):

const mkdirp = require('mkdirp');
const hex = require('hexy').hexy;
const dir = __dirname + '/weird \uc3a4\uc3ab\uc3af characters \u00e1\u00e2\u00e3';

mkdirp.sync(dir);
process.chdir(dir);

console.log(hex(new Buffer(dir)));
console.log(hex(new Buffer(process.cwd())));
console.log(dir.length, dir);
console.log(process.cwd().length, process.cwd());

The strings look identical on the terminal, but the bytes differ. Here's the output:

00000000: 2f55 7365 7273 2f73 696c 7665 7277 696e  /Users/silverwin
00000010: 642f 6769 742f 6368 6469 722f 7765 6972  d/git/chdir/weir
00000020: 6420 ec8e a4ec 8eab ec8e af20 6368 6172  d.l.$l.+l./.char
00000030: 6163 7465 7273 20c3 a1c3 a2c3 a3         acters.C!C"C#

00000000: 2f55 7365 7273 2f73 696c 7665 7277 696e  /Users/silverwin
00000010: 642f 6769 742f 6368 6469 722f 7765 6972  d/git/chdir/weir
00000020: 6420 e184 8ae1 85a7 e186 abe1 848a e185  d.a..a.'a.+a..a.
00000030: a7e1 86b2 e184 8ae1 85a7 e186 b620 6368  'a.2a..a.'a.6.ch
00000040: 6172 6163 7465 7273 2061 cc81 61cc 8261  aracters.aL.aL.a
00000050: cc83                                     L.

52 '/Users/silverwind/git/chdir/weird 쎤쎫쎯 characters áâã'
61 '/Users/silverwind/git/chdir/weird 쎤쎫쎯 characters áâã'

cc: @evanlucas

@Fishrock123

Just for comparision, could someone post the output of above script on 10.10? Maybe @Fishrock123?

output on 10.10.4:

00000000: 2f55 7365 7273 2f6d 7a61 7373 6f2f 7465  /Users/mzasso/te
00000010: 7374 2f77 6569 7264 20ec 8ea4 ec8e abec  st/weird.l.$l.+l
00000020: 8eaf 2063 6861 7261 6374 6572 7320 c3a1  ./.characters.C!
00000030: c3a2 c3a3                                C"C#

00000000: 2f55 7365 7273 2f6d 7a61 7373 6f2f 7465  /Users/mzasso/te
00000010: 7374 2f77 6569 7264 20ec 8ea4 ec8e abec  st/weird.l.$l.+l
00000020: 8eaf 2063 6861 7261 6374 6572 7320 c3a1  ./.characters.C!
00000030: c3a2 c3a3                                C"C#

43 '/Users/mzasso/test/weird 쎤쎫쎯 characters áâã'
43 '/Users/mzasso/test/weird 쎤쎫쎯 characters áâã'

@silverwind

@silverwind Maybe we can check what is at the code points using String.prototype.codePointAt(n) or similar? There are different ways of denoting unicode via code points, and it looks like El Capitan is expanding them. If so, that may actually be an OS bug.

@silverwind

@silverwind note that your script has a bug: you should use mkdirp.sync or it fails the first time because the directory doesn't exist yet when process.chdir is executed

Perhaps this can help:

const mkdirp = require('mkdirp');
const dir = __dirname + '/weird \uc3a4\uc3ab\uc3af characters \u00e1\u00e2\u00e3';

mkdirp.sync(dir);
process.chdir(dir);

function getChars(str) {
  var chars = [];
  for (var c of str) chars.push(c);
  return chars;
}

var dirC = getChars(dir);
var cwdC = getChars(process.cwd());

console.log(dirC.length);
console.log(cwdC.length);

for (var i = 0; i < dirC.length; i++) {
  if (dirC[i].codePointAt(0) !== cwdC[i].codePointAt(0))
    throw `Different code point at ${i}: ${dirC[i]} - ${cwdC[i]}`;
}

console.log('strings are identical');

On 10.10, strings are identical.

@targos

@targos fixed the mkdirp. Here's your script's output on 10.11:

52
61

/Users/silverwind/git/chdir/codepoints.js:21
    throw `Different code point at ${i}: ${dirC[i]} - ${cwdC[i]}`;
                                                               ^
Different code point at 34: 쎤 - ᄊ

@saghul

Either the bug is in uv_cwd or (more likely) in the OS. cc: @saghul

@silverwind

@silverwind could you modify it again to make the output run through String.fromCodePoint()?

Edit: Actually, it would be more helpful do get more out than just one of the conflicts... so log instead of throw also.

@Fishrock123

@Fishrock123 not sure if fromCodePoint is of much help. I don't think I can feed bytes into it. For simplicy, here's just a single character compared:

const mkdirp = require('mkdirp');
const hex = require('hexy').hexy;
const dir = __dirname + '/\uc3a4';

mkdirp.sync(dir);
process.chdir(dir);

const a = dir.substring(__dirname.length);
const b = process.cwd().substring(__dirname.length);

console.log(hex(new Buffer(a)));
console.log(hex(new Buffer(b)));
console.log(a.length, a);
console.log(b.length, b);

Output:

00000000: ec8e a4                                  l.$

00000000: e184 8ae1 85a7 e186 ab                   a..a.'a.+

1 '쎤'
3 '쎤'

It goes from 3 to 9 bytes. What kind of unicode encoding is that?

I meant in logging the points ala

for (var i = 0; i < dirC.length; i++) {
  if (dirC[i].codePointAt(0) !== cwdC[i].codePointAt(0))
    throw `Different code point at ${i}: ${dirC[i]} - ${cwdC[i]}`;
}

@Fishrock123

@Fishrock123 just comparing the 6 unicode characters (\uc3a4\uc3ab\uc3af\u00e1\u00e2\u00e3):

00000000: ec8e a4ec 8eab ec8e afc3 a1c3 a2c3 a3    l.$l.+l./C!C"C#

00000000: e184 8ae1 85a7 e186 abe1 848a e185 a7e1  a..a.'a.+a..a.'a
00000010: 86b2 e184 8ae1 85a7 e186 b661 cc81 61cc  .2a..a.'a.6aL.aL
00000020: 8261 cc83                                .aL.

6 '쎤쎫쎯áâã'
15 '쎤쎫쎯áâã'
Different code point at 0: 쎤 - ᄊ
Different code point at 1: 쎫 - ᅧ
Different code point at 2: 쎯 - ᆫ
Different code point at 3: á - ᄊ
Different code point at 4: â - ᅧ
Different code point at 5: ã - ᆲ

I don't know how chinese works but it's like if OSX decomposed the character into smaller pieces.
Look how the 3 first on the right (ᄊ, ᅧ, ᆫ) can be used together to create the first one on the left (쎤).

What happens with a character like 🚀 '\uD83D\uDE80' ?

@targos

@targos lol, didn't notice it actually was the three parts combined. 🚀 looks fine:

00000000: f09f 9a80                                p...

00000000: f09f 9a80                                p...

2 '🚀'
2 '🚀'

Noticed the same "assembling" happens on upper ascii characters like umlauts öäü, where the dots are a separate character following the ascii letter. I've seen this before on bugged webfonts, not sure what this mechanism is called.

Is there any disadvantage to using string.normalize() on mac? To my knowledge it should to what's correct per system, and will just return the string unmodified if node is built without Intl?

Yes, string.normalize should be a noop without Intl.

Remember thought, that for comparison to work, both strings must be in the same form. If we decide to normalize all file names returned by the system to NFC, there's still the possibilty that the user tries to compare it to NFD obtained through other means, which would fail.

Maybe the best course of action is to advice to do str1.normalize() === str2.normalize() when comparing any unicode strings, regardless of origin.

If we decide to normalize all file names returned by the system to NFC

This would not be correct, i.e. the same as making fs.readFile normalize CRLF line endings to LF because people are not aware of the difference. Filenames should be treated as data.

Maybe the best course of action is to advice to do str1.normalize() === str2.normalize() when comparing any unicode strings, regardless of origin.

Spot on. Although, when comparing file system strings, one must take into account whether the file system preserves form or not. e.g. When comparing filenames on Windows or Linux (or on any form-preserving filesystem), do not use normalize to compare, use ===. And on HFS+ or any non-form-preserving filesystem use normalize('NFD') to compare. Windows or Linux will preserve and return NFC or NFD as per Ben's comment above so using normalize on those filesystems would be wrong as two similar looking files could exist in different Unicode forms.

@Fishrock123

Is there any disadvantage to using string.normalize() on mac? To my knowledge it should to what's correct per system, and will just return the string unmodified if node is built without Intl?

@Fishrock123 this would fix the test where normalize works. But to fix the test to work where there is no ICU, it would be better to just change the test to create a NFD form of the directory to start with. That will work on all OS X versions.

Did some more testing. It seems that Windows and Linux file systems allow files with either NFD or NFC form and they are both preserved. On Mac, I can create either a file in NFD or NFC form but in both cases it ends up to be just one file in NFD form. That also means, if any String.normalize() is being used, it must only be used on Mac because the other OS allow for both forms.

I am still a bit puzzled why Mac behaves this way. If you compare this behavior to how case-sensitive file names are handled it would mean that Mac only allows 1 casing and it would always lowercase your filename.

Am I missing something?

That's exactly right. Apple may have thought that some users would be confused by different normalization forms, and they also wanted to be form-insensitive, so instead of preserving the form, they just convert abc to ABC and lose the form in the process. Meanwhile, the other file-systems preserve form and are sensitive to form.

One can think of it exactly as dealing with a non-case-preserving file-system.

One can think of it exactly as dealing with a non-case-preserving file-system.

I think a better comparision would be when you think of your data as case-insensitve (you wouldn't use uppercase for files in your project, would you?), and the OS starts to return uppercase for certain letters based on the characters used. It's a mess in my eyes.

WIP fix for this test: #3007
Documentation suggestion: nodejs/docs#42

test: use normalize() for unicode paths

Fixed the test in 81e98e9. Docs regarding this will follow - see nodejs/docs#42.

test: use normalize() for unicode paths

A guide is now up that explains how to work with filesystems that do Unicode normalization: https://nodejs.org/en/docs/guides/working-with-different-filesystems/

fix(macos): fullscreen titlebar reserve, instant Finder path paste, N…

Navigation Menu

Repository navigation

OS X 10.11: system calls return non-normalized unicode strings #2165

57 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

OS X 10.11: system calls return non-normalized unicode strings #2165

Description

Activity

silverwind commented on Jul 12, 2015

targos commented on Jul 12, 2015

Fishrock123 commented on Jul 12, 2015

targos commented on Jul 12, 2015

targos commented on Jul 12, 2015

silverwind commented on Jul 12, 2015

silverwind commented on Jul 12, 2015

Fishrock123 commented on Jul 12, 2015

silverwind commented on Jul 12, 2015

Fishrock123 commented on Jul 12, 2015

silverwind commented on Jul 12, 2015

targos commented on Jul 12, 2015

silverwind commented on Jul 12, 2015

silverwind commented on Jul 12, 2015

57 remaining items

Fishrock123 commented on Aug 24, 2015

silverwind commented on Aug 24, 2015

jorangreef commented on Aug 24, 2015

jorangreef commented on Aug 24, 2015

jorangreef commented on Aug 25, 2015

silverwind commented on Aug 25, 2015

silverwind commented on Sep 22, 2015

silverwind commented on Oct 6, 2015

jorangreef commented on Nov 17, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions