Store file names in POSIX format

[jfontan]

Current implementation stores file names in the index in the format sent by the program using the library. This means that the format stored in Windows and Linux differ. This fact makes incompatible using siva files created in different OS as path separator is not the same. For example, Index.Glob did not work correctly as it was not able to find directory separators.

This change makes old siva files created in Windows incompatible but is always compatible when the file was created in a POSIX OS.

Now the index header names are written in POSIX format and sanitized with ToSafePath. Also Find and Glob convert input paths with it.

This change makes interoperation between Windows and Linux possible.

[ajnavarro]

LGTM!

Shall we change siva spec to reflect that changes? https://github.com/src-d/go-siva/blob/master/SPEC.md#index

[erizocosmico]

UNIX or POSIX?

[jfontan]

I've used UNIX as POSIX also mentions remote paths (https://en.wikipedia.org/wiki/Path_(computing)#POSIX_pathname_definition) and UNIX is already mentioned in file mode. I don't have a preference for any of them.

[creachadair]

Another concern is that normalization affects path encoding, and in particular path comparison. That's not something this PR needs to address, but it's something to keep in mind for any storage format.

[jfontan]

Both calls to Find and Glob that do searches in the file index use the same normalization for input so maybe this is enough.

• Add user\files\test.txt
• It's normalized and saved as user/files/test.txt
• Call Gob("user\\files\\*.txt"), converted to user/files/.txt so the file is found

@creachadair is this what you mean?

[creachadair]

@creachadair is this what you mean?

That's part of it. I also mean normalization at the Unicode level: e.g., é can be represented as either LATIN SMALL LETTER E plus COMBINING ACUTE ACCENT (decomposed) or as LATIN SMALL LETTER E WITH ACUTE (composed). The two are equivalent but not byte-for-byte equal in UTF-8 (\xc3\xa9 vs. \x65\xcc\x81).

Worse, some filesystems (notably Apple's HFS family) require a particular normalization and will canonicalize paths when they are opened. So if you are storing and querying path, you need to either decode both the needle and the haystack and compare rune-by-rune, or remember which encoding you used originally so you can canonicalize the needle at query time. (The former is easy to describe but hard to do efficiently)

[edit: fix typo]

[jfontan]

@smola, @mcuadros friendly ping