Use a GUID when creating the temp path

[jwood803]

Addresses #568.

@justinormont Not sure if the placement of it is the best, but it should still prevent the collisions.

Edit: Fixes #568 (added so that issue #568 will close automatically when this PR is successfully merged.)

[harishsk]

Can you please help me understand the problem you are trying to fix?

[justinormont]

Currently, there's a race condition in creating temporary files, causing a naming collision.

The paths are created as C:\Users\jormont\AppData\Local\Temp\2\TLC_25553D77\0, where:

1. TLC_ is fixed
2. 25553D77 is a seeded random in hex (bad)
3. 0 is the count [0..N] of folders existing in the parent folder (also bad).

The prefix TLC_ (1) is fine, though should be updated to the current name, perhaps to ML_NET_. Or Microsoft_ML_, though I'd worry about creating too long of paths under Windows ~260 chars, especially as a we move towards a GUID. Note users can have long usernames, plus the names of the files placed in this path.

First issue -- The seeded random number (2) is an issue since running parallelly (or rerunning) with the same seed creates the same "random" folder name. This part is a deterministic collision.

Second issue -- The race condition then occurs when counting how many folders (3) are in the parent folder. When multiple processes/threads check the contents of the folder, they all receive the same number. The purposed fix replaces this count [0..N] with a GUID.

The need is to have unique folders for creating temporary files, otherwise they clobber each other. This is causing a crash and has the possibility of silent failures, especially on macOS/Linux where you can delete and overwrite files with currently open handles.

---

Currently the way I run into this, is running N parallel copies of the AutoML․NET CLI, which I do for validating changes to the ML․NET AutoML API on ~30 datasets. Some of the processes crash with the error in the linked issue, unless they are started staggered to avoid the race condition.

[justinormont]

Pushing deeper into the code, the random number (2) is not seeded, but uses the system seed which is set from the current time. I expect the collision occurs due the remark in System.Random docs, "On most Windows systems, Random objects created within 15 milliseconds of one another are likely to have identical seed values." (src). Hence another race condition in (2).

Hopefully the .NET GUIDs don't have the same 15ms race condition as the RNG.

[harishsk]

In the current code, it looks like we are using our own random number generation, which seems odd.
It is okay to use GUIDs. But they seem a bit of overkill.

Path.GetRandomFileName seems to be intended for this kind of scenario.
Would that work better?

[justinormont]

@harishsk: You're right, Path.GetRandomFileName() seems more purpose built than a GUID.

Code behind this function -- https://github.com/dotnet/runtime/blob/f57d49bb711fda8ac0bd231229d91919f4fca8e8/src/libraries/System.Private.CoreLib/src/System/IO/Path.cs#L249-L265

[jwood803]

@harishsk I had no idea that method existed. Thanks for letting me know! Updated to use it.

[codecov]

Codecov Report

Merging #4645 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4645   +/-   ##
=======================================
  Coverage   75.84%   75.84%           
=======================================
  Files         947      947           
  Lines      172184   172184           
  Branches    18579    18579           
=======================================
  Hits       130595   130595           
  Misses      36416    36416           
  Partials     5173     5173

Flag	Coverage Δ
#Debug	75.84% <0%> (ø)	⬆️
#production	71.43% <0%> (ø)	⬆️
#test	90.7% <0%> (ø)	⬆️

[justinormont]

The current function here, GetPath(), creates the path. It's called below in CreateEntry():

machinelearning/src/Microsoft.ML.Core/Data/Repository.cs

Lines 335 to 339 in c29a00b

	GetPath(out pathEnt, out pathTemp, dir, name, true);
	if (PathMap.ContainsKey(pathEnt))
	throw ExceptionContext.ExceptParam(nameof(name), "Duplicate entry: '{0}'", pathEnt);
	else
	PathMap.Add(pathEnt, pathTemp);

There another minor race condition in how the new path is being checked/added to the dictionary. It's a nit, and minor enough to ignore if you want.

If we want to handle in this PR, we could fix by either adding a lock around the IF/ELSE, or by moving to a ConcurrentDictionary and using its TryAdd() to decide if to throw.

(I tagged this on a line of code to allow for a threaded discussion)

[jwood803]

Went the ConcurrentDictionary route, but let me know if that's not the correct implementation of it.

[justinormont]

Assuming I'm reading the docs right, TryAdd() returns false when it doesn't add (docs).

Hence need to invert the return value.

Suggested change

	if (PathMap.TryAdd(pathEnt, pathTemp))
	if (!PathMap.TryAdd(pathEnt, pathTemp))

[jwood803]

Whoops! I thought I added that, but I must have missed it. Thanks for catching!

[justinormont]

To maintain similar to the previous behavior, we'll have to throw if the TryAdd() returns false.

The previous Dictionary.Add() throws an ArgumentException if the value being added already exists (docs).

Suggested change

	PathMap.TryAdd(pathLower, pathTemp);
	if (!PathMap.TryAdd(pathLower, pathTemp))
	throw ExceptionContext.ExceptParam(nameof(name), "Duplicate entry: '{0}'", pathLower);

[jwood803]

Updated to throw exception.

[justinormont]

I restarted the failed tests, hopefully they succeed.

[mstfbl]

Restarting failed tests one more time. There is currently an issue with the CI "Windows_x64_NetCoreApp21 Release_Build".

[justinormont]

@jwood803: Can you pull in changes from the master branch? This should allow the CI tests to pass (various flaky unit tests were temporarily disabled).