Support upload from named pipes #748
Conversation
|
|
||
| // accesses the ReadableStream that was passed into sendStream | ||
| // eslint-disable-next-line @typescript-eslint/unbound-method | ||
| const stream = mocked(HttpClient.prototype.sendStream).mock.calls[0][2] |
There was a problem hiding this comment.
Pretty cool trick, didn't know about this 👍
| @@ -174,6 +179,43 @@ describe('Upload Tests', () => { | |||
| expect(uploadResult.uploadSize).toEqual(expectedTotalSize) | |||
| }) | |||
|
|
|||
| const onLinuxIt = platform() === 'linux' ? it : it.skip | |||
There was a problem hiding this comment.
Is there are a reason why this test is only on Linux and are there any platform limitations that can arise? We try to remain platform agnostic as much as possible.
I think it should also be able to at least run on Mac just fine (on Windows named pipes are a considerably different with the \\.\pipe\ file location so I think it's fine to avoid a test for that) but I think you could do something like:
const notWindowsIt = process.platform !== 'win32'
There was a problem hiding this comment.
Thanks, good point. I think I can do one better, there is also other Unixes where this should work, so testing if mkfifo exists is better. Now there is a test with which mkfifo to see if mkfifo can be found.
zregvart
force-pushed
the
pr/named-pipe-upload
branch
8 times, most recently
from
bf1f1c7
to
c919018
Compare
|
@zregvart sorry for the delays. Could you sync up with the main branch, CI should go green now (we made some fixes). Afterwards this PR should be able to go in 😃 |
Named pipes report file size as 0, which leads to reading the whole content into memory (0 is less than 64K). This adds additional check to make sure that the passed in path is not a named pipe, and in that case opts for the create-temp-file-to-gzip code path. When running on GitHub Actions infrastructure on `windows` node, named pipes can be created using `mkfifo` from MSYS2. In that case `fs.Stats`s `isFIFO()` returns `false`, and not `true` as expected. This case is detected by `process.platform` being `win32` and the passed file having length of 0. As a side note, when MSYS2's `mkfifo` is run, a pipe file is created: ``` prw-rw-rw- 1 User None 0 Mar 31 12:58 pipe ``` If `fs.stat` is invoked at this point `ENOENT` error will be thrown. As soon as the pipe is written to, this pipe file is replaced by two same- named files: ``` -rw-r--r-- 1 User None 0 Mar 31 13:00 pipe -rw-r--r-- 1 User None 0 Mar 31 13:00 pipe ``` And at this point `fs.stat` `isFIFO()` returns `false`. Even though the file acts as a named pipe.
zregvart
force-pushed
the
pr/named-pipe-upload
branch
from
c919018
to
4c49a3d
Compare
|
@konradpabjan I've rebased, the workflow run requires approval to run. |
| // on Windows with mkfifo from MSYS2 stats.isFIFO returns false, so we check if running on Windows node and | ||
| // if the file has size of 0 to compensate | ||
| const isFIFO = | ||
| fileStat.isFIFO() || (process.platform === 'win32' && totalFileSize === 0) |
There was a problem hiding this comment.
Changes look good to me! Would like to get one more pair of eyes on this before merging
@yacaovsnc could you quickly glance over this PR? The only slight concern I have is the (process.platform === 'win32' && totalFileSize === 0) block. I'm not sure if there are any other edge cases where on Windows the file size would be reported as 0 and we don't want to go with an in-memory buffer.
There was a problem hiding this comment.
I think this is 👍🏼 . The logic here will make sure any file with size 0 to use the file on disk approach which should be safer compared to the in-memory buffer.
|
@konradpabjan I'm afraid the Windows support needs to be dropped from this see actions/upload-artifact#281, I can prepare a pull request for that. |
|
Thanks @zregvart If you could work on a fix for Windows that would be great! I'm currently on vacation and away from my computer but on Monday I can review the changes and release a new version of the package and then sync the changes with upload-artifact. cc @yacaovsnc for visibility |
Named pipes report file size as 0, which leads to reading the whole content into memory (0 is less than 64K). This adds additional check to make sure that the passed in path is not a named pipe, and in that case opts for the create-temp-file-to-gzip code path. When running on GitHub Actions infrastructure on `windows` node, named pipes can be created using `mkfifo` from MSYS2. In that case `fs.Stats`s `isFIFO()` returns `false`, and not `true` as expected. This case is detected by `process.platform` being `win32` and the passed file having length of 0. As a side note, when MSYS2's `mkfifo` is run, a pipe file is created: ``` prw-rw-rw- 1 User None 0 Mar 31 12:58 pipe ``` If `fs.stat` is invoked at this point `ENOENT` error will be thrown. As soon as the pipe is written to, this pipe file is replaced by two same- named files: ``` -rw-r--r-- 1 User None 0 Mar 31 13:00 pipe -rw-r--r-- 1 User None 0 Mar 31 13:00 pipe ``` And at this point `fs.stat` `isFIFO()` returns `false`. Even though the file acts as a named pipe.
Named pipes report file size as 0, which leads to reading the whole
content into memory (0 is less than 64K). This adds additional check to
make sure that the passed in path is not a named pipe, and in that case
opts for the create-temp-file-to-gzip code path.
When running on GitHub Actions infrastructure on
windowsnode, namedpipes can be created using
mkfifofrom MSYS2. In that casefs.StatssisFIFO()returnsfalse, and nottrueas expected. This case isdetected by
process.platformbeingwin32and the passed file havinglength of 0.
As a side note, when MSYS2's
mkfifois run, a pipe file is created:If
fs.statis invoked at this pointENOENTerror will be thrown. Assoon as the pipe is written to, this pipe file is replaced by two same-
named files:
And at this point
fs.statisFIFO()returnsfalse. Even though thefile acts as a named pipe.