Implement llvm-foreach tool

[Fznamznon]

The tool has simple functionality: execute specified command for each
file in input file list.
The tool will be used in the clang driver for cases when number of
action's outputs is unknown and used tools don't support file lists.

Signed-off-by: Mariya Podchishchaeva [email protected]

[asavonic]

Please state the motivation for this patch in the commit message (and in the PR description).

The tool looks good overall, and seems to do its job. However, the its command line interface is totally different from *nix xargs. I think that the tool should either be renamed (llvm-foreach?), or (probably better) its interface should be aligned with xargs.

[asavonic]

Just by looking at the help message, I cannot figure out what this option does.

[Fznamznon]

Tried to improve.

[keryell]

It is not clear why you need to reimplement xargs if it is just like xargs...
Just install xargs on the system? Recompile it on Windows? Use WSL? Use a .bat script doing the same?

[Fznamznon]

It is not clear why you need to reimplement xargs if it is just like xargs...
Just install xargs on the system? Recompile it on Windows? Use WSL? Use a .bat script doing the same?

The idea was taken from unix xargs but I'm not sure it's exactly xargs. It's xargs with additional functionality - generation of output file list. I think, I'm going to rename it to something like "llvm-foreach" as @asavonic suggests...

Actually it's needed for SYCL device code split feature #631 .
Basically split action will have multiple outputs and number of these outputs is unknown. All these outputs should be passed through some part of SYCL toolchain by the clang driver. The problem is in clang driver. When number of outputs is unknown before compilation invocation, the clang driver can operate only with file lists. But a lot of compilation tools don't have support for file lists, so we are implementing a new tool which will execute other tools for each file from specified list.
So, this tool (llvm-foreach?) will be used in the clang driver to execute tools which don't have support for multiple inputs/filelists. So, to make this tool working with the clang driver on any system, we just implement it.
I'll add this information to commit message soon.

[Fznamznon]

Applied comments.
Also I added support of multiple file lists and arguments containing "=".

[asavonic]

Search for the same substring in the same string happens 3 times: here in contains, in Arg.find and then in Arg.find_last_of.

[Fznamznon]

Now it's 1 time.

[asavonic]

It is enough to just store an index where Replace string starts, and it's length.
No need to split a string into two allocations.

[Fznamznon]

Okay, done.

[asavonic]

NumOfLines seems to be the same as FileLists[i].size().

[Fznamznon]

Yep. Removed NumOfLines.

[asavonic]

There is no need to keep concatenating file names into a string, you can just write them to OutputFileList file.

[Fznamznon]

Okay, done.

[asavonic]

Just to be clear: I strongly believe that the clang driver is supposed to handle this use case, and making a tool to cover this particular deficiency is a workaround.
Having said that, I've no major objections to the code itself.

[Fznamznon]

Just to be clear: I strongly believe that the clang driver is supposed to handle this use case, and making a tool to cover this particular deficiency is a workaround.
Having said that, I've no major objections to the code itself.

Yes, It's some kind of workaround now. But I'm afraid that to support such use cases clang driver needs some significant changes, probably redesign. As you can see, not only we faced such problems with the driver - triSYCL/sycl#73 .
So, I think we can maybe try to start a talk with llvm community about it.

[Fznamznon]

Ping.