[UX bug] dotnet tool dirty state causes paket to install twice or not at all #3493

haf · 2019-01-26T19:01:10Z

Description

The build can end up in a bad state when using paket from dotnet tool.

dotnet tool install runs twice, second time around it exists with 1 failing the build.

This happens in docker builds too.

Details:

/usr/share/dotnet/sdk/2.2.103/MSBuild.dll -consoleloggerparameters:Summary -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll -maxcpucount -property:Configuration=Release -restore -target:Build -verbosity:m /bl:/tmp/tmpWznPq1.tmp.binlog src/Logary.sln
  Installing paket.bootstrapper
  You can invoke the tool using the following command: paket.bootstrapper
  Tool 'paket.bootstrapper' (version '5.180.414') was successfully installed.
  Running paket.bootstrapper to install paket
  Checking Paket version (downloading latest stable)...
  Starting download from https://github.com/fsprojects/Paket/releases/download/5.195.7/Paket.5.195.7.nupkg
  Running dotnet tool install paket --version 5.195.7 --tool-path "/build/.paket" --configfile "/tmp/tmpvOxVqQ.tmp"
  Tool 'paket' is already installed.
/build/.paket/paket.bootstrapper.proj(32,9): error MSB3073: The command "/build/.paket/paket.bootstrapper --as-tool --output-dir=/build/.paket" exited with code 1.

Build FAILED.

/build/.paket/paket.bootstrapper.proj(32,9): error MSB3073: The command "/build/.paket/paket.bootstrapper --as-tool --output-dir=/build/.paket" exited with code 1.
    0 Warning(s)
    1 Error(s)

Another error, after erasing .paket/.store:

➜  logary ☁️ :(gke_voi-stage_europe-west3_voi-stage-cluster) 🐐:(master) ✗ ./fake.sh build
+ dotnet tool install --tool-path /Users/h/dev/qv2/logary/.paket paket.bootstrapper
error NU1212: Invalid project-package combination for paket.bootstrapper 5.164.0. DotnetToolReference project style can only contain references of the DotnetTool type
The tool package could not be restored.
Tool 'paket.bootstrapper' failed to install. This failure may have been caused by:

* You are attempting to install a preview release and did not use the --version option to specify the version.
* A package by this name was found, but it was not a .NET Core tool.
* The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
* You mistyped the name of the tool.

After erasing ~/.dotnet:

/Users/h/dev/qv2/logary> "/usr/local/share/dotnet/dotnet" build src/Logary.sln --configuration Release /bl:/var/folders/t8/zgc3yjy16m5_c7g4s0knbdx80000gn/T/tmp1mV737.tmp.binlog (In: false, Out: false, Err: false)
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/local/share/dotnet/sdk/2.2.103/MSBuild.dll -consoleloggerparameters:Summary -maxcpucount -property:Configuration=Release -restore -target:Build -verbosity:m /bl:/var/folders/t8/zgc3yjy16m5_c7g4s0knbdx80000gn/T/tmp1mV737.tmp.binlog src/Logary.sln
  Paket ready, checking version:
  The application to execute does not exist: '/Users/h/dev/qv2/logary/.paket/.store/paket/5.195.7/paket/5.195.7/tools/netcoreapp2.1/any/paket.dll'.
/Users/h/dev/qv2/logary/.paket/paket.bootstrapper.proj(37,9): error MSB3073: The command "/Users/h/dev/qv2/logary/.paket/paket --version" exited with code 154.

Build FAILED.

/Users/h/dev/qv2/logary/.paket/paket.bootstrapper.proj(37,9): error MSB3073: The command "/Users/h/dev/qv2/logary/.paket/paket --version" exited with code 154.
    0 Warning(s)
    1 Error(s)

/build> "/usr/bin/dotnet" build src/Logary.sln --configuration Release /bl:/tmp/tmpD1rPpU.tmp.binlog (In: false, Out: false, Err: false)
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/share/dotnet/sdk/2.2.103/MSBuild.dll -consoleloggerparameters:Summary -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/2.2.103/dotnet.dll -maxcpucount -property:Configuration=Release -restore -target:Build -verbosity:m /bl:/tmp/tmpD1rPpU.tmp.binlog src/Logary.sln
  Installing paket.bootstrapper
  Tool 'paket.bootstrapper' is already installed.
  Running paket.bootstrapper to install paket
  /bin/sh: 2: /tmp/tmpc8eec127c1cb45e4b6ff041ace976e71.exec.cmd: /build/.paket/paket.bootstrapper: not found
/build/.paket/paket.bootstrapper.proj(32,9): error MSB3073: The command "/build/.paket/paket.bootstrapper --as-tool --output-dir=/build/.paket" exited with code 127.

Expected behavior

Paket should not crash so much.

Actual behavior

It does crash.

Known workarounds

git clean -fxd && ./fake.sh build

The text was updated successfully, but these errors were encountered:

haf · 2019-01-27T00:03:32Z

d31b312e reproduces nicely when running make image

haf · 2019-01-27T12:59:46Z

I think the solution to this is very careful ignoring of what's in .paket/, namely, paket{,.bootstrapper{,.exe}},.store/

haf · 2019-01-28T10:53:53Z

So the only thing this issue showcases is the friction people will experience and the lack of docs on what to ignore and how to use paket with docker builds.

It's not a "bug" per-se but a rather obtuse experience.

forki · 2019-01-29T06:46:28Z

What can we do here to improve the docs?

haf · 2019-01-29T07:10:48Z

You can document what should be ignored when using Paket with Dockerfiles
You can document where .net tool installs it
You can document that there's no need for either paket nor paket bootstrapper to be present in the source code tree any more since they're automatically downloaded
You can document that paket is now a per-platform binary that must be cleaned before switching platforms
You could document the proper naming cross platform (you've chosen paket.exe vs paket)
You could fix and/or document the insistent (600x of them) warnings about LANG=C not existing (when paket runs)

forki · 2019-01-29T07:14:04Z

That clang warning - any ideas how to fix it? Am Di., 29. Jan. 2019, 08:10 hat Henrik Feldt <[email protected]> geschrieben:

…

1. You can document what should be ignored when using Paket with Dockerfiles 2. You can document where .net tool installs it 3. You can document that there's no need for either paket nor paket bootstrapper to be present in the source code tree any more since they're automatically downloaded 4. You can document that paket is now a per-platform binary that must be cleaned before switching platforms 5. You could document the proper naming cross platform (you've chosen paket.exe vs paket) 6. You could fix and/or document the insistent (600x of them) warnings about LANG=C not existing (when paket runs) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3493 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNBQAZIcEkb5nA7FWYiJvDVE6UHMXks5vH_P4gaJpZM4aUUw4> .

haf · 2019-01-29T07:45:57Z

I've tried forcing LANG with LC_ALL and LANG in the container, but the problem is that the compiler or paket sets another LANG environment variable before calling the compiler/dep resolver. The solution is probably to install en_US on the container, but in this instance my google-fu failed me and I failed to find how exactly to do that. The systematic way to do that is to see what container base image MSFT uses for the .net core image and search its documentation for both the method and packages required, and then replace C in my Dockerfile with en_US, while documenting it online in Paket's docs, perhaps with a sample Dockerfile to go with it.

forki · 2019-01-29T08:41:54Z

I don't think it's only a container issue. I see it in our linux builds as well (outside of containers)

matthid · 2019-01-29T08:50:06Z

Yes I see this on some CI servers with fake as well. I have started several attempts at this to change the environment but couldn't fix this yet (I think gitlab shows this iirc)

haf · 2019-01-29T10:28:17Z

If you'd find a way to install en_US properly, then perl would probably find it and not complain about it, no matter the linux.

inosik · 2019-02-05T20:29:53Z

Re: LANG warning:

It's reproducible with MSBuild and this project:

<Project>
  <Target Name="Build">
    <Exec StandardOutputImportance="Low" Command="shasum $(MSBuildThisFile)" ConsoleToMSBuild='true'>
      <Output TaskParameter="ConsoleOutput" PropertyName="Hash" />
    </Exec>
  </Target>
</Project>

I tested this on GitLab CI with this .gitlab-ci.yml:

image: microsoft/dotnet:2.1-sdk

test:
  script: dotnet msbuild -t:Build test.proj

I reduced down what Paket.Restore.targets does to determine if we should perform a restore or not. It has something to do with MSBuilds Exec-Task and how it invokes programs. And apparently shasum is written in Perl and thus we see a Perl warning.

inosik · 2019-02-05T20:40:07Z

MSBuild sets LANG and LC_ALL to en_US.UTF-8 before invoking programs on Unix: https://github.com/Microsoft/msbuild/blob/f136edbfecec171d73cb4c7fe1c936f6b2bf60a5/src/Tasks/Exec.cs#L597

This gets written to a temporary shell script, which MSBuild actually invokes.

matthid · 2019-02-05T21:55:58Z

wow very nice hunting. Any idea how to fix/workaround?

inosik · 2019-02-07T06:58:53Z

Short term: no idea. Some thoughts:

While digging in the SDKs targets files, I found a Hash task. The GenerateAssemblyInfo target (or its CreateGeneratedAssemblyInfoInputsCacheFile sub target) uses it, but I didn't find it in the docs.
There's a new GetFileHash task, but apparently it's not supposed to be used until MSBuild 16.0 comes out.
~~We could add a custom Hash task to Paket.Restore.targets using C# code, like in this example: http://blog.ylett.com/2012/05/msbuild-task-to-generate-hash-of-files.html~~

Ignore that: PaketRestoreLockFileHash = PaketRestoreCachedHash slow #2796 (comment)

inosik · 2019-02-24T11:59:56Z

We can suppress the warnings by redirecting stderr of shasum to /dev/null, like this:

shasum file.txt 2> /dev/null

matthid · 2019-07-02T19:00:16Z

That shasum thing should be solved with #3608

haf mentioned this issue Jan 26, 2019

[MAYBE] Paket for .Net Core crashes in Docker causiq/logary#385

Closed

haf changed the title ~~Paket for .Net Core on Alpine and Debian crashes~~ Paket for .Net Core installs twice Jan 26, 2019

haf changed the title ~~Paket for .Net Core installs twice~~ dotnet tool dirty state causes paket to install twice Jan 26, 2019

haf changed the title ~~dotnet tool dirty state causes paket to install twice~~ dotnet tool dirty state causes paket to install twice or not at all Jan 26, 2019

haf changed the title ~~dotnet tool dirty state causes paket to install twice or not at all~~ [UX bug] dotnet tool dirty state causes paket to install twice or not at all Jan 28, 2019

matthid closed this as completed Jul 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UX bug] dotnet tool dirty state causes paket to install twice or not at all #3493

[UX bug] dotnet tool dirty state causes paket to install twice or not at all #3493

haf commented Jan 26, 2019 •

edited

Loading

haf commented Jan 27, 2019

haf commented Jan 27, 2019

haf commented Jan 28, 2019

forki commented Jan 29, 2019

haf commented Jan 29, 2019

forki commented Jan 29, 2019 via email

haf commented Jan 29, 2019 •

edited

Loading

forki commented Jan 29, 2019

matthid commented Jan 29, 2019

haf commented Jan 29, 2019

inosik commented Feb 5, 2019

inosik commented Feb 5, 2019

matthid commented Feb 5, 2019

inosik commented Feb 7, 2019

inosik commented Feb 24, 2019

matthid commented Jul 2, 2019

[UX bug] dotnet tool dirty state causes paket to install twice or not at all #3493

[UX bug] dotnet tool dirty state causes paket to install twice or not at all #3493

Comments

haf commented Jan 26, 2019 • edited Loading

Description

Details:

Expected behavior

Actual behavior

Known workarounds

haf commented Jan 27, 2019

haf commented Jan 27, 2019

haf commented Jan 28, 2019

forki commented Jan 29, 2019

haf commented Jan 29, 2019

forki commented Jan 29, 2019 via email

haf commented Jan 29, 2019 • edited Loading

forki commented Jan 29, 2019

matthid commented Jan 29, 2019

haf commented Jan 29, 2019

inosik commented Feb 5, 2019

inosik commented Feb 5, 2019

matthid commented Feb 5, 2019

inosik commented Feb 7, 2019

inosik commented Feb 24, 2019

matthid commented Jul 2, 2019

haf commented Jan 26, 2019 •

edited

Loading

haf commented Jan 29, 2019 •

edited

Loading