contentFiles, project.json and ASP.NET (5/core 1.0) projects

[skyflyer]

Goal: to be able to create a nuget package, which can:

1. Copy content (CSS/images/CSHTML files/...) to a project
2. Reference set of dependencies (zero or more)
3. Transform existing files (config files, source files)
4. Run pre-post install scripts

Seems like all of this is planned and some of it is supposed to be implemented, but I can't get it to work. I've been testing with a very
simple nuget package in an simple ASP.NET 5 project and in an UWP Universal Class Library project, given the discussion and instructions here (by @yishaigalatzer and @rhyous).

I understand there's a lot of confusion around the "new" project.json based format, but the tooling support is rather lacking. I'm not even sure which part of the nuget (client, I suppose) is responsible for taking the nuget package and restoring content into appropriate places. Given that the nuget pack created SharedContent.1.0.0.nupkg file actually contains the any/any/images/image.png file, I'm left wondering who is supposed to pick that file after it has been Install-Package-ed.

[yishaigalatzer]

1, 2, 4 are not supported and not planned to be supported.

[skyflyer]

@yishaigalatzer,

I'm surprised. The page (http://docs.nuget.org/Create/Creating-and-Publishing-a-Package#powershell) specifically mentions:

Some packages contain more than just assemblies. They may contain

1. Content and source code that should be injected into the target project.
2. PowerShell scripts and executables.
3. Configuration file and source code transformations.

Am I reading this backwards?

Given the existing issues #1523 and #627, this seems to have been discussed, but no definite conclusion yet? Some issues (#1908, #1792, #1080 refer to functionality number 1 in "my list", at least the way I understand it.

While brining CSS/images to the (web) project is trivial using bower, adding common config files, DLLs, transitive dependencies and other artefacts is not.

Could you please clarify and expand on that?

Thanks!

[yishaigalatzer]

In a way you are, and we should probably do better with the docs.

There are two worlds for NuGet the packages.config world and the project.json world. What you read is referring to the packages.config world. The unfortunate fact is that powershell scripts is a feature that only work in visual studio and sometimes only in specific versions of visual studio. It does not work from command line, and it does not work in any cross platform way. It is a source of a constant stream of bugs around updating in the command line or on a CI machine, it creates unpredictable changes to the project that may never be properly updated/reverted. Similarly the old content model suffers from similar problems. All that stuff worked because NuGet was sort of a "scripting" addon to visual studio. This approach causes merge conflict issues, the inability to move the packages folder/share projects without breaking the world.

Our decision going forward is that NuGet's resposibility it to figure out dependencies and download the packages, leaving a lock file and stepping out of the way.

So the state of things right now is:
We support describing references in a NuGet package
We support describing ** immutable ** content in NuGet package

We are discussing possible options for support for a scaffolding model - But we haven't reached any useful conclusion on what that would look like, so this is not completely dismissed, but it is not in the plan for the RTM release of .net core.

Transformations are not something that reliably work, because the model does not describe upgrades/downgrades, but just restoring packages. And a key component for transformation is the addition but also removal of the changes, which does not exist in the model.

So the model in the immediate term is supporting references, and immutable content (see ContentFiles), but nothing more, longer term we might find a way to support mutable content, and a bit less likely a transformation model although it is something we are still open to if a good model can be designed.

Last - I think we covered the rest of the items, so with your permission I would like to close this issue, because it is not actionable. We can of course continue the discussion.

[skyflyer]

@yishaigalatzer,

thank you for taking the time to elaborate. I will go ahead, and close the ticket.

Additionally,

1. Could you point me to the reference (documentation, example) on "immutable content"? As I explained, I was unable to add content to the project. I would also like to find out on how the infrastructure plans to ensure that the content (once brought into the target project) remains immutable.
2. How will scenarios like the NLog issue be handled?
3. Do you know how to handle the case when one would like to import "layout" and shared DLL into an ASP.NET MVC 6 project?

ps: The issue 1080 referenced before was from another repo. This is the correct one and I've edited the previous comment.

[yishaigalatzer]

1. Immutable content is explained on our blog (content files demistified) they are not yet supported asp.net core, the work for that is being done by the asp.net team as we speak.
2. The scenario doesn't seem related, can you find the right issue?
3. I didn't walk through the layout scenario (I assume you mean a common razor file?) @davidfowl you guys should have a sample for this we can reference once Pavel is done
4. Immutable files are not included in the project, they are left in the packages folder and the project references them
5. Common DLL, why not add it as a reference or runtime?

[skyflyer]

Ad 1) I've tried importing files as per "Content files demistified" blog post to UWP and ASP.NET 5 projects. I get that ASP is not supported (yet), but I've tried the universal class library as well. I've mentioned my steps and included the nuget package in my initial comment. No image in my (Windows Universal Class Library) project.

Ad 2) I've fixed the links to issue number 1080 again -- I must've pasted the wrong ones before. And I'm pasting the full URL here just in case: NLog/NLog#1080

Ad 3) That would be really great @davidfowl ; As far as I am reading though, I don't think this scenario will be supported. Hope I'm mistaken?

Ad 4 & 5) Yes, of course, makes perfect sense. But regarding image assets, the loaders need to look into the "assets" folders etc (for instance, for UWP apps). So, for "immutable content" to reside in the package directory, that doesn't really help, unless those sources are moved to asset folders at build time? Same goes for web projects, I suppose.

[yishaigalatzer]

So for now the items don't show up in the solution explorer but they get compiled. The c# team would address it at some point. @Pilchie

[yishaigalatzer]

The blog bug says the same thing we discussed, mutable config files are not supported at this point

[frankabbruzzese]

I understand the problem created by "mutable files" in the new Nuget framework. However, "mutable files" are needed for scaffolding. Now asp.net core projects are very modular and use typical Javascript package managers to handle client side stuffs. All good things, but configuring complex projects with tools provided by the community, through different sources may become a very costly operation that might discourage the move toward Asp.net core of many developers.
Thus we need asomething like "scaffolding packages", and a clean framework to handle them, otherwise the only way would be the usage of either VS project templates or YEOMAN templates. Nothing bad, but one can't combine several project templates coming from several contributors, so if every contributor start using templates to deliver its modules, in a user friedly way, developers will not be able to use modules from different authors in their projects.
At moment I am using npm to add content files to asp.net core projects in my modules, install commands + gulp have enough flexibility to adapt easily the content to the project (namespace names etc.) but there is no clean framework behind this job, just copying files and instantiating them! So I think that an ad hoc general purpose tool is needed.

[yishaigalatzer]

There is work being done for scaffolding by the asp.net tooling team. I'll loop them in here (can't find the github ID at the moment for sayed hashimi

[frankabbruzzese]

Great, it would be nice to know more about that, such as timing, and the basic ideas this tool is based on, so I may coordinate my schedule and architecture with the job being done by the team on the subject.

[rhyous]

I dropped any attempt to use project.json because it just is a major step backwards when it comes to including source code in the NuGet package content.

As a former FreeBSD guy, I applaud the open source efforts. However, it is a poor decision IMHO to drop support for something that will work for 90% of users who write C# in Visual Studio on Windows just because 10% of users (or is it less?) might write C# for an open source platform.

I have a handful of packages that have source code as content. I am using packages.config and install/uninstall.ps1 scripts.

If you want to see what I am doing, you can read about it here:
http://www.rhyous.com/2016/03/03/nuget-for-source-using-add-as-link-1/

And see some of my projects on GitHub here:
https://github.com/rhyous

For example, SimpleArgs is designed for single file executables. It kind of defeats the purpose of a single file executable to provide a dll to reference. So I provide a NuGet package called Rhyous.SimpleArgs.Sources.
https://github.com/rhyous/SimpleArgs

One of the primary issues I had to solve was when I was creating a Visual Studio solution with a half dozen projects that were to compile as single file executables. I wanted them all to share the same source files and only have one copy of the source. When I first started adding source in NuGet packages, there was a copy of the source files in the packages directory and a duplicate copy in each project. So instead of 1 copy of the source files, I had 7 copies. I realized I needed to add the source to each project with "Add as link" which I got done with Install.ps1. I also uninstall with uninstall.ps1. It works very well.

From what I understand and from what I have heard, NuGet using packages.config will be around a long, long time. So I am not worried about depreciation for now.

Hopefully, my efforts will provide the NuGet team a feature set they know they have to match before they drop support for powershell scripts.

[yishaigalatzer]

You can absolutely support adding source code through project.json references and this change has nothing you do with open source vs. not open source.

The big difference is project.json is that it doesn't alter your project and hence allows for revving dependencies on a ci where you never want to re-checkin your code or see diffs or for that matter even have visual studio installed.

This mode allows also to support multiple languages in the same package, that was previously not supported.

I'd love to hear what is it that you are blocked from doing and in what scenarios so we can
A. Make sure we capture the impediments and improve them
B. Make sure the documentation can guide you to success.

Cc @harikm

[rhyous]

@yishaigalatzer: "You can absolutely support adding source code through project.json references and this change has nothing you do with open source vs. not open source."

Is that universally true? Or only true for Universal Apps? (ha, ha, see what I did there?)

Anyway, remember, I tried and failed. Remember, I worked with you and your the one who told me that it currently only works for Universal Windows Apps. You did say that you could alter a .proj/.sln to use project.json, but then we have to do a bunch of complex manual steps every time a new project is created. I believe you said that every project template Visual Studio has will have to be updated. And worse, anyone trying to consume our NuGet package would have to do the same complex steps. That just isn't practical. It is a show stopper.

If this change has nothing to do with becoming cross platform, why are you doing it? You gave two reasons:

1. It doesn't alter your project file

Instead, it alters a project.json file. Edit my .proj, edit my project.json, there is still an edit. Nothing was solved, it was just moved. Even if some obsure issue was solved, most people didn't even consider this a problem. You fixed something almost nobody even considered broken.

1. Language

I think this could have been added without a massive breaking change to the current package management system.

So far the solution is worse, not better. Sometimes it is a bad idea to re-write.

1. Every project template needs to be updated for project.json to work.
2. Switching from packages.xml to project.json is a major breaking change an there is zero benefit from changing the format. Everything done could have been done with Xml.

The desired feature set is simple.

1. Open Visual Studio.
2. Open a Project or Create a new Project: Command line, dll, Asp.Net, WCF, etc..
3. Manage NuGet packages.
4. Install a NuGet package that includes source so that only one copy of the source exists.

Even if you do update the project templates in Visual Studio 15 (not to be confused with VS 2015), then you still leave everyone who hasn't upgraded yet running VS 2012, 2013, 2015 broken.

This feels like some newbie developer jumped and said, hey this will be fun to re-write to use json, without really thinking through the entire landscape.

[yishaigalatzer]

1. We are adding an auto conversion tool in the next release, so it is not a show stopper.
2. It will work for every project type
3. There is a huge difference between updating a project file or project.json and you can use a * notation to float forward.

At the end of the day is your choice, just note that packages.config is the past and we are going to invest most of our efforts in the new model going forward.

[frankabbruzzese]

@rhyous , @yishaigalatzer ,
In my opinion Nuget changes were a "must". Just a matter of time. The wave of changes triggered by .Net Core was just a good occasion to do it. <= 3 versions mixed two purposes with conflictiong needs, namely:

1. Adding new functionalities, and tracking/updating their versions.
2. Performing scaffolding

Now scaffoldin for its very nature is not reversible (since developer or other packages might modify the same file). Thus it interferes with the version/Tracking and update process. Up to now the negative effect of this problem was negligible, but with software being split in always smaller modules, and with the always more frequent updates,it for sure would have created issues, thus preventing user from updating their software as needed.

Immutable files,and dlls are compatible with version tracking/updates that's why they remain in the new versions.

However, we need also scaffolding "packages" and "packeges sources" that obey to different logics, since one usually doesnt expect scaffolded files be updated automatically passing to a new version. Thus a similar package system should just alert the user a previous version of scaffolded files is already intsalled and jump or stop the update.

Tools like Yeoman or VS project templates doesnt help too much since two different templates can't be applied one after the other. Now since, software configuration is becoming always more complex scaffolding is the only way to simplify it...so we need scaffolding packages, and we need to combine several templates!

[PraveenPeri]

@yishaigalatzer Any update on the scaffolding model work on asp net core?

[yishaigalatzer]

That work is not happening as part of nuget. I'm not sure how far it progressed so far. I'll forwarded this thread around

[PraveenPeri]

Thanks!

Just to reiterate, if we want to share some typescript code across different teams, it is not currently possible via nuget, is it?

[Hecatron]

I'm currently running into this problem while trying to use the liblog Nuget package with a core project

[rhyous]

Unfortunately, the project.json version of NuGet removes some awesome features, such as including source and running scripts. If you use NuGet without the project.json changes, then you can.
http://www.rhyous.com/2016/03/03/nuget-for-source-using-add-as-link-1/

@frankabbruzzese thinks that ". . . scaffoldin for its very nature is not reversible" which is clearly untrue and that it "interferes with the version/Tracking and update process" which also is untrue.

Like everything, if you look at the problem, it is solvable. My method of adding scaffolding is both reversable and has zero interference with version/tracking. My NuGet packages restore just fine.

It also makes sure that if multiple projects in the same solution reference the same source-containing NuGet package, there is only one copy of the source file by using "Add as link" in the project.

Give it a look. With my solution added to the original NuGet feature set, you can share typescript code across different teams.

[leftler]

This problem also happens with VS 2017 and projects referenced in the new .csproj format. Content files don't get copied.