Add ability to transform other files than Web.config

[bartlomiejmucha]

In Habitat, there are transform files for Domains.config and Layers.config that I would like to be applied during the publishing process. At the moment only transforms files for Web.config are applied.

[richardszalay]

You should actually be able to add this yourself by registering a custom target into the PreTransformWebConfigDependsOn property and adding an entry into the WebConfigsToTransform item.

Here's the target that registers Web.config:

<!--********************************************************************--> <!--Target CollectWebConfigsToTransform --> <!--********************************************************************--> <PropertyGroup> <CollectWebConfigsToTransformDependsOn> $(OnBeforeCollectWebConfigsToTransform); $(CollectWebConfigsToTransformDependsOn); PipelineCollectFilesPhase; </CollectWebConfigsToTransformDependsOn> </PropertyGroup> <Target Name="CollectWebConfigsToTransform" DependsOnTargets="$(CollectWebConfigsToTransformDependsOn)" Condition="'$(CollectWebConfigsToTransform)' != 'false'"> <!-- Gather Sources, Transforms, and Destinations for the TransformXml task --> <ItemGroup Condition="'$(ProjectConfigTransformFileName)'!=''"> <WebConfigsToTransform Include="@(FilesForPackagingFromProject)" Condition="'%(FilesForPackagingFromProject.Filename)%(FilesForPackagingFromProject.Extension)'=='$(ProjectConfigFileName)'"> <TransformFile>$([System.String]::new($(WebPublishPipelineProjectDirectory)\$([System.IO.Path]::GetDirectoryName($([System.String]::new(%(DestinationRelativePath)))))).TrimEnd('\'))\$(ProjectConfigTransformFileName)</TransformFile> <TransformOriginalFolder>$(TransformWebConfigIntermediateLocation)\original</TransformOriginalFolder> <TransformFileFolder>$(TransformWebConfigIntermediateLocation)\assist</TransformFileFolder> <TransformOutputFile>$(TransformWebConfigIntermediateLocation)\transformed\%(DestinationRelativePath)</TransformOutputFile> <TransformScope>$([System.IO.Path]::GetFullPath($(WPPAllFilesInSingleFolder)\%(DestinationRelativePath)))</TransformScope> </WebConfigsToTransform> <_WebConfigsToTransformOuputs Include="@(WebConfigsToTransform->'%(TransformOutputFile)')" /> </ItemGroup> <CallTarget Targets="$(OnAfterCollectWebConfigsToTransform)" RunEachTargetSeparately="False" /> </Target> <!--********************************************************************--> <!--Target PreTransformWebConfig --> <!--********************************************************************--> <PropertyGroup> <PreTransformWebConfigDependsOn> CollectWebConfigsToTransform; </PreTransformWebConfigDependsOn> </PropertyGroup>

[richardszalay]

(sorry, on phone - you may want to format that 😂)

[bartlomiejmucha]

Thanks, I will try that:)

[richardszalay]

Let me know how you go, but don't close the issue even if it works because it's something that would make sense to support out of the box.

[bartlomiejmucha]

I have some prototype, but it's not done yet. Maybe I will be able to finish it on Monday. I will let you know.

[bartlomiejmucha]

hey, I created PR for this #33

The current implementation requires that the *.xdt file from modules project, have XmlTransform set to True.
It's not ideal. And I need to rethink how to do it better.

[richardszalay]

Thanks for that, it should provide a really good basis for the feature.

I'm thinking it might better to explicitly include additional paths that can be transformed, that way we can combine transforms from different modules (for the same target file) like we do with Web.config, and that would mean we could reuse what we've got. We'll also need some unit tests for it.

[bartlomiejmucha]

We have a few options here so it's not easy to decide:

• we can explicitly include additional paths as you did with Web.config,
• we can use Content and set XmlTransform like I did in prototype,
• we can create separate NuGet package with msubild extension that should be installed to projects with transform files. The extension will add a new Build Action like "HelixXmlTransform" or similar and new Target that we can consume in WebRoot project, (thanks to this the file will not be a Content file and will not be published when you do publish directly from the module, and you can set this build action from Visual Studio properties window)

With transform files there are also a few scenarios to handle:

• Web.config and Web.config transform files exists in WebRoot project, no more transforms in modules - it works out of the box
• Web.config and Web.config transform files exists in WebRoot project, but we have other transform files in modules (it works with your Web.Helix.config files)
• We want to use remote Web.config file (from the publish folder) and we have transform files in our modules and WebRoot - it also works with your HPP pipeline

So for web.config all scenarios are working fine. For config files other than Web.config, we have similar scenarios:

• Config file exists in any module, with transform file, but no other transform files in other modules
• Config file exists in any module, with transform file and other modules have config transform files,
• Config file does not exist in solution, we want to transform remote Config file (from publish folder) and we have transform files in one or more modules.

[kmac23va]

What about SlowCheetah, or will that not work with this setup? I'm moving away from it for Sitecore 9 because of roles, but you mentioned two cases outside the App_Config that won't be affected by patch files.

For domains.config, there's a way to configure a domain without getting into that file (https://kamsar.net/index.php/2016/08/Configuring-domains-from-patch-files/). Layers.config, yeah, you probably need to use a transform on that one, or just include it in your "publisher" project (so it isn't regularly touched, like the web.config) and modify it directly with comments to note the changes for upgrade efficiency.

[bartlomiejmucha]

I analyzed how SlowCheetah extends msbuild and what it does.
The pros of SlowCheetah:

• It is maintained by Microsoft,
• It supports different versions of VisualStudio and MSBuild,
• It supports XML transforms and JSON transforms,
• It supports build configuration transforms like Web.Release.config, publish transforms like Web.Local.config
• It also has Visual Studio integration, where you can preview transforms

I managed to write an extension for SlowCheetah that supports our all scenarios, except merging multiple transform files into a single one.

I think that we should use SlowCheetah and extend it. What do you think @richardszalay ?

[richardszalay]

I'm going to have to get back on this when I'm back from paternity leave (~3 weeks) as this is a bigger topic than I'm able to process properly via my phone.

[richardszalay]

I really appreciate everyone's involvement, by the way. It's great to have so many people (compared to to my other stuff) passionate about a project. My concern with this one is maintaining both backwards compatibility with what's there now as well as keeping the documentation/messaging to the user as simple as possible

[vitaliitylyk]

Hi @richardszalay , did you have a chance to look into this? It would be great indeed if SlowCheetah transformations for custom files were supported.

P.S. Thanks for your work! We were testing HPP on our solution and noticed 2x speed improvement in comparison to Habitat-like Gulp scripts ;)

[richardszalay]

I've looked into SlowCheetah and I just don't think it's going to work. Moving aside the issue of introducing a new dependency, SC itself just doesn't seem to have the extensibility points I'd need to make the experience work.

For content files, HPP exploits a target that exists in the standard MSBuild SDK that returns all of the content items The use of returns is key, since it enables HPP to locate the content values without resorting to pulling apart the csproj files (there be dragons) and it's the reason HPP is fast: it only actually publishes one project. SlowCheetah contains no such targets, so I'm not able to "find all of the transforms" in each of the Helix modules.

Rest assured that I am still thinking about this issue and how best to implement it, and when I think of the right approach (and I might be close) it'll get done pretty quickly.

I'd actually like everyone's feedback on something, if you don't mind:

Web.config transforms in HPP are handled by "federating" the transform out into the modules, merging the transform into a single file. The idea is that Web.config is a common resource and each module has it's own reasons for modifying it (adding an HttpModule, for example) so it makes sense not to hack that all together at the (Helix) Project level.

I can see Views\Web.config receiving the same treatment, but would it be fair to say that everyone's intention for "transform additional files" would have both the configs and single transform files in their respective modules? My original thoughts were to apply the federated approach used on Web.config, but I can't help but think that it's not really how people would want the feature to work.

Would love all of your feedback.

[vitaliitylyk]

My use case is: in some Feature/Foundation modules we use *.Release transforms for Sitecore config include files to replace certain elements with Octopus Deploy tokens. These tokens will be later replaced during deployment process with values specific to each environment.

So indeed, in my scenario configs and their transform files are in their respective modules.

[bartlomiejmucha]

Hi Richard,
Here is my use case for transform files:

In every website, I worked on, I always use NWebSec NuGet. It adds an NWebSec.config file where I can configure security options. I would like to have that NWebSec in a foundation project, let's say: Foundation.Security. (So the original, full NWebSec.config file will be in that module).
And then in other Feature projects, I would like to add options to the original NWebSec.config using transform files.

In Helix projects I see three scenarios that have to be supported (*By File.config I mean any XML or JSON file that we want to transform.):

1. File.config and transform file for that config are located in a single project. No other transform files in other projects.
2. File.config located in one of the projects in the solution and there are one or more transform files for that config file in solution.
3. File.config not in the solution, but there are transform files that we want to apply on File.config located in publish directory,
   A. We publish directly to the publish directory and we want to apply transform files,
   B. We generate a deployment package and we want to include transform files in the package, so we can apply during deployment.

On my blog I decribed these three scenarios and how I implemented them by extending SlowCheetah (except 3B, there was no need for that yet): https://bartlomiejmucha.com/en/blog/msbuild/how-to-extend-msbuild-publish-pipeline-to-apply-transform-files/

SlowCheetah has a few adventages:

• it is maintaned by Microsfot,
• it support's different versions of Visual Studio
• it allows to transform JSON files as well

I really hope that you figure out the best solution. This is the last missing piece for me in your library.
If you need any help or would like to discuss the topic I will be happy to help.

[richardszalay]

Ok based on feedback, here's some initial thoughts on design that doesn't break compatibility with the way that Web.config transforms works currently:

Discovery of which config files to transform is beyond the scope of these intial thoughts. Might be automatic, might be metadata based, might be opt in with an item path.

For each file to be transformed, we look for a transform in the same actual folder as the file with the name (file).(configuration).config, (file).(publishprofile).config, or (file).(publishprofile).(configuration).config. Regardless of whether that is found we also look for a (file).Helix.config (plus configuration/profile combos) in any of the modules. Whatever transforms are found are merged, and local publish will skip the file copy if the transformed result is binary equivalent to the target (to prevent app pool recycles).

Integration with SlowCheetah (and thus json transforms) will be considered for a future release, but will likely either be automatically detected or an opt in property.

Thoughts?

[richardszalay]

An additional note on the "discovery" side of things. We do have access to any metadata assigns to Content items in any of the projects, but not None items. This means we can almost certainly auto discover "configs to transform" using the item metadata added by SlowCheetah, but the transforms themselves will need to be located by checking that the file exists based on the convention.

[bartlomiejmucha]

Will this work if we have only (file).Helix.config and the file to be transformed is not in the solution? (Only in publish folder)? Other than sounds great.

Re the metadata of the None items I did a trick like this:

<MSBuild Projects="@(ProjectReference)" Targets="GetTransformFilesToApplyOnPublish" Properties="CustomBeforeMicrosoftCSharpTargets=$(MSBuildThisFileDirectory)Helix.Module.targets">
        <Output TaskParameter="TargetOutputs" ItemName="TransformFilesToApplyOnPublish" />
</MSBuild>

I used CustomBeforeMicrosoftCSharpTargets and inside Helix.Module.targets I added GetTransformFilesToApplyOnPublish target that returns None items with metadata.

[richardszalay]

It will. Would you be happy with that rule being all or nothing (all transforms are published and not applied, or none are)?

The GetTransformFilesToApplyOnPublish approach could work, but it would require that SlowCheetah was installed on 100% of your Helix modules, otherwise the whole publish will fail. I'm always trying to avoid forcing anything (especially HPP) itself to be installed per module because of the additional effort required. Still, it's definitely something that we can expose behind an option for people using SC extensively.

[bartlomiejmucha]

It will. Would you be happy with that rule being all or nothing (all transforms are published and not applied, or none are)?

I think that all or nothing is ok. It can be improved in the future if necessary I think.

The GetTransformFilesToApplyOnPublish approach could work, but it would require that SlowCheetah was installed on 100% of your Helix modules, otherwise the whole publish will fail. I'm always trying to avoid forcing anything (especially HPP) itself to be installed per module because of the additional effort required. Still, it's definitely something that we can expose behind an option for people using SC extensively.

Sorry, I did not express myself very clearly. With approach I suggested it's not requrie to install SlowCHeetah in every Helix module.
What I ment is that in your main HPP nuget that you install in WebRoot project, you create a new plugin similar to CollectFilesFromHelixModules.Content.targets, where you execute MSBuild for every helix module, and you add this property Properties="CustomBeforeMicrosoftCSharpTargets=$(MSBuildThisFileDirectory)HPP.Module.targets" you add a path to file named for example: HPP.Modules.targets. This file is still part of your main nuget installed in WebRoot project, but it will be loaded by and thanks to this you can extend module's project from WebRoot with custom target ;)
And you don't need SlowCheetah for this, it can work with your approach as well.

Not sure if this is the best solution, however it works for me.

Alternatively I was thinking to use this: https://docs.microsoft.com/pl-pl/visualstudio/msbuild/customize-your-build?view=vs-2017

You could create a Directory.Build.targets file and put it in the src directory and all modules should load that automatically, but I not tested that yet and I'm not sure if it is possible to add that file with the nuget package.

[bartlomiejmucha]

One more thing that you can consider. What I did for myself is that for every transform file in solution, I check if the original file is present in $(_PackageTempDir) and in $(PublishUrl). If it's not in the package temp dir then I copy it from publish dir before applying transforms.

[richardszalay]

That's a neat trick! That will come in handy, I think

That PublishUrl-based approach could be good too, behind an option. I'm always concerned with using something that may already be transformed as a transform input, though. Alternatively, you can also set it yourself via the ReplacementFilesForPackaging feature of HPP.

[richardszalay]

Hey guys, I've just pushed a branch that contains the initial implementation. It's got a few rough edges, so I'd like to either get a "beta" out or (better yet) setup a MyGet for nightly builds. Until then, feel free to build it yourself to test.

The implementation attempts to find transforms for all files with a .config extension. Files in the Website directory are preferred over module files (mainly to prevent issues with Web.config and Views\Web.config), but otherwise the behaviour for duplicates is "undefined".

Transform files can be found at:

1. Next to the source file as "Filename[.PublishProfile].Configuration.ext". (eg. Web.Debug.config, Web.Dev.Debug.config)
2. In any other module (including the HPP\Website project) as "Filename.Helix[[.PublishProfile].Configuration].ext" (eg. "Web.Helix.config", "Web.Helix.Debug.config", "Web.Helix.Dev.Debug.config")

The major issues I'd like to investigate before merging are:

• I'd rather people "opt in" to transforming particular files by path/glob (or via the SlowCheetah detection later on)
• I'm not sure what kind of performance impact the current approach has.

And finally I'd like to save SlowCheetah detection/integration for the next release.

[richardszalay]

(There are also a few broken tests that need resolving before I can merge)

[rroman81]

There is a bit discussion here that i am yet to investigate, but was just wondering how I would actually add layers.helix.config type transformation via custom PreTransformWebConfigDependsOn target. it's not immediately apparent.

[nightcrawler19]

(There are also a few broken tests that need resolving before I can merge)

@richardszalay - If the issue has been resolved,, can you please merge this feature branch "additional-transforms" to master ?

[squadwuschel]

Hi, is this implemented, that I can transform xdt to specific configs and where is the documentation for this?