Mapping Jira Custom Fields to Azure DevOps History

[masteragi]

Dear Community,

I hope you are doing well! 🤓

In our Jira we have some custom fields which we don't want to replicate in the Azure DevOps instance but would like to keep as historical data. For this reason it would be sufficient to store their values (e.g. in the format "field name: field value") in the History/Discussion as separate comments.

I have defined the mappings like following:

      {
        "original-name": "Geplanter Aufwand h",
	"source": "customfield_16862",
        "target": "Microsoft.VSTS.Scheduling.Effort"
      },
      {
        "original-name": "Melder",
	"source": "customfield_10160",
        "target": "System.History"
      },
      {
        "original-name": "Fehlerart",
	"source": "customfield_16324",
        "target": "System.History"
      },
      {
        "original-name": "Fachbereich",
	"source": "customfield_10460",
        "target": "System.History"
      },
      {
        "original-name": "Anforderungsart",
	"source": "customfield_16324",
        "target": "System.History"
      },

However the tool seems to ignore the mappings (or at least I am not able to find the error) and I can see following entries in the log:
[I][15:13:38] Connecting to Jira...
[I][15:13:38] Retrieving Jira fields...
[I][15:13:38] Retrieving Jira link types...
[I][15:13:38] Export started. Selecting 1 items.
[I][15:13:38] Initializing Jira field mapping...
[W][15:13:42] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:28] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:32] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:33] Ignoring target mapping with key: 'System.History', because it is already configured.

Should my approach work or I am doing something completely wrong?

Thank you in advance for your help. 🙏

[Alexander-Hjelm]

However the tool seems to ignore the mappings (or at least I am not able to find the error) and I can see following entries in the log:
[I][15:13:38] Connecting to Jira...
[I][15:13:38] Retrieving Jira fields...
[I][15:13:38] Retrieving Jira link types...
[I][15:13:38] Export started. Selecting 1 items.
[I][15:13:38] Initializing Jira field mapping...
[W][15:13:42] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:28] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:32] Ignoring target mapping with key: 'System.History', because it is already configured.
[W][15:16:33] Ignoring target mapping with key: 'System.History', because it is already configured.

Migrating multiple source fields to the same target fields is not supported. I will mark this as a feature request.

:)

[masteragi]

OK, thank you. I need to think of a workaround here... Is a change required only on the export side or also on the import side?

[Alexander-Hjelm]

Pretty sure it would only be on the export side!

Sure! A temporary but rather annoying workaround would be to set the target field as a temporary token and then search-replace all such tokens with "System.History".

For example I could see you doing something like this:

      {
        "original-name": "Melder",
	"source": "customfield_10160",
        "target": "UniqueReplace1"
      },
      {
        "original-name": "Fehlerart",
	"source": "customfield_16324",
        "target": "UniqueReplace2"
      },
      {
        "original-name": "Fachbereich",
	"source": "customfield_10460",
        "target": "UniqueReplace3"
      },
      {
        "original-name": "Anforderungsart",
	"source": "customfield_16324",
        "target": "UniqueReplace4"
      },

Then after the Export, go inside an IDE like VS Code and do a folderwide search-replace in your folder (with regex enabled):

• UniqueReplace\d+ -> System.History

Finally go ahead with the Import.

This should take care of it in theory, but I have never tried before, so I cannot say with 100% certainty. You are welcome to try though!

[masteragi]

First thing I tried is to add another History entry in the exported Jira json file, like this:

        {
          "ReferenceName": "System.History",
          "Value": "Value 1"
        },
	{
          "ReferenceName": "System.History",
          "Value": "Value 2"
        },

And after the import just "Value 2" gets written in the Azure DevOps work item.

It doesn't seem to be a big code-change on the export side so I might take a swing at it but I haven't checked the import side yet...

[Alexander-Hjelm]

Correct, each update would need to be in it's own separate revisions! It can only write to any single Work Item Field once in each revision.

[masteragi]

Correct, each update would need to be in it's own separate revisions! It can only write to any single Work Item Field once in each revision.

I see... In this case it is a bigger code-change, but I can only say for sure after I understand the code bahind revisions. 😄

[masteragi]

@Alexander-Hjelm
I am trying to make this work and have made following changes in the code to make this work:

Added the option to mapping configuration in the config.json file (additional "customFieldsList" with the list of my custom fields, mapped to "System.History" with the new "MapMultiple" mapper):

{
"source": "multiple",
"customFieldsList": [
	"Melder",
	"Fehlerart",
	"Fachbereich",
	"Anforderungsart"
],
"target": "System.History",
"mapper": "MapMultiple"
}

Then I added the "customFieldsList" as a property in the Field class:

[JsonProperty("customFieldsList")]
public List<string> CustomFieldsList { get; set; }

Added the handling of the "MapMultiple" mapper in JiraMapper::InitializeFieldMappings:

case "MapMultiple":
    value = r => FieldMapperUtils.MapMultipleValues(r, item.CustomFieldsList, item.Target, _config, _jiraProvider);
    break;

Implemented the concatenation of custom field "name: value" pairs in the MapMultipleValues method:

public static (bool, object) MapMultipleValues(JiraRevision r, List<string> customFields, string targetItem, ConfigJson config, IJiraProvider jiraProvider)
{
    if (r == null)
        throw new ArgumentNullException(nameof(r));

    if (config == null)
        throw new ArgumentNullException(nameof(config));

    var mappedValues = string.Empty;
    var customFieldId = string.Empty;
    var customFieldValue = string.Empty;
    //go through the list of source fields
    foreach (var customFieldName in customFields)
    {
        //fetch the Jira Custom Field ID via the Jira Rest API
        customFieldId = jiraProvider.GetCustomId(customFieldName);
        //fetch the field value from the list of fields in the Jira Revision
        customFieldValue = r.GetFieldValue(customFieldId);

        //add an entry only if the custom Field has a value in the Jira Revision
        if (customFieldValue != null)
        {
            //add a breakpoint before every new key-value pair (except for the first one)
            mappedValues += (mappedValues != string.Empty) ? "</br>" : "";
            //put name and value together as "name: value"
            mappedValues += customFieldName + ": " + customFieldValue;
        }
    }

    return (true, mappedValues);
}

I've probably butchered the code a lot and left some loose ends unhandled but this is as far as I got so far to get multiple custom fields mapped to the System.History field.

What I am experiencing at the moment is that all the original Jira comments get overwritten by those same custom field values:

Before I go and lose a lot of time debugging the code, do you know where I got it wrong and where I might need to adjust the code (or maybe add some additional forks/checks)?

Thank you in advance for your support. 🙏

Cheers,
Marko

[Alexander-Hjelm]

My best guess is that it would be the JiraMapper blocking the old field mapping for System.History because it contains the same target property as your new field mapping. This will cause one to override the other. Currently only one field mapping is allowed per field per Work Item Type.

jira-azuredevops-migrator/src/WorkItemMigrator/JiraExport/JiraMapper.cs

Lines 151 to 177 in 589c4ab

	foreach (var wit in currentWorkItemTypes)
	{
	try
	{
	if (wit == "All" || wit == "Common")
	{
	commonFields.Add(item.Target, value);
	}
	else
	{
	// If we haven't mapped the Type then we probably want to ignore the field
	if (typeFields.TryGetValue(wit, out FieldMapping<JiraRevision> fm))
	{
	fm.Add(item.Target, value);
	}
	else
	{
	Logger.Log(LogLevel.Warning, $"No target type '{wit}' is set, field {item.Source} cannot be mapped.");
	}
	}
	}
	catch (Exception)
	{
	Logger.Log(LogLevel.Warning, $"Ignoring target mapping with key: '{item.Target}', because it is already configured.");
	}
	}

[masteragi]

Thank you, @Alexander-Hjelm. Yes, that's what I first thought as well - that it is caused by the limitation of the Dictionary to hold one mapping for each key (target work item type). What would you recommend as the easiest workaround?

[Alexander-Hjelm]

I can think of a simple workaround which would be best implemented as a post-migration task. I would have mapped Comment to let's say "target": "System.History" and your new aggregated field to a unique token like "target": "System.History1". You can then do a folder-wide search replace in the workspace folder using e.g. an IDE or a script. You would replace all ocurences of System.History1" with "System.History". I think that this should adequately solve your problem, unless there are any instances where the the Jira Comment and your aggregated field are updated on the same revision, which I assume is not the case.

We would probably need to do a major redesign of the JiraMapper component if we want to properly support this scenario, plus think about all possible conflict scenarios. I am thus leaving this issue as a Feature Request for when we can commit to it! :)

[masteragi]

That's a good idea and easy to implement - as one additional line of code before the WiItem gets serialized to the JSON file. Will do that... Thank you so much!