[JENKINS-73687] Make deserialization of `Map` fields in XML files more robust #9653

dwnusbaum · 2024-08-23T23:08:36Z

I was recently made aware of a change in hashicorp-vault plugin that accidentally broke serial compatibility of a folder property in a way that unexpectedly caused the folder to become very corrupted after deserialization. See jenkinsci/hashicorp-vault-plugin#326 (comment) for details, but the summary is that config.xml for the folder had an element for a map which looked like this associated with a field of type Map<String, Calendar>, which is not correct.

<tokenExpiry>
  <time>1724357123299</time>
  <timezone>America/New_York</timezone>
</tokenExpiry>

The correct format would have been something like this:

<tokenExpiry>
  <entry>
    <string>key</string>
    <gregorian-calendar>
      <time>1724357123299</time>
      <timezone>America/New_York</timezone>
    </gregorian-calendar>
  </entry>
</tokenExpiry>

This led to a bare RuntimeException being thrown here as a result of this call, which expects that it has just read the <entry> element and needs to move down to read the key, but in this case has in fact just read the element named <time>, and so there are no children to read:

jenkins/core/src/main/java/hudson/util/RobustMapConverter.java

Line 67 in fb09eaf

reader.moveDown();

Since this is a RuntimeException, it actually escapes quite far up through the converter hierarchy until it is caught here and then the wrapped exception is caught in RobustReflectionConverter.doUnmarshal, but at that point things go pretty badly wrong and the content that XStream is reading is associated with the wrong field in the class, leading to further problems and eventually a corrupted object. In the original issue, I saw this warning in the logs: Cannot convert type hudson.util.DescribableList to type hudson.model.ViewGroup, indicating pretty bad type confusion.

Here is the full stack trace of the OldDataMontior warning:

java.lang.RuntimeException
     at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:105)
     at com.thoughtworks.xstream.io.ReaderWrapper.moveDown(ReaderWrapper.java:36)
     at com.thoughtworks.xstream.io.path.PathTrackingReader.moveDown(PathTrackingReader.java:37)
     at hudson.util.RobustMapConverter.read(RobustMapConverter.java:67)
     at hudson.util.RobustMapConverter.putCurrentEntryIntoMap(RobustMapConverter.java:49)
     at com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:99)
     at com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:93)
     at com.thoughtworks.xstream.converters.collections.MapConverter.unmarshal(MapConverter.java:88)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
 Caused: com.thoughtworks.xstream.converters.ConversionException: 
 ---- Debugging information ----
 cause-exception     : java.lang.RuntimeException
 cause-message       : null
 class               : java.util.HashMap
 required-type       : java.util.HashMap
 converter-type      : hudson.util.RobustMapConverter
 path                : /com.cloudbees.hudson.plugins.folder.Folder/properties/com.cloudbees.hudson.plugins.folder.FolderTest$Property/fieldB/time
 line number         : 6
 -------------------------------
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:81)
     at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
     at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
     at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
     at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
     at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
     at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
     at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:117)
     at hudson.util.CopyOnWriteList$ConverterImpl.unmarshal(CopyOnWriteList.java:203)
     at hudson.util.DescribableList$ConverterImpl.unmarshal(DescribableList.java:285)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
     at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
     at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
     at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
     at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
     at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
     at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
     at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
     at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464)
     at hudson.util.XStream2.unmarshal(XStream2.java:230)
     at hudson.util.XStream2.unmarshal(XStream2.java:201)
     at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441)
     at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1330)
     at hudson.XmlFile.read(XmlFile.java:165)
     at hudson.model.Items.load(Items.java:375)
     at hudson.model.ItemGroupMixIn$2.call(ItemGroupMixIn.java:284)
     at hudson.model.ItemGroupMixIn$2.call(ItemGroupMixIn.java:282)
     at hudson.model.Items.whileUpdatingByXml(Items.java:132)
     at hudson.model.ItemGroupMixIn.createProjectFromXML(ItemGroupMixIn.java:282)
     at jenkins.model.Jenkins.createProjectFromXML(Jenkins.java:4291)
     at com.cloudbees.hudson.plugins.folder.FolderTest.badProperty(FolderTest.java:610)
     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
     at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
     at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:655)
     at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
     at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
     at java.base/java.lang.Thread.run(Thread.java:833)

Here are some things that I still want to check:

Are there other cases where XStream throws RuntimeException where we'd like for one of our Robust*Converter classes to instead just ignore the offending element?
- See [JENKINS-73687] Make deserialization of Map fields in XML files more robust #9653 (comment). I did not see any other obvious things that we can improve here.
Are there cases where the proposed fix leads to worse problems than without the fix?
- Not that I can think of.
Can we improve the error message in this case to list the path to the problematic element?
- See 41433d8

Testing done

Proposed changelog entries

Make deserialization of Map fields in XML files more robust.

Proposed upgrade guidelines

N/A

Submitter checklist

Give feedback

The Jira issue, if it exists, is well-described.
The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples). Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
There is automated testing or an explanation as to why this change has no tests.
New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
For dependency updates, there are links to external changelogs and, if possible, full differentials.
For new APIs and extension points, there is a link to at least one consumer.
Options

Desired reviewers

Before the changes are marked as ready-for-merge:

Maintainer checklist

Give feedback

There are at least two (2) approvals for the pull request and no outstanding requests for change.
Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
Proper changelog labels are set so that the changelog can be generated automatically.
If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).
Options

jglick · 2024-08-24T13:25:57Z

See x-stream/xstream#106 and https://issues.jenkins.io/browse/JENKINS-49169 and #3248 for some context.

dwnusbaum · 2024-08-26T16:01:48Z

Yes, I think x-stream/xstream#106 would have prevented the main issue here.

Are there other cases where XStream throws RuntimeException where we'd like for one of our Robust*Converter classes to instead just ignore the offending element?

With this patch, Jenkins core no longer has any calls to HierarchicalStreamReader.moveDown that do not check HierarchicalStreamReader.hasMoreChildren first, apart from this call in XStreamDOM, which appears to be unused outside of https://plugins.jenkins.io/recipe/, which is obsolete, and a test in https://plugins.jenkins.io/nowsecure-auto-security-test/, so we shouldn't see a RuntimeException from that method any more:

jenkins/core/src/main/java/jenkins/util/xstream/XStreamDOM.java

Line 536 in a6d093d

r.moveDown();

There are some converters in plugins though which call moveDown without checking for children. So far, most of the calls in significant plugins seem to be ok, or the calls are in implementations of classes like SecurityRealm or AuthorizationStrategy where we expect problems to be fatal. Perhaps though we could use a ReaderWrapper here in core to check moveDown calls and throw XStreamException directly in case there are no children in the hopes that an enclosing class is likely to catch the XStreamException (vs the RuntimeException) and handle it in a recoverable way.

dwnusbaum · 2024-08-26T18:06:04Z

There are some converters in plugins though which call moveDown without checking for children. So far, most of the calls in significant plugins seem to be ok, or the calls are in implementations of classes like SecurityRealm or AuthorizationStrategy where we expect problems to be fatal.

After some filtering, I think there are really only 4 semi-relevant plugins, but none of them seem particularly problematic:

datadog preemptively created versioned Converter implementations for all of its types, and in some cases the plugin calls this utility method that avoids the typical while (reader.hasMoreChildren()) pattern to read fields, so I think that could potentially lead to issues. My guess though is that the custom Converter implementations probably mean that accidental serial compatibility mistakes are unlikely.
global-build-stats has two bare calls to moveDown (here and here) with no obvious guards, but they are in custom Converter implementations in code that is trying to handle migrations from older version formats, so they seem unlikely to cause problems.
bitbucket-oauth: Multiple calls to moveDown here, but it is a SecurityRealm, so perhaps any RuntimeException will be handled ok by the time it is caught by RobustReflectionConverter as a ConversionException because the securityRealm field will be considered critical. I have not tested this though, and the type confusion in TreeUnmarshaller may be relevant. The plugin is not under active development and the serial format seems unlikely to change.
test-stability: These calls could be a problem. This plugin went unmodified for 7 years though until it recently got updated for compatibility with a recent change to junit, so I doubt anyone will change the serial format going forward.

Usage in other plugins was either guarded by hasMoreChildren or peekNextChild, or the plugin was obsolete/archived.

dwnusbaum · 2024-08-26T20:00:37Z

Here is an example of the error message with 41433d8 showing that the path to the relevant element is included in the message:

Aug 26, 2024 3:53:16 PM hudson.diagnosis.OldDataMonitor report
WARNING: could not read hudson.util.RobustMapConverterTest$Data@61ce23ac (and Jenkins did not start up)
com.thoughtworks.xstream.converters.ConversionException: Invalid map entry
---- Debugging information ----
message             : Invalid map entry
path                : /hudson.util.RobustMapConverterTest$Data/map/string
line number         : 3
-------------------------------
    at hudson.util.RobustMapConverter.read(RobustMapConverter.java:69)
    at hudson.util.RobustMapConverter.putCurrentEntryIntoMap(RobustMapConverter.java:51)
    at com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:99)
    at com.thoughtworks.xstream.converters.collections.MapConverter.populateMap(MapConverter.java:93)
    at com.thoughtworks.xstream.converters.collections.MapConverter.unmarshal(MapConverter.java:88)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
    at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
    at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
    at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
    at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464)
    at hudson.util.XStream2.unmarshal(XStream2.java:230)
    at hudson.util.XStream2.unmarshal(XStream2.java:201)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441)
    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1321)
    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1312)
    at hudson.util.RobustMapConverterTest.robustAgainstInvalidEntry(RobustMapConverterTest.java:166)
    ...

jglick

Not sure I followed all the discussion.

dwnusbaum · 2024-09-12T16:03:50Z

Not sure what the preferred process is here. The PR has the necessary approvals. Should I just add the ready-for-merge label?

MarkEWaite · 2024-09-12T17:58:54Z

Not sure what the preferred process is here. The PR has the necessary approvals. Should I just add the ready-for-merge label?

Yes, you can add the ready-for-merge label and start the 24 merge clock yourself.

timja · 2024-09-12T18:10:20Z

see https://github.com/jenkinsci/jenkins/blob/master/docs/MAINTAINERS.adoc#acceptance

Improve resilience against malformed XML files during deserialization

a2a80d3

dwnusbaum added the bug For changelog: Minor bug. Will be listed after features label Aug 23, 2024

dwnusbaum mentioned this pull request Aug 23, 2024

Upgrading to the lastest plugin (364.vf5d54b_3dc313) breaks any Org or pipeline that uses with Vault. jenkinsci/hashicorp-vault-plugin#326

Open

dwnusbaum changed the title ~~Improve resilience against malformed XML files during deserialization~~ Make deserialization of Map fields in XML files more robust Aug 26, 2024

Improve error message

41433d8

dwnusbaum changed the title ~~Make deserialization of Map fields in XML files more robust~~ [JENKINS-73687] Make deserialization of Map fields in XML files more robust Aug 26, 2024

dwnusbaum marked this pull request as ready for review August 26, 2024 20:10

jglick approved these changes Aug 26, 2024

View reviewed changes

jeromepochat approved these changes Aug 27, 2024

View reviewed changes

NotMyFault approved these changes Aug 28, 2024

View reviewed changes

dwnusbaum requested a review from a team September 12, 2024 16:02

dwnusbaum added the ready-for-merge The PR is ready to go, and it will be merged soon if there is no negative feedback label Sep 12, 2024

jglick mentioned this pull request Sep 12, 2024

[JENKINS-63343] Validate element types for collections and maps when deserializing XML files #9727

Merged

MarkEWaite merged commit dab6597 into jenkinsci:master Sep 13, 2024
16 checks passed

dwnusbaum deleted the robustmapconverter-improvements branch September 16, 2024 13:58

This was referenced Dec 4, 2024

Recover views after error in Jenkins.load #10023

Merged

Defend against null AbstractFolder.primaryView jenkinsci/cloudbees-folder-plugin#447

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JENKINS-73687] Make deserialization of `Map` fields in XML files more robust #9653

[JENKINS-73687] Make deserialization of `Map` fields in XML files more robust #9653

dwnusbaum commented Aug 23, 2024 •

edited by MarkEWaite

Loading

Submitter checklist

Maintainer checklist

jglick commented Aug 24, 2024

dwnusbaum commented Aug 26, 2024

dwnusbaum commented Aug 26, 2024 •

edited

Loading

dwnusbaum commented Aug 26, 2024

jglick left a comment

dwnusbaum commented Sep 12, 2024

MarkEWaite commented Sep 12, 2024

timja commented Sep 12, 2024

[JENKINS-73687] Make deserialization of Map fields in XML files more robust #9653

[JENKINS-73687] Make deserialization of Map fields in XML files more robust #9653

Conversation

dwnusbaum commented Aug 23, 2024 • edited by MarkEWaite Loading

Testing done

Proposed changelog entries

Proposed upgrade guidelines

Submitter checklist

Desired reviewers

Maintainer checklist

jglick commented Aug 24, 2024

dwnusbaum commented Aug 26, 2024

dwnusbaum commented Aug 26, 2024 • edited Loading

dwnusbaum commented Aug 26, 2024

jglick left a comment

Choose a reason for hiding this comment

dwnusbaum commented Sep 12, 2024

MarkEWaite commented Sep 12, 2024

timja commented Sep 12, 2024

[JENKINS-73687] Make deserialization of `Map` fields in XML files more robust #9653

[JENKINS-73687] Make deserialization of `Map` fields in XML files more robust #9653

dwnusbaum commented Aug 23, 2024 •

edited by MarkEWaite

Loading

dwnusbaum commented Aug 26, 2024 •

edited

Loading