SharedTree: Encountering 0x379 Error During Map Modification, Leading to Document Corruption

[charokn]

Describe the bug

An error is encountered in SharedTree when modifying a Map under the following scenario:

• Client 1, acting as the elected summarizer, makes a change to a Map and generates a summary.
• Meanwhile, Client 2, which is connected to the same session, goes offline and makes the same change locally.
• After Client 1 submits the summary, Client 2 comes back online and resynchronizes with the session.

This sequence of events results in a 0x379 error, which leads to document corruption.

To Reproduce

Steps to reproduce the behavior:

Create a schema like below:

export class BugSchema extends sf.object('BugSchema', {
  mapSchema: sf.map(MapSchema),
}) {}

export class MapSchema extends sf.object('MapSchema', {
  x: sf.required(sf.number),
  y: sf.required(sf.number),
  z: sf.required(sf.number),
}) { }

Repro Steps:

1. Connect two clients to the same Fluid session.
2. From either client, add two BugSchema entries with IDs 1 and 2.
3. Before adding the third BugSchema entry, take the second client (the one not elected as summarizer) offline.
4. Add the third BugSchema entry with the same ID from both clients (while the second client is offline).
5. Allow the first client (the summarizer) to generate and upload a summary.
6. Bring the second client back online.

Expected Result:
Client resynchronizes without issues.

Actual Result:
Upon reconnection, both clients encounters error 0x379, which leads to document corruption.

Logs

[kaischroeder]

I would like to add that we keep running into this regularly. This is unfortunately not an edge case we can ignore, and we are not aware of a workaround.

[CraigMacomber]

Thanks for reporting the issue. Sorry for the slow response.

I'll attempt to create a test covering this scenario today.

In the event that does not get to the bottom of this, some further information that could be useful if you have it:

• What version(s) of the Fluid Framework Client libraries have you observed this with?
• When you say "which leads to document corruption", do you mean that any client trying to load the document also hits assert 0x379?

As a mitigation/experiment, you can try using configuredSharedTree to select ForestTypeExpensiveDebug. Its possible that the additional validation there might better diagnose the issue, and might even error early enough to prevent the document from getting corrupted.

[neerajcharokar]

@CraigMacomber , We have tested with client versions 2.43.0 and 2.50.0, and we are encountering the same error in both.
Interestingly, while testing with the latest version, I observed that the error occurs even when the same client inserts the same entry into the map repeatedly.

Repro Steps:

1. Client 1 inserts a BugSchema entry with ID 1.
2. Client 1 inserts the same BugSchema entry again with ID 1.

This results in the same error.

[CraigMacomber]

@CraigMacomber , We have tested with client versions 2.43.0 and 2.50.0, and we are encountering the same error in both. Interestingly, while testing with the latest version, I observed that the error occurs even when the same client inserts the same entry into the map repeatedly.

Repro Steps:

1. Client 1 inserts a BugSchema entry with ID 1.
2. Client 1 inserts the same BugSchema entry again with ID 1.

This results in the same error.

I'm not sure I'm understanding these steps correctly.

By "BugSchema entry" do you mean a MapSchema instance in the map node stored in a BugSchema node's mapSchema field?

To me that sounds like:

const sf = new SchemaFactory("test");
class MapSchema extends sf.object("MapSchema", {
	x: sf.required(sf.number),
	y: sf.required(sf.number),
	z: sf.required(sf.number),
}) {}
class BugSchema extends sf.object("BugSchema", {
	mapSchema: sf.map(MapSchema),
}) {}
const config = new TreeViewConfiguration({ schema: BugSchema });
const view = independentView(config, { forest: ForestTypeExpensiveDebug });
view.initialize({ mapSchema: {} });
const bug1 = new MapSchema({ x: 1, y: 2, z: 3 });
view.root.mapSchema.set("1", bug1);
view.root.mapSchema.set("1", bug1);

This however is invalid, since inserting something which is already in the tree is invalid, so an error is correctly thrown with "A node may not be inserted into the tree more than once".

If I instead create a new node each time, it seems to work as expected:

view.root.mapSchema.set("1", new MapSchema({ x: 1, y: 2, z: 3 }));
view.root.mapSchema.set("1", new MapSchema({ x: 1, y: 2, z: 3 }));

Can you still reproduce this issue with a simplified schema like this one?

const sf = new SchemaFactory("test");
class Child extends sf.object("Child", {}) {}
class ParentMap extends sf.map("ParentMap ", Child }) {}
const config = new TreeViewConfiguration({ schema: ParentMap });

Can you provide actual code which triggers this?
A call stack from the exception would also be very helpful.

When you say "take the second client (the one not elected as summarizer) offline" that means to put the container in a disconnected state, and then have it reconnect later right? My understanding is that should result in it downloading the ops which happened while it was offline, then running its own op through resubmit. If thats the case, then the summary should not matter: do you know if the summarizing step is important for this issue to reproduce? Perhaps your issue is not with container offline, but an actual full reconnect and using stashed ops?

So far I have been unable to reproduce this bug. My attempt to reproduce the issue with both sets of reproduction steps. My attempts are included in are included in main...CraigMacomber:FluidFramework:corruptionRepro

If you can see what I'm going that does not match your case which hits the bug, please let me know. Updating one of these tests so that it reproduced the bug would be ideal.