Fix method calls during reconnect

[OleksandrChekhovskyi]

User could enter irrecoverable broken state after doing any action while
reconnecting (specifically between stream reset and quiescence).
In this state some or all documents in collections would be missing,
because the loop processing buffered messages is interrupted with an
exception in _process_added.

This commonly happened on slow/bad networks, such as mobile.

[glasser]

Hi @OleksandrChekhovskyi! I think you've correctly identified a real and serious bug, and written a good fix and great test.

But I don't think your fix has the ideal semantics. Helpfully, I can point exactly to where it differs from what I think it should do in your test!

When a stub writes to a document, we're supposed to continue to see the stub value until the method returns. So I think your addition of _pushUpdate in this case is wrong, and at line 1853 we should still be seeing a value of 222.

Maybe to make your test more complete you could (a) add an assertion that the value becomes 222 as soon as the conn.call is done and (b) make the server send back 333 instead of 222 at the end, and we can see that the value goes from 111 before the call to 222 after the call to 333 after the method is done without ever "flickering" back to 111. (And I think the proper change to the actual code is just to not throw the error if _resetStores is set but otherwise still set the server document and not push the update for immediate application.)

[OleksandrChekhovskyi]

I've updated the patch to ensure that stub-written values survive reset.
It's a bit more complicated than what you've described though. _pushUpdate call has to be there, otherwise the store reset will simply remove the document. So to keep stub-written values, we still push the update, but with current document's fields.
Also added more comments in the code to clarify details.

[glasser]

maybe even verify that this message isn't what changes it (though this isn't relevant to your fix)

[glasser]

OK, this looks good. I'm a little concerned about what happens if the method returns before the subs all get ready (ie, before quiescence) but that can be a later issue to fix...

[dgreensp]

@OleksandrChekhovskyi, thank you for this PR!

[OleksandrChekhovskyi]

I'm a little concerned about what happens if the method returns before the subs all get ready

That should ideally be tested as well, but right now it's a pure theoretical concern, as in practice the method won't even be sent until quiescence (blocked by login wait method, injected from onReconnect callback).

[sasikanth513]

we're getting the error server sent add for exsting item id:: xxxx most of the times when we disconnect and reconnect again and most of the times our app crashes because of this issue.

Hoping this PR will solve that issue. 👍