NODE-2274: unified failover server loss

[mbroadst]

Description

The fundamental issue here is that when using the unified topology, if a node is "lost" (due to some network error, etc) it is never regained. This was due to some complicated error handling, primarily around the events propagated from the connection pool. I'll try to summarize in the next section what changed (or needed to be changed)

What changed?

• State machines for the Pool, Server and Topology types were formalized, now using the same method to generate the machines. The Pool gained a new state DRAINING which is used to distinguish whether new commands can be written to the pool or not, in particular during a scenario when fire-and-forget messages are sent right before pool destruction
• The Topology no longer listens for error or close events from the Pool. All errors are propagated through callbacks now, which allows us to be very specific about how we handle types of errors. This was the root cause of the issue addressed by this PR, all errors used to be funneled through a common error handler which would reset the server in SDAM flow, even during monitoring checks
• Topology no longer uses a pool in forced reconnect mode. This was the second part of the root cause, failed attempts to do something like monitoring would cause retries which would trigger the SDAM flow. We now depend on the retry loop of retryable operations for retries explicitly.
• The logic for "resetting a server" was factored, and made explicit during a server description update handling process.
• As a part of this change NODE-2214 was resolved, where Unknown servers were erroneously removed from the topology description

Are there any files to ignore?
A number of test changes were required that were out of scope for this change, but caused test failures nonetheless. Specifically, ignore changes to the transactions tests, and an added tool called run_each_test.sh which is just a helper for identifying individual test files which leak handles in node

[daprahamian]

nit: replace all of this with:

filter(test) {
  const unifiedTopology = test.metadata && 
    test.metadata.requires &&
    test.metadata.requires.unifiedTopology;
  return typeof unifiedTopology !== 'boolean' ||
    unifiedTopology === process.env.MONGODB_UNIFIED_TOPOLOGY;
}

[mbroadst]

done

[daprahamian]

Do we want connections to be creatable when pool is DESTROYING

[imlucas]

nit: it would be swell to have more explicit error messages for debuggability. For example:

callback(new MongoError('Cannot execute a command when the server is closed'));

instead of:

callback(new MongoError('server is closed'));

[mbroadst]

Thanks @imlucas. This particular error should only ever show up as a reason, so I think we're safe being less specific here.

As an aside, I think the error is also sufficient from the perspective of what the stack trace would look like now that we have a very descriptive command dispatch path. It would be like executeOperation => selectServer => command => error server is closed.

[imlucas]

@mbroadst thanks!

🤔 It would be sweet to have a little util to inspect the stack → message that includes the dispatch path. Could solve a problem we have in Compass today where we're only looking/showing at err.message. Adding to my list to think more about and move off this thread.