Fix: Aurora Connection Tracker invalidates the incorrect host after failover

[karenc-bq]

Integration Test Run

https://github.com/Bit-Quill/aws-advanced-jdbc-wrapper/actions/runs/4878201119

Issues

1. After failover Aurora Connection Tracker incorrectly invalidates the newly promoted writer
2. After failover Aurora Connection Tracker incorrectly invalidates a random reader that is neither the old or new writer.

Issue 1 Solution

Issue one is caused by the following check.

  private boolean isRoleChanged(final EnumSet<NodeChangeOptions> changes) {
    return changes.contains(NodeChangeOptions.PROMOTED_TO_WRITER)
        || changes.contains(NodeChangeOptions.PROMOTED_TO_READER);
  }

We need to remove the check for PROMOTED_TO_WRITER

Issue 2 Solution

The tracker plugin calls invalidateAllConnections in 2 locations, in one of the location it passes the original connection url to invalidateAllConnections. If the url is a cluster endpoint the plugin will look up its aliases and pick the first instance endpoint it finds.

This is a problem if the cluster has gone through many changes and has several instance endpoints in the aliases:

    [2023-05-04 00:24:20.970] [FINE   ] software.amazon.jdbc.plugin.AuroraConnectionTrackerPlugin execute
     Invalidate all connections for original host HostSpec[host=database-mysql.cluster-xxx.us-east-2.rds.amazonaws.com, port=3306, WRITER, AVAILABLE, weight=100] after exception: The active SQL connection has changed due to a connection failure. Please re-configure session state if required. 
    [2023-05-04 00:24:20.970] [FINE   ] software.amazon.jdbc.plugin.AuroraConnectionTrackerPlugin execute
     Invalidate all connections for original host with aliases: [mysql-instance-3.xxx.us-east-2.rds.amazonaws.com:3306, mysql-instance-5.xxx.us-east-2.rds.amazonaws.com:3306, mysql-instance-4.xxx.us-east-2.rds.amazonaws.com:3306] 

Extra Information: Logs observed

Before Failover

Topology:
HostSpec[instance-1, READER]
HostSpec[instance-2, READER]
HostSpec[instance-5, READER]
HostSpec[instance-3, READER]
HostSpec[instance-4, WRITER]

After Failover

New writer = instance-5

Changes detected

[2023-05-03 18:34:28.187] [FINEST ] software.amazon.jdbc.hostlistprovider.AuroraHostListProvider forceRefresh
 Topology: 
   HostSpec[host=database-pg-instance-5, WRITER]
   HostSpec[host=database-pg-instance-3, READER]

#### Invalidating Connections After Failover
[2023-05-03 18:34:28.192] [FINEST ] software.amazon.jdbc.plugin.OpenedConnectionTracker logConnectionQueue
 Invalidating opened connections to host: 'database-pg-instance-4[
  org.postgresql.jdbc.PgConnection@28ee7bee
  org.postgresql.jdbc.PgConnection@73ae0257
  org.postgresql.jdbc.PgConnection@5762658b
  org.postgresql.jdbc.PgConnection@2629d5dc
  org.postgresql.jdbc.PgConnection@2596d7f4
  org.postgresql.jdbc.PgConnection@6aa3bfc
]' 

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[karenc-bq]

#431