Skip to content

Comments

Allow filtering cassandra nodes by datacentre#1886

Merged
jschaul merged 6 commits intodevelopfrom
bm-14
Nov 10, 2021
Merged

Allow filtering cassandra nodes by datacentre#1886
jschaul merged 6 commits intodevelopfrom
bm-14

Conversation

@jschaul
Copy link
Member

@jschaul jschaul commented Oct 26, 2021

Context: https://wearezeta.atlassian.net/browse/BM-14 and related PR https://github.com/zinfra/cailleach/pull/703

  • cassandra-util: add a dcAwarePolicy for talking to cassandra
  • Spar/brig/galley/gundeck: allow filtering nodes by datacentre

Example: galley/gundeck don't configure this option, spar configures a valid datacentre, brig configures an invalid datacentre:

[galley] I, Using the following cassandra load balancing options ('Policy'):, filter_datacentre=Nothing
[gundeck] I, Using the following cassandra load balancing options ('Policy'):, filter_datacentre=Nothing
[spar] I, Using the following cassandra load balancing options:, filter_datacentre=Just "datacenter1"

[brig] I, Using the following cassandra load balancing options ('Policy'):, filter_datacentre=Just "datacenter2"
[brig] I, logger=cassandra.brig, Known hosts: []
[brig] I, logger=cassandra.brig, New control connection: datacenter1:rack1:127.0.0.1:9042#<socket: 11>
[brig] E, logger=cassandra.brig, No known hosts.
[brig] I, logger=cassandra.brig, Renewing control connection with initial contacts ...

(...)
brig: Failed to read schema version from meta table. Error was: cql-io: no host available
[brig] E, logger=cassandra.brig, No known hosts.
[brig] I, logger=cassandra.brig, Renewing control connection with initial contacts ...

So the behaviour is:

  • if not configured, fallback to random policy (i.e. no change from now)
  • if configured but wrongly, the service will fail to start up
  • if configured correctly, only nodes in the datacentre that is configured will be used as coordinator nodes, so another datacentre going down will have no effect.

Checklist

  • The PR Title explains the impact of the change.
  • The PR description provides context as to why the change should occur and what the code contributes to that effect. This could also be a link to a JIRA ticket or a Github issue, if there is one.
  • changelog.d contains the following bits of information:
    • A file with the changelog entry in one or more suitable sub-sections. The sub-sections are marked by directories inside changelog.d.

@jschaul jschaul changed the title Allow filtering cassandra nodes by datacentre WIP [skip ci] Allow filtering cassandra nodes by datacentre Nov 3, 2021
parseAWSEndpoint :: ReadM AWSEndpoint
parseAWSEndpoint = readerAsk >>= maybe (error "Could not parse AWS endpoint") return . fromByteString . fromString

cassandraParser :: Parser CassandraOpts
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was dead code not used anywhere.

@jschaul jschaul merged commit 922ec36 into develop Nov 10, 2021
@jschaul jschaul deleted the bm-14 branch November 10, 2021 11:01
@akshaymankar akshaymankar mentioned this pull request Nov 15, 2021
@smatting smatting mentioned this pull request Dec 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants