integration: Add test to verify behaviour with offline backends by akshaymankar · Pull Request #3501 · wireapp/wire-server

akshaymankar · 2023-08-14T12:35:39Z

Tracked by https://wearezeta.atlassian.net/browse/WPB-2890.

Checklist

~~Add a new entry in an appropriate subdirectory of changelog.d~~ No changelog
Read and follow the PR guidelines

… when fed policy is AllowAll

mdimjasevic

I'm yet to wrap my head around the new test.

Some comments for the rest are inlined.

mdimjasevic · 2023-08-21T08:20:19Z

hack/helmfile.yaml

+      - name: brig.config.optSettings.setFederationDomainConfigs[2].domain
+        value: {{ .Values.dynBackendDomain1 }}
+      - name: brig.config.optSettings.setFederationDomainConfigs[3].domain
+        value: {{ .Values.dynBackendDomain2 }}
+      - name: brig.config.optSettings.setFederationDomainConfigs[4].domain
+        value: {{ .Values.dynBackendDomain3 }}


I don't understand this, but two lines above the index was 0, and this block starts with 2. Is index 1 skipped on purpose?

In hack/helm_vars/wire-server/values.yaml.gotmpl we have configured this:

setFederationDomainConfigs: # 'setFederationDomainConfigs' is deprecated as of https://github.com/wireapp/wire-server/pull/3260. See # https://docs.wire.com/understand/federation/backend-communication.html#configuring-remote-connections # for details. - domain: integration.example.com search_policy: full_search - domain: federation-test-helper.{{ .Release.Namespace }}.svc.cluster.local search_policy: full_search # Remove these after fixing https://wearezeta.atlassian.net/browse/WPB-3796 - domain: dyn-backend-1 search_policy: full_search - domain: dyn-backend-2 search_policy: full_search - domain: dyn-backend-3 search_policy: full_search

Notice how except for the second domain, everything else is placeholders. Here we just replace the placeholders. This is a workaround for this issue: https://wearezeta.atlassian.net/browse/WPB-3796
Eventually we should delete all these explicit search policy stuff.

mdimjasevic · 2023-08-21T08:20:27Z

hack/helmfile.yaml

+      - name: brig.config.optSettings.setFederationDomainConfigs[2].domain
+        value: {{ .Values.dynBackendDomain1 }}
+      - name: brig.config.optSettings.setFederationDomainConfigs[3].domain
+        value: {{ .Values.dynBackendDomain2 }}
+      - name: brig.config.optSettings.setFederationDomainConfigs[4].domain
+        value: {{ .Values.dynBackendDomain3 }}


mdimjasevic · 2023-08-21T08:28:28Z

integration/test/Test/Federation.hs

+allPreds :: (Applicative f) => [a -> f Bool] -> a -> f Bool
+allPreds [] _ = pure True
+allPreds [p] x = p x
+allPreds (p1 : ps) x = (&&) <$> p1 x <*> allPreds ps x
+
+isDeleteUserNotif :: MakesValue a => a -> App Bool
+isDeleteUserNotif n =
+  nPayload n %. "type" `isEqual` "user.delete"
+
+isNewMessageNotif :: MakesValue a => a -> App Bool
+isNewMessageNotif n = fieldEquals n "payload.0.type" "conversation.otr-message-add"
+
+isMemberJoinNotif :: MakesValue a => a -> App Bool
+isMemberJoinNotif n = fieldEquals n "payload.0.type" "conversation.member-join"
+
+isConvLeaveNotif :: MakesValue a => a -> App Bool
+isConvLeaveNotif n = fieldEquals n "payload.0.type" "conversation.member-leave"
+
+isNotifConv :: (MakesValue conv, MakesValue a) => conv -> a -> App Bool
+isNotifConv conv n = fieldEquals n "payload.0.qualified_conversation" (objQidObject conv)
+
+isNotifForUser :: (MakesValue user, MakesValue a) => user -> a -> App Bool
+isNotifForUser user n = fieldEquals n "payload.0.data.qualified_user_ids.0" (objQidObject user)
+
+fieldEquals :: (MakesValue a, MakesValue b) => a -> String -> b -> App Bool
+fieldEquals a fieldSelector b = do
+  ma <- lookupField a fieldSelector `catchAll` const (pure Nothing)
+  case ma of
+    Nothing -> pure False
+    Just f ->
+      f `isEqual` b


These general utilities should be in a separate module, shouldn't they?

Good idea, moved.

mdimjasevic · 2023-08-21T09:18:43Z

I managed to get this test to fail locally once:

----- Federation.testNotificationsForOfflineBackends FAIL (4.02 s) -----
assertion failure:
Expected 1 matching events, but got 0.
Details:
AwaitResult
    success: False
    matches:

    non-matches:
    {
        "id": "d342d3df-4002-11ee-8001-960000bf3df4",
        "payload": [                                                                                                                                                                                      {
                "conversation": "3ecb40c6-d3cd-489f-af4d-c55b11e40722",                                                                                                                                       "data": {
                    "user_ids": [
                        "97a6eafc-49d4-425c-9322-488ae218cb01"
                    ],
                    "users": [
                        {                                                                                                                                                                                                 "conversation_role": "wire_admin",
                            "id": "97a6eafc-49d4-425c-9322-488ae218cb01",
                            "qualified_id": {
                                "domain": "b.example.com",
                                "id": "97a6eafc-49d4-425c-9322-488ae218cb01"
                            }                                                                                                                                                                                         }
                    ]
                },
                "from": "9acfd557-c55e-43a3-9285-ef558d1c7a59",
                "qualified_conversation": {
                    "domain": "example.com",                                                                                                                                                                      "id": "3ecb40c6-d3cd-489f-af4d-c55b11e40722"
                },
                "qualified_from": {
                    "domain": "example.com",
                    "id": "9acfd557-c55e-43a3-9285-ef558d1c7a59"
                },                                                                                                                                                                                            "time": "2023-08-21T09:12:17.347Z",
                "type": "conversation.member-join"                                                                                                                                                        }
        ]
    }
    {
        "id": "d3458a9a-4002-11ee-8001-960000bf3df4",
        "payload": [                                                                                                                                                                [178/1958]
            {
                "conversation": "3ecb40c6-d3cd-489f-af4d-c55b11e40722",
                "data": {
                    "qualified_user_ids": [
                        {
                            "domain": "b.example.com",
                            "id": "97a6eafc-49d4-425c-9322-488ae218cb01"
                        }
                    ],
                    "user_ids": [
                        "97a6eafc-49d4-425c-9322-488ae218cb01"
                    ]
                },
                "from": "9acfd557-c55e-43a3-9285-ef558d1c7a59",
                "qualified_conversation": {
                    "domain": "example.com",
                    "id": "3ecb40c6-d3cd-489f-af4d-c55b11e40722"
                },
                "qualified_from": {
                    "domain": "example.com",
                    "id": "9acfd557-c55e-43a3-9285-ef558d1c7a59"
                },
                "time": "2023-08-21T09:12:17.367Z",
                "type": "conversation.member-leave"
            }
        ]
    }
    {
        "id": "d35032d6-4002-11ee-8001-960000bf3df4",
        "payload": [
            {
                "conversation": "3ecb40c6-d3cd-489f-af4d-c55b11e40722",
                "data": {
                    "qualified_user_ids": [
                        {
                            "domain": "example.com",
                            "id": "9acfd557-c55e-43a3-9285-ef558d1c7a59"
                        }
                    ],
                    "user_ids": [
                        "9acfd557-c55e-43a3-9285-ef558d1c7a59"
                    ]
                },
                "from": "9acfd557-c55e-43a3-9285-ef558d1c7a59",
                "qualified_conversation": {
                "qualified_conversation": {
                    "domain": "example.com",
                    "id": "3ecb40c6-d3cd-489f-af4d-c55b11e40722"
                },
                "qualified_from": {
                    "domain": "example.com",
                    "id": "9acfd557-c55e-43a3-9285-ef558d1c7a59"
                },
                "time": "2023-08-21T09:12:17.436Z",
                "type": "conversation.member-leave"
            }
        ]
    }

                                                                                                                                                             
call stack:                                                                                                                                                   
1. assertFailure at test/Testlib/Cannon.hs:420                                                                                                                
     assertFailure $ unlines [msgHeader, details]                                                                                                                                                                                                                                                                           
2. assertAwaitResult at test/Notifications.hs:21                                                                                                              
     assertAwaitResult =<< go tSecs since0 (AwaitResult False n [] [])                                                                                        
                                                                                                                                                              
3. awaitNotifications at test/Notifications.hs:53                                                                                                             
     head <$> awaitNotifications user client since0 tSecs 1 selector                                                                                                                                                                                                                                                        
4. awaitNotification at test/Test/Federation.hs:106                                                                                                           
     delUserDeletedNotif <- nPayload $ awaitNotification otherUser otherClient (Just newMsgNotif) 1 isDeleteUserNotif

…ught up

akshaymankar · 2023-08-21T10:07:34Z

@mdimjasevic Can you also post the stack trace for the failing test, it is very hard to tell just from the failure because almost all assertions are about expecting some notification.

mdimjasevic · 2023-08-21T11:29:15Z

@mdimjasevic Can you also post the stack trace for the failing test, it is very hard to tell just from the failure because almost all assertions are about expecting some notification.

Ah, my bad. I've added it to the original report.

mdimjasevic · 2023-08-21T12:37:10Z

There was a flaky test failure. I've retriggered the CI.

----- Brig.testRemoteUserSearch FAIL (5.35 s) -----

assertion failure:

Actual:

533

Expected:

201


call stack: 

1. assertFailure at test/Testlib/Assertions.hs:52


2. shouldMatch at test/Testlib/Assertions.hs:98


3. shouldMatchInt at test/SetupHelpers.hs:52


4. connectUsers at test/SetupHelpers.hs:63


5. createAndConnectUsers at test/Test/Brig.hs:191


6. k at test/Testlib/ModService.hs:151


7. startDynamicBackends at test/Test/Brig.hs:188




request: 

POST http://127.0.0.1:35161/v4/connections/dynamic-backend-2.test-7ubmn9qetcap.svc.cluster.local/89fd2029-8917-454a-968e-8fd1ec21bd1b

request headers: 

Z-Connection: conn

Z-User: 8f3cbdd6-b5b7-4faa-9ce4-a640a3f816a3

request body:


response status: 533

response body:

{

    "code": 533,

    "data": {

        "domain": "dynamic-backend-2.test-7ubmn9qetcap.svc.cluster.local",

        "path": "/federation/api-version",

        "type": "federation"

    },

    "label": "federation-remote-error",

    "message": "A remote federator failed with status code 503"

}

Went away in the new CI run.

* Fix bug: federatorInternal host not set for background-worker (#3516) * WPB-3916: Filtering out duplicate members when sending defederation notifications (#3515) * integration: Add test to verify behaviour with offline backends (#3501) * background-worker: Make push backoff times configurable * brig/getFederationStatus: Always return NonConnectedBackends as empty when fed policy is AllowAll * integration: Use separate vHosts for backendA and B. * integration/RunServices: Add hack to make federation work * integration: Add test to verify behaviour with offline backends * helm-var-integration: Workaround bug with federation * integration-test.sh: Run new integration test suite first --------- Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com> * Distinguish between update and upsert cassandra commands (#3513) * Remove billing-team-member-backfill tool (#3520) * dockerephemeral: Increase nofile ulimits for ES and Fake DynamoDB (#3521) * [WPB 3842] Federation completeness check (#3514) * WPB-3842: Improving checks for adding users to a conversation. Added a check to `ensureAllowed` that checks for full federation connections for domains in a conversation, including the domains for new users. * WPB-3842: Adding the changelog * WPB-3842: Moving where the extra domain checks are being performed. Updating integration tests to reflect the updated semantics of conversation join semantics. Many of them weren't expecting errors relating to unreachable domains, and had to be updated to reflect this. * Fix asserted domains in an integration test * Integration test: assert on non-federating domains * WPB-3842: Changing parallel testing to sequential testing --------- Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com> * WPB-3798 incorrect json field names (#3518) * WPB-3798: Updating code and tests after renaming fields * WPB-3798: More updates to names after finding more JSON prefix mangling * WPB-3798: Fixing schema instances for SAML data * WPB-3798: Fixing instances that had errors, found by tests * WPB-3798: Adding changelogs * WPB-3798: PR feedback. * WPB-3798: Fixing an error with a field called `data'` The trailing ' would end up in the JSON representation. I've changed it to use a leading `_` like other structures, and wrote a newtype to handle the minimal prefix stripping. Also cleaning up the diff in regards to imports. * WPB-3798: Cleaning up imports to minimise the diff * nit-picks (#3519) * Remove unneeded -Wwarn (re-enabeling -Werror in those modules). * Makefile: fix hspec_options overloading in .envrc.local. * integration: Fix testAddingUserNonFullyConnectedFederation and testNotificationsForOfflineBackends (#3529) * integration: Fix testAddingUserNonFullyConnectedFederation * integration: Don't allow adding users to conv when one of the pariticipating backends is down * integration: Add retries to get around problem of federation domain sync threads * Introduce API v5 (#3527) * Introduce development version 5 * Specialise API to a specific version * Use versioned swagger for galley * Use version swagger for all other services * Collect all service Swaggers into a typeclass * Fix swagger integration tests * Revert any changes to API versions before 5 * Remove promotion of isDevelopmentVersion * Add CHANGELOG entry * stern: Optimize RAM usage of /i/users/meta-info (#3522) * stern: Fetch only the notifications that are needed * stern: Fetch only the conversations that are needed * Integration tests: use static ports (#3536) * [WPB-3799] cannot fetch conversation details after connection request (#3538) * brig-integration: Fix flaky tests for API.Federation (#3539) * brig-integration: Don't assume only 1 result in search by display name Display names are random strings from 2 to 128 characters. If a 2 string name gets generated it is likely that it matches some name generated in another test. * brig-integration: Mark test not flaky It didn't fail after runnning it 1000 times. * Integration suite: Fix bug in local setup: wrong port for nginz http2 (#3543) * [WPB-662] servantify brig provider bot api (#3540) * Fix broken "we are hiring" link (#3549) * Multi-ingress guest links (#3546) * Check validity of notification IDs (#3550) * Check validity of notification IDs * Add CHANGELOG entry * fixup! Add CHANGELOG entry * fixup! fixup! Add CHANGELOG entry * WPB-633 Servantify Brig/Provider.Service API (#3554) * WPB-1214: Servantify Brig/Provider.Service API - Moving the routes over to servant, and removing the old routing code. - Adding new instances to types that needed them for servant. * WPB-663: Removing a redundant TODO comment, adding changelog * Fix ES migration script. (#3558) * Revert "WPB-633 Servantify Brig/Provider.Service API (#3554)" This reverts commit 3653d56. * Integration tests: delete all rabbitmq queues during dynamic backends setup phase (#3523) * [WPB-4406] federator improve logging (#3556) * Makefile: Avoid executing the hint (#3564) Backticks execute the command even when they are in quotes. * Finalise v4 (#3545) * Remove MLS endpoints from the API They will be reintroduced when merging the mls branch. These endpoints are not currently functional on develop, so removing them from here will reduce the amount of conflicts. * Finalise v4 * Add CHANGELOG entry * Add pregenerated swagger for v4 * Delete MLS tests in brig * Remove more MLS endpoints from v4 * Set default API version to 5 in integration tests * Update the documentation on API versioning --------- Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com> * Fix: SCIM user lookup after changing IdP issuer ID (#3473) * doc: document webapp configuration for multi-ingress environments (#3569) --------- Co-authored-by: Sven Tennie <sven.tennie@gmail.com> * [WPB-4361] upgrade jwt-tools (#3559) * cassandra: Add column and table names in parsing error messages (#3555) * s/CORS/CSP/ as mentionned by Sven in WPB-2912 * Replace broken integrations with links see WPB-3599 * replace all instances of example.com with wire.example as per wpb-2621, in charts only * change back from wire.example to example.com as this was mistakenly commit to develop instead of to the proper branch * add documentation on creating a first user * reverting previous commit as sent to wrong branch * Update sftd docs: include uri scheme in allowOrigin (#3584) * Update sftd docs: include uri scheme in allowOrigin * fixup * WPB-4629 impossible to add users to a conversation if one of the members is from an offline backend (#3585) * fake-aws-s3 chart: Upgrade to minio 5.0.13 (#3565) * Disable de-federation to avoid running into a scalability issue (#3582) https://wearezeta.atlassian.net/browse/WPB-4668 Co-authored-by: Akshay Mankar <akshay@wire.com> * [WPB-3664] Bug fix: Notify remote backends of their users removed from conversation when reachable again (#3537) * Formatting * Test utilities for changing a conv name * Add a test confirming the bug report * An action to enqueue notifications concurrently * Enqueue member removal notification for remotes * Add a changelog * Test case formatting * Migrate test roleUpdateWithRemotesUnavailable * Migrate test putReceiptModeWithRemotesOk * Migrate test putReceiptModeWithRemotesUnavailable * Migrate test testRoleUpdateWithRemotesOk * Migrate test roleUpdateRemoteMember * Migrate test putQualifiedConvRenameWithRemotesUnavailable This one is already covered by testSynchroniseUserRemovalNotification * Migrate test putQualifiedConvRenameWithRemotesOk * Migrate test deleteLocalMemberConvLocalQualifiedOk * Migrate test deleteRemoteMemberConvLocalQualifiedOk * Migrate test deleteUnavailableRemoteMemberConvLocalQualifiedOk * Add the copyright header to a test module * Move a test utility (allPreds) * Test utility: create a team with members * Migrate test testAccessUpdateGuestRemoved * Migrate test messageTimerChangeWithRemotes * Migrate test messageTimerUnavailableRemotes * Migrate test testAccessUpdateGuestRemovedRemotesUnavailable * Migrate test accessUpdateWithRemotes * Migrate test testAddRemoteMember * Migrate test testDeleteTeamConversationWithRemoteMembers * Migrate test testDeleteTeamConversationWithUnavailableRemoteMembers * Move a test utility (assertLeaveNotification) * Migrate test "POST /federation/leave-conversation : Success" * Migrate test "POST /federation/on-user-deleted-conversations : Remove deleted remote user from local conversations" * Migrate test updateConversationByRemoteAdmin * Tests: support giving a role when adding * Use cannon API for notifications when possible * Use startDynamicBackends when possible * Fix assertion * Migrate test testAddRemoteUsersToLocalConv * Test add member endpoint at version 1 * Add return value to enqueueNotification * Use cannon assertions in offline backends test * Check that remote notifications are received * Test removal of users from unreachable backends * Use correct domains for default backends Taking the domains in the `backendA` and `backendB` resources only works locally. * fixup! Use cannon assertions in offline backends test --------- Co-authored-by: Paolo Capriotti <paolo@capriotti.io> Co-authored-by: Akshay Mankar <akshay@wire.com> * WPB-4240: Migrate from swagger2 to openapi3 (#3570) --------- Co-authored-by: Igor Ranieri Elland <54423+elland@users.noreply.github.com> Co-authored-by: Igor Ranieri <igor@elland.me> * Remove mocked MLS member add test * Resolve conflict in pregenerated swagger * Remove MLS end2end tests --------- Co-authored-by: Stefan Matting <smatting@users.noreply.github.com> Co-authored-by: Owen Harvey <owenlharvey@gmail.com> Co-authored-by: Akshay Mankar <akshay@wire.com> Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com> Co-authored-by: fisx <mf@zerobuzz.net> Co-authored-by: Igor Ranieri Elland <54423+elland@users.noreply.github.com> Co-authored-by: Leif Battermann <leif.battermann@wire.com> Co-authored-by: Jappie Klooster <jappieklooster@hotmail.com> Co-authored-by: Leif Battermann <leifbattermann@gmail.com> Co-authored-by: Thomas Belin <thomasbelin4@gmail.com> Co-authored-by: Sven Tennie <sven.tennie@gmail.com> Co-authored-by: Arthur Wolf <wolf.arthur@gmail.com> Co-authored-by: Igor Ranieri <igor@elland.me>

zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Aug 14, 2023

akshaymankar force-pushed the integration-offline-backend branch from 835d9ec to 0d9a739 Compare August 14, 2023 12:53

akshaymankar marked this pull request as ready for review August 14, 2023 13:53

akshaymankar added 8 commits August 21, 2023 09:40

background-worker: Make push backoff times configurable

96ab1d0

brig/getFederationStatus: Always return NonConnectedBackends as empty…

97becbc

… when fed policy is AllowAll

integration: Use separate vHosts for backendA and B.

45c958b

integration/RunServices: Add hack to make federation work

e4d1967

integration: Add test to verify behaviour with offline backends

4ddb67d

regen nix

93d9d83

Little hack

542f4dd

helm-var-integration: Workaround bug with federation

3db9c13

mdimjasevic force-pushed the integration-offline-backend branch from 5816501 to 3db9c13 Compare August 21, 2023 08:17

mdimjasevic reviewed Aug 21, 2023

View reviewed changes

Integration test: fix asserted status code

de56d22

integration: Assert about down backend conv after down backend is bro…

54b3587

…ught up

akshaymankar added 2 commits August 21, 2023 12:26

integration: Move utility functions in appropriate modules

370a8e4

integration-test.sh: Run new integration test suite first

6e9c2dd

mdimjasevic approved these changes Aug 21, 2023

View reviewed changes

akshaymankar merged commit 52d69ca into develop Aug 21, 2023

akshaymankar deleted the integration-offline-backend branch August 21, 2023 12:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

integration: Add test to verify behaviour with offline backends#3501

integration: Add test to verify behaviour with offline backends#3501
akshaymankar merged 12 commits intodevelopfrom
integration-offline-backend

akshaymankar commented Aug 14, 2023 •

edited by mdimjasevic

Loading

Uh oh!

mdimjasevic left a comment

Uh oh!

mdimjasevic Aug 21, 2023

Uh oh!

akshaymankar Aug 21, 2023

Uh oh!

mdimjasevic Aug 21, 2023

Uh oh!

mdimjasevic Aug 21, 2023

Uh oh!

akshaymankar Aug 21, 2023

Uh oh!

mdimjasevic commented Aug 21, 2023 •

edited

Loading

Uh oh!

akshaymankar commented Aug 21, 2023

Uh oh!

mdimjasevic commented Aug 21, 2023

Uh oh!

mdimjasevic commented Aug 21, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

akshaymankar commented Aug 14, 2023 • edited by mdimjasevic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

mdimjasevic left a comment

Choose a reason for hiding this comment

Uh oh!

mdimjasevic Aug 21, 2023

Choose a reason for hiding this comment

Uh oh!

akshaymankar Aug 21, 2023

Choose a reason for hiding this comment

Uh oh!

mdimjasevic Aug 21, 2023

Choose a reason for hiding this comment

Uh oh!

mdimjasevic Aug 21, 2023

Choose a reason for hiding this comment

Uh oh!

akshaymankar Aug 21, 2023

Choose a reason for hiding this comment

Uh oh!

mdimjasevic commented Aug 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akshaymankar commented Aug 21, 2023

Uh oh!

mdimjasevic commented Aug 21, 2023

Uh oh!

mdimjasevic commented Aug 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

akshaymankar commented Aug 14, 2023 •

edited by mdimjasevic

Loading

mdimjasevic commented Aug 21, 2023 •

edited

Loading

mdimjasevic commented Aug 21, 2023 •

edited

Loading