Crash in Realm Sync, realm::SyncSession::handle_location_update_failed, realm_core_v_13_26_0

[gyratorycircus]

How frequently does the bug occur?

Sometimes

Description

Since releasing an update with Realm Swift 10.47.0 a week ago, we're receiving a ton of crash reports for this issue. We've received 1273 crash reports across across 300 users. This is 10x higher than any other crash, and also 10x higher than all crashes from the previous version of Realm Swift in use, 10.43.0.

Stacktrace & log output

Logs:
2024-03-08 13:41:55:417         <1877439> init(type:)(RealmManager.swift 27)
2024-03-08 13:41:55:417  [INFO] REALM[ERROR] App: request location failed (CustomError -1200): An SSL error has occurred and a secure connection to the server cannot be made.
2024-03-08 13:41:55:418         <1877428> handleSyncError(_:session:)(RealmCoordinator.swift 91)
2024-03-08 13:41:55:418 [ERROR] Connection Failed: related decl 'e' for RLMSyncError(_nsError: Error Domain=io.realm.sync Code=12 "Unable to reach the server: An SSL error has occurred and a secure connection to the server cannot be made." UserInfo={NSLocalizedDescription=Unable to reach the server: An SSL error has occurred and a secure connection to the server cannot be made.})
2024-03-08 13:41:55:589         <1877855> init(type:)(RealmManager.swift 27)
2024-03-08 13:41:55:589  [INFO] REALM[ERROR] App: request location failed (CustomError -1200): An SSL error has occurred and a secure connection to the server cannot be made.

==============================================================================================
Crash Report:
Incident Identifier: 78ab8d4c-dccb-4eea-a024-d693500d9f9d
CrashReporter Key:   6F19E5C8-EF0F-4AD7-B843-9643590523AD
Hardware Model:      iPhone14,3
Process:         xxxxxxxxxxxxxxxxxxx [4973]
Path:            /private/var/containers/Bundle/Application/FE17F96D-8201-4975-BFE5-963B2B7EF5D8/xxxxxxxxxxxxxxxxxxx.app/xxxxxxxxxxxxxxxxxxx
Identifier:      xxxxxxxxxxxxxxxxxxx
Version:         2024.2.3 (500)
Code Type:       arm64
Parent Process:   [1]

Date/Time:       2024-03-08T13:41:55.999Z
Launch Time:     2024-03-08T13:40:31Z
OS Version:      iPhone OS 17.4 (21E219)
Report Version:  104

Exception Type:  SIGABRT
Exception Codes: #0 at 0x1edaf6974
Crashed Thread:  12

Thread 12 Crashed:
0   libsystem_kernel.dylib               0x00000001edaf6974 __pthread_kill + 8
1   libsystem_c.dylib                    0x00000001ad53fb80 abort + 176
2   Realm                                0x000000010276a450 please_report_this_issue_in_github_realm_realm_core_v_13_26_0 + 8
3   Realm                                0x000000010276a784 realm::util::terminate_internal(std::__1::basic_stringstream<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) + 240
4   Realm                                0x000000010276a5f4 realm::util::terminate_with_info(char const*, char const*, long, char const*, std::initializer_list<realm::util::Printable>&&) + 396
5   Realm                                0x000000010276a464 realm::util::terminate(char const*, char const*, long, std::initializer_list<realm::util::Printable>&&) + 16
6   Realm                                0x00000001028f41c8 realm::SyncSession::handle_location_update_failed(realm::Status) + 1004
7   Realm                                0x00000001029011e0 realm::util::UniqueFunction<void (std::__1::optional<realm::app::AppError>)>::SpecificImpl<realm::SyncSession::handle_refresh(std::__1::shared_ptr<realm::SyncSession> const&, bool)::$_2>::call(std::__1::optional<realm::app::AppError>&&) + 996
8   Realm                                0x000000010290b668 realm::util::UniqueFunction<void (std::__1::optional<realm::app::AppError>)>::SpecificImpl<realm::SyncUser::refresh_custom_data(bool, realm::util::UniqueFunction<void (std::__1::optional<realm::app::AppError>)>)::$_7>::call(std::__1::optional<realm::app::AppError>&&) + 316
9   Realm                                0x00000001028c75d8 realm::util::UniqueFunction<void (realm::app::Response const&)>::SpecificImpl<realm::app::App::refresh_access_token(std::__1::shared_ptr<realm::SyncUser> const&, bool, realm::util::UniqueFunction<void (std::__1::optional<realm::app::AppError>)>&&)::$_12>::call(realm::app::Response const&) + 156
10  Realm                                0x00000001028c6718 realm::util::UniqueFunction<void (std::__1::optional<realm::app::AppError>)>::SpecificImpl<realm::app::App::update_location_and_resend(realm::app::Request&&, realm::util::UniqueFunction<void (realm::app::Response const&)>&&, std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&&)::$_8>::call(std::__1::optional<realm::app::AppError>&&) + 176
11  Realm                                0x00000001028c5a14 realm::util::UniqueFunction<void (realm::app::Request&&, realm::app::Response const&)>::SpecificImpl<realm::app::App::request_location(realm::util::UniqueFunction<void (std::__1::optional<realm::app::AppError>)>&&, std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&&, std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&&, int)::$_7>::call(realm::app::Request&&, realm::app::Response const&) + 816
12  Realm                                0x00000001024c167c ___ZN12_GLOBAL__N_121CocoaNetworkTransport22send_request_to_serverERKN5realm3app7RequestEONS1_4util14UniqueFunctionIFvRKNS2_8ResponseEEEE_block_invoke (RLMApp.mm:78)
13  Realm                                0x00000001024fe9b0 -[RLMSessionDelegate URLSession:task:didCompleteWithError:] (RLMNetworkTransport.mm:0)
14  CFNetwork                            0x00000001a66d798c _CFNetworkSetHSTSStoragePath + 55244
15  libdispatch.dylib                    0x00000001ad48513c _dispatch_call_block_and_release + 28
16  libdispatch.dylib                    0x00000001ad486dd4 _dispatch_client_callout + 16
17  libdispatch.dylib                    0x00000001ad48e400 _dispatch_lane_serial_drain + 744
18  libdispatch.dylib                    0x00000001ad48ef64 _dispatch_lane_invoke + 428
19  libdispatch.dylib                    0x00000001ad499cb4 _dispatch_root_queue_drain_deferred_wlh + 284
20  libdispatch.dylib                    0x00000001ad499528 _dispatch_workloop_worker_thread + 400
21  libsystem_pthread.dylib              0x0000000201574f20 _pthread_wqthread + 284
22  libsystem_pthread.dylib              0x0000000201574fc0 start_wqthread + 4

Thread 7:
0   libsystem_kernel.dylib               0x00000001edafb8cc kevent + 8
1   Realm                                0x00000001027bf8a8 realm::sync::network::Service::IoReactor::wait_and_advance(std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, std::__1::chrono::time_point<std::__1::chrono::steady_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >, bool&, realm::sync::network::Service::OperQueue<realm::sync::network::Service::AsyncOper>&) + 40
2   Realm                                0x00000001027c1bfc realm::sync::network::Service::Impl::run_impl(bool) + 348
3   Realm                                0x00000001027b28ec realm::sync::websocket::DefaultSocketProvider::event_loop() + 216
4   Realm                                0x00000001027b4a8c void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (realm::sync::websocket::DefaultSocketProvider::*)(), realm::sync::websocket::DefaultSocketProvider*> >(void*) + 60
5   libsystem_pthread.dylib              0x0000000201575a90 _pthread_start + 132
6   libsystem_pthread.dylib              0x0000000201574fcc thread_start + 4

Thread 8:
0   libsystem_kernel.dylib               0x00000001edafb8cc kevent + 8
1   Realm                                0x00000001028424c4 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, realm::_impl::ExternalCommitHelper::ExternalCommitHelper(realm::_impl::RealmCoordinator&, realm::RealmConfig const&)::$_0> >(void*, void*) + 48
2   libsystem_pthread.dylib              0x0000000201575a90 _pthread_start + 132
3   libsystem_pthread.dylib              0x0000000201574fcc thread_start + 4

Can you reproduce the bug?

No

Reproduction Steps

No response

Version

10.47.0

What Atlas Services are you using?

Atlas Device Sync

Are you using encryption?

No

Platform OS and version(s)

iOS (iPhone and iPad), 17.4, 17.3.1, 17.3, 17.2, 17.1.1, 16.3.1

Build environment

Xcode version: 15.2
MacOS version: 14.1.3
Dependency manager and version: CocoaPods 1.15.0

[sync-by-unito]

➤ PM Bot commented:

Jira ticket: RCOCOA-2302

[nirinchev]

@michael-wb can you take a look at this one?

[sync-by-unito]

➤ michael-wb commented:

This looks like it is related to RCORE-1982 that I am currently working on. The error reported by core is not fatal, but the sync session will not attempt to reconnect. Both of these will be addressed.

The current workaround is to is to ignore the error for now and the reconnect should occur (at least on Cocoa) once the device is connected to the network.

[nirinchev]

I think the problem is that we're hitting an assertion failure in handle_location_update_failed, so there's no obvious way to ignore the error here.

[gyratorycircus]

Thanks all! We've received more information from users, and it seems to be occurring when a network connection appears to exist (local wifi), but that connection may be restricted to most outside sources. Unfortunately this is fairly common use case for our users, so it's quite impactful at the moment.

It looks like the assertion was introduced in realm-core 13.26.0 in this commit.

I was wondering if it would be safe for a production app to rollback from Realm Swift v10.47 to v10.45.3, reverting to the previous version of realm-core?

[sync-by-unito]

➤ michael-wb commented:

Thanks for the update - the error happens any time the location update fails (e.g. no internet) when a realm is opened. One workaround is to capture the error and just ignore it, as described from a different ticket (copied below). A fix is in progress and I am currently adding tests to verify.

{quote}
If they keep using 10.45.3 (https://github.com/realm/realm-swift/releases/tag/v10.45.3)
A workaround on the meantime is to ignore connection errors while we work on a fix for this, there is already a Core ticket for it https://jira.mongodb.org/browse/RCORE-1982.
An example of the code
{quote}

app.syncManager.errorHandler = { syncError, syncSession in
   if let syncError = syncError as? SyncError {
      if syncError.code == .connectionFailed {
          return
      }
      self.error = syncError
   }
} 

{quote}
This will hide the error message while offline. The sync session will be resumed when the device goes online without the need of any extra code (Like previous behaviour).
{quote}

[gyratorycircus]

That doesn't appear to be our experience. No sync error is reported, and the application is killed from the realm sync thread.

[sync-by-unito]

➤ michael-wb commented:

Sorry for the inconvenience this has caused.
Are you able to tell if the crash is coming from this line? (fwiw - the handle_location_update_failed function this is part of is being removed as part of the fix):
https://github.com/realm/realm-core/blob/master/src/realm/object-store/sync/sync_session.cpp#L253
REALM_ASSERT(m_state == State::WaitingForAccessToken);

[gyratorycircus]

It seems likely. The next call in the stack trace is to realm::util::terminate, which would track with that REALM_ASSERT macro. Is it possible to disable assertions from the Realm Swift SDK (installed via CocoaPods)?

I'm also happy to rollback to v10.45.3 for a while if there wouldn't be any compatibility issues with apps that had already been running v10.47

[michael-wb]

Hi @gyratorycircus,
This has been updated in the Swift SDK v10.49.1 (based on Realm v14.4.1) to no longer throw the fatal error if the location update fails while offline. However, there is still an outstanding issue (realm/realm-core#7527) regarding @AutoOpen, where it will not exit the .connecting state if the location needs to be updated and the client is offline. There is a fix for this issue in review and will be merged shortly.

[michael-wb]

@gyratorycircus - Sorry for the delay, the Realm Swift v10.50.0 release includes a fix (in Realm Core PR #7528) that is part of Core v16.5.2 so the @AutoOpen class so it will fall through and open the local realm if the asynchronous sync/update is not able to successfully update the realm (e.g. currently offline).

Please reopen this ticket or file a new one if you find that it is still not working for you.