Requesting more memory than is currently allowed for backlink array due to memory allocation logic for links.

[Cliff-F]

SDK and version

SDK : Realm Swift for Mac
Version: 10.39.0 (realm-core-13.10.0)

Observations

• How frequent do the crash occur? Always
• Does it happen in production or during dev/test? in dev
• Can the crash be reproduced by you? Yes
• Can you provide instructions for how we can reproduce it?

Crash log / stacktrace

/Users/realm/workspace/realm_realm-core_release_13.10.0/src/realm/alloc_slab.cpp:211: [realm-core-13.10.0] Assertion failed: size < (1 << section_shift) with (size, get_file_path_for_assertions()) =  [67194816, "/Users/masa/Desktop/All Laws from eLaws Output.noindex/index.realm"]
0   Realm                               0x00000001053ecef0 _ZN5realm4utilL18terminate_internalERNSt3__118basic_stringstreamIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 28
1   Realm                               0x00000001053ece38 _ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE + 400
2   Realm                               0x000000010525328c _ZNK5realm9SlabAlloc28get_file_path_for_assertionsEv + 0
3   Realm                               0x0000000105253050 _ZN5realm9SlabAlloc8do_allocEm + 552
4   Realm                               0x0000000105255308 _ZN5realm9SlabAlloc10do_reallocEmPcmm + 76
5   Realm                               0x00000001053ae1dc _ZN5realm16WrappedAllocator10do_reallocEmPcmm + 32
6   Realm                               0x00000001052ffbd8 _ZN5realm4Node5allocEmm + 236
7   Realm                               0x0000000105267da4 _ZN5realm5Array5allocEmm + 96
8   Realm                               0x000000010525a420 _ZN5realm5Array6insertEmx + 352
9   Realm                               0x0000000105264244 _ZN5realm13ArrayBacklink3addEmNS_6ObjKeyE + 288
10  Realm                               0x000000010530e1a4 _ZN5realm3Obj12add_backlinkENS_6ColKeyENS_6ObjKeyE + 300
11  Realm                               0x000000010530e688 _ZNK5realm3Obj12set_backlinkENS_6ColKeyENS_7ObjLinkE + 444
12  Realm                               0x000000010530c6d8 _ZN5realm3Obj3setINS_6ObjKeyEEERS0_NS_6ColKeyET_b + 460
13  Realm                               0x0000000105117274 _ZN5realm12_GLOBAL__N_112ValueUpdaterIU8__strongP11objc_object18RLMAccessorContextEclEPNS_3ObjE + 300
14  Realm                               0x0000000105116024 _ZN5realm6Object23set_property_value_implIU8__strongP11objc_object18RLMAccessorContextEEvRT0_RKNS_8PropertyET_NS_12CreatePolicyEb + 964
15  Realm                               0x000000010510d500 _ZN5realm6Object6createIU8__strongP11objc_object18RLMAccessorContextEES0_RT0_RKNSt3__110shared_ptrINS_5RealmEEERKNS_12ObjectSchemaET_NS_12CreatePolicyENS_6ObjKeyEPNS_3ObjE + 1220
16  Realm                               0x000000010510ccc0 _ZN18RLMAccessorContext12createObjectEP11objc_objectN5realm12CreatePolicyEbNS2_6ObjKeyE + 776
17  Realm                               0x0000000105163098 RLMAddObjectToRealm + 192
18  eLawsDownloader                     0x0000000102b02fc4 $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKFyyKXEfU_ySayAA3TocCG_SayAA3RowCGs5Error_pSgtcfU0_yyXEfU_yyXEfU3_TA + 216
19  RealmSwift                          0x0000000103863a48 $s10RealmSwift0A0V5write16withoutNotifying_xSaySo20RLMNotificationTokenCG_xyKXEtKlF + 112
20  eLawsDownloader                     0x0000000102aec92c $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKFyyKXEfU_ySayAA3TocCG_SayAA3RowCGs5Error_pSgtcfU0_yyXEfU_ + 7132
21  eLawsDownloader                     0x0000000102b005c0 $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKFyyKXEfU_ySayAA3TocCG_SayAA3RowCGs5Error_pSgtcfU0_TA + 124
22  eLawsDownloader                     0x0000000102a68b2c $s15eLawsDownloader18ProcessSentenceXMLC6divide3xml11reconstruct8progress10completiony10Foundation4DataV_SbySScySayAA3TocCG_SayAA3RowCGs5Error_pSgtctFyAN_So25NSMutableAttributedStringCAStcfU_TA + 2064
23  eLawsDownloader                     0x0000000102a5b134 $s15eLawsDownloader18ProcessSentenceXMLC6parser_13didEndElement12namespaceURI13qualifiedNameySo11NSXMLParserC_S2SSgAJtFyyXEfU_ + 19512
24  eLawsDownloader                     0x0000000102a5c22c $s15eLawsDownloader18ProcessSentenceXMLC6parser_13didEndElement12namespaceURI13qualifiedNameySo11NSXMLParserC_S2SSgAJtFTo + 144
25  Foundation                          0x000000018e356190 _endElementNs + 660
26  libxml2.2.dylib                     0x0000000195156944 xmlParseEndTag2 + 668
27  libxml2.2.dylib                     0x000000019514d3a0 xmlParseTryOrFinish + 2848
28  libxml2.2.dylib                     0x000000019514bfc0 xmlParseChunk + 708
29  Foundation                          0x000000018e35517c -[NSXMLParser parseData:] + 120
30  Foundation                          0x000000018e355310 -[NSXMLParser parseData:] + 524
31  Foundation                          0x000000018e354ecc -[NSXMLParser parseFromStream] + 172
32  eLawsDownloader                     0x0000000102a55f9c $s15eLawsDownloader18ProcessSentenceXMLC5parse4data11reconstruct10completiony10Foundation4DataV_SbySayAA3TocCG_So25NSMutableAttributedStringCs5Error_pSgtctF + 1816
33  eLawsDownloader                     0x0000000102a53580 $s15eLawsDownloader18ProcessSentenceXMLC6divide3xml11reconstruct8progress10completiony10Foundation4DataV_SbySScySayAA3TocCG_SayAA3RowCGs5Error_pSgtctF + 292
34  eLawsDownloader                     0x0000000102aeacf8 $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKF + 700
35  eLawsDownloader                     0x0000000102aea4d0 $s15eLawsDownloader11AppDelegateC13buildLinksSubyyFyycfU_ySiXEfU_yyXEfU_ + 240
36  eLawsDownloader                     0x0000000102b03118 $s15eLawsDownloader11AppDelegateC13buildLinksSubyyFyycfU_ySiXEfU_TA + 60
37  libswiftDispatch.dylib              0x00000001a05c2280 $sSiIgy_SiIegy_TRTA + 28
38  libswiftDispatch.dylib              0x00000001a05c22ac $sSiIegy_SiIyBy_TR + 32
39  libdispatch.dylib                   0x0000000103206a14 _dispatch_client_callout2 + 20
40  libdispatch.dylib                   0x0000000103221bf4 _dispatch_apply_invoke + 252
41  libdispatch.dylib                   0x00000001032069d4 _dispatch_client_callout + 20
42  libdispatch.dylib                   0x000000010321eb88 _dispatch_root_queue_drain + 996
43  libdispatch.dylib                   0x000000010321f560 _dispatch_worker_thread2 + 196
44  libsystem_pthread.dylib             0x0000000102e0bcec _pthread_wqthread + 228
45  libsystem_pthread.dylib             0x0000000102e13a7c start_wqthread + 8
!!! IMPORTANT: Please report this at https://github.com/realm/realm-core/issues/new/choose2023-05-05 15:56:02.742388+0900 eLawsDownloader[19936:8796200] /Users/realm/workspace/realm_realm-core_release_13.10.0/src/realm/alloc_slab.cpp:211: [realm-core-13.10.0] Assertion failed: size < (1 << section_shift) with (size, get_file_path_for_assertions()) =  [67194816, "/Users/masa/Desktop/All Laws from eLaws Output.noindex/index.realm"]
0   Realm                               0x00000001053ecef0 _ZN5realm4utilL18terminate_internalERNSt3__118basic_stringstreamIcNS1_11char_traitsIcEENS1_9allocatorIcEEEE + 28
1   Realm                               0x00000001053ece38 _ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE + 400
2   Realm                               0x000000010525328c _ZNK5realm9SlabAlloc28get_file_path_for_assertionsEv + 0
3   Realm                               0x0000000105253050 _ZN5realm9SlabAlloc8do_allocEm + 552
4   Realm                               0x0000000105255308 _ZN5realm9SlabAlloc10do_reallocEmPcmm + 76
5   Realm                               0x00000001053ae1dc _ZN5realm16WrappedAllocator10do_reallocEmPcmm + 32
6   Realm                               0x00000001052ffbd8 _ZN5realm4Node5allocEmm + 236
7   Realm                               0x0000000105267da4 _ZN5realm5Array5allocEmm + 96
8   Realm                               0x000000010525a420 _ZN5realm5Array6insertEmx + 352
9   Realm                               0x0000000105264244 _ZN5realm13ArrayBacklink3addEmNS_6ObjKeyE + 288
10  Realm                               0x000000010530e1a4 _ZN5realm3Obj12add_backlinkENS_6ColKeyENS_6ObjKeyE + 300
11  Realm                               0x000000010530e688 _ZNK5realm3Obj12set_backlinkENS_6ColKeyENS_7ObjLinkE + 444
12  Realm                               0x000000010530c6d8 _ZN5realm3Obj3setINS_6ObjKeyEEERS0_NS_6ColKeyET_b + 460
13  Realm                               0x0000000105117274 _ZN5realm12_GLOBAL__N_112ValueUpdaterIU8__strongP11objc_object18RLMAccessorContextEclEPNS_3ObjE + 300
14  Realm                               0x0000000105116024 _ZN5realm6Object23set_property_value_implIU8__strongP11objc_object18RLMAccessorContextEEvRT0_RKNS_8PropertyET_NS_12CreatePolicyEb + 964
15  Realm                               0x000000010510d500 _ZN5realm6Object6createIU8__strongP11objc_object18RLMAccessorContextEES0_RT0_RKNSt3__110shared_ptrINS_5RealmEEERKNS_12ObjectSchemaET_NS_12CreatePolicyENS_6ObjKeyEPNS_3ObjE + 1220
16  Realm                               0x000000010510ccc0 _ZN18RLMAccessorContext12createObjectEP11objc_objectN5realm12CreatePolicyEbNS2_6ObjKeyE + 776
17  Realm                               0x0000000105163098 RLMAddObjectToRealm + 192
18  eLawsDownloader                     0x0000000102b02fc4 $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKFyyKXEfU_ySayAA3TocCG_SayAA3RowCGs5Error_pSgtcfU0_yyXEfU_yyXEfU3_TA + 216
19  RealmSwift                          0x0000000103863a48 $s10RealmSwift0A0V5write16withoutNotifying_xSaySo20RLMNotificationTokenCG_xyKXEtKlF + 112
20  eLawsDownloader                     0x0000000102aec92c $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKFyyKXEfU_ySayAA3TocCG_SayAA3RowCGs5Error_pSgtcfU0_yyXEfU_ + 7132
21  eLawsDownloader                     0x0000000102b005c0 $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKFyyKXEfU_ySayAA3TocCG_SayAA3RowCGs5Error_pSgtcfU0_TA + 124
22  eLawsDownloader                     0x0000000102a68b2c $s15eLawsDownloader18ProcessSentenceXMLC6divide3xml11reconstruct8progress10completiony10Foundation4DataV_SbySScySayAA3TocCG_SayAA3RowCGs5Error_pSgtctFyAN_So25NSMutableAttributedStringCAStcfU_TA + 2064
23  eLawsDownloader                     0x0000000102a5b134 $s15eLawsDownloader18ProcessSentenceXMLC6parser_13didEndElement12namespaceURI13qualifiedNameySo11NSXMLParserC_S2SSgAJtFyyXEfU_ + 19512
24  eLawsDownloader                     0x0000000102a5c22c $s15eLawsDownloader18ProcessSentenceXMLC6parser_13didEndElement12namespaceURI13qualifiedNameySo11NSXMLParserC_S2SSgAJtFTo + 144
25  Foundation                          0x000000018e356190 _endElementNs + 660
26  libxml2.2.dylib                     0x0000000195156944 xmlParseEndTag2 + 668
27  libxml2.2.dylib                     0x000000019514d3a0 xmlParseTryOrFinish + 2848
28  libxml2.2.dylib                     0x000000019514bfc0 xmlParseChunk + 708
29  Foundation                          0x000000018e35517c -[NSXMLParser parseData:] + 120
30  Foundation                          0x000000018e355310 -[NSXMLParser parseData:] + 524
31  Foundation                          0x000000018e354ecc -[NSXMLParser parseFromStream] + 172
32  eLawsDownloader                     0x0000000102a55f9c $s15eLawsDownloader18ProcessSentenceXMLC5parse4data11reconstruct10completiony10Foundation4DataV_SbySayAA3TocCG_So25NSMutableAttributedStringCs5Error_pSgtctF + 1816
33  eLawsDownloader                     0x0000000102a53580 $s15eLawsDownloader18ProcessSentenceXMLC6divide3xml11reconstruct8progress10completiony10Foundation4DataV_SbySScySayAA3TocCG_SayAA3RowCGs5Error_pSgtctF + 292
34  eLawsDownloader                     0x0000000102aeacf8 $s15eLawsDownloader11AppDelegateC12extractLinks5lawNo0G4Name0G7Edition2at_ySS_S2S10Foundation3URLVSitKF + 700
35  eLawsDownloader                     0x0000000102aea4d0 $s15eLawsDownloader11AppDelegateC13buildLinksSubyyFyycfU_ySiXEfU_yyXEfU_ + 240
36  eLawsDownloader                     0x0000000102b03118 $s15eLawsDownloader11AppDelegateC13buildLinksSubyyFyycfU_ySiXEfU_TA + 60
37  libswiftDispatch.dylib              0x00000001a05c2280 $sSiIgy_SiIegy_TRTA + 28
38  libswiftDispatch.dylib              0x00000001a05c22ac $sSiIegy_SiIyBy_TR + 32
39  libdispatch.dylib                   0x0000000103206a14 _dispatch_client_callout2 + 20
40  libdispatch.dylib                   0x0000000103221bf4 _dispatch_apply_invoke + 252
41  libdispatch.dylib                   0x00000001032069d4 _dispatch_client_callout + 20
42  libdispatch.dylib                   0x000000010321eb88 _dispatch_root_queue_drain + 996
43  libdispatch.dylib                   0x000000010321f560 _dispatch_worker_thread2 + 196
44  libsystem_pthread.dylib             0x0000000102e0bcec _pthread_wqthread + 228
45  libsystem_pthread.dylib             0x0000000102e13a7c start_wqthread + 8
!!! IMPORTANT: Please report this at https://github.com/realm/realm-core/issues/new/choose

Steps & Code to Reproduce

I posted #6531. This may be related or may be not. This crash occurs in a different situation.

This program for Mac that we are developing reads 60,000+ local xml files, analyzes them, and builds a realm database that will have 20,000,000+ records. The program continuously reads/writes the database from multiple threads at almost 100% cpu usage for 5-6 hours. After several hours when 80% of the tasks have finished, the program crashes atrealm.add(obj). The memory usage is modest (500MB-2GB, running on 16GB MacBook Pro). The realm file is not broken. All the records we are trying to add to the database are not ususual. (after the crash, we can continue analyzing and adding the remaining records to the database when restarted.)

[Cliff-F]

After some investigations I found out in this code realmTestMain() crashes 100%

public class EGovLaw: Object {
  @objc dynamic public var title: String!
}

final public class EGovLawLink: Object {
  @objc dynamic public var law: EGovLaw!
}

func realmTestMain() {
  let dbUrl = FileManager.default.urls(for: .documentDirectory,  in: .userDomainMask)[0]
  
  let config = Realm.Configuration(fileURL: dbUrl.appendingPathComponent("index.realm"), readOnly: false, objectTypes:[EGovLaw.self, EGovLawLink.self])
  let realm = try! Realm(configuration: config)
  
  try! realm.write() {
    let law = EGovLaw()
    realm.add(law)
  }
    
  let debugCount = 30_000_000
  var totalCount = 0
  DispatchQueue.concurrentPerform(iterations: debugCount)  { (i:size_t) in
    autoreleasepool {
      
      let realm = try! Realm(configuration: config)
      let law = realm.objects(EGovLaw.self).first
      var linksToAdd: [EGovLawLink] = []
      let loopCount = 50_000
      
      for _ in 0..<loopCount {
        let link = EGovLawLink()
        link.law = law
        linksToAdd.append(link)
      }
      
      try! realm.write() {
        totalCount += loopCount
        print("[\(totalCount)]")
        
        linksToAdd.forEach {
          realm.add($0) // Crashes at 8750000
        }
      }
    }
  }
}

[nicola-cab]

Hello, thanks for reporting this, this issue is definitely very similar to #6531 ... I haven't found a way to reproduce it yet, thus we still don't have a fix for it.
However, looking a both core dumps that you posted, it seems that either you have 2 threads that are running the same code (this should not be possible because of our locking logic) or the same stack trace was pasted twice.
Can you please confirm this?
I am going to use the snipped provided in order to see if I can reproduce the same issue and eventually find the root cause.

[Cliff-F]

It's not pasted twice. This appeares on the console.

[nicola-cab]

OK, this should not be possible. I hope to be able to reproduce it and fix it soon.

[ironage]

I was able to reproduce this in a unit test. We are requesting more memory than is currently allowed in a single array due to the constraint of having to make it fit in a single mapping. The array growing is the backlink array with size of 8388608, and width of 32. The DB is around 100Mb. The orig_capacity_bytes was 66800160 and Node::alloc tries to double that which gives the request for 133600320 which hits the assertion. We don't actually need that much memory though so I don't think we should be doubling array capacity for reallocations of an already large chunk. With unrestricted growth of a single array we will eventually hit capacity again unless we redesign backlinks to use a b-tree or something else, but a simple fix for now could be to only request the exact amount of memory required past some threshold (say 2Mb). Passing to @finnschiermer for further analysis.

ONLY(Transactions_ThreadedTransactionsWithLinks)
{
    SHARED_GROUP_TEST_PATH(path);
    {
        std::unique_ptr<Replication> hist_w(make_in_realm_history());
        DBRef db = DB::create(*hist_w, path);
        {
            auto wt = db->start_write();
            auto top_table = wt->add_table("class_EGovLaw");
            top_table->add_column(type_String, "title");
            auto linked_table = wt->add_table("class_EGovLawLink");
            linked_table->add_column(*top_table, "law");
            wt->commit();
        }

        {
            auto wt = db->start_write();
            TableRef table = wt->get_table("class_EGovLaw");
            table->create_object();
            wt->commit();
        }
    }
    const int num_threads = 500;
    std::thread writers[num_threads];
    for (int i = 0; i < num_threads; ++i) {
        writers[i] = std::thread([&] {
            std::unique_ptr<Replication> hist_w(make_in_realm_history());
            DBRef db = DB::create(*hist_w, path);
            auto rt = db->start_read();
            TableRef table = rt->get_table("class_EGovLaw");
            auto first_law = *table->begin();

            auto wt = db->start_write();
            TableRef link_table = wt->get_table("class_EGovLawLink");
            ColKey link_col = link_table->get_column_key("law");
            for (size_t i = 0; i < 50'000; ++i) {
                link_table->create_object().set(link_col, first_law.get_key());
            }
            wt->commit();
        });
    }
    for (int i = 0; i < num_threads; ++i) {
        writers[i].join();
    }
}

[sync-by-unito]

➤ jedelbo commented:

I will have a look at this.

[jedelbo]

@Cliff-F unfortunately we have a limit in our implementation that limits the number of backlinks to a given object to 8388606. This means that you can only have so many objects linking to one specific object. We can change the implementation so that we don't have this limit, but it will be a file format breaking change, and we don't want to create too many of those. So I cannot promise that this can be done soon enough to solve your problem.

Apart from the fact that this actually makes your program crash, it must also cause some performance issues. Could you perhaps change your design so that you instead of linking to that object stored an id for the object?

[jedelbo]

Fixed by #6673