Panic when force killing the process

[atlas-comstock]

2022/05/19 06:15:55 opening badger with options {Dir:/data/logs ValueDir:/data/logs SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0x4000b0c020 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:65536 LevelSizeMultiplier:3 TableSizeMultiplier:2 MaxLevels:25 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:268435456 IndexCacheSize:0 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:5000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:1 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
panic: while opening memtables error: while opening fid: 2 error: while updating skiplist error: mremap size mismatch: requested: 20 got: 134217728

goroutine 1 [running]:
github.com/kelindar/talaria/internal/storage/disk.Open(0x4000fa0143, 0x5, 0x1d2b516, 0x4, 0x2006498, 0x4000b5c0c0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/talaria/internal/storage/disk/disk.go:55 +0x114
github.com/kelindar/talaria/internal/table/log.New(0x4000b0c390, 0x1fdef80, 0x400039c1b0, 0x2006498, 0x4000b5c0c0, 0x4000ad4020)
        /go/src/talaria/internal/table/log/log.go:42 +0xc0
main.main()
        /go/src/talaria/main.go:54 +0x5c8

[atlas-comstock]

2022/05/19 06:55:22 opening badger with options {Dir:/data/logs ValueDir:/data/logs SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0x4000bcc030 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:65536 LevelSizeMultiplier:3 TableSizeMultiplier:2 MaxLevels:25 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:268435456 IndexCacheSize:0 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:5000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:1 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
2022/05/19 06:55:22 [error] ServerError: target=storage/writer/writer.go.173, reason=Internal Server Error, msg=stream: writer was not configured
2022/05/19 06:55:22 [error] ServerError: target=table/timeseries/timeseries.go.124, reason=Internal Server Error, msg=error reading from uri  due to scheme  is not supported
2022/05/19 06:55:22 server: opening table eventlog...
2022/05/19 06:55:22 opening badger with options {Dir:/data/eventlog ValueDir:/data/eventlog SyncWrites:false NumVersionsToKeep:1 ReadOnly:false Logger:0x4000bccf50 Compression:1 InMemory:false MetricsEnabled:true NumGoroutines:8 MemTableSize:67108864 BaseTableSize:2097152 BaseLevelSize:65536 LevelSizeMultiplier:3 TableSizeMultiplier:2 MaxLevels:25 VLogPercentile:0 ValueThreshold:1048576 NumMemtables:5 BlockSize:4096 BloomFalsePositive:0.01 BlockCacheSize:268435456 IndexCacheSize:0 NumLevelZeroTables:5 NumLevelZeroTablesStall:15 ValueLogFileSize:1073741823 ValueLogMaxEntries:5000 NumCompactors:4 CompactL0OnClose:false LmaxCompaction:false ZSTDCompressionLevel:1 VerifyValueChecksum:false EncryptionKey:[] EncryptionKeyRotationDuration:240h0m0s BypassLockGuard:false ChecksumVerificationMode:0 DetectConflicts:true NamespaceOffset:-1 managedTxns:false maxBatchCount:0 maxBatchSize:0 maxValueThreshold:0}
panic: while opening memtables error: while opening fid: 273 error: while updating skiplist error: mremap size mismatch: requested: 67115867 got: 134217728

goroutine 1 [running]:
github.com/kelindar/talaria/internal/storage/disk.Open(0x400049e133, 0x5, 0x400049e158, 0x8, 0x2006498, 0x4000c08980, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/talaria/internal/storage/disk/disk.go:55 +0x114
main.openTable(0x400049e158, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x400049e147, 0x7, ...)
        /go/src/talaria/main.go:103 +0xf8
main.main()
        /go/src/talaria/main.go:68 +0xaec

Another error

[tardunge]

Can you provide more information on how to reproduce this?
I've tried Ctrl-C to interrupt the running process and also kill -p $PID, but i've not encountered this error.

[atlas-comstock]

@tardunge hi, do you have data ingested already and amended to the DB? I am afraid that this is necessary/

[atlas-comstock]

@tardunge Hi, I found the root cause: https://discuss.dgraph.io/t/error-mremap-size-mismatch-on-arm64/15333

[atlas-comstock]

dgraph-io/ristretto#281 Fix the issue.

[atlas-comstock]

The solution requires:

• Fork https://github.com/dgraph-io/ristretto/ and make a release
• Fork badger, import our fork ristretto and make a release
• Talaria update badger dep

[atlas-comstock]

This issue #97 maybe precisely has RC too:
2022/06/03 07:11:57 [error] ServerError: target=storage/disk/disk.go.160, reason=Internal Server Error, msg=unable to append due to Unable to truncate file: "/data/eventlog/000001.vlog" error: mremap size mismatch: requested: 1074437932 got: 2147483646

Same mremap size mismatch issue

[tardunge]

@tardunge Hi, I found the root cause: https://discuss.dgraph.io/t/error-mremap-size-mismatch-on-arm64/15333

[atlas-comstock]

Fix in #98