Skip to content

Run zip import in a transaction.#3031

Closed
losfair wants to merge 1 commit intoTriliumNext:masterfrom
losfair:import-zip-in-txn
Closed

Run zip import in a transaction.#3031
losfair wants to merge 1 commit intoTriliumNext:masterfrom
losfair:import-zip-in-txn

Conversation

@losfair
Copy link

@losfair losfair commented Aug 1, 2022

On SQLite VFSes where transaction start and commit are high-latency operations, this improves ZIP import performance a lot. Should help under the default settings too, as transaction commit involves at least one fsync.

With mvsqlite this reduces the time for importing demo.zip from tens of seconds to ~0.6sec.

Related: #3029

@zadam
Copy link
Contributor

zadam commented Aug 1, 2022

Hi, in general having transactional import would be beneficial. But one significant problem is that this approach forces the import mechanism to first read the whole ZIP into memory and removes the streaming aspect from the equation. This will cause problems when importing large ZIPs.

@zDEFz
Copy link

zDEFz commented Aug 3, 2022

Hi, in general having transactional import would be beneficial. But one significant problem is that this approach forces the import mechanism to first read the whole ZIP into memory and removes the streaming aspect from the equation. This will cause problems when importing large ZIPs.

Can't you just add more swap then or memory?

@eliandoran eliandoran deleted the branch TriliumNext:master June 21, 2025 14:45
@eliandoran eliandoran closed this Jun 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants