OOM crash when performing search on large input file

[766F6964]

Summary

Opening a large text file, ~2GB with helix works perfectly fine.
However, once I try to initiate a search using /, helix freezes and then gets killed by system-D with a out-of-memory (OOM)
message. The file I tested is 2.35GB and my system has 16GB of memory. It can open it just fine, I don't see how the search should cause a OOM crash.

Reproduction Steps

1. Download a large text file. The file I used, that triggered the crash was the 2.35GB word list file from the kaonashi project. It can be found here: https://github.com/kaonashi-passwords/Kaonashi/tree/master/wordlists
2. After downloading the file, extract the archive to obtain the .txt file
3. Open it with Helix: helix kaonashi.txt (This will take a few seconds, but it opens)
4. Press / to trigger a search. Notice that helix freezes and eventually gets killed by system-D.

Helix log

The helix.log file does not contain any useful information, since the crash doesn't happen on an application level, but on a system level. The linux kernel OOM logic terminates the process. Regardless, here is the log:

2023-09-15T01:42:13.458 helix_view::clipboard::provider [DEBUG] No native clipboard provider found. Yanking by OSC 52 and pasting will be internal to Helix
2023-09-15T01:42:23.388 helix_vcs [INFO] Error {
    context: "failed to open git repo",
    source: Discover(
        NoGitRepository {
            path: "/home/void/Downloads",
        },
    ),
}
2023-09-15T01:42:23.388 helix_vcs [INFO] failed to open diff base for /home/void/Downloads/kaonashi.txt
2023-09-15T01:42:23.388 helix_vcs [INFO] Error {
    context: "failed to open git repo",
    source: Discover(
        NoGitRepository {
            path: "/home/void/Downloads",
        },
    ),
}
2023-09-15T01:42:23.388 helix_vcs [INFO] failed to obtain current head name for /home/void/Downloads/kaonashi.txt
2023-09-15T01:42:23.389 helix_view::editor [DEBUG] editor status: Loaded 1 file.
2023-09-15T01:42:23.400 helix_tui::backend::crossterm [DEBUG] The keyboard enhancement protocol is not supported in this terminal (checked in 10.353824ms)
2023-09-15T01:42:23.401 helix_view::document [DEBUG] id 1 modified - last saved: 0, current: 0
2023-09-15T01:42:23.402 helix_term::application [DEBUG] received editor event: IdleTimer

What is more interesting is the system-D log, which gives the following info:

Sep 15 01:37:21 arch kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker-d162cc2af8da3dbd26b7736e113b93865e4e02de71046d76d2acffd161b8bbc1.scope,mems_allowed=0,global_oom,task_mem>
Sep 15 01:37:21 arch kernel: Out of memory: Killed process 43377 (helix) total-vm:13630608kB, anon-rss:12671780kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:24960kB oom_score_adj:200
Sep 15 01:37:21 arch systemd[1]: [email protected]: A process of this unit has been killed by the OOM killer.
░░ Subject: A process of [email protected] unit has been killed by the OOM killer.
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A process of unit @UNIT has been killed by the Linux kernel out-of-memory (OOM)
░░ killer logic. This usually indicates that the system is low on memory and that
░░ memory needed to be freed. A process associated with [email protected] has been determined
░░ as the best process to terminate and has been forcibly terminated by the
░░ kernel.
░░ 
░░ Note that the memory pressure might or might not have been caused by [email protected].
Sep 15 01:37:21 arch systemd[1973]: vte-spawn-1921d522-db42-4911-b9bc-31fc5d3f3905.scope: A process of this unit has been killed by the OOM killer.
░░ Subject: A process of UNIT unit has been killed by the OOM killer.
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A process of unit @UNIT has been killed by the Linux kernel out-of-memory (OOM)
░░ killer logic. This usually indicates that the system is low on memory and that
░░ memory needed to be freed. A process associated with UNIT has been determined
░░ as the best process to terminate and has been forcibly terminated by the
░░ kernel.
░░ 
░░ Note that the memory pressure might or might not have been caused by UNIT.
Sep 15 01:37:21 arch systemd[1973]: vte-spawn-1921d522-db42-4911-b9bc-31fc5d3f3905.scope: Failed with result 'oom-kill'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit UNIT has entered the 'failed' state with result 'oom-kill'.
Sep 15 01:37:21 arch systemd[1973]: vte-spawn-1921d522-db42-4911-b9bc-31fc5d3f3905.scope: Consumed 11.006s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit UNIT completed and consumed the indicated resources.

Platform

Arch Linux

Terminal Emulator

xterm-256color

Helix Version

helix 23.05

[pascalkuthe]

you missread something here. The file you linked is 2.35GB large when compressed.

its 10GB large uncompressed. Helix currently must copy the entire document contents when starting a regex search as the rust regex engine can only operate on contiguous memory. That means the memory requirement well be at least 20GB (which obviously leads to OOM on your system, I have 64GB and helix works fine with 20GB memory consumption).

We do want to avoid the clone eventually but that requires partially rewriting the regex engine. i have made some headway a while ago on that but its not done yet.

This is a duplicate of #185