Beets uses all available memory during import and then crashes

[hjmuller]

Problem

Beets program crashes after all memory appears to be consumed. I noticed that as an import continues, my memory usage increases. Once all system memory is consumed, beets crashes (see end of verbose output below). If I allocate more memory, then it only prolongs the crash. Doesn't seem to be related to specific files. Disabling all plugins seems to prevent the memory consumption and crash from happening.

Running this command in verbose (-vv) mode:

$ beet -vv import -qp /mnt/music/

Led to this problem:

scrub: auto-scrubbing /mnt/downloads/new_music_collection/Avantasia/The Mystery of Time_ A Rock Epic [2013]/10 - The Great Mystery.mp3
scrub: writing new tags after scrub
Sending event: write
Sending event: after_write
scrub: restoring art
scrub: auto-scrubbing /mnt/downloads/new_music_collection/Avantasia/The Mystery of Time_ A Rock Epic [2013]/07 - Invoke the Machine.mp3
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/04 - Carry Me Over.mp3'
chroma: no match found
scrub: writing new tags after scrub
Sending event: write
Sending event: after_write
scrub: restoring art
Sending event: album_imported
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/05 - What Kind of Love.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/06 - Another Angel Down.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/07 - The Toy Master.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/08 - Devil in the Belfry.mp3'
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/09 - Cry Just a Little.mp3'
chroma: no match found
chroma: chroma: fingerprinted "/mnt/music/Avantasia/The Scarecrow [2008]/10 - I Don't Believe in Your Love.mp3"
chroma: no match found
chroma: chroma: fingerprinted '/mnt/music/Avantasia/The Scarecrow [2008]/11 - Lost in Space.mp3'
chroma: no match found
Looking up: /mnt/music/Avantasia/The Scarecrow [2008]
Tagging Avantasia - The Scarecrow
Searching for discovered album ID: f72ea7dc-5316-36ac-9833-2476d88b62e6
Sending event: albuminfo_received
Candidate: Avantasia - The Scarecrow
Success. Distance: 0.00
Album ID match recommendation is Recommendation.strong
ID match.
Sending event: import_task_start

/mnt/music/Avantasia/The Scarecrow [2008] (11 items)
Tagging:
    Avantasia - The Scarecrow
URL:
    http://musicbrainz.org/release/f72ea7dc-5316-36ac-9833-2476d88b62e6
(Similarity: 100.0%) (CD, 2008, BR, Nuclear Blast)
Sending event: import_task_choice
Sending event: import_task_apply
Killed

Setup

• OS: Debian GNU/Linux 9.6 (stretch)
• Python version: Python 2.7.13
• beets version: 1.3.18
• Turning off plugins made problem go away (yes/no): yes

My configuration (output of beet config) is:

## beets config file
## See https://beets.readthedocs.io for more info

directory: /mnt/downloads/new_music_collection
library: ~/.config/beets/library.db
threaded: yes
ui:
    color: yes
    colors:
        text_success: green
        text_warning: yellow
        text_error: red
        text_highlight: red
        text_highlight_minor: lightgray
        action_default: turquoise
        action: blue
import:
    write: yes ## Either yes or no, controlling whether metadata (e.g., ID3) tags are written to files when using beet import. Defaults to yes.
#    copy: no ## Either yes or no, indicating whether to copy files into the library directory when using beet import. Defaults to yes.
#    move: yes ## Either yes or no, indicating whether to move files into the library directory when using beet import. Defaults to no.
    resume: ask ## Controls whether interrupted imports should be resumed.
    log: ~/.config/beets/log.txt
    quiet_fallback: skip
    duplicate_action: skip

match:
    strong_rec_thresh: 0.02

paths:
    default: $albumartist/$album [$year]/$track - $title
    singleton: Non-Album/$artist/$title
    comp: Compilations/$album [$year]/$track - $title
    albumtype:soundtrack: Soundtracks/$album [$year]/$track - $title

plugins:
    chroma
    fromfilename
    scrub
    info
    missing
    duplicates

[sampsyo]

Hi! Is it true, as your issue form implied, that this occurs the same way even if you disable all the plugins?

[hjmuller]

Hi! Is it true, as your issue form implied, that this occurs the same way even if you disable all the plugins?

I thought I had disabled them and still had the issue, but I tried again with disabled plugins and the crash/memory consumption doesn't occur.

I'm not sure which plugin is the culprit.

[sampsyo]

OK! Please post again if you can narrow it down.

[hjmuller]

Looks like the chroma plugin is the problem. I re-enabled only that plugin, ran the import again, and the issue occurred. All memory was consumed and then beets crashed.

[sampsyo]

Got it; thanks! Any chance you've got GStreamer (and its Python bindings) installed? It's probably the same memory leak reported here: beetbox/audioread#62

[hjmuller]

Got it; thanks! Any chance you've got GStreamer (and its Python bindings) installed? It's probably the same memory leak reported here: beetbox/audioread#62

I have a few gstreamer packages installed. Is there a recommended workaround?

$ dpkg-query -l | grep gstreamer
ii  gir1.2-gstreamer-1.0                 1.10.4-1                       amd64        GObject introspection data for the GStreamer library
ii  gstreamer1.0-plugins-base:amd64      1.10.4-1                       amd64        GStreamer plugins from the "base" set
ii  libgstreamer-plugins-base1.0-0:amd64 1.10.4-1                       amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.0-0:amd64              1.10.4-1                       amd64        Core GStreamer libraries and elements

[sampsyo]

Well, if it is in fact that very memory leak, one thing you could do is uninstall the Python bindings for GStreamer (more specifically, the pygobject library). Or, if you're feeling like it, maybe you can help us track down what's causing the leak, as the aforelinked thread discusses!

[motorific]

I also have this issue, running in a docker container. I'm happy to do testing that could assist with my limited knowledge of programming.

I have to abort the process when the memory gets high and then restart. It's corrupted the database a few times and i've had to start from scratch.

I am using the linuxserver version of the container.

[sampsyo]

@motorific, a great place to start would be to make sure this can be blamed on the chroma plugin. If so, the next place to look would be to investigate which audio-reading backend is being used—as mentioned above, to try uninstalling GStreamer, for example, to see if it's possible to blame that.

[motorific]

@sampsyo It's for sure chroma.

I added this to my config.yaml:

chroma:
    auto: no

and there are no longer memory leak issues. Please let me know if i can provide other information that would be useful.

[sampsyo]

Yo! Can you try uninstalling GStreamer and seeing what happens?

[ssssam]

I suspect this is the issue: beetbox/audioread#84

[dustinmhorvath]

Juts wanted to point out that I've been running into this issue with the beets:latest container as well from dockerhub. This is running on a Centos7 vm with Docker Swarm, alongside 11 other containers. 4GB ram and 2 cores.

Disabling the chroma plugin seems to resolve the issue as well for me. This has been such a plague of an issue, and finding this thread was difficult enough that I thought it worth commenting here.

If you guys need any other host info or logs that could be helpful, I'll do what I can.

[ssssam]

If you guys need any other host info or logs that could be helpful, I'll do what I can.

Thanks!
If you install the latest (as yet unreleased) version of audioread, do you still see the issue? Try running pip3 install --upgrade git+https://github.com/beetbox/audioread/

[tryfan]

@ssssam I just tried this and it works great. Memory was under 105MB during an import, where previously after importing several albums it would be over 2GB. I'm using a docker container and the only modification I made was to install that audioread package.

[lpcvoid]

In case anybody getting here via google needs to do what @ncelebic said, here's the command :

sudo docker exec -u abc -it [CONTAINERNAME] pip3 install --user --upgrade https://github.com/beetbox/audioread/archive/master.zip

[CONTAINERNAME] is, well, your container name.

Cheers

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[wisp3rwind]

Reading the comments, this seems to be fixed by updating audioread.