Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runs out of memory while doing local scan of big library on Raspberry PI Zero W #22

Open
Caenhorn opened this issue Nov 30, 2017 · 21 comments
Labels
A-scan Area: Media scanning C-bug Category: This is a bug

Comments

@Caenhorn
Copy link

Last output:

INFO     Scanned 18700 of 26522 files in 12201s, ~5103s left.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/DIRECTSO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FILEIO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FLASHASS.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTXTRA.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/INETURL.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MACROMIX.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MIXSERVI.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MPEG3IMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MUIDIALO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETFILE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETLINGO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/QT3ASSET.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDCON.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDIMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACNVRT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWADCMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAIMPOR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAOPT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWASTRM.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTXTRA.X32: No audio found in file.
INFO     Scanned 18800 of 26522 files in 12234s, ~5025s left.

(python:2337): GLib-ERROR **: /build/glib2.0-F5w919/glib2.0-2.50.3/./glib/gmem.c:100: failed to allocate 11017050 bytes
Trace/breakpoint trap

By the way, log file didn't catch that for some reason. Last line in log file is last progress report on scanning.

I'm running:

pi@mopidy-zero:/var/log/mopidy $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.1 (stretch)
Release:        9.1
Codename:       stretch
pi@mopidy-zero:/var/log/mopidy $ uname -a
Linux mopidy-zero 4.9.59+ mopidy/mopidy#1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux

pi@mopidy-zero:/var/log/mopidy $ dpkg -s mopidy
Package: mopidy
Status: install ok installed
Priority: optional
Section: sound
Installed-Size: 853
Maintainer: Stein Magnus Jodal <[email protected]>
Architecture: all
Version: 2.1.0-1
@Caenhorn Caenhorn changed the title Runs out of memory while doing local scan of big library on Raspberry Zero W Runs out of memory while doing local scan of big library on Raspberry PI Zero W Nov 30, 2017
@Caenhorn
Copy link
Author

Effective configuration:

pi@mopidy-zero:/var/log/mopidy $ sudo mopidyctl config
Running "/usr/bin/mopidy --config /usr/share/mopidy/conf.d:/etc/mopidy/mopidy.conf config" as user mopidy
[core]
cache_dir = /var/cache/mopidy
config_dir = /etc/mopidy
data_dir = /var/lib/mopidy
max_tracklist_length = 10000
restore_state = false

[logging]
color = true
console_format = %(levelname)-8s %(message)s
debug_format = %(levelname)-8s %(asctime)s [%(process)d:%(threadName)s] %(name)s\n  %(message)s
debug_file = /var/log/mopidy/mopidy-debug.log
config_file = /etc/mopidy/logging.conf

[audio]
mixer = software
mixer_volume =
output = autoaudiosink
buffer_time =

[proxy]
scheme =
hostname =
port =
username =
password =

[moped]
enabled = true

[local-images]
enabled = true
library = json
base_uri = /images/
image_dir =
album_art_files =
  *.jpg
  *.jpeg
  *.png

[iris]
enabled = true
country = NZ
locale = en_NZ
spotify_authorization_url = https://jamesbarnsley.co.nz/auth_spotify.php
lastfm_authorization_url = https://jamesbarnsley.co.nz/auth_lastfm.php

[mpd]
enabled = true
hostname = ::
port = 6600
password =
max_connections = 20
connection_timeout = 60
zeroconf = Mopidy MPD server on $hostname
command_blacklist =
  listall
  listallinfo
default_playlist_scheme = m3u

[http]
enabled = true
hostname = ::
port = 6680
static_dir =
zeroconf = Mopidy HTTP server on $hostname

[stream]
enabled = true
protocols =
  http
  https
  mms
  rtmp
  rtmps
  rtsp
metadata_blacklist =
timeout = 5000

[m3u]
enabled = true
base_dir =
default_encoding = latin-1
default_extension = .m3u8
playlists_dir = /var/lib/mopidy/playlists

[softwaremixer]
enabled = true

[file]
enabled = true
media_dirs =
  $XDG_MUSIC_DIR|Music
  ~/|Home
excluded_file_extensions =
  .jpg
  .jpeg
show_dotfiles = false
follow_symlinks = false
metadata_timeout = 1000

[local]
enabled = true
library = json
media_dir = /mnt/music
scan_timeout = 15000
scan_flush_threshold = 100
scan_follow_symlinks = false
excluded_file_extensions =
  .directory
  .html
  .jpeg
  .jpg
  .log
  .nfo
  .png
  .txt
  .cue

@kingosticks
Copy link
Member

Have you tried with mopidy-local-sqlite? That is the preferred local library to use.

@Caenhorn
Copy link
Author

Thanks for the tip! I installed mopidy-local-sqlite, added it as a library of choice in config.
Alas, it crashes with sqlite too.

INFO     Scanned 18500 of 26212 files in 11936s, ~4976s left.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/CDPLUS/Sub_Info.ja: gst-stream-error-quark: Could not determine type of stream. (4)
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/faceplate.dxr: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/media.cst: gst-stream-error-quark: Could not determine type of stream. (4)
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/MOVIES/BLANK.MOV: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/MOVIES/ozzy.mov: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/readme.htm: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00J: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00N: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00S: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00T: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/WIN_QT4/QuickTimeInstallCache.qdat: gst-stream-error-quark: Could not determine type of stream. (4)
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/BUDAPI.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/DIRECTSO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FILEIO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FLASHASS.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTXTRA.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/INETURL.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MACROMIX.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MIXSERVI.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MPEG3IMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MUIDIALO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETFILE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETLINGO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/QT3ASSET.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDCON.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDIMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACNVRT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWADCMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAIMPOR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAOPT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWASTRM.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTXTRA.X32: No audio found in file.
INFO     Scanned 18600 of 26212 files in 11967s, ~4897s left.
WARNING  Failed local:track:Pain%20of%20Salvation/2004%20-%20Be/01%20-%20Prologue-Animae%20Partus%20%28%27I%20Am%27%29.flac: Timeout after 15000ms

(python:1142): GLib-ERROR **: /build/glib2.0-F5w919/glib2.0-2.50.3/./glib/gmem.c:100: failed to allocate 11017050 bytes
Trace/breakpoint trap

@kingosticks
Copy link
Member

26k files should be fine. This has been used with larger collections. Can you provide some data on the process size? How much ram have you allocated to system memory? Is it crashing when it hits a limit?

@Caenhorn
Copy link
Author

Caenhorn commented Dec 1, 2017

I've collected some data on process size and memory dynamics. It looks like it does run out of free+cached+swap memory. Judging by log-file, process itself does not crash.
process memory
memory dynamics

Log-file:
mopidy.log

Here is netdata snapshot. You can use any netdata demo site to import it and check metrics on your own.
netdata-mopidy-zero-20171201-221224-9204.snapshot.gz

Most of the files in my library are lossless, if it matters. I used stretch lite image for mopidy installation. It is a dedicated system for mopidy, so only services coming by default in stretch lite are running. My config.txt is almost vanilla, except for disabled onboard audio and enabled Allo Mini Boss gpio extension board. I could free up some memory by allocating less GPU mem next time.

@mabeltron
Copy link

I can confirm I have seen this problem with mopidy and with other MPD servers. The scan function appears to have an in-memory cache and will fill memory to the point where the hardware will become unresponsive - I had this on a server with 8GB of RAM as well as RPis. I suspect there needs to be a method of flushing to disk more frequently.

@monte-monte
Copy link

I have the same behavior on a server using mopidy-local-sqlite with option "scan_flush_threshold = 50". It doesn't seem to free memory until it finishes scan. Had to hard reset machine several times before I finished initial scan. This for sure must be fixed.
mopidy version 2.2.2

@kingosticks

This comment has been minimized.

@kingosticks

This comment has been minimized.

@jodal jodal transferred this issue from mopidy/mopidy Dec 21, 2019
@jodal jodal added C-bug Category: This is a bug A-scan Area: Media scanning labels Dec 21, 2019
@01tot10
Copy link

01tot10 commented Jan 21, 2020

I think I'm experiencing similar behavior with my instance of Mopidy + Mopidy-local.

I've had to setup Mopidy twice - once with Mopidy on top of Python2, and now with Python3.
Both times, when adding Mopidy-local into the picture and doing an initial scan of my music library of ~3000 files, the system becomes unresponsive and supposedly hangs near the end of the scan.
Luckily, after doing a hard power cycle and attempting to do the scan again, it continues from where it left off and the scan runs till the end. Still, not how it should be I suppose and something should be done.

I'm running Mopidy on Raspberry 3+ with 1Gb of RAM and Raspbian Stretch.

@Uatschitchun
Copy link

Experiencing the same with mopidy 3.0.1 on a Rpi3 raspbian buster.
It's using up more and more memory during the initial scan. Needs app. 30min to fill the RAM until swapping starts and the PI gets stuck

@Uatschitchun
Copy link

It's not possible to scan in more than 4000 songs without sucking up the RPi3's full RAM and having to kill mopidy.
Haven't tested with 'limit' yet...

@Uatschitchun
Copy link

When using 'limit' (3000), the scan works, but that means to restart the scan over and over until all files are scanned.

@Uatschitchun
Copy link

Using limit 5000 works, but with 150k files it's not the glorious solution

@holtermp
Copy link

Hi, I face the same problem here
Trying to do an initial scan of 68450 files on a
Raspberry Pi 3 Model B Rev 1.2
with Mopidy 3.0.1
always results in swapping after about 20 min. I than power cycle and restart and hope to get through eventually.

Any ideas on how to tackle this would be helpful. @Uatschitchun : What do you mean by "limit", i could find no such parameter in the documentation.

@holtermp
Copy link

Ok, just found it myself, it is a parameter when calling mopidyctl local scan:
local scan [--limit LIMIT] [--force]

  Scan local media files and populate the local library.

    --limit LIMIT  Maximum number of tracks to scan
    --force        Force rescan of all media files

@holtermp
Copy link

holtermp commented Mar 30, 2020

It seemed to me that memory filled up very quickly when I had scan timeouts. And I had lots of them. I do not understand, how timeouts can occur on a local file system, but then I do not know how the mopidy scanner works. Anyway, I set the timeout to 10 seconds:

[local]
scan_timeout = 10000

And ran the scan in a loop of 1000 tracks each

#!/bin/bash
while :
do      
        mopidyctl local scan --limit=1000
        echo "         Press [CTRL+C] to stop.."
        sleep  5
done

That finally helped to complete the scan.

@mrwrighty
Copy link

mrwrighty commented May 22, 2020

Sadly the script didn't do it for me. I only have 11500 files and its bombing every 1000 or so. I have a timeout of 10000 and a limit of 1000 as per script, but no joy. Pi B3+

@01tot10
Copy link

01tot10 commented Jun 13, 2020

In proceedings to the topic:

I had to refresh my collection today, and tried out the limit-trick that was proposed in this thread.
'--limit=1000' didn't quite prevent the crash in my instance of Mopidy, running on Raspberry Pi 3+, 1Gb of RAM, but I do have a good amount of processes running in the background.
I continued with '--limit=500' for the rest of the scan and, at least with this batch, didn't crash the system anymore.

Anyways, just wanted to keep the thread alive and well, since the freeze takes down my whole server if I'm not careful and is quite a showstopper.

@Trotter73
Copy link

Hi,

Yet another "Met to".. Running on a PiZero a scan would hang the Pi after an hour or so, the script above seems to be working great, I started low with a limit of 200 but after some monitoring took it up to 500, I could probably go more but just wanted something reliable.

It's taken a while, 16H or so but I've scanned nearly 28K files...

Big thanks to @holtermp !

@tljohnsn
Copy link

tljohnsn commented Sep 28, 2020

Thanks everyone who commented. I am having the same issue. I modified @holtermp 's script a bit:

#!/bin/bash                                                                                                                                                                 
media_dir=`sudo mopidyctl config |grep media_dir | cut -d = -f 2 | sed -e 's/\s//g'`
numfiles=`find /home/ftp/local -type f -name "*.mp3" |wc -l`
echo "found $numfiles files in /home/ftp/local"
loops=`echo $(( numfiles / 1000 ))`
for i in $(seq 0 $loops); do
    echo $i
    echo mopidyctl local scan --limit=1000
done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-scan Area: Media scanning C-bug Category: This is a bug
Projects
None yet
Development

No branches or pull requests