A couple integrity questions #6834
Comments
|
Same problems here |
|
Also getting the same issue any time a video is played (see Other Errors section) |
|
Bump?? These are serious bugs. |
|
@Pheggas, I agree, and we have put more effort into hardening the storage template mechanism. I think you and Andrew was one of the first users of Immich so there could have been artifacts from before we put in the hardening process as well. We keep this open to make sure to revisit this mechanism cc: @zackpollard |
|
Yea, I would be interested if this issue has gotten worse after our storage migration hardening efforts. On the integrity tooling side, having a automated scan of integrity is something that's on my to-do list. As we have the hash of all files, doing this should be pretty simple overall. The problem is there are many things on my to-do list 😅 |
|
I'm seeing the write epipe errors as well. I'm new to the app, is this relates to using the storage template? If I wasn't log watching while setting up and running my imports, I never would have known there was an issue. The videos play and assets load, but it is accompanied by the errors. |
|
I am on v1.101.0 2 of these have the video transcribe error described here, one does not. I don't understand the underlying "integrity" problem well enough so I'm not sure what might be going on Thanks <3 |
Hey, the overall idea is a tool that will incrementally scan all assets on the server over time, recomputing the hash of the asset on disk and comparing it to the hash in the database to make sure it hasn't changed. However before we do this work we wanted to make changes to the current repair page to improve that so it works in the background too, as for big instances this page consistently times out as it computes everything when you load the page. |
|
I just discovered there is a repair page XD I see about 413 untracked files on mine at the moment, I have never modified the underlying storage or DB, is this because of the file integrity changing? Is the idea to similarly show the files which have changed integrity over time? I'll check out how the repair page works & if I have any ideas, thank you! If there is a discussion open you can link me to, I'll catch up on it. Minor update: One of the assets of mine which couldn't be moved might be a deleted photo that is in the thrash but I haven't visually confirmed it yet. |
|
I am seeing this EPIPE issue as well. New user since 1.91. I've had to wait 25 seconds for the large image thumbnail to load. |
|
Hey, I have created a brand new instance and I have imported all my data by using the |
|
@MiguelNdeCarvalho As you can see, your files are missing the moov atom. Therefore, you won't be able to transcode them as they are missing that important data. That data describes what is inside the mp4/mov container. You can get files like this typically when recording is terminated before it could be written to disk as the moov atom is usually written last. (For example if the card ran out of space, camera battery died during recording, corrupted files, etc) Immich should mark items it cannot transcode as such, so it does not add them back into the queue perpetually. Background: I work in broadcast |
Thanks for the explanation! |
|
I have noticed a few files with this error |
|
Regarding repairing the files, I’ve had success with Easeus software. You supply it the corrupted files but also known good files in the same format (same recording device, resolution, frame-rate, bitrate etc) and if you are lucky it can recover some of the corrupted files. Because it is corrupted data, it is not a sure thing though. To remove the files from Immich I believe you should be able to delete them from the UI so they are also marked as removed in the database. If they haven’t been ingested, then a simple disk delete should be fine. |
|
Thanks I noticed I can pull them up with the asset Id in the URL, I wasnt able to find them with search so this is helpful. |
|
Closing in favor of #12293 |
The bug
In discussion with @jrasm91 wanted to enumerate a couple integrity related log errors on my instance, these are likely instance specific "holes" that can be fallen into (especially if you were an early adopter). Tooling for examining instance integrity would be fantastic but realizing that it's potentially a lower priority than, say, search at this time.
On this instance all jobs have been re-run fully on the entire dataset.
I have never manipulated the underlying files or database, these errors in state were achieved through likely having a long running instance and not getting on top of these errors when they initially emerged.
Most likely these are all missing files that have "ghosted" over time. Some excerpt errors, will happily dig into each of these in more detail. There appears to be many (~300 within a ~200k) instances of these two types of errors which looks like missing assets or potentially originating file formats that immich isn't prepared for (e.g. VHS conversions of home movies where I may have ffmpeg'ed poorly):
Thumbnail Generation
Video Transcoding
Likely ~400 of these on my instance. We've always had pixel devices (or "real" cameras) so I think this may be a different bug than the recent #6337.
Other Errors
While browsing I see this, maybe nginx headers not configured properly?
The OS that Immich Server is running on
Centos 9 Stream, Podman, systemd
Version of Immich Server
v1.94.1
Version of Immich Mobile App
v1.94.1.build.121
Platform with the issue
The text was updated successfully, but these errors were encountered: