Files erased from docserver

[pirolen]

I could reproduce this now (own installation of FLAT, using nginx), still in test mode, so nothing important got lost.
Context:
while I was test-annotating a file logged in as a specific user, I restarted the service backend, since I changed something in the config file and wanted to update the settings (and forgot that I was still testing the annotating). I guess this is not best practice, but perhaps a realistic worst-case scenario?

This caused that all the files disappeared that were in the directory of the user who was editing, but this was only visible in the backend, in the GUI the user got a warning and could not undertake any action anymore (which I unfortunately did not screenshot.)
Shall I share the logs with you (all of it?).
Thanks very much in case this is solvable.

[proycon]

All existing files in the document root actually got lost?? They got removed from disk completely or are just no longer showing up in the index? That former sounds very serious, I can't really imagine what process could cause that. The only thing I can imagine is that a file you're currently annotating not getting saved to disk properly if the backend service is suddenly restarted too 'roughly' (e.g. SIGKILL).

Yes, please send the foliadocserve.log and an ls -laR of the document root. We definitely need to get to the bottom of this.

[pirolen]

All existing files in the document root actually got lost?? They got removed from disk completely

Yes, this is how I experienced it. But only for that user (as said, this was luckily not 'in production', so no loss). The files that another user had are left intact.
Shall I attach the log in its entirety or per email? (It includes also stuff from last year...)
Unfortunately I cannot tell on which day this happened, I think it was even last week, I was logged in as flatuser and I think it might involved the 'forcibly unload documents' parts for that user.

[pirolen]

if the backend service is suddenly restarted too 'roughly' (e.g. SIGKILL).

I did either sudo systemctl stop/start or restart.
(I know sudo is unsafe, will need to adjust this.)

Another suspicious detail might be that while doing the annotation, I was logged in with the same credentials as the FLAT superuser.

Yes, please send the foliadocserve.log and an ls -laR of the document root. We definitely need to get to the bottom of this.

Please find attached, and let me know in case I could supply more data.
I suspect it happened on 31 July.
docroot_filelist.txt
foliadocserve_PL.log.zip

[proycon]

Thanks for the logs. There is indeed a mention of a deletion in there on July 31st:

2021-07-31 15:05:18 - Documents loaded: 1
2021-07-31 15:05:37 - Delete, namespace=flatuser
2021-07-31 15:05:37 - Unloading flatuser/FA-MBK-4-3_035245008_0019_abpproc
2021-07-31 15:05:37 - Removing /home/flatuser/flateditor/foliadocuments/flatuser/FA-MBK-4-3_035245008_0019_abpproc.folia.xml
2021-07-31 15:05:37 - Doing git commit for /home/flatuser/flateditor/foliadocuments/flatuser/FA-MBK-4-3_035245008_0019_abpproc.folia.xml -- Removed document
2021-07-31 15:05:37 - Delete, namespace=flatuser
2021-07-31 15:05:37 - Removing /home/flatuser/flateditor/foliadocuments/flatuser/FA-MBK-4-3_035245008_0020_abpproc.folia.xml
2021-07-31 15:05:37 - Doing git commit for /home/flatuser/flateditor/foliadocuments/flatuser/FA-MBK-4-3_035245008_0020_abpproc.folia.xml -- Removed document

Something similar happened earlier in march:

2021-03-01 18:20:40 - Delete, namespace=piro

At both times, an entire namespace got deleted, which should only happen after an explicit request from the user to do so of course. I'm kind of wondering whether you were perhaps trying to delete anything using the file management functions in FLAT, perhaps a single document rather than all document? Or even through a possible misclick on the delete or move buttons? They don't currently ask for confirmation, perhaps I should implement that.

I'll do some further investigation to get to the bottom of this and to be sure it can't have been executed as a side-effect of some other action.

On the bright side, the git backend was correctly configured so all the deleted files are still in git and can be restored from there.

[pirolen]

Thanks very much!

Something similar happened earlier in march:

Yes indeed. The logs rock! :-)

At both times, an entire namespace got deleted

Yes, this is how it felt.

which should only happen after an explicit request from the user to do so of course. I'm kind of wondering whether you were perhaps trying to delete anything using the file management functions in FLAT, perhaps a single document rather than all document? Or even through a possible misclick on the delete or move buttons?

In March I am not sure, but this time I am quite sure I was not managing files. In the browser there was a single F:AT window open, where I was annotating a document. I pretty much use the management window for uploading only.

On the bright side, the git backend was correctly configured so all the deleted files are still in git and can be restored from there.

That's nice! How is the git backend working with FLAT? How safe are the files/versions? Would be great to understand more of it, so that I can communicate it to the users, or take extra backuping/synching actions if needed.

[proycon]

In March I am not sure, but this time I am quite sure I was not managing files. In the browser there was a single F:AT window open, where I was annotating a document. I pretty much use the management window for uploading only.

Hmm, ok, good to know. The upload invokes a different endpoint so shouldn't really cause a conflict. I do see that after both deletions, about 30s later you indeed do an upload.

That's nice! How is the git backend working with FLAT? How safe are the files/versions?
Would be great to understand more of it, so that I can communicate it to the users, or take extra backuping/synching actions if needed.

Basically whenever a document is unloaded from the server (after inactivity or on an explicit save), it is saved to disk and a git commit is being done that captures all the changes made in that particular session. Though you did report an instance where the git commit seems to fail (issue #169) that I'll have to look into deeper as well. From the interface you can roll-back to previous versions of a document using the "History & Undo", that's as far as the interface goes though. To roll back deleted files etc you need to use git from the command line.

Extra backup/sync actions are a good idea yes. FLAT basically creates multiple git repositories, each user namespace, or even each directory, is its own git repository. You can set up git remotes for these repositories and periodically push to them (e.g. from cron). The FLAT interface doesn't offer any high-level tooling for this either though, but since the it's just powered by git a lot is possible.

[pirolen]

Hmm, ok, good to know. The upload invokes a different endpoint so shouldn't really cause a conflict. I do see that after both deletions, about 30s later you indeed do an upload.

The reason why I am typically not doing file deletion in the GUI is that it often suffices to upload another version of the same document, which FLAT assigns the same name and thus rewrites the old version.

I seem to remember that in March, after some documents were deleted from the piro namespace, to make a clean start I also deleted (all?)the remaining files from the backend.

Extra backup/sync actions are a good idea yes. FLAT basically creates multiple git repositories, each user namespace, or even each directory, is its own git repository. You can set up git remotes for these repositories and periodically push to them (e.g. from cron).

That sounds good! What would be the path/highest directory of such a repository to push to the remote?

[proycon]

What would be the path/highest directory of such a repository to push to the remote?

You can configure the mode with the --gitmode parameter to foliadocserve (though you can't really switch if you're not starting from scratch). The default mode is user, which means that each user namespace is its own git repository. There is also monolithic which has one git repository for all users (to the root would be a level up), but this is not recommended except if you have few users/documents because it doesn't scale well. There is also nested in which case each directory is its own git repository (for maximum scalability).

[pirolen]

Thanks!
It looks important to use git with this parameter. What would it take to start from scratch? (sorry for the overhead)

[proycon]

The parameter is to foliadocserve (not git itself). You're currently using the default --gitmode user, which I think is the best fit.

Starting from scratch would mean starting with an empty document root, but I don't think that's necessary, your setup should be good to go as it is.

[proycon]

I implemented an extra confirmation stage and another precaution to hopefully prevent this issue from happening, even though the cause hasn't been explicitly identified and reproduced.

[pirolen]

Just mentioning another detail, probably irrelevant: when restarting the FLAT systemd service, I also always restart the uwsgi process. This restarting was/is sometimes quite frequent, as annotation set definition changes require fully stopping and starting the service to take effect.

[proycon]

That shouldn't make a diferrence indeed

[proycon]

FLAT v0.10.1 is released now which has some measures to prevent this issue (please reopen if it occurs still)