Update Virtuoso

[Windvis]

This updates Virtuoso from 7.2.5.1 to 7.2.9 by switching to the redpencil/virtuoso:1.1.0 image.

[Windvis]

Not sure if the data backup part is suited for the hosted environments, but this is what I did locally and it seemed to work.

[Windvis]

a new version is out, so we should update to that one instead.

Edit, not tagged yet, so I guess we wait until it is?

[cecemel]

We keep this one open until we have a sane version of virtuoso. I want this for documentation purpose.

[cecemel]

@Riadabd

• the new version should be here: apply workarounds for user rights issue with sparql endpoint user redpencilio/docker-virtuoso#19
• to test in docker-compose.override.yml from dev-loket
• @claire-lovisa has some deploy instructions to do it in a clean way, but no idea where.

[claire-lovisa]

Hello !

Before updating the version of virtuoso:

docker-compose exec database isql-v
    exec('checkpoint');
    exec('shutdown');

rm data/db/virtuoso.trx

Then you can bump the version and up the triplestore.

Note that I didn't do it for a while, so maybe try it out locally before doing it for real :)
You could also, if it's on prod, generate backups after doing the checkpoint, check the crontab of prod servers to see how the virtuoso-backup.sh script is called for your project

[Riadabd]

I'll put a couple of comments to document the process since it was buried inside Jira comments. The steps below involve the usual database update (some stuff will be edited after confirmation), and steps for nquad dumping and size saving will be posted in a separate comment.

Upgrade with database size optimizations:

Before any upgrade starts, it's best to back up both virtuoso and publication-triplestore:

• Stop the stack: docker compose down
• Start the maintenance frontend banner
• Start the backup process using virtuoso-backup.sh inside the /data/useful-scripts/ directory: /data/useful-scripts/virtuoso-backup.sh [container-name]

The upgrade section in this README describes the process needed to upgrade the database and reduce its size. The steps have been copied from Niel's notes with minor adjustments and are as follows:

Steps for Virtuoso:

Dump N-Quads (output files should have an .nq.gz extension)

When upgrading it's recommend (and sometimes required!) to first dump to quads using the dump_nquads procedure:

docker compose up -d virtuoso
docker compose exec virtuoso isql-v
SQL> dump_nquads ('dumps', 1, 1000000000, 1);

Stop virtuoso

docker compose down (Stops all containers in the project)

Remove the old db and corresponding files

After stopping the containers move the dumps folder to the toLoad folder and make sure to delete the following files:

• .data_loaded
• .dba_pwd_set
• virtuoso.db
• virtuoso.trx
• virtuoso.pxa
• virtuoso-temp.db

mv data/db/dumps data/db/toLoad
rm -i data/db/virtuoso.{db,trx,pxa} data/db/virtuoso-temp.db data/db/.data_loaded data/db/.dba_pwd_set

Update the virtuoso version in docker-compose.yml

virtuoso:
-    image: redpencil/virtuoso:1.0.0
+    image: redpencil/virtuoso:1.2.0-rc.1

Start the database again and monitor logs for any weird behavior:

docker compose up -d virtuoso && docker compose logs -ft virtuoso

Steps for Publication-triplestore:

Dump N-Quads (output files should have an .nq.gz extension)

When upgrading it's recommend (and sometimes required!) to first dump to quads using the dump_nquads procedure:

docker compose up -d publication-triplestore
docker compose exec publication-triplestore isql-v
SQL> dump_nquads ('dumps', 1, 1000000000, 1);

Stop publication-triplestore

docker compose down (Stops all containers in the project)

Remove the old db and corresponding files

After stopping the containers move the dumps folder to the toLoad folder and make sure to delete the following files:

• .data_loaded
• .dba_pwd_set
• virtuoso.db
• virtuoso.trx
• virtuoso.pxa
• virtuoso-temp.db

mv data/publication-triplestore/dumps data/publication-triplestore/toLoad
rm -i data/publication-triplestore/virtuoso.{db,trx,pxa} data/publication-triplestore/virtuoso-temp.db data/publication-triplestore/.data_loaded data/publication-triplestore/.dba_pwd_set

Update the virtuoso version in docker-compose.yml

publication-triplestore:
-    image: redpencil/virtuoso:1.0.0
+    image: redpencil/virtuoso:1.2.0-rc.1

Start the database again and monitor logs for any weird behavior:

docker compose up -d publication-triplestore
docker compose logs -f publication-triplestore

You can monitor the dump loading state as follows:

docker compose exec virtuoso isql-v
SQL> select * from DB.DBA.load_list;

The query will output a table; you can validate the process by checking the ll_state of the load. A ll_state of 2 means a file has been loaded.

[Riadabd]

Upgrade without database size optimizations:

Before any upgrade starts, it's best to back up both virtuoso and publication-triplestore:

• Stop the stack: docker compose down
• Start the maintenance frontend banner
• Start the backup process using virtuoso-backup.sh inside `/data/useful-scripts/ or by following the below steps:
• docker compose exec -i virtuoso_container mkdir -p backups (virtuoso_container can be virtuoso or publication-triplestore)
• Place backups inside the backups/ folder: docker compose exec -i virtuoso_container isql-v
  • exec('checkpoint');
  • backup_context_clear();
  • backup_online('backup_',30000,0,vector('backups'));
  • exit;

Steps for Virtuoso:

• Stop the stack: docker compose down (The stack should already be down at this point if the backup step was performed)
• Start the virtuoso container: docker compose up -d virtuoso
• Create a new checkpoint to clear the transaction log:
  • Enter the isql-v environment: docker compose exec virtuoso isql-v
  • Create a checkpoint: exec('checkpoint');
  • Exit the isql-v environment: exec('shutdown');
• Stop the virtuoso service: docker compose down
• Remove the transaction log file: rm data/db/virtuoso.trx
• Bump virtuoso to the new image
• docker compose up -d virtuoso to pick up the new version (currently redpencil/virtuoso:1.2.0-rc.1)
• Perform database workflows relating to your specific module
• Check the logs to monitor any irregularity

Steps for publication-triplestore:

The steps are the same for publication-triplestore with some slight adjustments for the virtuoso log deletion:

• Stop the stack: docker compose down
• Start the publication-triplestore container: docker compose up -d publication-triplestore
• Create a new checkpoint to clear the transaction log:
  • Enter the isql-v environment: docker compose exec publication-triplestore isql-v
  • Create a checkpoint: exec('checkpoint');
  • Exit the isql-v environment: exec('shutdown');
• Stop the publication-triplestore service: docker compose down
• Remove the transaction log file: rm data/publication-triplestore/virtuoso.trx
• Bump publication-triplestore to the new image (currently redpencil/virtuoso:1.2.0-rc.1)
• docker compose up -d publication-triplestore to pick up the new version
• Perform workflows that go through publication-triplestore
• Check the logs to monitor any irregularity

[Riadabd]

The upgrade process with and without applying the database size optimization process have been split for better clarity and separation.