Use Grok instead of OpenJpeg #2

whikloj · 2017-08-25T15:58:25Z

Partially Resolves? Islandora/documentation#703

This removes the OpenJpeg library and instead downloads and compiles the Grok fork for use with Cantaloupe.

whikloj · 2017-08-25T16:05:45Z

Oops forgot instructions.

Checkout this branch
vagrant up --no-provision
ansible-playbook -i inventory/vagrant bootstrap.yml
ansible-playbook -i inventory/vagrant playbook.yml --tags cantaloupe,tomcat8,grok
vagrant ssh
sudo mkdir -p /var/www/html/drupal/web/
Put test images in /var/www/html/drupal/web (tiff, jp2, etc)
Check that you can see them at: http://localhost:8080/cantaloupe/iiif/2/FILE.NAME/full/full/0/default.jpg
Check logs at: /var/log/cantaloupe

whikloj · 2017-08-25T16:12:52Z

And it works with @dannylamb's terrible Tokyo JP2 from Islandora/documentation#572 (comment)

jonathangreen

Just a few little nits the with the playbook, but it mostly looks good.

jonathangreen · 2017-08-25T16:06:04Z

inventory/vagrant/group_vars/all.yml

@@ -10,3 +10,7 @@ islandora_extra_packages:
  - unzip
  - build-essential
  - vim
+  - cmake


Since these packages are specific to the grok role. I would install them inside the grok role.

Maybe inside install.yml using a structure like this:
https://github.com/Islandora-Devops/claw-playbook/blob/66ef19d323a3c9cc1e1d3e88438d99d3abb0d6e1/roles/internal/openjpeg/tasks/install.yml#L3-L8

jonathangreen · 2017-08-25T16:06:46Z

roles/internal/grok/files/etc/ld.so.conf.d/libgrok.conf

@@ -0,0 +1 @@
+/usr/local/lib


Is this actually needed? I believe /usr/local/lib is in the library path by default.

I seemed to need it to decode JP2s, but Tiffs worked without it. Perhaps I just need the to run ldconfig?

For sure will need ldconfig to load the libraries.

I think it should be included by default, I guess its not a big deal to define it twice.

jonathangreen · 2017-08-25T16:13:01Z

roles/internal/grok/tasks/install.yml

+
+- name: Add Grok libraries to ld.so.conf.d
+  copy: src={{ item }} dest=/{{ item }} owner=root group=root mode=0444
+  with_items:


See comment about this not being necessary above. If we do include it, I wouldn't use a loop to install one item.

I'd rather see something like:

- name: Add Grok libraries to ld.so.conf.d copy: src: libgrok.conf dest: /etc/ld.so.conf.d/libgrok.conf owner: root group: root mode: "0444"

or alternatively in the single line syntax

- name: Add Grok libraries to ld.so.conf.d copy: src=libgrok.conf dest=/etc/ld.so.conf.d/libgrok.conf owner=root group=root mode=0444

I personally prefer the YAML syntax to the single line, but thats just a style thing.

jonathangreen · 2017-08-25T16:19:37Z

roles/internal/grok/vars/main.yml

@@ -0,0 +1,4 @@
+grok_clone_directory: /opt/grok
+grok_version_tag: v2.3.0


I would put both of these in the roles/internal/grok/defaults/main.yml file, as they are super hard to override where they are. Here is a good reference talking about where to put variables. The TL;DR is that you should always use the defaults, not vars.

Right, I did have these there and then I stupidly moved the entire file to cause the below Cantaloupe option to get set. I'll fix them both.

jonathangreen · 2017-08-25T16:21:40Z

roles/internal/grok/vars/main.yml

+grok_clone_directory: /opt/grok
+grok_version_tag: v2.3.0
+
+cantaloupe_OpenJpegProcessor_path_to_binaries: /usr/local/bin


This shouldn't be defined in this playbook for modularity. You want this role to stand on its own not to be setting variables from other roles. I would set this in this with the rest of the cantaloupe variables in:
https://github.com/Islandora-Devops/claw-playbook/blob/master/inventory/vagrant/group_vars/tomcat.yml#L37-L53

jonathangreen · 2017-08-25T16:24:37Z

roles/internal/grok/tasks/install.yml

+  make:
+    chdir: "{{ grok_clone_directory }}/build"
+
+- name: Install grok


Did you run the role twice to make sure this is idempotent? Taking a quick glance at the cmake library that is included, I don't think that this will be idempotent without the creates statement, but I could be reading the code wrong and I haven't actually run it yet.

You can tell if its idempotent by running it twice and seeing if ansible reports any tasks as changed.

So it does seem to have 'changed' it. But I also realized reading the code that he has a creates option, so I'll try adding that.

jonathangreen · 2017-08-25T16:34:05Z

One other minor thing I forgot to mention is that I try to avoid python libraries as much as possible, simply because of the extra cognitive overhead of having to deal with python as well as all the ansible yaml, unless the library buys something thats hard to do in YAML. The big on is if a library provides idempotency, its probably worth it.

In this case I'm not sure it buys us idempotency and I'm not sure its much more convenient then just doing something like:

- name: Cmake grok
  command:
    cmake blah blah
  args:
    creates: file blah blah

Just something else to consider.

whikloj · 2017-08-25T20:04:37Z

@jonathangreen ok I think I got it all. I do need to run ldconfig at the end or the new libraries are not picked up. This makes that process have one always changed task. But otherwise it seems to be idempotent now.

whikloj · 2017-08-25T20:07:42Z

Forgot some unused/unneeded variables and files.

jonathangreen · 2017-08-25T22:48:28Z

roles/internal/grok/tasks/install.yml

+    chdir: "{{ grok_clone_directory }}/build"
+    creates: /usr/local/bin/opj_decompress
+
+- name: Update ldconfig


So to get rid of the last idempotency thing, I'd do something like this:

- name: Install grok command: make install args: chdir: "{{ grok_clone_directory }}/build" creates: /usr/local/bin/opj_decompress register: grok_make_install - name: Update ldconfig command: ldconfig when: grok_make_install.changed is defined AND grok_make_install.changed

Your when: line above caused an error when I tried to run it. I was able (I think) to get the expected process with when: grok_make_install.changed. Does that seem viable?

jonathangreen

Looks good man. Couple other small things, but after that it should be good to go.

jonathangreen · 2017-08-25T22:57:17Z

roles/internal/grok/tasks/install.yml

+  file:
+    path: "{{ grok_clone_directory }}/build"
+    state: directory
+    mode: 0755


weird ansible gotcha here, but always put the mode in quotes since its an octal number or it'll be interpreted as decimal, and shit will get weird haha.

ansible/ansible#9196
https://blog.tyk.nu/blog/ansible-and-unix-file-permissions/

whikloj added 2 commits August 25, 2017 08:06

Replace openjpeg with grok

98880c0

Update libraries include path

94be730

jonathangreen suggested changes Aug 25, 2017

View reviewed changes

whikloj added 2 commits August 25, 2017 13:56

Make idempotent, correct variables, remove library

61e7085

Need to update ldconfig

fe0e06e

whikloj added 2 commits August 25, 2017 15:05

Remove cantaloupe variable

3021dd8

Remove unneeded file

9940835

jonathangreen reviewed Aug 25, 2017

View reviewed changes

Fix mode and make ldconfig run only when needed

80fe9e2

jonathangreen approved these changes Aug 28, 2017

View reviewed changes

jonathangreen merged commit 3efb132 into Islandora-Devops:master Aug 28, 2017

jonathangreen mentioned this pull request Aug 28, 2017

Consider using grok instead of openjp2 with cantaloupe Islandora/documentation#703

Closed

whikloj deleted the issue-703 branch September 24, 2021 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Grok instead of OpenJpeg #2

Use Grok instead of OpenJpeg #2

whikloj commented Aug 25, 2017

whikloj commented Aug 25, 2017

whikloj commented Aug 25, 2017

jonathangreen left a comment

jonathangreen Aug 25, 2017

jonathangreen Aug 25, 2017

whikloj Aug 25, 2017

jonathangreen Aug 25, 2017

jonathangreen Aug 25, 2017

jonathangreen Aug 25, 2017

whikloj Aug 25, 2017

jonathangreen Aug 25, 2017

jonathangreen Aug 25, 2017

whikloj Aug 25, 2017

jonathangreen commented Aug 25, 2017

whikloj commented Aug 25, 2017

whikloj commented Aug 25, 2017

jonathangreen Aug 25, 2017

whikloj Aug 28, 2017

jonathangreen left a comment

jonathangreen Aug 25, 2017

		@@ -0,0 +1,4 @@
		grok_clone_directory: /opt/grok
		grok_version_tag: v2.3.0

		@@ -0,0 +1 @@
		/usr/local/lib

Use Grok instead of OpenJpeg #2

Use Grok instead of OpenJpeg #2

Conversation

whikloj commented Aug 25, 2017

whikloj commented Aug 25, 2017

whikloj commented Aug 25, 2017

jonathangreen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathangreen commented Aug 25, 2017

whikloj commented Aug 25, 2017

whikloj commented Aug 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathangreen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment