Get OTA running in docker compose

[michielbdejong]

On my machine I can do:

michiel-2:edit.tosdr.org michiel$ docker ps -a
CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS                        PORTS     NAMES
01b32f844d25   d53961c856d4                      "/bin/sh -c '. $NVM_…"   6 minutes ago   Exited (1) 6 minutes ago                edittosdrorg-ota-1
1c0fa2d65139   ota                               "/bin/sh -c '. $NVM_…"   3 days ago      Exited (255) 11 minutes ago             elastic_engelbart
bd42ff74b366   6bd90a35d7fc                      "/bin/sh -c '. $NVM_…"   3 days ago      Exited (1) 3 days ago                   clever_hawking
9a351658b210   1ec693117994                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   optimistic_roentgen
133ec2dfaa1c   1ec693117994                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   wonderful_hugle
22f0b2aaf3f8   2f8c38c8d75d                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   nice_einstein
b4addc33fb4b   74961f58b71e                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   exciting_torvalds
399e5ab6acb8   74961f58b71e                      "/bin/sh -c 'node se…"   3 days ago      Exited (1) 3 days ago                   flamboyant_spence
bf6002fa0831   46f48478527e                      "/bin/sh -c 'node se…"   3 days ago      Exited (1) 3 days ago                   infallible_ride
a367016ce2b0   b02aa118fb0d                      "/bin/sh -c 'node se…"   3 days ago      Exited (1) 3 days ago                   nervous_morse
5bef5dce3d51   5063bf567f3b                      "/bin/sh -c 'node se…"   3 days ago      Exited (127) 3 days ago                 zealous_goodall
de264f0bf91f   1525b244be1d                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   heuristic_mestorf
4bd74eda885f   edittosdrorg-web                  "bash docker-entrypo…"   3 days ago      Exited (137) 2 seconds ago              edittosdrorg-web-1
c53bc33e2d93   hypothesis/elasticsearch:latest   "/usr/local/bin/dock…"   3 days ago      Exited (143) 2 seconds ago              edittosdrorg-elasticsearch-1
44f4a00bf1db   postgres:11.5-alpine              "docker-entrypoint.s…"   3 days ago      Exited (0) 3 seconds ago                db
7ea7aad8572a   adminer                           "entrypoint.sh php -…"   3 days ago      Exited (0) 3 seconds ago                edittosdrorg-adminer-1
michiel-2:edit.tosdr.org michiel$ docker rm `docker ps -qa`
01b32f844d25
1c0fa2d65139
bd42ff74b366
9a351658b210
133ec2dfaa1c
22f0b2aaf3f8
b4addc33fb4b
399e5ab6acb8
bf6002fa0831
a367016ce2b0
5bef5dce3d51
de264f0bf91f
4bd74eda885f
c53bc33e2d93
44f4a00bf1db
7ea7aad8572a
michiel-2:edit.tosdr.org michiel$ docker pull pondersource/tosdr-ota
Using default tag: latest
latest: Pulling from pondersource/tosdr-ota
Digest: sha256:7b383c068355a823b3a22dcdabd5fd066d250423eeceb4c5e55137170042d143
Status: Image is up to date for pondersource/tosdr-ota:latest
docker.io/pondersource/tosdr-ota:latest

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview pondersource/tosdr-ota
michiel-2:edit.tosdr.org michiel$ docker compose up
[+] Running 5/5
 ✔ Container edittosdrorg-adminer-1        Created                                                                                                                                                                                             0.6s 
 ✔ Container edittosdrorg-web-1            Created                                                                                                                                                                                             0.6s 
 ✔ Container edittosdrorg-ota-1            Created                                                                                                                                                                                             0.6s 
 ✔ Container edittosdrorg-elasticsearch-1  Created                                                                                                                                                                                             0.6s 
 ✔ Container db                            Created                                                                                                                                                                                             0.6s 
Attaching to db, adminer-1, elasticsearch-1, ota-1, web-1
adminer-1        | [Mon Sep 16 08:41:12 2024] PHP 7.4.33 Development Server (http://[::]:8080) started
db               | 2024-09-16 08:41:12.231 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db               | 2024-09-16 08:41:12.231 UTC [1] LOG:  listening on IPv6 address "::", port 5432
[...]

And then in a second terminal window:

michiel-2:edit.tosdr.org michiel$ docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS                    PORTS                                NAMES
3a81e20c4924   pondersource/tosdr-ota:latest     "/bin/sh -c '. $NVM_…"   12 seconds ago   Up 11 seconds             127.0.0.1:3000->3000/tcp             edittosdrorg-ota-1
4bd74eda885f   edittosdrorg-web                  "bash docker-entrypo…"   3 days ago       Up 11 seconds             0.0.0.0:9090->3000/tcp               edittosdrorg-web-1
c53bc33e2d93   hypothesis/elasticsearch:latest   "/usr/local/bin/dock…"   3 days ago       Up 11 seconds             127.0.0.1:9200->9200/tcp, 9300/tcp   edittosdrorg-elasticsearch-1
44f4a00bf1db   postgres:11.5-alpine              "docker-entrypoint.s…"   3 days ago       Up 11 seconds (healthy)   127.0.0.1:5432->5432/tcp             db
7ea7aad8572a   adminer                           "entrypoint.sh php -…"   3 days ago       Up 11 seconds             0.0.0.0:1337->8080/tcp               edittosdrorg-adminer-1
michiel-2:edit.tosdr.org michiel$ docker exec -it edittosdrorg-web-1 curl -d'{"fetch":"http://example.com","select":"body"}' http://ota:3000
{"fetch":"http://example.com","select":"body"}"<h1 id=\"exampledomain\">Example Domain</h1>\n<p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p>\n<p><a href=\"https://www.iana.org/domains/example\">More information…</a></p>"

@madoleary is this working for you too?

I just did a docker push so I think a docker pull pondersource/tosdr-ota should give you:

latest: digest: sha256:7b383c068355a823b3a22dcdabd5fd066d250423eeceb4c5e55137170042d143 size: 7836

and then it should work!

[madoleary]

I pulled the image with the matching hash, but I still have the same error.

When I run: docker exec -it edittosdrorg_web_1 curl -d'{"fetch":"http://example.com","select":"body"}' http://ota:3000

I get: curl: (18) transfer closed with outstanding read data remaining

And the output of docker logs edittosdrorg_ota_1 is:

Node.js v20.17.0
listening on port 3000
{ fetch: 'http://example.com', select: 'body' }
/home/crawler/engine/node_modules/@puppeteer/browsers/lib/cjs/launch.js:310
                reject(new Error([
                       ^

Error: Failed to launch the browser process!
[263:263:0916/102307.128939:FATAL:zygote_host_impl_linux.cc(126)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.


TROUBLESHOOTING: https://pptr.dev/troubleshooting

    at ChildProcess.onClose (/home/crawler/engine/node_modules/@puppeteer/browsers/lib/cjs/launch.js:310:24)
    at ChildProcess.emit (node:events:531:35)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)

Node.js v20.17.0

[michielbdejong]

which commit of edit.tosdr.org are you on? any uncommitted changes?
did you also remove the old containers? you need to make sure docker ps -a is empty before running docker compose up otherwise it will just start up the old ones

[michielbdejong]

you can run docker stop `docker ps -q` to stop all running containers (should not be necessary after docker compose down) and then docker rm `docker ps -qa` to remove them so they will be created fresh in your next docker compose up. This is one of the annoying things in docker compose which is why I don't really like it :)

[madoleary]

I'm on commit:

commit 90edfb47a1ec845b73b621091bbe15613e6e1ab1 (HEAD -> ota-engine-integration, origin/ota-engine-integration)
Author: Michiel de Jong <[email protected]>
Date:   Fri Sep 13 13:59:08 2024 +0200

    Add ota server

I followed your steps and I still have the same error.

[michielbdejong]

Hm, weird! Maybe it will go away.
Once I get the production server ready you'll be able to use that.

[michielbdejong]

I pushed the ota image version 1.1, you should now see:

docker exec -it edittosdrorg-web-1 curl -d'{"fetch":"http://example.com","select":"body"}' http://ota:3000
Please send an Authorization header with the API_SECRET from this script's run environment

[michielbdejong]

I pushed the ota image version 1.2, you should now see:

$ docker exec -it edittosdrorg-web-1 curl -d'{"fetch":"http://example.com","select":"body"}' -H 'Authorization: Bearer foo' http://ota:3000
{"fetch":"http://example.com","select":"body"}"<h1 id=\"exampledomain\">Example Domain</h1>\n<p>This domain is for 
[...]

[michielbdejong]

OK we both saw this fail in gitpod, but it seems to work in Docker Desktop on Mac. Su.per.weird. Maybe the sandboxing is different if the host system is on ubuntu?