add explicit `libxml2` and `libffi` dependencies to LLVM 11.x/12.x/14.x by Thyre · Pull Request #23603 · easybuilders/easybuild-easyconfigs

Thyre · 2025-08-08T08:24:37Z

(created using eb --new-pr)

Adding both to all EasyConfigs might be too much for some versions. We can verify this with builds on jsc-zen3 first, and remove things where necessary.

Partially resolves #23601

Related PRs:

{compiler}[GCCcore/12.2.0,GCCcore/12.3.0] Add libxml2/zstd/libffi dependencies to LLVM 14.x/15.x/16.x #23608
{compiler}[GCCcore/13.2.0,GCCcore/13.3.0] Add libxml2/libffi/zstd dependencies to minimal/llvmlite LLVM #23609
{compiler}[GCCcore/13.3.0] Add libxml2 and zstd dependencies to LLVM 18.1.8/19.1.7 #23610
{compiler}[GCCcore/13.3.0] Add libxml2 and zstd dependencies to LLVM 20.1.4/20.1.5 #23611
{compiler}[GCCcore/13.2.0] Add libxml2 and zstd dependencies to LLVM 20.1.5 #23612
{compiler}[GCCcore/14.3.0] Add libxml2 and zstd dependencies to LLVM 20.1.8 #23613
add sanity check to link system libraries to LLVM easyblock easybuild-easyblocks#3877

Thyre · 2025-08-08T08:42:14Z

Going through the LLVM sources, LLVM 15 is the first one adding zstd as part of llvm::Support::Compression. I'll remove this dependency from the older versions then. libxml2 can be found even in LLVM 11.

zstd compression was added in LLVM 15 Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>

Thyre · 2025-08-08T10:07:54Z

@boegelbot please test @ jsc-zen3
CORE_CNT=16
EB_ARGS="LLVM-11.1.0-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-11.2.0.eb LLVM-14.0.3-GCCcore-11.3.0.eb"

boegelbot · 2025-08-08T10:10:10Z

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23603 EB_ARGS="LLVM-11.1.0-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-10.3.0.eb LLVM-12.0.1-GCCcore-11.2.0.eb LLVM-14.0.3-GCCcore-11.3.0.eb" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23603 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 7523

Test results coming soon (I hope)...

Details

- notification for comment with ID 3167319735 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>

boegelbot · 2025-08-08T11:09:07Z

Test report by @boegelbot
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/547af282a7f0dc9ec6e464aa3fc6a63b for a full test report.

Thyre · 2025-08-08T11:19:18Z

Test report by @boegelbot
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/547af282a7f0dc9ec6e464aa3fc6a63b for a full test report.

We've seen this test fail before...
#20902 (comment)

Flamefire · 2025-08-08T11:58:20Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3875
FAILED
Build succeeded for 0 out of 4 (4 easyconfigs in total)
n1509 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/319eb845bfb511fcdcb417ba0b6e3d65 for a full test report.

Thyre · 2025-08-08T14:15:35Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3875
FAILED Build succeeded for 0 out of 4 (4 easyconfigs in total)
n1509 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/319eb845bfb511fcdcb417ba0b6e3d65 for a full test report.

All four failed 😕
Unfortunately, there are no logs in the Gist. Do you know why they failed?

Flamefire · 2025-08-08T14:43:42Z

Unfortunately, there are no logs in the Gist. Do you know why they failed?

This is weird. Is this an EB bug?

Failures are all the same:

llvm-ar: error while loading shared libraries: libffi.so.8: cannot open shared object file: No such file or directory

libffi gets pulled in through Python during buildtime. I only see libLLVM-11.so requires it, not why.

Thyre · 2025-08-08T14:46:31Z

Unfortunately, there are no logs in the Gist. Do you know why they failed?

This is weird. Is this an EB bug?

Maybe because of the options you've set, but I would still expect to see the Gist...

Sounds like a bug (or at least oversight) to me

Failures are all the same:

llvm-ar: error while loading shared libraries: libffi.so.8: cannot open shared object file: No such file or directory

libffi gets pulled in through Python during buildtime. I only see libLLVM-11.so requires it, not why.

Probably caused by easybuilders/easybuild-easyblocks#3849. So we either need an explicit dependency here, or update the check in the EasyBlock to set that configure option based on dependencies

Flamefire · 2025-08-08T14:51:00Z

Probably caused by easybuilders/easybuild-easyblocks#3849. So we either need an explicit dependency here, or update the check in the EasyBlock to set that configure option based on dependencies

Both, I'd say.

From the log:

-- Performing Test HAVE_FFI_CALL
-- Performing Test HAVE_FFI_CALL - Success
-- Found FFI: /data/horse/ws/s3248973-EasyBuild/easybuild-rapids/software/libffi/3.4.2-GCCcore-11.3.0/lib64/libffi.so

And there is seemingly an additional place where it might be pulled in in the omptarget plugin: https://github.com/llvm/llvm-project/blob/e64224a224a9891d01ad6eec84921bca1c1f0441/offload/cmake/Modules/LibomptargetGetDependencies.cmake#L42

Hence I guess a dependency is required.

Signed-off-by: Jan Andre Reuter <jan@zyten.de>

Thyre · 2025-08-09T05:46:30Z

Probably caused by easybuilders/easybuild-easyblocks#3849. So we either need an explicit dependency here, or update the check in the EasyBlock to set that configure option based on dependencies

Both, I'd say.

From the log:
-- Performing Test HAVE_FFI_CALL
-- Performing Test HAVE_FFI_CALL - Success
-- Found FFI: /data/horse/ws/s3248973-EasyBuild/easybuild-rapids/software/libffi/3.4.2-GCCcore-11.3.0/lib64/libffi.so  
And there is seemingly an additional place where it might be pulled in in the omptarget plugin: https://github.com/llvm/llvm-project/blob/e64224a224a9891d01ad6eec84921bca1c1f0441/offload/cmake/Modules/LibomptargetGetDependencies.cmake#L42

Hence I guess a dependency is required.

While a minimal LLVM should not get to the offload CMake module (which only exists since LLVM 19), I agree that we should just define the libffi dependency in all LLVM EasyConfigs to avoid issues like you've seen.

I updated all the PRs accordingly. For full LLVM builds, libffi was already added.

Flamefire · 2025-08-09T09:11:15Z

I guess we should still update the easyblock as this seems to be a regression introduced by that change. Could you do that? I could also fold it into the parallel-test PR to reduce testing.

I updated all the PRs accordingly. For full LLVM builds, libffi was already added.

Can you link the related PRs in the description for easier discoverability?

Thyre · 2025-08-09T09:28:07Z

I guess we should still update the easyblock as this seems to be a regression introduced by that change. Could you do that? I could also fold it into the parallel-test PR to reduce testing.

Agreed, I can look into this tomorrow.

Can you link the related PRs in the description for easier discoverability?

Done

Flamefire · 2025-08-11T07:35:42Z

Agreed, I can look into this tomorrow.

I did a more complete change in easybuilders/easybuild-easyblocks#3881 extending the same logic to all dependencies. This should avoid similar issues.

Thyre · 2025-08-18T14:38:52Z

@boegelbot please test @ jsc-zen3
CORE_CNT=16

boegelbot · 2025-08-18T14:40:10Z

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23603 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23603 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 7655

Test results coming soon (I hope)...

Details

- notification for comment with ID 3197211400 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegel · 2025-08-18T15:07:25Z

Test report from bot was triggered before easybuilders/easybuild-easyblocks#3881 was merged, not sure if that's wise...

So, just to make sure:

@boegelbot please test @ jsc-zen3
EB_ARGS="--installpath /tmp/$USER/pr23603 --include-easyblocks-from-pr 3881"
CORE_CNT=16

boegelbot · 2025-08-18T15:38:24Z

Test report by @boegelbot
FAILED
Build succeeded for 1 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/5ee0ae2ec96bc1f90169ebcae366cdbd for a full test report.

Thyre · 2025-08-18T15:45:07Z

Test report by boegelbot
FAILED
Build succeeded for 1 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/5ee0ae2ec96bc1f90169ebcae366cdbd for a full test report.

Hm, looks like ld.gold is broken for those older toolchains on jsc-zen3, while there's still the one failing test for the third LLVM. The fourth works fine.

boegel · 2025-08-18T16:29:52Z

Hm, looks like ld.gold is broken for those older toolchains on jsc-zen3, while there's still the one failing test for the third LLVM. The fourth works fine.

@Thyre We can try to rebuild GCCcore/10.3.0 under /tmp to see if that would fix the problem?

Thyre · 2025-08-18T16:31:34Z

Hm, looks like ld.gold is broken for those older toolchains on jsc-zen3, while there's still the one failing test for the third LLVM. The fourth works fine.

@Thyre We can try to rebuild GCCcore/10.3.0 under /tmp to see if that would fix the problem?

I'll try a manual rebuild of GCC & LLVM in my home folder at jsc-zen3. Might take a bit though.

Thyre · 2025-08-19T06:36:41Z

@boegelbot please test @ jsc-zen3
EB_ARGS="binutils-2.36.1.eb GCCcore-10.3.0.eb binutils-2.36.1-GCCcore-10.3.0.eb LLVM-11.1.0-GCCcore-10.3.0.eb --installpath=/tmp/$USER/ecpr-23603"
CORE_CNT=16

boegelbot · 2025-08-19T06:40:11Z

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23603 EB_ARGS="binutils-2.36.1.eb GCCcore-10.3.0.eb binutils-2.36.1-GCCcore-10.3.0.eb LLVM-11.1.0-GCCcore-10.3.0.eb --installpath=/tmp/$USER/ecpr-23603" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23603 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 7704

Test results coming soon (I hope)...

Details

- notification for comment with ID 3199420047 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2025-08-19T07:34:17Z

Test report by @boegelbot
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/fa8106d60df2271c0e125982d2b53372 for a full test report.

Thyre · 2025-08-19T07:39:48Z

Looking at the failure closer again:

g++ minimal.cpp -o minimal_cpp $(llvm-config --link-static --system-libs all) failed with exit code 1 (output: /usr/bin/ld.gold: /tmp/boegelbot/ecpr-23603/software/GCCcore/10.3.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/bin/ld.gold)

Why are we using /usr/bin/ld.gold?

boegel · 2025-08-19T08:43:45Z

Looking at the failure closer again:

g++ minimal.cpp -o minimal_cpp $(llvm-config --link-static --system-libs all) failed with exit code 1 (output: /usr/bin/ld.gold: /tmp/boegelbot/ecpr-23603/software/GCCcore/10.3.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /usr/bin/ld.gold)

Why are we using /usr/bin/ld.gold?

Yeah, that sure is fishy...

There's an ld.gold in the corresponding binutils installation though:

$ ls -l $EASYBUILD_PREFIX/software/binutils/2.36.1-GCCcore-10.3.0/bin/ld.gold
-rwxr-xr-x. 2 boegelbot def-maintainers 6466368 Dec 12  2023 /project/def-maintainers/boegelbot/rocky9/zen3/software/binutils/2.36.1-GCCcore-10.3.0/bin/ld.gold

Thyre · 2025-08-19T08:50:12Z

Yeah, that sure is fishy...

There's an ld.gold in the corresponding binutils installation though:

$ ls -l $EASYBUILD_PREFIX/software/binutils/2.36.1-GCCcore-10.3.0/bin/ld.gold
-rwxr-xr-x. 2 boegelbot def-maintainers 6466368 Dec 12  2023 /project/def-maintainers/boegelbot/rocky9/zen3/software/binutils/2.36.1-GCCcore-10.3.0/bin/ld.gold

Ah, I understand why...
GCCcore and LLVM both have binutils as a builddependency. That check is ran as a sanity check though, where binutils isn't loaded anymore. So it's using the system one instead.
Can we load a module during our sanity check?

Flamefire · 2025-08-26T14:28:58Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3899
FAILED
Build succeeded for 3 out of 4 (4 easyconfigs in total)
n1678 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/f907a7f98e1605d50089084f9498852d for a full test report.

Thyre · 2025-08-26T18:54:50Z

@boegelbot please test @ jsc-zen3
EB_ARGS="--installpath=/tmp/$USER/ecpr-23603 --include-easyblocks-from-pr=3899"
CORE_CNT=16

boegelbot · 2025-08-26T19:00:10Z

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=23603 EB_ARGS="--installpath=/tmp/$USER/ecpr-23603 --include-easyblocks-from-pr=3899" EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_23603 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 7758

Test results coming soon (I hope)...

Details

- notification for comment with ID 3225353947 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2025-08-26T19:59:31Z

Test report by @boegelbot
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3899
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.6, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.21
See https://gist.github.com/boegelbot/7bfc97b6c77dce46742f53dc38b8f6c8 for a full test report.

Flamefire · 2025-08-27T13:04:45Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3899
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
n1071 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/54d0a9d4d64a5d6c0b5b54368a0288b4 for a full test report.

Flamefire · 2025-08-27T16:39:13Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3899
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
n1005.barnard.hpc.tu-dresden.de - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/9fd0f32453c38f6340dc17a138523d3c for a full test report.

Flamefire · 2025-08-28T09:27:51Z

Test report by @Flamefire
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3899
SUCCESS
Build succeeded for 4 out of 4 (4 easyconfigs in total)
n1226 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (sapphirerapids), Python 3.9.18
See https://gist.github.com/Flamefire/84723f3648c14fcd49271ce380cf3c8e for a full test report.

boegel

lgtm

boegel · 2025-09-10T09:59:09Z

Going in, thanks @Thyre!

Add explicit libxml2 and zstd dependencies to LLVM

8c028d7

github-actions bot added the change label Aug 8, 2025

Thyre changed the title ~~Add explicit libxml2 and zstd dependencies to LLVM~~ Add explicit libxml2 and zstd dependencies to LLVM Aug 8, 2025

Thyre added the bug fix label Aug 8, 2025

Drop zstd from LLVM for version 14 and below

dac5103

zstd compression was added in LLVM 15 Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>

boegel added this to the next release (5.1.2) milestone Aug 8, 2025

boegel removed the change label Aug 8, 2025

Remove changed ported to separate PRs

1ddd2cd

Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>

Thyre changed the title ~~Add explicit libxml2 and zstd dependencies to LLVM~~ {compiler}[GCCcore/10.3.0--11.3.0] Add explicit libxml2 and zstd dependencies to LLVM Aug 8, 2025

Thyre changed the title ~~{compiler}[GCCcore/10.3.0--11.3.0] Add explicit libxml2 and zstd dependencies to LLVM~~ {compiler}[GCCcore/10.3.0-11.3.0] Add explicit libxml2 and zstd dependencies to LLVM Aug 8, 2025

Thyre added 2021a 2021b issues & PRs related to 2021b 2022a labels Aug 8, 2025

github-actions bot added the change label Aug 8, 2025

Thyre mentioned this pull request Aug 8, 2025

add explicit enabling/disabling of libffi with custom paths in LLVM easyblock easybuilders/easybuild-easyblocks#3849

Merged

Add explicit libffi dependency

77e8c24

Signed-off-by: Jan Andre Reuter <jan@zyten.de>

Thyre changed the title ~~{compiler}[GCCcore/10.3.0-11.3.0] Add explicit libxml2 and zstd dependencies to LLVM~~ {compiler}[GCCcore/10.3.0-11.3.0] Add explicit libxml2 and libffi dependencies to LLVM Aug 9, 2025

Thyre mentioned this pull request Aug 9, 2025

{compiler}[GCCcore/12.2.0,GCCcore/12.3.0] Add libxml2/zstd/libffi dependencies to LLVM 14.x/15.x/16.x #23608

Merged

boegel changed the title ~~{compiler}[GCCcore/10.3.0-11.3.0] Add explicit libxml2 and libffi dependencies to LLVM~~ {compiler}[GCCcore/10.3.0-11.3.0] Add explicit libxml2 and libffi dependencies to LLVM 11.x/12.x/14.x Aug 9, 2025

Flamefire mentioned this pull request Aug 11, 2025

Only consider explicit dependencies in LLVM easyblock easybuilders/easybuild-easyblocks#3881

Merged

Thyre mentioned this pull request Aug 19, 2025

[Regression]: LLVM sanity check may fail due to system ld being used easybuilders/easybuild-easyblocks#3891

Closed

Thyre mentioned this pull request Aug 27, 2025

Fix sanity check step of LLVM by loading binutils as extra module easybuilders/easybuild-easyblocks#3899

Merged

boegel approved these changes Sep 10, 2025

View reviewed changes

boegel merged commit 89fee9a into easybuilders:develop Sep 10, 2025
8 checks passed

boegel changed the title ~~{compiler}[GCCcore/10.3.0-11.3.0] Add explicit libxml2 and libffi dependencies to LLVM 11.x/12.x/14.x~~ add explicit libxml2 and libffi dependencies to LLVM 11.x/12.x/14.x Sep 24, 2025

Conversation

Thyre commented Aug 8, 2025 • edited by boegel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thyre commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thyre commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegelbot commented Aug 8, 2025

Uh oh!

boegelbot commented Aug 8, 2025

Uh oh!

Thyre commented Aug 8, 2025

Uh oh!

Flamefire commented Aug 8, 2025

Uh oh!

Thyre commented Aug 8, 2025

Uh oh!

Flamefire commented Aug 8, 2025

Uh oh!

Thyre commented Aug 8, 2025

Uh oh!

Flamefire commented Aug 8, 2025

Uh oh!

Thyre commented Aug 9, 2025

Uh oh!

Flamefire commented Aug 9, 2025

Uh oh!

Thyre commented Aug 9, 2025

Uh oh!

Flamefire commented Aug 11, 2025

Uh oh!

Thyre commented Aug 18, 2025

Uh oh!

boegelbot commented Aug 18, 2025

Uh oh!

boegel commented Aug 18, 2025

Uh oh!

boegelbot commented Aug 18, 2025

Uh oh!

Thyre commented Aug 18, 2025

Uh oh!

boegel commented Aug 18, 2025

Uh oh!

Thyre commented Aug 18, 2025

Uh oh!

Thyre commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegelbot commented Aug 19, 2025

Uh oh!

boegelbot commented Aug 19, 2025

Uh oh!

Thyre commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boegel commented Aug 19, 2025

Uh oh!

Thyre commented Aug 19, 2025

Uh oh!

Flamefire commented Aug 26, 2025

Uh oh!

Thyre commented Aug 26, 2025

Uh oh!

boegelbot commented Aug 26, 2025

Uh oh!

boegelbot commented Aug 26, 2025

Uh oh!

Flamefire commented Aug 27, 2025

Uh oh!

Flamefire commented Aug 27, 2025

Uh oh!

Flamefire commented Aug 28, 2025

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

boegel commented Sep 10, 2025

Thyre commented Aug 8, 2025 •

edited by boegel

Loading

Thyre commented Aug 8, 2025 •

edited

Loading

Thyre commented Aug 8, 2025 •

edited

Loading

Thyre commented Aug 19, 2025 •

edited

Loading

Thyre commented Aug 19, 2025 •

edited

Loading