Firefox 68.x won't run (from inside docker) #104
Comments
|
I'll have to try that image later to see what's going on. There's no problem with docker, per se, however. See #93 (comment) for an example of some steps to run |
|
In issue #93 it runs with xvfb instead of headless. Here is some more logging: what is odd is that the same test runs fine on my mac (e.g. not in docker) |
|
@birtles |
|
There were a few changes to startup discussed on dev-platform in March but most of those seemed to land in Firefox 67 (e.g. bug 1535144 and bug 1535021). The next step is probably to look through bugs in the Startup and Profile System component that landed in the 68 timeframe. Maybe @Mossop might have some pointers about where to start. |
|
@birtles but why does it work fine outside of docker on my Mac? |
|
If it is working with xvfb but not headless then this is likely a headless widget issue, probably accidentally depending on a windowing system existing somwhere. @brendandahl might have ideas.
Inside docker you have no windowing system and are presumably running the linux version of Firefox. Outside you have a windowing system and are running the osx version. Those are two very different situations. |
|
I'm not familiar with karma-runner, but is there a way to capture the startup output of Firefox in the docker image? That would probably tells us where it's crashing. Alternatively, you could just try to start firefox inside your docker image in headless mode |
|
@brendandahl see comment above for debug log output |
|
It doesn't look like that is capturing all of stdout/stderr from Firefox. There should hopefully be a message |
|
@davidkarlsen I tried debugging your image today but it seems to work for me (had to disable the proxy however). Steps I ran: |
|
How strange - I'll do some other testing myself. |
|
OK, on my desktop (MacOS) in runs fine, on CI server it stalls at: it runs docker: |
|
A bit impatient, it got further but eventually failed: |
|
So, just to check, on your desktop you're also running it inside docker, right? I thought docker was supposed to help creating a reproducible environment ;) That's certainly an older version of docker but I wonder if CI is setting environment variables or otherwise interacting with the container in a way that might affect the runtime environment. I've seen karma-firefox-launcher just not respond and timeout when it can't find the firefox binary so if CI is clobbering your |
|
I had the same problem with Firefox 68.0.1, but 68.0.2 seems to work. Edit: I should mention that I already had the shm set to 2gb. 68.0.1 failed to launch, 68.0.2 seems to be ok. |
|
@birtles yes, docker on desktop too, CI does not alter env. |
|
If the same image running the same steps is producing different results, then there must be something different about the environment, even if it's the docker version, network speed, etc. but without access to your CI setup I'm not sure there's much I can do to help other than dig through changes made in 68. If 68.0.2 is indeed working then that might provide a clue. Digging through the changes since 68.0.1, the most obvious candidate is Bug 1567614 but that should only affect Windows as far as I can tell. |
|
68.0.2 fails as well, 67.0.4 works fine - it is definitely tied to ffox version. |
|
Note for anyone testing the images: the |
|
@glandium Are you aware of any changes to launching Firefox on Debian introduced in 68? This is about |
|
Not aware of anything. |
|
The thing I notice is that it just seems incredibly slow - it hogs CPU and runs on close to 100% CPU but still takes an age to get running - and fails due to this it seems. Was anything changed in use of libraries, or other capabilities docker might slow down? |
|
I am also seeing this: |
Oh, that's super helpful. So know we now that at least it is finding the binary. Digging into related issues, this seems really promising: Specifically, it points to problems with shared memory affecting firefox running in docker containers and includes:
The solution seems to be running docker with See:
Ultimately, though it seems like this is tracked by: |
|
I have also seen those issues and tried running with |
|
Ok, I suggest trying some of those other solutions. I don't think there's anything more we can do here for now. |
|
Would someone be able to try a debug build of Firefox? |
|
|
hang on! |
|
Yes - that version consistently works OK! |
|
That's unfortunate, I was hoping we could get a backtrack from where it's crashing. |
|
Hm - seems it is still flaky - got a few green passes yesterday - but today it broke again: should there be more debug? |
|
I'd expect more output from a debug build, they're pretty noisy. Are you logging stdout and stderr? |
|
I run it as |
|
Hello, Same problem for us, all was fine during a lot of monthes and since previous friday (13/09) karma fails with the |
|
This is basically blocked by Firefox bug 1464690. I expect something else in your config caused you to start hitting out-of-memory conditions triggering this bug. |
I don't understand.
This is the correct solution (why is it not the default in the first place?). Anything else will be a gross hack and workaround in the browser basically duplicating a pile of glibc code to put shared memory exactly in the wrong place. Is there a reason why fixing this bug in docker is not possible? I'm considering whether it's possible to make Firefox spew out warnings when it detects this docker misconfiguration. But if the above docker switch doesn't help, that's no use. |
That's just saying we're waiting to see what the outcome is of the analysis in that bug (and bug 1245239) to know if there's anything that can be done on the karma-firefox-launcher side. Comment 22 in particular suggested the handling there may change. I think the warning for a docker misconfiguration would be helpful. We have one report that |
Yes, anything can change but that won't help Firefox 68.x -> 72.x who allocate shared memory in the way a Linux application should, i.e. by using shmem_open() which will create a mapping in /dev/shm. jld told me he's restarting memfd_create() work, so I'll hold off to see if we can get that working (Chromium seems to have given up on it, hmmm...). But again, as long as docker/kubernetes misconfigure /dev/shm, it's going to be misery for applications that use shared memory. The fix here is to fix the misconfiguration, not hope that a potential Firefox change ends up working around the brokenness. |
|
Facing the same problem with gitlab-ci using firefox-esr 68 from debian buster. Setting |
|
Facing the same problem with Firefox 68+ and karma-firefox-launcher 1.3 |
|
I am running pretty much the same setup and receiving the same error, with firefox 77. Although, it works if the docker container is run as a root user. I am aware of the fact that running containers as root is a bad idea. I have tried setting Environment When I run as a root user, I see these lines the debug log When the container is not run as a root user, this is what I see |
Karma Firefox launcher ikssue: karma-runner/karma-firefox-launcher#104
Karma Firefox launcher ikssue: karma-runner/karma-firefox-launcher#104
Karma Firefox launcher ikssue: karma-runner/karma-firefox-launcher#104
|
We keep getting questions about this, so I want to make this very clear: the problem is that docker by default misconfigures shared memory. You have to fix this in the docker config by specifying See also: #104 (comment) MOZ_FORCE_DISABLE_E10S=true only hides the problem, it will disable multiprocessing, ruining your performance and won't even work in recent Firefox versions, so it will just break on you. Do not do this. Fix the Docker config. |
Can you try the fix that has been indicated? If you're sure your Docker instance has sufficient shared memory, it might be worth filing a separate issue to look into the non-root problem. This one is far too generic ("won't run") and the majority of the discussion revolves around the SHM size bug. |
|
I've mounted the /dev/shm to the host directory via docker-compose but I'm still getting this issue. See the /dev/shm inside of the docker container should have enough space: Here is the output I get (Click to expand)[... Test] Test Suite If I run the command with Any suggestions? |
There's nothing more in your log than just the message that the connection is reset. Hard to debug what's up just from that. Something that catches my attention is that you're running a 3.10 kernel, which dates back to 2013. I don't know the reasons for that, but more recent kernels will definitely improve performance. |
|
Firefox 84 (current Nightly) and later will bypass /dev/shmem for shared memory allocation on a new enough kernel [1] and thus end up working around the reports in this bug that were caused by improperly sized shared memory under docker/kubernetes/etc. It is a large enough change that it doesn't look like it can make it into an ESR 78 update though. [1] memfd support was added in kernel 3.17, released in October 2014. |
|
Firefox 84 with the workarounds is now shipping in release. |
|
I haven't heard of any new reports of this and various fixes have landed in Firefox as mentioned by @gcp (notably bug 1440203 in Firefox 84 and also bug 1582954 in Firefox 80). Those fixes should now be in Firefox ESR 91 and 102 too so I'm closing this for now. |
I've used FirefoxHeadless inside docker just fine for a long time - but with v68 of ffox it just won't work.
Karma config:
Image I am using:
https://hub.docker.com/r/evryfs/node-dev-docker/ (node12 tag)
The text was updated successfully, but these errors were encountered: