busybox:uclibc (aka latest) only allows 3 DNS search paths

[thockin]

busybox:glibc works correctly, busybox:uclibc does not. :latest is currently tagged as :uclibc - it looks like this was updated today.

https://git.uclibc.org/uClibc/tree/include/resolv.h#n93

uclibc code shows support for 6

# docker run --dns=10.0.0.10 --dns-search=f1 --dns-search=f2 --dns-search=google.internal busybox:glibc nslookup metadata
Unable to find image 'busybox:glibc' locally
glibc: Pulling from library/busybox

Digest: sha256:820e88a5350fab82a8bccad595ffbcd668197abb5b4a718bfe576dd87ab60977
Status: Downloaded newer image for busybox:glibc
Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      metadata
Address 1: 169.254.169.254 metadata.google.internal



# docker run --dns=10.0.0.10 --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=google.internal busybox:glibc nslookup metadata
Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      metadata
Address 1: 169.254.169.254 metadata.google.internal



# docker run --dns=10.0.0.10 --dns-search=f1 --dns-search=f2 --dns-search=google.internal busybox:uclibc nslookup metadataUnable to find image 'busybox:uclibc' locally
uclibc: Pulling from library/busybox

Digest: sha256:51fef9ff5e7a2b6fc3bd21f4f5542cc56329a1abf5eb336613a6c1731ef681ae
Status: Downloaded newer image for busybox:uclibc
Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      metadata
Address 1: 169.254.169.254 metadata.google.internal



# docker run --dns=10.0.0.10 --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=google.internal busybox:uclibc nslookup metadata
Server:    10.0.0.10
Address 1: 10.0.0.10

nslookup: can't resolve 'metadata'

[thockin]

@tianon re jpetazzo/docker-busybox#19

[thockin]

I swear that at some point, :latest was -> :glibc because we had images built from that that showed a 2 MB base.

[thockin]

It looks like buildroot will pull http://downloads.uclibc-ng.org/releases/1.0.9/ which should be 6. I have to disappear for a while - anything you can find will be appreciated.

[tianon]

https://github.com/docker-library/official-images/commits/master/library/busybox is the history for when this image has been updated

:latest has been based on uClibc directly since docker-library/official-images@ab7d119, but was actually just a packaged version of buildroot itself before that, so it's been based on uClibc for a long time 😢

[tianon]

I'm going to go re-build the latest image on the official build server just to make sure we didn't somehow get a tag crossed or mis-built (since that does happen occasionally).

[thockin]

I SWEAR this used to work. Can we point :latest at :glibc, since it is the only one that actually seems to work?

[tianon]

The glibc variant is dynamically compiled with Debian's glibc embedded directly, so it's not a faithful representation of the spirit of "just busybox" as an image, and I'm really keen on figuring out what happened here and fixing it (especially since the :latest variant has been based on uClibc/buildroot since May 22, 2014; docker-library/official-images@164f4f7).

I'm going to spend some time working on reproducing this to see if I can figure out what's going on. In the meantime, I'd honestly recommend not to use the bare :latest tag anywhere that explicit features or shortcomings in the libc implementation are potentially show-stopping.

[thockin]

yeah, it was reported by someone, so "don't use it" is not a viable answer. :) I'm kicking off a build of busybox, too. Watch this space.

[tianon]

Oh yeah, I agree -- that'd be my suggestion, not trying to deflect. 😄 ❤️ I'm definitely keen on figuring out WTF is happening.

Is there an easy way to reproduce this publicly without using internal DNS servers? 😄 (I'm working on doing so, but if you've got a quick and easy one-liner or something that'd be helpful 😄)

[thockin]

You should just need a DNS server that you know will do a short-name resolution, and then put it at the end of your 4 dns-search flags.

[tianon]

Perfect, got it:

tianon@biers:~$ docker run --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=f4 --dns-search=google.com busybox:uclibc nslookup www
Server:    172.18.42.1
Address 1: 172.18.42.1

nslookup: can't resolve 'www'
tianon@biers:~$ docker run --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=google.com busybox:uclibc nslookup www
Server:    172.18.42.1
Address 1: 172.18.42.1

nslookup: can't resolve 'www'
tianon@biers:~$ docker run --dns-search=f1 --dns-search=f2 --dns-search=google.com busybox:uclibc nslookup www
Server:    172.18.42.1
Address 1: 172.18.42.1

Name:      www
Address 1: 2607:f8b0:4007:809::2004 lax02s21-in-x04.1e100.net
Address 2: 216.58.216.4 lax02s21-in-f4.1e100.net

Now to dig in. 😞

[thockin]

LOL, that is exactly what was going to send you

[tianon]

🤘 ❤️

[tianon]

Looks like buildroot is actually bringing in uClibc-ng, which is giving us http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/include/resolv.h#n91 (3). This isn't a new problem per-se, but it is definitely a problem, looking into what we can do at the buildroot level before I resort to hacking that value (and looking for uclibc-ng upstream bugs about this).

[tianon]

Oh no, I was looking at the wrong number; http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/include/resolv.h#n93 is correct, which is 6.

[thockin]

http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/include/resolv.h#n93 is the relevant line

[tianon]

I cloned the correct version of buildroot and ran make uclibc-patch in it after setting up the config, and I confirmed that output/build/uclibc-1.0.9/include/resolv.h has the correct value too. 😱 😞

[tianon]

Even going directly into the completed build container for busybox itself:

$ docker run -it --rm busybox:uclibc-builder
root@260a23b1d4ff:/usr/src/busybox# grep MAXDNSRCH ../buildroot/output/build/uclibc-1.0.9/include/resolv.h 
# define MAXDNSRCH      6   /* max # domains in search path */
    char    *dnsrch[MAXDNSRCH + 1]; /* components of domain to search */

[tianon]

And yet:

$ docker run -it --rm --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=f4 --dns-search=google.com busybox:uclibc-builder
root@6308be122ffe:/usr/src/busybox# ./busybox nslookup www 
Server:    172.18.42.1
Address 1: 172.18.42.1

nslookup: can't resolve 'www'

[tianon]

I don't think this has worked on :latest since 2014 (the update I referenced earlier) -- the old latest (before this repo's refactor) was also tagged as :buildroot-2014.02 which is still available and which I just tested:

tianon@biers:~$ docker run --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=f4 --dns-search=google.com busybox:buildroot-2014.02 nslookup www
Server:    172.18.42.1
Address 1: 172.18.42.1

nslookup: can't resolve 'www'

[tianon]

Testing the oldest digest I can find for busybox:latest (busybox@sha256:c451012efb6e79b9cf93f48a326a195acfcdf01cadf4271d678d03e031c214d3) isn't quite old enough, but it's interesting anyhow:

tianon@biers:~$ docker run --dns-search=f1 --dns-search=f2 --dns-search=f3 --dns-search=f4 --dns-search=google.com busybox@sha256:c451012efb6e79b9cf93f48a326a195acfcdf01cadf4271d678d03e031c214d3 nslookup www
Server:    172.18.42.1
Address 1: 172.18.42.1

nslookup: can't resolve 'www'

(found the digest via docker-library/docs@56ccc70#diff-9b603735f24f234123ee1abfb20b099bR115; Sep 18, 2015)

[thockin]

I (hackily) instrumented uclibc to printf stuff while iterating search domains; but now it works. Trying a more hermetic build, but the way this builder runs I am having a hard time applying a patch to uclibc as part of the dockerfile.

[thockin]

it's a bug in uclibc. You're right that it never could have worked. The bug is that it decrements the retry count on search-path misses in addition to server failures.

[tianon]

Ouch, is there a tracker for upstream you can submit it to? (and a link we can follow along with? 😄)

[thockin]

I sent a patch to their mailing list

[thockin]

http://mailman.uclibc-ng.org/pipermail/devel/2016-February/000775.html

[tianon]

Ah, excellent; thanks for seeing this through!

[tianon]

FYI, this has unfortunately regressed again; tracking here in #48 and upstream in https://bugs.busybox.net/show_bug.cgi?id=11161.