Install linux distribution-specific versions #40

p-mongo · 2018-06-27T19:53:42Z

This is needed for enterprise server versions to reference the correct libsnmp binary, for instance.

stennie · 2018-06-28T06:09:25Z

@p-mongo The Linux distro is generally useful to detect and use for both community & enterprise builds as the distro-specific MongoDB tarballs can include additional dependencies that are not in the generic 64-bit Linux legacy MongoDB builds (for example, TLS/SSL). I think it would be better to add distro detection with a fallback to generic builds if a distro tarball doesn't exist.

There may also be a more generic approach (perhaps an lsb_release incantation) to streamline detection.

Would you be interested in creating a more generic PR to detect the distro version?

Regards,
Stennie

p-mongo · 2018-06-28T15:45:18Z

OK will work on that.

p-mongo · 2018-07-12T01:58:02Z

Neither suse 12 (via evergreen) nor opensuse 13 (via https://app.vagrantup.com/bento) nor amazon linux have lsb_release on them, apparently.

p-mongo · 2018-07-12T03:45:14Z

I am only going to handle distros listed on https://www.mongodb.com/download-center/enterprise/releases/archive. Other users can send PRs to handle their systems (for example, I'm not going to do anything for Fedora seeing how rhel and centos are detected in entirely different ways).

p-mongo · 2018-07-12T03:51:28Z

Handy reference courtesy of Brian: https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/platform.rb

p-mongo · 2018-07-12T04:09:13Z

@stennie Tested this on:

redhat62
suse12
amazon
debian71

Added non-enterprise support.

I think this is usable as is. The one caveat is if there isn't an exact match between host OS and what mongodb is built for and e.g. suse11 package is installed on suse12, this isn't guaranteed to work.

stennie · 2018-07-12T10:04:26Z

Thanks @p-mongo .. looks great! Will sanity check on supported distros before merging.

Regards,
Stennie

devkev

Thanks very much for this submission! I had a look and have a few comments about how the code could be improved. Are you happy to go ahead make these changes? Most of them should be fairly straightforward. But if you don't have bandwidth then just let us know, and we should be able to wrangle them.

devkev · 2018-07-27T04:06:47Z

bin/m

+        distro_version=12 ;;
+    esac
+    distro_id=debian
+  elif test -f /etc/os-release; then


If this file is present, then this will also happen even if lsb_release gave results earlier. So I think we want something like:

if test -z "$distro_version"; then if test -f /etc/debian_version; then ... elif test -f /etc/os-release; then ... fi fi

My impression is lsb_release and os-release are mutually exclusive on systems I tested, hence if you do make this change I would advise checking what lsb_release would produce on systems that os-release currently covers.

devkev · 2018-07-27T04:08:58Z

bin/m

+    # which is not good
+    distro_version=`cat /etc/debian_version`
+    case "$distro_version" in
+      etch*)


I'm not super familiar with Debian and the exact format of this file, but perhaps slightly safer might be etch|etch/*) (and same for all the below)?

I guess, though I don't really see this making a difference in practice.

devkev · 2018-07-27T04:37:39Z

bin/m

@@ -316,13 +315,144 @@ install_bin() {
    sslbuild=$os
  fi

+  local distro_id= distro_version=
+  if lsb_release >/dev/null 2>&1; then


It's actually possible in some cut-down distros (eg. Docker Ubuntu images) for the lsb_release utility to not be installed (or the lsb-release package might have been uninstalled). On Ubuntu this causes m to download the Debian packages, which don't work (thanks to @kevinadi for finding this!).

Therefore, in the fallback section below, we should probably look for and use /etc/lsb-release before looking for /etc/debian_version and /etc/os-release.

This is surely a good thing to do, I seem to recall a suggestion to just use lsb_release hence the implementation I produced favors it.

devkev · 2018-07-27T04:50:16Z

bin/m

+    distro_id=debian
+  elif test -f /etc/os-release; then
+    # Suse, opensuse, amazon linux, centos (but not redhat)
+    distro_id=`(. /etc/os-release && echo $ID) | tr '[:upper:]' '[:lower:]'`


Not thrilled by sourcing this file, it feels like it could be an attack vector if malicious commands can be injected into it. The uppercasing also isn't necessary here, since it happens in all cases below.

So better would probably be something like:

distro_id=`sed -ne 's/^ID=//p' /etc/os-release | sed -e 's/^["'"'"']//' -e 's/["'"'"']$//' -e 's/\\$.$/\1/g'`

And similarly to parse VERSION_ID below, and also in the newly-added /etc/lsb-release fallback section above.

I agree in theory, in practice I don't think this matters hence the simpler implementation of sourcing the file.

devkev · 2018-07-27T04:56:46Z

bin/m

+
+  distro_id=`echo "$distro_id" | tr '[:upper:]' '[:lower:]'`
+  distro_version=`echo "$distro_version" | tr '[:upper:]' '[:lower:]'`
+  if test "$distro_id" = sles || test "$distro_id" = opensuse; then


A case statement is probably marginally cleaner, for these.

devkev · 2018-07-27T05:08:13Z

bin/m

+  # builds available for distributions that the latest version supports.
+  #
+  # The logic generally is to start with the correct distribution, then try
+  # one version older and one version newer. This should handle most cases


Can the version built for the newer distro ever work in the distro? My expectation is that this would basically never work, eg. I wouldn't expect the debian92 build to work in debian-8. In which case, if downloading it is doomed to fail, it would be better to get the legacy MongoDB build instead.

m currently installs rhel70 builds everywhere, if that works there are good chances of debian 9 builds installing on debian 8. I do favor older builds over newer builds.

devkev · 2018-07-27T05:09:12Z

bin/m

+    ubuntu-16*)
+      distros="ubuntu1604 ubuntu1404 ubuntu1204" ;;
+    ubuntu-18*)
+      distros="ubuntu1604 ubuntu1404 ubuntu1204" ;;


We have (or will soon have) ubuntu1804 builds, so this should be pre-pended.

devkev · 2018-07-27T05:14:00Z

bin/m

+    ubuntu-18*)
+      distros="ubuntu1604 ubuntu1404 ubuntu1204" ;;
+    ubuntu-*)
+      distros="ubuntu1604 ubuntu1404 ubuntu1204" ;;


And ubuntu1804 here also.

devkev · 2018-07-27T05:14:48Z

bin/m

+
+    amzn-*)
+      if test "$community" = 1; then
+        distros=amazon


Double-quotes.

devkev · 2018-07-27T05:58:57Z

bin/m

+    *)
+      distros="" ;;
+  esac
+


It'd be good to have a failsafe option to disable all distro-specific stuff, in case some user is having problems and just wants to get the legacy version, same as earlier versions of m. Perhaps here we can check if --legacy (or something similar like --no-distro) has been specified, and if so, set distros=""?

p-mongo · 2018-07-27T17:24:22Z

If you can make these changes that would be my preference, thanks.

devkev · 2018-07-31T01:34:20Z

Sure, no problem.

I carefully checked the version compatibility against a variety of Docker-based distros. Results summary, (alternate view). The testing mechanism was to use this check-all-mongodb-versions script:

#!/bin/bash

apt-get update
apt-get install -y openssl curl
apt-get install -y libcurl3
#apt-get install -y libcurl4
echo
echo

function runit {
    #"$v"/bin/mongod --version
    #"$v"/bin/mongod --version >/dev/null 2>&1
    output="$("$v"/bin/mongod --version 2>&1)"
}

cd /mnt/omnimongo/omnimongo/sw/linux/x86_64
for i in amazon* debian* rhel55 rhel62 rhel70 suse1* ubuntu1*; do
    for vbase in 3.0 3.2 3.4 3.6 4.0; do
        v="$(ls -dvr $i/"$vbase"*[0-9] 2>/dev/null | head -1)"
        if [ -x "$v"/bin/mongod ]; then
            if runit; then
                echo "$v YES"
            else
                echo "$v NO"
                echo "$output"
            fi
        else
            echo "$i/$vbase* -"
        fi
    done
    echo
done

and run it with:

for distro in amazonlinux:{1,2} debian:{7,8,9} centos:{5,6,7} opensuse:42.3 ubuntu:1{2,4,6,8}.04; do
	docker run --rm -v /mnt/omnimongo:/mnt/omnimongo:ro -v $PWD:/data "$distro" /data/check-all-mongodb-versions
done

Future versions tend to only work in older distros sporadically/rarely, so I've taken them out. The rhel70 default was just because there needed to be a default for Enterprise, I think (which has never had a "legacy" build). I've otherwise put in "backups" between rhel/amazon and ubuntu/debian where they seem to be reliable across MongoDB versions. I also added support for Amazon Linux 2.

@kevinadi, can you please test the updated code at https://github.com/devkev/m/tree/distros for me, and confirm that it works as expected?

p-mongo · 2018-07-31T17:27:55Z

The diff is looking good, it's clearer too except for the sed seas of quotes as would be expected.

If you are handling the ubuntu without lsb_release situation the comment saying it isn't handled should probably be removed.

devkev · 2018-08-01T01:31:17Z

Good catch, thanks - I've updated the comments accordingly.

kevinadi · 2018-08-08T06:15:18Z

I pushed a new branch on my fork at https://github.com/kevinadi/m/tree/distros which fixes some minor issues.

I tested the script and check to see if the installed mongod version matches the OS using docker images. Here are the results:

With lsb_release:

Distro	3.2.20	3.4.16	3.6.6	4.0.0	4.0.1
ubuntu-1204	debian71	generic	generic	generic	generic
ubuntu-1404	ubuntu1404	ubuntu1404	ubuntu1404	ubuntu1404	ubuntu1404
ubuntu-1604	ubuntu1604	ubuntu1604	ubuntu1604	(x) ubuntu1604	(x) ubuntu1604
ubuntu-1804	(x) ubuntu1604	(x) ubuntu1604	(x) ubuntu1604	(x) ubuntu1604	ubuntu1804
debian-7	debian71	generic	generic	generic	generic
debian-8	debian81	debian81	debian81	debian81	debian81
debian-9	(x) debian81	(x) debian81	debian92	debian92	debian92
centos-6	rhel62	rhel62	rhel62	rhel62	rhel62
centos-7	rhel70	rhel70	rhel70	rhel70	rhel70
opensuse-leap	suse12	suse12	suse12	suse12	suse12
opensuse-tumbleweed	suse12	suse12	suse12	suse12	suse12
amazonlinux-1	generic	generic	generic	generic	generic
amazonlinux-2	generic	generic	generic	generic	generic

(x) doesn't work out of the box. Requires missing mongod dependencies (libcurl3, libssl.so.1.0.0, glibc, etc.)

Amazon Linux 1 lsb_release outputs AmazonAMI, but identifies itself properly in /etc/os-release
Amazon Linux 2 lsb_release outputs n/a, but identifies itself properly in /etc/os-release

Without lsb_release:

Distro	3.2.20	3.4.16	3.6.6	4.0.0	4.0.1
ubuntu-1204	debian71	generic	generic	generic	generic
ubuntu-1404	ubuntu1404	ubuntu1404	ubuntu1404	ubuntu1404	ubuntu1404
ubuntu-1604	ubuntu1604	ubuntu1604	ubuntu1604	(x) ubuntu1604	(x) ubuntu1604
ubuntu-1804	(x) ubuntu1604	(x) ubuntu1604	(x) ubuntu1604	(x) ubuntu1604	ubuntu1804
debian-7	debian71	generic	generic	generic	generic
debian-8	debian81	debian81	debian81	debian81	debian81
debian-9	(x) debian81	(x) debian81	debian92	debian92	debian92
centos-6	generic	generic	generic	generic	generic
centos-7	rhel70	rhel70	rhel70	rhel70	rhel70
opensuse-leap	generic	generic	generic	generic	generic
opensuse-tumbleweed	generic	generic	generic	generic	generic
amazonlinux-1	amazon	amazon	amazon	(x) amazon2	(x) amazon2
amazonlinux-2	amazon	amazon	amazon	amazon2	amazon2

(x) doesn't work out of the box. Requires missing mongod dependencies (libcurl3, libssl.so.1.0.0, glibc, etc.)

Missing m requirements from official Docker images:

Distro	Missing requirements
ubuntu-1204	curl
ubuntu-1404	curl
ubuntu-1604	curl
ubuntu-1804	curl
debian-7	curl
debian-8	curl
debian-9	curl
centos-6	curl which
centos-7	curl which
opensuse-leap	curl which tar (libopenssl1_0_0)
opensuse-tumbleweed	curl which tar (libopenssl1_0_0)
amazonlinux-1	which tar gzip
amazonlinux-2	which tar gzip

@devkev @stennie please review the results above. Is there anything else that we can do to make the experience better? Some docker images provides a very bare bone installation that lacks basic tools such as which, tar, or gzip, but there's not much we can do about them.

devkev · 2018-08-08T06:39:01Z

That Amazon Linux lsb_release behaviour is really annoying.

devkev · 2018-08-08T06:59:04Z

I reviewed the commit on your branch, and left 2 comments.

kevinadi · 2018-08-09T02:16:54Z

Addressed the comments in my branch. Rerun the tests just in case. Verified that all is well and the test results are unchanged from the earlier table I posted.

devkev · 2018-08-09T04:15:27Z

Yep this looks good now, thanks! We can move forward to pushing the extra commits onto @p-mongo's branch, or just creating another PR, and then merging - assuming that this is ok with @stennie?

p-mongo · 2018-08-09T16:01:11Z

No love for debian 10?

stennie · 2018-08-14T01:52:01Z

Thanks for everyone's help testing the Linux distro pantheon! @kevinadi included all the changes from @p-mongo and @devkev in PR #40. I'm going to update the README to mention the new --legacy option and we'll ship this as a 1.5.0 release given the new distro magic.

No love for debian 10?

Debian 10 isn't GA yet, so there aren't any associated distro-specific MongoDB packages to test.

Regards,
Stennie

p added 2 commits July 10, 2018 16:06

Install debian versions on debian

d237346

Expand the cases

02dce6a

p-mongo force-pushed the debian branch from 2689193 to 02dce6a Compare July 10, 2018 21:29

p added 2 commits July 11, 2018 23:21

More distros

a9f2840

Button up

965594d

Oopsie

c8693d0

p added 3 commits July 11, 2018 23:53

Be more lenient

f9d7b5a

Handle non-enterprise versions

da67983

A wart

9a397bf

p-mongo force-pushed the debian branch from b1807d6 to 9a397bf Compare July 12, 2018 04:05

p-mongo changed the title ~~Install debian versions on debian~~ Install linux distribution-specific versions Jul 12, 2018

stennie self-requested a review July 12, 2018 10:04

devkev suggested changes Jul 27, 2018

View reviewed changes

kevinadi mentioned this pull request Aug 14, 2018

Linux distro-specific support #49

Merged

stennie closed this Aug 14, 2018

p-mongo mentioned this pull request Mar 18, 2020

Added Support to Download Enterprise Build (Binary Version) for Linux Distributions #60

Closed

Install linux distribution-specific versions #40

Install linux distribution-specific versions #40

Conversation

p-mongo commented Jun 27, 2018

stennie commented Jun 28, 2018

p-mongo commented Jun 28, 2018

p-mongo commented Jul 12, 2018 • edited Loading

p-mongo commented Jul 12, 2018

p-mongo commented Jul 12, 2018

p-mongo commented Jul 12, 2018

stennie commented Jul 12, 2018

devkev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p-mongo Jul 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p-mongo commented Jul 27, 2018

devkev commented Jul 31, 2018

p-mongo commented Jul 31, 2018

devkev commented Aug 1, 2018

kevinadi commented Aug 8, 2018

devkev commented Aug 8, 2018

devkev commented Aug 8, 2018

kevinadi commented Aug 9, 2018

devkev commented Aug 9, 2018

p-mongo commented Aug 9, 2018

stennie commented Aug 14, 2018

p-mongo commented Jul 12, 2018 •

edited

Loading

p-mongo Jul 27, 2018 •

edited

Loading