Add libffi as dependency of datadog-pip on arm platforms

[adrienkohlbecker]

Running the agent omnibus build raises the following error:

  [Software: datadog-pip] I | 2018-10-17T23:09:02+00:00 | Building because `datadog-agent' dirtied the cache
[GitFetcher: datadog-pip] I | 2018-10-17T23:09:02+00:00 | Cleaning existing clone
   [Builder: datadog-pip] I | 2018-10-17T23:09:02+00:00 | Starting build
   [Builder: datadog-pip] I | 2018-10-17T23:09:02+00:00 | $ "/opt/datadog-agent/embedded/bin/pip" install --disable-pip-version-check --no-cache --upgrade /root/.omnibus/src/datadog-pip/p
ip/[tuf-in-toto]
   [Builder: datadog-pip] I | 2018-10-17T23:10:11+00:00 | Execute: `"/opt/datadog-agent/embedded/bin/pip" install --disable-pip-version-check --no-cache --upgrade /root/.omnibus/src/datad
og-pip/pip/[tuf-in-toto]': 68.7528s
   [Builder: datadog-pip] I | 2018-10-17T23:10:11+00:00 | Build datadog-pip: 68.7551s
The following shell command exited with status 1:

    $ "/opt/datadog-agent/embedded/bin/pip" install --disable-pip-version-check --no-cache --upgrade /root/.omnibus/src/datadog-pip/pip/[tuf-in-toto]

Output:

    Processing /root/.omnibus/src/datadog-pip/pip
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Collecting tuf<0.12,>=0.11.2.dev1 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/b2/d8/2600772351d4da07fd4414cf6e6ea696265ac4124cc9003e21a54c75e2b3/tuf-0.11.2.dev1-py2.py3-none-any.whl (154kB)
Collecting in-toto<0.3,>=0.2.3 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/06/4d/4d32a374793f87b26a350fe659fc4abdc92a41fa9178f26af6683e40cf26/in-toto-0.2.3.tar.gz (64kB)
Collecting securesystemslib[crypto,pynacl]<0.12,>=0.11.3 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/6d/63/3fcd6b258e9717e32cf86e7cce5a04970bcc2aee4988e96b1844c250f97f/securesystemslib-0.11.3.tar.gz (70kB)
Collecting cryptography==2.3 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/79/a2/61c8625f96c8582d3053f89368c483ba62e56233d055e58e372f94a393f0/cryptography-2.3.tar.gz (449kB)
Collecting asn1crypto==0.24.0 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
Collecting cffi==1.11.5 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/e7/a7/4cd50e57cc6f436f1cc3a7e8fa700ff9b8b4d471620629074913e3735fb2/cffi-1.11.5.tar.gz (438kB)
Collecting enum34==1.1.6 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
Collecting ipaddress==1.0.22 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting pycparser==2.18 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/8c/2d/aad7f16146f4197a11f8e91fb81df177adcc2073d36a17b1491fd09df6ed/pycparser-2.18.tar.gz (245kB)
Collecting requests==2.19.1 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)
Collecting six==1.11.0 (from pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting iso8601>=0.1.12 (from tuf<0.12,>=0.11.2.dev1->pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/ef/57/7162609dab394d38bbc7077b7ba0a6f10fb09d8b7701ea56fa1edc0c4345/iso8601-0.1.12-py2.py3-none-any.whl
Collecting attrs (from in-toto<0.3,>=0.2.3->pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/3a/e1/5f9023cc983f1a628a8c2fd051ad19e76ff7b142a0faf329336f9a62a514/attrs-18.2.0-py2.py3-none-any.whl
Collecting python-dateutil (from in-toto<0.3,>=0.2.3->pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl (211kB)
Collecting pathspec (from in-toto<0.3,>=0.2.3->pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/84/2a/bfee636b1e2f7d6e30dd74f49201ccfa5c3cf322d44929ecc6c137c486c5/pathspec-0.5.9.tar.gz
Collecting subprocess32; python_version < "3" (from in-toto<0.3,>=0.2.3->pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/be/2b/beeba583e9877e64db10b52a96915afc0feabf7144dcbf2a0d0ea68bf73d/subprocess32-3.5.3.tar.gz (96kB)
Collecting colorama>=0.3.9 (from securesystemslib[crypto,pynacl]<0.12,>=0.11.3->pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/0a/93/6e8289231675d561d476d656c2ee3a868c1cca207e16c118d4503b25e2bf/colorama-0.4.0-py2.py3-none-any.whl
Collecting pynacl>1.2.0 (from securesystemslib[crypto,pynacl]<0.12,>=0.11.3->pip===18.1.tuf-in-toto)
  Downloading https://files.pythonhosted.org/packages/61/ab/2ac6dea8489fa713e2b4c6c5b549cc962dd4a842b5998d9e80cf8440b7cd/PyNaCl-1.3.0.tar.gz (3.4MB)

Error:

    Could not find a version that satisfies the requirement cffi>=1.4.1 (from versions: )
No matching distribution found for cffi>=1.4.1

After infestigation this is due to a missing libffi-dev package in the omnibus environment. Indeed, the cffi package doesn't come with pre-built ARM wheels, and building it from source requires that library.

[truthbk]

Once we verify the arm architecture conditional works, this appears to be 👌

[truthbk]

Let's hope this works! Otherwise maybe ohai['kernel']['machine'] cuts it?

[adrienkohlbecker]

The fix as written doesn't actually work, looking to remove the datadog-pip dependency for now

[adrienkohlbecker]

@truthbk I was able to fix the patch in this PR, and confirmed I can now build the agent with the datadog-pip dependency

[adrienkohlbecker]

@truthbk any chance you could give this another look?

[adrienkohlbecker]

Updated to also work on arm64 platforms

[albertvaka]

Hi Adrien! The patch looks good. Could you fix the linter issues pointed out by the CI?

[adrienkohlbecker]

Should be good now :-)

[albertvaka]

Thanks a lot!