Python 3.7 support and embedded python

README.md

@@ -3,6 +3,62 @@
 
 InstallApplications is an alternative to tools like [PlanB](https://github.com/google/macops-planb) where you can dynamically download packages for use with `InstallApplication`. This is useful for DEP bootstraps, allowing you to have a significantly reduced initial package that can easily be updated without repackaging your initial package.
 
+## Embedded Python
+As of v2.0, InstallApplications now uses its own embedded python. This is due to Apple's upcoming removal of Python2.
+
+Gurl has been updated from the Munki 3.7 release and tested with HTTPs and Basic Authentication. Further testing would be appreciate by the community.
+
+### Embedded Modules
+To help admins with their scripts, the following modules have been added:
+PyObjC (required for gurl)
+Requests (for API driven tools)
+
+Should the need come up for more modules, a PR should be made against the repo with proper justification
+
+### 2to3
+`installapplications.py` and `postinstall` have been ran through 2to3 to automatically convert for Python3 compatibility.
+
+### Building embedded python framework
+
+To reduce the size of the git repository, you **must** create your own Python. To do this, simply run the `./build_python_framework.sh` script within the repository.
+
+This process was tested on Catalina only.
+
+```
+./build_python_framework.sh
+
+Cloning relocatable-python tool from github...
+Cloning into '/tmp/relocatable-python-git'...
+remote: Enumerating objects: 20, done.
+remote: Counting objects: 100% (20/20), done.
+remote: Compressing objects: 100% (14/14), done.
+remote: Total 70 (delta 7), reused 16 (delta 6), pack-reused 50
+Unpacking objects: 100% (70/70), done.
+Downloading https://www.python.org/ftp/python/3.7.4/python-3.7.4-macosx10.9.pkg...
+
+...
+
+Done!
+Customized, relocatable framework is at ./Python.framework
+Moving Python.framework to InstallApplications payload folder
+```
+
+### Package size increases
+Unfortunately due to the embedded python, InstallApplications has significantly grown in size, from approximately 35Kb to 27.5 MB. The low size of InstallApplications has traditionally been one of it's greatest strengths, given how fragile `mdmclient` can be, but there is nothing that can be done here.
+
+### Pinning python user/root scripts to embedded Python
+Python user/root scripts should be pinned to the embedded Python framework. Moving forward, **scripts not pinned will be unsupported**.
+
+It is recommended that you run `2to3` against your scripts to make them python3 compliant.
+
+`/usr/local/bin/2to3 -w /path/to/script`
+
+Then simply update the shebang on your python scripts to pin against the InstallApplications python framework.
+
+`#!/Library/installapplications/Python.framework/Versions/3.7/bin/python3`
+
+You can find an example on how this was done by looking at InstallApplications' own `postinstall`
+
 ## MDMs that support Custom DEP
 - AirWatch
 - FileWave (please contact them for instructions)

build-info.json

@@ -10,5 +10,5 @@
         "timestamp": true
     },
     "suppress_bundle_relocation": true,
-    "version": "1.2.4"
+    "version": "2.0"
 }

build_python_framework.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+#
+# Build script for Python 3 framework for InstallApplications
+# Taken from https://github.com/munki/munki/blob/Munki3dev/code/tools/build_python_framework.sh
+
+TOOLSDIR=$(dirname $0)
+REQUIREMENTS="${TOOLSDIR}/py3_requirements.txt"
+PYTHON_VERSION=3.7.4
+CODEDIR=$(dirname "${TOOLSDIR}")
+MUNKIROOT=$(dirname "${CODEDIR}")
+
+# Sanity checks.
+GIT=$(which git)
+WHICH_GIT_RESULT="$?"
+if [ "${WHICH_GIT_RESULT}" != "0" ]; then
+    echo "Could not find git in command path. Maybe it's not installed?" 1>&2
+    echo "You can get a Git package here:" 1>&2
+    echo "    https://git-scm.com/download/mac"
+    exit 1
+fi
+if [ ! -f "${REQUIREMENTS}" ]; then
+    echo "Missing requirements file at ${REQUIREMENTS}." 1>&2
+    exit 1
+fi
+
+# clone our relocatable-python tool
+PYTHONTOOLDIR="/tmp/relocatable-python-git"
+if [ -d "${PYTHONTOOLDIR}" ]; then
+    rm -rf "${PYTHONTOOLDIR}"
+fi
+echo "Cloning relocatable-python tool from github..."
+git clone https://github.com/gregneagle/relocatable-python.git "${PYTHONTOOLDIR}"
+CLONE_RESULT="$?"
+if [ "${CLONE_RESULT}" != "0" ]; then
+    echo "Error cloning relocatable-python tool repo: ${CLONE_RESULT}" 1>&2
+    exit 1
+fi
+
+# remove existing Python.framework if present
+if [ -d "${MUNKIROOT}/payload/Library/installapplications/Python.framework" ]; then
+    rm -rf "${MUNKIROOT}/payload/Library/installapplications/Python.framework"
+fi
+
+# build the framework
+"${PYTHONTOOLDIR}/make_relocatable_python_framework.py" \
+    --python-version "${PYTHON_VERSION}" \
+    --pip-requirements "${REQUIREMENTS}" \
+    --destination "${MUNKIROOT}"
+
+# move the framework
+echo "Moving Python.framework to InstallApplications payload folder"
+mv "${MUNKIROOT}/Python.framework" "${MUNKIROOT}/payload/Library/installapplications"

generatejson.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 
 # Generate Json file for installapplications
@@ -57,30 +57,29 @@ def getpkginfopath(filename):
             elif entry.endswith('.pkg/PackageInfo'):
                 return entry
     else:
-        print "Error: %s while extracting BOM for %s" % (err, filename)
+        print("Error: %s while extracting BOM for %s" % (err, filename))
 
 
 def extractpkginfo(filename):
     '''Takes input of a file path and returns a file path to the
     extracted PackageInfo file.'''
+    cwd = os.getcwd()
+
     if not os.path.isfile(filename):
         return
     else:
         tmpFolder = tempfile.mkdtemp()
+        os.chdir(tmpFolder)
         # need to get path from BOM
         pkgInfoPath = getpkginfopath(filename)
 
         extractedPkgInfoPath = os.path.join(tmpFolder, pkgInfoPath)
-        cmd = [
-            '/usr/bin/xar',
-            '-x',
-            '-C', tmpFolder,
-            '-f', filename,
-            pkgInfoPath]
+        cmd = ['/usr/bin/xar', '-xf', filename, pkgInfoPath]
         proc = subprocess.Popen(cmd,
                                 stdout=subprocess.PIPE,
                                 stderr=subprocess.PIPE)
         out, err = proc.communicate()
+        os.chdir(cwd)
         return extractedPkgInfoPath
 
 
@@ -157,11 +156,11 @@ def main():
         elif fileExt == '.pkg':
             itemJson['type'] = itemType = 'package'
         else:
-            print 'Could not determine package type for item or unsupported: \
-            %s' % str(item)
+            print('Could not determine package type for item or unsupported: \
+            %s' % str(item))
             exit(1)
         if itemType not in ('package', 'rootscript', 'userscript'):
-            print 'item-type malformed: %s' % str(item['item-type'])
+            print('item-type malformed: %s' % str(item['item-type']))
             exit(1)
 
         # Determine the stage of the item to process - default to userland
@@ -171,7 +170,7 @@ def main():
                 itemStage = item['item-stage']
                 pass
             else:
-                print 'item-stage malformed: %s' % str(item['item-stage'])
+                print('item-stage malformed: %s' % str(item['item-stage']))
                 exit(1)
         except KeyError:
             itemStage = 'userland'
@@ -196,10 +195,10 @@ def main():
         if itemType in ('rootscript', 'userscript'):
             if itemType == 'userscript':
                 # Pass the userscripts folder path
-                itemJson['file'] = '/Library/Application Support/'\
+                itemJson['file'] = '/Library/'\
                     'installapplications/userscripts/%s' % fileName
             else:
-                itemJson['file'] = '/Library/Application Support/'\
+                itemJson['file'] = '/Library/'\
                     'installapplications/%s' % fileName
             # Check crappy way of doing booleans
             try:
@@ -209,16 +208,16 @@ def main():
                     if item['script-do-not-wait'] in ('true', 'True', '1'):
                         itemJson['donotwait'] = True
                 else:
-                    print 'script-do-not-wait malformed: %s ' % str(
-                    item['script-do-not-wait'])
+                    print('script-do-not-wait malformed: %s ' % str(
+                    item['script-do-not-wait']))
                     exit(1)
             except:
                 itemJson['donotwait'] = False
 
         # If packages, we need the version and packageid
         elif itemType == 'package':
             (pkgId, pkgVersion) = getpkginfo(filePath)
-            itemJson['file'] = '/Library/Application Support/'\
+            itemJson['file'] = '/Library/'\
                 'installapplications/%s' % fileName
             itemJson['packageid'] = pkgId
             itemJson['version'] = pkgVersion
@@ -238,10 +237,10 @@ def main():
         with open(savePath, 'w') as outFile:
             json.dump(stages, outFile, sort_keys=True, indent=2)
     except IOError:
-        print '[Error] Not a valid directory: %s' % savePath
+        print('[Error] Not a valid directory: %s' % savePath)
         exit(1)
 
-    print 'Json saved to %s' % savePath
+    print('Json saved to %s' % savePath)
 
 
 if __name__ == '__main__':

payload/Library/LaunchAgents/com.erikng.installapplications.plist

@@ -6,8 +6,8 @@
 	<string>com.erikng.installapplications</string>
 	<key>ProgramArguments</key>
 	<array>
-		<string>/usr/bin/python</string>
-		<string>/Library/Application Support/installapplications/installapplications.py</string>
+		<string>/Library/installapplications/Python.framework/Versions/3.7/bin/python3</string>
+		<string>/Library/installapplications/installapplications.py</string>
 		<string>--userscript</string>
 	</array>
 	<key>KeepAlive</key>
@@ -18,11 +18,11 @@
        <true/>
      </dict>
   </dict>
-  <key>OnDemand</key>
-  <true/>
+	<key>OnDemand</key>
+	<true/>
 	<key>StandardOutPath</key>
-	<string>/var/tmp/installapplications/installapplications.user.log</string>
+	<string>/var/log/installapplications/installapplications.user.log</string>
 	<key>StandardErrorPath</key>
-	<string>/var/tmp/installapplications/installapplications.user.log</string>
+	<string>/var/log/installapplications/installapplications.user.log</string>
 </dict>
 </plist>

payload/Library/LaunchDaemons/com.erikng.installapplications.plist

@@ -6,8 +6,8 @@
 	<string>com.erikng.installapplications</string>
 	<key>ProgramArguments</key>
 	<array>
-		<string>/usr/bin/python</string>
-		<string>/Library/Application Support/installapplications/installapplications.py</string>
+		<string>/Library/installapplications/Python.framework/Versions/3.7/bin/python3</string>
+		<string>/Library/installapplications/installapplications.py</string>
 		<string>--jsonurl</string>
 		<string>https://domain.tld</string>
 		<!-- <string>--iapath</string> -->
@@ -31,8 +31,8 @@
 	<key>RunAtLoad</key>
 	<true/>
 	<key>StandardOutPath</key>
-	<string>/var/log/installapplications.log</string>
+	<string>/var/log/installapplications/installapplications.log</string>
 	<key>StandardErrorPath</key>
-	<string>/var/log/installapplications.log</string>
+	<string>/var/log/installapplications/installapplications.log</string>
 </dict>
 </plist>