Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build platform-specific wheels containing libmagic #294

Open
wants to merge 38 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
ec952d7
Build platform-specific wheels containing libmagic
ddelange Sep 7, 2023
a437409
Move persmissions into job scope, remove ABI3 reference
ddelange Sep 15, 2023
4a715e2
Switch to PyPI trusted publishing
ddelange Nov 13, 2023
1adc0a5
Add CIBW_TEST_COMMAND and indent Makefile
ddelange Jan 25, 2024
20e2dc9
Merge branch 'master' of https://github.com/ahupp/python-magic into a…
ddelange Jan 25, 2024
090b1d4
Fix CI for macos
ddelange Jan 25, 2024
20d8fee
Add dependabot.yml
ddelange Jan 25, 2024
85d4422
Migrate actions/upload-artifact@v4
ddelange Jan 25, 2024
2efa36d
Ensure magic.mgc packaged in wheel gets recognised
ddelange Jan 25, 2024
0b43bc6
Add note about --no-binary to the installation instructions
ddelange Apr 1, 2024
05df4f9
Separate mac versions
ddelange Apr 4, 2024
d2972b9
Update cibuildwheel
ddelange Apr 4, 2024
e182ae1
Bump pypa/[email protected]
ddelange Apr 4, 2024
94718d5
Bump cibuildwheel docker images
ddelange Apr 11, 2024
359e007
Revert "Bump cibuildwheel docker images"
ddelange Apr 11, 2024
bb9c685
Move magic.mgc injection into Magic class
ddelange Apr 11, 2024
b0fddf3
Build on more recent cibw images
ddelange Apr 11, 2024
dc075e9
Use hls mp4 (recent libmagic only) for testing
ddelange Apr 11, 2024
144132d
Revert "Use hls mp4 (recent libmagic only) for testing"
ddelange Apr 11, 2024
fe62a26
Install from source
ddelange Apr 23, 2024
f7bbb03
Documentation and readability
ddelange Apr 24, 2024
2e6104e
Build macos wheels with maximum backwards compatibility
ddelange May 6, 2024
ca4def3
Use CIBW_SKIP
ddelange May 6, 2024
ba87ffd
Apply suggestions from code review
ddelange May 20, 2024
e112de3
Merge branch 'master' of ahupp/python-magic into abi3-wheels
ddelange May 22, 2024
eba05b6
Fix compat.py now that bundled libmagic is preferred
ddelange May 22, 2024
8381a96
Fix https://github.com/ahupp/python-magic/issues/321
ddelange May 22, 2024
9c5f955
Use sudo on ubuntu-latest in ci.yml
ddelange May 22, 2024
e6d5ed0
Fix sudo not available on windows-latest
ddelange May 22, 2024
50504a2
Merge branch 'ahupp:master' into abi3-wheels
ddelange May 22, 2024
9357f27
Add entries in CHANGELOG
ddelange May 23, 2024
53d099b
Merge branch 'master' of https://github.com/ahupp/python-magic into a…
ddelange May 26, 2024
9bf2e9c
Fix test
ddelange May 26, 2024
f7341ce
PR Suggestions
ddelange May 26, 2024
da5b330
Apply suggestions from code review
ddelange May 28, 2024
258efa4
Revert partially: fix install on Windows
ddelange May 29, 2024
3a55538
Merge branch 'master' into abi3-wheels
ddelange Jun 18, 2024
65fb61c
Apply suggestions from code review
ddelange Jun 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"
groups:
github-actions:
patterns:
- "*"
6 changes: 2 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,11 @@ jobs:
with:
python-version: ${{ matrix.python-version }}
allow-prereleases: true
- name: Copy libmagic into magic dir
run: ${{ ( startsWith( matrix.os, 'windows' ) && 'bash add_libmagic.sh' ) || 'sudo -E bash add_libmagic.sh' }}
- run: pip install --upgrade pip
- run: pip install --upgrade pytest
- run: pip install --editable .
- if: runner.os == 'macOS'
run: brew install libmagic
- if: runner.os == 'Windows'
run: pip install python-magic-bin
- run: LC_ALL=en_US.UTF-8 pytest
shell: bash
timeout-minutes: 15 # Limit Windows infinite loop.
141 changes: 141 additions & 0 deletions .github/workflows/wheels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
name: wheels

on:
pull_request:
push:
branches: master
release:
types: [released, prereleased]
workflow_dispatch: # allows running workflow manually from the Actions tab

jobs:

build-sdist:
runs-on: ubuntu-latest

env:
PIP_DISABLE_PIP_VERSION_CHECK: 1

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.x'

- run: sudo apt-get install -y libmagic1

- name: Build source distribution
run: |
pip install --upgrade setuptools wheel pip
python setup.py sdist

- uses: actions/upload-artifact@v4
with:
name: dist
path: dist/*.tar.*


build-wheels-matrix:
runs-on: ubuntu-latest
outputs:
include: ${{ steps.set-matrix.outputs.include }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.x'
- run: pip install cibuildwheel==2.17.0 # sync version with pypa/cibuildwheel below
- id: set-matrix
env:
# only mention one (trivial) python version, as py2.py3 wheels only need to be build once per arch
CIBW_PROJECT_REQUIRES_PYTHON: '==3.12.*'
# skip PyPy wheels for now, and skip i686 wheels because pytest is failing
CIBW_SKIP: pp* *i686
run: |
MATRIX_INCLUDE=$(
{
cibuildwheel --print-build-identifiers --platform linux --arch all | jq -nRc '{"only": inputs, "os": "ubuntu-latest"}' \
&& cibuildwheel --print-build-identifiers --platform macos --arch x86_64 | jq -nRc '{"only": inputs, "os": "macos-13"}' \
&& cibuildwheel --print-build-identifiers --platform macos --arch arm64 | jq -nRc '{"only": inputs, "os": "macos-14"}' \
&& cibuildwheel --print-build-identifiers --platform windows --arch x86,AMD64 | jq -nRc '{"only": inputs, "os": "windows-latest"}'
} | jq -sc
)
echo "include=$MATRIX_INCLUDE" >> $GITHUB_OUTPUT


build-wheels:
name: build ${{ matrix.only }}
needs: build-wheels-matrix
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.build-wheels-matrix.outputs.include) }}

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up QEMU
if: runner.os == 'Linux'
uses: docker/setup-qemu-action@v3

- uses: pypa/[email protected] # sync version with pip install cibuildwheel above
timeout-minutes: 10
with:
only: ${{ matrix.only }}
env:
CIBW_BUILD_VERBOSITY: 1
# add compiled libmagic to the build directory (to include in the wheel)
CIBW_BEFORE_BUILD: ${{ ( startsWith( matrix.os, 'macos' ) && 'sudo -E bash add_libmagic.sh' ) || 'bash add_libmagic.sh' }}
# build macos wheels with maximum backwards compatibility (gcc -mmacosx-version-min flag)
MACOSX_DEPLOYMENT_TARGET: ${{ ( endsWith( matrix.only, 'arm64' ) && '11.0' ) || '10.9' }}
# simple smoke test run on each wheel: this is an HLS MP4 video, only recognised in recent versions of libmagic
CIBW_TEST_COMMAND: python -c "import magic; assert magic.Magic(mime=True).from_buffer(b'\x00\x00\x00\x1cftypiso5\x00\x00\x00\x01isomiso5hlsf\x00\x00') == 'video/mp4'"

- uses: actions/upload-artifact@v4
with:
name: dist-${{ matrix.only }}
path: wheelhouse/*.whl


publish:
if: github.event_name == 'release'
needs: [build-sdist, build-wheels]
runs-on: ubuntu-latest

permissions:
contents: write # softprops/action-gh-release
id-token: write # pypa/gh-action-pypi-publish

steps:
- uses: actions/setup-python@v5
with:
python-version: 3.x

- uses: actions/download-artifact@v4
with:
path: dist/
pattern: dist-*
merge-multiple: true

- run: ls -ltra dist/

- run: pip install --upgrade python-magic --find-links ./dist

- name: Smoketest
run: python -c "import magic; magic.Magic()"

- name: Upload release assets
uses: softprops/[email protected]
with:
files: dist/*

- name: Publish package distributions to PyPI
uses: pypa/[email protected]
12 changes: 8 additions & 4 deletions CHANGELOG
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
Changes to 0.4.29:
Changes to 0.4.28:
ddelange marked this conversation as resolved.
Show resolved Hide resolved

- libmagic and magic.mgc now come bundled in the wheels on PyPI, and will be copied
into site-packages/magic along with the Python files of this library
- magic.loader.load_lib now first searches for libmagic in the same directory as the
Python files, then in the current working directory, and only then in standard paths
- magic.Magic(magic_file=...) and magic.compat.Magic.load(magic_file=...) will now
prefer "magic.mgc" in the same directory as the Python files, only if left
unspecified by the user (and the MAGIC env var is empty or not set)
- support MAGIC_SYMLINK (via follow_symlink flag on Magic constructor)
- correctly throw FileNotFoundException depending on flag

Changes to 0.4.28:

- support "magic-1.dll" on Windows, which is produced by vcpkg
- add python 3.10 to tox config
- update test for upstream gzip extensions
Expand Down
62 changes: 43 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
[![ci](https://github.com/ahupp/python-magic/actions/workflows/ci.yml/badge.svg)](https://github.com/ahupp/python-magic/actions/workflows/ci.yml)
[![Join the chat at https://gitter.im/ahupp/python-magic](https://badges.gitter.im/ahupp/python-magic.svg)](https://gitter.im/ahupp/python-magic?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

python-magic is a Python interface to the libmagic file type
identification library. libmagic identifies file types by checking
[python-magic](https://github.com/ahupp/python-magic) is a Python interface to the libmagic file type
identification library. libmagic identifies file types by checking
their headers according to a predefined list of file types. This
functionality is exposed to the command line by the Unix command
`file`.
[`file`](https://www.darwinsys.com/file/).

## Usage

Expand All @@ -31,8 +31,7 @@ will fail throw if this is attempted.
```python
>>> f = magic.Magic(uncompress=True)
>>> f.from_file('testdata/test.gz')
'ASCII text (gzip compressed data, was "test", last modified: Sat Jun 28
21:32:52 2008, from Unix)'
'ASCII text (gzip compressed data, was "test", last modified: Sat Jun 28 21:32:52 2008, from Unix)'
```

You can also combine the flag options:
Expand All @@ -45,27 +44,53 @@ You can also combine the flag options:

## Installation

The current stable version of python-magic is available on PyPI and
can be installed by running `pip install python-magic`.
This module is a simple [CDLL](https://docs.python.org/3/library/ctypes.html) wrapper around the libmagic C library.
The current stable version of python-magic is available on [PyPI](http://pypi.python.org/pypi/python-magic/)
and can be installed by running `pip install python-magic`.

Other sources:
Compiled libmagic and the magic database come bundled in the wheels on PyPI.
You can use your own `magic.mgc` database by setting the `MAGIC`
environment variable, or by using `magic.Magic(magic_file='path/to/magic.mgc')`.
If you want to compile your own libmagic, circumvent the wheels
by installing from source: `pip install python-magic --no-binary python-magic`.

- PyPI: http://pypi.python.org/pypi/python-magic/
- GitHub: https://github.com/ahupp/python-magic
For systems not supported by the wheels, pip installs from source,
requiring libmagic to be available before installing python-magic:

This module is a simple wrapper around the libmagic C library, and
that must be installed as well:
### Linux

### Debian/Ubuntu
The Linux wheels should run on most systems out of the box.

Depending on your system and CPU architecture, there might be no compatible wheel uploaded.
However, precompiled libmagic might still be available for your system:

```sh

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be beneficial to add a library installation guide for SUSE as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you provide the relevant command?

fwiw, I think mostly all linux flavours will be covered by the wheels in the PR description, so those users won't be needing the install from source instructions provided here.

Copy link

@Privat33r-dev Privat33r-dev Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that it would be
zypper install file-devel
it might as well be required to do (in vivo test is required though)
zypper install file-magic

Currently I don't have OpenSUSE at my disposal for tests and it's likely that it would be a default package. I don't promise anything, but I might find time soon-ish to test it.

# Debian/Ubuntu
apt-get update && apt-get install -y libmagic1
# Alpine
apk add --update libmagic
# RHEL
ddelange marked this conversation as resolved.
Show resolved Hide resolved
dnf install file-libs
```
sudo apt-get install libmagic1
```

### Windows

The DLLs that are bundled in the Windows wheels are compiled by @julian-r
and are hosted at https://github.com/julian-r/file-windows/releases.

For ARM64 Windows, you'll need to compile libmagic from source.

### OSX

- When using Homebrew: `brew install libmagic`
- When using macports: `port install file`
The Mac wheels are compiled with maximum backward compatibility.
For older Macs, you'll need to install libmagic from source:

```sh
# homebrew
brew install libmagic
# macports
port install file
```

If python-magic fails to load the library it may be in a non-standard location, in which case you can set the environment variable `DYLD_LIBRARY_PATH` to point to it.

Expand All @@ -78,7 +103,7 @@ If python-magic fails to load the library it may be in a non-standard location,
- 'MagicException: could not find any magic files!': some
installations of libmagic do not correctly point to their magic
database file. Try specifying the path to the file explicitly in the
constructor: `magic.Magic(magic_file="path_to_magic_file")`.
constructor: `magic.Magic(magic_file='path/to/magic.mgc')`.

- 'WindowsError: [Error 193] %1 is not a valid Win32 application':
Attempting to run the 32-bit libmagic DLL in a 64-bit build of
Expand All @@ -88,7 +113,6 @@ If python-magic fails to load the library it may be in a non-standard location,
- 'WindowsError: exception: access violation writing 0x00000000 ' This may indicate you are mixing
Windows Python and Cygwin Python. Make sure your libmagic and python builds are consistent.


## Bug Reports

python-magic is a thin layer over the libmagic C library.
Expand Down
71 changes: 71 additions & 0 deletions add_libmagic.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
#!/usr/bin/env bash

set -euxo pipefail

install_source() {
# install from source
# https://www.darwinsys.com/file/
# https://github.com/file/file/blob/FILE5_45/INSTALL#L51
(
version="file-5.45" &&
tmpfile="$(mktemp)" &&
curl -sSLo "${tmpfile}" "https://astron.com/pub/file/${version}.tar.gz" &&
Copy link

@Privat33r-dev Privat33r-dev Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried it on ubuntu's Docker, fails here :)
bash: curl: command not found

install_precompiled would work though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, the script is only intended for github actions default runner images and the cibuildwheel docker images (both have curl and make)

tar xvf "${tmpfile}" &&
Comment on lines +11 to +13

This comment was marked as resolved.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@estarfoo

Not a big deal, but if going with the piped curl output, personally I'd pair that with an explicit tar xvf -. Again, for the paranoid.

cd "${version}" &&
./configure &&
make &&
Copy link

@Privat33r-dev Privat33r-dev Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dev dependencies to use "make" and similar commands are not shipped by default in some distros, especially in those that are usually used by Docker. I'd suggest to ensure that necessary dependencies are installed first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wished set -euxo pipefail also effects commands inside function definitions. then we could remove && from every line here (and the subshell parentheses here and the || on L21), and we could test which make on top of this function or similar.

now, it simply falls back to install_precompiled (L69) when curl or make or compilation fails (error will be in output).

if make is not available, in the current setup we will only have done an unnecessary curl. not the worst but could be avoided.

any ideas?

Copy link

@Privat33r-dev Privat33r-dev Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wished set -euxo pipefail also effects commands inside function definitions. then we could remove && from every line here (and the subshell parentheses here and the || on L21), and we could test which make on top of this function or similar.

any ideas?

It appears to be doing so already. Try this (and also the same script, but remove the first line or replace exit 1 to exit 0):

set -euxo pipefail
func(){
  echo 1
  (exit 1)
  echo 2
}

func

You made me think about other topic when you mentioned that curl can be executed. There is no cleanup on any step, it might be logical to make a ZIP (or gzip, doesn't matter) file cleanup from external function and do it regardless of the outcome. Same for windows. Pseudocode:

install_stuff(){
 real_install()
 rm potential_zip
}

make install &&
make installcheck &&
cd .. &&
rm -r "${version}"
) || ( cd .. && false )
}

install_precompiled() {
# Mac https://formulae.brew.sh/formula/libmagic
# Debian https://packages.ubuntu.com/libmagic1
# Alpine https://pkgs.alpinelinux.org/package/libmagic
# RHEL https://git.almalinux.org/rpms/file
# Windows https://github.com/julian-r/file-windows
if [ -n "$(which brew)" ]; then
brew install libmagic
elif [ -n "$(which apt-get)" ]; then
apt-get update
apt-get install -y libmagic1
elif [ -n "$(which apk)" ]; then
apk add --update libmagic
elif [ -n "$(which dnf)" ]; then
dnf --setopt install_weak_deps=false -y install file-libs
else
# windows (no install, just download into current working directory)
# could also consider install using `pacman`: https://packages.msys2.org/base/mingw-w64-file
# which would require an update of copy_libmagic below to account for new magic.mgc paths
python <<EOF
import platform, sysconfig, io, zipfile, urllib.request
assert platform.system() == "Windows"
machine = "x86" if sysconfig.get_platform() == "win32" else "x64"
url = f"https://github.com/julian-r/file-windows/releases/download/v5.44/file_5.44-build104-vs2022-{machine}.zip"

This comment was marked as resolved.

print("Downloading", url)
zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(url).read())).extractall(".")
EOF
# check what was copied
ls -ltra
fi
}

copy_libmagic() {
# on cibuildwheel, the lib needs to exist in the project before running setup.py
# copy lib into the magic dir, regardless of platform
# this python command relies on current working directory containing `./magic/loader.py`
libmagic_path="$(python -c 'from magic.loader import load_lib; print(load_lib()._name)')" &&
cp "${libmagic_path}" "magic" &&
# only on linux/macos: additionally copy compiled db into magic dir (prefer the one installed by install_source)
( ( cp "/usr/local/share/misc/magic.mgc" "magic" || cp "/usr/share/misc/magic.mgc" "magic" ) || true ) &&
# check what was copied
ls -ltra magic
}

# prefer a recent build from source
install_source || install_precompiled
# files to be copied into the wheel
copy_libmagic
6 changes: 6 additions & 0 deletions magic/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,12 @@ def __init__(self, mime=False, magic_file=None, mime_encoding=False,
self.cookie = magic_open(self.flags)
self.lock = threading.Lock()

if magic_file is None and not os.environ.get("MAGIC"):
# wheels package the mime database in this directory
# prefer it when no magic file is specified by the user
mime_db = os.path.join(os.path.dirname(__file__), 'magic.mgc')
if os.path.exists(mime_db):
magic_file = mime_db
magic_load(self.cookie, magic_file)

# MAGIC_EXTENSION was added in 523 or 524, so bail if
Expand Down
Loading