Makefile depends on GNU core utilities ('make install' fails on OSX)

[blmorris]

The reason appears to be that the GNU and BSD versions of 'find' require different options:

bryans-computer:micropython-lib bryan$ make install
find: illegal option -- m
find: illegal option -- a
find: illegal option -- e
find: illegal option -- p
find: illegal option -- t
find: illegal option -- h
find: 1: No such file or directory

I haven't yet figured out how to modify the command to work on OSX, will continue to look into it.
Reference for BSD (OSX) find (the man page on OSX is basically the same):
https://www.freebsd.org/cgi/man.cgi?query=find(1)&sektion=
Reference for GNU find:
http://linux.die.net/man/1/find

[blmorris]

I haven't gotten it to work yet, but I now have a bit more insight as to why it isn't working on OSX.
In addition to find, xargs and cp also use different calling conventions between the GNU and BSD (OSX) versions.

• the BSD version of xargs does not accept the --no-run-if-empty option
• the BSD version of find objects when no path is specified (line 12, find -maxdepth 1 …)
• the BSD version of cp does not accept the -t <directory> option

The first two problems are easy fixes; just add the . path for find and go without the --no-run-if-empty option. The last problem is trickier, I need to find a way to construct CMD so that $(PREFIX) comes at the end of the list of source directories fed to cp by xargs
(The way it is currently constructed, if the -t option is removed and I run
make install MOD=uasyncio
then the content of PREFIX is interpreted as the source and micropython-lib/uasyncio/uasyncio/lib/ is interpreted as the destination)

PREFIX = ~/.micropython/lib

all:

# Installs all modules to a lib location, for development testing
CMD="find . -maxdepth 1 -mindepth 1 \( -name '*.py' -not -name 'test_*' -not -name 'setup.py' \) -or \( -type d -not -name 'dist' -not -name '*.egg-info' -not -name '__pycache__' \)| xargs --no-run-if-empty cp -r -t $(PREFIX)"
install:
    @mkdir -p $(PREFIX)
    @if [ -n "$(MOD)" ]; then \
        (cd $(MOD); sh -c $(CMD)); \
    else \
        for d in $$(find -maxdepth 1 -type d ! -name ".*"); do \
            echo $$d; \
            (cd $$d; sh -c $(CMD)); \
        done \
    fi

[blmorris]

I have given up for now on trying to modify the Makefile to work around the different behavior of the BSD versions of find,cp, and xargs

Instead I will just tweak the Makefile on my own system to use the gnu coreutils package installed using Macports - this package provides GNU versions of all the basic shell utilities, with each program prefixed by a 'g'.
Therefore, substitute gfind, gcp, and gxargs; make install will then copy everything over to the destination directory as it should. I won't make this into a pull request; it isn't entirely satisfactory as it requires installation of non-default software on OSX. We can probably close this for now.

[pfalcon]

Please don't close this, I guess it's worth to at least add GNU tools requirements to README.

[blmorris]

@pfalcon - Yes, of course you're right - the issue isn't really resolved, I just got frustrated with my inability to find a simple modification to make the same code work with both the BSD and GNU utilities. When I finally realized that I already had the GNU utilities installed, I was more than ready to just leave it for now.
I am renaming the issue to 'Makefile depends on GNU core utilities' to give a more specific description of the problem and to suggest how it could be resolved. I'll also write a paragraph for the README describing how to install and use the GNU utilities.
The remaining issue with cp is that the BSD version doesn't accept the cp -t <target directory> form. Without that, there needs to be a way to add the target directory specified by $(PREFIX) as the last item on the list of files generated by find … and piped to xargs. There may be a simple way to do this that works with both the GNU and BSD utilities, but I don't know what it is.

[pfalcon]

Sorry, I'm behind time on this issue (just to clarify, I'm working on many issues in uPy in parallel, and in reality it takes prioritized round-robin schedule, so some issue may wait for months).

[pfalcon]

Ok, so my idea was to add defines like "FIND = find" to Makefile, but I see that won't really help, because I used makefile to run dirty magic shell code.

Ok, if system utils are not portable enough, what do you do? You rewrite it in Python! I'd gladly accept such solution.

[blmorris]

Sounds like a better plan. Not that I know how to do it, but I'll see if there is anything in the main uPy repo to use as a guide for invoking python from make (or the other way around)

[blmorris]

Sorry, I'm behind time on this issue (just to clarify, I'm working on many issues in uPy in parallel,
and in reality it takes prioritized round-robin schedule, so some issue may wait for months).

Ha! No worries - I don't expect anyone to drop everything to solve my problems just because I can't figure them out right away.
As for this issue, once I got my workaround of using the GNU coreutils, it became clear that I just need a script that can scan the directory tree for all of the actual library files (ignoring everything else) and copy them to a destination directory. Seems like it should be straightforward to write a Python script that does that, and just have 'make install' call this script and pass any options to it. Since it seems pretty simple, it will probably only take me a few days to figure out the details ;)

[pfalcon]

Yes, everything like you write. And you probably can use https://github.com/micropython/micropython-lib/blob/master/make_metadata.py as a starting point.

[jonnor]

No motion over the last 9 years. I am pretty sure that Mac OSX builds have worked for many years already.