mtree bugs with filenames containing wildcards

[jgoerzen]

Let's say I have a directory containing these files, one of which contains an asterisk:

file*
file2.txt
file.txt
file3.txt

Further assume they are different sizes and have different content.

If, in the output from mtree -c, file* is emitted before the others, the later mtree verify will apparently match all four files using the rule file* and emit errors on the other three. I observed this in the wild.

A workaround is to manually edit the emitted mtree data and move file* to be after the other non-wildcard files; then it will properly verify.

Additionally, the support for wildcards in the mtree format is not documented that I can see, and probably should be. Or perhaps it should be removed, since AFAICT, no tools are generating mtree files intentionally using this feature.

I observe a call to fnmatch in verify.c which MAY be the cause of this issue. (I am not certain).

[bsiegert]

Isn't mtree part of the base system? Could you send this as a bug report upstream please? (https://www.netbsd.org/support/send-pr.html)

[jgoerzen]

Sure - pkg/56733 submitted. I'm not quite sure where to go with all these things; my apologies.

[sskras]

I also submitted an upstream issue (about another problem) on 2022-01-10.
Received zero response so far.

So I am not sure what is the workflow here.

[bsiegert]

@sskras If your bug report was about a different problem, then it does not belong here.

There is no SLO on replies to bug reports.

[jgoerzen]

@bsiegert So I just realized something:

There is mtree in two places:

https://github.com/NetBSD/pkgsrc/tree/trunk/pkgtools/mtree/files (which is what this issue pertains to)

and

https://github.com/NetBSD/src/tree/trunk/usr.sbin/mtree

(and of course their CVS origins)

I note that these two trees aren't identical, and in fact are rather divergent. The pkgsrc version -- this one -- is portable, and in fact I maintain mtree-netbsd for Debian based on this version.

The NetBSD/src version is not portable, but appears newer (for instance, having added -F).

I did have engagement on 56733 but did not at the time realize that we were talking about two different mtrees!

So I am very much talking about the pkgsrc one, which is https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/pkgtools/mtree/index.html . Presumably it is portable because pkgsrc itself is portable, to *BSD, Linux, and so forth.

I don't know how the NetBSD/src mtree flows into the pkgsrc mtree -- perhaps, at the moment, the answer is "it doesn't" - fair enough. All I know is that this patch fixes the issue for me:

--- verify.c.orig       2018-08-22 15:48:38.000000000 -0500
+++ verify.c    2022-03-01 09:45:04.854574448 -0600
@@ -142,9 +142,7 @@
                if (specdepth != p->fts_level)
                        goto extra;
                for (ep = level; ep; ep = ep->next)
-                       if ((ep->flags & F_MAGIC &&
-                           !fnmatch(ep->name, p->fts_name, FNM_PATHNAME)) ||
-                           !strcmp(ep->name, p->fts_name)) {
+                       if (!strcmp(ep->name, p->fts_name)) {
                                ep->flags |= F_VISIT;
                                if (compare(ep, p))
                                        rval = MISMATCHEXIT;

[jgoerzen]

Noting this from an email conversation:

There are some other mtree implementations and forks from NetBSD; for instance:

• https://github.com/archiecobbs/mtree-port (fork from pkgsrc; very old, doesn't even have -C)
• https://github.com/archiecobbs/nmtree (more current fork from pkgsrc)

Like the pkrsrc mtree, these two seem to be outdated relative to the
usr.sbin mtree.

Other implementations:

• libarchive, of course. (does not do verification)
• FreeBSD has an semi-compatible mtree. I found it has issues with
  filenames containing tildes, so I avoided it.
• https://github.com/mratajsky/libmtree (appears to originate from
  FreeBSD; apparently abandoned)
• https://github.com/eric-switzer/mtree-parse (Python)
• https://github.com/vbatts/go-mtree (re-implementation in Go)
• https://github.com/vbatts/go-mtree (Rust parser)

The NetBSD pkgsrc tree is directly used as part of the Arch Linux
packaging format; https://aur.archlinux.org/packages/nmtree for details.
They also have a "git" mtree derived from archiecobbs/nmtree as above.

So effectively NetBSD pkgsrc tree is the ultimate upstream for a number
of other projects. It is probably reasonable to point many of those to
use archiecobbs/nmtree but I thought I ought to write down all the
options here for others that may be looking.