Seems like synth is not seeing a package that it built, and therefore deletes packages that need it due to "failed dependency check"

[rwv37]

Everything is building OK for me, but when I do a synth rebuild-repository, synth says that four items (which are all among those it just built) "failed dependency check" and deletes them from /var/synth/live_packages/All. Three of them are failing the check simply because they depend upon the first. But for the first (subversion), this is what 06_obsolete_packages.log says:

devel/subversion package has less dependencies than the port requires (8)
Query: devel/apr1:apr-1.7.0.1.6.1_2
textproc/expat2:expat-2.5.0
security/gnupg:gnupg-2.3.8
archivers/liblz4:liblz4-1.9.4,1
www/serf:serf-1.3.9_6
databases/sqlite3:sqlite3-3.40.1,1
textproc/utf8proc:utf8proc-2.8.0

subversion-1.14.2_2.pkg failed dependency check.

At first, I thought synth was saying I was missing one of those, but I do have all of those packages, with those same versions (both installed on my machine and also within /var/synth/live_packages/All). Then I noticed there are only seven listed, rather than the eight that the log message says. So, I went to the FreshPorts page for Subversion, which seems to show that the missing dependency is devel/gettext-runtime (for libintl.so). But again, I have the latest version of that (again, both installed on my machine and in /var/synth/live_packages/All):

# pkg version -v | grep gettext
gettext-runtime-0.21.1             =   up-to-date with index
# find . | grep gettext
./gettext-runtime-0.21.1.pkg
./gettext-tools-0.21.1.pkg

And I don't think this matters with respect to the problem I'm facing, but I also have the actual required library package on my machine, and it was installed by the appropriate port:

# locate libintl.so
/usr/local/lib/libintl.so
/usr/local/lib/libintl.so.8
/usr/local/lib/libintl.so.8.3.0
# pkg which /usr/local/lib/libintl.so
/usr/local/lib/libintl.so was installed by package gettext-runtime-0.21.1

So it seems like maybe synth can't see (or for some reason is ignoring) the gettext-runtime-0.21.1.pkg that it just built?

[jrmarino]

it's a bug in the port. the port author has to fix it.
I don't know what specifically the problem is and it's not something poudriere can detect. But I've seen dozens of these and they are always mistakes in the port makefile.

[jrmarino]

the problem is subversion port lists 8 dependencies but for some reason only 7 of those are registered in the package. So subversion will fail the dep check every single time.

[jrmarino]

often what happens is the makefile lists a file that's actually present on the system so it doesn't get built.

[jrmarino]

here's the probable bug, gettext like you mention:

NLS_CONFIGURE_OFF=		--disable-nls
NLS_USES=			gettext

so gettext is only used for NLS
but NLS must be configured off in your case.
So gettext is not used.
therefore it's not linked in.
So the makefile has 8 dependencies but only creates package with 7.

The author needs to set "NLS_USES_ON=gettext" or whatever the syntax is for adding gettext ONLY IF NLS option is set on. That's the bug.

[rwv37]

OK, great, thanks - I'll try to get in touch with the port author to let them know.

[jrmarino]

let me look a few minutes more to be sure.

[rwv37]

OK, will do. Thanks again.

[jrmarino]

can you show me the options set for the subversion package you built?
There's some kind of "pkg info" command that can show it

[jrmarino]

and do you have NLS turned off globally somehow that you know of?

[jrmarino]

I'm not so sure it's gettext anymore because I think NLS_USES only adds gettext if NLS option is set on.

[rwv37]

I'm trying to figure out how to show the options that are set, but I saw your question about NLS, and I know the answer for that: Yes, I have it turned off in LiveSystem-make.conf. Here's the whole file:

OPTIONS_UNSET=
OPTIONS_UNSET+=CUPS
OPTIONS_UNSET+=DEBUG
OPTIONS_UNSET+=DOCS
OPTIONS_UNSET+=EXAMPLES
OPTIONS_UNSET+=GUI
OPTIONS_UNSET+=IPV6
OPTIONS_UNSET+=NLS
OPTIONS_UNSET+=X11

Do you know of any reason why I shouldn't just turn it back on to try to get things working (as a short-term fix)?

[rwv37]

I think this is what you asked for?

# pkg options devel/subversion
subversion - BDB: off
subversion - DOCS: off
subversion - FREEBSD_TEMPLATE: off
subversion - GPG_AGENT: on
subversion - NLS: off
subversion - SASL: off
subversion - SERF: on
subversion - STATIC: off
subversion - SVNSERVE_WRAPPER: off
subversion - TEST: off
subversion - TOOLS: on

[jrmarino]

yes, that's what I needed.

[jrmarino]

FYI the freshports showing gettext is misleading. You can see the freshports version was built with NLS=ON so that's why it includes gettext.

[jrmarino]

This is harder than I thought.
It might be python causing it.

USES=python add is a run-depends:

so python is added a run depends that somehow isn't specified in the package manifest. Maybe.

I think pkg will list out library and run dependencies of a package. can you figure out how and display that here?

[rwv37]

I'll try. But to be clear, you want to see that list for the svn package, right?

[jrmarino]

yes, I want to see the libraries and run dependencies the svn package is asking for

[rwv37]

# pkg info -dependencies devel/subversion
subversion-1.14.2_2:
        serf-1.3.9_6
        utf8proc-2.8.0
        expat-2.5.0
        gnupg-2.3.8
        apr-1.7.0.1.6.1_2
        sqlite3-3.40.1,1
        liblz4-1.9.4,1
# pkg info --required-shlibs devel/subversion
subversion-1.14.2_2:
        libutf8proc.so.2
        libsqlite3.so.0
        libserf-1.so.1
        liblz4.so.1
        libexpat.so.1
        libaprutil-1.so.0
        libapr-1.so.0

[jrmarino]

okay, so python isn't registered as a run dependency but synth expects that it will be.
So I just looked up what "env" actually means.

so synth doesn't know this (it must be new) so it's expecting a python dependency that doesn't get included.

To test the theory, you change change the subversion "Makefile" line from
USES+= shebangfix python:3.5+,env
to
USES+= shebangfix python:3.5+,build

Then the problem will probably go away.
Synth would need to be modified to understand python:env is this is if this is what is happening.

Which also means this is NOT a port bug as I expected

[rwv37]

OK, I'll give it a shot and let you know what happens

[jrmarino]

I'm not 100% sure it is python:env causing this. I may have to really dig into what's happening in the framework - hopefully I can reproduce it. I haven't used ports in years ...

[rwv37]

OK, thanks for letting me know. BTW I've been waiting on synth status to scan the ports tree, and it just finished a moment ago. The build should only take a minute or two.

[rwv37]

No luck - same thing:

# synth rebuild-repository
Stand by, prescanning existing packages.
Stand by, recursively scanning 264 ports serially.
Scanning existing packages.
subversion-1.14.2_2.pkg failed dependency check.
py39-subversion-1.14.2_2.pkg failed dependency check.
py39-viewvc-devel-1.3.0.20230104.pkg failed dependency check.
mod_dav_svn-1.14.2.pkg failed dependency check.
Packages validated, rebuilding local repository.
Local repository successfully rebuilt

[rwv37]

Just to be explicit, the same details in 06_obsolete_packages.log, too.

[jrmarino]

okay. I might actually be glad about that.
Right now my freebsd machine is actually churning on netbsd projects (I swapped SSD out), but I do have access to a dragonfly machine. Let me see if I can reproduce it there. One I reproduce it, I can probably figure out what odd dependency is and go from there. Whatever this is, it's not trivial like I assumed.

[rwv37]

For what it's worth, I've done a manual make install clean (which I know is a synth no-no, but hey), and it seems to have worked fine.

[jrmarino]

so I was not able to reproduce using dports with the exact same profile makefile options. So I'll have to try again when I get access to the freebsd again.

[jrmarino]

I could not recreate this on freebsd either.

am I missing something?

[rwv37]

OK, I've reopened it.

Regarding "I still think INFO is getting defined somehow", yes, I was trying to say that it was happening right there in that code, but I miswrote something important and garbled the message. When I wrote:

thus causing it to add the value of my ${GPG_AGENT_INFO} to ${flags} as ${opt}.

I meant:

thus causing it to add the value of my ${GPG_AGENT_INFO} to ${flags}

That is, in that particular iteration, it set ${flags} (which had been "INFO") to "INFO /root/.gnupg/S.gpg_agent::". I don't know exactly how the syntax works there, but I do know that immediately before execution of that ${flags} += line, the value of $INFO was not set, whereas immediately after the execution of that same line, the value was equal to the value of my ${GPG_AGENT_INFO}.

[jrmarino]

1. where is GPG_AGENT_INFO defined?
2. do you have an idea why I'm not seeing it? Is ther something specific about your setup?

[rwv37]

Oh, sorry, and:

just curious - is subversion the only port that exhibited this behavior?

Yes, at least for me.

where is GPG_AGENT_INFO defined?

I have it defined during my bash setup for interactive shells (as suggested here in the gnupg documentation):

GPG_AGENT_INFO=$(gpgconf --list-dirs agent-socket | tr -d '\n' && printf ::)

do you have an idea why I'm not seeing it? Is there something specific about your setup?

I don't know. You're using an earlier version of FreeBSD; maybe something in the make system changed? Or maybe something in the Subversion makefile? Or maybe there's a bug in my bash startup, where I'm actually doing setting $GPG_AGENT_INFO for more than just interactive shells? I'll check on that.

[jrmarino]

Hmmm - so if your local environment is affecting this (probably using the local pkg program) then I need to adjust synth to erase that environment first. so that would fall into the category of a bug.

[rwv37]

It's not just synth, though - this happens when I do a raw make from the command line too. The only difference is that synth cares that make is saying svn requires print/indexinfo but simultaneously svn was not compiled to refer to print/indexinfo (it decides to delete synth because of this). Make doesn't actually care that svn isn't compiled to refer to print/indexinfo, despite the fact that make itself is what decided svn should be compiled so.

[rwv37]

Do you have $GNUPG_AGENT_INFO set on your machine? I thought I remembered seeing it in one of your screenshots earlier, but I haven't yet been able to find it again just now.

[jrmarino]

No. Anyway, that solves the mystery.
The package is built in a controlled environment, so it's built correctly.
The validation must be using the host pkg, and thus exposed to local environment, and it's corrupting check.

I'm looking for command to test this theory, I just need a few minutes.

[jrmarino]

okay, here's the test:

1. revert that change you made to bsd.port.mk
2. build devel/subversion
3. run this exactly:

env -i TERM=xterm synth rebuild-repository

I did not need to use the absolute path. Maybe you have to, not sure.
If your theory is correct, I expect it will not delete subversion this time.

[rwv37]

I don't use X. I'm not sure that that matters, but I want to point it out before deciding to issue that command just in case.

[jrmarino]

I don't think it matters. TERM just needs some definition or synth complains

[rwv37]

OK, I'll try it and let you know.

[rwv37]

While waiting for the regeneration of the flavor index, I figured I might as well try this:

[root@silicon ~]# cat ./blah
#!/usr/local/bin/bash

echo $GPG_AGENT_INFO
[root@silicon ~]#
[root@silicon ~]#
[root@silicon ~]# ./blah
/root/.gnupg/S.gpg-agent::
[root@silicon ~]# env -i ./blah

[root@silicon ~]#

[rwv37]

OK, first, control test just to be sure:

(1) synth rebuild-repository
(2) synth just-build devel/subversion
(3) synth just-build devel/subversion

Behavior same as previously:

(1) synth deletes subversion
(2) synth builds subversion
(3) synth deletes then builds subversion

And now the test you asked for:

(4) synth rebuild-repository
(5) synth just-build devel/subversion
(6) env -i TERM=xterm synth rebuild-repository

Behavior different than previously:

(4) synth deletes subversion
(5) synth builds subversion
(6) synth does not delete subversion

[jrmarino]

so that confirms your theory
I'll see if I can fix synth so you don't need this workaround.

[rwv37]

Thanks, but hold on, I'm not so sure that it's synth's problem... I'm in the middle of typing something up about this, but I just saw your comment and figured I should say something before finishing mine.

[rwv37]

So it seems like this is due to the combination of several factors, all of which are necessary for the problem to happen:

(1) Subversion's makefile using the option GPG_AGENT

(2) GNUPG-AGENT requiring (or at least suggesting, I dunno) the exported environment variable $GPG_AGENT_INFO

(3) FreeBSD's make stuff deciding that since (A) GPG_AGENT is one of the options for Subversion, and (B) the environment variable $GPG_AGENT_INFO exists, it should therefore set the variable INFO to the value of $GPG_AGENT_INFO

(4) FreeBSD's make stuff further deciding that since INFO is set, Subversion therefore requires print/indexinfo

(5) Subversion not actually needing print/indexinfo

(6) Synth deciding that since it was told Subversion needs print/indexinfo but that the Subversion executable as built does not actually need print/indexinfo, it should therefore delete Subversion

From all that, it seems to me that Synth is doing exactly what it should be doing based upon what it has been told by make and upon what making Subversion actually results in.

I gather that you're intending to change Synth so that it wipes out the local environment... but isn't that potentially problematic in the event that some port actually does require print/indexinfo?

I guess I don't know enough about all this, but I'm wondering if perhaps the actual problem is that Make shouldn't be saying Subversion requires print/indexinfo. If GNUPG-AGENT requires it, then GNUPG-AGENT's make process should say so, not Subversion's -- even if that means that Subversion requires print/indexinfo indirectly.

Am I mistaken here?

[jrmarino]

I gather that you're intending to change Synth so that it wipes out the local environment... but isn't that potentially problematic in the event that some port actually does require print/indexinfo?

yes, that's exactly what I going to do.
No, it is not problematic because INFO is now set through the port framework like it's supposed to be.

You are correct in saying it's not strictly a synth issue because GNUPG-AGENT should affect ports live as well. But the whole point of synth is to control the environment as it does during the build. Really a builder should be opened for the make queries to avoid environment corruption, but it was easier to use the host make.

So that's why I consider it a synth bug -- because it failed to control the environment.

Plus I just realized that 2 major bug fixes were never published. 2.13 needs to come out anyway.

[rwv37]

OK, great. Thanks. In the meantime, doing the workaround's no big deal.

Incidentally, just as another little point of evidence for the theory: All this did start happening right around the time that I started using GNUPG-AGENT with Subversion, so I guess that the problem (whatever it actually is at the lowest level) might have been very longstanding rather than due to something like a change in FreeBSD's make, or Subversion's makefile.

[jrmarino]

yes, it just took a special combination to hit.
So it looks like I am using the chroot make but the environment is probably bleeding through. I'll test to be sure, then work on the fix.

[jrmarino]

confirmed, it's all bleeding through.

[jrmarino]

looks good:

[jrmarino]

I'll tag this v2.13 if you can verify it works.

Go to the ports-mgmt/synth and modify the makefile by adding:
GH_TAGNAME= cc05e2d
and changing DISTVERSION to 2.13
and changing PORTREVISION to 0

then generate a new distinfo, build, and test.

[rwv37]

Great, thanks so very much!

I will definitely give it a try and let you know, but I'm not very confident that I will be able to do so tonight.

[rwv37]

Works great! I think... I noticed one thing that I'm guessing might just be an unimportant thing caused by the abnormal way I did this (i.e. directly changing Synth's Makefile), but I figured I should run it past you to be sure. After having done a synth status, synth just-build, synth rebuild-repository, and pkg upgrade:

[root@silicon /stuff/packageInfo]# pkg version -v > targets/host/installedPackages
[root@silicon /stuff/packageInfo]# grep -v up-to-date targets/host/installedPackages
synth-2.13                         >   succeeds index (index has 2.12_1)
[root@silicon /stuff/packageInfo]#

So whatever "index" this is referring to was not updated as a part of all this. I am guessing that that's expected and nonproblematic, but again, I just want to run it past you to be sure.

[jrmarino]

it's fine. The pkg program is checking the published ports index to try to determine which installed packages are out of date, and it's squawking that this port is newer than what's published. expected; not an issue.

[jrmarino]

I tagged it and published it as a release.
Would you do me a favor and contact the freebsd port maintainer for synth and ask him to update to that release?
You can close this issue too.

[rwv37]

Will do.

Once again, thanks very much!