compilation conflicts

[kfirufk]

Hi :)
I cloned this project from git, tried to compile with "make" and I received several conflicts related error messages:

EXPORT PGLOADER.SOURCES::GET-COLUMN-LIST causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.MSSQL"> between the following symbols:
    PGLOADER.SOURCES::GET-COLUMN-LIST, PGLOADER.SOURCE.MSSQL::GET-COLUMN-LIST

EXPORT PGLOADER.SOURCES::GET-COLUMN-LIST causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.MYSQL"> between the following symbols:
    PGLOADER.SOURCES::GET-COLUMN-LIST, PGLOADER.SOURCE.MYSQL::GET-COLUMN-LIST

EXPORT PGLOADER.SOURCES::GET-COLUMN-SQL-EXPRESSION causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.MSSQL"> between the following symbols:
    PGLOADER.SOURCES::GET-COLUMN-SQL-EXPRESSION,
    PGLOADER.SOURCE.MSSQL::GET-COLUMN-SQL-EXPRESSION

THREAD "main thread" RUNNING {10005205B3}>:
  EXPORT PGLOADER.SOURCES::GET-COLUMN-SQL-EXPRESSION causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.MYSQL"> between the following symbols:
    PGLOADER.SOURCES::GET-COLUMN-SQL-EXPRESSION,
    PGLOADER.SOURCE.MYSQL::GET-COLUMN-SQL-EXPRESSION

 EXPORT PGLOADER.SOURCES::FILTER-LIST-TO-WHERE-CLAUSE causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.MSSQL"> between the following symbols:
    PGLOADER.SOURCES::FILTER-LIST-TO-WHERE-CLAUSE,
    PGLOADER.SOURCE.MSSQL::FILTER-LIST-TO-WHERE-CLAUSE

EXPORT PGLOADER.SOURCES::FILTER-LIST-TO-WHERE-CLAUSE causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.SQLITE"> between the following symbols:
    PGLOADER.SOURCES::FILTER-LIST-TO-WHERE-CLAUSE,
    PGLOADER.SOURCE.SQLITE::FILTER-LIST-TO-WHERE-CLAUSE

#<THREAD "main thread" RUNNING {10005205B3}>:
  EXPORT PGLOADER.SOURCES::FILTER-LIST-TO-WHERE-CLAUSE causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.MYSQL"> between the following symbols:
    PGLOADER.SOURCES::FILTER-LIST-TO-WHERE-CLAUSE,
    PGLOADER.SOURCE.MYSQL::FILTER-LIST-TO-WHERE-CLAUSE

after i resolve these conflicts by selecting the relevant function for that database type instead of generics i get the following errors:

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {10005205B3}>:
  attempt to redefine the STRUCTURE-OBJECT class SCHEMA incompatibly with the
  current definition

#<THREAD "main thread" RUNNING {10005205B3}>:
  PGLOADER.SOURCE.SQLITE::FILTER-LIST-TO-WHERE-CLAUSE already names an ordinary
  function or a macro.

#<THREAD "main thread" RUNNING {10005205B3}>:
  PGLOADER.SOURCE.MSSQL::FILTER-LIST-TO-WHERE-CLAUSE already names an ordinary
  function or a macro.

#<THREAD "main thread" RUNNING {10005205B3}>:
  PGLOADER.SOURCE.MSSQL::GET-COLUMN-SQL-EXPRESSION already names an ordinary
  function or a macro.

 PGLOADER.SOURCE.MSSQL::GET-COLUMN-LIST already names an ordinary function or
  a macro.

 PGLOADER.SOURCE.MYSQL::FILTER-LIST-TO-WHERE-CLAUSE already names an ordinary
  function or a macro.

 PGLOADER.SOURCE.MYSQL::GET-COLUMN-SQL-EXPRESSION already names an ordinary
  function or a macro.

  PGLOADER.SOURCE.MYSQL::GET-COLUMN-LIST already names an ordinary function or
  a macro.

I tried to compile with ccl instead of sbcl and I still got conflict messages

after it finished compiling (with sbcl) and I try to run the app without any parameters I got the following:

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {10006405B3}>:
  Problem running initialization hook #<FUNCTION OPEN-FOREIGN-LIBS>:
  Undefined foreign library: CL+SSL::LIBEAY32

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE] Skip this initialization hook.
  1: [ABORT   ] Exit from the current thread.

(SB-INT:CALL-HOOKS "initialization" (#<FUNCTION OPEN-FOREIGN-LIBS>) :ON-ERROR :ERROR)

[mrPsycho]

master branch is not for production. try released versions: https://github.com/dimitri/pgloader/releases/

[whitslack]

@mrPsycho: The main README.md specifically states:

When building from sources, you should always build from the current git HEAD as it's basically the only source that is managed in a way to ensure it builds aginst current set of dependencies versions.

[Maxzor]

Bump, same issue on 26cc9ca

[dimitri]

I locally have CCL Version 1.12-dev and SBCL 1.5.2 and I can't reproduce. I'm told it's possible to reproduce with SBCL from Debian at version 1.4.16; but I don't have that locally and can't see a very easy way to play around with it. Contributions welcome, or meanwhile upgrade your copy of http://www.sbcl.org to something recent and maintained?

[sjl]

I ran into this and managed to track down what's happening. Although I've figured out why the error happens, I don't grok why the problematic line was added in the first place. If it was just a copy/paste error the fix is just deleting a single line in the Makefile, but if there's something more subtle in the build process that I don't understand then maybe the fix needs to be something more complicated.

Taking the first error as an example:

EXPORT PGLOADER.SOURCES::GET-COLUMN-LIST causes name-conflicts in
  #<PACKAGE "PGLOADER.SOURCE.MSSQL"> between the following symbols:
    PGLOADER.SOURCES::GET-COLUMN-LIST, PGLOADER.SOURCE.MSSQL::GET-COLUMN-LIST

This is saying that the PGLOADER.SOURCE.MSSQL package has a conflict between the following two symbols:

• PGLOADER.SOURCE.MSSQL::GET-COLUMN-LIST
• PGLOADER.SOURCES::GET-COLUMN-LIST

Note that the first of those two symbols is in the package itself. The second is coming from the (:use … #:pgloader.sources) inside of (defpackage #:pgloader.source.mssql …):

pgloader/src/package.lisp

Lines 783 to 785 in 3b5c29b

	(:use #:cl
	#:pgloader.params #:pgloader.utils #:pgloader.connection
	#:pgloader.sources)

Having a conflict in package X for a symbol FOO defined in both packages X and Y usually happens when you're developing and forget to :use a package, and then read a form with the un-package-qualified symbol FOO. The reader interns FOO into X, then later down the line evaluation fails because that FOO isn't the one you actually wanted. So then you fix the defpackage to :use Y and reevaluate it, but now you get the conflict because of the mistakenly-interned FOO from before.

However: looking back on the actual error, it's complaining about the export of that symbol, which is very strange. You can see this by examining the stack trace by typing backtrace at the SBCL debugger you get tossed into and noticing lines like (SB-C::EVAL-COMPILE-TOPLEVEL ((SB-IMPL::%DEFPACKAGE #1="PGLOADER.SOURCES". (Why ASDF doesn't bother showing you the problematic form in the error message is beyond me.)

If you read through package.lisp the package with the conflict (PGLOADER.SOURCE.MSSQL) isn't even defined at the point where the export of that symbol happens. This clued me in that the build process is doing something very unusual, and I started digging around in the Makefile.

The Makefile for pgloader uses quicklisp to retrieve dependencies, but it creates a fresh quicklisp directory in the build/ directory to isolate it from the rest of the system (which is a good idea). It then pulls down the dependencies (and writes a cookie file so make won't do it twice) with the following chunk of code:

pgloader/Makefile

Lines 102 to 110 in 3b5c29b

	$(LIBS): $(QLDIR)/setup.lisp
	$(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp \
	--eval '(push :pgloader-image *features*)' \
	--eval '(setf *print-circle* t *print-pretty* t)' \
	--eval '(ql:quickload "pgloader")' \
	--eval '(push "$(PWD)/" ql:*local-project-directories*)' \
	--eval '(ql:quickload "pgloader")' \
	--eval '(quit)'
	touch $@

The three lines in the middle there are the problem:

;; Quickload pgloader (the version in the quicklisp dist).
(ql:quickload "pgloader")

;; Add the repo to local-projects.
(push "$(PWD)/" ql:*local-project-directories*)

;; Quickload pgloader again, but this time the version from the git repo!
(ql:quickload "pgloader")

So two separate versions of pgloader are loaded. The version in the dist is older (I don't know how to ping down the exact commit the dist is using — I've pinged Xach on IRC to try to find out), and if we compare their package.lisp files, we can see that the symbols in question were added at some point after the dist version and before now:

sjl at ouroboros in ~/src/pgloader on master?
><((°> diff -U3 build/quicklisp/dists/quicklisp/software/pgloader-v3.6.1/src/package.lisp src/package.lisp
--- build/quicklisp/dists/quicklisp/software/pgloader-v3.6.1/src/package.lisp	2020-02-16 20:51:59.311204800 -0500
+++ src/package.lisp	2020-02-13 11:15:28.238113763 -0500

… a bunch of extra lines elided…

@@ -560,6 +566,17 @@
            #:instanciate-table-copy-object
            #:concurrency-support

+           #:filter-list-to-where-clause
+           #:fetch-columns
+           #:fetch-indexes
+           #:fetch-foreign-keys
+           #:fetch-comments
+           #:get-column-sql-expression
+           #:get-column-list
+           #:format-matview-name
+           #:create-matviews
+           #:drop-matviews
+
            ;; database cast machinery
            #:*default-cast-rules*
            #:*cast-rules*

… a bunch of extra lines elided…

So the problem is:

1. The old version of pgloader (from the dist) gets quickloaded.
   1. package.lisp is loaded. This version does not export get-column-list from pgloader.sources.
   2. mssql-schema.lisp is loaded. This version defines the function get-column-list in PGLOADER.SOURCE.MSSQL, which interns the symbol.
2. The new version of pgloader gets quickloaded.
   1. package.lisp is loaded. This version tries to export get-column-list from pgloader.sources. But this conflicts with the previously-interned symbol, causing the original error.

It looks like the extra quickload was added in this commit: 25c9378 — I don't know if this was just a copy/paste mistake, or if there's some subtle reason pgloader needs to be loaded twice. Maybe @dimitri can clarify. But removing the first (ql:quickload "pgloader") (the one that was loading the old version of the code) fixes the issue for me and seems to let me successfully build again.

[sfrost]

Thank you for looking into this and working on it! I can confirm that this fixes the build for me also. @dimitri is there any reason you can recall that we needed to have the quicklisp pgloader load done twice..?

[dimitri]

Thanks for detailed analysis! I can’t recall of any reason why I would have loaded the pgloader sources twice in the build process step. Looks like a bug/typo to me!

[popsection]

@dimitri Can this change be merged into master?

[phoe]

Seems like I've independently found the same issue and made a PR at #1091

[dimitri]

Now that #1091 is merged, closing this one.