require and macro dependencies

[brandonwillard]

What are the expectations and conventions surrounding require-ed macros and their dependencies?

For example, consider the following:

In test-module.hy

(require [hy.contrib.walk [let]])

(defmacro test-module-macro [a]
  `(let [b 1] (+ b ~a)))

In some-file.hy

(require [test-module [*]])

(test-module-macro 1)

In its current state, running some-file.hy will produce

  => let([b, 1], b + 1)

  Traceback (most recent call last):
    File "/home/bwillard/projects/code/python/hy/hy/importer.py", line 199, in hy_eval
      return eval(ast_compile(expr, "<eval>", "eval"), namespace)
    File "<eval>", line 1, in <module>
  NameError: name 'let' is not defined

This result implies that (require [test-module [*]]) does not account for the required module's requirements (i.e. let in hy.contrib.walk).

How should dependencies like this be handled? More importantly, what would it take to automatically include macro dependencies?

FYI: This question started here, where @gilch provided a lot of important considerations and an approach.

[Kodiologist]

Automatically determining which macros a piece of code depends upon is, I suspect, undecidable. But there's no reason in principle that we couldn't make require able to pull in macros that are themselves required by the named file. #1268 is related.

Generally, macro subroutines are better written as functions than macros. Unfortunately, that's not an option when you're using a preexisting macro like let. You could try using macroexpand.

[brandonwillard]

The idea of require-ing a module's requires sounds helpful (in this situation), but perhaps not scalable — among other things.

My first response to this situation involved things like macroexpand, and @gilch provided some potential shortcomings in this comment.

Given that Hy seems partly inspired by Clojure, how does it handle these things? (Unfortunately, I'm not familiar enough with Clojure to readily find that answer in its source.)

[Kodiologist]

Not scalable? Do you mean you expect there to be some kind of performance problem with allowing require to take macros that are required by the named file? Why so?

[refi64]

I doubt it'd really be any slower than Python's imports already are TBH.

[brandonwillard]

@Kodiologist, sure, but for no reason other than generic pessimism. Well, I would also include "complexity" (of code changes, comprehensibility, maintenance, etc.) into that scaling. Regardless, if there's negligible performance and/or complexity hits, then I'm all for it!

I'm walking through the require process in the debugger right now, so hopefully I'll have a better sense of what's going on (both in general and as it pertains to this issue).

[Kodiologist]

I don't anticipate a performance issue, no, and the resulting Hy programs will obviously not be more complex. I don't think the implementation will be complex, either, but figuring out how to do it could take some work.

[gilch]

Given that Hy seems partly inspired by Clojure, how does it handle these things?

Clojure namespaces ("qualifies") symbols. So if you do a syntax quote, it automatically prepends the symbol's namespace to prevent conflicts. (Special forms are immune). If we had Clojure's system, then given your example,

(defmacro test-module-macro [a]
  `(let [b 1] (+ b ~a)))

If invoked like

(macroexpand-1 '(test-module-macro q))

Would expand to something like

(hy.contrib.walk/let [test-module/b 1] (+ test-module/b q))

Let's try it in Clojure. First change namespace.

Clojure 1.6.0
user=> (ns walk)
nil

Clojure already has let built in, so let's make a new one.

walk=> (defmacro LET [& body] `(let ~@body))
#'walk/LET
walk=> (ns test)
nil
test=> (refer 'walk :only '[LET])
nil
test=> (defmacro test-macro [a]
         `(LET [b 1] (+ b ~a)))
#'test/test-macro
test=> (macroexpand-1 '(test-macro q))
(walk/LET [test/b 1] (clojure.core/+ test/b q))

Of course, this doesn't actually work in Clojure.

test=> (test-macro q)
CompilerException java.lang.RuntimeException: Can't let qualified name: test/b, compiling:(NO_SOURCE_PATH:8:1)

That's because it was almost certainly a mistake. The b should have been a gensym to avoid accidental capture. That's why Clojure's let won't accept a qualified symbol as a target. This is easy to fix.

test=> (defmacro test-macro [a]
         `(LET [b# 1] (+ b# ~a)))
#'test/test-macro
test=> (macroexpand-1 '(test-macro q))
(walk/LET [b__36__auto__ 1] (clojure.core/+ b__36__auto__ q))
test=> (test-macro 41)
42

Hy uses defmacro/g! instead of the # suffix. Also + is a special form in Hy, so it wouldn't get qualified (But it would become hy.core.shadow/+ if not used in the function posiiton.)

If you actually want to capture (like for an anaphoric macro), you can do that too.

test=> (defmacro test-macro [a]
         `(LET [~'b 1] (+ ~'b ~a)))
#'test/test-macro
test=> (macroexpand-1 '(test-macro q))
(walk/LET [b 1] (clojure.core/+ b q))
test=> (test-macro 41)
42

Of course, all this is currently Hypothetical in Hy. Heh. But you can see how a proper namespacing implementation would be easier to use than my current recommendation,

(defmacro/g! test-module-macro [a]
  `(do
     (require [hy.contrib.walk [let :as ~g!let]])
     (~g!let [~g!b 1]
       (+ ~g!b ~a))))