`require` doesn't compile to anything, which interacts badly with bytecode #1268

Kodiologist · 2017-04-08T03:47:20Z

Here's an example with Python 2. If you have a Hy program foo.hy like this:

(require [hy.extra.anaphoric [ap-if]])
(print (eval '(ap-if (+ "a" "b") (+ it "c"))))

then hy foo.hy works fine and prints "abc". But if you say hyc foo.hy and then python foo.pyc, you get NameError: name 'ap_if' is not defined.

Here's why this happens. Currently, require doesn't contribute anything to the AST: its compilation method returns an empty Result. So foo.pyc ends up containing the same thing as if you compiled just (print (eval '(ap-if (+ "a" "b") (+ it "c")))). The only reason hy foo.hy works is that the macros that Hy loaded during compile-time are still hanging around during run-time when we get to the eval.

The most obvious solution to this is to make require forms produce a piece of AST that does the same thing at run-time that the form does at compile time, namely, load the requested macros. The annoying thing about this approach is that it would give byte-compiled modules a dependency that they won't need unless they use a macro inside an eval, which is not something that most programs will do. That makes me think we should do the opposite, and prevent (eval '(some-imported-macro ...)) from working at all. Then, if you want to use an imported macro in an eval, you'll need a corresponding require form in there, too.

The text was updated successfully, but these errors were encountered:

refi64 · 2017-04-08T14:29:16Z

As a side note, I'm starting the get the vibe that we need some tests specifically for hyc. This is around the third issue we've had recently involving something that either doesn't work or is just plain broken.

Kodiologist · 2017-04-08T14:30:46Z

We have some in test_bin.hy, but they're not much. However, I'm soon going to submit a PR to make Hy read and write bytecode automatically, which will make bytecode issues much more visible in general.

Importing or executing a Hy file now loads the byte-compiled version if it exists and is up to date, and if not, the source is byte-compiled after it's parsed. This change can speed up Hy a lot. Here are some examples comparing run times of the current master (491b474) to this commit, on my laptop with Python 3.6: - `nosetests --exclude='test_bin'` goes from 3.8 s to 0.7 s (a 5-fold speedup) - `hy -c '(print "hello world")` goes from 0.47 s to 0.20 s (a 2-fold speedup) - Rogue TV's startup goes from 3.6 s to 0.4 s (a 9-fold speedup) Accompanying changes include: - `setup.py` now creates and installs bytecode for `hy.core`, `hy.contrib`, and `hy.extra`. - The `hyc` command under Python 3 now creates bytecode in `__pycache__`, as usual for Python 3, instead of putting the `.pyc` right next to the source file like Python 2 does. I've removed a test of `hy.extra.anaphoric.a-if` that triggers hylang#1268 when the test file is byte-compiled and then hits some weird `macroexpand` bug or something when I try to work around that—Nose crashes when trying to produce an error message, and I can't seem to replicate the bug without Nose.

brandonwillard · 2018-09-23T19:18:44Z

I have a mostly work implementation of something like this here: https://github.com/brandonwillard/hy/blob/macro-changes/hy/compiler.py#L1197

I'll put in a PR when I get all the tests passing again (only a few to go, though).

This commit adds just enough namespacing to resolve a macro first in the Hy compilation namespace (i.e. the module assigned to the `HyASTCompiler`), then in the namespace it was defined. Namespacing is accomplished by adding a `module` attribute to `HySymbol`, so that `HyExpression`s can be checked for this definition namespace attribute and their car symbol resolved per the above. As well, a test is included that covers module-level macros (i.e. via `__macros` and `__tags__`), AST generated for `require`, and the non-local macro namespace resolution described above. Closes hylang#1268, closes hylang#1650, closes hylang#1416.

This commit adds just enough namespacing to resolve a macro first in the macro's defining module's namespace (i.e. the module assigned to the `HyASTCompiler`), then in the namespace/module it's evaluated in. Namespacing is accomplished by adding a `module` attribute to `HySymbol`, so that `HyExpression`s can be checked for this definition namespace attribute and their car symbol resolved per the above. As well, a comprehensive test has been added that covers - the loading of module-level macros - e.g. that only macros defined in the `require`d module are added - the AST generated for `require` - using macros loaded from modules imported via bytecode - the non-local macro namespace resolution described above - a `require`d macro that uses a macro `require` exclusively in its module-level namespace - and that (second-degree `require`d) macros can reference variables within their module-level namespaces. Closes hylang#1268, closes hylang#1650, closes hylang#1416.

This commit adds just enough namespacing to resolve a macro first in the macro's defining module's namespace (i.e. the module assigned to the `HyASTCompiler`), then in the namespace/module it's evaluated in. Namespacing is accomplished by adding a `module` attribute to `HySymbol`, so that `HyExpression`s can be checked for this definition namespace attribute and their car symbol resolved per the above. As well, a couple tests have been added that cover - the loading of module-level macros - e.g. that only macros defined in the `require`d module are added - the AST generated for `require` - using macros loaded from modules imported via bytecode - the non-local macro namespace resolution described above - a `require`d macro that uses a macro `require` exclusively in its module-level namespace - and that (second-degree `require`d) macros can reference variables within their module-level namespaces. There's also a test--expected to fail--for hylang#1414. Closes hylang#1268, closes hylang#1650, closes hylang#1416.

This commit adds just enough namespacing to resolve a macro first in the macro's defining module's namespace (i.e. the module assigned to the `HyASTCompiler`), then in the namespace/module it's evaluated in. Namespacing is accomplished by adding a `module` attribute to `HySymbol`, so that `HyExpression`s can be checked for this definition namespace attribute and their car symbol resolved per the above. As well, a couple tests have been added that cover - the loading of module-level macros - e.g. that only macros defined in the `require`d module are added - the AST generated for `require` - using macros loaded from modules imported via bytecode - the non-local macro namespace resolution described above - a `require`d macro that uses a macro `require` exclusively in its module-level namespace - and that (second-degree `require`d) macros can reference variables within their module-level namespaces. Closes hylang#1268, closes hylang#1650, closes hylang#1416.

Kodiologist added the bug label Apr 8, 2017

Kodiologist mentioned this issue Apr 8, 2017

Automatically read and write bytecode #1269

Merged

Kodiologist mentioned this issue Jul 3, 2017

[hy version 0.13.0] macro import error causes module with imported macros output incorrect compiled byte code #1316

Closed

Kodiologist changed the title ~~When a program is byte-compiled, require doesn't make macros available at run-time~~ require doesn't compile to anything, which interacts badly with bytecode Jul 3, 2017

gilch mentioned this issue Jul 15, 2017

.pyc files don't get recreated when they use a macro from a file that was changed #1324

Closed

Kodiologist mentioned this issue Sep 8, 2017

Consider moving macros to a __macros__ object in each module #1416

Closed

Kodiologist mentioned this issue Jun 29, 2018

require and macro dependencies #1650

Closed

brandonwillard mentioned this issue Aug 20, 2018

New patch importer #1672

Merged

brandonwillard mentioned this issue Sep 24, 2018

Macro processing updates and fixes #1682

Merged

3 tasks

Kodiologist closed this as completed in 010986e Nov 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`require` doesn't compile to anything, which interacts badly with bytecode #1268

`require` doesn't compile to anything, which interacts badly with bytecode #1268

Kodiologist commented Apr 8, 2017 •

edited

Loading

refi64 commented Apr 8, 2017

Kodiologist commented Apr 8, 2017

brandonwillard commented Sep 23, 2018

require doesn't compile to anything, which interacts badly with bytecode #1268

require doesn't compile to anything, which interacts badly with bytecode #1268

Comments

Kodiologist commented Apr 8, 2017 • edited Loading

refi64 commented Apr 8, 2017

Kodiologist commented Apr 8, 2017

brandonwillard commented Sep 23, 2018

`require` doesn't compile to anything, which interacts badly with bytecode #1268

`require` doesn't compile to anything, which interacts badly with bytecode #1268

Kodiologist commented Apr 8, 2017 •

edited

Loading