Simple file import system

[OAGr]

It seems like it's fairly important to have some sort of simple file import system.

One tricky thing is that the import system clearly requires both interpreter work, but also some command line interface / bash work. I'm not used to these things, so am guessing on what a good way would be for this.

Key Steps

My guess is something like:

1. People write imports like, import AIModel from "models/ai_model_1.squiggle", at the top of their Squiggle files. All file paths are absolute, from the root of the main directory (for simplicity, I'm not sure though).
2. The Squiggle parser parses import statements.
3. The CLI can call the parser and get the imports() of each file, without needing to run() each file. This would allow it to build up an import tree/graph. Maybe there's a parseAndListImports command.
4. When the CLI calls run() on a file, it can pass in the import values. Something like,
   run(squiggleString, bindings, environment, jsonImports, Imports: [{fileName: "foo/bar.squiggle", contents: "..."}]). We rename the current imports to jsonImports. (Or maybe we say staticImports vs. dynamicImports or something)
5. When these values are passed in, they will be treated just like code on the top of the file, but wrapped into a library/record for a namespace.

One question is how to handle situations where A imports B which imports C. In this case, either all of that information could be brought into squiggleImports, or the CLI could be responsible for packaging up everything in A into B, before passing into C.

CLI Work

The CLI would have a specific process for doing all of this work. First, it would have a watcher to check:

1. Do all .squiggle files parse properly?
2. Is there a dependency cycle?
3. Start compiling files in order of their hierarchy.

Bundling

We might want to make it possible to bundle a squiggle file; replacing all the imports with the actual code. This would make it easier to copy+paste the results into other systems. The process is pretty simple.

[umuro]

My idea is this:

• A preprocessor parses the file import statements and makes a call-out. The preprocessor is parseAndListImports above.
• The call-out gets a file path array and answers with an array of text
• Then either the callout or the preprocessor prepends a file directive to each source-code
• All is appended and given to the squiggle language. Because of the file directives squiggle can use the correct file name in errors.

@import /a/b/c //for preprocessor
@file /a/b/c //for squiggle

Cons:

• Doing it outside the existing parser makes sure that nothing is destabilized.-

• It is trivial to add a file directive to the parser

• In terms of peggy this is 5 lines of code for all directives

• The call-out can be fully unit tested and we can be sure that all is working even before integration

• Because the actual squiggle parser is working on the whole source code, each imported file can make use of / depend on definitions made in previous files.

• Compiling above is already achieved.

• I don't suggest running each imported file one by one as they are imported. Much more memory consuming. And file directives solve the error reporting with correct file name problem anyway

• If imported files can have other import directives (dangerously) then preprocessor we needs to keep track of files loaded not to continue a cycle. The Call-out can assume that responsibility also.

[umuro]

run squiggle -> preprocessor -> callout (loads files) -> (go to preprocessor until there are no loads) -> squiggle parser -> execute

Uh, the best part is that I can write and unit test the preprocessor using a dummy callout. And, repeating, without touching the existing squiggle code ever (except filenames in error reports).

[umuro]

Differences from Ozzie's initial description

• I am not changing the run command. All imports are preprocessor directives
• The CLI is not making a run call and passing the imports.
• Existing parser does not change
• There is no file by file run. All source code is constructed first.
• I believe this way the changes are contained and without side effects.

[OAGr]

Thanks for the take here! Some questions:

"makes a call-out"

-> Does "call-out" here mean a "callback"?

I don't suggest running each imported file one by one as they are imported

-> When you say "run"; do you mean like "run all the math", or just "load the text of the imports"?

[OAGr]

I realized that what we really have now is a very simple bundler, not really a compiler at all. Bundling is fairly different from regular importing. At this point, I think that having good bundling support is probably the important most thing, in this area, but it would later of course be good to have imports work without full bundling.

By bundling I mean, Taking the import code and merging it into the dependant, as a new file

[berekuk]

I'm a bit confused by some of the complexity in the proposals above, feels like I'm missing something about why this is more complex than it seems, but I'll dump my impressions on this topic here anyway, hope it'll help.

Imports probably shouldn't depend on CLI / CLI code shouldn't implement importing logic, since there can be other execution environments in which imports are needed.

I believe import statements shouldn't include the raw filesystem paths, there should be an abstraction layer + probably something like SQUIGGLEPATH env variable (like it's usually done in other languages, e.g. PYTHONPATH).

I don't understand why preprocessor is needed. I imagine something like:

• Squiggle (peggy parser) turns strings into ASTs; AST can include import nodes, which could be substituted with AST from files they refer to, recursively, on the second pass over the AST. So first there's an original AST for a single file and after the second pass there's a full AST which can be turned into an expression and executed.
• For resolving source code from a module's name there could be a "resolver" object, similar to Umur's fileSpace from Reducer: Return the locations of runtime errors #763 (comment); so when parser encounters an import statement, it passes the name of the module to the resolver, which looks up a file on disk based on module's name, taking SQUIGGLEPATH in consideration, and then returns the source code for that file. The only thing CLI should do is initialize the resolver properly and pass it to run for context.
• If resolver is undefined then any attempt to import anything fails; so the old approach of running raw strings is still available, they are just limited in functionality because they don't have a context set up for doing imports.
• Every node in AST should include the location, which should include the name of the file it came from, or an object which includes both the module's abstract name and the file location on disk. This is easily achieved by passing the filename to the parse calls.

[umuro]

Notes.
I am about to make running with bindings impossible. (Leopold Aschenbrenner and Ted Suzman suffer from problems about running with bindings). Source code chaining will be the responsibility of Reducer...
Instead I will provide a runtime system that you can plugin an include handler...
And yes every parser node should/will have a location eventually

[leopoldaschenbrenner]

Thanks! Will you update the cli/explain how to do imports?

[OAGr]

Will do!

[umuro]

Of course

[mlao-pdx]

Being new to Squiggle, I'm trying to build a simple cost estimation for our IT department using the VS Code extension. How would I import my "common" functions (common.squiggle) into an actual estimate file?

[OAGr]

Hi!

In Squiggle Hub this is pretty easy. Also, on Squiggle Hub, you can make private & group accounts. You can self-host Squiggle Hub if you want.

We used to have a hacky solution for files I could dig up. We could also see about better supporting imports locally.

Do you think Squiggle Hub could work for you? If not, how comfortable are you with kind-of-janky solutions?

[mlao-pdx]

Hi,

For now, I'm up for trying a kind-of-janky solution or plain stick with copy/paste. So, don't go spelunking for me, but if you have it near, I'll give it a try.

I am using the VS Code extension to do some proof of concept/feasibility/introduction work to sell the idea of self-hosting Squiggle Hub as an end goal.

[berekuk]

VS Code imports are not implemented yet; #1332. Mostly because it's not clear how to sync up the playground webview with the state of all imported files, especially since some of them can be unsaved and exist only in VS Code memory.

Another thing you could do to run Squiggle code with imports locally is to use CLI. You won't have access to the components, so the usability would be much worse, but imports do work there:

$ cat >1.squiggle
import "./2.squiggle" as two
import "./3.squiggle" as three
two.x + three.x
$ cat >2.squiggle
export x = 2
$ cat >3.squiggle
export x = 3
$ npx squiggle run ./1.squiggle
5

[OAGr]

We have something for this now (mainly for squiggle hub), will close. Feel free to make new issues/discussions for changes/improvements!