Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A pipeline to import content into memories #55

Open
11 of 20 tasks
jkomoros opened this issue Jul 12, 2023 · 0 comments
Open
11 of 20 tasks

A pipeline to import content into memories #55

jkomoros opened this issue Jul 12, 2023 · 0 comments

Comments

@jkomoros
Copy link
Owner

jkomoros commented Jul 12, 2023

It could be as simple as a utility seed that does it (possibly using fetch). Ideally we'll need to figure out read/write access control to memories.

Memories are controlled by the user, but they also want ways to make sure that seeds they run don't read from (or espcially write to) memories that they don't intend them to. How to do that?

The engine could check the namespace matches the remote location of the memory (and let local ones read/write any ones they want)?

Seed_type import. When called, the harness shows "system" UI allowing the user to select which file to import (for example the CLI might use inquirer to list files in the data folder). Then the harness loads the file and returns it as a string. You could use a json extract to parse the content.

In web contexts, the UI would be to show a <input type='file'> and then load up the file and pass to the engine.

  • Stub out profile.import()
  • ProfileFilesystem.import()
  • seed_type import + documentation
  • In the web implementation, add an implementaion for import that uses <input type='file'>

The above I think is obsolete because we can just use fetch with a local file.

  • A cache of embeddings in profile (Add caching and intermediate results #25)
  • Make import-polymath process each bit and do a title\n\nbody
  • When memorizing something skip if already memorized
  • Add a memorize-items (meta-seed pattern)
  • Add a import-polymath
  • Document how to import a polymath instance
  • Add logging on progress for import
  • Why does the return value of import-polymath have all undefined?
  • Add a suggest-titles which selects titles and then asks for new ones
  • Fix the extract used in suggest-titles
  • remove-numbers should handle empty lines at the end (needs a Add a map and filter seed_type #46 filter)
  • Update remove-numbers to be a split/join loop with an individual remove-number
  • Add a draft-content given a title from suggest titles
  • suggest-titles should stop relying on polymath-import memory name and have some other way of detecting which memory to use (perhaps enumerating current memory names?)
  • (Add a list seed, which can have a type of memories or stores)
  • Make a thing to store a memory (ask for a name)
jkomoros added a commit that referenced this issue Jul 22, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
Not particularly secure or anything, just looking for something consistent.

Used the same one as card-web.

Part of #55. Part of #25.
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
This means that if the text has ever been asked for from that model, it will return it.

Part of #55. Part of #25.
jkomoros added a commit that referenced this issue Jul 23, 2023
Make it produce blobs of text with structured Title and Content.

Part of #55.
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
…only show up when --verbose is passed.

Part of #55.
jkomoros added a commit that referenced this issue Jul 23, 2023
It doesn't actually work because the extraction fails.

And it also hard-codes a default of `polymath-import` for main.

Part of #55.
jkomoros added a commit that referenced this issue Jul 23, 2023
…default.

Added a test for the behavior.

Part of #17.

This also fixes the `suggest-titles` seed.

Part of #55.
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 23, 2023
jkomoros added a commit that referenced this issue Jul 30, 2023
This is necessary in order to create an `enumerate` seed_type soon.

We run every memoryID and storeID through `safeName`, which is a one-way mangle of the name, so to enumerate ids we need to store them whwn we save.

Note that this will make old profiles before this change fail to load and validate. I'm not bothering to update the profile metadata version number and do upgrading machinery for this small change.
If you have a profile, delete it before running.

Part of #55.
jkomoros added a commit that referenced this issue Jul 30, 2023
It allows enumerating either all memories or all stores.

Part of #55.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant