Support multiple notes in one org file

[bdarcus]

Summary

Given that org-roam has removed the file-per-note restriction (see also #517), we might do the same.

Details

I'm just not sure how best to do that ATM, as I don't have a ton of experience coding and working with org structure.

There are also, I think, some design nuances to consider; for example:

If we remove this restriction, seems it should be more flexible than only a single org file. For example, maybe I want to mix single-per-file and multi-per-file (though not sure that's a great idea in org-roam given its robust linking, along with transclusion)?

Regardless, how to best configure what UX here?

OTOH, maybe if people want that sort of flexibility, they should store the note file paths in the actual record?

Counterpoint

One could reasonably argue this is not worth the hassle, and that we have a design now that works well for one-file-per-note/reference use cases, that this is increasingly what we're seeing in diverse new note-taking packages, and that users here aren't actually requesting it. For that reason, I've marked the milestone as "future" (post 1.0).

[ademberbic]

Hi @bdarcus, thank you so much for your work on this package, it's been a joy to use so far! For whatever it's worth, I recently slapped together a few functions to add this functionality to citar à la ivy-bibtex, which I'd been using before. Needless to say that it's very quick-and-dirty code, and you may well have had a completely different implementation in mind, but here it is nonetheless. Hope it proves useful, and thanks again.

[bdarcus]

Thanks @ademberbic. Based on this, do you have any input on my design questions above?

I don't actually recall how ivy-bibtex handles this.

[ademberbic]

Good question @bdarcus—I'm not at all familiar with org-roam myself, so I'm afraid I can only be of so much use here.

With ivy-bibtex and helm-bibtex, it's an either/or choice: if bibtex-completion-notes-path is a filename, it assumes one-file-for-notes; if it's a directory, it assumes one-file-per-note. Their implementation is here: if multiple keys are selected (which, out of laziness, my code assumes will never happen), it either opens all the notes files or opens an org-capture-style narrowed buffer for each, depending on the system in use.

In terms of mixing one-file-for-notes and one-note-per-file, I suppose I could imagine a setup where citar-notes-paths takes filenames for the former as well as directories for the latter: existing notes are found by scanning everything in the list for org properties holding the BibTeX key, as in ivy-bibtex; new notes are added to the first entry in the list, be it a file or a directory. But that's a pretty cludgy, convoluted set-up (e.g. what if a user wants to choose how and where a new note is stored?), and I have no idea how well it would play with org-roam.

Maybe you're right that, if a user wants that much flexibility, they should store the note path in the record itself. I can imagine situations where it'd be useful (e.g. a 'master' notes directory, and then files that collect notes on everything I'm reading for a given project)—but I can equally imagine it all getting very messy, very quick. I think this one's your call.

[bdarcus]

Good question @bdarcus—I'm not at all familiar with org-roam myself, so I'm afraid I can only be of so much use here.

Just to clarify, I use the org-roam example just because it's the most widely used package for this, and I use it.

But it's really a general question.

With ivy-bibtex and helm-bibtex, it's an either/or choice: if bibtex-completion-notes-path is a filename, it assumes one-file-for-notes; if it's a directory, it assumes one-file-per-note.

Right; that's what I was assuming. And you then can't specify more than one file?

Their implementation is here: if multiple keys are selected (which, out of laziness, my code assumes will never happen), it either opens all the notes files or opens an org-capture-style narrowed buffer for each, depending on the system in use.

So it could open the same file in multiple buffers then?

In terms of mixing one-file-for-notes and one-note-per-file, I suppose I could imagine a setup where citar-notes-paths takes filenames for the former as well as directories for the latter: existing notes are found by scanning everything in the list for org properties holding the BibTeX key, as in ivy-bibtex; new notes are added to the first entry in the list, be it a file or a directory. But that's a pretty cludgy, convoluted set-up (e.g. what if a user wants to choose how and where a new note is stored?), and I have no idea how well it would play with org-roam.

I would also worry about the performance implications of that.

Aside: I noticed that Emacs recently picked up sqlite support.

[ademberbic]

Right; that's what I was assuming. And you then can't specify more than one file?

Correct, the variable bibtex-completion-notes-path expects a string, and refuses to accept lists.

So it could open the same file in multiple buffers then?

In theory, yes, but I've just given it a try and it looks like their implementation is buggy. It spawns an org-capture-style window to edit notes, but if you pass multiple keys to bibtex-completion-edit-notes, it'll loop over them and keep re-expanding and re-narrowing the notes file until you're left looking at the final key you requested.

I imagine you could implement this properly with indirect buffers, each narrowed to the relevant subheading—I think that's how org-capture itself does it—but if each key the user selected spawns its own window, there's the question of how to lay them all out.

[alarsyo]

Hey @bdarcus, thanks a lot for citar. I'm chiming in with my use-case, maybe this will help shaping up this functionality:

I'm using a single bibliography.org file, which is inside my org-roam directory. This file contains, roughly, one Org header per bibliography entry, each node having its own Org Roam ID.

My reason for having everything in one file instead of using one file per entry is that all entries have a bibtex source block that gets tangled to a "biblio.bib" file, and Org mode doesn't tangling to a single output file from multiple Org files.

Sample of what my file looks like:

:PROPERTIES:
:ID:       d55f36aa-f97f-4978-b82d-133c2fe750d5
:END:
#+title: Thesis Bibliography
#+property: header-args :tangle thesis.bib :exports code
#+startup: hideblocks content
#+bibliography: thesis.bib

* Paper 1
:PROPERTIES:
:ID:       fad561fd-0f37-4d9c-8c91-e39637dadaca
:ROAM_REFS: @doe.06.conference
:END:
#+begin_src bibtex
@InProceedings{doe.06.conference,
  [...]
}
#+end_src

Notes on Doe '06 here...

* Paper 2
:PROPERTIES:
:ID:       cdfdafb6-12a2-491a-9538-5c834b9fec25
:ROAM_REFS: @doe.07.conference
:END:
#+begin_src bibtex
@InProceedings{doe.07.conference,
  [...]
}
#+end_src

Notes on Doe '07 here...

[alarsyo]

Currently I think I could use the trick mentioned in the README, with setting citar-open-note-function to orb-citar-edit-note, but that doesn't cover use cases like citar-open, which displays citekey.org files as note files for the corresponding entry, but shows nothing in my case because it doesn't know about my note layout in a single file.

Maybe a solution would be providing a citar-find-resources-function or something along these lines, that would allow us to find custom resources for a citekey? It would return a list, with elements of the list being a pair of:

• some text representing the "location" of the resource, so for my notes that would be something like "file:/path/to/org-roam-dir/bibliography.org::* Paper 1" (using something similar to org-mode links, though this is just here as a visual representation of the note's location to present it to the user for selection).
• a lambda that would be called on resource selection, and open the note correctly (a lambda would be necessary to call something like org-roam-node-open, maybe there are better ideas though)

[bdarcus]

Maybe a solution would be providing a citar-find-resources-function or something along these lines, that would allow us to find custom resources for a citekey?

That would likely make sense, as a generalization of the functions we use now to find related files (see citar-file--files-for-entry etc).

... some text representing the "location" of the resource, so for my notes that would be something like "file:/path/to/org-roam-dir/bibliography.org::* Paper 1"

That seems a little awkward. OR v2 uses org-ids to identify all nodes, which is at least consistent and unambiguous.

As is occurs to me now, it might be worth mentioning that if we're concerned about performance here, maybe it's time to consider a SQLite solution, as org-roam does? Emacs now has out-of-box support for SQLite, presumably for just this sort of situation.

Which makes me also wonder about something more tightly coupled to OR; say a citar-org-roam.

[bdarcus]

Hey @bdarcus, thanks a lot for citar. I'm chiming in with my use-case, maybe this will help shaping up this functionality:

Hi @alarsyo!

To be clear, I totally get the sort of use cases, and have no objection in principle.

To me it's just a question of how to do it elegantly, that fits well with the existing design.

And since I myself haven't seen a need for it, and have other issues that are higher priority, I don't have a huge incentive to figure it out and code it.

Which is to say, happy to accept PRs ;-)

[bdarcus]

Which makes me also wonder about something more tightly coupled to OR (v2); say a citar-org-roam.

@jethrokuan - any thoughts or tips on this?

I'm just thinking that, rather than add more code and complexity to citar proper to support this use case, can instead have a small (tiny?) little package that relies on OR v2 for handing the notes and gluing it to citar. The resulting package should also work better with OR.

Might even be org-roam-citar, rather than citar-org-roam?

Seems one obvious needed change here would be something like a configurable citar-has-notes-function variable. One could then swap one that queried org-roam instead to create the candidate data.

On the org-roam end, seems it would be helpful to be helpful to have the function for adding a ref to a node configurable?

See #602 for futther discussion.

cc @roshanshariff

[bdarcus]

I've decided to close this issue, and instead make citar a bit more configurable. So folks that want this functionality would simply use citar with org-roam, which would take care of those details. Or they could write their own functions, examples of which we can put on the wiki, to open notes and update candidates.

See the linked issues.

[bdarcus]

For the adventurous, if you want to test this, I think it should be ready (though don't be surprised if there are some hiccups).

#609

[alarsyo]

I'll try to test this tomorrow, thanks!

[bdarcus]

No huge rush; it will probably take a week or so before I've fixed some things I know need fixing.

And it could take longer to address a couple of issues in org-roam.

[bdarcus]

Closing this again; this will be handled in https://github.com/emacs-citar/citar-org-roam.