Error printed if key not in Zotero, but in metadata or bibliography

[fiapps]

With version 1.0.0 of the filter, if a key is cited that is not in Zotero, but is defined in a bibliography file or in references metadata, the filter prints an error and a stack trace from Zotero. I think it should print nothing in this case. A warning that the reference is not in Zotero would be OK, but I think it is unnecessary, because if the key is not defined elsewhere citeproc will already warn about this. The error that is currently printed does not make it clear what is wrong, and the stack trace is unnecessary.

Sample to reproduce the problem:

Cite something not in Zotero[@no_such_entry 123].

---
references:
- id: no_such_entry
  author:
    - literal: Somebody
  issued:
    - year: 2021
  publisher: Some Press
  publisher-place: Somewhere
  title: Such Entry
  type: book
...

Place the above in foo.md and run pandoc foo.md -L pandoc-zotxt.lua --citeproc -o foo.pdf:

pandoc-zotxt.lua: Library ID not providedError: Library ID not provided
    Zotero.DataObjects.prototype.getIDFromLibraryAndKey@chrome://zotero/content/xpcom/data/dataObjects.js:377:25
    Zotero.DataObjects.prototype.getByLibraryAndKeyAsync<@chrome://zotero/content/xpcom/data/dataObjects.js:349:12
    tryCatcher@resource://zotero/loader.jsm -> resource://zotero/bluebird/util.js:16:16
    module.exports/Promise.method/<@resource://zotero/loader.jsm -> resource://zotero/bluebird/method.js:15:21
From previous event:
    captureStackTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:915:23
    CapturedTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:807:5
    longStackTracesCaptureStackTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:482:19
    module.exports/Promise.prototype._then@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:261:9
    module.exports/Promise.prototype._passThrough@resource://zotero/loader.jsm -> resource://zotero/bluebird/finally.js:94:12
    module.exports/Promise.prototype.finally@resource://zotero/loader.jsm -> resource://zotero/bluebird/finally.js:103:12
    PromiseSpawn@resource://zotero/loader.jsm -> resource://zotero/bluebird/generators.js:36:25
    module.exports/Promise.coroutine/<@resource://zotero/loader.jsm -> resource://zotero/bluebird/generators.js:197:21
    Zotero.Server.DataListener.prototype._headerFinished@chrome://zotero/content/xpcom/server.js:302:5
    Zotero.Server.DataListener.prototype.onDataAvailable@chrome://zotero/content/xpcom/server.js:208:7

[odkr]

Good point! pandoc-zotxt.lua behaves this way because zotxt used to return error messages, not strack traces in that case. That no longer makes sense. Also, pandoc-zotxt.lua shouldn't try to look up keys that are in the references or in bibliography files at any rate. I don’t know when I'll get around to it, but I’ll fix that.

[odkr]

I haven’t fully tested it, but as of commit 9226cdc, pandoc-zotxt.lua should fetch sources from Zotero if and only if they are defined neither in the references metadata field nor in any CSL JSON or CSL YAML bibliography file. And it should no longer print stack traces.

[odkr]

I made a pre-release, if you want to test it. https://github.com/odkr/pandoc-zotxt.lua/releases/tag/v1.0.1b

[fiapps]

I downloaded v1.0.1b. It works on the test case above, but when I run it on the document that led me to report this issue, it fails like this:

Error running filter pandoc-zotxt.lua:
pandoc-zotxt.lua:681: attempt to get length of a nil value (local 'sources')
stack traceback:
	pandoc-zotxt.lua:681: in local 'get_def_citekeys'
	pandoc-zotxt.lua:705: in global 'get_undef_citekeys'
	pandoc-zotxt.lua:875: in function <pandoc-zotxt.lua:874>

That document has a YAML bibliography specified in its metadata, and does not use the references metadata field.

Here's an MWE to produce the above error:

Cite something not in Zotero[@no_such_entry 123].

---
bibliography:
  - bibliography.yaml
...

[odkr]

Sorry, I was busy. I’ve added your MWE to the test suite, which now passes. Have a look at https://github.com/odkr/pandoc-zotxt.lua/releases/tag/v1.1.0b and tell me if it works. Opening this again for the time being.

[fiapps]

I found time to test this, and the test case does work.

When I tested it on a larger document, I got pandoc-zotxt.lua: bibliography.yaml: Parse error. This was caused by the fact that bibliography.yaml had --- as the first line and ... as the last line. When II removed these lines (unnecessary, but valid YAML), pandoc.zotxt.lua processed the document correctly.

[odkr]

Thanks for testing this again. I'll have to think about how to handle --- and .... Pandoc could do that, but it parses text to its own metadata data types, which isn't handy. And the YAML library I am using is incomplete (doing everything the spec asks for in Lua, with its limited string handling capabilities, is hard).

[odkr]

So, switching to Pandoc for parsing YAML, while a good idea in general, wasn't quite as useful as expected. The Lua library can't handle --- and ..., and Pandoc insists on them. Either way, I have to add a hacky check for ---. Still, hacky as it may be, it should work now. Have a look at https://github.com/odkr/pandoc-zotxt.lua/releases/tag/v1.1.0b2.

[fiapps]

The filter now seems happy with my document, which gets some items from its YAML bibliography, and others from Zotero via the filter. I first tested without --- and ..., which I had removed to make the YAML bibliography work with 1.1.0b. Then I restored --- and ... by reverting the commit that deleted them, and ran again.

I noticed that if Zotero is not running, I get this error:

Error running filter /Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:
/Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:282: attempt to index a pandoc error value (local 'msg')
stack traceback:
	/Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:282: in global 'printf'
	/Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:1774: in function </Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:1761>

I easily guessed that the problem was that i had forgotten to start Zotero, but you should probably have a clearer error message for that case.

[fiapps]

I'm still installing the script manually. This time I tried make install, but it failed like this:

/bin/sh: oksh: command not found
install.sh: pandoc-zotxt.lua-1.1.0b2: Wrong name.

This is on macOS 11.4.

My manual installation method is to unpack the zip under my Pandoc Lua filters directory and then copy pandoc-zotxt.lua into the filter directory. The variable VERSION in 1.1.0b2 was still "1.1.0b", so it was getting its luarocks from the previous release. I changed the variable to "1.1.0b2" so that it would get its luarocks from the right place.

[odkr]

The filter now seems happy with my document, which gets some items from its YAML bibliography, and others from Zotero via the filter. I first tested without --- and ..., which I had removed to make the YAML bibliography work with 1.1.0b. Then I restored --- and ... by reverting the commit that deleted them, and ran again.

Great!

I noticed that if Zotero is not running, I get this error:

Error running filter /Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:
/Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:282: attempt to index a pandoc error value (local 'msg')
stack traceback:
	/Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:282: in global 'printf'
	/Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:1774: in function </Users/jlm/.local/share/pandoc/filters/pandoc-zotxt.lua:1761>

I easily guessed that the problem was that i had forgotten to start Zotero, but you should probably have a clearer error message for that case.

That’s not so much an error message as a bug in the error handling mechanism ;-).

I'm still installing the script manually. This time I tried make install, but it failed like this:

/bin/sh: oksh: command not found
install.sh: pandoc-zotxt.lua-1.1.0b2: Wrong name.

I’m not surprised that the installer is still buggy. That bug in particular does surprise me though. I'll write some tests for the installer.

[odkr]

I had another look at those error message, and they should be more legible now. And I decided to drop the installer. You can find a new beta version at https://github.com/odkr/pandoc-zotxt.lua/releases/tag/v1.1.0b3.

[odkr]

Closing this because v1.1.0b7 is stable enough. I'll release it once I am sure it will work with the next release of Pandoc.

[fiapps]

I finally got back to a project using this filter, and updated to 1.1.0b7.
If the key cannot be found in Zotero or in the bibliography, I get an error like this:

pandoc-zotxt.lua: carlo_cammino_1664: got unparsable response: Library ID not providedError: Library ID not provided
    Zotero.DataObjects.prototype.getIDFromLibraryAndKey@chrome://zotero/content/xpcom/data/dataObjects.js:377:25
    Zotero.DataObjects.prototype.getByLibraryAndKeyAsync<@chrome://zotero/content/xpcom/data/dataObjects.js:349:12
    tryCatcher@resource://zotero/loader.jsm -> resource://zotero/bluebird/util.js:16:16
    module.exports/Promise.method/<@resource://zotero/loader.jsm -> resource://zotero/bluebird/method.js:15:21
From previous event:
    captureStackTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:915:23
    CapturedTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:807:5
    longStackTracesCaptureStackTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:482:19
    module.exports/Promise.prototype._then@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:261:9
    module.exports/Promise.prototype._passThrough@resource://zotero/loader.jsm -> resource://zotero/bluebird/finally.js:94:12
    module.exports/Promise.prototype.finally@resource://zotero/loader.jsm -> resource://zotero/bluebird/finally.js:103:12
    PromiseSpawn@resource://zotero/loader.jsm -> resource://zotero/bluebird/generators.js:36:25
    module.exports/Promise.coroutine/<@resource://zotero/loader.jsm -> resource://zotero/bluebird/generators.js:197:21
    Zotero.Server.DataListener.prototype._headerFinished@chrome://zotero/content/xpcom/server.js:302:5
    Zotero.Server.DataListener.prototype.onDataAvailable@chrome://zotero/content/xpcom/server.js:208:7

[fiapps]

MWE for a test case:
echo '@no_such_key' | pandoc -L pandoc-zotxt.lua
The stack trace 1.1.0b7 produces is similar to the above.

On the positive side:

• If Zotero is not running, the error message is clear.
• The filter does not complain about keys that are not in Zotero but are in references metadata.
• The filter works when it should and fails when it should: the only problem is the clarity of an error message.

[odkr]

Thanks for taking the time! I, too, have noticed this error message in testing. zotxt throws this exception when a user runs a query with a -- syntactically invalid -- Zotero item ID that contains an underscore. However, if a citation key matches no item, pandoc-zotxt.lua simply re-runs the query, declaring the citation key to be of a different type. So, if a citation key is undefined, it will run a query declaring that key to be a Zotero item ID eventually. I have fixed that in the develop branch already, so it will be in the next release. I don't know when that will be, however. If it bothers you a lot, the develop branch as per commit cd3ef1f should be stable ;-).

[odkr]

There's a new beta that addresses this (and other) issues: https://github.com/odkr/pandoc-zotxt.lua/releases/tag/v1.2.0b

[fiapps]

I installed and tried 1.2.0 today, and encountered no issues.

[odkr]

Great :-).