Using msgpackr independantly with lmdb

[cysabi]

Hey there, I am trying to serialize some data with msgpackr, and both send it over a webhook, as well as persist it inside of lmdb. I've found some helpful resources but am struggling with the specifics about how to string them together:

• I know that msgpackr lets you work with persisted data https://github.com/kriszyp/msgpackr?tab=readme-ov-file#shared-record-structures
• I also know that you can specify a key to automatically manage shared structures with lmdb https://github.com/kriszyp/lmdb-js?tab=readme-ov-file#shared-structures

but will there be conflicts when attempting to connect both at the same time? aka:

    const db = open({
      path: fp,
      sharedStructuresKey: Symbol.for("structs"),
    });
    const packr = new Packr({
      getStructures: () => db.get(Symbol.for("structs")) || [],
      saveStructures: (s) => db.putSync(Symbol.for("structs"), s),
    });

It would be lovely to be able to pass in my own Encoder object as a custom encoder into lmdb.
What is the right way to do this?

p.s: side question, i am using const prev = db.getKeys({reverse: true, limit: 1}).at(0) ?? -1 to create incremental keys, is this terribly inefficient? what alternatives do I have (attempting to use lmdb as an event store / append only log)?

[kriszyp]

I think they may conflict, or result in unsynced structures. I would recommend you simply use the encoder that lmdb-js creates:

const packr = db.encoder; // this will have the structure persistent set up for you

And you can serialize new records and they will be written from the database (and deserializing will read the structs from the database as needd).

db.getKeys({reverse: true, limit: 1}) ... is this terribly inefficient?

No, getting the iterable entry from this query is quite efficient, it should pretty efficiently find the largest/last key. I don't think there is an at method on RangeIterator though (although it is a good idea), you would just need to iterate and use the first/only iterated result.

[cysabi]

it doesn't look like .encoder exists

[kriszyp]

Addressed with 11eea88#diff-7aa4473ede4abd9ec099e87fec67fd57afafaf39e05d493ab4533acc38547eb8R382