custom encoding options on db from client.

[dweinstein]

if you pass require('bytewise/hex') as the keyEncoding in a .batch, .put, .createReadStream... (as passed in the opts parameter) on the client it'll just receive an error back saying .encode is not a method.

I'm guessing this isn't something that could be stored in the manifest. So is this something that one would have to tweak/setup server-side? Shim the client's methods to do the key-encoding client side?

The documentation says binary encoding is supported, but maybe it would help other's to say custom encodings aren't. Bytewise seems to be a pretty fundamental encoding for level... I almost wish it were native like json and binary but maybe it's bloat. Thoughts?

Probably this is related to #17 and #12 but not certain.

[dweinstein]

The test I used initially is this:

var getDb = require('./util').getDb;
var test = require('tape');
var binary = require('bops');
var bytewise = require('bytewise/hex');

  test('batch', function (t) {
    var opts = { keyEncoding: bytewise, valueEncoding : 'binary' };
    getDb(function (db, dispose) {
      db.put(['test', 'foo'], binary.from([1,2,3]), opts, function (err) {
        t.error(err);
        db.get(['test', 'foo'], opts, function (err, value) {
          t.error(err);
          t.ok(binary.is(value));
          t.equal(binary.to(value), binary.to(binary.from([1,2,3])));
          dispose();
          t.end();
        });
      });
    });
  });

error:

...
not ok 5 Error: Object #<Object> has no method 'encode'
...

[dominictarr]

ah... I think that multilevel doesn't actually implement this correctly. The quick fix is to encode your own keys using bytewise/hex and then pass that to multilevel as a string. The better fix is to make a pull request to multilevel that does this for all requests... in a way that is consistent with levelup.

[dweinstein]

I set a couple breakpoints and:

break in node_modules/multilevel/lib/server.js:35
 33       if (method.type == 'async') {
 34         server.createLocalCall(name, function (args, cb) {
 35           debugger;
 36           access(server.sessionData, db, k, args);
 37           args.push(cb);
debug> repl
Press Ctrl + C to leave debug repl
> args
[ [ { type: 'put',
      key: [Object],
      value: 0 },
    { type: 'put',
      key: [Object],
      value: 0 },
    { type: 'put',
      key: [Object],
      value: 0 } ],
  { keyEncoding: { buffer: false, type: 'bytewise-hex' } } ]
 //                              ^-- missing .decode/.encode

I'll see if I can figure out what's going on but based on this the decode method didn't make it across (not super familiar with how the RPC stuff works here yet but I'll figure it out). I guess stepping back I'm not sure I'd expect the .encode and .decode methods from the the client to magically appear at the server, so maybe this is expected.

Hopefully I can work towards that PR... we'll see. :-)

[dominictarr]

Whats happening is the object is encoded with json which removes the function (that would be a massive security hole, like an sql injection)

Basically you need to encode the object before you send it, or make it so that multilevel encodes the object for you...

hmm, now that I think about it, multilevel does use an encoding that supports buffers, so if you set keyEncoding: require('bytewise/hex') on the server instance, then I think this will all work for you as is.

[juliangruber]

that will work but multilevel should still support encodings the same way as levelup, so i'll be working on this now

[juliangruber]

i'm working on an implementation that will work, but i'm thinking more and more that multilevel as a leveldown implementation will work better, so we don't need to reinvent levelup logic

[juliangruber]

relevant: #64 and Level/levelup#313

[mafintosh]

does this mean i can't do this anymore?

db.get('foo', {valueEncoding: myOwnEncoder}, cb)

i usually do that when using protocol-buffers, https://github.com/mafintosh/protocol-buffers#leveldb-encoding-compatibility

[juliangruber]

currently you can't do that from the client, but i don't remember this ever working with multilevel

[mafintosh]

@juliangruber ah whoops. meant to add the comment to Level/levelup#313 (added it there as well now)