Add support for Uint8Array's in place of node Buffer's to ipfs-unixfs-importer

[Gozala]

I glanced through the implementation and it seems that dependence is inherited from use of following libraries

• bl
• Support for plain array buffer multiformats/js-multihashing-async#33

There are few lines that also use Buffer directly but there Uint8Array would have done just fine as well.

Trying to actually remove here might be difficult, however we could probably just remove it from the API surface and turn Uint8Arrays into Buffers internally where needed.

[Gozala]

Actually I think I have overlooked chunk validation phase that seems to normalize each chunk

js-ipfs-unixfs/packages/ipfs-unixfs-importer/src/dag-builder/validate-chunks.js

Lines 8 to 19 in 8e8d83d

	for await (const content of source) {
	if (content.length === undefined) {
	throw errCode(new Error('Content was invalid'), 'ERR_INVALID_CONTENT')
	}
	if (typeof content === 'string' || content instanceof String) {
	yield Buffer.from(content, 'utf8')
	} else if (Array.isArray(content)) {
	yield Buffer.from(content)
	} else {
	yield content
	}

Although I'm also starting to suspect that ipfs.add([ new ArrayBuffer(1024) ]) would cause content was invalid error because ArrayBuffer because ArrayBuffers seem to be passed through although they do not have .length property.

P.S.: I'll verify that assumption and submit test / fix as necessary.

[Gozala]

@achingbrain is this the right type signature for the importer:

declare function importer(source:AsyncIterable<ImportEntry>, ipld:IPLDResolver, options?:ImporterOptions)

type ImportEntry =
  | ImportFile
  | ImportDir

type ImportFile = {
  path: string|void,
  content: Content,
  mtime?: Mtime,
  mode?: Mode
}

type ImortDir = {
  path: string,
  content?:void,
  mtime?: Mtime,
  mode?: Mode
}


type Content =
  | Iterable<string>
  | Iterable<ArrayBuffer>
  | Iterable<ArrayBufferView>
  | AsyncIterable<string>
  | AsyncIterable<ArrayBuffer>
  | AsyncIterable<ArrayBufferView>

type Mode = string | number
type MTime = number | Date | UnixFSTime | HRTime
type UnixFSTime = {
  secs: number,
  nsecs?: number 
}
type HRTime = [number, number]

type IPLDResolver = {
 // ...
}

type ImporterOptions = {
  // ...
}

[achingbrain]

The type it expects to be yielded by the source is:

{
  path: String,
  mode: Number (optional),
  mtime: {
    secs: Number,
    nsecs: Number (optional)
  },  (optional)
  content: AsyncIterator<Buffer> (optional)
}

It handles more, as you found, maybe it shouldn't.

[Gozala]

In that case can we loosen up it a bit to accept Uint8Arrays in place of buffers ?

[achingbrain]

Of course, we shouldn't require the input to be Buffers anywhere. I think the only place that might get painful is the use of bl, which we have to avoid unnecessary copying, but as long as we end up with similar performance characteristics then all good.

[Gozala]

fixed by #69