How to do local file bundling

[designermonkey]

Firstly, sorry if this is documented and I can't see it (wood for the trees problem).

I'm struggling to understand how to properly bundle and validate with local files.

Our API schema documents are never going to live at a real URL as they are part of a proprietary system. According to the spec, they don't have to be a real URL as the $id is not meant to be network addressable.

I wanted to use $id values that were simply path based, which I assume is allowed as a "relative uri" by the spec, and would resolve to the most parent document.

The issue I've run into is that I can't addSchema using these IDs as they are always rejected as unreachable over the network. I've tried to use the retrievalUri property but I don't understand how to.

---

This isn't an issue so I've raised this as a discussion, but I would like to understand how I am supposed to bundle and validate a path based tree of schemas that are filesystem only.

Here are some things I want to be able to do:

1. I want to use onloy path IDs so a schema could have $id: /my/path/based/schema and be referenced as $ref: /my/path/based/schema from any nested path level.
2. With each schema having a path that starts at the same root level, I would like to know that root level is taken into account, so if I validated a document using the schema at /my/path/based/schema it would be able to find other schemas that I have added based on that root level.

I don't even know if this is achievable, and would love some advice on how this should be done. There is no https:// based root level for our setup, and I'm averse to faking one just for the sake of it.

[designermonkey]

I should add, for clarity, there isn't always one root schema in our case. Some schemas reference others as mixins for example, but each schema we have is useable in isolation as part of a tree of schemas where the path tree is about the concept the schema belongs to.

[jdesrosiers]

The way this library works, it bundles everything you need to evaluate a given schema. So, it sounds like you might want a bundle for each potential root schema. If what you want is a complete bundle of all of your schemas regardless of association, this library isn't going to give what you want, but the good news is it's so easy you don't need a library. The only remotely hard things this library does for you is finding the set of schemas that are needed to evaluate a given schema. Constructing that into a bundle is trivial.

It's literally just this,

bundled = {};
LOOP recursively over a folder with schemas:
  bundled.$defs[path] = schema
END LOOP

[designermonkey]

I got each schema to bundle any references into itself, so I have a tree of bundled schemas now, which is what I was after. Thanks.

[designermonkey]

After a little playing around I figured out how to do it. Example here: https://github.com/micron-research/bundle-example

My main complaint about this is that the bundle process is removing IDs of the bundled files. Why would it do this? If I ever did decide to deliver these bundled schema over the web, I can't identify them any more. Is there a way of preventing this behaviour?

Another observation at least is that the IDs added to the $defs clearly don't match those of the $refs as the preceeding slash is removed.

Would it not be better in both cases not to mess with existing values?

[jdesrosiers]

The $ids are getting removed/rewritten because you are using file: URIs. Things look strange because you're going against the grain by adding schemas to the schemaStore with file: $ids. When the bundler sees file: URIs, it's expected that you are actually working with the filesystem and takes into account the special considerations necessary to account for when using the filesystem.

Using a full file: URI is a problem because it couples the schema identifier to your personal filesystem. If someone else checks out the repo with your schemas, those $ids will be wrong because their filesystem layout isn't identical to your filesystem layout. It can also be a security risk for the files in your repo to expose your personal filesystem layout.

Using relative URI solves those problems, but creates more of it's own. If you have a relative URI, there must be a retrieval URI in order to resolve that $id, which means the retrieval URI and the schema identifier can potentially disagree and cause problems (as it does with the schemas in your example repo).

Therefore, the only way forward is to not use $ids and rely on retrieval URIs. This avoids all of the problems and has no downsides. It's also much more natural to reference based on the filesystem that some hierarchy encoded in the the schemas that may not have any relation to how they are stored on the filesystem.

The $ids used for the embedded schemas is a little different story. Those $ids are necessary for the bundling process. We still don't want full file: URIs for all the same reasons. That means we have to use relative URIs but it's important that the resolve to the actual location of the schema being embedded. The relative $id generated for those schemas is the path from the enclosing schema's location on the filesystem to the embedded schema's location on the filesystem. If it were anything else we would again having the problem of mismatching schema identifier and retrieval URIs.

I hope that explains why it works the way it does. Again, using http(s) paths and loading them into the schemaStore gives you more control over identifiers if that's what you need, but I don't think it should matter.

[jdesrosiers]

If I ever did decide to deliver these bundled schema over the web, I can't identify them any more.

If you host them on the web, you don't want a $id for the same reasons you don't want one for files. The identifier is the URI used to retrieve schema. Including an $id is unnecessary duplication that can only be a potential source of bugs if the retrieval URI and the schema identifier don't match.

[designermonkey]

I know I'm dumping loads into this thread, sorry...

I've added a branch openapi to that repo where I've tried to bundle into an openapi document, but it hasn't worked in the output file. Any advice on what I should be doing?

I expected a $defs block but I got nothing.

[jdesrosiers]

I never implemented support for bundling into an OpenAPI document. It's not possible for OpenAPI 3.0 because that version doesn't support $id. For OpenAPI 3.1, it's possible.
I think the missing piece was support for traversing the OpenAPI document to find references, which is different than finding references in JSON Schema.

I expected a $defs block but I got nothing.

For an OpenAPI document, schemas would go into /components/schemas instead of $defs.

[jdesrosiers]

Let's cover some fundamentals first. I'll get to actually answering the question of bundling from files next. (It's way easier than you think)

Our API schema documents are never going to live at a real URL as they are part of a proprietary system. [...] There is no https:// based root level for our setup, and I'm averse to faking one just for the sake of it.

It's 100% normal and expected that your schemas aren't hosted on the web at the URI that identifies them. Don't think of it as "faking it", that's the intended way to use JSON Schema.

According to the spec, they don't have to be a real URL as the $id is not meant to be network addressable. I wanted to use $id values that were simply path based, which I assume is allowed as a "relative uri" by the spec, and would resolve to the most parent document.

If I understand you correctly, you have that right. But, let me add that every schema needs to be identified by a non-relative URI. $id can be relative, but it must resolve to a non-relative URI which is the true identifier of the schema. If the $id is in a sub-schema (as it is with bundled schemas), the base URI used to resolve the $id is the identifier of it's parent schema resource. If the $id is at the root, the base URI is the URI used to retrieve the schema.

https://example.com/schema/main

{
  "$id": "/schema/main",

  "$ref": "/schema/foo",

  "$defs": {
    "foo": {
      "$id": "foo",
      "type": "string"
    }
  }
}

In this example, the schema is retrieved from the schemaStore with the URI https://example.com/schema/main. This is the base URI used to resolve $id. Fortunately, the resolved $id is the same as the retrieval URI so there's only one URI identifying this schema. The $id at #/$defs/foo is resolved against the base URI of the main schema resulting in https://example.com/schema/foo. The $ref is also resolved against the base URI resulting in https://example.com/schema/foo, which matches the identifier of the embedded schema.

So, JSON Schema is always working with non-relative URIs. Relative URIs are allowed almost everywhere for convenience, but they have to be resolved before they can be used.

[jdesrosiers]

Especially when bundling schemas from files or schemas that are hosted on the web, I strongly recommend not using $id in your schemas. The URI used to retrieve the schema is sufficient for identifying the schema. At best, $id duplicates the retrieval URI and at worst, it contradicts it! Here's a small example.

/path/to/schemas/main.schema.json

{
  "$ref": "components/foo.schema.json"
}

/path/to/schemas/components/foo.schema.json

{ "type": "string" }

const bundledSchema = bundle("file:///path/to/schemas/main.schema.json");

That's it. You don't need to load schemas with addSchema. The library will get the schemas directly from the file system and your references can be relative to the schema on the filesystem.

Here are some things I want to be able to do

It sounds like you want to define a given path that all of your reference paths append onto. URIs don't work that way. They can only be relative to the current document. If relative paths aren't good enough for some reason, you can go back to using $id with made up http(s) to get full control of what your root path would be.

[designermonkey]

Thanks @jdesrosiers, for all your answers! I'll have to digest them all to make sure I understand correctly, but I'm sure I'll need more clarification 😄

[designermonkey]

Ok, so after all your informative answers, I've given up on the idea of local file paths as retrieval URIs and have decided to go with a dynamic domain name so I can run it as a local server to bundle files on the fly, depending on what URI is requested.

I guess the last outstanding thing is about OpenAPI 3.1 support. I know you're a busy chap; will there be support for this spec in the future?

[jdesrosiers]

Glad you came up with something that works for your needs.

I guess the last outstanding thing is about OpenAPI 3.1 support. I know you're a busy chap; will there be support for this spec in the future?

It's not high on my todo list. It will likely get done at some point, but not any time soon.