Implementing JSON Schema for Provider

[thekaveman]

This is for #46 and more of a test of the idea than fully-baked implementation.

There is plenty of room for improvement:

• Define the top-level envelope structure once:

  {
      "version": "...",
      "data": {},
      "links": {}
  }

  and re-use it across different parts of the overall MDS schema.

• Define the common objects (route, vehicle_type, propulsion_type, etc.) once and reference

• Reference (existing?) geoJSON schemas if applicable

I'm sure there are others.

[ian-r-rose]

This is awesome @thekaveman. I know you are into the versioning of these specs: what do you think about being even stricter with the version field?

I think it's a good idea to include the schemas for the GeoJSON componenents. The good news is that the definers of the format have provided schemas. Here is code to auto-generate them. The README links to the actual schemas. As far as I can tell, the JSON schema spec allows you to reference other schemas by URI, so it should be relatively straightforward to link to those. For example, here is the schema for Point:
http://geojson.org/schema/Point.json

[ian-r-rose]

I think this pattern can be even stronger than this. We know that this schema is 0.1.0, not just any string. We can also expect various non-breaking changes that result in patches to this schema, and therefore could change the third number. So the pattern could be something stricter like

^0\.1\.[0-9]+$

Thereby enforcing the major version number, and letting the minor version be any number.

Alternatively, we could store the major and minor version numbers as numbers, rather than strings, and have something like

       "majorVersion": {
            "description": "Major version number",
            "type": "integer",
            "minimum": 0,
            "maximum": 0
        },
       "minorVersion": {
            "description": "Minor version number",
            "type": "integer",
            "minimum": 1,
            "maximum": 1
        },
       "patchVersion": {
            "description": "Patch version number",
            "type": "integer",
            "minimum": 0
        }

[thekaveman]

Good point, I just left the default there but SemVer is well-defined enough that we could easily tighten it.

I don't know about hard-coding the minorVersion in a pattern like ^0\.1\.[0-9]+$, but maybe with some kind of generation tool it wouldn't be that big of a deal.

[ian-r-rose]

I think that the version of the schema could be incremented with each publishing of a new version. The publisher would decide: "is this a bugfix, a backwards compatible change, or a breaking change?", and then change the pattern (or integer field) accordingly.

[ian-r-rose]

Boy, the UUID pattern is not messing around, is it?

[ian-r-rose]

My understanding of the narrative description is that the JSON payloads are different between whether they hit the trips or status_changes endpoints. So having the JSON schema describe both of them at once seems like it is not the right thing. Instead, I would think that there would be a separate schema for each of the endpoints (maybe both deriving from a common base schema?).

Or am I misunderstanding things?

[thekaveman]

No, it's one of the things I meant to put on my list above... definitely needs to be cleaned up.

Reading about combining schemas got me thinking that something along those line might work. Just haven't thought about how to do it yet.

[ian-r-rose]

Ah, I see some of my comments were also addressed in your initial statement, sorry.

[hunterowens]

@thekaveman @ian-r-rose status on this? Would love to merge into dev soon.

[ian-r-rose]

I think CityofSantaMonica#1 is close to ready. @thekaveman and I were uncertain about whether it was best to include some of the tooling (validation + schema generation scripts) in this repo, or an associated one. Also happy to open that PR against this repo to supersede this one.

[thekaveman]

@hunterowens I thought this was good for 0.1.x since it is more of a clarification/expression of the existing spec than a change? I'd like to work from this as I'm building out the ingestion framework if possible (since we didn't have clarification on some field types for example).

Outstanding concerns to address for this to be ready:

• event_location is a GeoJSON Feature Clarify event_location type in status changes #74
• timestamps are float Unix epoch seconds (Unix Timestamp clarification in Provider API #87 ) unless anyone has other thoughts?

[ian-r-rose]

Also, are you using dev to track feature changes, while master is the official spec? If so, we should reopen this PR to be against dev.

[hunterowens]

@ian-r-rose going forward that is the case. I just changed this PR to be against dev rather than master.

[thekaveman]

I'm going to rebase this on dev, stand by...

[thekaveman]

Ok, I think we still need this PR to address the following:

• the latest clarifications from Clarifying location type #94 (I suggest creating a corollary FeatureMDS that we can use)
• update the version string to 0.2.0
• update the README to address the schema, link to it, etc.

(I'm going to go through it again and potentially add to this list, feel free to do so anyone else @hunterowens @ian-r-rose)

[ian-r-rose]

Thanks for updating @thekaveman, these changes look good to me. Just one future-proofing thought.

[hunterowens]

🌮 💯 💯 🥇 to @thekaveman and @ian-r-rose