[APM] migrate to io-ts

[dgieselaar]

I've migrated some of our routes to io-ts, and with the help of some TypeScript magic and a quirky way of registering our routes, we now should have end-to-end runtime and type coverage without having to duplicate things for client and server.

Opening a draft PR to get some early feedback. I migrated the error and index pattern routes.

To summarize:

• we replace joi with io-ts
• instead of letting the routes register themselves, we export the route objects, import them in a root file and then register them at once via a router vessel. that router will inherit the types of the routes, and merge them all together.
• we create route objects via a createRoute call. this allows us to get fully typed parameters in the request handler, inferred via the io-ts types.
• the type of the router vessel is used to create a fully typed callApmApi method, that uses pathname and method as indexers to get inferred types for path, query and body params, and return types.

[dgieselaar]

this and below is just dummy stuff to test out the types.

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[dgieselaar]

@sqren We can also use this approach to generate the methods we now have in /rest, if we add a name property to the route description. Basically the TS transformation would look like:

{
  name: 'getErrorDistribution',
  path: `/api/apm/services/{serviceName}/errors/distribution`,
  method: 'GET',
  params: {
	path: t.type({
      serviceName: t.string
    }),
    query: t.intersection([
      t.partial({
        groupId: t.string
      }),
      defaultApiTypes
    ])
  },
  handler: async (req, params) => {
    ...
  }
}

to

const { client } = api;

client.getErrorDistribution({ serviceName, groupId, start, end, uiFilters })

And it would be fully typed.

Thoughts?

[sorenlouv]

const { client } = api;

client.getErrorDistribution({ serviceName, groupId, start, end, uiFilters })

And it would be fully typed.

Thoughts?

My knee-jerk reaction was that I did not like this. Where's the http method, the url, potentially headers if we want to add those?
But then I thought a little more about it, and I think I really like it. All the things I mentioned above are just implementation details. I like how we can avoid all the ceremony of REST, have the transport details declare in a single config, and then just worry about calling the endpoint with the right arguments.

Controversial idea: we could take a page from graphql's book and get rid of path, method and headers entirely (all requests go to a single endpoint). We could keep name to identify the request handler on the backend. This would determine which params the client had to pass.

Btw. I am too young to remember xml-rpc but I believe that is what we are reinventing. I love it! :D

[dgieselaar]

@sqren maybe we should use /api/apm/${apiName} for debugging purposes? For network inspection, curl, postman etc I imagine semi-RESTful endpoints are still useful. I'm also wary of losing some benefits of conventions around GET/POST/PUT/DELETE etc. But it might also not be entirely clear to me what the value of one endpoint without headers etc would be.

[sorenlouv]

maybe we should use /api/apm/${apiName} for debugging purposes? For network inspection, curl, postman etc I imagine semi-RESTful endpoints are still useful. I'm also wary of losing some benefits of conventions around GET/POST/PUT/DELETE etc

I also don't think it would be a great idea. For one it becomes much harder to cache content if everything goes to the same endpoint.

But it might also not be entirely clear to me what the value of one endpoint without headers etc would be

The benefit would be that it would simplify the API. Right now a route is identified by both a name, a path and a method. If we decided on a single endpoint for all routes, any route could be identified purely from the name.

With the rpc style approach you suggested last (which I liked) we don't have a single identifier. Eg. from the client we call it like

client.getErrorDistribution({ serviceName, groupId, start, end, uiFilters })

So the unique identifier for the route is getErrorDistribution, right? But how would it be called via curl? It will be something completely different. There is no correlation between the name identifier and the method/path.

[dgieselaar]

So the unique identifier for the route is getErrorDistribution, right? But how would it be called via curl? It will be something completely different. There is no correlation between the name identifier and the method/path.

I guess that's up to the implementation. Nothing is holding us back from using name as a suffix for path. Ie, { name: 'getErrorsDistribution' } would register a route for '/api/apm/getErrorsDistribution' (and we'd no longer have to define path).

[sorenlouv]

I guess that's up to the implementation. Nothing is holding us back from using name as a suffix for path. Ie, { name: 'getErrorsDistribution' } would register a route for '/api/apm/getErrorsDistribution' (and we'd no longer have to define path).

Apart from the odd camelCased paths we'd end up with I think I like this approach. What do you think?

[dgieselaar]

Yeah, I like it. Maybe I can demo what I have in today's standup, I'd like to know what @ogupte thinks about it as well.

[dgieselaar]

Thinking about this some more: can we import server code (specifically the route descriptions) for client code? I mean runtime code, not just types. If we can't, we have to consider that the client call that we make needs to know enough to construct a request object. Ie, it needs to know the pathname, the request method, and the parameters. So that would limit us to the following API I think:

[RouteName]: ( params:Params, method:HttpMethod = 'GET' ) => ReturnTypeOfRoute

So that means even if we only have a POST route for RouteName, we would still need to explicitly define 'POST' as a method, because the client needs to know that it should send a POST to that API.

(this also means that we still have to differentiate between path, query and body parameters).

[dgieselaar]

It does seems like I'm not able to import server-side code. The errors are really vague, but I'm going to assume that it just doesn't work for the time being. That also means I have to implement the pattern that we discussed client.getErrorDistribution({}) with a Proxy:

export const calls: Client<APMAPI> = new Proxy(
  {},
  {
    get: (obj, prop) => {
      return ({
        query = {},
        body = {},
        method = 'GET'
      }) => {
        return callApi({
          method,
          pathname: `/api/apm/${prop.toString()}`,
          query,
          body: body ? JSON.stringify(body) : undefined
        }) as any;
      };
    }
  }
) as Client<APMAPI>;

We can also keep it simple and opt for client('getErrorDistribution', {}).

[sorenlouv]

It does seems like I'm not able to import server-side code.

You might already do this but just to make sure we are on the same page: Typescript imports (interfaces, types etc) should work across /public and /server. If you want to share actual runtime code between client and server it should live in /common (instead of /public or /server)

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[sorenlouv]

Number(value.toFixed(3)) === value looks so much nicer than the one I suggested 👍

[sorenlouv]

You used to do:

              params: {
                query: params.query
                  ? t.intersection([params.query, debugRt])
                  : debugRt
              },

Which made sense to me. But with this change, if the route doesn't define a query will _debug still be added?

[dgieselaar]

No, that's a good catch actually. The way it's written now none of the {query,path,body} parameters get validated if there's no param types. Let me fix that (and write a test for it). Thanks!

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[dgieselaar]

retest

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request