Fix API Swagger specs, add OpenAPI v3

[valentijnscholten]

The API is very important, but currently we get feedback that:

• The SwaggerUI is incomplete/incorrect with regards to field types, parameter types, etc.
• Client generation either fails or generates incorrect code due to the schema not being correct;

This PR should help to fix that:

Swagger (OpenAPIv2) fixes:

• Fix all issues reported by test_swagger_schema;
• Fix notes() endpoints to have correct spec and consistent response type (instead of empty list when there are no notes);
• Fix accept_risk endpoints to have correct spec;
• Fix testdata for authorized_users in products, and for JIRA_Instance in test data and sample data;
• Support password as writeOnly field;
• Fix AppAnalysis and Finding_Template Tag serialization;
• Fix Finding request_response serialization specs;
• Fix response status codes to comply with RFC (no content = 204, created = 201)
• Rename get_duplicate_status to get_duplicate_cluster
• Remove left-over / incorrectly used TagList class
• Incorrect schema for reverse ManyToMany relationships (Incorrect prefetch schema for reverse ManyToMany fields #4618)

Some stuff is hard to fix in Swagger/OpenAPIv2. For example the NumberInFilter (comma seperated list of integers) is not correctly specced in the Swagger (v2) schema. It's better to migrate to OpenAPI v3 instead of having to fix all this inside django-filters itself.

The OpenAPI v3 schema generator built in to DRF doesn't work it seems, so I chose the most popular one: drf-spectacular (https://github.com/tfranzel/drf-spectacular)

• Add OpenAPI v3 spec using drf-spectacular (experimental);
• Port all @swagger-auto-schema decorators to @extend-schema
• Port schema validation code from test_swagger_schema.py to test_rest_framework.py;
• Add check for warnings in schema to start of unit tests;
• I implemented the prefetch parameter / response mechanism using a drf-spectacular post processor. This doesn't feel as nice as the composable schema implemented for swagger in the past. On the other hand my implementation is a lot simpler and less bound to which library we are using for the schema generation. Open to suggestions / improvements in follow up PRs.

Future:

• get descriptions for enums, I added/suggested this feature to drf-spectacular: DjangoFilters: Add description for enum choices tfranzel/drf-spectacular#403

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[valentijnscholten]

back to draft as I saw a problem

[alles-klar]

Good work @valentijnscholten!

• you added OpenAPI v3 (which is awesome!) with a different URL. Should we document and present it as our v3 API?
  I mean, OpenAPI and Swagger are very different. it will confuse users if we are considering the both our "v2" API

The API is still more or less the same, just upgrading from openapiv2 to v3 doesn't mean we have a v3 API. I would stick to v2 as API Version and just remove the openapiv2 documentation in the future.

[valentijnscholten]

It now looks like this:

After 1 or 2 months we could remove or hide the "old" swagger docs.

[alles-klar]

It now looks like this:

After 1 or 2 months we could remove or hide the "old" swagger docs.

We should rename Swagger docs to OpenAPI2 Docs. See https://swagger.io/blog/api-strategy/difference-between-swagger-and-openapi/

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[valentijnscholten]

We should rename Swagger docs to OpenAPI2 Docs. See https://swagger.io/blog/api-strategy/difference-between-swagger-and-openapi/

Done.

Also fixed the last open issue around selecting the correct serializer for the prefetch fields, so ready for review @alles-klar

[madchap]

It seems that the oa3 does not log the user in automatically, is it an oversight or just something that may no longer be possible?

[valentijnscholten]

It seems that the oa3 does not log the user in automatically, is it an oversight or just something that may no longer be possible?

What do you mean by "log in automatically"? When I browse to the swagger docs of OA3 I can execute requests without having to login or provide tokens.

[StefanFl]

@valentijnscholten Is it intentional, that there are no prefetches for one-to-one relationships anymore, for both OpenAPI V2 and V3? They are very useful, for example when you want to get the name of a role for a product member.

[valentijnscholten]

@valentijnscholten Is it intentional, that there are no prefetches for one-to-one relationships anymore, for both OpenAPI V2 and V3? They are very useful, for example when you want to get the name of a role for a product member.

No, it's not. I noticed they weren't there in the OpenAPI v2 so didn't think I needed to make them work in OpenAPI v3 and thought maybe it was intended as we also have the related_fields mechanism. But it looks like my PR somehow made them disappear in v2 already. Without this PR they are there. Will take a look.