[dart][dart-dio] Next-gen dart-dio generator #8869

kuhnroyal · 2021-03-02T15:20:44Z

WIP!

This is work in progress but examples compile and tests run locally. This needs an updated Dart version in CI.

Description

This PR establishes a next-gen dart-dio generator with breaking changes around null-safety which can not be implemented in backwards-compatible ways.

Features & changes

Dart 2.12.0+ requirement
Dio 4.0.0+ requirement
null-safety implemented
new package layout with a single exported file
wrapping of serialization errors in DioError
named parameters for all API params
support for different serialization libraries with built_value being the current only and default implementation
enforce dart format in CI (waiting on dart format --set-exit-if-changed not really setting the status code dart-lang/sdk#44582)

TODO

need a solution for serializing null with built_value (Support serializing null on a per field basis - nullable vs required google/built_value.dart#912)
need to handle unused_import analyzer warning in polymorph model classes
wait for timemachine to support NNBD (Any plans to migrate this package to dart sound null safety Dana-Ferguson/time_machine#55)

PR checklist

Read the contribution guidelines.
Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
Run the following to build the project and update samples:
```
./mvnw clean package 
./bin/generate-samples.sh
./bin/utils/export_docs_generators.sh
```
Commit all changed files.
This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
These must match the expectations made by your contribution.
You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
For Windows users, please run the script in Git BASH.
File the PR against the correct branch: master, 5.1.x, 6.0.x
If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

auto-labeler · 2021-03-02T15:20:48Z

👍 Thanks for opening this issue!
🏷 I have applied any labels matching special text in your issue.

The team will review the labels and make any necessary changes.

kuhnroyal · 2021-03-02T15:21:28Z

CC @agilob @josh-burton

robrbecker · 2021-03-03T06:02:09Z

I'm having a heck of a time getting this to build locally. I have tried to build with openjdk8 and oracle java 14.0.2.
Had to comment out the javadocs plugin from pom.xml as that was crashing with an exception.
I'm able to run mvn clean package successfully though.
Still getting an exception when running ./bin/generate-samples.sh bin/configs/dart-dio-next*

[main] INFO  o.o.codegen.TemplateManager - writing file samples/openapi3/client/petstore/dart-dio-next/petstore_client_lib_fake/doc/DogAllOf.md
[main] WARN  o.o.c.languages.AbstractDartCodegen - 200_response (model name starts with number) cannot be used as model name. Renamed to Model200Response
[main] ERROR o.o.c.api.TemplatingEngineAdapter - Failed to read full template enum_inline.mustache, enum_inline.mustache
[main] ERROR o.o.c.api.TemplatingEngineAdapter - Failed to read full template enum_inline.mustache, enum_inline.mustache
Exception in thread "main" java.lang.RuntimeException: Could not generate model 'EnumArrays'
	at org.openapitools.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:544)
	at org.openapitools.codegen.DefaultGenerator.generate(DefaultGenerator.java:875)
	at org.openapitools.codegen.cmd.Generate.execute(Generate.java:440)
	at org.openapitools.codegen.cmd.OpenApiGeneratorCommand.run(OpenApiGeneratorCommand.java:32)
	at org.openapitools.codegen.OpenAPIGenerator.main(OpenAPIGenerator.java:66)
Caused by: org.openapitools.codegen.templating.TemplateNotFoundException: enum_inline
	at org.openapitools.codegen.templating.MustacheEngineAdapter.findTemplate(MustacheEngineAdapter.java:79)
	at org.openapitools.codegen.templating.MustacheEngineAdapter.lambda$compileTemplate$0(MustacheEngineAdapter.java:61)
	at com.samskivert.mustache.Mustache$IncludedTemplateSegment.execute(Mustache.java:756)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:845)
	at com.samskivert.mustache.Mustache$InvertedSegment.execute(Mustache.java:910)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:845)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:870)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:845)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:866)
	at com.samskivert.mustache.Template.executeSegs(Template.java:157)
	at com.samskivert.mustache.Mustache$IncludedTemplateSegment.execute(Mustache.java:774)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:845)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:881)
	at com.samskivert.mustache.Template.executeSegs(Template.java:157)
	at com.samskivert.mustache.Mustache$IncludedTemplateSegment.execute(Mustache.java:774)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:845)
	at com.samskivert.mustache.Mustache$InvertedSegment.execute(Mustache.java:910)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:845)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:881)
	at com.samskivert.mustache.Mustache$BlockSegment.executeSegs(Mustache.java:845)
	at com.samskivert.mustache.Mustache$SectionSegment.execute(Mustache.java:866)
	at com.samskivert.mustache.Template.executeSegs(Template.java:157)
	at com.samskivert.mustache.Template.execute(Template.java:134)
	at com.samskivert.mustache.Template.execute(Template.java:125)
	at org.openapitools.codegen.templating.MustacheEngineAdapter.compileTemplate(MustacheEngineAdapter.java:65)
	at org.openapitools.codegen.TemplateManager.write(TemplateManager.java:163)
	at org.openapitools.codegen.DefaultGenerator.processTemplateToFile(DefaultGenerator.java:1017)
	at org.openapitools.codegen.DefaultGenerator.generateModel(DefaultGenerator.java:382)
	at org.openapitools.codegen.DefaultGenerator.generateModels(DefaultGenerator.java:535)

robrbecker · 2021-03-03T06:17:52Z

I think I got it running enough to generate my project by running something like
java -jar openapi-generator/modules/openapi-generator-cli/target/openapi-generator-cli.jar generate ... <params>

kuhnroyal · 2021-03-03T09:47:58Z

Hmm, the error looks like I might have messed something up with the templates but the example generates, not sure.

I think I got it running enough to generate my project by running something like
java -jar openapi-generator/modules/openapi-generator-cli/target/openapi-generator-cli.jar generate ... <params>

That is the way I am using it outside of the build in examples.

vishna · 2021-03-07T12:52:53Z

Just stumbled upon this, looks promising - good job 👍 I was wondering if it would be easier to use freezed instead of build_value?

kuhnroyal · 2021-03-07T12:57:42Z

Good question and one I have evaluated before. freezed does not support the level of enums that are required for OpenAPI - see rrousselGit/freezed#74

But this new version of the generator will allow to add different serialization libraries in the future. You are welcome to start working on a freezed implementation and see where that leads you.

This solves the problem of not being able to serialize null.

agilob · 2021-03-18T15:40:07Z

modules/openapi-generator/src/main/resources/dart/libraries/dio/auth/basic_auth.mustache

 }

 class BasicAuthInterceptor extends AuthInterceptor {
-    Map<String, BasicAuthInfo> authInfo = {};
+  Map<String, BasicAuthInfo> authInfo = {};


final for consistency?

Sure, this was just a copy/paste job from dart-dio.

agilob · 2021-03-18T15:44:20Z

modules/openapi-generator/src/main/resources/dart/libraries/dio/auth/oauth.mustache

 import 'package:dio/dio.dart';
 import 'package:{{pubName}}/src/auth/auth.dart';

 class OAuthInterceptor extends AuthInterceptor {
-    Map<String, String> tokens = {};
+  Map<String, String> tokens = {};


final for consistency?

This is much faster than individual files. The committed samples should be formatted since Dart is very opinionated and it makes diffs in PRs much easier to read. Due to this, we also need to format in CI, otherwise there is a git diff.

jannikrh · 2021-03-26T12:18:37Z

If the additional library timemachine is not used, the Date format cannot be used. A complete DateTime string is always generated. Is it possible to distinguish between Date and DateTime like in the "normal" Dart generator?

As an example the following formatter is used in the Dart generator: final _dateFormatter = DateFormat('yyyy-MM-dd');
The example of the distinction between Date and DateTime is in the following file starting at line 36.
openapi-generator/modules/openapi-generator/src/main/resources/dart2/serialization/native/native_class.mustache

agilob · 2021-03-26T13:44:16Z

If the additional library timemachine is not used, the Date format cannot be used. A complete DateTime string is always generated. Is it possible to distinguish between Date and DateTime like in the "normal" Dart generator?

Could you open an issue for that and call me and @kuhnroyal?

kuhnroyal · 2021-03-26T14:29:29Z

Don't think this is possible unfortunately, but I have not tried. built_value can only have one serializer per class and date/datetime is the same class DateTime in plain Dart.

jannikrh · 2021-03-26T14:37:45Z

I will open an issue later today and tag you. The implementation might be possible by extending Iso8601DateTimeSerializer in combination with FullType parameters. The DateTime serializer would be called in each case. Once as now and once with the parameter date or similar.

agilob · 2021-03-26T14:46:53Z

lets not comment here again, we will lose context of the problem. call us to the issue

TimWhiting · 2022-05-30T18:59:58Z

@kuhnroyal
I'd love to have freezed support. I'm not understanding the relation of rrousselGit/freezed#74 to enum support in OpenAPI. Is it a problem with modeling enums or serializing? Serializing should be taken care of by JsonSerializable shouldn't it? Can you elaborate more on the issues with freezed support?

Abhilash-Chandran · 2022-05-31T12:20:35Z

@TimWhiting I already have a version that uses freezed. I just need to clean it up a bit and raise a PR which probably @kuhnroyal could validate. I am using normal enum and seems to work well. Will create a PR today... 😄

TimWhiting · 2022-06-02T03:52:33Z

@Abhilash-Chandran I'm happy to test it on my apis to see if it makes sense. Can you push what you have so far to a branch?

dastein1 · 2022-06-21T21:05:11Z

@Abhilash-Chandran I'd be interested in this as well.

TimWhiting · 2022-06-22T04:45:26Z

@dastein1 @Abhilash-Chandran
Abhilash, I've taken your branch that you gave me the link to, and squashed your commits, and rebased on master. It has a much cleaner PR / diff now. However, I think I broke something in the rebase. Anyways, for those who are interested, Abhilash's branch does work even if it is a bit out of date / sync with the rest of the repo:

Here is a link to @Abhilash-Chandran's branch:
https://github.com/bahag-chandrana/openapi-generator
Here is the link to the rebased changes:
https://github.com/TimWhiting/openapi-generator/tree/freezed_2

If anyone wants to try to get the rebased branch working & a PR opened, that would probably be a good place to start. I'll see what time I have, but I can't make any promises. I was hoping to see allOf and oneOf support with inheritance & freezed unions, but unfortunately that looks like it will be more work.

Here are the instructions I got from @Abhilash-Chandran to run his branch:
(Build the project from source as explained in the readme)
You can find the jar inside modules/openapi-generator-cli/target/openapi-generator-cli.jar

java -jar ../openapi-generator-cli.jar generate \
      -i api_spec.yaml \
      -g dart-dio-next \
      -c generator_config.yaml

with the generator_config.yaml containing the additional property to enable freezed.

additionalProperties:
  serializationLibrary: freezed

Abhilash-Chandran · 2022-06-23T07:28:22Z

I saw some simiallar issues while building after rebasing. I will try to rework on this from the fresh branch. But no promises yet. Especially the way the templates are referred to has changed since I began working on this. So my idea is to port my changes directly on a fresh fork and hence avoid a host of other errors.

TimWhiting · 2022-06-24T03:55:36Z

@Abhilash-Chandran I did end up copying the json_serializable/api folder to the freezed folder. That seemed to be the only major change needed to get it to compile & generate valid dart files. However, I have not tried using the generated api yet. The changes are on my branch.

auto-labeler bot added the Client: Dart label Mar 2, 2021

kuhnroyal mentioned this pull request Mar 2, 2021

[DART] Add null-safety for dart 2.12+ #8278

Closed

kuhnroyal added 11 commits March 7, 2021 14:48

[dart-dio] Remove old Dart 1.x templates

d2b2dc1

[dart-dio] Initial version of a next-gen Dart Dio generator

0cb718f

Fix a couple readme links and examples

c8bc3ba

Fix import in testcase

7fd2b4a

Add integration tests to master POM

69ab071

Run initial dart format and improve some base formatting

8b70359

Generate docs

ff27b0b

Use integer matchers in test for content-length

3f9f78b

Separate more built_value specific files

e41a1cc

Fix mustache partial path

c22138e

Update pubspec template

84f59de

kuhnroyal force-pushed the dio-next-gen branch from 972f19e to 84f59de Compare March 7, 2021 15:15

kuhnroyal added 6 commits March 7, 2021 16:27

Fix tests after mock lib updates

6e6607a

Generate with new built_value pre-release

331ceaf

This solves the problem of not being able to serialize null.

Update built_value dependency

26e7270

Merge master

db3711b

Regenerate after merge

82303c3

Regenerate tests

16c32ec

kuhnroyal mentioned this pull request Mar 12, 2021

[dart] Split mustache file by serialization #8958

Merged

5 tasks

agilob mentioned this pull request Mar 15, 2021

Fix #8971 Updating pubspec.yaml template for dart2 generator #8978

Closed

Nikarous mentioned this pull request Mar 16, 2021

[BUG][DART] Update Intl package version #8971

Closed

agilob reviewed Mar 18, 2021

View reviewed changes

kuhnroyal added 6 commits March 18, 2021 19:16

Make a couple variables final

b20769a

Update and freeze dio dependency

e97c40e

Update generator doc

9e6c44a

Test some CI stuff

08ae169

Merge branch 'master' into dio-next-gen

8240a4f

wing328 marked this pull request as ready for review March 19, 2021 17:02

petrleocompel mentioned this pull request Mar 19, 2021

Code Generation stucks after Flutter 2.0 Upgrade gibahjoe/openapi-generator-dart#38

Closed

wing328 added the Enhancement: New generator label Mar 20, 2021

wing328 added this to the 5.1.0 milestone Mar 20, 2021

wing328 merged commit 3d038b7 into OpenAPITools:master Mar 20, 2021

kuhnroyal deleted the dio-next-gen branch March 20, 2021 12:29

jannikrh mentioned this pull request Mar 26, 2021

[dart-dio][dart-dio-next] Support Date Type #9095

Closed

kuhnroyal mentioned this pull request Mar 29, 2021

[Dart-dio] Generator force using built_value #9082

Closed

NANASHI0X74 mentioned this pull request Apr 15, 2021

[BUG] [DART] The default readme file has missing/misleading information #9186

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dart][dart-dio] Next-gen dart-dio generator #8869

[dart][dart-dio] Next-gen dart-dio generator #8869

kuhnroyal commented Mar 2, 2021 •

edited

Loading

auto-labeler bot commented Mar 2, 2021

kuhnroyal commented Mar 2, 2021

robrbecker commented Mar 3, 2021 •

edited

Loading

robrbecker commented Mar 3, 2021

kuhnroyal commented Mar 3, 2021

vishna commented Mar 7, 2021

kuhnroyal commented Mar 7, 2021

agilob Mar 18, 2021

kuhnroyal Mar 18, 2021

agilob Mar 18, 2021

jannikrh commented Mar 26, 2021 •

edited

Loading

agilob commented Mar 26, 2021 •

edited

Loading

kuhnroyal commented Mar 26, 2021

jannikrh commented Mar 26, 2021

agilob commented Mar 26, 2021

TimWhiting commented May 30, 2022

Abhilash-Chandran commented May 31, 2022

TimWhiting commented Jun 2, 2022

dastein1 commented Jun 21, 2022

TimWhiting commented Jun 22, 2022 •

edited

Loading

Abhilash-Chandran commented Jun 23, 2022

TimWhiting commented Jun 24, 2022

[dart][dart-dio] Next-gen dart-dio generator #8869

[dart][dart-dio] Next-gen dart-dio generator #8869

Conversation

kuhnroyal commented Mar 2, 2021 • edited Loading

WIP!

Description

TODO

PR checklist

auto-labeler bot commented Mar 2, 2021

kuhnroyal commented Mar 2, 2021

robrbecker commented Mar 3, 2021 • edited Loading

robrbecker commented Mar 3, 2021

kuhnroyal commented Mar 3, 2021

vishna commented Mar 7, 2021

kuhnroyal commented Mar 7, 2021

agilob Mar 18, 2021

Choose a reason for hiding this comment

kuhnroyal Mar 18, 2021

Choose a reason for hiding this comment

agilob Mar 18, 2021

Choose a reason for hiding this comment

jannikrh commented Mar 26, 2021 • edited Loading

agilob commented Mar 26, 2021 • edited Loading

kuhnroyal commented Mar 26, 2021

jannikrh commented Mar 26, 2021

agilob commented Mar 26, 2021

TimWhiting commented May 30, 2022

Abhilash-Chandran commented May 31, 2022

TimWhiting commented Jun 2, 2022

dastein1 commented Jun 21, 2022

TimWhiting commented Jun 22, 2022 • edited Loading

Abhilash-Chandran commented Jun 23, 2022

TimWhiting commented Jun 24, 2022

kuhnroyal commented Mar 2, 2021 •

edited

Loading

robrbecker commented Mar 3, 2021 •

edited

Loading

jannikrh commented Mar 26, 2021 •

edited

Loading

agilob commented Mar 26, 2021 •

edited

Loading

TimWhiting commented Jun 22, 2022 •

edited

Loading