Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core][bug] FILES is now path relative with no prefixes #7271

Merged
merged 4 commits into from
Aug 22, 2020
Merged

[core][bug] FILES is now path relative with no prefixes #7271

merged 4 commits into from
Aug 22, 2020

Conversation

jimschubert
Copy link
Member

some Java implementations don't honor .relativize documentation fully.
When outDir is /a/b and the input is /a/b/c/d, the result should be c/d.
Some implementations make the output ./c/d which seems to mix the logic
as documented for symlinks. So we need to trim any / or ./ from the start,
as nobody should be generating into system root and our expectation is no ./

This resolves regeneration issues for those on such Java
implementations, although we've not been able to track down the exact
vendor or configurations which might lead to these differences.

cc @OpenAPITools/generator-core-team

cc @sebastien-rosset I believe you were affected by this FILES issue? If so, would you mind checking against this branch to see if it's resolved?

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • If contributing template-only or documentation-only changes which will change sample output, build the project beforehand.
  • Run the shell script ./bin/generate-samples.shto update all Petstore samples related to your fix. This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master. These must match the expectations made by your contribution. You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*. For Windows users, please run the script in Git BASH.
  • File the PR against the correct branch: master
  • Copy the technical committee to review the pull request if your PR is targeting a particular programming language.

@jimschubert
FILES is now path relative with no prefixes
04cc4be 
some Java implementations don't honor .relativize documentation fully.
When outDir is /a/b and the input is /a/b/c/d, the result should be c/d.
Some implementations make the output ./c/d which seems to mix the logic
as documented for symlinks. So we need to trim any / or ./ from the start,
as nobody should be generating into system root and our expectation is no ./

This resolves regeneration issues for those on such Java
implementations, although we've not been able to track down the exact
vendor or configurations which might lead to these differences.
@auto-labeler
Copy link

auto-labeler bot commented Aug 22, 2020

👍 Thanks for opening this issue!
🏷 I have applied any labels matching special text in your issue.

The team will review the labels and make any necessary changes.

@sebastien-rosset
Copy link
Contributor

Yes, I see this problem when generating the samples in my local workspace (but not in CI). I'll apply your changes to my branch now to check.

@jimschubert
Copy link
Member Author

@sebastien-rosset I appreciate it. I noticed another issue related to FILES on Windows while fixing the cross-version check, and pushed a fix for this to this PR around the time of your comment. I think you said you were on Mac, but just wanted to give a heads just in case.

@sebastien-rosset
Copy link
Contributor

@sebastien-rosset I appreciate it. I noticed another issue related to FILES on Windows while fixing the cross-version check, and pushed a fix for this to this PR around the time of your comment. I think you said you were on Mac, but just wanted to give a heads just in case.

I'm building in a CentOS Linux VM.

@sebastien-rosset
Copy link
Contributor

I pulled your branch into my local branch, ran mvn package, then bin/generate-samples.sh and I do see './'. I'll do it one more time just to be sure I have the changes from your branch.

#	modified:   samples/client/petstore/typescript-angular-v10-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v10-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-not-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-not-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-not-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-not-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v8-provided-in-root/builds/single-request-parameter/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v8-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v8-provided-in-root/builds/with-prefixed-module-name/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v9-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v9-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-node/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-node/npm/.openapi-generator/FILES
#	modified:   samples/server/petstore/python-blueplanet/.openapi-generator/FILES
index bc66e2a..7f11560 100644
--- a/samples/client/petstore/typescript-angular-v10-provided-in-root/builds/default/.openapi-generator/FILES
+++ b/samples/client/petstore/typescript-angular-v10-provided-in-root/builds/default/.openapi-generator/FILES
@@ -9,11 +9,11 @@ configuration.ts
 encoder.ts
 git_push.sh
 index.ts
-model/./apiResponse.ts
-model/./category.ts
-model/./order.ts
-model/./pet.ts
-model/./tag.ts
-model/./user.ts
+model/apiResponse.ts
+model/category.ts
+model/order.ts
+model/pet.ts
+model/tag.ts
+model/user.ts
 model/models.ts
 variables.ts

@sebastien-rosset
Copy link
Contributor

sebastien-rosset commented Aug 22, 2020 •
edited
Loading

This is what I did:

git merge --no-commit --squash upstream/fix-relative-path-in-files-cache
git status
# On branch add-model-cache
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	modified:   modules/openapi-generator/src/main/java/org/openapitools/codegen/DefaultGenerator.java
#	modified:   samples/server/petstore/php-symfony/SymfonyBundle-php/.openapi-generator/FILES
mvn package
bin/generate-samples.sh
git status
 On branch add-model-cache
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	modified:   modules/openapi-generator/src/main/java/org/openapitools/codegen/DefaultGenerator.java
#	modified:   samples/server/petstore/php-symfony/SymfonyBundle-php/.openapi-generator/FILES
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   samples/client/petstore/typescript-angular-v10-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v10-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-not-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-not-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v6-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-not-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-not-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v7-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v8-provided-in-root/builds/single-request-parameter/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v8-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v8-provided-in-root/builds/with-prefixed-module-name/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v9-provided-in-root/builds/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-angular-v9-provided-in-root/builds/with-npm/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-node/default/.openapi-generator/FILES
#	modified:   samples/client/petstore/typescript-node/npm/.openapi-generator/FILES
#	modified:   samples/server/petstore/python-blueplanet/.openapi-generator/FILES

I do see your changes in my branch (uncommitted).

@jimschubert
Copy link
Member Author

@sebastien-rosset the green in your diff is the new changes correct? If so, then it looks like it's working as expected. Or did I misunderstand your response?

@sebastien-rosset
Copy link
Contributor

@sebastien-rosset the green in your diff is the new changes correct? If so, then it looks like it's working as expected. Or did I misunderstand your response?

ah ok, yes. So you didn't run bin/generate-samples.sh and commit the changes yet, right?

@jimschubert
Copy link
Member Author

@sebastien-rosset no, I did. There was only one change on my branch in a php file. Yours and mine might have different merge bases and the changes you're seeing maybe l may have been fixed already in latest master, or you might have those changes from an earlier commit of yours. CI on this branch is also looking good.

I've been trying to determine if it's a specific version of Java causing this, but wasn't able to find one that's faulty. I did try ubuntu, alpine, and centos so I'm not sure it's caused by the OS.

@jimschubert
Copy link
Member Author

Wait, that's weird. So I do see the ./ inside files on master as seen here. This PR should not have affected path relativization within those… so I'm not sure why it would have fixed it in any of your existing files. this should only remove those from the beginning.

Based on your diff, it seems like that functionality might be random and only for a handful of generators? I'd need to dig into that one a little more to resolve the issue in the middle of the paths.

@sebastien-rosset
Copy link
Contributor

Should we call java.nio.normalize?

@jimschubert
Copy link
Member Author

jimschubert commented Aug 22, 2020 •
edited
Loading

Yeah. Let me try that and see if it solves the problem. Everything outside of this call works off File or Path, so it's a little weird that it's happening here. It must be that some generators are embedding ./ and it's retaining that relative path when constructing new File/Path instances.

@jimschubert
Copy link
Member Author

@sebastien-rosset normalize seems to do the trick. I'll go ahead and merge once CI completes successfully.

image

I'm still curious why this occurs seemingly randomly (such as your diff above).

Thanks for testing it out and recommending the normalize operation for the mid-path fix.

@jimschubert jimschubert merged commit da84d8e into master Aug 22, 2020
@jimschubert jimschubert deleted the fix-relative-path-in-files-cache branch August 22, 2020 19:49
jimschubert added a commit that referenced this pull request Aug 23, 2020
@jimschubert
Merge branch 'master' into addl-props-import
1faa26f 
* master: (720 commits)
  [docs] Update README badges (#7276)
  Update apiInvoker.mustache and sample file for akka-scala client for issue #7258 fix (#7259)
  [Dart] Get all enum values in a list (#7166)
  Update .gitattributes
  [ci] Set ubuntu workflow verification to autoclrf=true, safeclrf=false
  Update check-supported-versions.yaml
  [ci] Update gitattributes and allow skipping docs generation for Windows CI workflows (#7273)
  [core][bug] FILES is now path relative with no prefixes (#7271)
  Update check-supported-versions.yaml
  Update check-supported-versions.yaml (#7268)
  [Java][jersey2] Add jersey injection dependencies (#7240)
  [C][Clang Static Analyzer] Remove the useless variable when assembling URL (#7255)
  Date format dart (#6389)
  minor enhancement to java client generator (#7253)
  typescript: Fix Union Types Import Issue (#6789)
  Modifying the es5 and es6 templates for javascript to handle default values (#6649)
  [python-exp] simplify examples (#7157)
  Support for KumuluzEE microprofile runtime (#5944)
  [C#][netcore] minor improvements and bug fixes (#7244)
  Deprecate Flash (ActionScript) client generator (#7231)
  ...
jimschubert added a commit to mohamedelhabib/openapi-generator that referenced this pull request Aug 24, 2020
@jimschubert
Merge branch 'master' into fix-double-slash-into-markdown-models-links
3219ef8 
* master: (219 commits)
  [java] Appropriate instantiation of model with dynamic properties (OpenAPITools#6052)
  [docs] Update README badges (OpenAPITools#7276)
  Update apiInvoker.mustache and sample file for akka-scala client for issue OpenAPITools#7258 fix (OpenAPITools#7259)
  [Dart] Get all enum values in a list (OpenAPITools#7166)
  Update .gitattributes
  [ci] Set ubuntu workflow verification to autoclrf=true, safeclrf=false
  Update check-supported-versions.yaml
  [ci] Update gitattributes and allow skipping docs generation for Windows CI workflows (OpenAPITools#7273)
  [core][bug] FILES is now path relative with no prefixes (OpenAPITools#7271)
  Update check-supported-versions.yaml
  Update check-supported-versions.yaml (OpenAPITools#7268)
  [Java][jersey2] Add jersey injection dependencies (OpenAPITools#7240)
  [C][Clang Static Analyzer] Remove the useless variable when assembling URL (OpenAPITools#7255)
  Date format dart (OpenAPITools#6389)
  minor enhancement to java client generator (OpenAPITools#7253)
  typescript: Fix Union Types Import Issue (OpenAPITools#6789)
  Modifying the es5 and es6 templates for javascript to handle default values (OpenAPITools#6649)
  [python-exp] simplify examples (OpenAPITools#7157)
  Support for KumuluzEE microprofile runtime (OpenAPITools#5944)
  [C#][netcore] minor improvements and bug fixes (OpenAPITools#7244)
  ...
@wing328 wing328 added this to the 5.0.0 milestone Sep 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Enhancement: General Issue: Bug
Projects
None yet
Milestone
5.0.0
Development

Successfully merging this pull request may close these issues.
3 participants
@jimschubert @sebastien-rosset @wing328