[SegFormer] TensorFlow port by sayakpaul · Pull Request #17910 · huggingface/transformers

sayakpaul · 2022-06-28T05:25:46Z

This PR adds the SegFormer model in TensorFlow (probably the first Transformer-based segmentation model in TensorFlow for which we have PT weights available).

TODOs

Write the foundation components
Write the image classification layer
Write components w.r.t semantic segmentation
Write the semantic segmentation layer
Add code examples after call() methods where relevant
Write tests
Modify other related utilities
Create Space to allow users to try out the models (preferably with ONNX to reduce the time?)
Create a Colab Notebook

The final two points are unrelated to the merging of this PR.

Cc: @deep-diver

HuggingFaceDocBuilderDev · 2022-06-28T05:34:46Z

The documentation is not available anymore as the PR was closed or merged.

LysandreJik · 2022-06-28T08:34:57Z

Hey @sayakpaul! The code quality error comes from a new release from black. Rebasing on main should solve the issue as you'll benefit from #17918.

Co-authored-by: matt <rocketknight1@gmail.com>

src/transformers/models/segformer/modeling_tf_segformer.py

sayakpaul · 2022-06-30T08:55:50Z

@Rocketknight1 @sgugger this PR is now ready for your review. Some things to note:

This is the first segmentation model on the TF side that has pre-trained segmentation checkpoints available. Hopefully, it serves as a good foundation for devs contributing TF segmentation models in the future.
@deep-diver and I will work on creating a Space and Colab notebook (for off-the-shelf inference and fine-tuning) to allow users to take advantage of a state-of-the-art segmentation model like this one in TF via transformers.

~~@NielsRogge even though the error in the CI is coming from run_tests_pipelines_tf it seems like the PT test is what is originating the error. Do you mind taking a look?~~

src/transformers/models/segformer/modeling_tf_segformer.py

sgugger

Very clean!
The error you mention is caused by the fact that in test_modeling_segformer.py, the config is only imported under an is_torch_available() (which is a mistake). This test file is used by the TF pipeline test to get the small segformer config, hence the errors. Could you just put that import at the top level of this test file? It should resolve the failure.

src/transformers/models/segformer/modeling_tf_segformer.py

sgugger

@Rocketknight1 Could you have a look as well?

sayakpaul · 2022-07-01T13:15:33Z

Yes @Rocketknight1's comments are needed here as well.

Rocketknight1 · 2022-07-01T18:17:20Z

Yes, I'm sorry! I went deep on a couple of PRs yesterday and today - one for datasets, the other for XLA in transformers, and haven't had time to review this properly yet. I'll get to it ASAP, though!

Rocketknight1

I just read through the model code, and I'm going to go through the test notes now! Overall, I think the code is very high-quality, and I didn't see any obvious issues anywhere.

If I have one nitpick, it's that I prefer operators like / and > to functions like tf.divide and tf.greater - I think they read more clearly, and they're computationally identical. Still, this is purely an aesthetic preference - if you prefer tf.greater, feel free to keep it that way!

tests/models/segformer/test_modeling_tf_segformer.py

gante

(Approving model_doc/segformer.mdx only, as per @sayakpaul review request) 👍

sayakpaul · 2022-07-14T12:06:17Z

Thanks, @gante. There were no changes except that. Those were reviewed and approved by @Rocketknight1 and @ydshieh.

ydshieh

2 tiny naming issues. Thanks @sayakpaul !

src/transformers/models/segformer/modeling_tf_segformer.py

NielsRogge · 2022-07-19T18:17:01Z

utils/documentation_tests.txt

 src/transformers/models/speech_encoder_decoder/modeling_speech_encoder_decoder.py
 src/transformers/models/speech_to_text/modeling_speech_to_text.py
 src/transformers/models/speech_to_text_2/modeling_speech_to_text_2.py
+src/transformers/models/segformer/modeling_tf_segformer.py


I see that the doc example for semantic segmentation doesn't actually test anything.

You only test something if you include an expected output. We can include the expected output shape.

Feel free to update the PyTorch one as well

Both the semantic segmentation classes don't have the add_code_sample_docstrings decorator. Any suggestion on where to best include the expected output shape?

It should be added below the code snippet. You can find an example here:

transformers/src/transformers/models/detr/modeling_detr.py

Line 1413 in bc6fe6f

['cat', 'cat', 'couch', 'remote', 'remote']

I don't see anything in SegformerForSemanticSegmentation.

I did include a comment though:

>>> # logits are of shape (batch_size, num_labels, height, width)

Could you provide an example where the expected shape has been provided in the way you're suggesting?

Nevermind. Added something. Let me know.

Yes looks good! Should be added for the PyTorch one as well

I have added it. Refer here: https://github.com/huggingface/transformers/pull/17910/files#diff-3b11fdec06eefd9ffc22616fda87863b8bc2c54ea175c63ec08333490116861a

src/transformers/models/segformer/modeling_tf_segformer.py

NielsRogge

Very clean implementation, thanks a lot! Left some minor comments.

src/transformers/models/segformer/modeling_tf_segformer.py

* add: segformer utils and img. classification. * add: segmentation layer. * feat: working implementation of segformer. * chore: remove unused variable. * add test, remaining modifications. * remove: unnecessary files. * add: rest of the files. Co-authored-by: matt <rocketknight1@gmail.com> * chore: remove ModuleList comment. * chore: apply make style. * chore: apply make fixup-copies. * add to check_repo.py * add decode head to IGNORE_NON_TESTED * chore: run make style. * chore: PR comments. * chore: minor changes to model doc. * tests: reduction across samples. * add a note on the space. * sort importats. * fix: reduction in loss computation. * chore: align loss function with that of NER. * chore: correct utils/documentation_tests.txt Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * chore: simplify the interpolation of logits in loss computation. * chore: return transposed logits when return_dict=False. * chore: add link to the tf fine-tuning repo. * address pr comments. * address niels's comments. * remove from_pt=True since tf weights are in. * remove comment from pt model. * address niels's comments. Co-authored-by: matt <rocketknight1@gmail.com> Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

add: segformer utils and img. classification.

b591887

add: segmentation layer.

a6f8781

sayakpaul and others added 6 commits June 28, 2022 15:10

feat: working implementation of segformer.

4ebb62a

Merge branch 'main' into tf-segformer

15c7271

chore: remove unused variable.

b994e35

add test, remaining modifications.

d472c38

remove: unnecessary files.

22db5d0

add: rest of the files.

90629ab

Co-authored-by: matt <rocketknight1@gmail.com>

sayakpaul marked this pull request as ready for review June 30, 2022 07:44

Merge branch 'main' into tf-segformer

59e365c

sayakpaul changed the title ~~[WIP] [SegFormer] TensorFlow port~~ [SegFormer] TensorFlow port Jun 30, 2022

chore: remove ModuleList comment.

8d913e6

sayakpaul commented Jun 30, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

sayakpaul added 5 commits June 30, 2022 13:35

chore: apply make style.

3152f05

chore: apply make fixup-copies.

129db92

add to check_repo.py

93fafd4

add decode head to IGNORE_NON_TESTED

eb33e0f

chore: run make style.

48f836f

deep-diver reviewed Jun 30, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

sgugger reviewed Jun 30, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

src/transformers/models/segformer/modeling_tf_segformer.py Show resolved Hide resolved

chore: PR comments.

828960d

sayakpaul requested a review from sgugger July 1, 2022 07:53

sgugger approved these changes Jul 1, 2022

View reviewed changes

Rocketknight1 approved these changes Jul 1, 2022

View reviewed changes

Rocketknight1 reviewed Jul 1, 2022

View reviewed changes

tests/models/segformer/test_modeling_tf_segformer.py Show resolved Hide resolved

sayakpaul added 2 commits July 8, 2022 10:16

chore: simplify the interpolation of logits in loss computation.

ba93bb4

chore: return transposed logits when return_dict=False.

6c97e8d

gante mentioned this pull request Jul 11, 2022

TF: remove graph mode distinction when processing boolean options #18102

Merged

sayakpaul added 2 commits July 13, 2022 14:33

Merge branch 'main' into tf-segformer

754b145

chore: add link to the tf fine-tuning repo.

4c85484

gante approved these changes Jul 14, 2022

View reviewed changes

ydshieh reviewed Jul 18, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

sayakpaul added 3 commits July 18, 2022 17:42

address pr comments.

4afb097

Merge branch 'main' into tf-segformer

05edc7b

Merge branch 'main' into tf-segformer

59c530e

NielsRogge reviewed Jul 19, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

NielsRogge reviewed Jul 19, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Show resolved Hide resolved

NielsRogge reviewed Jul 19, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

NielsRogge reviewed Jul 19, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

NielsRogge reviewed Jul 19, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

NielsRogge reviewed Jul 19, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

NielsRogge reviewed Jul 19, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

NielsRogge approved these changes Jul 19, 2022

View reviewed changes

sayakpaul added 3 commits July 20, 2022 07:17

address niels's comments.

8dd8b46

remove from_pt=True since tf weights are in.

52affaa

remove comment from pt model.

4a41bdc

NielsRogge reviewed Jul 20, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

NielsRogge reviewed Jul 20, 2022

View reviewed changes

src/transformers/models/segformer/modeling_tf_segformer.py Outdated Show resolved Hide resolved

sayakpaul added 2 commits July 20, 2022 17:07

address niels's comments.

9c1584c

Merge branch 'main' into tf-segformer

ebaec84

gante merged commit 561b9a8 into huggingface:main Jul 21, 2022

Conversation

sayakpaul commented Jun 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODOs

Uh oh!

HuggingFaceDocBuilderDev commented Jun 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LysandreJik commented Jun 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sayakpaul commented Jun 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sgugger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sgugger left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Jul 1, 2022

Uh oh!

Rocketknight1 commented Jul 1, 2022

Uh oh!

Rocketknight1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gante left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul commented Jul 14, 2022

Uh oh!

ydshieh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NielsRogge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

sayakpaul commented Jun 28, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Jun 28, 2022 •

edited

Loading

LysandreJik commented Jun 28, 2022 •

edited

Loading

sayakpaul commented Jun 30, 2022 •

edited

Loading

gante left a comment •

edited

Loading