Additonal Backbones for Object Detection Task

[aribornstein]

Object Detection task currently only seems to support FasterRCNN it would be good to have additional backbones such as RetinaNet or YoloV5 .

[zhiqwang]

Hi @aribornstein , I'm working on integrating YOLOv5 with Lightning here, I am learning the interface of Lightning and flash, and hope I could help to add YOLOv5 to flash.

Now I open a issue in my own repo to test the trainer and inference of Lightning. After this issue is done, I would like to submit a PR for the YOLOv5 task.

And this is my first attempt on Lightning and flash, if there is a problem with the handling, please point it out.

cc @tchaton

[aribornstein]

That's amazing news !!! Look forward to finally having a clean way of finetuning and comparing different object detectors on TPUs and Distributed Clusters.

[zhiqwang]

Hi @aribornstein

I follow the development philosophy of torchvision's retinanet or faster-rcnn, and try to keep the same interface as torchvision. For the aspects of the YOLOv5 model itself, the work on the inference procedure in done, it can load a checkpoint trained by ultralytics, the inference result is also same as ultralytics.

The work on the training procedure is in process. Although It can also be used for training, I didn't train a model on the actual datasets (neither COCO nor VOC). The author of ultralytics added a lot of tricks to his trainer mechanism. I guess there is still some distance to train a model as good as ultralytics .

[kaushikb11]

@aribornstein RetinaNet with other backbones added #121 :)
@zhiqwang That's great!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[edgarriba]

@zhiqwang what's the status on this. Do you have any progress ?

[zhiqwang]

Hi @edgarriba , I haven't completely solved the training problem ,( I hope I can resolve this problem before June.

[zhiqwang]

Hi, I postponed the DDL again.

I was a little busy with some other work and didn’t push this task forward. I’m now starting to come back to the development here. I'm very sorry that this affects your progress.

[aniketmaurya]

Hi, I have started working on YoloV5 integration with Flash.

[juliangrosshauser]

Integrating YOLOv5 into Lightning Flash would result in a license conflict, because YOLOv5 is licensed under GPL-3.0 and Lightning Flash under Apache 2.0. In order to comply with the GPL-3.0 license you need to disclose your source code, which means you wouldn't be able to use Lightning Flash in closed source projects anymore.

[ethanwharris]

@juliangrosshauser Presumably this would apply to copying the code only though right? Our intention here is simply to have yolov5 as a dependency which is fine no? E.g. we would be adding support for icevision, which lists yolov5 as a dependency and is itself licensed under Apache 2.0. Please correct me if I am mistaken 😃

[juliangrosshauser]

@ethanwharris I'm not a lawyer and this isn't legal advice. I think using YOLOv5 in IceVision makes it a derivative work, and IceVision therefore needs to be GPL-3.0 licensed too. If Flash uses IceVision it creates a derivative work again, which means Flash also needs to be distributed under the GPL-3.0 license. I'm not 100% sure that this is correct, but I wouldn't take the risk of integrating a GPL-3.0 licensed package, especially if I want companies to use my project.

Some relevant links:

• https://opensource.stackexchange.com/questions/1579/when-is-my-project-a-derivative-work-in-an-interpreted-language
• https://opensource.stackexchange.com/questions/2139/can-i-license-python-project-under-3-clause-bsd-while-it-has-gpl-based-dependenc
• https://opensource.stackexchange.com/questions/1640/if-im-using-a-gpl-3-library-in-my-project-can-i-license-my-project-under-mit-l
• http://www.gnu.org/licenses/gpl-faq.html

[ethanwharris]

@juliangrosshauser Thanks for the links. As far as the YoloV5 founder is concerned, it seems that using the library in a or your own trained models with different licensing would be fine: ultralytics/yolov5#967 (comment)

[juliangrosshauser]

Quoting ultralytics/yolov5#967 (comment):

If you simply use this repo without altering it, you own any resulting products (i.e. trained models) and are free to assign them any license you wish as I understand it.

That's probably true for trained models, as long as you're not distributing the models (by providing a REST API for example). AGPL closes this loophole. If you integrate the trained models into your mobile app for example, that you then distribute over app stores, you'd need to disclose the source code again though.

That said, all of that doesn't apply to integrating the package. Flash would still need to be licensed under GPL-3.0 as far as I understand. I might be wrong of course.

[zhiqwang]

Hi all here,

Do you plan to add a wrapper of https://github.com/Megvii-BaseDetection/YOLOX ?

[weiji14]

Hi all here,

Do you plan to add a wrapper of https://github.com/Megvii-BaseDetection/YOLOX ?

I'm interested in this question as well 😄. If my understanding is correct, the IceVision integration PR at #608 will allow Pytorch Lightning to use all the Object Detection models listed by IceVision under https://github.com/airctic/icevision/tree/0.8.1/icevision/models. And since IceVision has integrations with MMDetection, that would include YOLOX (and all the other listed under https://github.com/open-mmlab/mmdetection/tree/v2.15.1#benchmark-and-model-zoo)? Is that correct?

[ethanwharris]

Hi @weiji14 @zhiqwang yes I think this is in the process of being added to icevision via mmdet. Once the models are in icevision we'll be able to add them into flash 😃