torch.multiprocessing cannot pickle local object DataLoader (pointnet segmentation example) #366
Comments
|
It seems like you do need to set |
|
It was that, thank you @rusty1s for the quick reply ! |
|
I have the same issue. In my case, without multiprocessing the training speed would be much lower. Hope you can fix it. |
|
There's sadly nothing for me to fix since this is a PyTorch issue. |
|
Simply setting |
3 tasks
anthony-chaudhary
added a commit
to diffgram/diffgram
that referenced
this issue
Jun 24, 2022
-> pyg-team/pytorch_geometric#366 num_workers=0 -> appears to work without the color conversion. ` BGR to RGB` doesn't quite make sense since the images would be RGB , so if it is needed then probably need to invert it -> self.session was throwing pickle error from pytorch, after the fact that may be repeat of 366 above but just leaving it for now till can test again -> serialize_with_type() with type should handle the regen URL
PJEstrada
added a commit
to diffgram/diffgram
that referenced
this issue
Jul 14, 2022
* Add Score for better capturing ML prediction values
Clean old comments
* Add score
* Update annotation.py
* Move to Shared
Connections are used both in setup phases with default service and usage phases with other services
* Refactor
-> These files both seemed to do very similar things and their interaction was not super clear to me. I refactored it into one class that should hopefully be improved.
-> As an opinon thing, that magic package grabber with globals is a little "scary". At least for the current context I'm not clear the value of that vs a concrete import. Can discuss if I am missing something of course.
I left the datasaur and scaleai connections out since those are being refactored
#'datasaur': DatasaurConnector, # Being Refactored for migration
#'scale_ai': ScaleAIConnector, # Being Refactored for migration / actions
* Continue refactor
I believe this should be more clear what's happening
* Refactor continued
The labelbox webhook thing is actually deprecated / part of refactor. But just as an example.
* Remove deprecated launch strategies
If we we add this in future, e.g. with Scale AI etc it will be more from migration side, or actions side etc... For now just removing the imports so we can still add it back as needed
* Update task_template_launch_handler.py
* Update to new refactor pattern
* Create alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
* Update connection_strategy.py
* Refactor
For now to keep current pattern labelbox thing is in the strategy pattern
but would prefer to move towards more concrete thing where we just pass the class if needed. Moving the connection concepts to be just for Auth/Client/Admin not the functions done by the client.
* Refactor to shared continued
* Update labelbox_connector.py
* Update regular_api.py
* Move packet concept to shared
Context of continuing to refactor code that may be used by multiple services
Conceptually walrus specific concerns should be extracted
And the queue concept will keep progressing towards using rabbit, at least for the "ingest immediately" ones even if we keep the deferred processing ones in db for a while
Move to "ingest" name since technically "input" is reserved python name
* Refactor names
* Clean
* Protect undefined case
* Add example of more streamlined connection strategy
Also helps disambiguate the `client` vs the `connection` a little bit
* Update packet.py
* refactor fix
* Duplicate imports removal
* Register Concept (Addition)
Each action now supplies a Register function
That will check if it already exists, else create it
Provides a smoother way for developer to do it besides a database migration
A v1 more iteration for how we want to handle updates etc
* Support None
for example manual actions
* Add manual action concept WIP
* clean
* Rename to align
* Add to view as an example WIP
* Create huggingface_zeroshot.py
WIP
* Fix import timing assumptions
* Add shared packet method
* Clean
* WIP on manual
Should be most of the way there
* Update alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
Align
* Remove in shared context
These permissions concepts generally assume
* Deprecate 2 tests until new context created
* clean print statement
* fix
SkipTest is an exception class
* Fix None case
For whatever reason we are sometimes passing None here
So that overrides the default set, but later processing assumes it's a dict
So just putting the check outside signature for now
* Upgrade to use class method
* Deep Checks Action (#917)
* feat: adding better registration function and boilerplate action for reference
* fix: upgrade to flask 2 and add dataset class
* feat: add action run to execute_action
* feat: add action event base classes
* refactor: modify base class
* fix dependencies
Co-authored-by: Pablo <[email protected]>
* feat: action registration functionality
* feat: progress deep checks setup
Report is now generated but null values get. Still wip
* Fix global attributes on the Image interface (#918)
* Fix not showing globals on the image files
* align with LabelSchema.get_by_id() taking project
* Bug fix
Context before was trying to just return the "new" one
But now, we assign the Schema relation.
So if a user created a new group in Schema A, then moved to Schema B, it would not show a group properly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Add error handling for actions import
* Wire up Hugging face action on the backend
* Add right higging face icon to the action
* Add updating of the action and making action anavalible if they are not in the ACTION_RUNNERS_KIND_MAPPER
* feat: deepchecks now working, adding better composability to frontend
* fix: action creation and new base class for action config
* fix: method signatures
* feat: improvements to naming, frontend dev experience
* feat: configure deepchecks
* Fix name of the create task action
* Update avalablity of the action if it already exists
* Update create_task action on the frontend
* Set action key in ACTION_RUNNERS_KIND_MAPPER to the action kind
* Improve import of the actions
* Fix gun of updating avalablity if exists
* Fix type and icon of the Azure Sentiment analisys action
* feat: save html
* WIP: update action creation steps management
* Add new steps handler to the export action
* Generate right index for the steps
* Fix bug of updating the steps element
* feat: save html
* Fix Global Attributes Saving (#940)
* fix: global attributes saving
* clean log statement
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Resolve conflicts
* feat: populate event list from backend
* Fix merge errors
* Add frontend of the hugging face action
* Delete steps_config from the data and leave it only in the props
* feat: deepchecks now working
* Add docker compose file for development(it starts only rabbitMq, postgress and postgress migrations)
* Fix typo
* Update requirements.txt
* placeholder till this gets improved
in slack discussion we are going to move this to a new name and maybe data type
at the moment `trigger_event_name` isn't in the dict
* making it run, and run on windows
-> pyg-team/pytorch_geometric#366 num_workers=0
-> appears to work without the color conversion. ` BGR to RGB` doesn't quite make sense since the images would be RGB , so if it is needed then probably need to invert it
-> self.session was throwing pickle error from pytorch, after the fact that may be repeat of 366 above but just leaving it for now till can test again
-> serialize_with_type() with type should handle the regen URL
* Update ActionConsumer.py
* Fix action template event name
* Fix not updating trigger bug on the wizzard
* Fix unneded update of the action to the default
* Delete dev docker-compose
* Fix quering on the action model
* Add condition to trigger action
* Add zero shot to the action
* Perform zero shot on given selection attribute
* Finihs implememntation of the Hugging face zero shot classification
* Add attribute configuration for hugging face action
* Modify backend of the zero shot action
* Add configuration of the action attribute
* Add select of the attribute
* Fix shema update on the wizzard mount
* Show options that will be passed to the model
* Fix schema selection on reload
* Fix typo
* Add task_created triggier to the hf zero shot action triggers
* Add job selector to the trigger configs step
* Wire up zero shot classififcation to the cak_creation trigger
* Set deafult schma on the frontend if the hf action is triggered on task creation
* Fix bug of nulling attribute group
* Fix updatin job id on the wizzard
* Delete commented code
* Add data for stats to action run
* Add route to retrive action run output by id
* Add service to fetch actions stats and create base component on the frontend
* Populate chart from the backend stats
* Add some styles to the chart
* Fix few console errors
* Fix backend bug for file_upload trigger
* Refactore for CLI
* Update .gitignore [ci skip]
* Wordsmith
link to docs not active yet
* Align with current goals
Refocus on Install over premium
Enterprise (must be qualified Enterprise to get demo)
Also remove legacy super admin buttons
* Delete dunlicated import
* fix: e2e tests
* Fix issue of updating dataset (#960)
* Fix issue of updating dataset
* Predict update of the global state with uodate of the action
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* comments
* Pass action output data to the next action on the action_complete (#963)
* Pass action output data to the next action on the action_complete
* Add output_interface to the action table
* Define output interface on action
* Define output_interafce on actions
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Change output interafce structure
* Dynamically add/hide step on the create tasks action depends on the initial trigger
* Add route to get previus action by action id
* Add frontend selection of the condition
* Add precondition field to the db
* Add precondition to the action update (backend)
* Add precondition saving on the frontend
* Improve communication interface between actions
* Add precondition check for task creation
* Fix bug on the output type
* Fix frontend from crashing
* Final refactoring
* fix: typo
* fix: rename to extra_metadata
* fix: rename preconditions
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Pablo <[email protected]>
* Replace is with == just not to get warning
* fix: proxyfix
* Add conditional statement for passing user email to the event on attribute group creation
* Fix updateattribute route for the case if it called with the generated credeantials and does not have user
* fix: fixes on defaults and types for action completion event
* fix: bugfix duplicate events
* feat: fix requirements
* fix: unit tests
* fix: imports
* fix: imports
* fix: deps
* fix: imports
* fix: imports and depenedncies
* fix: imports
* feat: add open cv deps
* fix: os deps
* fix: improve logging
* fix: imports
* fix: imports
* debug
* feat: cache dir update
* fix: azure requirements and resources ram
* fix: logging
* fix: output fail
* fix: deepchecks shape
* fix: batch size
* Add link to docs
* fix: connection management for threaded context
Improves connection closing and opening mechanisms for error recovery, session management and reconnection for better availability.
* fix: min resources
* Fix multi select (#978)
* Fix handling multy select on the new command pattern
* Fix hugging face to apply multy select correctly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* Refactor
Co-authored-by: PJEstrada <[email protected]>
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
PJEstrada
added a commit
to diffgram/diffgram
that referenced
this issue
Jul 19, 2022
* Add Score for better capturing ML prediction values
Clean old comments
* Add score
* Update annotation.py
* Move to Shared
Connections are used both in setup phases with default service and usage phases with other services
* Refactor
-> These files both seemed to do very similar things and their interaction was not super clear to me. I refactored it into one class that should hopefully be improved.
-> As an opinon thing, that magic package grabber with globals is a little "scary". At least for the current context I'm not clear the value of that vs a concrete import. Can discuss if I am missing something of course.
I left the datasaur and scaleai connections out since those are being refactored
#'datasaur': DatasaurConnector, # Being Refactored for migration
#'scale_ai': ScaleAIConnector, # Being Refactored for migration / actions
* Continue refactor
I believe this should be more clear what's happening
* Refactor continued
The labelbox webhook thing is actually deprecated / part of refactor. But just as an example.
* Remove deprecated launch strategies
If we we add this in future, e.g. with Scale AI etc it will be more from migration side, or actions side etc... For now just removing the imports so we can still add it back as needed
* Update task_template_launch_handler.py
* Update to new refactor pattern
* Create alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
* Update connection_strategy.py
* Refactor
For now to keep current pattern labelbox thing is in the strategy pattern
but would prefer to move towards more concrete thing where we just pass the class if needed. Moving the connection concepts to be just for Auth/Client/Admin not the functions done by the client.
* Refactor to shared continued
* Update labelbox_connector.py
* Update regular_api.py
* Move packet concept to shared
Context of continuing to refactor code that may be used by multiple services
Conceptually walrus specific concerns should be extracted
And the queue concept will keep progressing towards using rabbit, at least for the "ingest immediately" ones even if we keep the deferred processing ones in db for a while
Move to "ingest" name since technically "input" is reserved python name
* Refactor names
* Clean
* Protect undefined case
* Add example of more streamlined connection strategy
Also helps disambiguate the `client` vs the `connection` a little bit
* Update packet.py
* refactor fix
* Duplicate imports removal
* Register Concept (Addition)
Each action now supplies a Register function
That will check if it already exists, else create it
Provides a smoother way for developer to do it besides a database migration
A v1 more iteration for how we want to handle updates etc
* Support None
for example manual actions
* Add manual action concept WIP
* clean
* Rename to align
* Add to view as an example WIP
* Create huggingface_zeroshot.py
WIP
* Fix import timing assumptions
* Add shared packet method
* Clean
* WIP on manual
Should be most of the way there
* Update alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
Align
* Remove in shared context
These permissions concepts generally assume
* Deprecate 2 tests until new context created
* clean print statement
* fix
SkipTest is an exception class
* Fix None case
For whatever reason we are sometimes passing None here
So that overrides the default set, but later processing assumes it's a dict
So just putting the check outside signature for now
* Upgrade to use class method
* Deep Checks Action (#917)
* feat: adding better registration function and boilerplate action for reference
* fix: upgrade to flask 2 and add dataset class
* feat: add action run to execute_action
* feat: add action event base classes
* refactor: modify base class
* fix dependencies
Co-authored-by: Pablo <[email protected]>
* feat: action registration functionality
* feat: progress deep checks setup
Report is now generated but null values get. Still wip
* Fix global attributes on the Image interface (#918)
* Fix not showing globals on the image files
* align with LabelSchema.get_by_id() taking project
* Bug fix
Context before was trying to just return the "new" one
But now, we assign the Schema relation.
So if a user created a new group in Schema A, then moved to Schema B, it would not show a group properly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Add error handling for actions import
* Wire up Hugging face action on the backend
* Add right higging face icon to the action
* Add updating of the action and making action anavalible if they are not in the ACTION_RUNNERS_KIND_MAPPER
* feat: deepchecks now working, adding better composability to frontend
* fix: action creation and new base class for action config
* fix: method signatures
* feat: improvements to naming, frontend dev experience
* feat: configure deepchecks
* Fix name of the create task action
* Update avalablity of the action if it already exists
* Update create_task action on the frontend
* Set action key in ACTION_RUNNERS_KIND_MAPPER to the action kind
* Improve import of the actions
* Fix gun of updating avalablity if exists
* Fix type and icon of the Azure Sentiment analisys action
* feat: save html
* WIP: update action creation steps management
* Add new steps handler to the export action
* Generate right index for the steps
* Fix bug of updating the steps element
* feat: save html
* Fix Global Attributes Saving (#940)
* fix: global attributes saving
* clean log statement
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Resolve conflicts
* feat: populate event list from backend
* Fix merge errors
* Add frontend of the hugging face action
* Delete steps_config from the data and leave it only in the props
* feat: deepchecks now working
* Add docker compose file for development(it starts only rabbitMq, postgress and postgress migrations)
* Fix typo
* Update requirements.txt
* placeholder till this gets improved
in slack discussion we are going to move this to a new name and maybe data type
at the moment `trigger_event_name` isn't in the dict
* making it run, and run on windows
-> pyg-team/pytorch_geometric#366 num_workers=0
-> appears to work without the color conversion. ` BGR to RGB` doesn't quite make sense since the images would be RGB , so if it is needed then probably need to invert it
-> self.session was throwing pickle error from pytorch, after the fact that may be repeat of 366 above but just leaving it for now till can test again
-> serialize_with_type() with type should handle the regen URL
* Update ActionConsumer.py
* Fix action template event name
* Fix not updating trigger bug on the wizzard
* Fix unneded update of the action to the default
* Delete dev docker-compose
* Fix quering on the action model
* Add condition to trigger action
* Add zero shot to the action
* Perform zero shot on given selection attribute
* Finihs implememntation of the Hugging face zero shot classification
* Add attribute configuration for hugging face action
* Modify backend of the zero shot action
* Add configuration of the action attribute
* Add select of the attribute
* Fix shema update on the wizzard mount
* Show options that will be passed to the model
* Fix schema selection on reload
* Fix typo
* Add task_created triggier to the hf zero shot action triggers
* Add job selector to the trigger configs step
* Wire up zero shot classififcation to the cak_creation trigger
* Set deafult schma on the frontend if the hf action is triggered on task creation
* Fix bug of nulling attribute group
* Fix updatin job id on the wizzard
* Delete commented code
* Add data for stats to action run
* Add route to retrive action run output by id
* Add service to fetch actions stats and create base component on the frontend
* Populate chart from the backend stats
* Add some styles to the chart
* Fix few console errors
* Fix backend bug for file_upload trigger
* Refactore for CLI
* Update .gitignore [ci skip]
* Wordsmith
link to docs not active yet
* Delete dunlicated import
* fix: e2e tests
* Fix issue of updating dataset (#960)
* Fix issue of updating dataset
* Predict update of the global state with uodate of the action
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* comments
* Pass action output data to the next action on the action_complete (#963)
* Pass action output data to the next action on the action_complete
* Add output_interface to the action table
* Define output interface on action
* Define output_interafce on actions
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Change output interafce structure
* Dynamically add/hide step on the create tasks action depends on the initial trigger
* Add route to get previus action by action id
* Add frontend selection of the condition
* Add precondition field to the db
* Add precondition to the action update (backend)
* Add precondition saving on the frontend
* Improve communication interface between actions
* Add precondition check for task creation
* Fix bug on the output type
* Fix frontend from crashing
* Final refactoring
* fix: typo
* fix: rename to extra_metadata
* fix: rename preconditions
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Pablo <[email protected]>
* Replace is with == just not to get warning
* fix: proxyfix
* Add conditional statement for passing user email to the event on attribute group creation
* Fix updateattribute route for the case if it called with the generated credeantials and does not have user
* fix: fixes on defaults and types for action completion event
* fix: bugfix duplicate events
* feat: fix requirements
* fix: unit tests
* fix: imports
* fix: imports
* fix: deps
* fix: imports
* fix: imports and depenedncies
* fix: imports
* feat: add open cv deps
* fix: os deps
* fix: improve logging
* fix: imports
* fix: imports
* debug
* feat: cache dir update
* fix: azure requirements and resources ram
* fix: logging
* fix: output fail
* fix: deepchecks shape
* fix: batch size
* Predict all the children of the tree to be rendreder to the dom and replace default search
* Fix loading all the children on the click
* Make search and load children work together
* Set selectable property of the tree to false and arr checkbox to the UI
* Add condition to open all if search is applied
* Remove logs
* Fix golabl attribues saving
* Uncommited merge conflict changes
* Wire up tree slection to the save
* Add tree construction after the search
* Rewrite find parent function
* Add lazy component to the tree attribuye
Co-authored-by: Anthony Sarkis <[email protected]>
Co-authored-by: PJEstrada <[email protected]>
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
PJEstrada
added a commit
to diffgram/diffgram
that referenced
this issue
Jul 19, 2022
* Add Score for better capturing ML prediction values
Clean old comments
* Add score
* Update annotation.py
* Move to Shared
Connections are used both in setup phases with default service and usage phases with other services
* Refactor
-> These files both seemed to do very similar things and their interaction was not super clear to me. I refactored it into one class that should hopefully be improved.
-> As an opinon thing, that magic package grabber with globals is a little "scary". At least for the current context I'm not clear the value of that vs a concrete import. Can discuss if I am missing something of course.
I left the datasaur and scaleai connections out since those are being refactored
#'datasaur': DatasaurConnector, # Being Refactored for migration
#'scale_ai': ScaleAIConnector, # Being Refactored for migration / actions
* Continue refactor
I believe this should be more clear what's happening
* Refactor continued
The labelbox webhook thing is actually deprecated / part of refactor. But just as an example.
* Remove deprecated launch strategies
If we we add this in future, e.g. with Scale AI etc it will be more from migration side, or actions side etc... For now just removing the imports so we can still add it back as needed
* Update task_template_launch_handler.py
* Update to new refactor pattern
* Create alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
* Update connection_strategy.py
* Refactor
For now to keep current pattern labelbox thing is in the strategy pattern
but would prefer to move towards more concrete thing where we just pass the class if needed. Moving the connection concepts to be just for Auth/Client/Admin not the functions done by the client.
* Refactor to shared continued
* Update labelbox_connector.py
* Update regular_api.py
* Move packet concept to shared
Context of continuing to refactor code that may be used by multiple services
Conceptually walrus specific concerns should be extracted
And the queue concept will keep progressing towards using rabbit, at least for the "ingest immediately" ones even if we keep the deferred processing ones in db for a while
Move to "ingest" name since technically "input" is reserved python name
* Refactor names
* Clean
* Protect undefined case
* Add example of more streamlined connection strategy
Also helps disambiguate the `client` vs the `connection` a little bit
* Update packet.py
* refactor fix
* Duplicate imports removal
* Register Concept (Addition)
Each action now supplies a Register function
That will check if it already exists, else create it
Provides a smoother way for developer to do it besides a database migration
A v1 more iteration for how we want to handle updates etc
* Support None
for example manual actions
* Add manual action concept WIP
* clean
* Rename to align
* Add to view as an example WIP
* Create huggingface_zeroshot.py
WIP
* Fix import timing assumptions
* Add shared packet method
* Clean
* WIP on manual
Should be most of the way there
* Update alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
Align
* Remove in shared context
These permissions concepts generally assume
* Deprecate 2 tests until new context created
* clean print statement
* fix
SkipTest is an exception class
* Fix None case
For whatever reason we are sometimes passing None here
So that overrides the default set, but later processing assumes it's a dict
So just putting the check outside signature for now
* Upgrade to use class method
* Deep Checks Action (#917)
* feat: adding better registration function and boilerplate action for reference
* fix: upgrade to flask 2 and add dataset class
* feat: add action run to execute_action
* feat: add action event base classes
* refactor: modify base class
* fix dependencies
Co-authored-by: Pablo <[email protected]>
* feat: action registration functionality
* feat: progress deep checks setup
Report is now generated but null values get. Still wip
* Fix global attributes on the Image interface (#918)
* Fix not showing globals on the image files
* align with LabelSchema.get_by_id() taking project
* Bug fix
Context before was trying to just return the "new" one
But now, we assign the Schema relation.
So if a user created a new group in Schema A, then moved to Schema B, it would not show a group properly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Add error handling for actions import
* Wire up Hugging face action on the backend
* Add right higging face icon to the action
* Add updating of the action and making action anavalible if they are not in the ACTION_RUNNERS_KIND_MAPPER
* feat: deepchecks now working, adding better composability to frontend
* fix: action creation and new base class for action config
* fix: method signatures
* feat: improvements to naming, frontend dev experience
* feat: configure deepchecks
* Fix name of the create task action
* Update avalablity of the action if it already exists
* Update create_task action on the frontend
* Set action key in ACTION_RUNNERS_KIND_MAPPER to the action kind
* Improve import of the actions
* Fix gun of updating avalablity if exists
* Fix type and icon of the Azure Sentiment analisys action
* feat: save html
* WIP: update action creation steps management
* Add new steps handler to the export action
* Generate right index for the steps
* Fix bug of updating the steps element
* feat: save html
* Fix Global Attributes Saving (#940)
* fix: global attributes saving
* clean log statement
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Resolve conflicts
* feat: populate event list from backend
* Fix merge errors
* Add frontend of the hugging face action
* Delete steps_config from the data and leave it only in the props
* feat: deepchecks now working
* feat: improve OIDC abstraction layers
* Add docker compose file for development(it starts only rabbitMq, postgress and postgress migrations)
* Fix typo
* Update requirements.txt
* placeholder till this gets improved
in slack discussion we are going to move this to a new name and maybe data type
at the moment `trigger_event_name` isn't in the dict
* making it run, and run on windows
-> pyg-team/pytorch_geometric#366 num_workers=0
-> appears to work without the color conversion. ` BGR to RGB` doesn't quite make sense since the images would be RGB , so if it is needed then probably need to invert it
-> self.session was throwing pickle error from pytorch, after the fact that may be repeat of 366 above but just leaving it for now till can test again
-> serialize_with_type() with type should handle the regen URL
* Update ActionConsumer.py
* Fix action template event name
* Fix not updating trigger bug on the wizzard
* Fix unneded update of the action to the default
* fix: unit tests
* Delete dev docker-compose
* Fix quering on the action model
* Add condition to trigger action
* Add zero shot to the action
* Perform zero shot on given selection attribute
* feat: adding connection to file and input
* Finihs implememntation of the Hugging face zero shot classification
* fix: reformat
* Add attribute configuration for hugging face action
* Modify backend of the zero shot action
* Add configuration of the action attribute
* feat: add bucket name
* Add select of the attribute
* feat: files from blob can now be uploaded
Fetching part pending
* Fix shema update on the wizzard mount
* fix: error display
* Show options that will be passed to the model
* Fix schema selection on reload
* Fix typo
* Add task_created triggier to the hf zero shot action triggers
* Add job selector to the trigger configs step
* Wire up zero shot classififcation to the cak_creation trigger
* Set deafult schma on the frontend if the hf action is triggered on task creation
* Fix bug of nulling attribute group
* Fix updatin job id on the wizzard
* Delete commented code
* Add data for stats to action run
* Add route to retrive action run output by id
* Add service to fetch actions stats and create base component on the frontend
* Populate chart from the backend stats
* Add some styles to the chart
* Fix few console errors
* Fix backend bug for file_upload trigger
* Refactore for CLI
* Update .gitignore [ci skip]
* Wordsmith
link to docs not active yet
* Delete dunlicated import
* fix: e2e tests
* Fix issue of updating dataset (#960)
* Fix issue of updating dataset
* Predict update of the global state with uodate of the action
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* wip: refactor regenerate url to a single point
To reuse it across all blob types in diffgram
* comments
* fix: typo merge conflict
* fix: add unit tests and refactor asset types
Still pending video/frame case since it regenerates some extra assets
* Pass action output data to the next action on the action_complete (#963)
* Pass action output data to the next action on the action_complete
* Add output_interface to the action table
* Define output interface on action
* Define output_interafce on actions
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Change output interafce structure
* Dynamically add/hide step on the create tasks action depends on the initial trigger
* Add route to get previus action by action id
* Add frontend selection of the condition
* Add precondition field to the db
* Add precondition to the action update (backend)
* Add precondition saving on the frontend
* Improve communication interface between actions
* Add precondition check for task creation
* Fix bug on the output type
* Fix frontend from crashing
* Final refactoring
* fix: typo
* fix: rename to extra_metadata
* fix: rename preconditions
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Pablo <[email protected]>
* Replace is with == just not to get warning
* fix: proxyfix
* Add conditional statement for passing user email to the event on attribute group creation
* Fix updateattribute route for the case if it called with the generated credeantials and does not have user
* fix: fixes on defaults and types for action completion event
* fix: bugfix duplicate events
* feat: fix requirements
* fix: unit tests
* fix: imports
* fix: imports
* fix: deps
* fix: imports
* fix: imports and depenedncies
* fix: imports
* feat: add open cv deps
* fix: os deps
* fix: improve logging
* fix: imports
* fix: imports
* debug
* feat: cache dir update
* fix: azure requirements and resources ram
* fix: logging
* fix: output fail
* fix: deepchecks shape
* fix: batch size
* Auto stash before merge of "DAD-305-add-connection-concept-to-upload-process" and "ai-workflow"
* wip: URL generation from connection ID
* feat: add signature
* feat: added support for images rendering on annotation core
Needed to get the width and height from the blob image at run time since we don't process the image at any time
* cleanup
* fix: unit tests
* example refactor to strategy
please update as needed
* An idea
not implemented or added to alembic
thought process is that it provides a check and balance
and could also help debugging without having to rely on a None value
but not 100% sure
* fix: use self in serializer
* fix: strategy pattern and fix e2e tests
Co-authored-by: Anthony Sarkis <[email protected]>
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
What about for MacOS please? |
|
I think |
|
Hi, I'm wondering if there is a fix on the multiprocessing issue? I'm using Pytorch1.12, Windows. |
|
Is this PyG related? Does |
Thank you for the reply @rusty1s ! I'm sorry that I didn't notice this is a PyG repo. I have fixed it(it's something on my dataset issue). |
PJEstrada
added a commit
to diffgram/diffgram
that referenced
this issue
Nov 16, 2022
* Add Score for better capturing ML prediction values
Clean old comments
* Add score
* Update annotation.py
* Move to Shared
Connections are used both in setup phases with default service and usage phases with other services
* Refactor
-> These files both seemed to do very similar things and their interaction was not super clear to me. I refactored it into one class that should hopefully be improved.
-> As an opinon thing, that magic package grabber with globals is a little "scary". At least for the current context I'm not clear the value of that vs a concrete import. Can discuss if I am missing something of course.
I left the datasaur and scaleai connections out since those are being refactored
#'datasaur': DatasaurConnector, # Being Refactored for migration
#'scale_ai': ScaleAIConnector, # Being Refactored for migration / actions
* Continue refactor
I believe this should be more clear what's happening
* Refactor continued
The labelbox webhook thing is actually deprecated / part of refactor. But just as an example.
* Remove deprecated launch strategies
If we we add this in future, e.g. with Scale AI etc it will be more from migration side, or actions side etc... For now just removing the imports so we can still add it back as needed
* Update task_template_launch_handler.py
* Update to new refactor pattern
* Create alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
* Update connection_strategy.py
* Refactor
For now to keep current pattern labelbox thing is in the strategy pattern
but would prefer to move towards more concrete thing where we just pass the class if needed. Moving the connection concepts to be just for Auth/Client/Admin not the functions done by the client.
* Refactor to shared continued
* Update labelbox_connector.py
* Update regular_api.py
* Move packet concept to shared
Context of continuing to refactor code that may be used by multiple services
Conceptually walrus specific concerns should be extracted
And the queue concept will keep progressing towards using rabbit, at least for the "ingest immediately" ones even if we keep the deferred processing ones in db for a while
Move to "ingest" name since technically "input" is reserved python name
* Refactor names
* Clean
* Protect undefined case
* Add example of more streamlined connection strategy
Also helps disambiguate the `client` vs the `connection` a little bit
* Update packet.py
* refactor fix
* Duplicate imports removal
* Register Concept (Addition)
Each action now supplies a Register function
That will check if it already exists, else create it
Provides a smoother way for developer to do it besides a database migration
A v1 more iteration for how we want to handle updates etc
* Support None
for example manual actions
* Add manual action concept WIP
* clean
* Rename to align
* Add to view as an example WIP
* Create huggingface_zeroshot.py
WIP
* Fix import timing assumptions
* Add shared packet method
* Clean
* WIP on manual
Should be most of the way there
* Update alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
Align
* Remove in shared context
These permissions concepts generally assume
* Deprecate 2 tests until new context created
* clean print statement
* fix
SkipTest is an exception class
* Fix None case
For whatever reason we are sometimes passing None here
So that overrides the default set, but later processing assumes it's a dict
So just putting the check outside signature for now
* Upgrade to use class method
* Deep Checks Action (#917)
* feat: adding better registration function and boilerplate action for reference
* fix: upgrade to flask 2 and add dataset class
* feat: add action run to execute_action
* feat: add action event base classes
* refactor: modify base class
* fix dependencies
Co-authored-by: Pablo <[email protected]>
* feat: action registration functionality
* feat: progress deep checks setup
Report is now generated but null values get. Still wip
* Fix global attributes on the Image interface (#918)
* Fix not showing globals on the image files
* align with LabelSchema.get_by_id() taking project
* Bug fix
Context before was trying to just return the "new" one
But now, we assign the Schema relation.
So if a user created a new group in Schema A, then moved to Schema B, it would not show a group properly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Add error handling for actions import
* Wire up Hugging face action on the backend
* Add right higging face icon to the action
* Add updating of the action and making action anavalible if they are not in the ACTION_RUNNERS_KIND_MAPPER
* feat: deepchecks now working, adding better composability to frontend
* fix: action creation and new base class for action config
* fix: method signatures
* feat: improvements to naming, frontend dev experience
* feat: configure deepchecks
* Fix name of the create task action
* Update avalablity of the action if it already exists
* Update create_task action on the frontend
* Set action key in ACTION_RUNNERS_KIND_MAPPER to the action kind
* Improve import of the actions
* Fix gun of updating avalablity if exists
* Fix type and icon of the Azure Sentiment analisys action
* feat: save html
* WIP: update action creation steps management
* Add new steps handler to the export action
* Generate right index for the steps
* Fix bug of updating the steps element
* feat: save html
* Fix Global Attributes Saving (#940)
* fix: global attributes saving
* clean log statement
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Resolve conflicts
* feat: populate event list from backend
* Fix merge errors
* Add frontend of the hugging face action
* Delete steps_config from the data and leave it only in the props
* feat: deepchecks now working
* Add docker compose file for development(it starts only rabbitMq, postgress and postgress migrations)
* Fix typo
* Update requirements.txt
* placeholder till this gets improved
in slack discussion we are going to move this to a new name and maybe data type
at the moment `trigger_event_name` isn't in the dict
* making it run, and run on windows
-> pyg-team/pytorch_geometric#366 num_workers=0
-> appears to work without the color conversion. ` BGR to RGB` doesn't quite make sense since the images would be RGB , so if it is needed then probably need to invert it
-> self.session was throwing pickle error from pytorch, after the fact that may be repeat of 366 above but just leaving it for now till can test again
-> serialize_with_type() with type should handle the regen URL
* Update ActionConsumer.py
* Fix action template event name
* Fix not updating trigger bug on the wizzard
* Fix unneded update of the action to the default
* Delete dev docker-compose
* Fix quering on the action model
* Add condition to trigger action
* Add zero shot to the action
* Perform zero shot on given selection attribute
* Finihs implememntation of the Hugging face zero shot classification
* Add attribute configuration for hugging face action
* Modify backend of the zero shot action
* Add configuration of the action attribute
* Add select of the attribute
* Fix shema update on the wizzard mount
* Show options that will be passed to the model
* Fix schema selection on reload
* Fix typo
* Add task_created triggier to the hf zero shot action triggers
* Add job selector to the trigger configs step
* Wire up zero shot classififcation to the cak_creation trigger
* Set deafult schma on the frontend if the hf action is triggered on task creation
* Fix bug of nulling attribute group
* Fix updatin job id on the wizzard
* Delete commented code
* Add data for stats to action run
* Add route to retrive action run output by id
* Add service to fetch actions stats and create base component on the frontend
* Populate chart from the backend stats
* Add some styles to the chart
* Fix few console errors
* Fix backend bug for file_upload trigger
* Refactore for CLI
* Update .gitignore [ci skip]
* Wordsmith
link to docs not active yet
* Align with current goals
Refocus on Install over premium
Enterprise (must be qualified Enterprise to get demo)
Also remove legacy super admin buttons
* Delete dunlicated import
* fix: e2e tests
* Fix issue of updating dataset (#960)
* Fix issue of updating dataset
* Predict update of the global state with uodate of the action
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* comments
* Pass action output data to the next action on the action_complete (#963)
* Pass action output data to the next action on the action_complete
* Add output_interface to the action table
* Define output interface on action
* Define output_interafce on actions
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Change output interafce structure
* Dynamically add/hide step on the create tasks action depends on the initial trigger
* Add route to get previus action by action id
* Add frontend selection of the condition
* Add precondition field to the db
* Add precondition to the action update (backend)
* Add precondition saving on the frontend
* Improve communication interface between actions
* Add precondition check for task creation
* Fix bug on the output type
* Fix frontend from crashing
* Final refactoring
* fix: typo
* fix: rename to extra_metadata
* fix: rename preconditions
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Pablo <[email protected]>
* Replace is with == just not to get warning
* fix: proxyfix
* Add conditional statement for passing user email to the event on attribute group creation
* Fix updateattribute route for the case if it called with the generated credeantials and does not have user
* fix: fixes on defaults and types for action completion event
* fix: bugfix duplicate events
* feat: fix requirements
* fix: unit tests
* fix: imports
* fix: imports
* fix: deps
* fix: imports
* fix: imports and depenedncies
* fix: imports
* feat: add open cv deps
* fix: os deps
* fix: improve logging
* fix: imports
* fix: imports
* debug
* feat: cache dir update
* fix: azure requirements and resources ram
* fix: logging
* fix: output fail
* fix: deepchecks shape
* fix: batch size
* Add link to docs
* fix: connection management for threaded context
Improves connection closing and opening mechanisms for error recovery, session management and reconnection for better availability.
* fix: min resources
* Fix multi select (#978)
* Fix handling multy select on the new command pattern
* Fix hugging face to apply multy select correctly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* Refactor
Co-authored-by: PJEstrada <[email protected]>
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
PJEstrada
added a commit
to diffgram/diffgram
that referenced
this issue
Nov 16, 2022
* Add Score for better capturing ML prediction values
Clean old comments
* Add score
* Update annotation.py
* Move to Shared
Connections are used both in setup phases with default service and usage phases with other services
* Refactor
-> These files both seemed to do very similar things and their interaction was not super clear to me. I refactored it into one class that should hopefully be improved.
-> As an opinon thing, that magic package grabber with globals is a little "scary". At least for the current context I'm not clear the value of that vs a concrete import. Can discuss if I am missing something of course.
I left the datasaur and scaleai connections out since those are being refactored
#'datasaur': DatasaurConnector, # Being Refactored for migration
#'scale_ai': ScaleAIConnector, # Being Refactored for migration / actions
* Continue refactor
I believe this should be more clear what's happening
* Refactor continued
The labelbox webhook thing is actually deprecated / part of refactor. But just as an example.
* Remove deprecated launch strategies
If we we add this in future, e.g. with Scale AI etc it will be more from migration side, or actions side etc... For now just removing the imports so we can still add it back as needed
* Update task_template_launch_handler.py
* Update to new refactor pattern
* Create alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
* Update connection_strategy.py
* Refactor
For now to keep current pattern labelbox thing is in the strategy pattern
but would prefer to move towards more concrete thing where we just pass the class if needed. Moving the connection concepts to be just for Auth/Client/Admin not the functions done by the client.
* Refactor to shared continued
* Update labelbox_connector.py
* Update regular_api.py
* Move packet concept to shared
Context of continuing to refactor code that may be used by multiple services
Conceptually walrus specific concerns should be extracted
And the queue concept will keep progressing towards using rabbit, at least for the "ingest immediately" ones even if we keep the deferred processing ones in db for a while
Move to "ingest" name since technically "input" is reserved python name
* Refactor names
* Clean
* Protect undefined case
* Add example of more streamlined connection strategy
Also helps disambiguate the `client` vs the `connection` a little bit
* Update packet.py
* refactor fix
* Duplicate imports removal
* Register Concept (Addition)
Each action now supplies a Register function
That will check if it already exists, else create it
Provides a smoother way for developer to do it besides a database migration
A v1 more iteration for how we want to handle updates etc
* Support None
for example manual actions
* Add manual action concept WIP
* clean
* Rename to align
* Add to view as an example WIP
* Create huggingface_zeroshot.py
WIP
* Fix import timing assumptions
* Add shared packet method
* Clean
* WIP on manual
Should be most of the way there
* Update alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
Align
* Remove in shared context
These permissions concepts generally assume
* Deprecate 2 tests until new context created
* clean print statement
* fix
SkipTest is an exception class
* Fix None case
For whatever reason we are sometimes passing None here
So that overrides the default set, but later processing assumes it's a dict
So just putting the check outside signature for now
* Upgrade to use class method
* Deep Checks Action (#917)
* feat: adding better registration function and boilerplate action for reference
* fix: upgrade to flask 2 and add dataset class
* feat: add action run to execute_action
* feat: add action event base classes
* refactor: modify base class
* fix dependencies
Co-authored-by: Pablo <[email protected]>
* feat: action registration functionality
* feat: progress deep checks setup
Report is now generated but null values get. Still wip
* Fix global attributes on the Image interface (#918)
* Fix not showing globals on the image files
* align with LabelSchema.get_by_id() taking project
* Bug fix
Context before was trying to just return the "new" one
But now, we assign the Schema relation.
So if a user created a new group in Schema A, then moved to Schema B, it would not show a group properly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Add error handling for actions import
* Wire up Hugging face action on the backend
* Add right higging face icon to the action
* Add updating of the action and making action anavalible if they are not in the ACTION_RUNNERS_KIND_MAPPER
* feat: deepchecks now working, adding better composability to frontend
* fix: action creation and new base class for action config
* fix: method signatures
* feat: improvements to naming, frontend dev experience
* feat: configure deepchecks
* Fix name of the create task action
* Update avalablity of the action if it already exists
* Update create_task action on the frontend
* Set action key in ACTION_RUNNERS_KIND_MAPPER to the action kind
* Improve import of the actions
* Fix gun of updating avalablity if exists
* Fix type and icon of the Azure Sentiment analisys action
* feat: save html
* WIP: update action creation steps management
* Add new steps handler to the export action
* Generate right index for the steps
* Fix bug of updating the steps element
* feat: save html
* Fix Global Attributes Saving (#940)
* fix: global attributes saving
* clean log statement
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Resolve conflicts
* feat: populate event list from backend
* Fix merge errors
* Add frontend of the hugging face action
* Delete steps_config from the data and leave it only in the props
* feat: deepchecks now working
* Add docker compose file for development(it starts only rabbitMq, postgress and postgress migrations)
* Fix typo
* Update requirements.txt
* placeholder till this gets improved
in slack discussion we are going to move this to a new name and maybe data type
at the moment `trigger_event_name` isn't in the dict
* making it run, and run on windows
-> pyg-team/pytorch_geometric#366 num_workers=0
-> appears to work without the color conversion. ` BGR to RGB` doesn't quite make sense since the images would be RGB , so if it is needed then probably need to invert it
-> self.session was throwing pickle error from pytorch, after the fact that may be repeat of 366 above but just leaving it for now till can test again
-> serialize_with_type() with type should handle the regen URL
* Update ActionConsumer.py
* Fix action template event name
* Fix not updating trigger bug on the wizzard
* Fix unneded update of the action to the default
* Delete dev docker-compose
* Fix quering on the action model
* Add condition to trigger action
* Add zero shot to the action
* Perform zero shot on given selection attribute
* Finihs implememntation of the Hugging face zero shot classification
* Add attribute configuration for hugging face action
* Modify backend of the zero shot action
* Add configuration of the action attribute
* Add select of the attribute
* Fix shema update on the wizzard mount
* Show options that will be passed to the model
* Fix schema selection on reload
* Fix typo
* Add task_created triggier to the hf zero shot action triggers
* Add job selector to the trigger configs step
* Wire up zero shot classififcation to the cak_creation trigger
* Set deafult schma on the frontend if the hf action is triggered on task creation
* Fix bug of nulling attribute group
* Fix updatin job id on the wizzard
* Delete commented code
* Add data for stats to action run
* Add route to retrive action run output by id
* Add service to fetch actions stats and create base component on the frontend
* Populate chart from the backend stats
* Add some styles to the chart
* Fix few console errors
* Fix backend bug for file_upload trigger
* Refactore for CLI
* Update .gitignore [ci skip]
* Wordsmith
link to docs not active yet
* Delete dunlicated import
* fix: e2e tests
* Fix issue of updating dataset (#960)
* Fix issue of updating dataset
* Predict update of the global state with uodate of the action
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* comments
* Pass action output data to the next action on the action_complete (#963)
* Pass action output data to the next action on the action_complete
* Add output_interface to the action table
* Define output interface on action
* Define output_interafce on actions
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Change output interafce structure
* Dynamically add/hide step on the create tasks action depends on the initial trigger
* Add route to get previus action by action id
* Add frontend selection of the condition
* Add precondition field to the db
* Add precondition to the action update (backend)
* Add precondition saving on the frontend
* Improve communication interface between actions
* Add precondition check for task creation
* Fix bug on the output type
* Fix frontend from crashing
* Final refactoring
* fix: typo
* fix: rename to extra_metadata
* fix: rename preconditions
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Pablo <[email protected]>
* Replace is with == just not to get warning
* fix: proxyfix
* Add conditional statement for passing user email to the event on attribute group creation
* Fix updateattribute route for the case if it called with the generated credeantials and does not have user
* fix: fixes on defaults and types for action completion event
* fix: bugfix duplicate events
* feat: fix requirements
* fix: unit tests
* fix: imports
* fix: imports
* fix: deps
* fix: imports
* fix: imports and depenedncies
* fix: imports
* feat: add open cv deps
* fix: os deps
* fix: improve logging
* fix: imports
* fix: imports
* debug
* feat: cache dir update
* fix: azure requirements and resources ram
* fix: logging
* fix: output fail
* fix: deepchecks shape
* fix: batch size
* Predict all the children of the tree to be rendreder to the dom and replace default search
* Fix loading all the children on the click
* Make search and load children work together
* Set selectable property of the tree to false and arr checkbox to the UI
* Add condition to open all if search is applied
* Remove logs
* Fix golabl attribues saving
* Uncommited merge conflict changes
* Wire up tree slection to the save
* Add tree construction after the search
* Rewrite find parent function
* Add lazy component to the tree attribuye
Co-authored-by: Anthony Sarkis <[email protected]>
Co-authored-by: PJEstrada <[email protected]>
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
PJEstrada
added a commit
to diffgram/diffgram
that referenced
this issue
Nov 16, 2022
* Add Score for better capturing ML prediction values
Clean old comments
* Add score
* Update annotation.py
* Move to Shared
Connections are used both in setup phases with default service and usage phases with other services
* Refactor
-> These files both seemed to do very similar things and their interaction was not super clear to me. I refactored it into one class that should hopefully be improved.
-> As an opinon thing, that magic package grabber with globals is a little "scary". At least for the current context I'm not clear the value of that vs a concrete import. Can discuss if I am missing something of course.
I left the datasaur and scaleai connections out since those are being refactored
#'datasaur': DatasaurConnector, # Being Refactored for migration
#'scale_ai': ScaleAIConnector, # Being Refactored for migration / actions
* Continue refactor
I believe this should be more clear what's happening
* Refactor continued
The labelbox webhook thing is actually deprecated / part of refactor. But just as an example.
* Remove deprecated launch strategies
If we we add this in future, e.g. with Scale AI etc it will be more from migration side, or actions side etc... For now just removing the imports so we can still add it back as needed
* Update task_template_launch_handler.py
* Update to new refactor pattern
* Create alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
* Update connection_strategy.py
* Refactor
For now to keep current pattern labelbox thing is in the strategy pattern
but would prefer to move towards more concrete thing where we just pass the class if needed. Moving the connection concepts to be just for Auth/Client/Admin not the functions done by the client.
* Refactor to shared continued
* Update labelbox_connector.py
* Update regular_api.py
* Move packet concept to shared
Context of continuing to refactor code that may be used by multiple services
Conceptually walrus specific concerns should be extracted
And the queue concept will keep progressing towards using rabbit, at least for the "ingest immediately" ones even if we keep the deferred processing ones in db for a while
Move to "ingest" name since technically "input" is reserved python name
* Refactor names
* Clean
* Protect undefined case
* Add example of more streamlined connection strategy
Also helps disambiguate the `client` vs the `connection` a little bit
* Update packet.py
* refactor fix
* Duplicate imports removal
* Register Concept (Addition)
Each action now supplies a Register function
That will check if it already exists, else create it
Provides a smoother way for developer to do it besides a database migration
A v1 more iteration for how we want to handle updates etc
* Support None
for example manual actions
* Add manual action concept WIP
* clean
* Rename to align
* Add to view as an example WIP
* Create huggingface_zeroshot.py
WIP
* Fix import timing assumptions
* Add shared packet method
* Clean
* WIP on manual
Should be most of the way there
* Update alembic_2022_06_07_16_51_a47884689fba_improve_ai.py
Align
* Remove in shared context
These permissions concepts generally assume
* Deprecate 2 tests until new context created
* clean print statement
* fix
SkipTest is an exception class
* Fix None case
For whatever reason we are sometimes passing None here
So that overrides the default set, but later processing assumes it's a dict
So just putting the check outside signature for now
* Upgrade to use class method
* Deep Checks Action (#917)
* feat: adding better registration function and boilerplate action for reference
* fix: upgrade to flask 2 and add dataset class
* feat: add action run to execute_action
* feat: add action event base classes
* refactor: modify base class
* fix dependencies
Co-authored-by: Pablo <[email protected]>
* feat: action registration functionality
* feat: progress deep checks setup
Report is now generated but null values get. Still wip
* Fix global attributes on the Image interface (#918)
* Fix not showing globals on the image files
* align with LabelSchema.get_by_id() taking project
* Bug fix
Context before was trying to just return the "new" one
But now, we assign the Schema relation.
So if a user created a new group in Schema A, then moved to Schema B, it would not show a group properly
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Add error handling for actions import
* Wire up Hugging face action on the backend
* Add right higging face icon to the action
* Add updating of the action and making action anavalible if they are not in the ACTION_RUNNERS_KIND_MAPPER
* feat: deepchecks now working, adding better composability to frontend
* fix: action creation and new base class for action config
* fix: method signatures
* feat: improvements to naming, frontend dev experience
* feat: configure deepchecks
* Fix name of the create task action
* Update avalablity of the action if it already exists
* Update create_task action on the frontend
* Set action key in ACTION_RUNNERS_KIND_MAPPER to the action kind
* Improve import of the actions
* Fix gun of updating avalablity if exists
* Fix type and icon of the Azure Sentiment analisys action
* feat: save html
* WIP: update action creation steps management
* Add new steps handler to the export action
* Generate right index for the steps
* Fix bug of updating the steps element
* feat: save html
* Fix Global Attributes Saving (#940)
* fix: global attributes saving
* clean log statement
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Anthony Sarkis <[email protected]>
* Resolve conflicts
* feat: populate event list from backend
* Fix merge errors
* Add frontend of the hugging face action
* Delete steps_config from the data and leave it only in the props
* feat: deepchecks now working
* feat: improve OIDC abstraction layers
* Add docker compose file for development(it starts only rabbitMq, postgress and postgress migrations)
* Fix typo
* Update requirements.txt
* placeholder till this gets improved
in slack discussion we are going to move this to a new name and maybe data type
at the moment `trigger_event_name` isn't in the dict
* making it run, and run on windows
-> pyg-team/pytorch_geometric#366 num_workers=0
-> appears to work without the color conversion. ` BGR to RGB` doesn't quite make sense since the images would be RGB , so if it is needed then probably need to invert it
-> self.session was throwing pickle error from pytorch, after the fact that may be repeat of 366 above but just leaving it for now till can test again
-> serialize_with_type() with type should handle the regen URL
* Update ActionConsumer.py
* Fix action template event name
* Fix not updating trigger bug on the wizzard
* Fix unneded update of the action to the default
* fix: unit tests
* Delete dev docker-compose
* Fix quering on the action model
* Add condition to trigger action
* Add zero shot to the action
* Perform zero shot on given selection attribute
* feat: adding connection to file and input
* Finihs implememntation of the Hugging face zero shot classification
* fix: reformat
* Add attribute configuration for hugging face action
* Modify backend of the zero shot action
* Add configuration of the action attribute
* feat: add bucket name
* Add select of the attribute
* feat: files from blob can now be uploaded
Fetching part pending
* Fix shema update on the wizzard mount
* fix: error display
* Show options that will be passed to the model
* Fix schema selection on reload
* Fix typo
* Add task_created triggier to the hf zero shot action triggers
* Add job selector to the trigger configs step
* Wire up zero shot classififcation to the cak_creation trigger
* Set deafult schma on the frontend if the hf action is triggered on task creation
* Fix bug of nulling attribute group
* Fix updatin job id on the wizzard
* Delete commented code
* Add data for stats to action run
* Add route to retrive action run output by id
* Add service to fetch actions stats and create base component on the frontend
* Populate chart from the backend stats
* Add some styles to the chart
* Fix few console errors
* Fix backend bug for file_upload trigger
* Refactore for CLI
* Update .gitignore [ci skip]
* Wordsmith
link to docs not active yet
* Delete dunlicated import
* fix: e2e tests
* Fix issue of updating dataset (#960)
* Fix issue of updating dataset
* Predict update of the global state with uodate of the action
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
* wip: refactor regenerate url to a single point
To reuse it across all blob types in diffgram
* comments
* fix: typo merge conflict
* fix: add unit tests and refactor asset types
Still pending video/frame case since it regenerates some extra assets
* Pass action output data to the next action on the action_complete (#963)
* Pass action output data to the next action on the action_complete
* Add output_interface to the action table
* Define output interface on action
* Define output_interafce on actions
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Add output intercae to the frontend action class and fix type on the passing it on the backend
* Change output interafce structure
* Dynamically add/hide step on the create tasks action depends on the initial trigger
* Add route to get previus action by action id
* Add frontend selection of the condition
* Add precondition field to the db
* Add precondition to the action update (backend)
* Add precondition saving on the frontend
* Improve communication interface between actions
* Add precondition check for task creation
* Fix bug on the output type
* Fix frontend from crashing
* Final refactoring
* fix: typo
* fix: rename to extra_metadata
* fix: rename preconditions
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Pablo <[email protected]>
* Replace is with == just not to get warning
* fix: proxyfix
* Add conditional statement for passing user email to the event on attribute group creation
* Fix updateattribute route for the case if it called with the generated credeantials and does not have user
* fix: fixes on defaults and types for action completion event
* fix: bugfix duplicate events
* feat: fix requirements
* fix: unit tests
* fix: imports
* fix: imports
* fix: deps
* fix: imports
* fix: imports and depenedncies
* fix: imports
* feat: add open cv deps
* fix: os deps
* fix: improve logging
* fix: imports
* fix: imports
* debug
* feat: cache dir update
* fix: azure requirements and resources ram
* fix: logging
* fix: output fail
* fix: deepchecks shape
* fix: batch size
* Auto stash before merge of "DAD-305-add-connection-concept-to-upload-process" and "ai-workflow"
* wip: URL generation from connection ID
* feat: add signature
* feat: added support for images rendering on annotation core
Needed to get the width and height from the blob image at run time since we don't process the image at any time
* cleanup
* fix: unit tests
* example refactor to strategy
please update as needed
* An idea
not implemented or added to alembic
thought process is that it provides a check and balance
and could also help debugging without having to rely on a None value
but not 100% sure
* fix: use self in serializer
* fix: strategy pattern and fix e2e tests
Co-authored-by: Anthony Sarkis <[email protected]>
Co-authored-by: Pablo <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Co-authored-by: Vitalii Bulyzhyn <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
🐛 Bug
Hi, when trying to run the pointnet++ example on segmentation, I get the following error in utils/data/dataloader.py:
Anyone had similar issues ? read pickle cant pickle lambda functions but I wonder why the default implementation of pytorch is made this way ?
Environment
The text was updated successfully, but these errors were encountered: