Fix/enable-handle-fs-connector

[kirtimanmishrazipstack]

What

• Azure FS, GCS have been enabled
• Made necessary changes to comprehend all unstract fs connector
• Remove path in minio, not used
• Remove overwrite parameter from destination connector

Why

• Remove path in minio, because path is not used. In test connection, we are only using bucket.
• Remove overwrite parameter from destination connector. This is done because all fsspec connector is not using overwrite parms. Connectors like Dropbox, minio, GCS, Azure FS will always overwrite the file if present. Where as Gdrive always creates a new file if already present. So this param is not needed in destination connector

How

• Azure FS, GCS have been enabled from FE
• Made necessary changes in backend/workflow_manager/endpoint/source.py and backend/workflow_manager/endpoint/destination.py to comprehend all unstract fs connector
• Remove path in minio from unstract/connectors/filesystems/minio/minio.py
• Remove overwrite param backend/workflow_manager/endpoint/static/dest/file.json to remove overwrite from destination connector

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

• This PR only affects the FS connector. All FS connector have tested on local

Database Migrations

• N/A

Env Config

• N/A

Relevant Docs

Related Issues or PRs

Dependencies Versions

Notes on Testing

• Tested all 5 FS connectors as a source and destination on workflow

Screenshots

Checklist

I have read and understood the Contribution Guidelines.

[chandrasekharan-zipstack]

Changes look good for the most part, however do confirm if this requires any migration to avoid regressions on removing the parameters. I believe we persist these details in the DB

[kirtimanmishrazipstack]

Changes look good for the most part, however do confirm if this requires any migration to avoid regressions on removing the parameters. I believe we persist these details in the DB

@chandrasekharan-zipstack I believe we are storing as json in workflow_endpoints table. When I remove the params, its will only change the json. So no migration changes as per my understanding.
@muhammad-ali-e let me know your thoughts

[muhammad-ali-e]

@kirtimanmishrazipstack Overall LGTM. But Please try to move the functions or methods that related to specific connectors in to their own classes or their base classes

[muhammad-ali-e]

@chandrasekharan-zipstack I believe we are storing as json in workflow_endpoints table. When I remove the params, its will only change the json. So no migration changes as per my understanding.
@muhammad-ali-e let me know your thoughts

Yes I beleive so.It will not affect anything if it not used anywhere.

[ritwik-g]

@kirtimanmishrazipstack I have only one comment. Other than that it looks fine.

But please make sure to address comments from @muhammad-ali-e before merging

[muhammad-ali-e]

LGTM, Please ensure thorough testing with both new and existing connectors.

[github-actions]

filepath	function	$$\textcolor{#23d18b}{\tt{passed}}$$	SUBTOTAL
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_logs}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_client\_init}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_image}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$	$$\textcolor{#23d18b}{\tt{test\_run\_container}}$$	$$\textcolor{#23d18b}{\tt{1}}$$	$$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$		$$\textcolor{#23d18b}{\tt{9}}$$	$$\textcolor{#23d18b}{\tt{9}}$$

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud