Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[New Feature]: Improve dependency management for MI Connectors #3614

Open
ShammiL opened this issue Sep 20, 2024 · 9 comments
Open

[New Feature]: Improve dependency management for MI Connectors #3614

ShammiL opened this issue Sep 20, 2024 · 9 comments
Assignees

Comments

@ShammiL
Copy link
Contributor

ShammiL commented Sep 20, 2024

Description

The current implementations in MI for adding connectors to an integration project and resolving connector dependencies involve several complex and manual steps from both the connector developer and the user sides. To mitigate the problems in the existing implementation, the current mechanisms need to be reformed.

Describe your problem(s)

  1. The VSCode extension uses a static JSON file containing all connector metadata. Developers are required to manually update this list every time a connector is updated or a new one is developed.
  2. When a user adds a connector to an integration project, the user is required to manually add all the dependency jars.
  3. The connector is downloaded and stored inside the project long before build time, when ideally connector and its dependencies should be resolved at build time.

Describe your solution(s)

  1. Come up with a proper API spec for the connector store that allows
    • Listing connectors based on a priority algorithm
    • Searching connectors and operations
    • fetching connector metadata including operations and versions released
  2. Implement workflows to automate releasing connectors to Maven and treat connectors as Maven dependencies of a project
  3. Implement a dependency resolution mechanism to handle connector dependencies and release the users from the burden of having to manually add jars.

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

@ShammiL ShammiL self-assigned this Sep 20, 2024
@ShammiL
Copy link
Contributor Author

ShammiL commented Oct 2, 2024

Following initial discussions with the teams overseeing the MI VSCode extension and the Connector Store, an API specification has been drafted outlining the required functionalities from the Connector Store side. The APIs will provide access to the following features:

  • Retrieve a list of all connectors along with their metadata.
  • Search for specific connectors.
  • Publish new connectors or release updates to existing connectors in the store.
  • Fetch metadata for previous versions of a connector.

The specification can be found here

A final specification will be prepared after reviewing the above draft

@ShammiL
Copy link
Contributor Author

ShammiL commented Oct 14, 2024

One goal of this feature is to automate the connector release process, which is currently done manually. At present, the procedure involves creating the connector ZIP artifacts locally and uploading them to manually created Git release tags. To streamline this process, the original plan was to implement Jenkins workflows to:

  • Build and package the connector into a ZIP artifact,
  • Release the ZIP artifact to the WSO2 public Maven repository,
  • Call the connector store's publish API,
  • Create a release in the corresponding Git repository.

As a preliminary step, before setting these workflows up in the main connector repositories and deploying artifacts to the public Maven repository, the process was tested using a local setup and a fork of a connector repository. This involved:

  • Installing Jenkins locally to create ZIP artifacts, push them to a locally hosted Nexus repository, and generate release tags in the forked Git repository.
  • Setting up a Git Webhook in the forked repository to trigger the Jenkins build process.

However, a decision was taken to transition from Jenkins to GitHub workflows for this task, aligning with the ongoing effort to migrate all MI-related automation tasks from Jenkins to GitHub workflows.

@ShammiL
Copy link
Contributor Author

ShammiL commented Oct 21, 2024

After finalizing the API spec for the connector store[1], as the proposed solution would require collaboration and consensus across multiple teams, and considering the time constraints, a milestone plan has also been outlined in this design document.[2]

[1] https://gist.github.com/ShammiL/606f1bc87af66a33b391864ff6fb12e8
[2] https://docs.google.com/document/d/1XlWv_71iE2VKQtpuWPIzKNpWY66RZcb8qx39Te2R96k/edit?usp=sharing

As our next step, we will be initiating discussions with the teams we believe will be impacted by the proposed changes to ensure alignment and address any concerns.

Simultaneously, other areas highlighted in the design will also be addressed.

@ShammiL
Copy link
Contributor Author

ShammiL commented Oct 22, 2024

After a discussion with members of the IS team looking over IS connectors, it was concluded that the changes proposed for the connector store in Milestone 1 will have no impact on the IS connectors or their current release workflows. Along with that, we also came to the following conclusions:

  1. The new GET API will be implemented aligning with the proposed API spec and the new optional fields in the meta files will be handled from the connector store. This will have no impact on the IS connectors and their release workflows.
  2. Decisions taken regarding the query parameters are as below:
  • The parameter “product” will continue to be used to filter connectors by the product (IS and MI)
  • The "category" parameter will be used exclusively to filter connectors by their type, rather than being utilized to categorize connectors based on various metrics as it is now.In the case of MI connectors there will be two types: Inbound and Outbound connectors
  • The MI and IS teams will define an ENUM of accepted values for the "category" field, which will be used by the connector store to validate and enforce the categories specified in connector metadata files.
  • In the Connector Store UI, filtering by the category will only be visible to the user once a product is selected.

However, a concern was raised about allocating resources for the implementations proposed for the connector store, due to the limited availability of members on the internal apps team.

@ShammiL
Copy link
Contributor Author

ShammiL commented Oct 28, 2024

A release workflow has been implemented in the Redis Connector repository, marking the initial step in integrating release workflows across all MI connector repositories. The workflow makes use of the Maven release plugin to perform the release.
PR - wso2-extensions/mi-connector-redis#60

This workflow will:

  • Create a ZIP release artifact through a Maven build
  • Publish the artifact to the WSO2 Nexus repository using the correct release versions
  • Generate release tags in the Git repository and attach the release artifacts
  • Prepare the connector for the next release.

The individual steps and the overall workflow were initially tested in a local environment and then on a forked repository of the Redis Connector. A locally configured Nexus server was used to simulate the WSO2 release repository.

Once the workflow has been tested on this repository, the same changes will be applied to all MI connector repositories

@ShammiL
Copy link
Contributor Author

ShammiL commented Nov 6, 2024

Due to the lack of resources to be allocated from the digital transformation team, the MI team will implement the new GET endpoint to retrieve all MI connector details to be used by the MI VS Code extension. [1].

This implementation will cover the following:

  1. Add two new tables to the connector store database mi_metadata and mi_operations.
  2. Add a new GET endpoint to retrieve connector metadata including the mi-specific metadata and operations
  3. Update the webhooks and the daily sync jobs to read the new metadata fields in the MI connector repositories and update the database accordingly.
    Tentative structures for the tables are as below:

mi_metadata
image

mi_operation
image

[1] https://github.com/wso2-enterprise/integration-product-management/issues/248

@ShammiL
Copy link
Contributor Author

ShammiL commented Nov 13, 2024

The implementation of the endpoint for retrieving a list of MI connectors can be found here - https://github.com/wso2-enterprise/digiops-marketing/pull/220
This also supports MI-specific metadata and MI connector operations

@ShammiL
Copy link
Contributor Author

ShammiL commented Nov 21, 2024

The implementation of the daily sync job for updating the connector store database has been updated to read MI metadata and operations - https://github.com/wso2-enterprise/digiops-engineering/pull/164

@ShammiL
Copy link
Contributor Author

ShammiL commented Dec 10, 2024

The GET endpoint and the daily sync for populating the connector store database has been updated with the required changes are available on the staging environments now [1] [2]

[1] https://github.com/wso2-enterprise/digiops-marketing/pull/220
[2] https://github.com/wso2-enterprise/digiops-engineering/pull/164

The metadata files of a selected set of connectors have been updated with the new metadata values and the release workflows have also been added for these connectors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants