Support Deploy/Undeploy Global Level Policies from UI/REST APIs

[YasasRangika]

Problem

In previous versions of API Manager releases, if we want to add a global-level policy to a gateway node, users have to put the manually written xml-type policy file to the location: <APIM_GATEWAY_NODE_HOME>/repository/deployment/server/synapse-configs/default/sequences with the naming pattern WSO2AM--Ext--.

An example Synapse configuration of a global policy is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="WSO2AM--Ext--Out">
   <property name="Category" value="Electronic" scope="transport"/>
</sequence>

But this approach is no longer promoted with the API Manager 4.3.0 release.

Solution

If the logged-in user has sufficient permission, future API Manager products (from APIM-4.3.0 onwards) will provide a separate UI for adding ( /enabling ) and removing ( /disabling ) the global-level policies under the gateway label.

Affected Component

APIM

Version

No response

Implementation

• Complete implementations for the initial flow of the REST API of saving the policy mapping added for a gateway to the database and sending the event to all available gateways.
• Completing the event flow and pulling artifacts via the internal REST API when deploying the policy mapped to a gateway
• Completing the policy aggregation process by creating the final artifact file to be deployed in the gateway when the internal REST API receives the pull request for artifacts.
• Deploy the artifact file in the gateway
• Engage the policy in the API path
• Artifact synchronization in server restart.
• UI designs and implementations

Related Issues

No response

Suggested Labels

No response

[YasasRangika]

Update (upto 06/04/2023)

Analyzed the flow of Operation level policies feature works and had a few meetings with Lakshitha and Isuru. Started this feature implementation from the REST level and added OAS changes to the publisher-API specification. Had an initial coding review with generated classes and DTOs with Lakshitha.
He suggested reducing some implementation at this level and pointed out the required REST APIs. Starting implementations of the persistence layer changes.

[YasasRangika]

Update (07/04/2023)

Completed:

• OAS spec support for global policies.
• Generated DTOs and validated affected DTOs: ProductAPIDTO and GraphQLValidationResponseGraphQLInfoDTO.
• Came up with the database structure for the feature.
• Implemented table creation queries and SQLConstants for basic and required CRUD operations with global policies.

In progress:

• DAO layer changes and the flow establishment to take required cloned policy details for new DAO methods

[YasasRangika]

Update (10/04/2023)

Completed:

• Added changes for REST API to engage new global policy to a gateway.
  Added the required beans.xml changes and tested the invocation
• Mappings to map objects retrieved as parameters
• Design doc changes

In progress:

• Impl layer changes to complete the flow of storing data coming through the post request
• Fixing design doc comments

[YasasRangika]

Update (upto 18/04/2023)

Completed:

• Completed flow to save the details of policy mappings and policy deployments to the database.
• Conducted the design review on the 12th of April 2023, and recorded all discussed facts in the mail [1].
• Changed the code to utilize the architectural decisions taken in the design review meeting.
• Introduced a new event: GatewayPolicyEvent and two new event types: DEPLOY_POLICY_MAPPING_IN_GATEWAY, REMOVE_POLICY_MAPPING_FROM_GATEWAY with new event notifier: GATEWAY_POLICY.
• Debugged the code and validated the information passing through event objects for each event type.

In progress:

Completing the policy aggregation process by creating the final artifact file to be deployed in the gateway.

[1] mail: "Re: [Architecture] [New Feature] Support Deploy/Undeploy Gateway-Specific Global Policies from UI/REST APIs"

[YasasRangika]

Update (20/04/2023)

Completed:

• J2 template file for gateway policies

In progress:

• TemplateBuilderUtil and SynapsePolicyAggregator code changes to retrieve and generate artifacts

[thisaltennakoon]

Update (09/10/2023 - 13/10/2023)

• Had a design review on 12/10/2023 with Sanjeewa Malalgoda, Thilini Shanika, Ashera Silva, Yasas Rangika Mendis and Lasantha Samarakoon (Recording)
• Decided to implement the UI as follows.

Source File of UI Sketches

[piyumaldk]

Update (23/10/2023 - 03/11/2023)

Implemented the basic UIs / Routing / Data handling and updating.

Please note that

• UIs still need improvements / validations / heavy refactoring (as 32+ files from API policy were reused / changed in the attempt)
• Since the backend implementation is not merged yet, implementation was done referring the shared openAPI specification examples. (Used as hardcode promises). Once the backend implementation is merged, need to connect those API calls and properly test / fix issues.
• Some concerns were found in the attempt of the UI implementation and those are needed to be properly discussed.

Please find the implemented UIs.

Landing Page

Create Page

Edit Page

[piyumaldk]

Update (06/11/2023 - 10/11/2023)

• Refactor, improve and clean the implementation.
• Introduced 6 reusable UI specific components (Yellow in the below diagram)
  • These UI components can be reused in API / resource level policies.
• Global specific components will be white in the below diagram.
  • Removed unnecessary components and functionalities.
• Introduced the Global Policy Context to handle the feature context.

Please find the below diagram in order to get a rough idea about the React:TSX component structure of the implementation.

• Improved the UI and implemented the front-end validations.

• Please note that since the backend implementation for this is still not merged, connecting the backend to frontend for dummy promises is yet to be done.

• Some concerns are needed to be clarify and depending on that, further implementation on front-end will be required if backend implementation is changed.

[YasasRangika]

Hi @piyumaldk,

First and foremost, I must commend you on the outstanding work you've done on the UI for the feature. Your design and implementation are truly impressive, and the refactoring you carried out will significantly improve the overall codebase.

I've just returned from medical leave today and haven't had the chance to review the UI changes you have made.

Given that I'm catching up after my absence, I think it would be helpful for us to have a quick call to sync up. I'd love to hear more about your thought process behind certain design decisions and any concerns you might have regarding the UI. Additionally, it would be a good way for me to provide insights into the backend changes I've made.

As for the backend, the majority of it is now complete, and I've conducted the final code review. There were some last-minute changes requested by the leads, and I incorporated those within the given timeframe. However, I also took some time to enhance certain aspects of the feature for better performance and maintainability. As a result, I had to revise the unit tests and integration tests.

Furthermore, due to the nature of the changes, I'll be testing the backend with all the supported databases to ensure compatibility. There were query-level modifications as well, as per the feedback received during the final code review. I appreciate your understanding, and I'm confident these refinements will contribute to the overall robustness of the feature.

Since I've allocated for patches this week, starting from next week onwards, I will also allocate for RnD, allowing me to give my full focus to this project. Until then, if the backend is a blocker for you to complete the feature, I can provide a working backend within a short timeframe.

Looking forward to our call and to further discussing about this feature.

Thanks and Regards,
Yasas

[piyumaldk]

Update (13/11/2023 - 17/11/2023)

• Discussed the concerns.
• Removed the Gateway Selector from the feature as I was informed that we are not supporting Choreo Connect in the feature.
• Did necessary changes (ex: Removing API context from all sub components) and improved the JSDocs.

Please find the new react component structure here.

• Please note that feature implementation was completed for the current implemented backend hence scheduled a code review next Monday @3.00PM @WSO2-LK-9th Floor-Grove considering the availability of the participants.

[piyumaldk]

Update (20/11/2023 - 21/11/2023)

• Had the code + UI/UX review on 20/11/2023 from 3.00PM to 4.30PM @WSO2-LK-9th Floor-Grove.
• There was a collective decision to change the design of the deploying and un-deploying UX/UI.
  • This decision was taken to improve the user experience.
• As I was requested to come up with a new design, on 21/11/2023, I was able to finalised the UI design.
• Please find the final version of the UI design.
  • The design changes have been reviewed and approved by the stakeholders who requested them during the code review process.

[piyumaldk]

[FRONT-END] Pending and Requested Tasks after the Code/UX Review

• Design a new UI/UX for deploying and un-deploying.

• To shorten the name content in Gateway chips, use CSS (text-overflow) rather than a TSX function.

  • This is no longer needed as we changed the design.

• Internationalise the component using React Intl.

• Remove response.body.message from Error Messages.

• Differentiate column header names from data of the table.

  • This was decided later (after discussion) as "Not to do" to keep the same style in all publisher tables.

• Scope for Global Policies.

  • New scope > Read-only.
  • Admin > Has the privilege to do deployments / un-deployments.
  • This also needs to be implemented in the back-end in order to support in front-end.

• Change the Gateway identifier from name to ID.

  • This was an early concern we discussed.
  • Earlier this was temporary implemented using the name as we only got name from the backend APIs.
  • Once the backend is changed to send / receive / handle gateways through IDs, frontend changes can be implemented.
  • Discussed some concerns and decided to use label as it's unique and encoded.

• Implement new design (For expand area (deploy) and un-deploy.

• Test the UI/UX.

• Write UI tests in cypress.

• Documentation.

[YasasRangika]

[Backend] Status Update as of 01/12/2023: Feature Enhancements and Refinements

Here is the progress on the feature implementation, incorporating various suggestions and feedback received during and after the code review:

• Completed the implementation of UUID generation for policy mapping. Also, altered the approach by separating the addition and update paths for the new policy mapping, aligning with the discussed code review feedback.

• Revised the update flow to perform delete and insert actions solely for the policy mapping table while conducting updates only on the policy metadata table, in accordance with review comments.

• Refactored HTTP response handling by changing the status from OK to 201 CREATED for new policy mapping creation. Moreover, the get-all policy mapping API was optimized to reduce response bulkiness by providing only metadata.

• Modified the handling of deletion for active deployments in the policy mapping, shifting from a 500 Internal Server Error to more appropriate 401 and 403 error codes for specific cases as suggested in the code review.

• Implemented query support for filtering deployments based on gateway labels, meeting the specific requirements identified in the review.

• Addressed comments from review meetings by incorporating persistence layer changes, enhancing error logs, adjusting user roles, and removing unnecessary validation parts.

• Added changes for gateway label identification based on the UI implementation, introducing a new property in the response object.

  • I made a change to introduce a new property in the response for using the Environment schema, which is already in use in the settings API. However, we noticed that the response object was carrying too much redundant data, so I introduced a new schema containing only the UUID of the gateway, name, and display name. I have added and refactored the code with these changes, and I need to discuss whether we can use the UUID instead of the gateway name in other use cases as well. i.e. completely removing the gateway label from the database table and just identifying it via the gateway label ID. It's worth noting that in API deployments, we only address gateway labels by their name and vhost.

To-Do:

• Consider using the gateway label ID instead of the name during deployment. Further discussion is needed on this topic.

• Incorporate necessary changes to user roles, introducing a new scope to fulfill a recent requirement.

• Revisit and reintroduce the policy artifact name change in Synapse, previously removed due to testing inconsistencies. Validate to ensure the issues no longer persist.

• Ensure all test changes cover all flows.

• Perform a round of DB tests with the recent backend changes, considering the updated functionalities and refinements introduced. This testing phase includes verification across all supported databases to ensure the functionality of the system with the recent modifications.

[YasasRangika]

[Backend] Status Update as of 08/12/2023

• Reverted the recent changes made to associate gateway label IDs with their respective names within gateway-related processes.
• Following offline discussions with Sanjeewa, Nandika, Tharindu, Piyumal, and myself, we implemented essential code modifications to enforce the validation of gateway labels before proceeding with any operations.
• Introduced new DAO and other methods for gateway label validations, ensuring their implementation across all relevant processes.
• Identified potential instances allowing modifications or deletions of existing gateway labels. Decided against implementing foreign key constraints to maintain data consistency due to conflicts with the policy feature.
• Implemented new methods aimed at validating gateway deployments in policy mappings. These methods have been incorporated into Environment-related operations to prevent scenarios such as the deletion of a gateway label when an active policy mapping exists and updating the gateway labels as well.
• Updated user roles to accommodate a recent requirement by introducing a new scope.
• Reintroduced the policy artifact name change within Synapse, which was previously removed due to inconsistencies during testing. After validating and addressing the issues, testing of each deploy, undeploy, and execution sequence order was performed.
• Completed validations across h2, mysql, and mssql databases.

To-Do:

• Finalize testing for databases: Oracle, DB2, Postgresql.
• Address test suite alterations and resolve test failures encountered during the latest test runs.
• Continuously engaging in feature testing while systematically addressing and adding possible enhancements.