Add Custom Backend for APIs #3007

BLasan · 2024-07-15T01:27:23Z

Problem

Some API developers want to use connectors within the api manager to implement mediation logic as they do not want to use an additional MI deployment if their use cases are simple. Within the API Manager, currently, we do not have a construct that will allow us to facilitate this requirement.

Solution

Introduce a way to define a Custom Backend for an API which gets added to the api.xml file of the API.

Affected Component

APIM

Version

No response

Implementation

No response

Related Issues

No response

Suggested Labels

No response

BLasan · 2024-09-03T06:20:52Z

Please find the below attachment related to the issue.
https://docs.google.com/document/d/1DxjI4f-XIwvr414PkT0Mo7C4vuJT_BCGzCidEKwSdF0/edit

BLasan · 2024-09-03T06:35:34Z

[Update 1]

As per the initial discussion, following was the proposed solution.

Introduce a new tab under Endpoints called Custom Sequence to upload a sequence.
Implement a new REST API to update the API with Custom Sequence provided. Following is the REST API format.

      requestBody:
        content:
          multipart/form-data:
            schema:
              properties:
                sequence:
                  type: string
                  description: The sequence that needs to be uploaded.
                  format: binary
                type:
                  type: string
                  description: Type of the Endpoint
                apiData:
                  type: string
                  description: API Data to be updated

Update velocity template of the API to override the existing configurations with Custom Backend provided.
This will update the api.xml provided with the custom backend without storing the provided sequence in sequences directory of the APIM pack. This might affect the reeadabiility of the api.xml.
On update, the endpointConfiguration object will look as follows.

	endpointConfig:
		endpoint_type: custom_sequence
		sequence: FILE_NAME

BLasan · 2024-09-03T06:41:32Z

[Update 2]

As per the second suggestion, changed the velocity template logic to refer the sequence file provided. Following is the approached followed.

Upload the file via the REST API and store under sequences directory which is used to load all sequences by synapse.
Velocity template refers to the Sequence key generated by the code which looks as below.

                #if( $endpoint_config.get("endpoint_type") == 'custom_sequence' )
                    #set( $sequenceKey = $endpoint_config.get("sequence_key") )
                    <sequence key="$sequenceKey"/>

This approach does not contain any tables to store uploaded custom backends. Only the API Artifact contains all necessary configurations of the endpoint.

BLasan · 2024-09-03T07:20:24Z

[Update 3]

Changed the REST API to support add, delete, get operations of the custom sequence as the UI requirement. Followings are the changes introduced.

Changed the REST API to only support Custom Backend upload. Following is the REST API format.

      requestBody:
        content:
          multipart/form-data:
            schema:
              properties:
                sequence:
                  type: string
                  description: The sequence that needs to be uploaded.
                  format: binary
                type:
                  type: string
                  description: Type of the Endpoint

Introduced a new table AM_API_CUSTOM_BACKEND to store sequence related data. Following is the proposed schema for the table.

CREATE TABLE IF NOT EXISTS AM_API_CUSTOM_BACKEND (
  ID INTEGER NOT NULL AUTO_INCREMENT,
  API_UUID VARCHAR(256) NOT NULL,
  REVISION_UUID VARCHAR(256) DEFAULT NULL,
  SEQUENCE MEDIUMBLOB,
  NAME VARCHAR(256) NOT NULL,
  TYPE VARCHAR(120) NOT NULL,
  PRIMARY KEY (ID),
  FOREIGN KEY (API_UUID) REFERENCES AM_API(API_UUID) ON DELETE CASCADE
)ENGINE INNODB;

New REST API supports the following operations related to the Custom Backend.

get
delete
update

When retrieving the API Artifact, we need to retrieve the corresponding sequence from the AM_API_CUSTOM_BACKEND table using API_UUID and REVISION_ID.

BLasan · 2024-09-04T04:33:45Z

[Update 4]

Changed the schema of the new table as follows.

CREATE TABLE IF NOT EXISTS AM_API_CUSTOM_BACKEND (
  ID VARCHAR(60) NOT NULL,
  API_UUID VARCHAR(256) NOT NULL,
  REVISION_UUID VARCHAR(256) DEFAULT NULL,
  SEQUENCE MEDIUMBLOB,
  NAME VARCHAR(256) NOT NULL,
  TYPE VARCHAR(120) NOT NULL,
  PRIMARY KEY (ID),
  FOREIGN KEY (API_UUID) REFERENCES AM_API(API_UUID) ON DELETE CASCADE
)ENGINE INNODB;

Name of the Custom Backend should be in the following format.
<API_UUID>-<ENDPOINT_TYPE>
Issue with the current table schema.

Cannot define a PRIMARY KEY as same policy ID will be replicated in the table on a revision deployment.
Cannot define a composite primary key as REVISION_UUID can be a null

Solutions for the schema issue.

Make REVISION_UUID not null and make REVISION_UUID = "1" for the undeployed version of the API and have a composite primary key (REVISION_UUID, ID, API_UUID)
Introduce two tables. One to store sequence data (this will make a Custom Backend globally available) another one to map Custom Backend to API & Revision.

BLasan · 2024-09-04T21:06:04Z

[Update 5]

Changed the endpointConfig object structure to the following as Custom Backends can be added to both Production and Sandbox endpoints.

endpointConfig: {
   "endpoint_type": "custom_backend",
   "production": {
         "sequence_id": xxxxx,
         "sequence_name": xxxxxx,
         "type": "PRODUCTION"
   },
  "sandbox": {
         "sequence_id": xxxxx,
         "sequence_name": xxxxxx,
         "type": "SANDBOX"
  }
}

2 Implemented get, delete REST APIs which fetches and deletes the Custom Backends from the API_UUID, BACKEND_UUID, TYPE and REVISION_UUID
3. Made default value as "0" for the REVISION_UUID. Also, new composite primary key is (REVISION_UUID, API_UUID, TYPE and ID)

BLasan · 2024-09-05T05:25:32Z

[Update 6]:

Tested Custom Backend upload, Get operations
Working on revision deployment. Fixing minor database issues.

Draft PR - wso2/carbon-apimgt#12524
Updated REST API - https://github.com/wso2/carbon-apimgt/pull/12524/files#diff-d8623bfc39bc03ceb819b7dff3e7cb16254f9eb97238c795f40d3b025a4a3747R491

BLasan · 2024-09-09T01:33:05Z

[Update 7]:

Tested the following flows against the new feature and working without an issue. (Tested for MySQL 8)

API creation
API deletion
Revision Deployment
Revision Restore
Revision Deletion
Changing Endpoint Configurations
API Invocation
API Import via APICTL

New REST API changes can be found via the following links

Tasks completed

REST APIs for Custom Backend
Revision related operations
API Velocity Template support for Custom Backend
API related operations (API Creation, Deletion)
APICTL support for Custom Backend (Import & Export)

Tasks remaining

UI for Custom Backend upload and Endpoint Configuration change
Test APICTL export API
Analytics related changes

BLasan · 2024-09-09T10:21:10Z

All scenarios to be tested

API Creation
Changing API Endpoint
Add Custom Backend, Delete Custom Backend, Change Endpoint Config, Download Sequence
API Revision Deployment, Undeploy revision, Delete revision, Restore revision
API Delete
API Invocation
APICTL Import, Export, Update
APIProduct creation, deploy, invoke, import & export

BLasan · 2024-09-22T09:02:35Z

Tasks Completed -

REST API support to define custom mediation logic by selecting endpoint type as "Sequence Backend".
Upload, Delete, Download sequence backend
Revision deployment, Revision deletion, Revision restore and API invocation with sequence backend
APICTL import/export operations for REST APIs with sequence backend

Future Work -

Improve validations in both UI and backend
Add support for API Product with sequence backend
Check analytics related operations

BLasan added the Type/NewFeature label Jul 15, 2024

github-actions bot added the Missing/Component label Jul 15, 2024

BLasan added Component/APIM 4.x.x and removed Missing/Component labels Jul 15, 2024

BLasan self-assigned this Jul 15, 2024

RakhithaRR added this to the APIM 4.4.0 milestone Jul 22, 2024

RakhithaRR added the 4.4.0 label Aug 1, 2024

BLasan changed the title ~~Support Broker APIS through Gateway~~ Add Custom Backend for APIs Sep 3, 2024

RakhithaRR modified the milestones: APIM 4.4.0, APIM 4.4.0 Alpha Sep 8, 2024

RakhithaRR added the 4.4.0-Alpha label Sep 11, 2024

BLasan mentioned this issue Sep 19, 2024

Add: Support Custom Sequence as a Backend of an API wso2/carbon-apimgt#12524

Merged

This was referenced Sep 23, 2024

[Feat]: Provide Sequence as a Backend for APIs wso2/apim-apps#743

Merged

[Doc]: Sequence as a Backend for REST APIs wso2/docs-apim#8386

Merged

RakhithaRR closed this as completed Sep 27, 2024

This was referenced Oct 11, 2024

Sequence as backend wso2/docs-apim#8470

Merged

Add: Sequence Backend Test wso2/product-apim#13574

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Custom Backend for APIs #3007

Add Custom Backend for APIs #3007

BLasan commented Jul 15, 2024 •

edited

Loading

BLasan commented Sep 3, 2024

BLasan commented Sep 3, 2024

BLasan commented Sep 3, 2024

BLasan commented Sep 3, 2024

BLasan commented Sep 4, 2024 •

edited

Loading

BLasan commented Sep 4, 2024 •

edited

Loading

BLasan commented Sep 5, 2024

BLasan commented Sep 9, 2024 •

edited

Loading

BLasan commented Sep 9, 2024 •

edited

Loading

BLasan commented Sep 22, 2024

Add Custom Backend for APIs #3007

Add Custom Backend for APIs #3007

Comments

BLasan commented Jul 15, 2024 • edited Loading

Problem

Solution

Affected Component

Version

Implementation

Related Issues

Suggested Labels

BLasan commented Sep 3, 2024

BLasan commented Sep 3, 2024

BLasan commented Sep 3, 2024

BLasan commented Sep 3, 2024

BLasan commented Sep 4, 2024 • edited Loading

BLasan commented Sep 4, 2024 • edited Loading

BLasan commented Sep 5, 2024

BLasan commented Sep 9, 2024 • edited Loading

BLasan commented Sep 9, 2024 • edited Loading

BLasan commented Sep 22, 2024

BLasan commented Jul 15, 2024 •

edited

Loading

BLasan commented Sep 4, 2024 •

edited

Loading

BLasan commented Sep 4, 2024 •

edited

Loading

BLasan commented Sep 9, 2024 •

edited

Loading

BLasan commented Sep 9, 2024 •

edited

Loading