Skip to content

Latest commit

 

History

History
128 lines (99 loc) · 4.78 KB

agora-api.mdx

File metadata and controls

128 lines (99 loc) · 4.78 KB

Optimism Agora API

This repository contains the OpenAPI specification and API pipeline manifest needed to create a RAG pipeline. This pipeline generates a knowledge base from RetroPGF projects and proposals within the OP collective.

Pre-requisites

To access this API, you'll need an API key. You can request one through the Agora's Discord server. You can run the rag-api-pipeline setup command to set the REST API Key, or your can directly store the key in the config/secrets/api-key file. A less secure option is to provide it using the --api-key CLI argument.

Defining the RAG API Pipeline Manifest

This pipeline will extract data related to DAO proposals (/proposals) and RetroPGF projects (/projects). Next, you can find an overview of the main sections in the API pipeline manifest.

Basic Configuration

Since no api_parameters are required, this section remains empty.

api_name: "optimism_agora_api"

api_parameters:

api_config:
  request_method: "get"
  content_type: "application/json"
  response_entrypoint_field: "data"

Connector Specification

The manifest then defines some metadata and the request parameters needed for making calls to the API. In this case, it only needs an api_key parameter for authentication:

spec:
  connection_specification:
    $schema: http://json-schema.org/draft-07/schema#
    additionalProperties: true
    properties:
      api_key:
        airbyte-secret: true
        description: Agora API Key.
        type: string
    required:
    - api_key
    title: Agora API Spec
    type: object
  documentation_url: https://docs.airbyte.com/integrations/sources/agora
  type: Spec

API Request Configuration

Below is the requester_base definition. The API implements a BearerAuthenticator schema and retrieves the api_token from the config object:

definition:
  requester_base:
    type: HttpRequester
    url_base: "https://vote.optimism.io/api/v1"
    http_method: "GET"
    authenticator: # Details at https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/authentication
      type: BearerAuthenticator
      api_token: "{{ config['api_key'] }}"

Record Selection and Pagination

The API uses an Offset-based pagination strategy. The page_size is set to 50, while offset and limit parameters are dynamically inserted into the URL as request parameters:

definition:
  paginator: # Details at https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination
    type: DefaultPaginator
    pagination_strategy: # Details at https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination#offset-increment
      type: "OffsetIncrement"
      page_size: 50
    page_token_option:
      type: "RequestOption"
      field_name: "offset"
      inject_into: "request_parameter"
    page_size_option:
      type: "RequestOption"
      inject_into: "request_parameter"
      field_name: "limit"

Endpoint Configuration

Below are the target endpoints with their respective schemas:

endpoints:
  /proposals:
    id: "proposals"
    primary_key: "id"
    responseSchema: "#/schemas/Proposal"
    textSchema:
      $ref: "#/textSchemas/Proposal"
  /projects:
    id: "projects"
    primary_key: "id"
    responseSchema: "#/schemas/Project"
    textSchema:
      $ref: "#/textSchemas/Project"

Using the RAG Pipeline to generate a Knowledge Base for the OP Collective

RAG Pipeline CLI

  1. Make sure to setup the pipeline initial settings by running the rag-api-pipeline setup command.
  2. Execute the following command:
rag-api-pipeline run all config/agora_api_pipeline.yaml config/agora_openapi.yaml

After execution, you'll find the processed data and compressed knowledge base snapshot in the output/optimism_agora_api folder.

Import the KB Snapshot into a Gaia Node

  1. Locate the generated snapshot in output/optimism_agora_api/ (named optimism_agora_api_collection-xxxxxxxxxxxxxxxx-yyyy-mm-dd-hh-mm-ss.snapshot.tar.gz) or download it from the HuggingFace link above.
  2. Follow the official knowledge base selection guide
  3. Configure your node using the recommended settings from the node deployment guide. Do not forget to update the custom prompts to mention optimism as the target DAO protocol.