[New Feature]: Providing support for observability, service catalog, runtime monitoring in Ballerina.toml

[gayaldassanayake]

Description

Observability

Need a way to import observability modules to a package. Although this can be done by asking the user to add a null import, this will introduce a build time feature to the source. Furthermore, a way to request the runtime to collect observability meta data is required. The observabilityIncluded compiler option currently does this.

service catalog

The requirement is to publish services in a package into the APIM service catalog/ ICP. Here also, the required dependency (eg:- wso2.apim.catalog) should be included as a dependency. The current design requires the users to add a null import and add an entry to the Ballerina.toml

import ballerinax/wso2.apim.catalog as _;

# Ballerina.toml
[[dependency]]
org = "ballerinax"
name = "wso2.apim.catalog"
version = "0.1.0"
repository="local"

[build-options]
remoteManagement=true

Describe your problem(s)

There is no extensible way for the developers to specify such imports/ metadata required for these features without introducing new fields to the Ballerina.toml (and PackageManifest) by changing the ballerina-lang.

Describe your solution(s)

Suggested designs

1. The below entry is added to the Ballerina.toml. This will add an import to be resolved during the dependency resolution. Adding this entry should also automatically toggle the observabilityIncluded flag so that the runtime can collect observability meta data.

[[observability.metrics]]
orgName = "ballerinax"  # Mandatory
moduleName = "prometheus"   # Mandatory
version = "0.2.0"       # Optional

Suggested by @NipunaMadhushan.

2. The user should add an import in the Ballerina.toml. This will be added to the dependency list of the user project.

End user ballerina.toml

[[features.dependency]]
orgName = "ballerinax"
moduleName = "newrelic"
version = "0.2.0"

In the Ballerina.toml of the newrelic module, the observability package developer must specify that the runtime of the user project should start collecting information for observability.

Newrelic package Ballerina.toml

[[runtime]]
observability = true
# stats = false

4. Same as above, but instead of specifying the import in the Ballerina.toml features.dependency array, the user should add a null import.
   End user's main.bal

import ballerinax/newrelic as _;

In the Ballerina.toml of the newrelic

[[runtime]]
observability = true
stats = false

2, 3 Suggested by @manuranga

Design documents

1. Observability - https://docs.google.com/document/d/17RocnId3UugzUGSPYscr2XcEZ8KZjs1EOFOvqnU_T_U/edit?usp=sharing
2. Service catalog - https://docs.google.com/document/d/1oLQLjXm2Uo1YJtqUO3mVnt_tPcRbrFJKloW7Jg3JrcU/edit?usp=sharing

Related area

-> Compilation

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[gayaldassanayake]

Relarted observability issue #41924