feat: improve URL handling

README.md

@@ -39,15 +39,15 @@ You can also completely remove `ethereum_package` from your configuration in whi
 
 Kurtosis packages are parameterizable, meaning you can customize your network and its behavior to suit your needs by storing parameters in a file that you can pass in at runtime like so:
 
-```bash
+```shell
 kurtosis run github.com/ethpandaops/optimism-package --args-file https://raw.githubusercontent.com/ethpandaops/optimism-package/main/network_params.yaml
 ```
 
 For `--args-file` parameters file, you can pass a local file path or a URL to a file.
 
 To clean up running enclaves and data, you can run:
 
-```bash
+```shell
 kurtosis clean -a
 ```
 
@@ -57,7 +57,7 @@ This will stop and remove all running enclaves and **delete all data**.
 
 If you are attempting to test any changes to the package code, you can point to the directory as the `run` argument
 
-```bash
+```shell
 cd ~/go/src/github.com/ethpandaops/optimism-package
 kurtosis run . --args-file ./network_params.yaml
 ```
@@ -77,8 +77,10 @@ The full YAML schema that can be passed in is as follows with the defaults provi
 optimism_package:
   # Observability configuration
   observability:
-    # Whether or not to configure observability (e.g. prometheus)
+    # Whether to provision an observability stack (prometheus, loki, promtail, grafana)
     enabled: true
+    # Whether to enable features exclusive to the K8s backend (ie log collection)
+    enable_k8s_features: false
     # Default prometheus configuration
     prometheus_params:
       storage_tsdb_retention_time: "1d"
@@ -91,15 +93,34 @@ optimism_package:
       min_mem: 128
       max_mem: 2048
       # Prometheus docker image to use
-      # Defaults to the latest image
-      image: "prom/prometheus:latest"
+      image: "prom/prometheus:v3.1.0"
+    # Default loki configuration
+    loki_params:
+      # Loki docker image to use
+      image: "grafana/loki:3.3.2"
+      # Resource management for loki container
+      # CPU is milicores
+      # RAM is in MB
+      min_cpu: 10
+      max_cpu: 1000
+      min_mem: 128
+      max_mem: 2048
+    # Default promtail configuration
+    promtail_params:
+      # Promtail docker image to use
+      image: "grafana/promtail:3.3.2"
+      # Resource management for promtail container
+      # CPU is milicores
+      # RAM is in MB
+      min_cpu: 10
+      max_cpu: 1000
+      min_mem: 128
+      max_mem: 2048
     # Default grafana configuration
     grafana_params:
-      # A list of locators for grafana dashboards to be loaded by the grafana service.
-      # Each locator should be a URL to a directory containing a /folders and a /dashboards directory.
-      # Those will be uploaded to the grafana service by using grizzly.
-      # See https://github.com/ethereum-optimism/grafana-dashboards-public for more info.
+      # A list of locators for grafana dashboards to be loaded be the grafana service
       dashboard_sources:
+        # Default public Optimism dashboards
         - github.com/ethereum-optimism/grafana-dashboards-public/resources
       # Resource management for grafana container
       # CPU is milicores
@@ -109,8 +130,7 @@ optimism_package:
       min_mem: 128
       max_mem: 2048
       # Grafana docker image to use
-      # Defaults to the latest image
-      image: "grafana/grafana:latest"
+      image: "grafana/grafana:11.5.0"
   # Interop configuration
   interop:
     # Whether or not to enable interop mode
@@ -623,7 +643,7 @@ optimism_package:
 
 Compile [tx-fuzz](https://github.com/MariusVanDerWijden/tx-fuzz) locally per instructions in the repo. Run tx-fuzz against the l2 EL client's RPC URL and using the pre-funded wallet:
 
-```bash
+```shell
 ./livefuzzer spam  --sk "0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80" --rpc http://127.0.0.1:<port> --slot-time 2
 ```
 
@@ -666,39 +686,85 @@ Please find examples of additional configurations in the [test folder](.github/t
 
 - List information about running containers and open ports
 
-```bash
+```shell
 kurtosis enclave ls
 kurtosis enclave inspect <enclave-name>
 ```
 
 - Inspect chain state.
 
-```bash
+```shell
 kurtosis files inspect <enclave-name> op-deployer-configs
 ```
 
 - Dump all files generated by kurtosis to disk (for inspecting chain state/deploy configs/contract addresses etc.). A file that contains an exhaustive
   set of information about the current deployment is `files/op-deployer-configs/state.json`. Deployed contract address, roles etc can all be found here.
 
-```bash
+```shell
 # dumps all files to a enclave-name prefixed directory under the current directory
 kurtosis enclave dump <enclave-name>
 kurtosis files download <enclave-name> op-deployer-configs <where-to-download>
 ```
 
 - Get logs for running services
 
-```bash
+```shell
 kurtosis service logs <enclave-name> <service-name> -f . # -f tails the log
 ```
 
 - Stop/Start running service (restart sequencer/batcher/op-geth etc.)
 
-```bash
+```shell
 kurtosis service stop <enclave-name> <service-name>
 kurtosis service start <enclave-name> <service-name>
 ```
 
+## Observability
+
+This package optionally provisions an in-enclave observability stack consiisting of Grafana, prometheus, promtail, and loki, which collects logs and metrics from the enclave.
+
+This feature is enabled by default, but can be disabled like so:
+
+```yaml
+optimism_package:
+  observability:
+    enabled: false
+```
+
+You can provide custom dashboard sources to have Grafana pre-populated with your preferred dashboards. Each source should be a URL to a Github repository directory containing at minimum a `dashboards` directory:
+
+```yaml
+optimism_package:
+  observability:
+    grafana_params:
+      dashboard_sources:
+        - github.com/<org>/<repo>/<path-to-resources>
+```
+
+See [grafana-dashboards-public](https://github.com/ethereum-optimism/grafana-dashboards-public) for more info.
+
+To access the Grafana UI, you can use the following command after starting the enclave:
+
+```shell
+just open-grafana <enclave name>
+```
+
+### Logs
+
+Note that due to `kurtosis` limitations, log collection is not enabled by default, and is only supported for the Kubernetes backend. To enable log collection, you must set the following parameter:
+
+```yaml
+optimism_package:
+  observability:
+    enable_k8s_features: true
+```
+
+Note that since `kurtosis` runs pods using the namespace's default `ServiceAccount`, which is not typically able to modify cluster-level resources, such as `ClusterRoles`, as the `promtail` Helm chart requires, you must also install the `ns-authz` Helm chart to the Kubernetes cluster serving as the `kurtosis` backend using the following command:
+
+```shell
+just install-ns-authz
+```
+
 ## Development
 
 ### Development environment
@@ -723,7 +789,7 @@ mise install
 
 If you have made changes and would like to submit a PR, test locally and make sure to run `lint` on your changes
 
-```bash
+```shell
 kurtosis lint --format .
 ```
 
@@ -733,7 +799,7 @@ kurtosis lint --format .
 
 We are using [`kurtosis-test`](https://github.com/ethereum-optimism/kurtosis-test) to run a set of unit tests against the starlark code:
 
-```bash
+```shell
 # To run all unit tests
 kurtosis-test .
 ```

justfile

@@ -0,0 +1,16 @@
+install-ns-authz:
+    helm install ns-authz util/ns-authz --namespace kube-system
+
+uninstall-ns-authz:
+    helm uninstall ns-authz --namespace kube-system
+
+get-service-url enclaveName serviceName portId:
+    kurtosis service inspect {{enclaveName}} {{serviceName}} | tail -n +2 | yq e - -o=json |\
+        jq -r --arg portId {{portId}} '.Ports[$portId]' | sed 's/.*-> //'
+
+open-service enclaveName serviceName:
+    open "$(just get-service-url {{enclaveName}} {{serviceName}} http)"
+
+open-grafana enclaveName:
+    just open-service {{enclaveName}} grafana
+

main.star

@@ -9,8 +9,6 @@ op_challenger_launcher = import_module(
 )
 
 observability = import_module("./src/observability/observability.star")
-prometheus = import_module("./src/observability/prometheus/prometheus_launcher.star")
-grafana = import_module("./src/observability/grafana/grafana_launcher.star")
 
 wait_for_sync = import_module("./src/wait/wait_for_sync.star")
 input_parser = import_module("./src/package_io/input_parser.star")
@@ -163,21 +161,9 @@ def run(plan, args):
                 observability_helper,
             )
 
-    if observability_helper.enabled and len(observability_helper.metrics_jobs) > 0:
-        plan.print("Launching prometheus...")
-        prometheus_private_url = prometheus.launch_prometheus(
-            plan,
-            observability_helper,
-            global_node_selectors,
-        )
-
-        plan.print("Launching grafana...")
-        grafana.launch_grafana(
-            plan,
-            prometheus_private_url,
-            global_node_selectors,
-            observability_params.grafana_params,
-        )
+    observability.launch(
+        plan, observability_helper, global_node_selectors, observability_params
+    )
 
 
 def get_l1_config(all_l1_participants, l1_network_params, l1_network_id):

mise.toml

@@ -3,3 +3,4 @@
 # Core dependencies
 "ubi:ethereum-optimism/kurtosis-test" = "0.0.1"
 "ubi:kurtosis-tech/kurtosis-cli-release-artifacts[exe=kurtosis]" = "1.4.4"
+"yq" = "v4.44.3"

src/batcher/op-batcher/op_batcher_launcher.star

@@ -6,6 +6,9 @@ ethereum_package_constants = import_module(
     "github.com/ethpandaops/ethereum-package/src/package_io/constants.star"
 )
 
+constants = import_module("../../package_io/constants.star")
+util = import_module("../../util.star")
+
 observability = import_module("../../observability/observability.star")
 prometheus = import_module("../../observability/prometheus/prometheus_launcher.star")
 
@@ -14,16 +17,13 @@ prometheus = import_module("../../observability/prometheus/prometheus_launcher.s
 # The Docker container runs as the "op-batcher" user so we can't write to root
 BATCHER_DATA_DIRPATH_ON_SERVICE_CONTAINER = "/data/op-batcher/op-batcher-data"
 
-# Port IDs
-BATCHER_HTTP_PORT_ID = "http"
-
 # Port nums
 BATCHER_HTTP_PORT_NUM = 8548
 
 
 def get_used_ports():
     used_ports = {
-        BATCHER_HTTP_PORT_ID: ethereum_package_shared_utils.new_port_spec(
+        constants.HTTP_PORT_ID: ethereum_package_shared_utils.new_port_spec(
             BATCHER_HTTP_PORT_NUM,
             ethereum_package_shared_utils.TCP_PROTOCOL,
             ethereum_package_shared_utils.HTTP_APPLICATION_PROTOCOL,
@@ -47,8 +47,6 @@ def launch(
     observability_helper,
     da_server_context,
 ):
-    batcher_service_name = "{0}".format(service_name)
-
     config = get_batcher_config(
         plan,
         image,
@@ -62,16 +60,12 @@ def launch(
         da_server_context,
     )
 
-    batcher_service = plan.add_service(service_name, config)
-
-    batcher_http_port = batcher_service.ports[BATCHER_HTTP_PORT_ID]
-    batcher_http_url = "http://{0}:{1}".format(
-        batcher_service.ip_address, batcher_http_port.number
-    )
+    service = plan.add_service(service_name, config)
+    service_url = util.make_service_http_url(service)
 
-    observability.register_op_service_metrics_job(observability_helper, batcher_service)
+    observability.register_op_service_metrics_job(observability_helper, service)
 
-    return "op_batcher"
+    return service_url
 
 
 def get_batcher_config(

src/cl/hildr/hildr_launcher.star

@@ -26,7 +26,6 @@ BEACON_DATA_DIRPATH_ON_SERVICE_CONTAINER = "/data/hildr/hildr-beacon-data"
 # Port IDs
 BEACON_TCP_DISCOVERY_PORT_ID = "tcp-discovery"
 BEACON_UDP_DISCOVERY_PORT_ID = "udp-discovery"
-BEACON_HTTP_PORT_ID = "http"
 
 # Port nums
 BEACON_DISCOVERY_PORT_NUM = 9003
@@ -43,7 +42,7 @@ def get_used_ports(discovery_port):
         BEACON_UDP_DISCOVERY_PORT_ID: ethereum_package_shared_utils.new_port_spec(
             discovery_port, ethereum_package_shared_utils.UDP_PROTOCOL, wait=None
         ),
-        BEACON_HTTP_PORT_ID: ethereum_package_shared_utils.new_port_spec(
+        constants.HTTP_PORT_ID: ethereum_package_shared_utils.new_port_spec(
             BEACON_HTTP_PORT_NUM,
             ethereum_package_shared_utils.TCP_PROTOCOL,
             ethereum_package_shared_utils.HTTP_APPLICATION_PROTOCOL,
@@ -84,7 +83,7 @@ def launch(
     #     endpoint="/",
     #     content_type="application/json",
     #     body='{"jsonrpc":"2.0","method":"opp2p_self","params":[],"id":1}',
-    #     port_id=BEACON_HTTP_PORT_ID,
+    #     port_id=constants.HTTP_PORT_ID,
     #     extract={
     #         "enr": ".result.ENR",
     #         "multiaddr": ".result.addresses[0]",
@@ -113,15 +112,11 @@ def launch(
         da_server_context,
     )
 
-    beacon_service = plan.add_service(service_name, config)
-
-    beacon_http_port = beacon_service.ports[BEACON_HTTP_PORT_ID]
-    beacon_http_url = "http://{0}:{1}".format(
-        beacon_service.ip_address, beacon_http_port.number
-    )
+    service = plan.add_service(service_name, config)
+    service_url = util.make_service_http_url(service)
 
     metrics_info = observability.new_metrics_info(
-        observability_helper, beacon_service, METRICS_PATH
+        observability_helper, service, METRICS_PATH
     )
 
     # response = plan.request(
@@ -135,9 +130,9 @@ def launch(
     return ethereum_package_cl_context.new_cl_context(
         client_name="hildr",
         enr="",  # beacon_node_enr,
-        ip_addr=beacon_service.ip_address,
-        http_port=beacon_http_port.number,
-        beacon_http_url=beacon_http_url,
+        ip_addr=service.ip_address,
+        http_port=util.get_service_http_port_num(service),
+        beacon_http_url=service_url,
         cl_nodes_metrics_info=[metrics_info],
         beacon_service_name=service_name,
     )
@@ -159,14 +154,8 @@ def get_beacon_config(
     observability_helper,
     da_server_context,
 ):
-    EXECUTION_ENGINE_ENDPOINT = "http://{0}:{1}".format(
-        el_context.ip_addr,
-        el_context.engine_rpc_port_num,
-    )
-    EXECUTION_RPC_ENDPOINT = "http://{0}:{1}".format(
-        el_context.ip_addr,
-        el_context.rpc_port_num,
-    )
+    EXECUTION_ENGINE_ENDPOINT = util.make_execution_engine_url(el_context)
+    EXECUTION_RPC_ENDPOINT = util.make_execution_rpc_url(el_context)
 
     ports = dict(get_used_ports(BEACON_DISCOVERY_PORT_NUM))