Skip to content

Fix Loki toolset docs for Grafana Cloud and trailing slash bug #1590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aantn wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Fix Loki toolset docs for Grafana Cloud and trailing slash bug #1590

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
33ae3d8
Fix Loki toolset docs for Grafana Cloud and trailing slash bug
claude Feb 19, 2026
7ce3762
Remove trailing slash warning from Loki docs since code handles it
claude Feb 19, 2026
7ddeeab
Restructure Loki docs: merge confusing 3 options into 2 clear ones
claude Feb 19, 2026
17fa0ee
Simplify Grafana Cloud Loki direct config to just api_url + api_key
claude Feb 19, 2026
96716a7
Fix token instructions order and remove Capabilities section
claude Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 84 additions & 40 deletions docs/data-sources/builtin-toolsets/grafanaloki.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,57 +17,107 @@ Connect HolmesGPT to Loki for log analysis through Grafana or direct API access.

## Configuration

Choose one of the following methods:

### Option 1: Through Grafana (Recommended)

HolmesGPT queries Loki through Grafana's datasource proxy. This works with any Grafana instance — self-hosted or Grafana Cloud.

**Required:**

- [Grafana service account token](https://grafana.com/docs/grafana/latest/administration/service-accounts/) with Viewer role
- Loki datasource UID from Grafana

**Find your Loki datasource UID:**
```bash
# Port forward to Grafana
kubectl port-forward svc/grafana 3000:80

# Get Loki datasource UID
curl -s -u admin:admin http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki") | .uid'
```
=== "Self-Hosted Grafana"

### Configuration (Grafana Proxy)
```bash
# Port forward to Grafana
kubectl port-forward svc/grafana 3000:80

```yaml-toolset-config
toolsets:
grafana/loki:
enabled: true
config:
api_key: <your grafana API key>
api_url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
grafana_datasource_uid: <the UID of the loki data source in Grafana>
# Get Loki datasource UID
curl -s -u admin:admin http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki") | .uid'
```

kubernetes/logs:
enabled: false # HolmesGPT's default logging mechanism MUST be disabled
```
=== "Grafana Cloud"

## Direct Connection
Create a [service account token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/service-accounts/): navigate to "Administration → Service accounts", create a new service account with "Viewer" role, and generate a token (starts with `glsa_`).

The toolset can directly connect to a Loki instance without proxying through a Grafana instance. This is done by not setting the `grafana_datasource_uid` field. Not setting this field makes HolmesGPT assume that it is directly connecting to Loki.
```bash
curl -H "Authorization: Bearer YOUR_GLSA_TOKEN" \
"https://YOUR-INSTANCE.grafana.net/api/datasources" | \
jq '.[] | select(.type=="loki") | {name, uid}'
```

### Configuration (Direct Connection)
**Configure HolmesGPT:**

```yaml-toolset-config
toolsets:
grafana/loki:
enabled: true
config:
api_url: http://loki.logging
additional_headers:
X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled
=== "Self-Hosted Grafana"

kubernetes/logs:
enabled: false # HolmesGPT's default logging mechanism MUST be disabled
```
```yaml-toolset-config
toolsets:
grafana/loki:
enabled: true
config:
api_key: <your grafana service account token>
api_url: http://grafana.monitoring.svc:3000
grafana_datasource_uid: <the UID of the loki data source in Grafana>

kubernetes/logs:
enabled: false # HolmesGPT's default logging mechanism MUST be disabled
```

=== "Grafana Cloud"

```yaml-toolset-config
toolsets:
grafana/loki:
enabled: true
config:
api_key: <your glsa_ service account token>
api_url: https://YOUR-INSTANCE.grafana.net
grafana_datasource_uid: <the UID of the loki data source in Grafana>

kubernetes/logs:
enabled: false # HolmesGPT's default logging mechanism MUST be disabled
```

!!! warning "Getting 404 errors?"
- **Use your Grafana instance URL** (`https://YOUR-INSTANCE.grafana.net`), not the Loki endpoint URL (`https://logs-prod-xxx.grafana.net`)
- **Verify the datasource UID** using the `curl` command above — a wrong UID is the most common cause of 404 errors

### Option 2: Direct to Loki

Connect directly to the Loki API without going through Grafana. Use this if you don't have Grafana or prefer not to proxy through it.

=== "Self-Hosted Loki"

```yaml-toolset-config
toolsets:
grafana/loki:
enabled: true
config:
api_url: http://loki.logging
additional_headers:
X-Scope-OrgID: "<tenant id>" # Set if Loki multitenancy is enabled

kubernetes/logs:
enabled: false # HolmesGPT's default logging mechanism MUST be disabled
```

=== "Grafana Cloud Loki"

Find your Loki URL in Grafana Cloud under "My Account → Loki":

```yaml-toolset-config
toolsets:
grafana/loki:
enabled: true
config:
api_url: https://logs-prod-XXX.grafana.net
api_key: <your Grafana Cloud API key>

kubernetes/logs:
enabled: false # HolmesGPT's default logging mechanism MUST be disabled
```
Comment on lines +106 to +120

@coderabbitai coderabbitai Bot Feb 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

find . -name "common.py" -path "*/grafana/*" | head -20

Repository: HolmesGPT/holmesgpt

Length of output: 106

🏁 Script executed:

rg -n "api_key|Bearer|basic_auth|Authorization" --type py -A3 -B3 holmes/plugins/toolsets/grafana/common.py

Repository: HolmesGPT/holmesgpt

Length of output: 1044

🏁 Script executed:

find . -name "*.py" -path "*/loki*" -o -name "*loki*" -type f | head -20

Repository: HolmesGPT/holmesgpt

Length of output: 392

🏁 Script executed:

rg -i "grafana.*loki|loki.*cloud|basic.*auth|instance.*id" --type py -A2 -B2 | head -100

Repository: HolmesGPT/holmesgpt

Length of output: 6629

🏁 Script executed:

cat -n holmes/plugins/toolsets/grafana/loki_api.py

Repository: HolmesGPT/holmesgpt

Length of output: 3362

🌐 Web query:

Grafana Cloud Loki authentication API key bearer token

💡 Result:

For Grafana Cloud Loki (logs), the “API key” you create is typically a Grafana Cloud Access Policy token. How you pass it depends on which endpoint you’re calling:

1) Calling Loki (read/write logs): use tenant/user + token

Grafana Cloud Loki expects your Logs tenant ID (a.k.a. logs user / tenant) plus the token.

Option A — HTTP Basic Auth (most common)

  • Username: your logs tenant ID
  • Password: your Access Policy token (Grafana Cloud token) (grafana.com)

Option B — Authorization: Bearer <user>:<token> (common for push examples)

Grafana’s own curl example for pushing logs uses:
Authorization: Bearer <LOGS_USERID>:<API_KEY> (grafana.com)

Example (push):

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <LOGS_TENANT_ID>:<ACCESS_POLICY_TOKEN>" \
  -d '{"streams":[{"stream":{"source":"curl"},"values":[["'"$(($(date +%s)*1000000000))"'","hello"]]}]}' \
  https://<your-loki-endpoint>/loki/api/v1/push

(grafana.com)

2) Calling the Grafana Cloud API (not Loki): use Bearer <token>

For Grafana Cloud API requests, you pass the Access Policy token as:
Authorization: Bearer <CLOUD ACCESS POLICY TOKEN> (grafana.com)

If you tell me whether you’re pushing logs (/loki/api/v1/push) or querying (/loki/api/v1/query / query_range), and paste your Loki endpoint host pattern (redact IDs), I can give you the exact curl for your case.

Citations:

🌐 Web query:

Grafana Cloud Loki query_range authentication tenant ID logs user

💡 Result:

To call Loki’s GET /loki/api/v1/query_range against Grafana Cloud Logs (Loki) you typically need (a) the logs tenant ID / logs userID and (b) a token (Cloud Access Policy token) with logs:read.

1) Endpoint

Use your stack’s Loki “Logs endpoint” (shown in the Grafana Cloud Portal). The range-query path is: .../loki/api/v1/query_range. [1]

2) Authentication (Grafana Cloud)

Grafana Cloud’s recommended setup is HTTP Basic Auth:

  • Username = log tenant ID (often shown as “Logs user” / “UserID” in the Cloud Portal)
  • Password = Grafana Cloud token (Cloud Access Policy token) [2]

3) Tenant / Org header (X-Scope-OrgID)

Loki is multi-tenant and, in general, identifies the tenant with the X-Scope-OrgID header. [3]
In Grafana Cloud, the fronting gateway commonly derives tenant from your credentials (so you often don’t need to manually add X-Scope-OrgID if you’re using the Cloud endpoint + basic auth), but if you’re hitting Loki behind your own proxy or a custom deployment, you may need to set it explicitly. [3]

4) Example curl (range query)

curl -u "$LOKI_TENANT_ID:$GRAFANA_CLOUD_TOKEN" \
  "https://<your-logs-endpoint>/loki/api/v1/query_range?query=%7Bjob%3D%22api%22%7D&start=1700000000000000000&end=1700003600000000000&limit=1000"

Common failure modes

  • Using a token without logs:read scope. [2]
  • Using the wrong “user/tenant ID” (must be the logs tenant ID, not your email/login). [2]
  • Missing/incorrect X-Scope-OrgID when talking to Loki without a gateway that injects it. [3]

Sources: [1] [2] [3]

🏁 Script executed:

cat -n holmes/plugins/toolsets/grafana/common.py | head -60

Repository: HolmesGPT/holmesgpt

Length of output: 2498

🏁 Script executed:

cat -n holmes/plugins/toolsets/grafana/loki/toolset_grafana_loki.py | head -100

Repository: HolmesGPT/holmesgpt

Length of output: 4074

The Grafana Cloud Loki configuration example is incomplete for direct-access authentication.

Grafana Cloud Loki requires both the logs tenant ID (numeric user ID) and an Access Policy token with logs:read scope. Official Grafana documentation specifies HTTP Basic Auth (username=tenant ID, password=token) or Authorization: Bearer <TENANT_ID>:<TOKEN>.

The current code implements only Authorization: Bearer <api_key>, omitting the tenant ID. This will fail for Grafana Cloud Loki direct access (logs-prod-XXX.grafana.net/loki/api/v1/...).

The configuration should:

  1. Clarify that a Grafana Cloud Access Policy token (not a service account token like glsa_*) is required with logs:read scope.
  2. Document that the logs tenant/user ID must be provided—either via a new tenant_id config field or through additional_headers using HTTP Basic Auth (Authorization: Basic <base64-encoded-tenant:token>).
  3. Update the example to show how users can configure direct Loki access with proper authentication.

Without this, users following the current example with a direct Loki endpoint will encounter authentication errors.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/data-sources/builtin-toolsets/grafanaloki.md` around lines 106 - 120,
Update the Grafana Cloud Loki docs for the grafana/loki toolset to require and
document the tenant ID and Access Policy token (with logs:read scope) instead of
only api_key: explain that Grafana Cloud requires either HTTP Basic Auth
(username=tenant_id, password=access_policy_token) or an Authorization: Bearer
header in the form TENANT_ID:TOKEN, and add a new config field suggestion
tenant_id (or show using additional_headers with Authorization: Basic <base64
tenant:token>) alongside api_key/access_policy_token; clarify that glsa_*
service account tokens are not acceptable and update the example config under
toolsets.grafana/loki to demonstrate the correct fields and authentication
method for direct Loki endpoints.


## Advanced Configuration

Expand Down Expand Up @@ -96,9 +146,3 @@ toolsets:
api_url: http://loki.internal:3100 # Internal URL for API calls
external_url: https://loki.example.com # URL for links in results
```

## Capabilities

| Tool Name | Description |
|-----------|-------------|
| fetch_pod_logs | Fetches pod logs from Loki |
5 changes: 3 additions & 2 deletions holmes/plugins/toolsets/grafana/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,11 @@ def build_headers(api_key: Optional[str], additional_headers: Optional[Dict[str,


def get_base_url(config: GrafanaConfig) -> str:
api_url = config.api_url.rstrip("/")
if config.grafana_datasource_uid:
return f"{config.api_url}/api/datasources/proxy/uid/{config.grafana_datasource_uid}"
return f"{api_url}/api/datasources/proxy/uid/{config.grafana_datasource_uid}"
else:
return config.api_url
return api_url


class GrafanaTempoLabelsConfig(ToolsetConfig):
Expand Down
2 changes: 1 addition & 1 deletion tests/plugins/toolsets/grafana/test_grafana.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ def test_loki_datasource_toolset_error_health_check():
toolset.check_prerequisites()

assert (
"Unable to connect to Loki.\nFailed to query Loki logs: 404 Client Error: Not Found for url: http://localhost:3000//api/datasources/proxy/uid/wrong-uid/loki/api/v1/query_range"
"Unable to connect to Loki.\nFailed to query Loki logs: 404 Client Error: Not Found for url: http://localhost:3000/api/datasources/proxy/uid/wrong-uid/loki/api/v1/query_range"
in toolset.error
)
assert toolset.status == ToolsetStatusEnum.FAILED
Loading