Skip to content

[ML] Adds ML tasks to the kibana audit log#195120

Merged
jgowdyelastic merged 24 commits intoelastic:mainfrom
jgowdyelastic:adding-ML-tasks-to-kibana-audit-log
Oct 14, 2024
Merged

[ML] Adds ML tasks to the kibana audit log#195120
jgowdyelastic merged 24 commits intoelastic:mainfrom
jgowdyelastic:adding-ML-tasks-to-kibana-audit-log

Conversation

@jgowdyelastic
Copy link
Copy Markdown
Member

@jgowdyelastic jgowdyelastic commented Oct 4, 2024 •
edited by kibanamachine
Loading

Adds a new MlAuditLogger service for logging calls to elasticsearch in kibana's audit log.
Not all calls are logged, only ones which make changes to ML jobs or trained models, e.g. creating, deleting, starting, stopping etc.

Calls to the es client are wrapped in a logging function so successes and failures can be caught and logged.

the audit log can be enabed by adding this to the kibana yml or dev.yml file
xpack.security.audit.enabled: true

An example log entry (NDJSON formatted to make it readable):

{
  "event": {
    "action": "ml_start_ad_datafeed",
    "type": [
      "change"
    ],
    "category": [
      "database"
    ],
    "outcome": "success"
  },
  "labels": {
    "application": "elastic/ml"
  },
  "user": {
    "id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
    "name": "elastic",
    "roles": [
      "superuser"
    ]
  },
  "kibana": {
    "space_id": "default",
    "session_id": "U6HQCDkk+fAEUCXs7i4qM2/MZITPxE02pp8o7h09P68="
  },
  "trace": {
    "id": "4f1b616b-8535-43e1-8516-32ea9fe76d19"
  },
  "client": {
    "ip": "127.0.0.1"
  },
  "http": {
    "request": {
      "headers": {
        "x-forwarded-for": "127.0.0.1"
      }
    }
  },
  "service": {
    "node": {
      "roles": [
        "background_tasks",
        "ui"
      ]
    }
  },
  "ecs": {
    "version": "8.11.0"
  },
  "@timestamp": "2024-10-11T09:07:47.933+01:00",
  "message": "Starting anomaly detection datafeed datafeed-11aaaa",
  "log": {
    "level": "INFO",
    "logger": "plugins.security.audit.ecs"
  },
  "process": {
    "pid": 58305,
    "uptime": 100.982390291
  },
  "transaction": {
    "id": "77c14aadc6901324"
  }
}

jgowdyelastic
jgowdyelastic commented Oct 9, 2024
View reviewed changes
Comment thread x-pack/plugins/ml/tsconfig.json
},
// add references to other TypeScript projects the plugin depends on
"@kbn/actions-plugin",
"@kbn/aiops-change-point-detection",
Copy link
Copy Markdown
Member Author

@jgowdyelastic jgowdyelastic Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorting these alphabetically

@jgowdyelastic jgowdyelastic self-assigned this Oct 10, 2024
@jgowdyelastic jgowdyelastic added :ml v9.0.0 release_note:feature Makes this part of the condensed release notes v8.16.0 backport:version Backport to applied version labels labels Oct 10, 2024
@jgowdyelastic jgowdyelastic marked this pull request as ready for review October 10, 2024 08:23
@jgowdyelastic jgowdyelastic requested a review from a team as a code owner October 10, 2024 08:23
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 10, 2024

Pinging @elastic/ml-ui (:ml)

peteharverson
peteharverson approved these changes Oct 11, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and LGTM. Just left a question about the type for one of the actions.

Comment thread x-pack/plugins/ml/server/lib/ml_client/ml_audit_logger.ts
darnautov
darnautov reviewed Oct 11, 2024
View reviewed changes
Comment thread x-pack/plugins/ml/server/lib/ml_client/ml_client.ts
async closeJob(...p: Parameters<MlClient['closeJob']>) {
await jobIdsCheck('anomaly-detector', p);
return mlClient.closeJob(...p);
return auditLogger.wrapTask(() => mlClient.closeJob(...p), 'ml_close_ad_job', p);
Copy link
Copy Markdown
Contributor

@darnautov darnautov Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider using Proxy instead of manually wrapping every call?

Copy link
Copy Markdown
Member Author

@jgowdyelastic jgowdyelastic Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had not, it would probably be just as much code, but could be neater and mean we don't have changes in the ml_client file.
I'm not sure how we'd be able to intercept the calling of the function to log the error if it fails.

darnautov
darnautov approved these changes Oct 11, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@darnautov darnautov left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I reckon we should consider refactoring using Proxy or decorators for easier maintenance in the future.

@peteharverson peteharverson changed the title [ML] Adding ML tasks to kibana audit log [ML] Adds ML tasks to the kibana audit log Oct 14, 2024
@jgowdyelastic jgowdyelastic added release_note:enhancement and removed release_note:feature Makes this part of the condensed release notes labels Oct 14, 2024
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 14, 2024

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @jgowdyelastic

@jgowdyelastic jgowdyelastic merged commit 923c450 into elastic:main Oct 14, 2024
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Oct 14, 2024

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11325840648

kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Oct 14, 2024
@jgowdyelastic @kibanamachine
[ML] Adds ML tasks to the kibana audit log (elastic#195120)
5ec58aa 
Adds a new `MlAuditLogger` service for logging calls to elasticsearch in
kibana's audit log.
Not all calls are logged, only ones which make changes to ML jobs or
trained models, e.g. creating, deleting, starting, stopping etc.

Calls to the es client are wrapped in a logging function so successes
and failures can be caught and logged.

the audit log can be enabed by adding this to the kibana yml or dev.yml
file
`xpack.security.audit.enabled: true`

An example log entry (NDJSON formatted to make it readable):
```
{
  "event": {
    "action": "ml_start_ad_datafeed",
    "type": [
      "change"
    ],
    "category": [
      "database"
    ],
    "outcome": "success"
  },
  "labels": {
    "application": "elastic/ml"
  },
  "user": {
    "id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0",
    "name": "elastic",
    "roles": [
      "superuser"
    ]
  },
  "kibana": {
    "space_id": "default",
    "session_id": "U6HQCDkk+fAEUCXs7i4qM2/MZITPxE02pp8o7h09P68="
  },
  "trace": {
    "id": "4f1b616b-8535-43e1-8516-32ea9fe76d19"
  },
  "client": {
    "ip": "127.0.0.1"
  },
  "http": {
    "request": {
      "headers": {
        "x-forwarded-for": "127.0.0.1"
      }
    }
  },
  "service": {
    "node": {
      "roles": [
        "background_tasks",
        "ui"
      ]
    }
  },
  "ecs": {
    "version": "8.11.0"
  },
  "@timestamp": "2024-10-11T09:07:47.933+01:00",
  "message": "Starting anomaly detection datafeed datafeed-11aaaa",
  "log": {
    "level": "INFO",
    "logger": "plugins.security.audit.ecs"
  },
  "process": {
    "pid": 58305,
    "uptime": 100.982390291
  },
  "transaction": {
    "id": "77c14aadc6901324"
  }
}
```

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 923c450)
@kibanamachine kibanamachine mentioned this pull request Oct 14, 2024
Merged
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Oct 14, 2024

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Oct 14, 2024
@kibanamachine @jgowdyelastic
[8.x] [ML] Adds ML tasks to the kibana audit log (#195120) (#196099)
d4f7bad 
# Backport

This will backport the following commits from `main` to `8.x`:
- [[ML] Adds ML tasks to the kibana audit log
(#195120)](#195120)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"James
Gowdy","email":"jgowdy@elastic.co"},"sourceCommit":{"committedDate":"2024-10-14T10:37:56Z","message":"[ML]
Adds ML tasks to the kibana audit log (#195120)\n\nAdds a new
`MlAuditLogger` service for logging calls to elasticsearch
in\r\nkibana's audit log.\r\nNot all calls are logged, only ones which
make changes to ML jobs or\r\ntrained models, e.g. creating, deleting,
starting, stopping etc.\r\n\r\nCalls to the es client are wrapped in a
logging function so successes\r\nand failures can be caught and
logged.\r\n\r\nthe audit log can be enabed by adding this to the kibana
yml or dev.yml\r\nfile\r\n`xpack.security.audit.enabled: true`\r\n\r\nAn
example log entry (NDJSON formatted to make it
readable):\r\n```\r\n{\r\n \"event\": {\r\n \"action\":
\"ml_start_ad_datafeed\",\r\n \"type\": [\r\n \"change\"\r\n ],\r\n
\"category\": [\r\n \"database\"\r\n ],\r\n \"outcome\": \"success\"\r\n
},\r\n \"labels\": {\r\n \"application\": \"elastic/ml\"\r\n },\r\n
\"user\": {\r\n \"id\":
\"u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0\",\r\n \"name\":
\"elastic\",\r\n \"roles\": [\r\n \"superuser\"\r\n ]\r\n },\r\n
\"kibana\": {\r\n \"space_id\": \"default\",\r\n \"session_id\":
\"U6HQCDkk+fAEUCXs7i4qM2/MZITPxE02pp8o7h09P68=\"\r\n },\r\n \"trace\":
{\r\n \"id\": \"4f1b616b-8535-43e1-8516-32ea9fe76d19\"\r\n },\r\n
\"client\": {\r\n \"ip\": \"127.0.0.1\"\r\n },\r\n \"http\": {\r\n
\"request\": {\r\n \"headers\": {\r\n \"x-forwarded-for\":
\"127.0.0.1\"\r\n }\r\n }\r\n },\r\n \"service\": {\r\n \"node\": {\r\n
\"roles\": [\r\n \"background_tasks\",\r\n \"ui\"\r\n ]\r\n }\r\n },\r\n
\"ecs\": {\r\n \"version\": \"8.11.0\"\r\n },\r\n \"@timestamp\":
\"2024-10-11T09:07:47.933+01:00\",\r\n \"message\": \"Starting anomaly
detection datafeed datafeed-11aaaa\",\r\n \"log\": {\r\n \"level\":
\"INFO\",\r\n \"logger\": \"plugins.security.audit.ecs\"\r\n },\r\n
\"process\": {\r\n \"pid\": 58305,\r\n \"uptime\": 100.982390291\r\n
},\r\n \"transaction\": {\r\n \"id\": \"77c14aadc6901324\"\r\n
}\r\n}\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"923c450c1b044a12dd938c0c5ea380a895eeaf88","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement",":ml","v9.0.0","v8.16.0","backport:version"],"title":"[ML]
Adds ML tasks to the kibana audit
log","number":195120,"url":"https://github.com/elastic/kibana/pull/195120","mergeCommit":{"message":"[ML]
Adds ML tasks to the kibana audit log (#195120)\n\nAdds a new
`MlAuditLogger` service for logging calls to elasticsearch
in\r\nkibana's audit log.\r\nNot all calls are logged, only ones which
make changes to ML jobs or\r\ntrained models, e.g. creating, deleting,
starting, stopping etc.\r\n\r\nCalls to the es client are wrapped in a
logging function so successes\r\nand failures can be caught and
logged.\r\n\r\nthe audit log can be enabed by adding this to the kibana
yml or dev.yml\r\nfile\r\n`xpack.security.audit.enabled: true`\r\n\r\nAn
example log entry (NDJSON formatted to make it
readable):\r\n```\r\n{\r\n \"event\": {\r\n \"action\":
\"ml_start_ad_datafeed\",\r\n \"type\": [\r\n \"change\"\r\n ],\r\n
\"category\": [\r\n \"database\"\r\n ],\r\n \"outcome\": \"success\"\r\n
},\r\n \"labels\": {\r\n \"application\": \"elastic/ml\"\r\n },\r\n
\"user\": {\r\n \"id\":
\"u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0\",\r\n \"name\":
\"elastic\",\r\n \"roles\": [\r\n \"superuser\"\r\n ]\r\n },\r\n
\"kibana\": {\r\n \"space_id\": \"default\",\r\n \"session_id\":
\"U6HQCDkk+fAEUCXs7i4qM2/MZITPxE02pp8o7h09P68=\"\r\n },\r\n \"trace\":
{\r\n \"id\": \"4f1b616b-8535-43e1-8516-32ea9fe76d19\"\r\n },\r\n
\"client\": {\r\n \"ip\": \"127.0.0.1\"\r\n },\r\n \"http\": {\r\n
\"request\": {\r\n \"headers\": {\r\n \"x-forwarded-for\":
\"127.0.0.1\"\r\n }\r\n }\r\n },\r\n \"service\": {\r\n \"node\": {\r\n
\"roles\": [\r\n \"background_tasks\",\r\n \"ui\"\r\n ]\r\n }\r\n },\r\n
\"ecs\": {\r\n \"version\": \"8.11.0\"\r\n },\r\n \"@timestamp\":
\"2024-10-11T09:07:47.933+01:00\",\r\n \"message\": \"Starting anomaly
detection datafeed datafeed-11aaaa\",\r\n \"log\": {\r\n \"level\":
\"INFO\",\r\n \"logger\": \"plugins.security.audit.ecs\"\r\n },\r\n
\"process\": {\r\n \"pid\": 58305,\r\n \"uptime\": 100.982390291\r\n
},\r\n \"transaction\": {\r\n \"id\": \"77c14aadc6901324\"\r\n
}\r\n}\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"923c450c1b044a12dd938c0c5ea380a895eeaf88"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195120","number":195120,"mergeCommit":{"message":"[ML]
Adds ML tasks to the kibana audit log (#195120)\n\nAdds a new
`MlAuditLogger` service for logging calls to elasticsearch
in\r\nkibana's audit log.\r\nNot all calls are logged, only ones which
make changes to ML jobs or\r\ntrained models, e.g. creating, deleting,
starting, stopping etc.\r\n\r\nCalls to the es client are wrapped in a
logging function so successes\r\nand failures can be caught and
logged.\r\n\r\nthe audit log can be enabed by adding this to the kibana
yml or dev.yml\r\nfile\r\n`xpack.security.audit.enabled: true`\r\n\r\nAn
example log entry (NDJSON formatted to make it
readable):\r\n```\r\n{\r\n \"event\": {\r\n \"action\":
\"ml_start_ad_datafeed\",\r\n \"type\": [\r\n \"change\"\r\n ],\r\n
\"category\": [\r\n \"database\"\r\n ],\r\n \"outcome\": \"success\"\r\n
},\r\n \"labels\": {\r\n \"application\": \"elastic/ml\"\r\n },\r\n
\"user\": {\r\n \"id\":
\"u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0\",\r\n \"name\":
\"elastic\",\r\n \"roles\": [\r\n \"superuser\"\r\n ]\r\n },\r\n
\"kibana\": {\r\n \"space_id\": \"default\",\r\n \"session_id\":
\"U6HQCDkk+fAEUCXs7i4qM2/MZITPxE02pp8o7h09P68=\"\r\n },\r\n \"trace\":
{\r\n \"id\": \"4f1b616b-8535-43e1-8516-32ea9fe76d19\"\r\n },\r\n
\"client\": {\r\n \"ip\": \"127.0.0.1\"\r\n },\r\n \"http\": {\r\n
\"request\": {\r\n \"headers\": {\r\n \"x-forwarded-for\":
\"127.0.0.1\"\r\n }\r\n }\r\n },\r\n \"service\": {\r\n \"node\": {\r\n
\"roles\": [\r\n \"background_tasks\",\r\n \"ui\"\r\n ]\r\n }\r\n },\r\n
\"ecs\": {\r\n \"version\": \"8.11.0\"\r\n },\r\n \"@timestamp\":
\"2024-10-11T09:07:47.933+01:00\",\r\n \"message\": \"Starting anomaly
detection datafeed datafeed-11aaaa\",\r\n \"log\": {\r\n \"level\":
\"INFO\",\r\n \"logger\": \"plugins.security.audit.ecs\"\r\n },\r\n
\"process\": {\r\n \"pid\": 58305,\r\n \"uptime\": 100.982390291\r\n
},\r\n \"transaction\": {\r\n \"id\": \"77c14aadc6901324\"\r\n
}\r\n}\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"923c450c1b044a12dd938c0c5ea380a895eeaf88"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: James Gowdy <jgowdy@elastic.co>
@jgowdyelastic jgowdyelastic mentioned this pull request Oct 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@peteharverson peteharverson peteharverson approved these changes

@darnautov darnautov darnautov approved these changes

Assignees

@jgowdyelastic jgowdyelastic

Labels

backport:version Backport to applied version labels :ml release_note:enhancement v8.16.0 v9.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jgowdyelastic @elasticmachine @kibanamachine @darnautov @peteharverson