-
Notifications
You must be signed in to change notification settings - Fork 302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MLOB-1555] add LLMObs writers #4699
base: sabrenner/llmobs-sdk-release
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
name: LLMObs | ||
|
||
on: | ||
pull_request: | ||
push: | ||
branches: [master] | ||
schedule: | ||
- cron: '0 4 * * *' | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.ref || github.run_id }} | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
ubuntu: | ||
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v4 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 Code VulnerabilityWorkflow depends on a GitHub actions pinned by tag (...read more)Pin third party actions by hash, or at least by tag for trusted sourcesWhen using a third party action, one needs to provide its GitHub path ( No pinned git ref means the action will use the latest commit of the default branch each time it runs, eventually running newer versions of the code that were not audited by Datadog. Specifying a git tag is better, but since they are not immutable, using a full length hash is recommended to make sure the action content is actually frozen to some reviewed state. Be careful however, as even pinning an action by hash can be circumvented by attackers still. For instance, if an action relies on a Docker image which is itself not pinned to a digest, it becomes possible to alter its behaviour through the Docker image without actually changing its hash. You can learn more about this kind of attacks in Unpinnable Actions: How Malicious Code Can Sneak into Your GitHub Actions Workflows. Pinning actions by hash is still a good first line of defense against supply chain attacks. Additionally, pinning by hash or tag means the action won’t benefit from newer version updates if any, including eventual security patches. Make sure to regularly check if newer versions for an action you use are available. For actions coming from a very trustworthy source, it can make sense to use a laxer pinning policy to benefit from updates as soon as possible. |
||
- uses: ./.github/actions/testagent/start | ||
- uses: ./.github/actions/node/setup | ||
- uses: ./.github/actions/install | ||
- uses: ./.github/actions/node/18 | ||
- run: yarn test:llmobs:ci | ||
- uses: ./.github/actions/node/20 | ||
- run: yarn test:llmobs:ci | ||
- uses: ./.github/actions/node/latest | ||
- run: yarn test:llmobs:ci | ||
- if: always() | ||
uses: ./.github/actions/testagent/logs | ||
- uses: codecov/codecov-action@v3 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 Code VulnerabilityWorkflow depends on a GitHub actions pinned by tag (...read more)Pin third party actions by hash, or at least by tag for trusted sourcesWhen using a third party action, one needs to provide its GitHub path ( No pinned git ref means the action will use the latest commit of the default branch each time it runs, eventually running newer versions of the code that were not audited by Datadog. Specifying a git tag is better, but since they are not immutable, using a full length hash is recommended to make sure the action content is actually frozen to some reviewed state. Be careful however, as even pinning an action by hash can be circumvented by attackers still. For instance, if an action relies on a Docker image which is itself not pinned to a digest, it becomes possible to alter its behaviour through the Docker image without actually changing its hash. You can learn more about this kind of attacks in Unpinnable Actions: How Malicious Code Can Sneak into Your GitHub Actions Workflows. Pinning actions by hash is still a good first line of defense against supply chain attacks. Additionally, pinning by hash or tag means the action won’t benefit from newer version updates if any, including eventual security patches. Make sure to regularly check if newer versions for an action you use are available. For actions coming from a very trustworthy source, it can make sense to use a laxer pinning policy to benefit from updates as soon as possible. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
'use strict' | ||
|
||
module.exports = { | ||
EVP_PROXY_AGENT_BASE_PATH: 'evp_proxy/v2', | ||
EVP_PROXY_AGENT_ENDPOINT: 'evp_proxy/v2/api/v2/llmobs', | ||
EVP_SUBDOMAIN_HEADER_NAME: 'X-Datadog-EVP-Subdomain', | ||
EVP_SUBDOMAIN_HEADER_VALUE: 'llmobs-intake', | ||
AGENTLESS_SPANS_ENDPOINT: '/api/v2/llmobs', | ||
AGENTLESS_EVALULATIONS_ENDPOINT: '/api/intake/llm-obs/v1/eval-metric', | ||
|
||
EVP_PAYLOAD_SIZE_LIMIT: 5 << 20, // 5MB (actual limit is 5.1MB) | ||
EVP_EVENT_SIZE_LIMIT: (1 << 20) - 1024, // 999KB (actual limit is 1MB) | ||
|
||
DROPPED_IO_COLLECTION_ERROR: 'dropped_io', | ||
DROPPED_VALUE_TEXT: "[This value has been dropped because this span's size exceeds the 1MB size limit.]" | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
'use strict' | ||
|
||
function encodeUnicode (str) { | ||
if (!str) return str | ||
return str.split('').map(char => { | ||
const code = char.charCodeAt(0) | ||
if (code > 127) { | ||
return `\\u${code.toString(16).padStart(4, '0')}` | ||
} | ||
return char | ||
}).join('') | ||
} | ||
|
||
module.exports = { | ||
encodeUnicode | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,108 @@ | ||
'use strict' | ||
|
||
const request = require('../../exporters/common/request') | ||
const { URL, format } = require('url') | ||
|
||
const logger = require('../../log') | ||
|
||
const { encodeUnicode } = require('../util') | ||
|
||
class BaseLLMObsWriter { | ||
constructor ({ interval, timeout, endpoint, intake, eventType, protocol, port }) { | ||
this._interval = interval || 1000 // 1s | ||
this._timeout = timeout || 5000 // 5s | ||
this._eventType = eventType | ||
|
||
this._buffer = [] | ||
this._bufferLimit = 1000 | ||
this._bufferSize = 0 | ||
|
||
this._url = new URL(format({ | ||
protocol: protocol || 'https:', | ||
hostname: intake, | ||
port: port || 443, | ||
pathname: endpoint | ||
})) | ||
|
||
this._headers = { | ||
'Content-Type': 'application/json' | ||
} | ||
|
||
this._periodic = setInterval(() => { | ||
this.flush() | ||
}, this._interval).unref() | ||
|
||
process.once('beforeExit', () => { | ||
this.destroy() | ||
}) | ||
|
||
this._destroyed = false | ||
|
||
logger.debug(`Started ${this.constructor.name} to ${this._url}`) | ||
} | ||
|
||
append (event, byteLength) { | ||
if (this._buffer.length >= this._bufferLimit) { | ||
logger.warn(`${this.constructor.name} event buffer full (limit is ${this._bufferLimit}), dropping event`) | ||
return | ||
} | ||
|
||
this._bufferSize += byteLength || Buffer.from(JSON.stringify(event)).byteLength | ||
this._buffer.push(event) | ||
} | ||
|
||
flush () { | ||
if (this._buffer.length === 0) { | ||
return | ||
} | ||
|
||
const events = this._buffer | ||
this._buffer = [] | ||
this._bufferSize = 0 | ||
const payload = this._encode(this.makePayload(events)) | ||
|
||
const options = { | ||
headers: this._headers, | ||
method: 'POST', | ||
url: this._url, | ||
timeout: this._timeout | ||
} | ||
|
||
request(payload, options, (err, resp, code) => { | ||
if (err) { | ||
logger.error( | ||
`Error sending ${events.length} LLMObs ${this._eventType} events to ${this._url}: ${err.message}` | ||
) | ||
} else if (code >= 300) { | ||
logger.error( | ||
`Error sending ${events.length} LLMObs ${this._eventType} events to ${this._url}: ${code}` | ||
) | ||
} else { | ||
logger.debug(`Sent ${events.length} LLMObs ${this._eventType} events to ${this._url}`) | ||
} | ||
}) | ||
} | ||
|
||
makePayload (events) {} | ||
|
||
destroy () { | ||
if (!this._destroyed) { | ||
logger.debug(`Stopping ${this.constructor.name}`) | ||
clearInterval(this._periodic) | ||
process.removeListener('beforeExit', this.destroy) | ||
this.flush() | ||
this._destroyed = true | ||
} | ||
} | ||
|
||
_encode (payload) { | ||
return JSON.stringify(payload, (key, value) => { | ||
if (typeof value === 'string') { | ||
return encodeUnicode(value) // serialize unicode characters | ||
} | ||
return value | ||
Comment on lines
+100
to
+103
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just for clarification, can you explain what exactly's happening here? Does json.stringify() get called first then we run the encodeUnicode() helper on the result afterwards? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it gets run as |
||
}).replace(/\\\\u/g, '\\u') // remove double escaping | ||
} | ||
} | ||
|
||
module.exports = BaseLLMObsWriter |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
'use strict' | ||
|
||
const { AGENTLESS_EVALULATIONS_ENDPOINT } = require('../constants') | ||
const BaseWriter = require('./base') | ||
|
||
class LLMObsEvalMetricsWriter extends BaseWriter { | ||
constructor (config) { | ||
super({ | ||
endpoint: AGENTLESS_EVALULATIONS_ENDPOINT, | ||
intake: `api.${config.site}`, | ||
eventType: 'evaluation_metric' | ||
}) | ||
|
||
this._headers['DD-API-KEY'] = config.llmobs?.apiKey || config.apiKey | ||
} | ||
|
||
makePayload (events) { | ||
return { | ||
data: { | ||
type: this._eventType, | ||
attributes: { | ||
metrics: events | ||
} | ||
} | ||
} | ||
} | ||
} | ||
|
||
module.exports = LLMObsEvalMetricsWriter |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
'use strict' | ||
|
||
const { EVP_SUBDOMAIN_HEADER_NAME, EVP_SUBDOMAIN_HEADER_VALUE, EVP_PROXY_AGENT_ENDPOINT } = require('../../constants') | ||
const LLMObsBaseSpanWriter = require('./base') | ||
|
||
class LLMObsAgentProxySpanWriter extends LLMObsBaseSpanWriter { | ||
constructor (config) { | ||
super({ | ||
intake: config.hostname || 'localhost', | ||
protocol: 'http:', | ||
endpoint: EVP_PROXY_AGENT_ENDPOINT, | ||
port: config.port | ||
}) | ||
|
||
this._headers[EVP_SUBDOMAIN_HEADER_NAME] = EVP_SUBDOMAIN_HEADER_VALUE | ||
} | ||
} | ||
|
||
module.exports = LLMObsAgentProxySpanWriter |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
'use strict' | ||
|
||
const { AGENTLESS_SPANS_ENDPOINT } = require('../../constants') | ||
const LLMObsBaseSpanWriter = require('./base') | ||
|
||
class LLMObsAgentlessSpanWriter extends LLMObsBaseSpanWriter { | ||
constructor (config) { | ||
super({ | ||
intake: `llmobs-intake.${config.site}`, | ||
endpoint: AGENTLESS_SPANS_ENDPOINT | ||
}) | ||
|
||
this._headers['DD-API-KEY'] = config.llmobs?.apiKey || config.apiKey | ||
} | ||
} | ||
|
||
module.exports = LLMObsAgentlessSpanWriter |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
'use strict' | ||
|
||
const { | ||
EVP_EVENT_SIZE_LIMIT, | ||
EVP_PAYLOAD_SIZE_LIMIT, | ||
DROPPED_VALUE_TEXT, | ||
DROPPED_IO_COLLECTION_ERROR | ||
} = require('../../constants') | ||
const BaseWriter = require('../base') | ||
const logger = require('../../../log') | ||
|
||
class LLMObsSpanWriter extends BaseWriter { | ||
constructor (options) { | ||
super({ | ||
...options, | ||
eventType: 'span' | ||
}) | ||
} | ||
|
||
append (event) { | ||
const eventSizeBytes = Buffer.from(JSON.stringify(event)).byteLength | ||
if (eventSizeBytes > EVP_EVENT_SIZE_LIMIT) { | ||
logger.warn(`Dropping event input/output because its size (${eventSizeBytes}) exceeds the 1MB event size limit`) | ||
event = this._truncateSpanEvent(event) | ||
} | ||
|
||
if (this._bufferSize + eventSizeBytes > EVP_PAYLOAD_SIZE_LIMIT) { | ||
logger.debug('Flusing queue because queing next event will exceed EvP payload limit') | ||
this.flush() | ||
} | ||
|
||
super.append(event, eventSizeBytes) | ||
} | ||
|
||
makePayload (events) { | ||
return { | ||
'_dd.stage': 'raw', | ||
event_type: this._eventType, | ||
spans: events | ||
} | ||
} | ||
|
||
_truncateSpanEvent (event) { | ||
event.meta.input = { value: DROPPED_VALUE_TEXT } | ||
event.meta.output = { value: DROPPED_VALUE_TEXT } | ||
|
||
event.collection_errors = [DROPPED_IO_COLLECTION_ERROR] | ||
return event | ||
} | ||
} | ||
|
||
module.exports = LLMObsSpanWriter |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
'use strict' | ||
|
||
const { | ||
encodeUnicode | ||
} = require('../../src/llmobs/util') | ||
|
||
describe('util', () => { | ||
describe('encodeUnicode', () => { | ||
it('should encode unicode characters', () => { | ||
expect(encodeUnicode('😀')).to.equal('\\ud83d\\ude00') | ||
}) | ||
|
||
it('should encode only unicode characters in a string', () => { | ||
expect(encodeUnicode('test 😀')).to.equal('test \\ud83d\\ude00') | ||
}) | ||
}) | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟠 Code Vulnerability
No explicit permissions set for at the workflow level (...read more)
Check the permissions granted to jobs
Datadog’s GitHub organization defines default permissions for the
GITHUB_TOKEN
to be restricted (contents:read
,metadata:read
andpackages:read
).Your repository may require different setup, but please consider defining permissions for each job following the least privilege principle to restrict the impact of a possible compromission.
You can find the list of all possible permissions in Workflow syntax for GitHub Actions - GitHub Docs. Please note they can be defined at the job or the workflow level.