Skip to content

[AWS logs forwarder] Support Python 3.12 #878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ge0Aja merged 3 commits into from
Dec 19, 2024
Merged

[AWS logs forwarder] Support Python 3.12 #878

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
60300eb
[WIP][AWS logs forwarder] Support Py 3.12
ge0Aja Dec 18, 2024
a2051ed
Update github workflow
ge0Aja Dec 18, 2024
1ec40c0
add missing package ujson
ge0Aja Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/integration_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
strategy:
max-parallel: 4
matrix:
python-version: ["3.10", "3.11"]
python-version: ["3.11", "3.12"]
steps:
- name: Checkout
uses: actions/checkout@v4
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
strategy:
max-parallel: 4
matrix:
python-version: ['3.10', '3.11']
python-version: ['3.11', '3.12']
steps:
- name: Checkout source
uses: actions/checkout@v3
Expand Down
9 changes: 5 additions & 4 deletions aws/logs_monitoring/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ bytecode
cattrs
certifi
charset-normalizer
datadog-lambda==5.87.0
datadog==0.48.0
ddsketch==2.0.4
ddtrace==2.6.5
datadog-lambda==6.104.0
datadog==0.50.2
ddsketch==3.0.1
ddtrace==2.17.3
deprecated
envier
exceptiongroup
Expand All @@ -22,3 +22,4 @@ urllib3>=1.26.19,<3.0
wrapt==1.14.0
xmltodict
zipp
ujson
Copy link

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 bot Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Library Vulnerability

ujson

Incorrect handling of invalid surrogate pair characters (...read more)

Impact

What kind of vulnerability is it? Who is impacted?

Anyone parsing JSON from an untrusted source is vulnerable.

JSON strings that contain escaped surrogate characters not part of a proper surrogate pair were decoded incorrectly. Besides corrupting strings, this allowed for potential key confusion and value overwriting in dictionaries.

Examples:

# An unpaired high surrogate character is ignored.
>>> ujson.loads(r'"\uD800"')
''
>>> ujson.loads(r'"\uD800hello"')
'hello'

# An unpaired low surrogate character is preserved.
>>> ujson.loads(r'"\uDC00"')
'\udc00'

# A pair of surrogates with additional non surrogate characters pair up in spite of being invalid.
>>> ujson.loads(r'"\uD800foo bar\uDC00"')
'foo bar𐀀'

Patches

Has the problem been patched? What versions should users upgrade to?

Users should upgrade to UltraJSON 5.4.0.

From version 5.4.0, UltraJSON decodes lone surrogates in the same way as the standard library's json module does, preserving them in the parsed output:

>>> ujson.loads(r'"\uD800"')
'\ud800'
>>> ujson.loads(r'"\uD800hello"')
'\ud800hello'
>>> ujson.loads(r'"\uDC00"')
'\udc00'
>>> ujson.loads(r'"\uD800foo bar\uDC00"')
'\ud800foo bar\udc00'

Workarounds

Is there a way for users to fix or remediate the vulnerability without upgrading?

Short of switching to an entirely different JSON library, there are no safe alternatives to upgrading.

For more information

If you have any questions or comments about this advisory:

View in Datadog  Leave us feedback  Documentation

4 changes: 2 additions & 2 deletions aws/logs_monitoring/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@
author="Datadog, Inc.",
author_email="[email protected]",
classifiers=[
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
],
keywords="datadog aws lambda layer",
python_requires=">=3.10, <3.12",
python_requires=">=3.11, <3.13",
extras_require={
"dev": ["nose2==0.9.1", "flake8==3.7.9", "requests==2.22.0", "boto3==1.10.33"]
},
Expand Down
18 changes: 15 additions & 3 deletions aws/logs_monitoring/template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,13 @@ Parameters:
- WARN
- ERROR
- CRITICAL
DdTraceEnabled:
Type: String
Default: "true"
AllowedValues:
- "true"
- "false"
Description: Set to false to disable trace forwarding.
Conditions:
IsAWSChina: !Equals [!Ref 'AWS::Partition', aws-cn]
IsGovCloud: !Equals [!Ref 'AWS::Partition', aws-us-gov]
Expand Down Expand Up @@ -348,6 +355,7 @@ Conditions:
- !Equals [!Join ["", !Ref VPCSubnetIds], ""]
SetDdLogLevel: !Not
- !Equals [!Ref DdLogLevel, ""]
SetDdTraceEnabled: !Equals [!Ref DdTraceEnabled, false]
Rules:
MustSetDdApiKey:
Assertions:
Expand Down Expand Up @@ -396,7 +404,7 @@ Resources:
- {DdForwarderVersion: !FindInMap [Constants, DdForwarder, Version]}
- ZipFile: " "
MemorySize: !Ref MemorySize
Runtime: python3.11
Runtime: python3.12
Timeout: !Ref Timeout
Tags:
- Key: "dd_forwarder_version"
Expand Down Expand Up @@ -540,6 +548,10 @@ Resources:
- SetDdLogLevel
- !Ref DdLogLevel
- !Ref AWS::NoValue
DD_TRACE_ENABLED: !If
- SetDdTraceEnabled
- !Ref DdTraceEnabled
- "true"
ReservedConcurrentExecutions: !If
- SetReservedConcurrentExecutions
- !Ref ReservedConcurrency
Expand Down Expand Up @@ -724,7 +736,7 @@ Resources:
- !Ref AWS::NoValue
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: true
- BucketKeyEnabled: true
ServerSideEncryptionByDefault:
SSEAlgorithm: "aws:kms"
PublicAccessBlockConfiguration:
Expand Down Expand Up @@ -788,7 +800,7 @@ Resources:
Properties:
Description: Copies Datadog Forwarder zip to the destination S3 bucket
Handler: index.handler
Runtime: python3.11
Runtime: python3.12
Timeout: 600
Code:
ZipFile: |
Expand Down
2 changes: 1 addition & 1 deletion aws/logs_monitoring/tools/build_bundle.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ else
VERSION=$1
fi

PYTHON_VERSION="${PYTHON_VERSION:-3.11}"
PYTHON_VERSION="${PYTHON_VERSION:-3.12}"
FORWARDER_PREFIX="aws-dd-forwarder"
FORWARDER_DIR="../.forwarder"

Expand Down
18 changes: 14 additions & 4 deletions aws/logs_monitoring/tools/integration_tests/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ services:
AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
AWS_SECURITY_TOKEN: "${AWS_SECURITY_TOKEN}"
AWS_SESSION_TOKEN: "${AWS_SESSION_TOKEN}"
DOCKER_LAMBDA_STAY_OPEN: 1
AWS_DEFAULT_REGION: us-east-1
DD_LOG_LEVEL: ${LOG_LEVEL:-info}
DD_API_KEY: abcdefghijklmnopqrstuvwxyz012345 # Must be 32 characters exactly
DD_URL: recorder # Used for logs intake
Expand All @@ -45,11 +45,18 @@ services:
DD_FETCH_LOG_GROUP_TAGS: "true"
DD_FETCH_STEP_FUNCTIONS_TAGS: "false" # intentionally set false to allow integration test for step function logs to run without hitting aws
DD_STORE_FAILED_EVENTS: "false"
DD_TRACE_ENABLED: "true"
expose:
- 9001
- 8080
depends_on:
recorder:
condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/2015-03-31/functions/function/invocations"]
interval: 10s
timeout: 5s
retries: 3


tester:
image: ${PYTHON_BASE}
Expand All @@ -60,8 +67,11 @@ services:
working_dir: /tester
environment:
RECORDER_URL: http://recorder:8080/recording
FORWARDER_URL: http://forwarder:9001/2015-03-31/functions/myfunction/invocations
FORWARDER_URL: http://forwarder:8080/2015-03-31/functions/function/invocations
UPDATE_SNAPSHOTS: ${UPDATE_SNAPSHOTS:-false}
SNAPSHOTS_DIR_NAME: ${SNAPSHOTS_DIR_NAME}
depends_on:
- forwarder
forwarder:
condition: service_healthy
recorder:
condition: service_healthy
14 changes: 8 additions & 6 deletions aws/logs_monitoring/tools/integration_tests/integration_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@

set -e

PYTHON_VERSION="python3.11"
PYTHON_IMAGE="python:3.11"
PYTHON_VERSION="python3.12"
PYTHON_VERSION_TAG="3.12"
PYTHON_IMAGE="python:3.12"
SKIP_FORWARDER_BUILD=false
UPDATE_SNAPSHOTS=false
LOG_LEVEL=info
Expand Down Expand Up @@ -38,10 +39,11 @@ for arg in "$@"; do

# -v or --python-version
# The version of the Python Lambda runtime to use
# Must be 3.9 or 3.10
# Must be 3.11 or 3.12
-v=* | --python-version=*)
PYTHON_VERSION="python${arg#*=}"
PYTHON_IMAGE="python:${arg#*=}"
PYTHON_VERSION_TAG="${arg#*=}"
shift
;;

Expand Down Expand Up @@ -78,8 +80,8 @@ for arg in "$@"; do
esac
done

if [ $PYTHON_VERSION != "python3.10" ] && [ $PYTHON_VERSION != "python3.11" ]; then
echo "Must use either Python 3.10 or 3.11"
if [ $PYTHON_VERSION != "python3.11" ] && [ $PYTHON_VERSION != "python3.12" ]; then
echo "Must use either Python 3.11 or 3.12"
exit 1
fi

Expand Down Expand Up @@ -152,7 +154,7 @@ cd $INTEGRATION_TESTS_DIR
echo "Building Docker Image for Forwarder with tag datadog-log-forwarder:$PYTHON_VERSION"
docker buildx build --platform linux/amd64 --file "${INTEGRATION_TESTS_DIR}/forwarder/Dockerfile" -t "datadog-log-forwarder:$PYTHON_VERSION" ../../.forwarder --no-cache \
--build-arg forwarder='aws-dd-forwarder-0.0.0' \
--build-arg image="mlupin/docker-lambda:${PYTHON_VERSION}-x86_64"
--build-arg image="public.ecr.aws/lambda/python:${PYTHON_VERSION_TAG}-x86_64"

echo "Running integration tests for ${PYTHON_VERSION}"
LOG_LEVEL=${LOG_LEVEL} \
Expand Down
72 changes: 22 additions & 50 deletions aws/logs_monitoring/tools/integration_tests/recorder/pb/span_pb2.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

35 changes: 16 additions & 19 deletions aws/logs_monitoring/tools/integration_tests/recorder/pb/trace_payload_pb2.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading