Validator: Date-time format fails validation

[j2clerck]

What were you trying to accomplish?
I am trying to validate an event against a json schema using the validate function.
validate(event=json_payload, schema=INPUT)

Expected Behavior

I expected the schema to be validated correctly.

Current Behavior

The validate function is raising the following error:
Error: argument of type 'NoneType' is not iterable

Possible Solution

Instead of using a default value of None for formats, formats should be an empty dict.
It seems the issues comes from fastjsonschema not handling custom format date-time in the schema. According from fastjsonschema repo the issues is fixed. It might be because the validate function set the formats to None whereas fastjsonschema.validate function sets the formats to an empty dict.

Steps to Reproduce (for bugs)

1. fastjsonschema.validate(definition=INPUT,data=event, formats=None)
2. fastjsonschema.validate(definition=INPUT,data=event, formats={})
3. validate(event=json_payload, schema=INPUT, formats={})

Environment

• Powertools version used: 1.17.0
• Packaging format (Layers, PyPi): PyPi
• AWS Lambda function runtime: Python3.8
• Debugging logs

How to enable debug mode**

# paste logs here

Some links:
horejsek/python-fastjsonschema#115

Schema example

{
    "$schema": "http://json-schema.org/draft-07/schema",
    "type": "object",
    "required": ["eventType"],
    "properties": {
        "eventType": {
            "$id": "#/properties/eventType",
            "type": "string",
            "pattern": "com.okta.event_hook",
        },
        "eventTypeVersion": {
            "$id": "#/properties/eventTypeVersion",
            "type": "string",
        },
        "cloudEventsVersion": {
            "$id": "#/properties/cloudEventsVersion",
            "type": "string",
        },
        "contentType": {
            "$id": "#/properties/contentType",
            "type": "string",
        },
        "eventId": {
            "$id": "#/properties/eventId",
            "type": "string",
        },
        "data": {
            "$id": "#/properties/data",
            "type": "object",
            "required": ["events"],
            "properties": {
                "events": {
                    "type": "array",
                    "items": {"type": "object"},
                }
            },
        },
    },
}

event example:

{
    "eventType": "com.okta.event_hook",
    "eventTypeVersion": "1.0",
    "cloudEventsVersion": "0.1",
    "source": "https://dev-XXXXXX.okta.com/api/v1/eventHooks/XXXXXXX",
    "eventId": "c9ad0d85-1ce9-4dfe-996c-8e5441214189",
    "data": {
        "events": [
            {
                "uuid": "242dbb7a-d50d-11eb-9787-79648ebc0d8c",
                "published": "2021-06-24T16:56:38.320Z",
                "eventType": "group.user_membership.add",
                "version": "0",
                "displayMessage": "Add user to group membership",
                "severity": "INFO",
                "client": {
                    "userAgent": "",
                    "zone": "",
                    "device": "",
                    "id": "",
                    "ipAddress": "",
                    "geographicalContext": "",
                    "ipChain": []
                },
                "device": "",
                "actor": {
                    "id": "",
                    "type": "User",
                    "alternateId": "[email protected]",
                    "displayName": "John Doe",
                    "detailEntry": ""
                },
                "outcome": {
                    "result": "SUCCESS",
                    "reason": ""
                },
                "target": [
                    {
                        "id": "",
                        "type": "User",
                        "alternateId": "[email protected]",
                        "displayName": "John Doe",
                        "detailEntry": ""
                    },
                    {
                        "id": "00ga8rdpafWMUl69l357",
                        "type": "UserGroup",
                        "alternateId": "unknown",
                        "displayName": "SomeGroup",
                        "detailEntry": ""
                    }
                ],
                "transaction": {
                    "type": "JOB",
                    "id": "gwja8rfhsaVSZvTn9357",
                    "detail": {}
                },
                "debugContext": {
                    "debugData": {}
                },
                "legacyEventType": "core.user_group_member.user_add",
                "authenticationContext": {
                    "authenticationProvider": "",
                    "credentialProvider": "",
                    "credentialType": "",
                    "issuer": "",
                    "authenticationStep": 0,
                    "externalSessionId": "",
                    "interface": ""
                },
                "securityContext": {
                    "asNumber": "",
                    "asOrg": "",
                    "isp": "",
                    "domain": "",
                    "isProxy": ""
                },
                "insertionTimestamp": ""
            }
        ]
    },
    "eventTime": "2021-06-29T14:46:06.804Z",
    "contentType": "application/json"
}

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

Hey @j2clerck Thanks for raising this - I've just managed to reproduce by re-adding the date-time format in the schema.

Full snippet to reproduce

from aws_lambda_powertools.utilities.validation import validate

schema = {
    "$schema": "http://json-schema.org/draft-07/schema",
    "type": "object",
    "required": ["eventType"],
    "properties": {
        "eventType": {
            "$id": "#/properties/eventType",
            "type": "string",
            "pattern": "com.okta.event_hook",
        },
        "eventTypeVersion": {
            "$id": "#/properties/eventTypeVersion",
            "type": "string",
        },
        "cloudEventsVersion": {
            "$id": "#/properties/cloudEventsVersion",
            "type": "string",
        },
        "contentType": {
            "$id": "#/properties/contentType",
            "type": "string",
        },
        "eventId": {
            "$id": "#/properties/eventId",
            "type": "string",
        },
        "data": {
            "$id": "#/properties/data",
            "type": "object",
            "required": ["events"],
            "properties": {
                "events": {
                    "type": "array",
                    "items": {"type": "object"},
                    "properties": {
                        "published": {
                            "id": "#/properties/data/events/published",
                            "type": "string",
                            "format": "date-time"
                        }
                    }
                }
            },
        },
    },
}

raw_event = {
    "eventType": "com.okta.event_hook",
    "eventTypeVersion": "1.0",
    "cloudEventsVersion": "0.1",
    "source": "https://dev-XXXXXX.okta.com/api/v1/eventHooks/XXXXXXX",
    "eventId": "c9ad0d85-1ce9-4dfe-996c-8e5441214189",
    "data": {
        "events": [
            {
                "uuid": "242dbb7a-d50d-11eb-9787-79648ebc0d8c",
                "published": "2021-06-24T16:56:38.320Z",
                "eventType": "group.user_membership.add",
                "version": "0",
                "displayMessage": "Add user to group membership",
                "severity": "INFO",
                "client": {
                    "userAgent": "",
                    "zone": "",
                    "device": "",
                    "id": "",
                    "ipAddress": "",
                    "geographicalContext": "",
                    "ipChain": []
                },
                "device": "",
                "actor": {
                    "id": "",
                    "type": "User",
                    "alternateId": "[email protected]",
                    "displayName": "John Doe",
                    "detailEntry": ""
                },
                "outcome": {
                    "result": "SUCCESS",
                    "reason": ""
                },
                "target": [
                    {
                        "id": "",
                        "type": "User",
                        "alternateId": "[email protected]",
                        "displayName": "John Doe",
                        "detailEntry": ""
                    },
                    {
                        "id": "00ga8rdpafWMUl69l357",
                        "type": "UserGroup",
                        "alternateId": "unknown",
                        "displayName": "SomeGroup",
                        "detailEntry": ""
                    }
                ],
                "transaction": {
                    "type": "JOB",
                    "id": "gwja8rfhsaVSZvTn9357",
                    "detail": {}
                },
                "debugContext": {
                    "debugData": {}
                },
                "legacyEventType": "core.user_group_member.user_add",
                "authenticationContext": {
                    "authenticationProvider": "",
                    "credentialProvider": "",
                    "credentialType": "",
                    "issuer": "",
                    "authenticationStep": 0,
                    "externalSessionId": "",
                    "interface": ""
                },
                "securityContext": {
                    "asNumber": "",
                    "asOrg": "",
                    "isp": "",
                    "domain": "",
                    "isProxy": ""
                },
                "insertionTimestamp": ""
            }
        ]
    },
    "eventTime": "2021-06-29T14:46:06.804Z",
    "contentType": "application/json"
}

validate(event=raw_event, schema=schema)

[heitorlessa]

PR created - will merge as soon as CI checks pass. I took a look upstream to further investigate None, and what happened there was that despite string being the type, it looks up for format keyword in the schema, and if it find anything it first tries to use a custom format input (None in this case), then fallback to lookup for built-in custom formats introduced in later drafts.

[heitorlessa]

Staging this for the next release - As there's a workaround (format={}) for now, I'm gonna finish other changes instead of making an emergency release I normally would -- Let me know otherwise!