Bug: Pydantic types that cannot be instantiated - deserialization #2334
Labels
Bug 🐛
This is something that is not working as expected
(De)serialization
This is related to our Serialization (msgspec)
Help Wanted 🆘
This is good for people to work on
Comments
geeshta
added
Bug 🐛
|
Schema validation of Pydantic and msgspec can't be used transparently/interchangeably: like pointed out by @geeshta, the type As far as I understand, currently in Litestar msgspec always does schema validation, irrespective if there's a validation later on by a plugin, e.g. Pydantic? The problem is that when a PydanticDTO is used, BOTH Pydantic and msgspec do a schema validation, which poses a problem for types like I think the behaviour should be that msgspec only does schema validation if there's no other plugin doing that instead. Would you agree? |
|
When the msgspec transfer models are created for the pydantic type, the type on the transfer model should be annotated E.g., pydantic model: class WithEmail(BaseModel):
email: EmailStrTransfer model produced by DTO should be: class WithEmailTransfer(msgspec.Struct):
email: strGiven:
The down-typing from litestar/litestar/contrib/pydantic/pydantic_dto_factory.py Lines 54 to 56 in 7c3d24f |
jw-can
added a commit
to jw-can/litestar
that referenced
this issue
Oct 20, 2023
…spec validation (resolves litestar-org#2334)
5 tasks
|
@peterschutt thanks for the hints! |
jw-can
added a commit
to jw-can/litestar
that referenced
this issue
Dec 2, 2023
…spec validation (resolves litestar-org#2334)
JacobCoffee
added
the
(De)serialization
4 tasks
|
Just ran into this, any update or workaround? EDIT: Not a general fix but at least for EmailStr this workaround seems to work for me: from pydantic import AfterValidator, validate_email
class EmailModel(BaseModel):
email: Annotated[str, AfterValidator(lambda v: validate_email(v)[1])] |
peterschutt
added a commit
that referenced
this issue
Mar 31, 2024
This PR simplifies the type that we apply to transfer models for pydantic field types in specific circumstances. - `JsonValue`: this field type is an instance of `TypeAliasType` at runtime, and contains recursive type definitions. Pydantic allow `list`, `dict`, `str`, `bool`, `int`, `float` and `None`, and the value types of `list` and `dict` are allowed to be the same. We type this as `Any` on the transfer models as this is pretty much the same thing for msgspec ([ref][1]). - `EmailStr`. These are typed as `EmailStr` which is a class at runtime which is not a `str`, however they are a string and `if TYPE_CHECKING` is used to get type checkers to play along. If we return a `str` from a msgspec decode hook for the type, it msgspec won't validate the input. So we must tell msgspec its a string. This also works well with encoding because it is one. - IP/Network/Interface types. These are represented by types such as `IPvAnyAddress`, but they are actually parsed into instances of stdlib types such as `IPv4Address` and others from the `ipaddress` module. Given that an instance of `IPvAnyAddress` cannot be meaningfully returned from a decoding hook, we have to tell msgspec that these are strings on the transfer models. Encoding the stdlib ip types to str is natively handled by msgspec. Closes #2334 1: https://jcristharif.com/msgspec/supported-types.html#any
peterschutt
added a commit
that referenced
this issue
Mar 31, 2024
This PR simplifies the type that we apply to transfer models for pydantic field types in specific circumstances. - `JsonValue`: this field type is an instance of `TypeAliasType` at runtime, and contains recursive type definitions. Pydantic allow `list`, `dict`, `str`, `bool`, `int`, `float` and `None`, and the value types of `list` and `dict` are allowed to be the same. We type this as `Any` on the transfer models as this is pretty much the same thing for msgspec ([ref][1]). - `EmailStr`. These are typed as `EmailStr` which is a class at runtime which is not a `str`, however they are a string and `if TYPE_CHECKING` is used to get type checkers to play along. If we return a `str` from a msgspec decode hook for the type, it msgspec won't validate the input. So we must tell msgspec its a string. This also works well with encoding because it is one. - IP/Network/Interface types. These are represented by types such as `IPvAnyAddress`, but they are actually parsed into instances of stdlib types such as `IPv4Address` and others from the `ipaddress` module. Given that an instance of `IPvAnyAddress` cannot be meaningfully returned from a decoding hook, we have to tell msgspec that these are strings on the transfer models. Encoding the stdlib ip types to str is natively handled by msgspec. Closes #2334 1: https://jcristharif.com/msgspec/supported-types.html#any
peterschutt
added a commit
that referenced
this issue
Mar 31, 2024
This PR simplifies the type that we apply to transfer models for pydantic field types in specific circumstances. - `JsonValue`: this field type is an instance of `TypeAliasType` at runtime, and contains recursive type definitions. Pydantic allow `list`, `dict`, `str`, `bool`, `int`, `float` and `None`, and the value types of `list` and `dict` are allowed to be the same. We type this as `Any` on the transfer models as this is pretty much the same thing for msgspec ([ref][1]). - `EmailStr`. These are typed as `EmailStr` which is a class at runtime which is not a `str`, however they are a string and `if TYPE_CHECKING` is used to get type checkers to play along. If we return a `str` from a msgspec decode hook for the type, it msgspec won't validate the input. So we must tell msgspec its a string. This also works well with encoding because it is one. - IP/Network/Interface types. These are represented by types such as `IPvAnyAddress`, but they are actually parsed into instances of stdlib types such as `IPv4Address` and others from the `ipaddress` module. Given that an instance of `IPvAnyAddress` cannot be meaningfully returned from a decoding hook, we have to tell msgspec that these are strings on the transfer models. Encoding the stdlib ip types to str is natively handled by msgspec. Closes #2334 1: https://jcristharif.com/msgspec/supported-types.html#any
peterschutt
added a commit
that referenced
this issue
Mar 31, 2024
feat: pydantic DTO with non-instantiable types. This PR simplifies the type that we apply to transfer models for pydantic field types in specific circumstances. - `JsonValue`: this field type is an instance of `TypeAliasType` at runtime, and contains recursive type definitions. Pydantic allow `list`, `dict`, `str`, `bool`, `int`, `float` and `None`, and the value types of `list` and `dict` are allowed to be the same. We type this as `Any` on the transfer models as this is pretty much the same thing for msgspec ([ref][1]). - `EmailStr`. These are typed as `EmailStr` which is a class at runtime which is not a `str`, however they are a string and `if TYPE_CHECKING` is used to get type checkers to play along. If we return a `str` from a msgspec decode hook for the type, it msgspec won't validate the input. So we must tell msgspec its a string. This also works well with encoding because it is one. - IP/Network/Interface types. These are represented by types such as `IPvAnyAddress`, but they are actually parsed into instances of stdlib types such as `IPv4Address` and others from the `ipaddress` module. Given that an instance of `IPvAnyAddress` cannot be meaningfully returned from a decoding hook, we have to tell msgspec that these are strings on the transfer models. Encoding the stdlib ip types to str is natively handled by msgspec. Closes #2334 1: https://jcristharif.com/msgspec/supported-types.html#any
cofin
pushed a commit
that referenced
this issue
Apr 1, 2024
feat: pydantic DTO with non-instantiable types. This PR simplifies the type that we apply to transfer models for pydantic field types in specific circumstances. - `JsonValue`: this field type is an instance of `TypeAliasType` at runtime, and contains recursive type definitions. Pydantic allow `list`, `dict`, `str`, `bool`, `int`, `float` and `None`, and the value types of `list` and `dict` are allowed to be the same. We type this as `Any` on the transfer models as this is pretty much the same thing for msgspec ([ref][1]). - `EmailStr`. These are typed as `EmailStr` which is a class at runtime which is not a `str`, however they are a string and `if TYPE_CHECKING` is used to get type checkers to play along. If we return a `str` from a msgspec decode hook for the type, it msgspec won't validate the input. So we must tell msgspec its a string. This also works well with encoding because it is one. - IP/Network/Interface types. These are represented by types such as `IPvAnyAddress`, but they are actually parsed into instances of stdlib types such as `IPv4Address` and others from the `ipaddress` module. Given that an instance of `IPvAnyAddress` cannot be meaningfully returned from a decoding hook, we have to tell msgspec that these are strings on the transfer models. Encoding the stdlib ip types to str is natively handled by msgspec. Closes #2334 1: https://jcristharif.com/msgspec/supported-types.html#any
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
When deserializing object, Litestar checks if a given
valuematches a giventarget_type. Additionaltype_decoderscan be provided to convert a type given after deserialization to the expected type.The problem is - when using Pydantic models and PydanticDTO's - Pydantic has some types that cannot be instantiated and are really not supposed to be used elsewhere other than model declaration - for example
EmailStr.When deserializing an object like:
using a DTO given in the MCVE -> the expected type of
emailduring runtime isEmailStr(see screenshot). Msgspec after deserialization returns astrso a decoder is needed. However, there is no way to instantiateEmailStr- it is only meant to be used as a type annotation for a Pydantic model.castdoesn't change the type at runtime it is only useful for type checkers. So such decoder is impossible.The solution should be changing the
target_typetostrin thedefault_deserializerfunction - line 91 oflitestar/serialization/msgspec_hooks.pyand consequently in the call tomsgspec.json.decodeat line 187 of the same file. Internally by Pydantic, they're treated as astrafter validation is performed.Which should be handled by the PydanticDTO. Also there's more types in Pydantic that cannot be instantiated like
ImportStrand maybe others. The target type for these should always be they're naive counterpart.URL to code causing the issue
No response
MCVE
Steps to reproduce
{"email": "[email protected]"}to/emailmsgspec.ValidationErroroccurstype_decoders, a similar error appearsScreenshots
Logs
Full traceback
2.0.1
Platform
Funding
The text was updated successfully, but these errors were encountered: