feat: add support for `NamedTuple` and `TypedDict` types #2216

PrettyWood · 2020-12-22T18:54:12Z

Change Summary

Also expose two utils

create_model_from_namedtuple
create_model_from_typeddict

Related issue number

fix #760
fix #1324
fix #1430

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)

codecov · 2020-12-22T18:55:48Z

Codecov Report

Merging #2216 (1e60f5a) into master (13a5c7d) will decrease coverage by 0.11%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##            master    #2216      +/-   ##
===========================================
- Coverage   100.00%   99.88%   -0.12%     
===========================================
  Files           21       23       +2     
  Lines         4199     4405     +206     
  Branches       854      886      +32     
===========================================
+ Hits          4199     4400     +201     
- Misses           0        5       +5

Impacted Files	Coverage Δ
pydantic/annotated_types.py	`100.00% <100.00%> (ø)`
pydantic/fields.py	`100.00% <100.00%> (ø)`
pydantic/schema.py	`100.00% <100.00%> (ø)`
pydantic/typing.py	`100.00% <100.00%> (ø)`
pydantic/validators.py	`100.00% <100.00%> (ø)`
pydantic/types.py	`100.00% <0.00%> (ø)`
pydantic/networks.py	`100.00% <0.00%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13a5c7d...1e60f5a. Read the comment docs.

samuelcolvin

This looks awesome, would be great if we could have a proper example in the docs.

samuelcolvin · 2021-01-01T19:22:14Z

docs/usage/types.md

 `typing.Dict`
 : see [Typing Iterables](#typing-iterables) below for more detail on parsing and validation

+`subclass of typing.TypedDict`


Suggested change

`subclass of typing.TypedDict`

`typing.TypedDict`

I think the subclass is implicit

It used to be the case for me too but I now tend to disagree since we added the support of enum.Enum to have a valid enum instance. We could someday add typing.TypedDict just to have a valid TypedDict instance for example. WDYT?

I would still prefer to remove subclass of and instead be explicity when we mean the actual type, like Enum.

samuelcolvin · 2021-01-01T19:22:45Z

docs/usage/types.md

@@ -85,9 +85,17 @@ with custom properties and validation.
 `typing.Tuple`
 : see [Typing Iterables](#typing-iterables) below for more detail on parsing and validation

+`subclass of typing.NamedTuple (or collections.namedtuple)`


Suggested change

`subclass of typing.NamedTuple (or collections.namedtuple)`

`typing.NamedTuple` (or `collections.namedtuple`)

again I think the "subclass of..." is implicit

samuelcolvin · 2021-01-01T19:30:30Z

pydantic/typing.py

+def is_typed_dict_type(type_: Type[Any]) -> bool:
+    from .utils import lenient_issubclass
+
+    return lenient_issubclass(type_, dict) and getattr(type_, '__annotations__', None)


Humm, I wonder if this is a water tight check. I can't think of a better solution though.

Currently there is no easier way afaik. There will be a public method in 3.10 though

pydantic/validators.py

samuelcolvin · 2021-01-01T19:37:02Z

tests/test_main.py

@@ -1425,3 +1434,90 @@ class M(BaseModel):
        a: int

    get_type_hints(M.__config__)
+
+
+def test_named_tuple():


I know test files are a mess, but please can you move this to somewhere else. I try to keep test_main.py for the most vanilla cases.

Sure thing! TBH I didn't know it was for vanilla cases. Maybe a module docstring on top of the module could help. I'll try to find a better place for these bad boys

I created a new dedicated file test_annotated_types. Suggestions welcome if you don't like it

PrettyWood · 2021-01-02T17:03:11Z

Ok I added a dedicated section in the doc for both annotated types and took your remarks into account except the subclass thing (cf my remark). I feel like it's ready for another review :)

RobertCraigie · 2021-01-11T05:24:01Z

This looks great, I'll be putting it to good use, thanks!

However there is an issue that this does not respect subclassing, for example, the following will raise a ValidationError that the name field is missing.

class OptionalUser(TypedDict, total=False):
    name: str

class User(OptionalUser):
    id: int

class Model(BaseModel):
    user: User

Model(user={'id': 1})

I had a quick look to try and fix this issue but couldn't find a complete solution.

There are __required_keys__ and __optional_keys__ attributes but they are not guaranteed to be present on all versions and we cannot calculate the optional / required keys ourselves by traversing the mro as it is discarded upon class creation.

If there is no universal solution for this issue then __required_keys__ and __optional_keys__ should be respected if they are present.

PrettyWood · 2021-01-11T09:20:48Z

Hi @RobertCraigie
First let me thank you for testing the code! It's so valuable to have feedback before a feature is actually merged.
You're right I didn't get that. My first implementation used __required_keys__ indeed but I got issues with the CI with older python versions that didn't support it.
I guess we could simpy use TypedDict from typing_extensions for python 3.9- if possible, which implements __required_keys__ and __optional_keys__ for python 3.6+. But it means the code will behave differently with and without the extension, which I don't like. Maybe we could add a warning in this case?

EDIT: I added a commit. Feedback welcome :)

pydantic/validators.py

RobertCraigie · 2021-01-11T16:49:46Z

Glad I could help!

I would also like to be able to explicitly forbid passing extra keys, is this feasible?

PrettyWood · 2021-01-12T18:00:34Z

@RobertCraigie Sure easy peasy ! :) Just needed to pass down the config like I did for dataclass.
I didn't pass it down to NamedTuple though as I feel like it makes no sense. It can probably be done in a future PR if needed.

PrettyWood · 2021-01-21T21:53:08Z

pydantic/annotated_types.py

+
+if TYPE_CHECKING:
+
+    class TypedDict(Dict[str, Any]):


We could use typing_extensions.TypedDict for python < 3.9 or typing.TypedDict for 3.9+ but we would need to add typing_extensions in linting requirements.

PrettyWood · 2021-01-21T21:54:25Z

requirements.txt

@@ -4,5 +4,5 @@ Cython==0.29.21;sys_platform!='win32'
 devtools==0.6.1
 email-validator==1.1.2
 dataclasses==0.6; python_version < '3.7'
-typing-extensions==3.7.4.1; python_version < '3.8'
+typing-extensions==3.7.4.3; python_version < '3.9'


Had to bump this to have the good version of TypedDict for python < 3.9

Thorbenl · 2021-02-05T12:09:55Z

Is there any timeframe for when we can expect this to get released? :)

samuelcolvin

this is going to be f***ing awesome. Can't wait for it.

Just a few small things.

samuelcolvin · 2021-02-11T18:13:52Z

docs/usage/types.md

 `typing.Dict`
 : see [Typing Iterables](#typing-iterables) below for more detail on parsing and validation

+`subclass of typing.TypedDict`


I would still prefer to remove subclass of and instead be explicity when we mean the actual type, like Enum.

docs/usage/types.md

samuelcolvin · 2021-02-11T18:18:29Z

pydantic/__init__.py

@@ -16,6 +17,9 @@
 # WARNING __all__ from .errors is not included here, it will be removed as an export here in v2
 # please use "from pydantic.errors import ..." instead
 __all__ = [
+    # annotated types utils
+    'create_model_from_namedtuple',


yes agree, perhaps we might want create_model_from which takes any of the three types, but we definitely want these standalone methods for performance.

Or we could go even further and extend create_model to take a single dataclass, named tuple or typed dict instead of fields?

pydantic/annotated_types.py

pydantic/validators.py

PrettyWood · 2021-02-11T20:35:42Z

@samuelcolvin Feedback taken except for the rewording on types.md.
I'm not really sure how to explain the Enum for example.
Would you mind update types.md for me please ? I'm sure it will be clearer than what I would come by with

samuelcolvin · 2021-02-12T00:22:49Z

@samuelcolvin Feedback taken except for the rewording on types.md.
I'm not really sure how to explain the Enum for example.
Would you mind update types.md for me please ? I'm sure it will be clearer than what I would come by with

Don't worry about it, the whole file needs rethinking, what's there is fine.

I'll merge tomorrow.

samuelcolvin · 2021-02-13T10:06:26Z

🎉 this is great, thank you so much.

PrettyWood force-pushed the feat/support-named-tuple branch from 16d53eb to da3d88f Compare December 23, 2020 01:03

PrettyWood changed the title ~~feat: add support for namedtuple type~~ feat: add support for NamedTuple and TypedDict types Dec 23, 2020

feat: add support for NamedTuple and TypedDict types

0cbe31f

PrettyWood force-pushed the feat/support-named-tuple branch from da3d88f to 0cbe31f Compare December 23, 2020 01:41

PrettyWood added 2 commits December 23, 2020 03:15

support total=False

c5e236a

tests: fix ci with python < 3.8 without typing-extensions

0f5cb59

PrettyWood force-pushed the feat/support-named-tuple branch from d472bf0 to 9a2dba9 Compare December 23, 2020 02:19

chore: improve mypy

de54b65

PrettyWood force-pushed the feat/support-named-tuple branch from 9a2dba9 to de54b65 Compare December 23, 2020 02:29

PrettyWood marked this pull request as ready for review December 23, 2020 02:34

samuelcolvin reviewed Jan 1, 2021

View reviewed changes

PrettyWood added 4 commits January 2, 2021 17:06

Merge branch 'master' into feat/support-named-tuple

1c1b499

chore: @samuelcolvin remarks

986feda

refactor: move tests in dedicated file

a7bf673

docs: add annotated types section with examples

6764357

PrettyWood added 3 commits January 11, 2021 11:18

feat: support properly required and optional fields

62b4feb

chore(deps-dev): bump typing_extensions

7b45015

docs: add a note for typing_extensions

7368bf9

RobertCraigie reviewed Jan 11, 2021

View reviewed changes

pydantic/validators.py Outdated Show resolved Hide resolved

chore: update message to be more accurate

63f4a1e

PrettyWood force-pushed the feat/support-named-tuple branch from d3477f1 to 63f4a1e Compare January 12, 2021 17:30

feat: pass down config to created models

ebfd440

PrettyWood force-pushed the feat/support-named-tuple branch from 9376bb5 to ebfd440 Compare January 12, 2021 18:03

feat: add util methods to create model from TypedDict or NamedTuple

4cb9dbd

PrettyWood force-pushed the feat/support-named-tuple branch from c68515d to 8221ce2 Compare January 17, 2021 22:27

PrettyWood added 3 commits January 17, 2021 23:36

refactor: rename utils to match the rest

0a250e1

chore: update change

3361496

docs: add section for create_model_from_{namedtuple,typeddict}

d737f07

PrettyWood force-pushed the feat/support-named-tuple branch from 10941ab to d737f07 Compare January 17, 2021 22:56

PrettyWood added the ready for review label Jan 19, 2021

PrettyWood added 2 commits January 21, 2021 22:50

refactor: rename typed_dict/named_tuple

3fe8ff9

feat: support schema with TypedDict

08868ab

PrettyWood force-pushed the feat/support-named-tuple branch from ae66729 to ee7033a Compare January 21, 2021 21:50

PrettyWood commented Jan 21, 2021

View reviewed changes

PrettyWood added 2 commits January 22, 2021 11:08

feat: support schema for NamedTuple

567bf4d

feat: add json support for NamedTuple

9481776

PrettyWood force-pushed the feat/support-named-tuple branch from ee7033a to 9481776 Compare January 22, 2021 10:10

samuelcolvin reviewed Feb 11, 2021

View reviewed changes

PrettyWood added 3 commits February 11, 2021 21:14

chore: rewording

1c6f369

refactor: use parse_obj

5db83e9

fix: add check for max items in tuple

4da1475

docs: separate typing.NamedTuple and collections.namedtuple

1e60f5a

samuelcolvin merged commit c314f5a into pydantic:master Feb 13, 2021

PrettyWood deleted the feat/support-named-tuple branch February 13, 2021 10:48

PrettyWood mentioned this pull request Feb 13, 2021

chore(deps-dev): fix typing-extensions version #2361

Merged

4 tasks

This was referenced Mar 6, 2021

Bump pydantic from 1.7.3 to 1.8.1 in /backend Vaarna/vaarna#139

Merged

Bump pydantic from 1.6.1 to 1.8.1 in /emmet-core rkingsbury/emmet#25

Closed

dependabot bot mentioned this pull request Mar 17, 2021

Bump pydantic[dotenv] from 1.7.3 to 1.8.1 enchant97/web-portal#1

Merged

KyleKing mentioned this pull request Oct 6, 2021

TypeError: TypedDict does not support instance and class checks beartype/beartype#55

Closed

prrao87 mentioned this pull request Jul 7, 2023

Pydantic v2 inheritance from TypedDict is flagged as error, even though it's valid Python microsoft/pylance-release#4584

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for `NamedTuple` and `TypedDict` types #2216

feat: add support for `NamedTuple` and `TypedDict` types #2216

PrettyWood commented Dec 22, 2020 •

edited

Loading

codecov bot commented Dec 22, 2020 •

edited

Loading

samuelcolvin left a comment

samuelcolvin Jan 1, 2021

PrettyWood Jan 2, 2021

samuelcolvin Feb 11, 2021

samuelcolvin Jan 1, 2021

samuelcolvin Jan 1, 2021

PrettyWood Jan 1, 2021 •

edited

Loading

samuelcolvin Jan 1, 2021

PrettyWood Jan 2, 2021

PrettyWood Jan 2, 2021

PrettyWood commented Jan 2, 2021

RobertCraigie commented Jan 11, 2021

PrettyWood commented Jan 11, 2021 •

edited

Loading

RobertCraigie commented Jan 11, 2021 •

edited

Loading

PrettyWood commented Jan 12, 2021

PrettyWood Jan 21, 2021

PrettyWood Jan 21, 2021

Thorbenl commented Feb 5, 2021 •

edited

Loading

samuelcolvin left a comment

samuelcolvin Feb 11, 2021

samuelcolvin Feb 11, 2021

PrettyWood commented Feb 11, 2021

samuelcolvin commented Feb 12, 2021 •

edited

Loading

samuelcolvin commented Feb 13, 2021

	`subclass of typing.NamedTuple (or collections.namedtuple)`
	`typing.NamedTuple` (or `collections.namedtuple`)

feat: add support for NamedTuple and TypedDict types #2216

feat: add support for NamedTuple and TypedDict types #2216

Conversation

PrettyWood commented Dec 22, 2020 • edited Loading

Change Summary

Related issue number

Checklist

codecov bot commented Dec 22, 2020 • edited Loading

Codecov Report

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PrettyWood Jan 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PrettyWood commented Jan 2, 2021

RobertCraigie commented Jan 11, 2021

PrettyWood commented Jan 11, 2021 • edited Loading

RobertCraigie commented Jan 11, 2021 • edited Loading

PrettyWood commented Jan 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thorbenl commented Feb 5, 2021 • edited Loading

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PrettyWood commented Feb 11, 2021

samuelcolvin commented Feb 12, 2021 • edited Loading

samuelcolvin commented Feb 13, 2021

feat: add support for `NamedTuple` and `TypedDict` types #2216

feat: add support for `NamedTuple` and `TypedDict` types #2216

PrettyWood commented Dec 22, 2020 •

edited

Loading

codecov bot commented Dec 22, 2020 •

edited

Loading

PrettyWood Jan 1, 2021 •

edited

Loading

PrettyWood commented Jan 11, 2021 •

edited

Loading

RobertCraigie commented Jan 11, 2021 •

edited

Loading

Thorbenl commented Feb 5, 2021 •

edited

Loading

samuelcolvin commented Feb 12, 2021 •

edited

Loading