Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pydantic for Aeroval setupclasses.py #1017

Merged
merged 62 commits into from
Apr 3, 2024
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
5597317
playing with pydantic for aeroval setupclasses
lewisblake Feb 29, 2024
d5245d7
continuing pydantic for setupclasses
lewisblake Mar 1, 2024
9a838b3
Merge branch 'main-dev' into pydantic-setupclasses
lewisblake Mar 7, 2024
eb0c309
test_setupclasses passing
lewisblake Mar 7, 2024
8c18795
rm setupclasses2.py
lewisblake Mar 7, 2024
b1e7e7f
rm unused imports
lewisblake Mar 7, 2024
ad50d4d
from __future__ import annotations
lewisblake Mar 7, 2024
0d0cda1
WIP EvalSetup
lewisblake Mar 8, 2024
aae4521
ExperimentOutput: json_basedir WIP
lewisblake Mar 8, 2024
52659d5
go back in ProjectOutput
lewisblake Mar 10, 2024
594836d
Attempting a pydantic ProjectOutput
lewisblake Mar 10, 2024
5923437
Attempting to remove DIrLoc from setupclasses
lewisblake Mar 10, 2024
339d450
allow io_aux_file to be a string
lewisblake Mar 10, 2024
5d947ab
using pydantic ValidationError in test_setup
lewisblake Mar 10, 2024
509625b
ObsCollection needs init?
lewisblake Mar 11, 2024
ea9e08d
use field_validators for Obs and Model Collection
lewisblake Mar 11, 2024
3f99eb2
go back tol old experiment_output
lewisblake Mar 11, 2024
55b5319
test_experiment_output don't use dict accessing
lewisblake Mar 11, 2024
5bc2024
rm _invalid
lewisblake Mar 11, 2024
1b304e1
args for EvalSetuip in test__processing_base
lewisblake Mar 11, 2024
6bab371
Merge branch 'main-dev' into pydantic-setupclasses
lewisblake Mar 15, 2024
5acb6b9
Merge branch 'main-dev' into pydantic-setupclasses
lewisblake Mar 16, 2024
781d7fc
time_cfg seems to be working
lewisblake Mar 16, 2024
03c0d7f
converting to json: Unexpected extra items present in tuple
lewisblake Mar 16, 2024
a93f0ff
type annotate tuple as any length- ColocationSetup not serializable
lewisblake Mar 16, 2024
ce7e410
ObsCollection and ModelCollection must be made JSON serializable
lewisblake Mar 16, 2024
095b3bd
custom serializers for BaseModel-based feilds in EvalSetup
lewisblake Mar 18, 2024
bd3bfb8
must specify arguments to eval_config before EvalSetup
lewisblake Mar 18, 2024
aa8df1f
Merge branch 'main-dev' into pydantic-setupclasses
lewisblake Mar 19, 2024
044a55d
OutputPath in EvalSetup not being set correctly
lewisblake Mar 19, 2024
44fae29
Merge branch 'main-dev' into pydantic-setupclasses
lewisblake Mar 24, 2024
a27e6d0
comment out AsciiFileLoc2
lewisblake Mar 24, 2024
d2cdeed
OutputPaths needed extra keys
lewisblake Mar 24, 2024
088f728
Make ProjectInfo and ExperimentInfo BaseModels
lewisblake Mar 24, 2024
abb0009
json_basedir computed property WIP
lewisblake Mar 24, 2024
d7a8ff1
json_basedir WIP
lewisblake Mar 24, 2024
cefc461
basedirs just needed a validator
lewisblake Mar 25, 2024
27b6196
rm dead code
lewisblake Mar 25, 2024
c62ea2b
Merge branch 'main-dev' into pydantic-setupclasses
lewisblake Mar 28, 2024
2d8e94a
assign ColocationSetup defaults if not given
lewisblake Mar 30, 2024
19dd2e1
test_reanalyse_existing
lewisblake Mar 30, 2024
a453d5b
clean up and linters
lewisblake Mar 30, 2024
dece69f
unrelated MEP path typo
lewisblake Mar 30, 2024
abe39cc
rm unused imports
lewisblake Mar 30, 2024
c258182
passing local tests
lewisblake Mar 30, 2024
cd93e96
linters
lewisblake Mar 30, 2024
58d7a6c
isort
lewisblake Mar 30, 2024
a371b0c
rm use of AsciiFileLoc
lewisblake Mar 30, 2024
9394bfb
linting
lewisblake Mar 30, 2024
e9971f0
rm comment
lewisblake Mar 30, 2024
e25539f
bring back model_only_stats
lewisblake Mar 30, 2024
f3956d9
rm dataclasses import
lewisblake Mar 31, 2024
ed92208
PEP 604
lewisblake Apr 2, 2024
f28a5c9
alvaro feedback
lewisblake Apr 2, 2024
d9299e3
attempt Self import based on python version
lewisblake Apr 2, 2024
67ee0f5
correct sys.version_info <3.11
lewisblake Apr 2, 2024
b6e531f
more alvaro updates
lewisblake Apr 3, 2024
00e8c67
linters
lewisblake Apr 3, 2024
49aeea5
typing-extensions for python <3.11
lewisblake Apr 3, 2024
db0f1f5
typing_extensions import improvements
lewisblake Apr 3, 2024
220b2e1
try importing Annotated from typing
lewisblake Apr 3, 2024
3d1e1c4
isort
lewisblake Apr 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions pyaerocom/aeroval/setupclasses.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
from __future__ import annotations

import logging
import os
import sys
from getpass import getuser
from pathlib import Path
from typing import Literal, Self

if sys.version_info < (3, 10):
from typing_extensions import Self

Check warning on line 8 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L8

Added line #L8 was not covered by tests
else:
from typing import Self

from typing import Literal
avaldebe marked this conversation as resolved.
Show resolved Hide resolved

from pydantic import (
BaseModel,
Expand Down Expand Up @@ -175,7 +180,7 @@
round_floats_precision: int | None = None

if round_floats_precision:
set_float_serialization_precision(round_floats_precision)

Check warning on line 183 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L183

Added line #L183 was not covered by tests


class TimeSetup(BaseModel):
Expand Down Expand Up @@ -338,7 +343,7 @@
subset_dict = {k: self.model_extra[k] for k in time_cfg_keys}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is time_cfg_keys really needed?
How is this different than:

    def time_cfg(self) -> TimeSetup:
        if not hasattr(self, "model_extra"):
            return TimeSetup()
        
        model_args = {key: val for key, val in self.model_extra.items() if key in TimeSetup.model_fields}
        TimeSetup(**model_args)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is cleaner. Changed.

return TimeSetup(**subset_dict)
else:
return TimeSetup()

Check warning on line 346 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L346

Added line #L346 was not covered by tests

@computed_field
@property
Expand All @@ -346,8 +351,8 @@
if hasattr(self, "model_extra") and bool(
cfg_extra_keys := set(self.model_extra) & set(ModelMapsSetup.model_fields)
):
subset_dict = {k: self.model_extra[k] for k in cfg_extra_keys}
return ModelMapsSetup(**subset_dict)

Check warning on line 355 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L354-L355

Added lines #L354 - L355 were not covered by tests
else:
return ModelMapsSetup()

Expand All @@ -371,7 +376,7 @@
subset_dict = {k: self.model_extra[k] for k in cfg_extra_keys}
return EvalRunOptions(**subset_dict)
else:
return EvalRunOptions()

Check warning on line 379 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L379

Added line #L379 was not covered by tests

@computed_field
@property
Expand All @@ -382,7 +387,7 @@
subset_dict = {k: self.model_extra[k] for k in cfg_extra_keys}
return StatisticsSetup(**subset_dict)
else:
return StatisticsSetup(weighted_stats=True, annual_stats_constrained=False)

Check warning on line 390 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L390

Added line #L390 was not covered by tests

##################################
## Non-BaseModel-based attributes
Expand Down Expand Up @@ -420,7 +425,7 @@
@field_validator("obs_cfg")
def validate_obs_cfg(cls, v):
if isinstance(v, ObsCollection):
return v

Check warning on line 428 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L428

Added line #L428 was not covered by tests
return ObsCollection(v)

@field_serializer("obs_cfg")
Expand All @@ -432,7 +437,7 @@
@field_validator("model_cfg")
def validate_model_cfg(cls, v):
if isinstance(v, ModelCollection):
return v

Check warning on line 440 in pyaerocom/aeroval/setupclasses.py

View check run for this annotation

Codecov / codecov/patch

pyaerocom/aeroval/setupclasses.py#L440

Added line #L440 was not covered by tests
return ModelCollection(v)

@field_serializer("model_cfg")
Expand Down
Loading