[WIP] Upgrade to Pydantic 2 #632
Conversation
sqlmodel/main.py
Outdated
| List, | ||
| Mapping, | ||
| NoneType, |
There was a problem hiding this comment.
NoneType has been moved to types in Python 3.10+.
https://docs.python.org/3/library/types.html#types.NoneType
sqlmodel/main.py
Outdated
|
|
||
| _TSQLModel = TypeVar("_TSQLModel", bound="SQLModel") | ||
|
|
||
|
|
||
| class SQLModel(BaseModel, metaclass=SQLModelMetaclass, registry=default_registry): |
There was a problem hiding this comment.
In the test tests\test_tutorial\test_delete, the following error occurred: AttributeError: 'Hero' object has no attribute '__pydantic_extra__'.
You can solve this problem by adding the following code to SQLmodel:
def __new__(cls, *args: Any, **kwargs: Any) -> Any:
new_object = super().__new__(cls)
object.__setattr__(new_object, "__pydantic_fields_set__", set())
return new_object
|
I got almost everything working except for Open API. It provides optional for the OpenAPI if you use a table as a request or response type. While this isn't correct in the strict sense, it is correct for Pydantic as empty initialisations need to be supported. I tried a couple of other ways (creating a second class for the optional initialisations or trying to override the OpenAPI generation) but none of them worked out. For now I commented the OpenAPI assertions, but @tiangolo will need to decide how to handle it. As he wrote FastAPI I'm sure he'll have better ideas 😄 I just hope I can give him some inspiration with this. |
|
the following test not pass: from sqlmodel import SQLModel, Field
class Article(SQLModel, table=True):
name : str | None = Field( description='the name of article')Error: |
|
@honglei try I think just one more case must be checked somewhere. |
I made a pull request to support it. mbsantiago#1 |
|
Hi! Just checking if there's still something to be done for this PR, more than willing to help out! |
|
Hey! I'm also willing to help out. |
|
@ma-ts @ogabrielluiz You can help by testing the PR in your projects, and reporting how well it works. |
|
@honglei Installed with poetry using |
|
@ogabrielluiz @honglei service_category: ServiceCategory = await self.get_object_or_404(id, company.id, async_db)
new_service_category = service_category.model_copy(update={'id': None})Use this test: def test_model_copy(clear_sqlmodel):
class Hero(SQLModel, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
name: str
secret_name: str
age: Optional[int] = None
hero = Hero(name="Deadpond", secret_name="Dive Wilson", age=25)
engine = create_engine("sqlite://")
SQLModel.metadata.create_all(engine)
with Session(engine) as session:
session.add(hero)
session.commit()
session.refresh(hero)
db_hero = session.get(Hero, hero.id)
copy = db_hero.model_copy(update={"name": "Deadpond Copy"})
assert copy.name == "Deadpond Copy" and \
copy.secret_name == "Dive Wilson" and \
copy.age == 25 |
fixed honglei#1 |
|
I wonder if @tiangolo will notice this sentence |
|
Not strictly a SQLModels problem, but if you try and use the current SQLModels 0.0.8 with FastAPI 0.103.0 where you've targeted fastapi[all] conflicts are created because of the pydantic version pins of SQLModel, as pydantic-settings requires a much later version than the max pinned. Just like to add a +1 for this PR being quite important to get merged. |
|
topic is very long, guys, possible to use with this PR? |
@MatsiukMykola I'm using it in my project. Everything's great |
|
The type |
|
@ikreb7 @50Bytes-dev support the following in my fork(https://github.com/honglei/sqlmodel), but how to change test.yml to do http: HttpUrl = Field(max_length=250)
email: EmailStr
name_email : NameEmail = Field( max_length=50)
import_string:ImportString = Field(max_length=200, min_length=100) |
@honglei pydantic = { version = "^2.1.1", extras = ["email"] } |
|
@50Bytes-dev thanks! |
|
@honglei what's the status of your fork? Can we use it till a stable version of SQLmodel is released? |
|
I used in my projects, it works fine.
wojciech-mazur-mlg ***@***.***> 于 2023年9月18日周一 下午7:30写道：
… @honglei <https://github.com/honglei> what's the status of your fork? Can
we use it till a stable version of SQLmodel is released?
—
Reply to this email directly, view it on GitHub
<#632 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHW5AVSOKK64F5EAV4P7LTX3AWDZANCNFSM6AAAAAA27PJTWM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I have tried it out a bit and it seems to work. Some other people are also saying it works. But I would say it's not official until @tiangolo approves it or makes his own version. I also won't be trying everything out to be honest 😅 |
|
After I updated, the "alias" stopped working The validation in create_user fails with "password ", but works with "password_hash" |
|
@BoManev @AntonDeMeester |
|
@honglei Thank you for the fast reply. In that case should I create a class using pydantic BaseModel for validation of my inputs and then map it with model_validate to my SQLModel. I also found this #374 EDIT: Generally, how to handle data validation when field names change and data needs to be modified. In my case I receive "password", but internally I want my types to carry "password_hash". In Rust, I can specify Into/To/From traits and convert between types. Can I do something similar with pydantic/SQLModel, define the conversion of 1 data schema to other. |
|
Hey folks, |
There was a problem hiding this comment.
Really looking forward to seeing these changes go in - carrying out an early preview as I'm already using the latest versions. One thing I've observed, I've been struggling a little with using the async db setup in the context of pulling in additional relationships. All of the other routes work fine from this tutorial, but when I call read_hero I get the following error:
fastapi.exceptions.ResponseValidationError: 1 validation errors:
{'type': 'get_attribute_error', 'loc': ('response', 'team'), 'msg': "Error extracting attribute: MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/
e/20/xd2s)", 'input': Hero(secret_name='Batman', id=1, team_id=1, name='Bruce Wayne', age=42), 'ctx': {'error': "MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https:
//sqlalche.me/e/20/xd2s)"}, 'url': 'https://errors.pydantic.dev/2.4/v/get_attribute_error'}This only happens when the response_model is set to HeroReadWithTeam, as the expectation is that we'd want to include teams with the hero. Using HeroRead works fine.
There was a problem hiding this comment.
This is now resolved with the help of another post here. For relationships between models you may need to leverage the correct loading styles through the sa_relationship_kwargs property that is offered by SQLModel. For the curious this is how I've built up the tutorial with async in mind below.
In SQL Alchemy there's more reading here on the subject of lazy loading.
Really happy, so will crack on :)
from contextlib import asynccontextmanager
from typing import List, Optional
from fastapi import Depends, FastAPI, HTTPException, Query
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from sqlmodel import Field, Relationship, SQLModel
from sqlalchemy.sql.expression import select
from sqlmodel.ext.asyncio.session import AsyncSession
from config import config
class TeamBase(SQLModel):
name: str = Field(index=True)
headquarters: str
class Team(TeamBase, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
heroes: List["Hero"] = Relationship(back_populates="team", sa_relationship_kwargs={"lazy": "selectin"})
class TeamCreate(TeamBase):
pass
class TeamRead(TeamBase):
id: int
class TeamUpdate(SQLModel):
id: Optional[int] = None
name: Optional[str] = None
headquarters: Optional[str] = None
class HeroBase(SQLModel):
name: str = Field(index=True)
secret_name: str
age: Optional[int] = Field(default=None, index=True)
team_id: Optional[int] = Field(default=None, foreign_key="team.id")
class Hero(HeroBase, table=True):
id: Optional[int] = Field(default=None, primary_key=True)
team: Optional[Team] = Relationship(back_populates="heroes", sa_relationship_kwargs={"lazy": "joined"})
class HeroRead(HeroBase):
id: int
class HeroCreate(HeroBase):
pass
class HeroUpdate(SQLModel):
name: Optional[str] = None
secret_name: Optional[str] = None
age: Optional[int] = None
team_id: Optional[int] = None
class HeroReadWithTeam(HeroRead):
team: Optional[TeamRead] = None
class TeamReadWithHeroes(TeamRead):
heroes: List[HeroRead] = []
engine = create_async_engine(config.DB_HOST, echo=True, future=True)
async def init_db():
async with engine.begin() as conn:
await conn.run_sync(SQLModel.metadata.create_all)
async def get_session() -> AsyncSession:
async_session = async_sessionmaker(engine, class_=AsyncSession, expire_on_commit=False)
async with async_session() as session:
yield session
@asynccontextmanager
async def lifespan(app: FastAPI):
# on startup
await init_db()
yield
# on shutdown
await engine.dispose()
app = FastAPI(lifespan=lifespan)
@app.post("/heroes", response_model=HeroRead)
async def create_hero(*, session: AsyncSession = Depends(get_session), hero: HeroCreate):
db_hero = Hero.model_validate(hero)
session.add(db_hero)
await session.commit()
await session.refresh(db_hero)
return db_hero
@app.get("/heroes", response_model=List[HeroRead])
async def read_heroes(
*,
session: AsyncSession = Depends(get_session),
offset: int = 0,
limit: int = Query(default=100, lte=100),
):
statement = select(Hero).offset(offset).limit(limit)
heroes = await session.execute(statement)
return heroes.scalars().all()
@app.get("/heroes/{hero_id}", response_model=HeroReadWithTeam)
async def read_hero(*, session: AsyncSession = Depends(get_session), hero_id: int):
hero = await session.get(Hero, hero_id)
if not hero:
raise HTTPException(status_code=404, detail="Hero not found")
return hero
@app.patch("/heroes/{hero_id}", response_model=HeroRead)
async def update_hero(
*, session: AsyncSession = Depends(get_session), hero_id: int, hero: HeroUpdate
):
db_hero = await session.get(Hero, hero_id)
if not db_hero:
raise HTTPException(status_code=404, detail="Hero not found")
hero_data = hero.model_dump(exclude_unset=True)
for key, value in hero_data.items():
setattr(db_hero, key, value)
session.add(db_hero)
await session.commit()
await session.refresh(db_hero)
return db_hero
@app.delete("/heroes/{hero_id}")
async def delete_hero(*, session: AsyncSession = Depends(get_session), hero_id: int):
hero = await session.get(Hero, hero_id)
if not hero:
raise HTTPException(status_code=404, detail="Hero not found")
await session.delete(hero)
await session.commit()
return {"ok": True}
@app.post("/teams", response_model=TeamRead)
async def create_team(*, session: AsyncSession = Depends(get_session), team: TeamCreate):
db_team = Team.model_validate(team)
session.add(db_team)
await session.commit()
await session.refresh(db_team)
return db_team
@app.get("/teams", response_model=List[TeamRead])
async def read_teams(
*,
session: AsyncSession = Depends(get_session),
offset: int = 0,
limit: int = Query(default=100, lte=100),
):
statement = select(Team).offset(offset).limit(limit)
teams = await session.execute(statement)
return teams.scalars().all()
@app.get("/teams/{team_id}", response_model=TeamReadWithHeroes)
async def read_team(*, team_id: int, session: AsyncSession = Depends(get_session)):
team = await session.get(Team, team_id)
if not team:
raise HTTPException(status_code=404, detail="Team not found")
return team
@app.patch("/teams/{team_id}", response_model=TeamRead)
async def update_team(
*,
session: AsyncSession = Depends(get_session),
team_id: int,
team: TeamUpdate,
):
db_team = await session.get(Team, team_id)
if not db_team:
raise HTTPException(status_code=404, detail="Team not found")
team_data = team.model_dump(exclude_unset=True)
for key, value in team_data.items():
setattr(db_team, key, value)
session.add(db_team)
await session.commit()
await session.refresh(db_team)
return db_team
@app.delete("/teams/{team_id}")
async def delete_team(*, session: AsyncSession = Depends(get_session), team_id: int):
team = await session.get(Team, team_id)
if not team:
raise HTTPException(status_code=404, detail="Team not found")
await session.delete(team)
await session.commit()
return {"ok": True}
|
Joining the party here trying to update a fairly large data model the fork. One thing I am working on resolving and curious if anyone else has run into is an issue with model class inheritance producing SQLAlchemy errors regarding column reuse: My class TimestampModel(BaseModel):
created_at: datetime.datetime = sqlmodel.Field(
sa_column=sqlalchemy.Column(
sqlalchemy.DateTime(timezone=True),
server_default=sqlalchemy.func.statement_timestamp(),
nullable=False
)
)
updated_at: typing.Optional[datetime.datetime] = sqlmodel.Field(
sa_column=sqlalchemy.Column(
sqlalchemy.DateTime(timezone=True),
onupdate=sqlalchemy.func.statement_timestamp(),
nullable=True
)
)It looks like the |
|
always was interesting how pull to sa.Column - column comment based on Field Description? |
|
➕ on thanks for putting this PR up, my team is a few weeks into a new project and I'd hate to be stuck behind a major on 2 core libraries this early on, so this is huge. I'm running into a strange issue creating a one-to-many relationship (using this branch), any ideas would be appreciated. Here's the error: the related models: class Organization(SQLModel, table=True):
id: Optional[int] = sqlmodel.Field(default=None, primary_key=True)
venues: List["Venue"] = sqlmodel.Relationship(back_populates="organization")
@classmethod
async def list(cls, session: AsyncSession):
query = select(cls)
result_from_db = await session.execute(query)
collection = result_from_db.fetchall()
return collection
class Venue(SQLModel, table=True):
id: Optional[int] = sqlmodel.Field(default=None, primary_key=True)
organization_id: int = sqlmodel.Field(foreign_key="organization.id")
organization: Organization = sqlmodel.Relationship(back_populates="venues")and the calling functionality: class DatabaseConnection:
def __init__(self):
url = sqlalchemy.URL.create(*url_args)
self.async_engine = sqlalchemy.ext.asyncio.create_async_engine(
url, echo=True, future=True
)
async def get_async_session(self) -> AsyncSession:
async_session = sessionmaker(
bind=self.async_engine, class_=AsyncSession, expire_on_commit=False
)
async with async_session() as session:
yield session
db_session = DatabaseConnection().get_async_session
@router.get("/")
async def index(*, session=Depends(db_session)):
return await Organization.list(session=session)I've tried changing annotation types, explicitly passing an |
|
It looks like the both of the goals prior to migrating to sqlalchemy 2.0, and pydantic 2.0 have been reached (supporting the latest pre 2.0 versions of each). The next goal in the roadmap is migrating each to 2.0. Any update here? |
|
📝 Docs preview for commit 7ecbc38 at: https://5165ba05.sqlmodel.pages.dev |
|
📝 Docs preview for commit ab07514 at: https://398280da.sqlmodel.pages.dev |
|
@AntonDeMeester I have been using an older commit from this repo, however today I tried to update my dependencies and got the following errors. Snippet from my pyproject.toml The older commit from this PR is using |
Ye I've been working on getting both Pydantic v1 and v2 to work at the same time, I might have broken some things. I'll change the PR to use the old commit again and push my changes to a new PR on top of this one until both work. |
|
📝 Docs preview for commit d0ae3e8 at: https://ff83edad.sqlmodel.pages.dev |
|
@AntonDeMeester This commit still doesn't work |
AntonDeMeester
changed the title
|
@AntonDeMeester sqlmodel has drop support for sqlalchemy<2.0, maybe next release will drop support for pydantic v1, |
|
@AntonDeMeester how are you? I've been using your fork internally for a while now and it seems stable. Do you see this PR being merged this week? Thanks in advance |
|
Thank you for your work @AntonDeMeester! I included your commits in #722 (you can see your badge now says "Contributor" 😎 ), I changed a few extra things, more details in that PR. It's now released and available as SQLModel 0.0.14! 🎉 🌮 |
|
Not sure if replying to a closed PR is the way, but it seems that the issue raised by @50Bytes-dev about Below is a quote of the mentionned error:
|
|
This error 'TypeError: issubclass() arg 1 must be a class' seems not fixed. ( I use Pydantic 2.7 and SQLModel 0.0.16. ) So I use Pydantic 1.0.11 and SQLModel 0.08. If anyone has same problem, hope it helps. |
Upgrades to Pydantic 2, using the new pydantic models. Switches for v1 and v2 can be built later
Builds on #563 to upgrade SQL Alchemy
Still WIP, did some uglier hacks which could provide some problems.
Most of it is just changing code with different naming of Pydantic 2. However, the validation of empty initialisations is broken because that now happens in rust and there is no static
validate_modelanymore. For now I set defaults to all provided Fields is there are none, so that it will always inititialise, but that screws a bit with the logic. This is also annoying forcls.model_validateas it screws up SQL Alchemy.