Add PyDantic v2 support

[loadams]

Pydantic completed its update to V2, they released a tool called bump-pydantic that might do this for you, this PR tries that.

[mrwyattii]

Note: Will need to merge deepspeedai/DeepSpeed-MII#213 before this one

[ringohoffman]

Note: Will need to merge microsoft/DeepSpeed-MII#213 before this one

@mrwyattii It looks like the tests for DeepSpeed and DeepSpeed-MII are interdependent on each other and also both depend on pydantic, so updating pydantic in one will make it incompatible with the other. What is the plan for continuing this upgrade?

[ringohoffman]

Here is LangChain's pydantic migration plan: https://github.com/langchain-ai/langchain/discussions/9337

They are able to support both pydantic v1 and v2. Perhaps DeepSpeed could follow suit? It's still unclear to me how to perform any kind of update when DeepSpeed and DeepSpeed-MII are interdependent.

[loadams]

Here is LangChain's pydantic migration plan: langchain-ai/langchain#9337

They are able to support both pydantic v1 and v2. Perhaps DeepSpeed could follow suit? It's still unclear to me how to perform any kind of update when DeepSpeed and DeepSpeed-MII are interdependent.

Hi @ringohoffman - DeepSpeed and DeepSpeed-MII do rely on each other for tests, but what we would do, to update to pydantic v2 is to push the update to DeepSpeed-MII first (once we've run tests with both branches updated, by changing this on the DeepSpeed-MII side for example). Then the DeepSpeed PR should be able to pass tests, and we can enable DeepSpeed-MII to use the DeepSpeed master branch again.

Does that help answer your question for how we will merge these?

We haven't looked into supporting both pydantic v1 and v2 simultaneously, we've had a fairly large amount of work with just adopting v2, but we will discuss internally and let you know on supporting both.