fix: Downgrade to marshmallow<4

[amotl]

Problem

With Marshmallow 4, just released, Superset 4 trips like:

TypeError: Field.__init__() got an unexpected keyword argument 'minLength'

Solution

Pin dependency to marshmallow<4.

References

• Make Superset compatible with marshmallow>=4.0.0 #33162

[korbit-ai]

I've completed my review and didn't find any issues.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

[mistercrunch]

Let's always put a comment above when setting an upper bound that's not based on pure semver assumptions, something like "marshmallow>=4 has shown issues related to 'minLength'", ...

[amotl]

Thank you. I didn't see such comments on other spots, but I've just added it, effectively linking to the corresponding issue on GitHub.

[mistercrunch]

LGTM - curious about your internal release process that lead to that issue, do you run your own uv pip compile --upgrade process internally? Or maybe you have your own snyk/dependabot-type setup in place and upgrade certain things?

In any case, would be great to centralize the dependency management and upgrades in the main repo as much as possible. Could certainly use help with the process if you wanted to get involved in dep management.

[amotl]

@mistercrunch: Thanks for merging.

curious about your internal release process that lead to that issue

You can inspect the testing rig which flagged this regression through its nightly CI runs over here. HTH, otherwise feel free to ask.

• https://github.com/crate/cratedb-examples/tree/main/application/apache-superset
• https://github.com/crate/cratedb-examples/actions/workflows/application-apache-superset.yml

[amotl]

Hi again. This still needs a release, in order to resolve the issue completely for average downstream users.

• Make Superset compatible with marshmallow>=4.0.0 #33162 (comment)

/cc @mistercrunch, @rusackas

[amotl]

In any case, would be great to centralize the dependency management and upgrades in the main repo as much as possible. Could certainly use help with the process if you wanted to get involved in dep management.

It looks like this might still be an unresolved issue? Maybe I can free up some time for that, as I guess it might be a one-time action? On the other hand, I strongly believe Superset has its intriciacies, so I certainly would need to learn a significant amount of your current philosophy and about your most prominent pain points in order to possibly improve things thereafter, if at all.

[mistercrunch]

This could be pull in our various active LTS branches though release managers are already pretty busy with everything and LTS-done-right is difficult, and I believe it's an area where only committers can do certain things (people without write access to the repo simply cannot push a release through for good reasons). Tagging the folks that have been doing release management over the past few releases here: @michael-s-molina @sadpandajoe

On your side though, wondering if your build process can build upon Superset's pinned dependencies. Say if you use pip compile or uv pip compile for you project, you may be able to add a constraint on our pinned deps in your build process. Notice the -c in this script that compiles our deps here -> https://github.com/apache/superset/blob/master/scripts/uv-pip-compile.sh#L28 as you may be able to do something similar. Now it may conflict with with core app's dependencies, but at least you would integrate our pinned requirements along yours, and have to resolve them pre-build.

Superset is/was designed to be in a fairly standalone mode, meaning we're assuming your deploying the Superset app, not another app that happens to be integrating Superset. The way I've seen Superset be a dependency of a wider project is typically by creating a wrapper/shell around it while bringing the Superset repo as a git submodule inside your project or repo. Typically the wrapper will have CI/cloud-provisioning things like terraform/docker and whatever is needed to integrate Superset in your organization's typical build process and requirements. Clearly there's a lot of different constraints for different organizations (everyone has different reqs on how they deploy services internally), and it's been difficult to share the various building blocks and reusable components that caters to such a wide variety of ways of doing things ... Even things like the Helm chart or Dockerifle often needs to be forked for things to work. Similar with terraform, usually there are more things in a typical terraform package that are related to your org's specificities as there are things related to "how Superset should be deployed".

Anyhow, open to helping out around it and offering the buildings blocks that are useful to most.

One thing I'd like to see if "reference implementations" OUTSIDE our repos showing various ways around how this can be done in different places...

[sadpandajoe]

I think I had already cherried this as part of 4.1.3rc3, so it should be there already.

[zhangyijun]

Lastest 5.0.0 version release, this bug is still exists. pip install will install marshmallow-4.0.0. CAN NOT work anything untill mannually run pip install -U marshmallow==3.26.1.

[betodealmeida]

Pinned it here: #33978