Apache Superset Make Superset compatible with marshmallow>=4.0.0

Bug description

On a fresh install of superset, since marshmallow 4.0.0 released https://pypi.org/project/marshmallow/, superset fails to init due to an error validating parameters

It is possible to work around this by manually installing marshmallow at 3.26.1

pip3 install marshmallow==3.26.1

It looks like the version is being set in requirements/development.txt, so I'm unsure as to where the version update is occurring

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.9

Node version

Browser

Not applicable

Additional context

File "/home/ec2-user/.local/lib/python3.9/site-packages/superset/charts/schemas.py", line 621, in ChartDataProphetOptionsSchema periods = fields.Integer( File "/home/ec2-user/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 977, in init super().init(as_string=as_string, kwargs) File "/home/ec2-user/.local/lib/python3.9/site-packages/marshmallow/fields.py", line 924, in init super().init(kwargs) TypeError: init() got an unexpected keyword argument 'min'

Checklist

[x] I have searched Superset docs and Slack and didn't find a solution to my problem.
[x] I have searched the GitHub issue tracker and didn't find a similar bug report.
[x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Comment From: dosubot[bot]

Hello @MattBodey! I'm Dosu, a bot here to help you with bugs, answer your questions, and guide you on your journey to becoming a contributor. I'm ready to assist while you wait for a human maintainer. Let’s tackle this together!

The issue you're encountering is due to a compatibility problem between Superset and Marshmallow 4.0.0. Specifically, the fields.Integer class in Marshmallow 4.0.0 no longer supports the min keyword argument, which is causing the error during Superset's initialization.

To resolve this issue, you can downgrade Marshmallow to version 3.26.1, where the min keyword argument is still supported. You can do this by running the following command:

pip3 install marshmallow==3.26.1

This should allow Superset to initialize without errors. It seems that the version of Marshmallow is specified in requirements/development.txt, but there might be an override or update happening elsewhere in the setup process. You may want to check other configuration files or scripts that could be specifying a different version of Marshmallow.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

Comment From: rusackas

Is there somewhere we need to pin the dependency? It looks like Superset already installs 3.26.1. I'm not sure about the details of the failure, but I know we also use marshmallow-sqlalchemy which doesn't seem to support marshmallow 4.0 yet if I'm understanding correctly.

Comment From: ColinMaudry

I followed the PyPi install instructions, and running pip install marshmallow==3.26.1 just before superset db upgrade did the trick.

Comment From: amotl

Hi there,

we are running a little integration test suite against Apache Superset and CrateDB, in order to validate both work together well. Two days ago, CI apparently started tripping on it.

TypeError: Field.__init__() got an unexpected keyword argument 'minLength'

Do you think we need to adjust a few details, or is it just about marshmallow-4.0.0, which has been released two days ago, as suggested above?

With kind regards, Andreas.

References

https://github.com/crate/cratedb-examples/tree/main/application/apache-superset
https://github.com/crate/cratedb-examples/actions/workflows/application-apache-superset.yml
https://github.com/crate/cratedb-examples/actions/runs/14545171337/job/40820574564#step:7:352

Comment From: ColinMaudry

@amotl I guess either

superset stop using minLength and generally adapts to Marshallow v4
or superset ensures Marshmallow v3.26.1 is pinned all the way
or Marshmallow readds minLength and others

In the meantime we must force Marshmallow version. How? Depends on how you install superset.

Comment From: amotl

Hi. Thank you very much.

In the meantime we must force Marshmallow version. How? Depends on how you install superset.

I don't completely understand this, yet. Isn't it possible to just add a corresponding version pinning constraint to Superset's main dependencies to handle the situation well, until Superset will be compatible with Marshmallow 4?

Comment From: ColinMaudry

Marshmallow is already pinned to the right version in development.txt, via marshmallow-sqlalchemy module requirements, but somehow v4.0.0 gets installed in the end.

Comment From: ColinMaudry

Depends on how you install Superset.

If you install it via pip install superset, Kubernetes or download the docker image, you won't force Marshmallow version the same way. I'm not fluent in Docker, even less in Kubernetes, so I went the PyPi way to fix it.

Comment From: amotl

Hi Colin, thanks for showing the right dependency pinning in development.txt. The same, even more important, should also be conducted for the runtime dependencies, right? Otherwise, it will leave the average user stranded like outlined in this thread.

Comment From: amotl

It looks like the dependency of Marshmallow is also pinned per base.txt ¹. Do you have an idea why it isn't used at runtime?

Comment From: amotl

Hi again. We submitted a patch to the 4.1 branch, in order to improve the situation for Superset 4.

GH-33216

Comment From: turingnixstyx

Hi is this fixed? Or can I be assigned this one?

Comment From: amotl

Hi. I think it's "in progress". I've just refreshed the patch GH-33216.

Comment From: amotl

Hi @mistercrunch. It's just a humble question about your plans: Is there a bugfix release for Apache Superset 4 scheduled, which would include the fix GH-33216? Or are other patches pending that want to be bundled into the same release?