This is my first time trying to deploy Superset, and I want to deploy it into a Kubernetes cluster. I'm following the guide on https://superset.apache.org/docs/installation/running-on-kubernetes and I believe I have done all the relevant config steps (I skipped the SMTP settings, System packages settings, Data sources, and Alerts and Reporting).

How to reproduce the bug

  1. Deploy the latest Helm chart (superset 0.8.10`) to Kubernetes with the following config (notice the placeholders):
values.yaml 
postgresql:
  postgresqlPassword: ${jsonencode(postgresql_password)}
configOverrides:
  secret: |
    SECRET_KEY = '${secret_key}'
  enable_oauth: |
    # This will make sure the redirect_uri is properly computed, even with SSL offloading
    ENABLE_PROXY_FIX = True

    from flask_appbuilder.security.manager import AUTH_OAUTH
    AUTH_TYPE = AUTH_OAUTH
    OAUTH_PROVIDERS = [
        {
            "name": "google",
            "icon": "fa-google",
            "token_key": "access_token",
            "remote_app": {
                "client_id": os.getenv("GOOGLE_KEY"),
                "client_secret": os.getenv("GOOGLE_SECRET"),
                "api_base_url": "https://www.googleapis.com/oauth2/v2/",
                "client_kwargs": {"scope": "email profile"},
                "request_token_url": None,
                "access_token_url": "https://accounts.google.com/o/oauth2/token",
                "authorize_url": "https://accounts.google.com/o/oauth2/auth",
                "authorize_params": {"hd": os.getenv("AUTH_DOMAIN", "")}
            },
        }
    ]

    # Map Authlib roles to superset roles
    AUTH_ROLE_ADMIN = 'Admin'
    AUTH_ROLE_PUBLIC = 'Public'

    # Will allow user self registration, allowing to create Flask users from Authorized User
    AUTH_USER_REGISTRATION = True

    # The default user self registration role
    AUTH_USER_REGISTRATION_ROLE = "Admin"
bootstrapScript: |
  #!/bin/bash
  pip install psycopg2==2.9.1 \
    authlib==1.2.0 \
    redis==3.2.1 \
    psycopg2-binary==2.9.6 \
    sqlalchemy-bigquery==1.5.0 \
    elasticsearch-dbapi==0.2.5 &&\
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi
extraEnv:
  AUTH_DOMAIN: ${jsonencode(domain_name)}
extraSecretEnv:
  GOOGLE_KEY: ${jsonencode(google_oauth_client_id)}
  GOOGLE_SECRET: ${jsonencode(google_oauth_client_secret)}

Expected results

All Pods come up healthy.

Actual results

The superset-worker Pod fails with the following stacktrace:

superset [2023-04-04 17:43:16,310: CRITICAL/MainProcess] Unrecoverable error: TypeError("__init__() got an unexpected keyword argument 'usern
superset Traceback (most recent call last):
superset   File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 925, in create_channel
superset     return self._avail_channels.pop()
superset IndexError: pop from empty list
superset
superset During handling of the above exception, another exception occurred:
superset
superset Traceback (most recent call last):
superset   File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 988, in get_connection
superset     connection = self._available_connections.pop()
superset IndexError: pop from empty list
superset
superset During handling of the above exception, another exception occurred:
superset
superset Traceback (most recent call last):
superset   File "/usr/local/lib/python3.8/site-packages/celery/worker/worker.py", line 203, in start
superset     self.blueprint.start(self)
superset   File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
superset     step.start(parent)
superset   File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 365, in start
superset     return self.obj.start()
superset   File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 326, in start
superset     blueprint.start(self)
superset   File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
superset     step.start(parent)
superset   File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/connection.py", line 21, in start
superset     c.connection = c.connect()
superset   File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 422, in connect
superset     conn = self.connection_for_read(heartbeat=self.amqheartbeat)
superset   File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 428, in connection_for_read
superset     return self.ensure_connected(
superset   File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 454, in ensure_connected
superset     conn = conn.ensure_connection(
superset   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 381, in ensure_connection
superset     self._ensure_connection(*args, **kwargs)
superset   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
superset     return retry_over_time(
superset   File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
superset     return fun(*args, **kwargs)
superset   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
superset     self._connection = self._establish_connection()
superset   File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
superset     conn = self.transport.establish_connection()
superset   File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 949, in establish_connection
superset     self._avail_channels.append(self.create_channel(self))
superset   File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 927, in create_channel
superset     channel = self.Channel(connection)
superset   File "/usr/local/lib/python3.8/site-packages/kombu/transport/redis.py", line 737, in __init__
superset     self.client.ping()
superset   File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 1037, in ping
superset     return self.execute_command('PING')
superset   File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 772, in execute_command
superset     connection = pool.get_connection(command_name, **options)
superset   File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 990, in get_connection
superset     connection = self.make_connection()
superset   File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 1025, in make_connection
superset     return self.connection_class(**self.connection_kwargs)
superset TypeError: __init__() got an unexpected keyword argument 'username'

Environment

(please complete the following information):

  • ~browser type and version:~
  • superset version: superset 2.0.1 (from Helm chart 0.8.10)
  • python version: 3.8.12
  • ~node.js version:~
  • ~any feature flags active:~

Checklist

Make sure to follow these steps before submitting your issue - thank you!

  • [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
  • [x] I have reproduced the issue with at least the latest released version of superset.
  • [x] I have checked the issue tracker for the same issue and I haven't found one similar.

Comment From: narioinc

@NiklasRosenstein just giving a +1 to this issue as the worker pod is giving the exact same issue on my minikube local installation.

minikube version minikube version: v1.30.1 commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION superset default 1 2023-04-12 07:51:13.2154475 +0530 IST failed superset-0.9.1 2.1.0

I am not attaching the logs here for brevity as it is the exact same as the one you posted. Going to take a look at the stacktrace and figure out where this is stemming from

Comment From: narioinc

@NiklasRosenstein
I got ahead on this just a few min back. so it seems that the celery[redis] handle was missing So here is my bootstrapScript section in values.yaml

bootstrapScript: | #!/bin/bash pip install psycopg2==2.9.1 \ redis==3.2.1 \ elasticsearch==7.10.1 \ elasticsearch-dbapi==0.2.10 &&\ pip uninstall celery &&\
pip install "celery[redis]" &&\ if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

I dont see the username error anymore, however unable to add an opendistro connection string yet. Hope this fix helps your usecase. let me know

Comment From: narioinc

I am not sure if the above is the correct fix, but atleast it let me progress further to test out opendistro connection issues.

Comment From: narioinc

@NiklasRosenstein

The final bootstrapscript version that worked for me:

bootstrapScript: | #!/bin/bash pip uninstall elasticsearch && \ pip install psycopg2==2.9.1 \ redis==3.2.1 \ elasticsearch==7.10.1 \ elasticsearch-dbapi==0.2.10 &&\ pip install "celery[redis]" &&\ if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

With this, i was able to connect to opensearch 2.6. Additionally i had to setup root CA access in superset "Database Security" settings page in UI, but once that was done, i was able to connect and i am now able to create dashboards, graphs etc.

Comment From: alpeshv

@NiklasRosenstein Remove redis==3.2.1 from your bootstrapScript and see if it gets installed as a part of other package installations. If not change the version to 3.5.3 and it should work.

Comment From: kli-bgl

@NiklasRosenstein Remove redis==3.2.1 from your bootstrapScript and see if it gets installed as a part of other package installations. If not change the version to 3.5.3 and it should work.

In my case, I can confirm the worker pod works after removing the Redis version from Bootstrap.

Comment From: juparog

Remove redis==3.2.1 in bootstrap script soved problem:

bootstrapScript: |
  #!/bin/bash
  pip install \
    psycopg2==2.9.1 \
    redis==3.2.1 \   # <<<--- remove this dependency
    PyAthena==2.25.2 \
    trino==0.321.0 &&\
  if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi

Comment From: rusackas

It looks like all the comments on this thread happened before the 3.0 release. We're no longer supporting Superset 2.x or prior, and it's been a while since this thread saw any activity, so I'm closing this as stale. If it is still an issue in Superset 3.x, we can re-open this, or feel free to open a new issue with updated context.

Comment From: olanipekunolabodeayoola

In my case, the minLength error was resolved by downgrading the version of marshmallow from the current 4.0.0 (as of the time of making this comment) to 3.20.1, which is a previous version.

To check the current version of marshmallow: pip show marshmallow

If higher than 3.20.1, specify the version of marshmallow to be re-installed: pip install marshmallow==3.20.1 Next, re-run: superset db upgrade

Hopefully, this resolves the error!

Credits: Zoetrope Studio's Apache Superset Installation on Windows 11 Using WSL — No Docker Needed! on YouTube (aka Sabbirz blog, Install superset on Windows with WSL and docker)