Bug description
After upgrade to 4.0.0, the workers keeps restarting
How to reproduce the bug
Install helm with this value.yaml
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Default values for superset.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# A README is automatically generated from this file to document it, using helm-docs (see https://github.com/norwoodj/helm-docs)
# To update it, install helm-docs and run helm-docs from the root of this chart
# -- Provide a name to override the name of the chart
nameOverride: ~
# -- Provide a name to override the full names of resources
fullnameOverride: ~
# -- User ID directive. This user must have enough permissions to run the bootstrap script
# Running containers as root is not recommended in production. Change this to another UID - e.g. 1000 to be more secure
runAsUser: 0
# -- Specify service account name to be used
serviceAccountName: ~
serviceAccount:
# -- Create custom service account for Superset. If create: true and serviceAccountName is not provided, `superset.fullname` will be used.
create: false
annotations: {}
# -- Install additional packages and do any other bootstrap configuration in this script
# For production clusters it's recommended to build own image with this step done in CI
# @default -- see `values.yaml`
bootstrapScript: |
#!/bin/bash
pip install --upgrade pip &&\
pip install sqlalchemy-bigquery redis \
pymssql pyodbc pymysql mysql-connector-python holidays==0.32 prophet==1.1.4 authlib
apt-get update
apt-get install -y wget zip libodbc2
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
apt-get install -y --no-install-recommends ./google-chrome-stable_current_amd64.deb
CHROMEVER=$(google-chrome --product-version | grep -o "[^\.]*\.[^\.]*\.[^\.]*")
DRIVERVER=$(curl -s "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_$CHROMEVER")
wget "https://storage.googleapis.com/chrome-for-testing-public/124.0.6367.60/linux64/chromedriver-linux64.zip"
unzip chromedriver-linux64.zip
ls -la
chmod +x chromedriver-linux64/chromedriver
mv chromedriver-linux64/chromedriver /usr/local/bin
apt-get autoremove -yqq --purge
apt-get clean
rm -f google-chrome-stable_current_amd64.deb chromedriver_linux64.zip
if [ ! -f ~/bootstrap ]; then echo "Running Superset with uid {{ .Values.runAsUser }}" > ~/bootstrap; fi
# -- The name of the secret which we will use to generate a superset_config.py file
# Note: this secret must have the key superset_config.py in it and can include other files as well
configFromSecret: '{{ template "superset.fullname" . }}-config'
# -- The name of the secret which we will use to populate env vars in deployed pods
# This can be useful for secret keys, etc.
envFromSecret: '{{ template "superset.fullname" . }}-env'
# -- This can be a list of templated strings
envFromSecrets: []
# -- Extra environment variables that will be passed into pods
extraEnv:
# Different gunicorn settings, refer to the gunicorn documentation
# https://docs.gunicorn.org/en/stable/settings.html#
# These variables are used as Flags at the gunicorn startup
# https://github.com/apache/superset/blob/master/docker/run-server.sh#L22
# Extend timeout to allow long running queries.
GUNICORN_TIMEOUT: 300
# Increase the gunicorn worker amount, can improve performance drastically
# See: https://docs.gunicorn.org/en/stable/design.html#how-many-workers
SERVER_WORKER_AMOUNT: 4
WORKER_MAX_REQUESTS: 0
WORKER_MAX_REQUESTS_JITTER: 0
SERVER_THREADS_AMOUNT: 20
GUNICORN_KEEPALIVE: 2
SERVER_LIMIT_REQUEST_LINE: 0
SERVER_LIMIT_REQUEST_FIELD_SIZE: 0
SUPERSET_WEBSERVER_PROTOCOL: "http"
WEBDRIVER_BASEURL_USER_FRIENDLY: "https://company-domain.***.com/"
SMTP_HOST: "mail.***.com"
SMTP_PORT: 2525
SMTP_STARTTLS: 'True'
SMTP_SSL_SERVER_AUTH: 'True'
SMTP_SSL: 'False'
SMTP_USER: "smtp@***.com"
SMTP_MAIL_FROM: "noreply@***.com"
EMAIL_REPORTS_SUBJECT_PREFIX: "[*** BI] "
# OAUTH_HOME_DOMAIN: ..
# # If a whitelist is not set, any address that can use your OAuth2 endpoint will be able to login.
# # this includes any random Gmail address if your OAuth2 Web App is set to External.
# OAUTH_WHITELIST_REGEX: ...
# -- Extra environment variables in RAW format that will be passed into pods
extraEnvRaw:
# Load DB password from other secret (e.g. for zalando operator)
- name: REDIS_PASS
valueFrom:
secretKeyRef:
name: redis
key: redis-password
# -- Extra environment variables to pass as secrets
extraSecretEnv:
SMTP_PASSWORD: "***"
MAPBOX_API_KEY: '***'
SUPERSET_SECRET_KEY: '***'
# MAPBOX_API_KEY: ...
# # Google API Keys: https://console.cloud.google.com/apis/credentials
# GOOGLE_KEY: ...
# GOOGLE_SECRET: ...
# # Generate your own secret key for encryption. Use openssl rand -base64 42 to generate a good key
# SUPERSET_SECRET_KEY: 'CHANGE_ME_TO_A_COMPLEX_RANDOM_SECRET'
# -- Extra files to mount on `/app/pythonpath`
extraConfigs:
custom_sso_security_manager.py: |
import logging
from superset.security import SupersetSecurityManager
logger = logging.getLogger("azure_ad")
class CustomSsoSecurityManager(SupersetSecurityManager):
from flask_appbuilder.security.manager import (AUTH_DB, AUTH_OAUTH)
def oauth_user_info(self, provider, response=None):
#In case if userDetail url is nor working case use response attribute to get the user details like name,mail etc
logger.debug("Oauth2 provider: {0}.".format(provider))
if provider == 'azure':
# As example, this line request a GET to base_url + '/' + userDetails with Bearer Authentication,
# and expects that authorization server checks the token, and response with user details
#me = self.appbuilder.sm.oauth_remotes[provider].get('userDetails').data
#logging.debug("user_data: {0}".format(me))
#check the values of user_name, mail and others values in me variable
#return { 'name' : me['User_name'], 'email' : me['mail'], 'id' : me['user_name'], 'username' : me['user_name'], 'first_name':'', 'last_name':''}
logger.debug("Azure response received : {0}".format(response))
id_token = response["id_token"]
logger.debug(str(id_token))
#logging.debug(" -- oauth user detail -- : {0}".format(self.appbuilder.sm.oauth_remotes[provider].get('userDetails')))
me = self._decode_and_validate_azure_jwt(id_token)
logger.debug("Parse JWT token : {0}".format(me))
return {
"name": me.get("name", ""),
"email": me["upn"],
"first_name": me.get("given_name", ""),
"last_name": me.get("family_name", ""),
"id": me["unique_name"],
"username": me["upn"],
"role_keys": me.get("roles", []),
}
# import_datasources.yaml: |
# databases:
# - allow_file_upload: true
# allow_ctas: true
# allow_cvas: true
# database_name: example-db
# extra: "{\r\n \"metadata_params\": {},\r\n \"engine_params\": {},\r\n \"\
# metadata_cache_timeout\": {},\r\n \"schemas_allowed_for_file_upload\": []\r\n\
# }"
# sqlalchemy_uri: example://example-db.local
# tables: []
# -- Extra files to mount on `/app/pythonpath` as secrets
extraSecrets: {}
extraVolumes:
- name: v
configMap:
name: v
items:
- key: k
path: v
# - name: customConfig
# configMap:
# name: '{{ template "superset.fullname" . }}-custom-config'
# - name: additionalSecret
# secret:
# secretName: my-secret
# defaultMode: 0600
extraVolumeMounts:
- name: v
mountPath: /tmp
subPath: v
readOnly: true
# - name: customConfig
# mountPath: /mnt/config
# readOnly: true
# - name: additionalSecret:
# mountPath: /mnt/secret
# -- A dictionary of overrides to append at the end of superset_config.py - the name does not matter
# WARNING: the order is not guaranteed
# Files can be passed as helm --set-file configOverrides.my-override=my-file.py
configOverrides:
feature_flags: |
import ast
FEATURE_FLAGS = {
"ALERT_REPORTS": True
}
SMTP_HOST = os.getenv("SMTP_HOST","localhost")
SMTP_STARTTLS = ast.literal_eval(os.getenv("SMTP_STARTTLS", "True"))
SMTP_SSL = ast.literal_eval(os.getenv("SMTP_SSL", "False"))
SMTP_USER = os.getenv("SMTP_USER","superset")
SMTP_PORT = os.getenv("SMTP_PORT",25)
SMTP_PASSWORD = os.getenv("SMTP_PASSWORD","superset")
SMTP_MAIL_FROM = os.getenv("SMTP_MAIL_FROM","superset@superset.com")
ALERT_REPORTS_NOTIFICATION_DRY_RUN = ast.literal_eval(os.getenv("ALERT_REPORTS_NOTIFICATION_DRY_RUN","False"))
SLACK_API_TOKEN = os.getenv("SLACK_API_TOKEN",None)
celery_conf: |
from celery.schedules import crontab
class CeleryConfig:
broker_url = f"redis://:{env('REDIS_PASS')}@{env('REDIS_HOST')}:{env('REDIS_PORT')}/0"
imports = (
"superset.sql_lab",
"superset.tasks.cache",
"superset.tasks.scheduler",
)
result_backend = f"redis://:{env('REDIS_PASS')}@{env('REDIS_HOST')}:{env('REDIS_PORT')}/0"
worker_prefetch_multiplier = 10
task_acks_late = True
task_annotations = {
"sql_lab.get_sql_results": {
"rate_limit": "100/s",
},
}
beat_schedule = {
"reports.scheduler": {
"task": "reports.scheduler",
"schedule": crontab(minute="*", hour="*"),
},
"reports.prune_log": {
"task": "reports.prune_log",
'schedule': crontab(minute=0, hour=0),
},
'cache-warmup-hourly': {
"task": "cache-warmup",
"schedule": crontab(minute="*/30", hour="*"),
"kwargs": {
"strategy_name": "top_n_dashboards",
"top_n": 10,
"since": "7 days ago",
},
}
}
CELERY_CONFIG = CeleryConfig
cache_config: |
from superset.tasks.types import ExecutorType
FILTER_CACHE_REDIS_URL = f"{REDIS_BASE_URL}/{env('REDIS_DB', 2)}{REDIS_URL_PARAMS}"
FILTER_STATE_CACHE_CONFIG = {
'CACHE_TYPE': 'RedisCache',
'CACHE_DEFAULT_TIMEOUT': 24*60*60,
'CACHE_KEY_PREFIX': 'superset_filter_state',
'CACHE_REDIS_URL': FILTER_CACHE_REDIS_URL,
}
THUMBNAIL_CACHE_REDIS_URL = f"{REDIS_BASE_URL}/{env('REDIS_DB', 3)}{REDIS_URL_PARAMS}"
THUMBNAIL_CACHE_CONFIG = {
'CACHE_TYPE': 'RedisCache',
'CACHE_DEFAULT_TIMEOUT': 24*60*60*7,
'CACHE_KEY_PREFIX': 'thumbnail_',
'CACHE_REDIS_URL': THUMBNAIL_CACHE_REDIS_URL,
}
THUMBNAIL_SELENIUM_USER = "admin"
THUMBNAIL_EXECUTE_AS = [ExecutorType.SELENIUM]
reports: |
EMAIL_PAGE_RENDER_WAIT = 60
WEBDRIVER_BASEURL = "http://{{ template "superset.fullname" . }}:{{ .Values.service.port }}/"
WEBDRIVER_BASEURL_USER_FRIENDLY = os.getenv("WEBDRIVER_BASEURL_USER_FRIENDLY", "https://www.example.com/")
WEBDRIVER_TYPE= "chrome"
WEBDRIVER_OPTION_ARGS = [
"--force-device-scale-factor=2.0",
"--high-dpi-support=2.0",
"--headless",
"--disable-gpu",
"--disable-dev-shm-usage",
# This is required because our process runs as root (in order to install pip packages)
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-extensions",
]
SCREENSHOT_LOCATE_WAIT = 100
SCREENSHOT_LOAD_WAIT = 1200
from superset.utils.urls import headless_url
from superset.utils.machine_auth import MachineAuthProvider
def auth_driver(driver, user):
# Setting cookies requires doing a request first, but /login is redirected to oauth provider, and stuck there.
driver.get(headless_url("/doesnotexist"))
cookies = MachineAuthProvider.get_auth_cookies(user)
for cookie_name, cookie_val in cookies.items():
driver.add_cookie(dict(name=cookie_name, value=cookie_val))
return driver
WEBDRIVER_AUTH_FUNC = auth_driver
extend_timeout: |
# Extend timeout to allow long running queries.
SUPERSET_WEBSERVER_TIMEOUT = 300
enable_oauth: |
import sys
sys.path.append('/app/configs/')
print (sys.path)
from flask_appbuilder.security.manager import AUTH_OAUTH
from custom_sso_security_manager import CustomSsoSecurityManager
CUSTOM_SECURITY_MANAGER = CustomSsoSecurityManager # For managing User data after fetching from SSO app
AUTH_TYPE = AUTH_OAUTH
OAUTH_PROVIDERS = [
{
'name':'azure',
'token_key':'access_token', # Name of the token in the response of access_token_url
'icon':'fa-address-card', # Icon for the provider
'remote_app': {
'client_id':'***', # Client Id (Identify Superset application)
'client_secret':'***', # Secret for this Client Id (Identify Superset application)
'client_kwargs':{
'scope': 'User.read name preferred_username email profile upn groups' # Scope for the Authorization
},
'access_token_method':'POST', # HTTP Method to call access_token_url
#'access_token_params':{ # Additional parameters for calls to access_token_url
# 'client_id':'myClientId'
#},
#'jwks_uri':'https://myAuthorizationServe/adfs/discovery/keys', # may be required to generate token
#'access_token_headers':{ # Additional headers for calls to access_token_url
# 'Authorization': 'Basic Base64EncodedClientIdAndSecret'
#},
'api_base_url':'https://login.microsoftonline.com/***/oauth2',
'access_token_url':'https://login.microsoftonline.com/***/oauth2/token',
'authorize_url':'https://login.microsoftonline.com/***/oauth2/authorize'
}
}
]
AUTH_ROLE_PUBLIC = 'Public'
AUTH_USER_REGISTRATION = True
additional_config: |
APP_NAME = '*** BI'
APP_ICON = "/static/assets/images/logo.png"
APP_ICON_WIDTH = 200
LOGO_TARGET_PATH = '/'
LOGO_TOOLTIP = '*** BI'
SUPERSET_WEBSERVER_PROTOCOL = os.getenv("SUPERSET_WEBSERVER_PROTOCOL","https")
EMAIL_REPORTS_SUBJECT_PREFIX = os.getenv("EMAIL_REPORTS_SUBJECT_PREFIX","*** BI] ")
ENABLE_PROXY_FIX = True
RATELIMIT_ENABLED = False
AUTH_RATE_LIMITED = False
DEFAULT_FEATURE_FLAGS: dict[str, bool] = {
"CLIENT_CACHE": False, # deprecated
"DISABLE_DATASET_SOURCE_EDIT": False, # deprecated
"DRUID_JOINS": False,
"DYNAMIC_PLUGINS": False,
"DISABLE_LEGACY_DATASOURCE_EDITOR": True,
"ENABLE_EXPLORE_JSON_CSRF_PROTECTION": False, # deprecated
"ENABLE_TEMPLATE_PROCESSING": False,
"ENABLE_TEMPLATE_REMOVE_FILTERS": True, # deprecated
"ENABLE_JAVASCRIPT_CONTROLS": False,
"KV_STORE": True,
"PRESTO_EXPAND_DATA": False,
"THUMBNAILS": True,
"THUMBNAILS_SQLA_LISTENERS": True,
"DASHBOARD_CACHE": False, # deprecated
"REMOVE_SLICE_LEVEL_LABEL_COLORS": False, # deprecated
"SHARE_QUERIES_VIA_KV_STORE": True,
"TAGGING_SYSTEM": True,
"SQLLAB_BACKEND_PERSISTENCE": True,
"LISTVIEWS_DEFAULT_CARD_VIEW": False,
"ESCAPE_MARKDOWN_HTML": False,
"DASHBOARD_NATIVE_FILTERS": True, # deprecated
"DASHBOARD_CROSS_FILTERS": True,
"DASHBOARD_NATIVE_FILTERS_SET": False, # deprecated
"DASHBOARD_FILTERS_EXPERIMENTAL": False, # deprecated
"DASHBOARD_VIRTUALIZATION": False,
"GLOBAL_ASYNC_QUERIES": False,
"VERSIONED_EXPORT": True, # deprecated
"EMBEDDED_SUPERSET": False,
"ALERT_REPORTS": True,
"DASHBOARD_RBAC": True,
"ENABLE_EXPLORE_DRAG_AND_DROP": True, # deprecated
"ENABLE_ADVANCED_DATA_TYPES": True,
"ALERTS_ATTACH_REPORTS": True,
"ALLOW_FULL_CSV_EXPORT": True,
"GENERIC_CHART_AXES": True, # deprecated
"ALLOW_ADHOC_SUBQUERY": True,
"USE_ANALAGOUS_COLORS": False,
"RLS_IN_SQLLAB": False,
"CACHE_IMPERSONATION": False,
"CACHE_QUERY_BY_USER": True,
"EMBEDDABLE_CHARTS": True,
"DRILL_TO_DETAIL": True,
"DRILL_BY": True,
"DATAPANEL_CLOSED_BY_DEFAULT": False,
"HORIZONTAL_FILTER_BAR": True,
"ESTIMATE_QUERY_COST": False,
"SSH_TUNNELING": False,
"AVOID_COLORS_COLLISION": True,
"MENU_HIDE_USER_INFO": False,
"ENABLE_SUPERSET_META_DB": True,
"PLAYWRIGHT_REPORTS_AND_THUMBNAILS": False,
}
# extend_timeout: |
# # Extend timeout to allow long running queries.
# SUPERSET_WEBSERVER_TIMEOUT = ...
# enable_oauth: |
# from flask_appbuilder.security.manager import (AUTH_DB, AUTH_OAUTH)
# AUTH_TYPE = AUTH_OAUTH
# OAUTH_PROVIDERS = [
# {
# "name": "google",
# "whitelist": [ os.getenv("OAUTH_WHITELIST_REGEX", "") ],
# "icon": "fa-google",
# "token_key": "access_token",
# "remote_app": {
# "client_id": os.environ.get("GOOGLE_KEY"),
# "client_secret": os.environ.get("GOOGLE_SECRET"),
# "api_base_url": "https://www.googleapis.com/oauth2/v2/",
# "client_kwargs": {"scope": "email profile"},
# "request_token_url": None,
# "access_token_url": "https://accounts.google.com/o/oauth2/token",
# "authorize_url": "https://accounts.google.com/o/oauth2/auth",
# "authorize_params": {"hd": os.getenv("OAUTH_HOME_DOMAIN", "")}
# }
# }
# ]
# # Map Authlib roles to superset roles
# AUTH_ROLE_ADMIN = 'Admin'
# AUTH_ROLE_PUBLIC = 'Public'
# # Will allow user self registration, allowing to create Flask users from Authorized User
# AUTH_USER_REGISTRATION = True
# # The default user self registration role
# AUTH_USER_REGISTRATION_ROLE = "Admin"
# secret: |
# # Generate your own secret key for encryption. Use `openssl rand -base64 42` to generate a good key
# SECRET_KEY = 'CHANGE_ME_TO_A_COMPLEX_RANDOM_SECRET'
# -- Same as above but the values are files
configOverridesFiles: {}
# extend_timeout: extend_timeout.py
# enable_oauth: enable_oauth.py
configMountPath: "/app/pythonpath"
extraConfigMountPath: "/app/configs"
image:
repository: apachesuperset.docker.scarf.sh/apache/superset
tag: 4.0.0-dev
pullPolicy: Always
imagePullSecrets: []
initImage:
repository: apache/superset
tag: dockerize
pullPolicy: Always
service:
type: ClusterIP
port: 8088
annotations: {}
# cloud.google.com/load-balancer-type: "Internal"
loadBalancerIP: ~
nodePort:
# -- (int)
http: nil
ingress:
enabled: true
ingressClassName: "public"
annotations:
kubernetes.io/tls-acme: "true"
## Extend timeout to allow long running queries.
nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
nginx.ingress.kubernetes.io/client_max_body_size : "50m"
nginx.ingress.kubernetes.io/client-body-buffer-size: "50m"
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
# kubernetes.io/tls-acme: "true"
## Extend timeout to allow long running queries.
# nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
# nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
# nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
path: /
pathType: ImplementationSpecific
hosts:
- ***.com
tls:
- hosts:
- ***.com
secretName: ***.com-tls
extraHostsRaw: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# The limits below will apply to all Superset components. To set individual resource limitations refer to the pod specific values below.
# The pod specific values will overwrite anything that is set here.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
# -- Custom hostAliases for all superset pods
## https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/
hostAliases: []
# - hostnames:
# - nodns.my.lan
# ip: 18.27.36.45
# Superset node configuration
supersetNode:
replicaCount: 1
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
# -- Sets the [pod disruption budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for supersetNode pods
podDisruptionBudget:
# -- Whether the pod disruption budget should be created
enabled: false
# -- If set, maxUnavailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
minAvailable: 1
# -- If set, minAvailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
maxUnavailable: 1
# -- Startup command
# @default -- See `values.yaml`
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; /usr/bin/run-server.sh"
connections:
# -- Change in case of bringing your own redis and then also set redis.enabled:false
redis_host: 'redis-***'
redis_port: "6379"
redis_user: ""
redis_password: ***
redis_cache_db: "1"
redis_celery_db: "0"
# Or SSL port is usually 6380
# Update following for using Redis with SSL
redis_ssl:
enabled: false
ssl_cert_reqs: CERT_NONE
# You need to change below configuration incase bringing own PostgresSQL instance and also set postgresql.enabled:false
db_host: 'postgresql-***'
db_port: "5432"
db_user: ***
db_pass: "***"
db_name: ***
env: {}
# -- If true, forces deployment to reload on each upgrade
forceReload: true
# -- Init containers
# @default -- a container waiting for postgres
initContainers:
- name: wait-for-postgres
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/sh
- -c
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -timeout 120s
# -- Launch additional containers into supersetNode pod
extraContainers: []
# -- Annotations to be added to supersetNode deployment
deploymentAnnotations: {}
# -- Labels to be added to supersetNode deployment
deploymentLabels: {}
# -- Affinity to be added to supersetNode deployment
affinity: {}
# -- TopologySpreadConstrains to be added to supersetNode deployments
topologySpreadConstraints: []
# -- Annotations to be added to supersetNode pods
podAnnotations: {}
# -- Labels to be added to supersetNode pods
podLabels: {}
startupProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
timeoutSeconds: 1
failureThreshold: 60
periodSeconds: 5
successThreshold: 1
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 15
successThreshold: 1
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 15
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 15
successThreshold: 1
# -- Resource settings for the supersetNode pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
podSecurityContext: {}
containerSecurityContext: {}
strategy: {}
# type: RollingUpdate
# rollingUpdate:
# maxSurge: 25%
# maxUnavailable: 25%
# Superset Celery worker configuration
supersetWorker:
replicaCount: 5
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
# -- Sets the [pod disruption budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for supersetWorker pods
podDisruptionBudget:
# -- Whether the pod disruption budget should be created
enabled: false
# -- If set, maxUnavailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
minAvailable: 1
# -- If set, minAvailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
maxUnavailable: 1
# -- Worker startup command
# @default -- a `celery worker` command
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app worker"
# -- If true, forces deployment to reload on each upgrade
forceReload: true
# -- Init container
# @default -- a container waiting for postgres and redis
initContainers:
- name: wait-for-postgres-redis
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/sh
- -c
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT" -timeout 120s
# -- Launch additional containers into supersetWorker pod
extraContainers: []
# -- Annotations to be added to supersetWorker deployment
deploymentAnnotations: {}
# -- Labels to be added to supersetWorker deployment
deploymentLabels: {}
# -- Affinity to be added to supersetWorker deployment
affinity: {}
# -- TopologySpreadConstrains to be added to supersetWorker deployments
topologySpreadConstraints: []
# -- Annotations to be added to supersetWorker pods
podAnnotations: {}
# -- Labels to be added to supersetWorker pods
podLabels: {}
# -- Resource settings for the supersetWorker pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
podSecurityContext: {}
containerSecurityContext: {}
strategy: {}
# type: RollingUpdate
# rollingUpdate:
# maxSurge: 25%
# maxUnavailable: 25%
livenessProbe:
exec:
# -- Liveness probe command
# @default -- a `celery inspect ping` command
command:
- sh
- -c
- celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME
initialDelaySeconds: 120
timeoutSeconds: 60
failureThreshold: 3
periodSeconds: 60
successThreshold: 1
# -- No startup/readiness probes by default since we don't really care about its startup time (it doesn't serve traffic)
startupProbe: {}
# -- No startup/readiness probes by default since we don't really care about its startup time (it doesn't serve traffic)
readinessProbe: {}
# Superset beat configuration (to trigger scheduled jobs like reports)
supersetCeleryBeat:
# -- This is only required if you intend to use alerts and reports
enabled: true
# -- Sets the [pod disruption budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for supersetCeleryBeat pods
podDisruptionBudget:
# -- Whether the pod disruption budget should be created
enabled: false
# -- If set, maxUnavailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
minAvailable: 1
# -- If set, minAvailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
maxUnavailable: 1
# -- Command
# @default -- a `celery beat` command
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid --schedule /tmp/celerybeat-schedule"
# -- If true, forces deployment to reload on each upgrade
forceReload: false
# -- List of init containers
# @default -- a container waiting for postgres
initContainers:
- name: wait-for-postgres-redis
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/sh
- -c
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT" -timeout 120s
# -- Launch additional containers into supersetCeleryBeat pods
extraContainers: []
# -- Annotations to be added to supersetCeleryBeat deployment
deploymentAnnotations: {}
# -- Affinity to be added to supersetCeleryBeat deployment
affinity: {}
# -- TopologySpreadConstrains to be added to supersetCeleryBeat deployments
topologySpreadConstraints: []
# -- Annotations to be added to supersetCeleryBeat pods
podAnnotations: {}
# -- Labels to be added to supersetCeleryBeat pods
podLabels: {}
# -- Resource settings for the CeleryBeat pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
podSecurityContext: {}
containerSecurityContext: {}
supersetCeleryFlower:
# -- Enables a Celery flower deployment (management UI to monitor celery jobs)
# WARNING: on superset 1.x, this requires a Superset image that has `flower<1.0.0` installed (which is NOT the case of the default images)
# flower>=1.0.0 requires Celery 5+ which Superset 1.5 does not support
enabled: false
replicaCount: 1
# -- Sets the [pod disruption budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for supersetCeleryFlower pods
podDisruptionBudget:
# -- Whether the pod disruption budget should be created
enabled: false
# -- If set, maxUnavailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
minAvailable: 1
# -- If set, minAvailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
maxUnavailable: 1
# -- Command
# @default -- a `celery flower` command
command:
- "/bin/sh"
- "-c"
- "pip install redis authlib flower && celery --app=superset.tasks.celery_app:app flower"
service:
type: ClusterIP
annotations: {}
loadBalancerIP: ~
port: 5555
nodePort:
# -- (int)
http: nil
startupProbe:
httpGet:
path: /api/workers
port: flower
initialDelaySeconds: 5
timeoutSeconds: 1
failureThreshold: 60
periodSeconds: 5
successThreshold: 1
livenessProbe:
httpGet:
path: /api/workers
port: flower
initialDelaySeconds: 5
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 5
successThreshold: 1
readinessProbe:
httpGet:
path: /api/workers
port: flower
initialDelaySeconds: 5
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 5
successThreshold: 1
# -- List of init containers
# @default -- a container waiting for postgres and redis
initContainers:
- name: wait-for-postgres-redis
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/sh
- -c
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT" -timeout 120s
# -- Launch additional containers into supersetCeleryFlower pods
extraContainers: []
# -- Annotations to be added to supersetCeleryFlower deployment
deploymentAnnotations: {}
# -- Affinity to be added to supersetCeleryFlower deployment
affinity: {}
# -- TopologySpreadConstrains to be added to supersetCeleryFlower deployments
topologySpreadConstraints: []
# -- Annotations to be added to supersetCeleryFlower pods
podAnnotations: {}
# -- Labels to be added to supersetCeleryFlower pods
podLabels: {}
# -- Resource settings for the CeleryBeat pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
podSecurityContext: {}
containerSecurityContext: {}
supersetWebsockets:
# -- This is only required if you intend to use `GLOBAL_ASYNC_QUERIES` in `ws` mode
# see https://github.com/apache/superset/blob/master/CONTRIBUTING.md#async-chart-queries
enabled: false
replicaCount: 1
# -- Sets the [pod disruption budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for supersetWebsockets pods
podDisruptionBudget:
# -- Whether the pod disruption budget should be created
enabled: false
# -- If set, maxUnavailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
minAvailable: 1
# -- If set, minAvailable must not be set - see https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
maxUnavailable: 1
ingress:
path: /ws
pathType: Prefix
image:
# -- There is no official image (yet), this one is community-supported
repository: oneacrefund/superset-websocket
tag: latest
pullPolicy: IfNotPresent
# -- The config.json to pass to the server, see https://github.com/apache/superset/tree/master/superset-websocket
# Note that the configuration can also read from environment variables (which will have priority), see https://github.com/apache/superset/blob/master/superset-websocket/src/config.ts for a list of supported variables
# @default -- see `values.yaml`
config:
{
"port": 8080,
"logLevel": "debug",
"logToFile": false,
"logFilename": "app.log",
"statsd": { "host": "127.0.0.1", "port": 8125, "globalTags": [] },
"redis":
{
"port": 6379,
"host": "127.0.0.1",
"password": "",
"db": 0,
"ssl": false,
},
"redisStreamPrefix": "async-events-",
"jwtSecret": "CHANGE-ME",
"jwtCookieName": "async-token",
}
service:
type: ClusterIP
annotations: {}
loadBalancerIP: ~
port: 8080
nodePort:
# -- (int)
http: nil
command: []
resources: {}
# -- Launch additional containers into supersetWebsockets pods
extraContainers: []
deploymentAnnotations: {}
# -- Affinity to be added to supersetWebsockets deployment
affinity: {}
# -- TopologySpreadConstrains to be added to supersetWebsockets deployments
topologySpreadConstraints: []
podAnnotations: {}
podLabels: {}
strategy: {}
podSecurityContext: {}
containerSecurityContext: {}
startupProbe:
httpGet:
path: /health
port: ws
initialDelaySeconds: 5
timeoutSeconds: 1
failureThreshold: 60
periodSeconds: 5
successThreshold: 1
livenessProbe:
httpGet:
path: /health
port: ws
initialDelaySeconds: 5
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 5
successThreshold: 1
readinessProbe:
httpGet:
path: /health
port: ws
initialDelaySeconds: 5
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 5
successThreshold: 1
init:
# Configure resources
# Warning: fab command consumes a lot of ram and can
# cause the process to be killed due to OOM if it exceeds limit
# Make sure you are giving a strong password for the admin user creation( else make sure you are changing after setup)
# Also change the admin email to your own custom email.
resources: {}
# limits:
# cpu:
# memory:
# requests:
# cpu:
# memory:
# -- Command
# @default -- a `superset_init.sh` command
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; . {{ .Values.configMountPath }}/superset_init.sh"
enabled: false
jobAnnotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": "before-hook-creation"
loadExamples: false
createAdmin: false
adminUser:
username: ***
firstname: ***
lastname: ***
email: ***
password: ***
# -- List of initContainers
# @default -- a container waiting for postgres
initContainers:
- name: wait-for-postgres
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/sh
- -c
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -timeout 120s
# -- A Superset init script
# @default -- a script to create admin user and initialize roles
initscript: |-
#!/bin/sh
pip install holidays==0.23
set -eu
echo "Upgrading DB schema..."
superset db upgrade
echo "Initializing roles..."
superset init
{{ if .Values.init.createAdmin }}
echo "Creating admin user..."
superset fab create-admin \
--username {{ .Values.init.adminUser.username }} \
--firstname {{ .Values.init.adminUser.firstname }} \
--lastname {{ .Values.init.adminUser.lastname }} \
--email {{ .Values.init.adminUser.email }} \
--password {{ .Values.init.adminUser.password }} \
|| true
{{- end }}
{{ if .Values.init.loadExamples }}
echo "Loading examples..."
superset load_examples
{{- end }}
if [ -f "{{ .Values.extraConfigMountPath }}/import_datasources.yaml" ]; then
echo "Importing database connections.... "
superset import_datasources -p {{ .Values.extraConfigMountPath }}/import_datasources.yaml
fi
# -- Launch additional containers into init job pod
extraContainers: []
## Annotations to be added to init job pods
podAnnotations: {}
podSecurityContext: {}
containerSecurityContext: {}
## Tolerations to be added to init job pods
tolerations: []
## Affinity to be added to init job pods
affinity: {}
# -- TopologySpreadConstrains to be added to init job
topologySpreadConstraints: []
# -- Configuration values for the postgresql dependency.
# ref: https://github.com/bitnami/charts/tree/main/bitnami/postgresql
# @default -- see `values.yaml`
postgresql:
##
## Use the PostgreSQL chart dependency.
## Set to false if bringing your own PostgreSQL.
enabled: false
## Authentication parameters
auth:
## The name of an existing secret that contains the postgres password.
existingSecret:
## PostgreSQL name for a custom user to create
username: ***
## PostgreSQL password for the custom user to create. Ignored if `auth.existingSecret` with key `password` is provided
password: ***
## PostgreSQL name for a custom database to create
database: ***
image:
tag: "14.6.0-debian-11-r13"
## PostgreSQL Primary parameters
primary:
##
## Persistent Volume Storage configuration.
## ref: https://kubernetes.io/docs/user-guide/persistent-volumes
persistence:
##
## Enable PostgreSQL persistence using Persistent Volume Claims.
enabled: true
##
## Persistent class
# storageClass: classname
##
## Access modes:
accessModes:
- ReadWriteOnce
## PostgreSQL port
service:
ports:
postgresql: "5432"
# -- Configuration values for the Redis dependency.
# ref: https://github.com/bitnami/charts/blob/master/bitnami/redis
# More documentation can be found here: https://artifacthub.io/packages/helm/bitnami/redis
# @default -- see `values.yaml`
redis:
##
## Use the redis chart dependency.
##
## If you are bringing your own redis, you can set the host in supersetNode.connections.redis_host
##
## Set to false if bringing your own redis.
enabled: false
##
## Set architecture to standalone/replication
architecture: standalone
##
## Auth configuration:
##
auth:
## Enable password authentication
enabled: false
## The name of an existing secret that contains the redis password.
existingSecret: ""
## Name of the key containing the secret.
existingSecretKey: ""
## Redis password
password: ***
##
## Master configuration
##
master:
##
## Image configuration
# image:
##
## docker registry secret names (list)
# pullSecrets: nil
##
## Configure persistance
persistence:
##
## Use a PVC to persist data.
enabled: false
##
## Persistent class
# storageClass: classname
##
## Access mode:
accessModes:
- ReadWriteOnce
nodeSelector: {}
tolerations: []
affinity: {}
# -- TopologySpreadConstrains to be added to all deployments
topologySpreadConstraints: []
Screenshots/recordings
Logs from the worker pod
logging was configured successfully
2024-05-13 02:26:25,220:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-05-13 02:26:25,222:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
2024-05-13 02:26:26,655:DEBUG:celery.utils.functional:
def chord(self, header, body, partial_args=0, interval=1, countdown=2, max_retries=3, eager=4, **kwargs):
return 1
2024-05-13 02:26:26,655:DEBUG:celery.utils.functional:
def accumulate(self, *args, **kwargs):
return 1
2024-05-13 02:26:26,656:DEBUG:celery.utils.functional:
def cache_chart_thumbnail(current_user, chart_id, force=0, window_size=1, thumb_size=2):
return 1
2024-05-13 02:26:26,656:DEBUG:celery.utils.functional:
def execute(self, report_schedule_id):
return 1
2024-05-13 02:26:26,656:DEBUG:celery.utils.functional:
def unlock_chord(self, group_id, callback, interval=0, max_retries=1, result=2, Result=3, GroupResult=4, result_from_tuple=5, **kwargs):
return 1
2024-05-13 02:26:26,656:DEBUG:celery.utils.functional:
def backend_cleanup():
return 1
2024-05-13 02:26:26,657:DEBUG:celery.utils.functional:
def cache_dashboard_thumbnail(current_user, dashboard_id, force=0, thumb_size=1):
return 1
2024-05-13 02:26:26,657:DEBUG:celery.utils.functional:
def xmap(task, it):
return 1
2024-05-13 02:26:26,657:DEBUG:celery.utils.functional:
def prune_log():
return 1
2024-05-13 02:26:26,657:DEBUG:celery.utils.functional:
def xstarmap(task, it):
return 1
2024-05-13 02:26:26,657:DEBUG:celery.utils.functional:
def cache_warmup(strategy_name, *args, **kwargs):
return 1
2024-05-13 02:26:26,658:DEBUG:celery.utils.functional: def get_sql_results(query_id, rendered_query, return_results=0, store_results=1, username=2, start_time=3, expand_data=4, log_params=5):
return 1
2024-05-13 02:26:26,658:DEBUG:celery.utils.functional:
def fetch_url(data, headers):
return 1
2024-05-13 02:26:26,658:DEBUG:celery.utils.functional:
def chunks(task, it, n):
return 1
2024-05-13 02:26:26,658:DEBUG:celery.utils.functional:
def group(self, tasks, result, group_id, partial_args, add_to_parent=0):
return 1
2024-05-13 02:26:26,659:DEBUG:celery.utils.functional:
def scheduler():
return 1
2024-05-13 02:26:26,659:DEBUG:celery.utils.functional:
def chain(*args, **kwargs):
return 1
/usr/local/lib/python3.10/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
['/usr/local/bin', '/app/pythonpath', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app', '/app/configs/']
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
-------------- celery@superset-worker-54bc9dffd4-skkk8 v5.3.6 (emerald-rush)
--- ***** -----
-- ******* ---- Linux-5.15.0-102-generic-x86_64-with-glibc2.36 2024-05-13 02:26:26
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: __main__:0x7f934c950a90
- ** ---------- .> transport: redis://:**@redis-master:6379/0
- ** ---------- .> results: redis://:**@redis-master:6379/0
- *** --- * --- .> concurrency: 24 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[2024-05-13 02:26:29,760: WARNING/MainProcess] /usr/local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
[2024-05-13 02:26:29,768: WARNING/MainProcess] /usr/local/lib/python3.10/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
Superset version
4.0.0
Python version
3.10
Node version
16
Browser
Chrome
Additional context
No response
Checklist
- [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [X] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
Comment From: dosubot[bot]
To address the issue of worker instances continuously restarting after upgrading to Superset version 4.0.0, consider the following steps:
-
Compatibility Check: Ensure all components, including Python 3.11 and Node 16, are compatible with Superset 4.0.0. While Python 3.11 is compatible, Node.js version 18.20.1 is recommended for the best development experience with Superset's frontend.
-
Review Custom Bootstrap Script: Inspect your custom bootstrap script for any conflicts or deprecated packages that might not be compatible with Superset 4.0.0.
-
Worker Configuration: Verify the
commandused to start worker instances in your helm chart is correct for Superset 4.0.0, focusing on the path tosuperset_bootstrap.shand the Celery command. -
Liveness and Readiness Probes: Ensure the configuration for these probes in the
supersetWorkersection is correctly set up for Superset 4.0.0, as incorrect configurations can lead to continuous restarts. -
Environment Variables and Configurations: Check that all necessary environment variables and configurations are correctly set up, especially those derived from secrets or config maps.
-
Log Analysis: Review the logs of the restarting worker instances for errors or warnings that could indicate the cause of the restarts.
-
Dependencies and Package Versions: Make sure all dependencies specified in your bootstrap script or helm chart are compatible with Superset 4.0.0.
-
Resource Limits: Review and adjust the resource limits and requests for the worker instances if they are being terminated due to resource constraints.
-
Review Helm Chart Annotations and Checksums: Update checksums for configurations if changes are made, ensuring Kubernetes recognizes and applies these changes.
-
Security Concerns with
runAsUser: Running containers as root (UID 0) is discouraged due to security risks. Consider running containers with a non-root user.
These steps should help identify and resolve the issue with worker instances restarting. If the problem persists, further investigation into Superset's change logs for version 4.0.0 may be necessary to identify any breaking changes or additional migration steps required.
#### Details
**Sources** - [supersetWorker Liveness Probes always fail causing restarts](https://github.com/apache/superset/issues/25225) - [helm/superset/values.yaml](https://github.com/apache/superset/blob/master/helm/superset/values.yaml) - [helm/superset/templates/deployment-worker.yaml](https://github.com/apache/superset/blob/master/helm/superset/templates/deployment-worker.yaml) - [helm/superset/templates/deployment.yaml](https://github.com/apache/superset/blob/master/helm/superset/templates/deployment.yaml) - [CHANGELOG/4.0.0.md](https://github.com/apache/superset/blob/master/CHANGELOG/4.0.0.md) - [pyproject.toml](https://github.com/apache/superset/blob/master/pyproject.toml) - [superset-frontend/.nvmrc](https://github.com/apache/superset/blob/master/superset-frontend/.nvmrc)**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.
Comment From: fzhan
It's an issue with Liveness probe :
Liveness probe failed: ['/usr/local/bin', '/app/pythonpath', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app', '/app/configs/'] Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py] logging was configured successfully 2024-05-13 02:49:48,445:INFO:superset.utils.logging_configurator:logging was configured successfully 2024-05-13 02:49:48,447:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'> Error: No nodes replied within time constraint
Comment From: fzhan
After running 'celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME' manually in worker pod, response was Error:
root@superset-worker-54fcfd4b-bzpzd:/app# celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME
['/usr/local/bin', '/app/pythonpath', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app', '/app/configs/']
Loaded your LOCAL configuration at [/app/pythonpath/superset_config.py]
logging was configured successfully
2024-05-13 02:58:06,359:INFO:superset.utils.logging_configurator:logging was configured successfully
2024-05-13 02:58:06,361:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
Error: No nodes replied within time constraint
Comment From: fzhan
After done some digging, it appears to be related to the Celery version issue with Redis. https://github.com/apache/superset/pull/28471
Comment From: fzhan
After upgrade to celery 5.4, flower can see workers listed, but the ping for liveness within worker still fails with no nodes replied.
Comment From: fzhan
LivenessProbe still fails with 'Error: No nodes replied within time constraint' as result from "celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME"
Comment From: rusackas
@dosu-bot
Comment From: akkbng
Was this problem resolved? I'm also getting the LivenessProbe still fails with 'Error: No nodes replied within time constraint' as result from "celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME" error on the worker on startup
Comment From: dosubot[bot]
The worker instances keep restarting after upgrading to Superset 4.0.0-dev likely due to the upgrade of Celery and its dependencies packages as mentioned in the changelog:
- [#26604](https://github.com/apache/superset/pull/26604) feat(celery): upgrade celery and its dependencies packages (@Musa10)
This upgrade might have introduced changes or incompatibilities that are causing the worker instances to restart. Additionally, the UPDATING.md file notes that the Celery upgrade to 5.X resulted in breaking changes to its command line invocation. You may need to adjust your command line invocation and consider migrating your Celery config as per the new guidelines [1][2].
For the specific configuration changes required for Celery 5.X in the superset_config.py file, you can refer to the following example:
class CeleryConfig:
broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}"
imports = (
"superset.sql_lab",
"superset.tasks.scheduler",
"superset.tasks.thumbnails",
"superset.tasks.cache",
)
result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_RESULTS_DB}"
worker_prefetch_multiplier = 1
task_acks_late = False
beat_schedule = {
"reports.scheduler": {
"task": "reports.scheduler",
"schedule": crontab(minute="*", hour="*"),
},
"reports.prune_log": {
"task": "reports.prune_log",
"schedule": crontab(minute=10, hour=0),
},
}
CELERY_CONFIG = CeleryConfig
This configuration includes settings for the broker URL, result backend, task imports, worker prefetch multiplier, task acknowledgments, and beat schedule [3][4].
Additionally, ensure that your command line invocation for the worker is updated to match the new Celery 5.X requirements. For example:
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app worker"
Make sure to review and adjust your configurations accordingly to align with the new Celery 5.X guidelines.
To continue talking to Dosu, mention @dosu.
Comment From: fzhan
@dosu, in this case, what is your cache_config?
here's mine:
from superset.tasks.types import ExecutorType
CACHE_REDIS_URL = f"redis://:{env('REDIS_PASSWORD')}@{env('REDIS_HOST')}:{env('REDIS_PORT')}/1"
CACHE_CONFIG = {
'CACHE_TYPE': 'redis',
'CACHE_DEFAULT_TIMEOUT': 60 * 60 * 24,
'CACHE_KEY_PREFIX': 'superset_',
'CACHE_REDIS_URL': CACHE_REDIS_URL
}
DATA_CACHE_REDIS_URL = f"redis://:{env('REDIS_PASSWORD')}@{env('REDIS_HOST')}:{env('REDIS_PORT')}/2"
DATA_CACHE_CONFIG = {
'CACHE_TYPE': 'redis',
'CACHE_DEFAULT_TIMEOUT': 60 * 60 * 24, # 1 day default (in secs)
'CACHE_KEY_PREFIX': 'superset_results',
'CACHE_REDIS_URL': DATA_CACHE_REDIS_URL
}
FILTER_CACHE_REDIS_URL = f"redis://:{env('REDIS_PASSWORD')}@{env('REDIS_HOST')}:{env('REDIS_PORT')}/3"
FILTER_STATE_CACHE_CONFIG = {
'CACHE_TYPE': 'RedisCache',
'CACHE_DEFAULT_TIMEOUT': 24*60*60,
'CACHE_KEY_PREFIX': 'superset_filter_state',
'CACHE_REDIS_URL': FILTER_CACHE_REDIS_URL,
}
THUMBNAIL_CACHE_REDIS_URL = f"redis://:{env('REDIS_PASSWORD')}@{env('REDIS_HOST')}:{env('REDIS_PORT')}/4"
THUMBNAIL_CACHE_CONFIG = {
'CACHE_TYPE': 'RedisCache',
'CACHE_DEFAULT_TIMEOUT': 24*60*60*7,
'CACHE_KEY_PREFIX': 'thumbnail_',
'CACHE_REDIS_URL': THUMBNAIL_CACHE_REDIS_URL,
}
THUMBNAIL_SELENIUM_USER = "user@company"
THUMBNAIL_EXECUTE_AS = [ExecutorType.SELENIUM]
Comment From: dosubot[bot]
The cache_config for Superset is as follows:
# Default cache for Superset objects
CACHE_CONFIG: CacheConfig = {
"CACHE_TYPE": "RedisCache",
"CACHE_DEFAULT_TIMEOUT": 300,
"CACHE_KEY_PREFIX": "superset_",
"CACHE_REDIS_HOST": os.getenv("REDIS_HOST", "redis"),
"CACHE_REDIS_PORT": os.getenv("REDIS_PORT", "6379"),
"CACHE_REDIS_DB": os.getenv("REDIS_RESULTS_DB", "1"),
}
# Cache for datasource metadata and query results
DATA_CACHE_CONFIG: CacheConfig = CACHE_CONFIG
# Cache for dashboard filter state. `CACHE_TYPE` defaults to `SupersetMetastoreCache`
# that stores the values in the key-value table in the Superset metastore, as it's
# required for Superset to operate correctly, but can be replaced by any
# `Flask-Caching` backend.
FILTER_STATE_CACHE_CONFIG: CacheConfig = {
"CACHE_TYPE": "SupersetMetastoreCache",
"CACHE_DEFAULT_TIMEOUT": int(timedelta(days=90).total_seconds()),
# Should the timeout be reset when retrieving a cached value?
"REFRESH_TIMEOUT_ON_RETRIEVAL": True,
# The following parameter only applies to `MetastoreCache`:
# How should entries be serialized/deserialized?
"CODEC": JsonKeyValueCodec(),
}
# Cache for explore form data state. `CACHE_TYPE` defaults to `SupersetMetastoreCache`
# that stores the values in the key-value table in the Superset metastore, as it's
# required for Superset to operate correctly, but can be replaced by any
# `Flask-Caching` backend.
EXPLORE_FORM_DATA_CACHE_CONFIG: CacheConfig = {
"CACHE_TYPE": "SupersetMetastoreCache",
"CACHE_DEFAULT_TIMEOUT": int(timedelta(days=7).total_seconds()),
# Should the timeout be reset when retrieving a cached value?
"REFRESH_TIMEOUT_ON_RETRIEVAL": True,
# The following parameter only applies to `MetastoreCache`:
# How should entries be serialized/deserialized?
"CODEC": JsonKeyValueCodec(),
}
Superset uses Flask-Caching for caching purposes. Flask-Caching supports various caching backends, including Redis (recommended), Memcached, SimpleCache (in-memory), or the local filesystem. Custom cache backends are also supported.
For example, to configure the filter state cache using Redis:
FILTER_STATE_CACHE_CONFIG = {
'CACHE_TYPE': 'RedisCache',
'CACHE_DEFAULT_TIMEOUT': 86400,
'CACHE_KEY_PREFIX': 'superset_filter_cache',
'CACHE_REDIS_URL': 'redis://localhost:6379/0'
}
Dependencies for dedicated cache stores include: - For Redis: the redis Python package - For Memcached: the pylibmc client library
If the required caches are undefined, Superset falls back to using a built-in cache that stores data in the metadata database. For example, to use the built-in cache to store chart data:
DATA_CACHE_CONFIG = {
"CACHE_TYPE": "SupersetMetastoreCache",
"CACHE_KEY_PREFIX": "superset_results",
"CACHE_DEFAULT_TIMEOUT": 86400,
}
The cache timeout for charts can be overridden by settings for an individual chart, dataset, or database. Setting the cache timeout to -1 disables caching for charting data.
Caching for SQL Lab query results is configured using RESULTS_BACKEND and requires a cachelib object.
For caching thumbnails, enable the feature flag and configure a cache system and celery workers. For example, to store images on S3:
from flask import Flask
from s3cache.s3cache import S3Cache
class CeleryConfig(object):
broker_url = "redis://localhost:6379/0"
imports = (
"superset.sql_lab",
"superset.tasks.thumbnails",
)
result_backend = "redis://localhost:6379/0"
worker_prefetch_multiplier = 10
task_acks_late = True
CELERY_CONFIG = CeleryConfig
def init_thumbnail_cache(app: Flask) -> S3Cache:
return S3Cache("bucket_name", 'thumbs_cache/')
THUMBNAIL_CACHE_CONFIG = init_thumbnail_cache
THUMBNAIL_SELENIUM_USER = "Admin"
Using the above example, cache keys for dashboards will be superset_thumb__dashboard__{ID}. You can override the base URL for selenium using WEBDRIVER_BASEURL. Additional selenium web drive configuration can be set using WEBDRIVER_CONFIGURATION.
The CacheManager class in superset/utils/cache_manager.py is responsible for initializing and managing these caches. It ensures that the required caches are properly configured and initialized, falling back to the built-in cache if necessary. The ExploreFormDataCache class extends the Cache class to handle specific logic for explore form data, such as renaming data keys and adding default datasource types [1][2][3][4].
To continue talking to Dosu, mention @dosu.
Comment From: fzhan
Was this problem resolved? I'm also getting the
LivenessProbe still fails with 'Error: No nodes replied within time constraint' as result from "celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME"error on the worker on startup
Nope, I have tried "celery==5.5.0b2" still doesn't work
Comment From: PaleNeutron
superset-worker continuous OOM at 4.1.0rc2-py310 with 4Gi memory.
Comment From: surajsrivathsa
still facing same issue celery worker dies often and it is a problem to deploy superset on production
Comment From: ignasf5
on setup error (Error: No nodes replied within time constraint) remains by causing restarts. superset Version: 4.1.1 celery 5.4.0 (opalescent)
Comment From: rusackas
This one's been quiet for a while. Anyone still facing this?
Perhaps @dosu has some additional helpful thoughts by now :D
Comment From: dosubot[bot]
Yes, several users are still seeing the Celery worker liveness probe fail with "No nodes replied within time constraint" on recent Superset versions, including 4.1.1 and Celery 5.4.0. This is a known pain point, and the root cause appears to be a combination of how Kubernetes probes interact with Celery's process model and how Celery responds to inspect ping during startup, shutdown, or under load.
Key findings and actionable notes:
- The liveness probe is still configured by default to use
celery -A superset.tasks.celery_app:app inspect ping -d celery@$HOSTNAME, and this command can fail even when the worker is otherwise healthy and processing tasks. This is a limitation of Celery's inspect mechanism, not unique to Superset. source - Some users report that even with this error, Celery continues to function, and Flower can see the workers. The probe failure is often a false positive. source
- A workaround is to relax the liveness probe (increase
initialDelaySeconds,timeoutSeconds, andfailureThreshold) or to use a custom probe script that checks for the Celery process rather than relying oninspect ping. - For graceful shutdown and to avoid zombie workers, add a Kubernetes lifecycle
preStophook to send a Celery control shutdown signal:
lifecycle: preStop: exec: command: - /bin/sh - '-c' - celery --app=superset.tasks.celery_app:app control shutdown --destination=celery@$HOSTNAMEThis helps ensure the worker deregisters cleanly and avoids some probe-related restarts. source - If you use SQLite as the metadata DB, switch to PostgreSQL—async queries and Celery require it. source
There have not been major changes to the liveness probe logic in recent Superset releases; the Helm chart and probe command remain the same. The best current advice is to tune probe settings and use the preStop hook for graceful shutdown. If the worker is otherwise healthy and tasks are running, the probe failure may be safely ignored, but keep an eye on your actual task throughput and logs.
If anyone has found a more robust probe or workaround, please share!
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other 
