Bug description

uv is not included in the lean superset image, so instructions such as https://github.com/apache/superset/blob/6612343f33fb88fd900b97a006a60876a12bf1fc/docs/docs/installation/kubernetes.mdx end up invalid.

Screenshots/recordings

No response

Superset version

4.1.1

Python version

Not applicable

Node version

Not applicable

Browser

Not applicable

Additional context

No response

Checklist

  • [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
  • [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
  • [x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Comment From: dosubot[bot]

Hello @kainz! I'm here to help you with any bugs, questions, or contributions you have for Apache Superset. Let's tackle this together!

The 'lean' Superset image does not include the 'uv' package, which is used to manage the Python environment in the installation process. This exclusion means that the standard installation instructions, which rely on 'uv', may not be applicable for the lean image. The lean image is designed to be a minimal version that does not contain database drivers, requiring users to install their own drivers for both analytics and metadata databases [1].

To address this issue, you may need to manually install the necessary Python packages and dependencies that are typically managed by 'uv' in the standard image. This could involve using alternative package management methods or modifying the installation scripts to accommodate the absence of 'uv'.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

Comment From: mistercrunch

As a quick workaround, recommending to simply git checkout {TAG} prior to running docker-compose. The core issue is around the fact that we mount the docker/ folder in the docker image when using docker-compose and the scripts in the local repo/branch may not be compatible with the image you're referencing. A more involved fix would bring the docker/* scripts inside the image so that they are aligned. Note that doing this in master now wouldn't fix older releases, but would provide good guarantees moving forward.

Another improvement would be to be more explicit about checking out the TAG prior to running docker-compose

Comment From: sfirke

I just ran into this, I am building my own Docker image successfully on top of 5.0.0rc1 and uv installing packages there. But when I switched to 4.1.2rc1 I had to change the Dockerfile to install the packages just with pip instead of uv.

The other thing that would help here is versioned docs, because you are seeing instructions that got introduced to the repo after 4.1.1. If you could see a 4.1.1 version of the docs it would say to just pip install - and then that, plus checking out the repo to the appropriate tag as @mistercrunch suggests, would do it.

Versioned docs should be in the pipeline with the Superset Improvement Proposal SIP-156: #32625

Comment From: bt-88

I just ran into this, I am building my own Docker image successfully on top of 5.0.0rc1 and uv installing packages there. But when I switched to 4.1.2rc1 I had to change the Dockerfile to install the packages just with pip instead of uv.

The other thing that would help here is versioned docs, because you are seeing instructions that got introduced to the repo after 4.1.1. If you could see a 4.1.1 version of the docs it would say to just pip install - and then that, plus checking out the repo to the appropriate tag as @mistercrunch suggests, would do it.

Versioned docs should be in the pipeline with the Superset Improvement Proposal SIP-156: #32625

Same here. I had to adjust the docker/docker-bootstrap.sh because it installs the packages inrequirements-local.txtvia uv.

if [ -f "${REQUIREMENTS_LOCAL}" ]; then
  echo "Installing local overrides at ${REQUIREMENTS_LOCAL}"
  # I added this if/else so it uses pip if uv is not available
  if command -v uv > /dev/null 2>&1; then
    # Use uv in newer images
    uv pip install --no-cache-dir -r "${REQUIREMENTS_LOCAL}"
  else
    # Use pip in older images
    pip install --no-cache-dir -r "${REQUIREMENTS_LOCAL}"
  fi
else
  echo "Skipping local overrides"
fi

Comment From: mistercrunch

Yup, solution here is to bundle the docker/ scripts INSIDE the image (instead of mounting them), but would only fix this family of issues moving forward - unless we go in previous release branches and change the Dockerfile/docker-compose files there...

Also noting that including more scripts in the Docker images themselves might have security implications, we'd have to be cautious around making them readonly, and maybe segmenting between those that can be run by root vs the superset user. Personally not planning on taking this on. Best solution/workaround is to checkout the branch related to the image you're working with/on.

Comment From: rusackas

Anyone here interested in opening a PR to resolve this? If not, I think we might just call the question answered and close the Issue as "not planned"

Comment From: kainz

Anyone here interested in opening a PR to resolve this? If not, I think we might just call the question answered and close the Issue as "not planned"

Best solution/workaround is to checkout the branch related to the image you're working with/on.

I believe there is a root problem here in that if you are not building a custom image, the standard images do not work with the helm chart. I can dig a bit later (tomorrow, probably) and produce a reproduction with a better illustration of the issue.

I think a good question to ask is: Is the superset team invested in Kubernetes instructions that work with a wide array of possible configurations, or only a single prototypical or example configuration?

If the answer is to support the generic case, either the docs should be adjusted, or the 'lean' image should have uv added.

  • For the docs side, is a different distributed image a better fit than the lean image?
  • On the image adjustment side, is the cost of adding uv to the 'lean' image untenable for some reason?

If the answer is to only support the specific case, whatever that case is, and its narrower applicability should be noted in the documentation, I believe.

Comment From: mistercrunch

Oh right, my comment assumed [wrongly] that it was a docker-compose-related issues where clearly the issue is about k8s/helm.

I think a good question to ask is: Is the superset team invested in Kubernetes instructions that work with a wide array of possible configurations, or only a single prototypical or example configuration?

Totally fair. On the www.preset.io side (represents a fair % of the active/sponsored committers), we forked the helm chart a long time ago and aren't using the helm chart in the repo, so we're not actively maintaining it. The reality is that it's extremely difficult to support the variety of custom requirement everyone has in their own environments (different metadata database, need for different analytics database drivers, different MQ systems for Celery, different observability stacks, ...). All this becomes really hard to make fully configurable, especially given that Superset is a fairly complex collection of services that are individually highly configurable...

My take, based on the activity in the repo, is that we should clarify a limited commitment on the "single prototypical or example configuration" (as you put it), or a simple/vanilla "reference implementation". Essentially assuming that you should probably just fork the helm chart and customize it to suit your needs.

Tagging @villebro as I believe he's been most active around the helm chart and might be using it for his organization's deployment.