On Superset 1.5.1 we have ca. 50 reports enabled that are mostly sent on a daily basis. Sometimes, a report is sent out mutliple times without an apparent reason (s. screenshot of the Report Execution Log), even though the report is set to only send it out once a day (s. screenshot).
This behavior has been seen on different reports on different days, with no apparent pattern. Just randomly, it seems that Superset decides to send out a report several times. The only notable circumstance is that this happens after the scheduled time.
How to reproduce the bug
- Set up a report with a schedule for once a day
Expected results
Each report is sent out according to schedule
Actual results
Some reports are sent out multiple times at random (but all around the scheudled time)
Screenshots
Environment
(please complete the following information):
- browser type and version: Any browser
- superset version: 1.5.1
Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [ ] I have reproduced the issue with at least the latest released version of superset.
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.
Additional context
Add any other context about the problem here.
Comment From: devaale
@mattitoo It's not related to your problem - but could you share information if you're able to send reports of charts as png with this version of superset ?
Comment From: mattitoo
@devaale Yes, it works nicely.
Comment From: devaale
@mattitoo By any chance you could share your cloned superset repo as public repository of your own ? I'm having difficulty getting png using API with the master branch.
Comment From: mattitoo
For organisational reasons I can not do that right now. But we did not implement any changes on the Reporting features or API whatsoever.
Comment From: devaale
Mhm - understandable, no changes required in superset_config.py file either - in order for API endpoint api/v1/chart/{pk}/cache_screenshot or reporting when sending chart as .PNG to work, right ? @mattitoo Thank you - i'll try out 1.5.1 version.
Comment From: chathawee
I am facing this issue too. Some day it sent out duplicately and some day did not sent
Comment From: mattitoo
We found out that is a timing issue when generating reports. Basically, every 60 seconds there is a check if a report is scheduled but not finished yet. But, it is not taken into account whether the report was triggered, but not finished yet. So, if reports take too long, they are not registered as finished, and the report is triggered again. Then, the first report is sent out (because it is finished now) and the second one as well.
Comment From: chathawee
@mattitoo Could you please suggest? Do you have any idea how to fix it?
Comment From: unnyns-307
Is this related to caching timeout in redis/celery worker? I found some related issue about celery worker that duplicate task https://github.com/celery/celery/issues/3270 and there is some comment suggested to extend VISIBILITY_TIMEOUT config of celery but I'm not sure where we could apply it in Superset
Comment From: mattitoo
We just upgraded to Superset 2.0.1 and wanted to see if the problem persists there. If it does, we will have a look at a possible fix.
Comment From: mdeshmu
@mattitoo we are facing same issue. did upgrade to 2.0.1 solve your problem?
Comment From: mattitoo
No, unfortunately this still happens.
Comment From: mdeshmu
For us, This was caused by value of visibility timeout being lower than time taken by task to complete the job. Increasing the sqs queue's visibility timeout stopped the duplicates.
Comment From: unnyns-307
Hi, we also tried extend VISIBILITY_TIMEOUT for celery and it resolved this issue. The report stop duplicating or skipping. Thank you @mdeshmu for your suggestion and thank you all for discussion on this issue. Cheers!
Comment From: rusackas
I'm tempted to close this as completed based on what I'm reading... is there anything that needs to be added to the docs and/or comments in config files so we can rest easier about doing so?
Comment From: zhaoyongjie
Hey @rusackas, as @unnyns-307 mentioned, we resolved this issue by appending the following line to the config.py file: broker_transport_options = {'visibility_timeout': 18000}. The snippet below might be helpful for other users experiencing the same issue.
class CeleryConfig(object):
broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
broker_transport_options = {'visibility_timeout': 18000}
imports = ("superset.sql_lab", "superset.tasks", "superset.tasks.thumbnails")
result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
Comment From: MONOLIFE-12
Hi @rusackas , @unnyns-307 , @zhaoyongjie , can someone help me?
I'm use as @unnyns-307 mentioned, but it does not resolve this issue.
broker_transport_options = {'visibility_timeout': 18000}
please help.
Comment From: sfirke
I looked at the Celery docs on Visibility Timeout and it says the default value is one hour. I take that to mean that you would only encounter the problematic behavior in this issue if your reports sometimes take over an hour to run -- can anyone confirm that was the situation or give a runtime value for one of the reports that was being sent in a loop?
I think this should be added to the Alerts & Reports documentation, but not the default config.py. One hour seems like a reasonable default. The "caveats" section of those docs say that increasing this value can have the negative side effect of reports being excessively delayed if Celery is restarted.
Would anyone in this thread be willing to contribute a brief PR to the Alerts & Reports docs page so we can close this? I think it should be a commented-out line of code in the config showing how to set a valid value in a way that works with the current version of Superset, and then another comment above it saying something like:
# if you have long-running reports that are being resent in a loop, extend the visibility timeout per https://github.com/apache/superset/issues/22664"
Comment From: wuqicyber
They are different task Ids. Have you started multiple Celery Beat Instance?
Comment From: mohitpaddhariya
@rusackas could you please assign me that issue?
Comment From: Payoshnik123
@mattitoo could you please assign me this issue?
Comment From: sfirke
@Payoshnik123 anyone is free to work on this issue, it has been open for two years. I'll assign the issue to you, assuming that means you'll be sending a pull request in the near future or post here if you're unable to do so.
Comment From: nkdanta1
facing same issue. is there any update on fix ?
Comment From: zoetw88
Hi @sfirke, I noticed this issue is still open and the problem persists. If @Payoshnik123 hasn’t had a chance to follow up, I’d be happy to take this on and submit a PR this week with visibility_timeout guidance added to the Alerts & Reports documentation.
Let me know if that works — happy to help move this forward. Thanks!
Comment From: sfirke
Thank you @zoetw88! That would be appreciated. You can ping me on the PR when it's ready for review or if you have questions.
Comment From: devyash786
Hi @yousoph @zoetw88 , Any updates on this issue? any ETA?
Comment From: nicolavolpini
They are different task Ids. Have you started multiple Celery Beat Instance?
This was indeed our problem. We run Superset as an AWS ECS deployment and had 2 Celery Beat instances running. Once dropped down to 1, duplicate reports stopped.
It is also recommended in the Superset docs:
There should only be one instance of celery beat running in your entire setup. If not, background jobs can get scheduled multiple times resulting in weird behaviors like duplicate delivery of reports, higher than expected load / traffic etc.
Source: https://superset.apache.org/docs/configuration/async-queries-celery/
Might not be the root cause for everyone but I thought it would be beneficial to report the solution nevertheless, for anyone who is reaching this ticket via web search.
Comment From: Nafizul02
Hi. Is this issue still open? I would like to contribute.
Comment From: rusackas
Anyone can pick this up, but I assigned to @Nafizul02 just since they're the last to volunteer. First PR to merge wins! :D
Comment From: YousufFFFF
Hi @rusackas / maintainers — I can work on this issue. Plan: first I’ll add extra logging to confirm whether duplication happens at the scheduler or worker level, then add a short-lived lock (Redis or DB advisory lock) in the report dispatcher to prevent duplicate sends. I’ll include tests and a short deployment note about ensuring a single scheduler/beat. If you want, assign this issue to me and I’ll open a PR with an initial instrumentation patch first.
Comment From: YousufFFFF
To clarify, I’ll start by adding instrumentation/logging first to confirm where the duplication originates (scheduler or worker). Once the behavior is clearly reproduced, I’ll propose the exact fix before implementing it. Happy to open a draft PR for early feedback.
Comment From: sfirke
The reason this is marked "good first issue" is that we're only looking for a short documentation change. Please see my comment above: https://github.com/apache/superset/issues/22664#issuecomment-2269191492
@YousufFFFF you are welcome to send that docs PR if you like
Comment From: YousufFFFF
@sfirke Thanks for the clarification! I’ll go ahead and prepare the documentation PR as suggested. Let me know if there’s anything specific you’d like me to include.