Hi ! I am currently trying to use a custom EVENT_LOGGER in superset, i added the code snippets from the official doc but it isn't working: I think the EVENT_LOGGER has changed and isn't the default DBEventLogger() anymore because we have some new logs on stdout, and there are no more "action logs" in the superset UI. However, my JSONStdOutEventLogger() isn't used either, i added some prints in the log method to see if it was even called, and it doesn't seem so.
Here are the code snippets i added:
event_logger.py: |-
from superset.utils.log import AbstractEventLogger
import json
class JSONStdOutEventLogger(AbstractEventLogger):
def log(self, user_id, action, *args, **kwargs):
records = kwargs.get('records', list())
dashboard_id = kwargs.get('dashboard_id')
slice_id = kwargs.get('slice_id')
duration_ms = kwargs.get('duration_ms')
referrer = kwargs.get('referrer')
print("bip boup 1")
for record in records:
log = dict(
action=action,
json=record,
dashboard_id=dashboard_id,
slice_id=slice_id,
duration_ms=duration_ms,
referrer=referrer,
user_id=user_id
)
print(json.dumps(log))
print("bip boup 2")
configFile: |-
...
from event_logger import JSONStdOutEventLogger
EVENT_LOGGER = JSONStdOutEventLogger()
Expected results
I expect the custom log method to be used (ie to see "bip boup" and "bip boup 2" on stdout)
Actual results
Some debug logs and logs on the requests done are on stdout (cf screenshot)
Screenshots
Environment
- superset version: 1.0.1
- python version: 3.8.8
- node.js version: 12
I hope i'm not missing something, thank in advance for your help !
Comment From: wiktor2200
Hello! Have you solved this issue somehow or found some workaround for that?
Comment From: bouleq
Hello, yes we managed to fix it. It turned out that logs were not written at the right place by the logging class. We added this to the code :
with open("/var/lib/superset/action.log","w") as f:
f.write("\n [LOG]: " + json.dumps(log) + "\n")
The full code snippet is now:
event_logger.py: |-
from superset.utils.log import AbstractEventLogger
import json
class JSONStdOutEventLogger(AbstractEventLogger):
def log(self, user_id, action, *args, **kwargs):
records = kwargs.get('records', list())
dashboard_id = kwargs.get('dashboard_id')
slice_id = kwargs.get('slice_id')
duration_ms = kwargs.get('duration_ms')
referrer = kwargs.get('referrer')
for record in records:
log = dict(
action=action,
json=record,
dashboard_id=dashboard_id,
slice_id=slice_id,
duration_ms=duration_ms,
referrer=referrer,
user_id=user_id
)
with open("/var/lib/superset/action.log","w") as f:
f.write("\n [LOG]: " + json.dumps(log) + "\n")
Comment From: wiktor2200
Thanks a lot for your answer. I think it's worth considering editing documentation here: https://github.com/apache/superset/blob/master/docs/docs/installation/event-logging.mdx and then it would be possible to close this issue. May I do it and link this issue to PR or do you want to open your own PR?
Comment From: bouleq
Sure you can do it, hope it can help people.
Comment From: wiktor2200
I've managed to do both JSONStdOutEventLogger and DBEventLogger at the same time. Now events are saved to logs in JSON format and they are still visible in Superset's dashboard in activity tab.
event_logger.py: |
from superset.utils.log import DBEventLogger
import json
class JSONStdOutEventLogger(DBEventLogger):
def log(self, user_id, action, *args, **kwargs):
records = kwargs.get('records', list())
dashboard_id = kwargs.get('dashboard_id')
slice_id = kwargs.get('slice_id')
duration_ms = kwargs.get('duration_ms')
referrer = kwargs.get('referrer')
super().log(user_id, action, *args, **kwargs)
for record in records:
if 'rison' in record:
del record["rison"]
log = dict(
action=action,
json=record,
dashboard_id=dashboard_id,
slice_id=slice_id,
duration_ms=duration_ms,
referrer=referrer,
user_id=user_id
)
print(json.dumps(log))
logger_settings: |
from event_logger import JSONStdOutEventLogger
EVENT_LOGGER = JSONStdOutEventLogger()
Comment From: wiktor2200
Hi! My last PR is merged, finally I've just updated docs. I think this issue could be closed now? @bouleq could you close it? Or is it anything more still needed for this issue?
Comment From: gugacavalieri
If anyone is coming here in 2024. This is what worked for us with the Superset Helm Chart 0.12.9 version: (We are running Superset in Kubernetes)
# values.yaml file:
configOverrides:
# We use a custom JSON logger here to output everything to stdout -> FluentBit
# Also, we prevent superset from using the default logger and flooding the DB with Logs
# More info: https://superset.apache.org/docs/configuration/event-logging/
logger_settings: |
# log everything as JSON
from superset.utils.log import AbstractEventLogger
import json
class JSONStdOutEventLogger(AbstractEventLogger):
def log(self, user_id, action, *args, **kwargs):
records = kwargs.get('records', list())
dashboard_id = kwargs.get('dashboard_id')
slice_id = kwargs.get('slice_id')
duration_ms = kwargs.get('duration_ms')
referrer = kwargs.get('referrer')
for record in records:
log = dict(
action=action,
json=record,
dashboard_id=dashboard_id,
slice_id=slice_id,
duration_ms=duration_ms,
referrer=referrer,
user_id=user_id
)
print(json.dumps(log))
# use JSON logger and disable logging to the DB Table
EVENT_LOGGER = JSONStdOutEventLogger()
This code creates a new logger that will log events as JSON in stdout and then prevent the DB Table from overflowing from time to time. The logs can then be flushed out from stdout to a search tool using a logger daemon.
If desired, I can help to update the Kubernetes Docs with this 🧡
Comment From: jgournet
@gugacavalieri : I tried your snippet (THANK YOU !) on superset v5 (I haven't tried it in v4), and I'm getting this error in all dashboards:
DB engine Error: Object of type SqlaTable is not JSON serializable
I've "fixed" it by changing to:
try:
print(json.dumps(log))
except:
print(log)
would you have any idea what could cause this ? Also: would you have a better solution ? I have no idea what I'm doing
Comment From: gugacavalieri
Hey @jgournet! Sorry, I haven't used Superset in a while, but I might have an idea on why that this error is showing.
The error is saying that something we are passing to json.dumps cannot be converted to JSON. I have a feeling that it could be the record itself that was changed in recent superset versions.
You solution works, but it will not log as JSON if that is a requirement for you. So you can try to change this line (remove the record):
log = dict(
action=action,
dashboard_id=dashboard_id,
slice_id=slice_id,
duration_ms=duration_ms,
referrer=referrer,
user_id=user_id
)
Or try to use Record as String (will give you better info in the logs):
log = dict(
action=action,
json=str(record),
dashboard_id=dashboard_id,
slice_id=slice_id,
duration_ms=duration_ms,
referrer=referrer,
user_id=user_id
)