Skip to content

[Alerting] Business: User Reward Delays

Archived (pre-2022)

Preserved for reference only -- likely outdated. View original | Last updated: May 2019

Alert example in slack:

image2018-8-22_12-8-6.png

image2018-8-22_12-8-49.png

image2018-8-22_12-9-45.png

Solution:

Restart local kafka on all workers (example with ansible, you can use cssh for this):

ansible all -i workers.ini -b -m shell -a "systemctl status kafka-server"

> cat workers.ini
wrk001.prd.fyber.com
wrk002.prd.fyber.com
wrk003.prd.fyber.com

After restart check /var/log/kafka/server.log that kafka was able to start because sometimes there could be corrupted topics you need to clean so kafka can start.