Beanstalkd app engine publisher callback failures¶

Archived (pre-2022)

Preserved for reference only -- likely outdated. View original | Last updated: November 2018

Alert Name:	Interstitial-Consumer Offset Lag alert
Notification Source:	Grafana, check_mk, OpsGenie
Component:	AD Delivery
Default Priority:	Medium
Investigation Process / Resolution:	Graphs: Callback's failures among all products Callbacks are not piling up Please refer to this article - Investigation of server-side callback failures Screenshot 2018-11-11 at 23.47.50.png Temporary fix is to increase number of workers per server, by default it is 40
`ssh engine@wrk001.prd.fyber.com cd current ENGINE_ENV=production bundle exec rake gearbox:publisher:dispatch_worker[100]`
Suggestion is to run multiple time same process in the screen mode (5 times for 100 workers on each server) Permanent solution is to ask developers to deploy engine with more workers. Example of change: config/workers.rb
`dispatcher: { rake_task: 'gearbox:publisher:dispatch_worker', threads: 100, processes: 70 },`
engine (Github)
Reference Incident:	INC-2749