Skip to content

Legacy Graphite

Imported from Confluence

Content may be outdated. Verify before following any procedures. View original | Last updated: January 2026

Known Issues

No Data alerts

We are getting No Data alerts when graphite is down . It's still running on VM - graphite-core.vm-offerwall-prod-useast1.fyber.com.

In case you are not able to ssh we need to do hard reset from the console. In such case temporary folders will be removed from OS.

Info

For SSH access, ask the DevOps BLN team

Graphite store PID number of the process in the file in one of temporary folders. In order to fix.

  1. Check that file exists, if not - create folder and change owner.
cat /var/run/carbon/carbon1.pid
cat /var/run/carbon/carbon2.pid

mkdir /var/run/carbon/
chown -R graphite:graphite /var/run/carbon
  1. Start graphite process
supervisorctl -c /etc/supervisord/supervisord.conf start all
  1. Check logs
tail -f /var/log/graphite-carbon-cache-1_error.log
tail -f /var/log/graphite-carbon-cache-1.log

Graphite host out of disk space

SSH to the instance and clean some space

If the above doesn't help, it might be because Graphite machine has run out of space. Check under /storage/statsite  to get some clue about the files that are getting accumulated. For example:

cd /storage/statsite/counts/k8/engine
find . -maxdepth 1 -type d -name 'engine-*' -newermt "2018-01-01" ! -newermt "2025-05-20" -exec rm -r {} +

or:

# Confirm first that blacklisted IPs are okay to remove
cd /storage/statsite/counts/sws/rails/vm-ofw-core-offerwall-prod-useast1-0/blacklisted_ips
rm -r *
cd /storage/statsite/counts/sws/rails/vm-ofw-core-offerwall-prod-useast1-1/blacklisted_ips
rm -r *
cd /storage/statsite/counts/sws/rails/vm-ofw-core-offerwall-prod-useast1-2/blacklisted_ips/
rm -r *
...
# Continue with the rest of machines

HostOutOfMemory

Alert in #bln-escalation Slack channel:

image-2025-11-24_18-3-47.png

Probably you need to roload or restart httpd service on VM.

sudo su -
systemctl status httpd
systemctl reload httpd
systemctl restart httpd

List Failed services and reset

systemctl --failed
systemctl reset-failed