dbs00* high cpu usage¶
Archived (pre-2022)
Preserved for reference only -- likely outdated. View original | Last updated: February 2019
High CPU usage on master or slave DB instances could be cause of specific user is trying to perform attack or spamming. Solution for this is to block user id and IP from which are spamming.
In addition we need to kill ancient processes to stop a loop which we are having in application. Login to all ecl severs ecl00*.prd.fyber.com
Search for a processes from user offer, kill processes that are older then others, e.g. kill processes from Feb07
IMPORTANT: Leave process Passenger AppPreloader: /var/app/offer/current, it should exists to spin up new workers.
root@ecl003 13:29:33 ~ # ps aux | grep offer
offer 6373 0.0 0.0 527508 68324 ? Sl 06:13 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 9213 12.0 9.1 12516428 12077168 ? Sl Feb07 162:24 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10556 0.0 0.0 527796 62500 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10586 0.0 0.0 527796 62508 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10610 0.0 0.0 527796 62500 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10629 0.0 0.0 527796 62504 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10664 0.0 0.0 527796 62508 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10693 0.0 0.0 527796 62504 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10704 0.0 0.0 527796 62504 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10720 0.0 0.0 527796 62500 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10734 0.0 0.0 527796 62504 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10766 0.0 0.0 527796 62508 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10803 0.0 0.0 527796 62512 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10812 0.0 0.0 527796 62516 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10821 0.0 0.0 527796 62508 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 10849 0.0 0.0 527796 62500 ? Sl 13:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
root 10861 0.0 0.0 112648 976 pts/0 S+ 13:29 0:00 grep --color=auto offer
offer 16091 0.0 0.0 527604 68468 ? Sl 07:29 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 21030 0.0 0.0 527604 65524 ? Sl 09:44 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 25185 0.0 0.0 527124 69176 ? Sl Feb07 0:01 Passenger RubyApp: /var/app/offer/current/public (production)
offer 26956 0.0 0.0 392612 61260 ? Sl Feb07 0:06 Passenger AppPreloader: /var/app/offer/current
offer 27356 14.1 9.6 13237936 12740472 ? Sl Feb07 175:28 Passenger RubyApp: /var/app/offer/current/public (production)
offer 27406 23.5 4.4 6557952 5907060 ? Sl Feb07 292:28 Passenger RubyApp: /var/app/offer/current/public (production)
offer 27424 5.8 8.9 12256436 11771448 ? Sl Feb07 71:59 Passenger RubyApp: /var/app/offer/current/public (production)
offer 27434 5.8 8.5 11932128 11297656 ? Sl Feb07 73:08 Passenger RubyApp: /var/app/offer/current/public (production)
offer 27445 3.8 0.0 0 0 ? Zl Feb07 47:26 [ruby] <defunct>
offer 27469 1.8 0.0 0 0 ? Zl Feb07 23:15 [ruby] <defunct>
offer 34530 0.0 0.0 527700 65796 ? Sl 13:00 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 38013 0.0 0.0 527316 68512 ? Sl 02:46 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 41459 0.0 0.0 527700 65868 ? Sl 11:12 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 42525 0.0 0.0 527412 68968 ? Sl 03:52 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 44254 0.0 0.0 527412 67492 ? Sl 05:59 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
offer 46735 0.0 0.0 527700 62980 ? Sl 13:14 0:00 Passenger RubyApp: /var/app/offer/current/public (production)
This should help to reduce CPU usage.
NOTE: contact ofw dev team to block ID and IP address of the attacker.
Similar incidents INC-2805 and INC-2826