Opened 11 years ago

Closed 7 years ago

#70 closed task (invalid)

Automatic restart of apache

Reported by: tobixen Owned by: tobixen
Priority: major Milestone:
Component: ServerSetup Keywords:

Description (last modified by tobixen)

We have experienced three times that apache has been eating too much memory, last time the server got so bloated that it was impossible to do anything for a whole hour, until the OOM killer finally kicked in.

My ad-hoc workaround idea is to create a script and doing something like:

while true:
  if apache is bloated:
     restart apache
     send email about it
     sleep 5 minutes
  sleep 30 seconds

The script should probably run with negative nice value, or protected from swapping somehow.

The algorithm and treshold level for "apache is bloated" has to be refined.

The "restart apache"-part has to make sure we do "kill -9" on all pids that won't terminate cleanly; probably it's best to run "/etc/init.d/httpd restart", sleep for a while, and then do kill -9 on all old apache processes.

Of course, we should also investigate why apache eats so much, and get it fixed - but that's another ticket.

Change History (3)

comment:1 Changed 11 years ago by anonymous

apache bloat detection could be a bit hard. What do you think?

comment:2 Changed 11 years ago by tobixen

  • Description modified (diff)
  • Owner changed from hannu to tobixen

I haven't given it that much a thought, but it shouldn't be that difficult to sum up the amount of memory used by apache.

Another idea (as commented in #73) is to periodically restart apache, i.e. 00:00 UTC every day.

Or a combination (the ordinary restart script to run every night, and the extraordinary restart script would hopefully never run).

Periodically restarts is a breeze to set up, I'll just throw it into /etc/cron.daily/ ... done.

I still think we should have that extraordinary script to actually check the memory consumption of apache.

comment:3 Changed 7 years ago by jsfan

  • Resolution set to invalid
  • Status changed from new to closed

Ticket was opened for deer. Closing because not relevant on puma/cub for now. If ever this problem resurfaces, Apache could run in daemontools/runit with memory limiting.

Note: See TracTickets for help on using tickets.