wiki:DeerToPuma

In June 2012 we ordered a new server at Hetzner. The new server is called Puma. The current live server Deer will be migrated to Puma on 22 June 2012 (see blog post). This page documents the progress. There is also a pad with a detailed log of geekisms.

How to test

Before DNS was switched the new environment could only be tested by following these instructions:

Add this to your hosts file:

5.9.10.110 www.bewelcome.org alpha.bewelcome.org

Afterwards you can access the BeWelcome website on the new server as usual by visiting http://www.bewelcome.org. You may need to restart your browser or operating system to make this work. If you want to visit the old live website, simply remove the above line from your hosts file again.

Thoroughly test all features and report findings in our IRC channel #bewelcome. All changes you make at this point (forum posts, friend requests etc.) will disappear again later today or tomorrow, 24 June.

Known issues

Migration steps

People responsible in square brackets. Add "DONE:" in front of items once they are done.

  1. DONE: Prepare Puma (see #1664, #1665 and possibly other tickets) [jsfan]
  2. DONE: Clone Rox code from master (www) and develop (alpha) [jsfan]
  3. DONE: Copy live database dumps from Deer to Puma for testing [jsfan]
  4. DONE: Copy and adjust Rox and Apache config files [jsfan, planetcruiser]
  5. DONE: Test all features thoroughly (with modified hosts, see "How to test" above) [jsfan, planetcruiser, matthias, others], special attention to:
    • Email features
    • Map features
    • HTTPS
    • Registration progress (verification link etc.) [matthias]
    • alpha.bewelcome.org
  6. DONE: Install and test cronjobs
    • 00 02 * * * bwbackup /home/bwbackup/db_backup.sh (DB backup) [jsfan]
    • 0 * * * * bwscripts /home/bwrox/www.bewelcome.org/setup/production_scripts/cronscripts/clean_guestsonline.sh (anonymous session clean-up) [jsfan]
    • 0,30 * * * * bwscripts /home/bwrox/www.bewelcome.org/setup/production_scripts/cronscripts/close_edit_forum.sh [jsfan]
    • */5 * * * * root /usr/local/bin/diskspace.sh [jsfan]
    • */5 * * * * bwrox cd /home/www/www.bewelcome.org ; env SERVER_NAME=www.bewelcome.org php /home/www/www.bewelcome.org/htdocs/bw/mailbot.php (mailbot) [planetcruiser]
    • 5 2 * * * bwrox /home/bwrox/www.bewelcome.org/setup/production_scripts/update_geo.sh www.bewelcome.org (Geonames import)
    • 0 7 * * * bwrox cd /home/bwrox/www.bewelcome.org ; env SERVER_NAME=www.bewelcome.org php /home/bwrox/www.bewelcome.org/htdocs/bw/updatestats.php (generate user statistics) [jsfan]
  7. DONE: Place holder page on www and alpha on Deer: "We will be back in 10 minutes" (state date and time) [jsfan]
    • Design place holder [jsfan]
    • Show place holder for all sub pages (url rewrite)
    • Activate placeholder
      • sudo a2dissite www.bewelcome.org alpha.bewelcome.org staging.bewelcome.org && sudo a2ensite placeholder-www.bewelcome.org && sudo /etc/init.d/apache2 reload
  8. DONE: Make sure all DB requests have been written to Deer MySQL
    • Wait a few minutes
  9. DONE: Make sure mailbot sends all pending messages (wait 5 minutes or start manually), these MySQL queries on Deer BW_MAIN should return 0 rows:
    • select * from messages where status="ToSend" and created>date_sub(now(),interval 10 minute);
    • select * from posts_notificationqueue where status="ToSend" and created>date_sub(now(),interval 10 minute);
  10. DONE: Disable mailbot cronjob [planetcruiser]
  11. DONE: Put Deer MySQL in read-only for last dumps
  12. DONE: Copy live DBs from Deer to Puma (dump and import) [jsfan]
    • After import, change routine definers to root@localhost
  13. DONE: Sync data directory from Deer to Puma (from puma) [jsfan]
    • rsync --delete-before -uvrl root@deer.bewelcome.org:/home/bwrox/www.bewelcome.org/data/ /var/rox/data/ && find /var/rox/data/ -user root -exec chown www-data:www-data {} \;
  14. DONE: Change DNS entry for www and alpha from Deer to Puma [planetcruiser]
  15. DONE: Clear DNS cache on Deer:
    • sudo rndc flushname www.bewelcome.org
  16. DONE: Add ProxyPass to Deer apache for www to redirect users with outdated DNS
    • Activate proxy
      • sudo a2dissite placeholder-www.bewelcome.org && sudo a2ensite proxied-www.bewelcome.org && sudo /etc/init.d/apache2 reload
    • Needed bind restart on Deer
    • SSL vhost needed SSLProxyEngine On
  17. DONE: Watch various logs on Puma and repair :)
  18. DONE DONE! :)

Things to do after migration of www

  • DONE: Fix deployment script [planetcruiser]
  • DONE: Change DNS for download.bewelcome.org and possibly other sub domains [planetcruiser]
  • DONE: Stop cronjobs on Deer
  • DONE: Verify backup cronjob on Puma [planetcruiser]
  • DONE: Rsync Deer to big partition on Puma [jsfan]
  • DONE: Have one last look through Deer to see if we forgot to migrate anything [planetcruiser]
  • Shred /home partition on Deer [planetcruiser]
  • Shred all partitions on Deer via rescue mode [jeanyves]
  • Shut down Deer [jeanyves]
  • DONE: Cancel (don't renew) contract for Deer [jilrev]
  • Delete worst useless disk space hogs one by one in Deer backup on Puma [planetcruiser]
  • Create repository for production scripts, configs etc. for Puma [planetcruiser]
  • Set up bullet-proof backup for Puma
Last modified 5 years ago Last modified on Jun 28, 2012 2:19:16 PM