Diaspora startup script / status cron job

EDIT: Further below is a more sophisticated version, suitable for setups that run multiple thin servers.

I found my Diaspora process missing, with no clues in the logs as to why. Hmm - think I forgot to use nohup. Anyway, I just wrote a cron job to check on the status of its child processes.

This also doubles as the initial script to call up Diaspora and get it running. It checks to see if all processes are running and if one is missing then it kills the remaining processes and starts Diaspora again. Cron will also send an email if it finds Diaspora not running.

I'm using Thin, but you can modify this to match your setup. Many thanks to justin at ser.endipito.us (https://ser.endipito.us/u/justin) for the mentoring.

I have the script saved as diasp_check and call it every five minutes with:

*/5 * * * * /home/clancy/scripts/diasp_check cron

in my crontab.

Single Thin Server Instance

#!/bin/bash

### options:
# path to diaspora script/server
  DIASPORA_PATH=/home/diaspora/script/server
# email for notifications
  EMAIL=me@example.com
# IMPORTANT: cron needs to know your path. Find out by typing "echo $PATH" in a terminal, and paste it here
  export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
###

RESQUE="$(ps ax | grep resque | awk '{print $1}')"
WEBSOCKET="$(ps ax | grep websocket_server | awk '{print $1}')"
THIN="$(ps ax | grep "thin server" | awk '{print $1}')"
DIASPORA="$(ps ax | grep $DIASPORA_PATH | awk '{print $1}')"

if [ "$RESQUE" != "" ] && [ "$WEBSOCKET" != "" ] && [ "$THIN" != "" ] && [ "$DIASPORA" != "" ] ; then
	# only echo if not called by cron (no args)
	if [ "$1" == "" ] ; then
		echo "Everything fine. I'm done here."
	fi
	exit 0;
fi

# from here on echo, for /var/mail records
STATUS="resque:'$RESQUE' websocket:'$WEBSOCKET' thin:'$THIN' diaspora:'$DIASPORA'"
echo "Status is $STATUS"
echo "Something is wrong. Killing old processes (if any).";

if [ "$RESQUE" != "" ] ; then
	echo "killing resque..."
	kill $RESQUE
fi

if [ "$WEBSOCKET" != "" ] ; then
	echo "killing websocket..."
	kill $WEBSOCKET
fi

if [ "$THIN" != "" ] ; then
	echo "killing thin..."
	kill $THIN
fi

if [ "$DIASPORA" != "" ] ; then
	echo "killing diaspora..."
	kill $DIASPORA
fi

echo "starting diaspora..."
nohup $DIASPORA_PATH &

# send an email if we're not cron
if [ "$1" != "" ] ; then
	echo "The Diaspora process ... isn't. It is now though. Status: $STATUS." | /usr/bin/mail -s "DIASPORA DOWN" "$EMAIL"
fi

Multiple Thin Servers

If you use many thin servers, then you don't need to grep for your diaspora script (it isn't there), but instead you need a while loop to kill all your thins. I now use 3 thin servers, so this is my current script.

It has benefited from a little more fine tuning than the one above, and you can also call it with "status" to see if Diaspora is up, down, or starting up.

#!/bin/bash

### options:
# path to diaspora script/server
  DIASPORA_PATH=/home/diaspora/script/server
# email for notifications
  EMAIL=dope_geek@example.com
# IMPORTANT: cron needs to know your path. Find out by typing "echo $PATH" in a terminal, and paste it here
  export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
###

BASENAME=`basename $0`;

if [ "$1" != 'start' ] && [ "$1" != 'stop' ]  && [ "$1" != 'status' ] && [ "$1" != 'cron' ] ; then
	echo "$BASENAME: invalid argument(s) '$1'";
	echo "Usage: $BASENAME [start|stop|status|cron]";
	echo "	- start   : check for diaspora processes; if any absent, kill remaining and start diaspora again";
	echo "	- stop    : kill all diaspora processes and exit";
	echo "	- status  : do quick check of processes, print and exit";
	echo "	- cron    : for cron only. Like [start], checks everything is ok and restarts if not.";
	exit 1;
fi

# so let's check for diaspora processes
RESQUE="$(ps ax | grep resque | awk '{print $1}')"
WEBSOCKET="$(ps ax | grep websocket_server | awk '{print $1}')"
THIN="$(ps ax | grep "thin server" | awk '{print $1}')"

# status string giving pids
STATUS="- resque: $RESQUE\n- websocket: $WEBSOCKET\n- thin: $THIN"

# up or down? or starting up?
D="down"
DSTART=
TSTART=
if [ "$RESQUE" != "" ] && [ "$WEBSOCKET" != "" ] && [ "$THIN" != "" ] ; then
	D="up"
else
	DSTART="$(ps ax | grep "$DIASPORA_PATH" | awk '{print $1}')";
	TSTART="$(ps ax | grep "thin start" | awk '{print $1}')";
fi

if [ "$1" == "status" ] ; then
	if [ "$D" == "up" ] ; then
		echo "All cool. Processes up:";
		echo -e $STATUS;
		exit 0;
	fi
	if [ "$DSTART" != "" ] || [ "$TSTART" != "" ] ; then
		echo "Diaspora is down but appears to be starting up. Processes up:";
		echo -e $STATUS;
		exit 0;
	fi
	echo "Diaspora is down.";
	exit 0;
fi

# If cron, check to see if manual restart is taking place
if [ "$1" == "cron" ] ; then

	START="$(ps ax | grep "$0 start" | awk '{print $1}')";
	STOP="$(ps ax | grep "$0 stop" | awk '{print $1}')";

	if [ "$START" != "" ] || [ "$STOP" != "" ] || [ "$DSTART" != "" ] || [ "$TSTART" != "" ] ; then
		# echo to /var/mail/~
		echo "$BASENAME cron found manual restart in progress. Aborted.";
		exit 0;
	fi
fi

# if diaspora is up and we're not stopping it then we're done.
if [ "$1" != 'stop' ] ; then

	if [ "$D" == "up" ] ; then
		# cron succeeds silently
		if [ "$1" != "cron" ] ; then
			echo "Everything is fine. I'm done here."
		fi
		exit 0;
	fi

	echo "Diaspora is down.";
fi

# WEBSOCKET HACK #######################
# Currently, websocket has a problem and regularly dies
# Monitor /var/mail/you for restart attempts. If none appear, remove this patch

if [ "$1" == "cron" ] && [ "$RESQUE" != "" ] && [ "$THIN" != "" ] ; then

	SCRIPT=$(dirname $DIASPORA_PATH);
	DIASPO=$(dirname $SCRIPT);

	cd $DIASPO

	RAILS_ENV=production nohup bundle exec ruby ./script/websocket_server.rb &

	echo "Attempted to restart websocket";
	exit 0;

fi

# END WEBSOCKET HACK #####################

# Kill kill kill.

if [ "$1" != 'start' ] ; then
	echo "Processes found:"
	echo -e $STATUS;
fi
echo "Killing old processes (if any) ...";

if [ "$WEBSOCKET" != "" ] ; then
	echo "- killing websocket..."
	kill $WEBSOCKET
fi

while [ "$THIN" != "" ] ; do
	echo "- killing thin..."
	kill $THIN
	sleep 1
	THIN="$(ps ax | grep "thin server" | awk '{print $1}')"
done

while [ "$RESQUE" != "" ] ; do
	echo "- killing resque..."
	kill $RESQUE
	sleep 1
	RESQUE="$(ps ax | grep resque | awk '{print $1}')"
done

echo "done."

if [ "$1" == 'stop' ] ; then
	exit 0;
fi

echo "starting diaspora..."
nohup $DIASPORA_PATH &

# send an email if we're cron
if [ "$1" == "cron" ] ; then
	echo -e "I just found Diaspora asleep at the wheel. I kicked it up the arse. \nProcesses found: \n$STATUS." | /usr/bin/mail -s "DIASPORA DOWN from $0" "$EMAIL"
	
else
# else be helpful at the command line
	echo "";
	echo "> Startup may take a minute or so. Do '$BASENAME status' to see how things are going.";
	echo "> Once up, thin may yet take a few more seconds to respond to HTTP requests.";
	echo "";
fi

exit 0;