PHP MogileFS Client from Erik Osterman modified

I am struggling with a high latency network, and in many cases the tracker simply doesn't respond in time, and the client throws a fatal error. So I did some modifications on the client from http://osterman.com/wordpress/2007/06/30/robust-php-mogilefs-client

Changelog:
* setFile now tries upload 10 times before failing - example; tracker dies during upload
* setResource with key == null now close file handle before throwing exception
* added pathcount and noverify options to getPaths (copied from perl client 1.08)
* unknown key, and empty file now produce a warning, and not fatal error in doRequest
* 0byte check now before upload, and not relying on tracker

My plan was to make it slightly more tolerant for errors from tracker.

Download it here: PHP MogileFS Client

Install MogileFS 2.34 from CPAN on Ubuntu 8.04 LTS with init scripts

This guide is very similar to my 'installing from source' post, but installing from CPAN makes more sence, since updates are issued there - making it much easier to upgrade using CPANs 'install MogileFS::Server' directly

In a perfect world, it should to just issue:

cpan
install MogileFS::Server

But version 2.34 is still imperfect..

On a clean Ubuntu install, start off by getting compilers etc.:

sudo apt-get install build-essential

Enter cpan, and install the following BEFORE mogilefs

cpan
install IO::AIO BSD::Resource

In that process the optimal would be to install DBD::mysql also - but that one failed for me, so go install the package from ubuntu repository instead:

sudo apt-get install libdbd-mysql-perl

Now you're ready to run the actual install command without it failing on you:

cpan
install MogileFS::Server

If you want additional features, you can install MogileFS::Client and MogileFS::Utils also

Now you have your mogstored and mogilefsd in /usr/local/bin/mogilefsd, but that's pretty much all you have. For them to be useful you would need config files:

Here's mine from /etc/mogilefs/mogilefsd.conf :

db_dsn = DBI:mysql:mogilefs:host=server1.example.com;port=3306;mysql_connect_timeout=10;mysql_ssl=1;mysql_compression=1
db_user = mog_user
db_pass = mog_pass
listen = 0.0.0.0:7001
node_timeout = 5
conf_port = 7001
listener_jobs = 10
delete_jobs = 1
replicate_jobs = 10
mog_root = /mnt/mogilefs
reaper_jobs = 1
rebalance_ignore_missing = 1

and from /etc/mogilefs/mogstored.conf :

maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /var/mogdata

Create users you want mogilefsd and mogstored to run under:

sudo adduser mogilefsd
sudo adduser mogstored

My init.d script is under /etc/init.d/mogilefsd and modified from an older init script found in a custom repository:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          mogilefsd
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start/Stop the mogilefsd daemon
# Description:       Start/Stop the mogilefsd daemon.
### END INIT INFO

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/mogilefsd
NAME=mogilefsd
DESC=mogilefsd
MOGILEFSD_RUNASUSER=mogilefsd
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable if it is present
if [ -z "$MOGILEFSD_RUNASUSER" ]; then
    echo "Cannot determine user to run as, recheck init script"
    exit 0
fi


# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

set -e

#
# Function that starts the daemon/service
#
do_start()
{
        if [ -e $PIDFILE ]
                then

                if [ -d /proc/`cat $PIDFILE`/ ]
                then

                        echo "$NAME already running."
                        exit 0;
                else
                        rm -f $PIDFILE
                fi

        fi

        start-stop-daemon --start --quiet --exec $DAEMON --pidfile $PIDFILE -b -m --name $NAME --chuid $MOGILEFSD_RUNASUSER
}

#
# Function that stops the daemon/service
#
do_stop()
{
        start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE --name $NAME --user $MOGILEFSD_RUNASUSER
        rm -f $PIDFILE
}

case "$1" in
  start)
        echo -n "Starting $DESC: "
        do_start
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        do_stop
        echo "$NAME."
        ;;

  restart|force-reload)
        #
        #       If the "reload" option is implemented, move the "force-reload"
        #       option to the "reload" entry above. If not, "force-reload" is
        #       just the same as "restart".
        #
        echo -n "Restarting $DESC: "
        do_stop
        sleep 1
        do_start
        echo "$NAME."
        ;;

  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac

:

and my mogstored init script in /etc/init.d/mogstored :

#! /bin/sh
### BEGIN INIT INFO
# Provides:          mogstored
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start/Stop the mogstored daemon
# Description:       Start/Stop the mogstored daemon.
### END INIT INFO

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/mogstored
NAME=mogstored
DESC=mogstored
MOGSTORED_RUNASUSER=mogstored
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable if it is present
if [ -z "$MOGSTORED_RUNASUSER" ]; then
    echo "Cannot determine user to run as, recheck init script"
    exit 0
fi


# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

set -e

#
# Function that starts the daemon/service
#
do_start()
{
        if [ -e $PIDFILE ]
                then

                if [ -d /proc/`cat $PIDFILE`/ ]
                then

                        echo "$NAME already running."
                        exit 0;
                else
                        rm -f $PIDFILE
                fi

        fi

        start-stop-daemon --start --quiet --exec $DAEMON --pidfile $PIDFILE -b -m --name $NAME --chuid $MOGSTORED_RUNASUSER
}
#
# Function that stops the daemon/service
#
do_stop()
{
        start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE --name $NAME --user $MOGSTORED_RUNASUSER
        rm -f $PIDFILE
}

case "$1" in
  start)
        echo -n "Starting $DESC: "
        do_start
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        do_stop
        echo "$NAME."
        ;;

  restart|force-reload)
        #
        #       If the "reload" option is implemented, move the "force-reload"
        #       option to the "reload" entry above. If not, "force-reload" is
        #       just the same as "restart".
        #
        echo -n "Restarting $DESC: "
        do_stop
        sleep 1
        do_start
        echo "$NAME."
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac

:

To make the init scripts run at boot, make them executable, and add to boot sequence:

sudo chmod +x /etc/init.d/mogilefsd
sudo update-rc.d mogilefsd defaults 
sudo chmod +x /etc/init.d/mogstored
sudo update-rc.d mogstored defaults 

varnish and sessions / cookies disappearing fix

I was having a problem with logged in users behind varnish, this is what I did to fix it:

My php code:

$value = 'access';
setcookie("auth_user", $value);
setcookie("auth_user", $value, time()+86400);
setcookie("auth_user", $value, time()+86400, "/", ".mysite.com", 1);

and then in /etc/varnish/default.vcl:

sub vcl_recv {
if (req.http.cookie && req.http.cookie ~ "auth_user") {
    pass;
   }
}

sub vcl_hash {
if (req.http.cookie && req.http.cookie ~ "auth_user") { set req.hash +=
"auth"; }
    set req.hash += req.url;

 hash;
}

Installing snmpd on Ubuntu 8.04LTS for cacti

To install it type:

sudo apt-get install snmpd

Edit file /etc/default/snmpd and make sure those values are set:

SNMPDRUN=yes
SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid 127.0.0.1'

If you want your server to listen on all interfaces, remove the 127.0.0.1 bit.

Edit /etc/snmp/snmpd.conf and change com2sec so that you have a private community

com2sec readonly default mycommunity

If you want a remote machine to be able to gather information for the community mycommunity, make sure you replace localhost by mynetwork, where mynetwork can be of the form: 1.1.1.1 or 1.1.0.0/16.

We are going to use the snmpwalk utility to verify that the server is working as wanted.

Here we want snmp to reply only to localhost for the community mycommunity.

From localhost

snmpwalk -Os -c mycommunity -v 1 localhost system

Should return a lot of output and:

snmpwalk -Os -c public -v 1 localhost system
Timeout: No Response from localhost

If the second command returns result, it might be because you did not comment the line starting with com2sec.

If you have a firewall running, you need to open UDP port 161 to allow external connections

Backup and restore mysql database using mysqldump gzip and scp

First create a dump of the database where faqs is the name of the database

mysqldump -u root -h localhost -pmypassword faqs | gzip -9 > faqs-db.sql.gz

This will create a gzipped file containing insert statements

To transfer the file to another computer

scp faqs-db.sql.gz root@my.other.host.com:/home/username/faqs-db.sql.gz

Create the database on the new sql you want to import to

mysql -uroot -p
create faqs;
exit

To import the file into a new mysql database named faqs

gunzip -c faqs-db.sql.gz | mysql -uroot -pmypassword faqs