How to backup my data (and why!)

It is the digital age, the amount of personal data we produce keeps going up: digital pictures, HD movies and documents take an ever increasing amount of space.
That's a lot a memories and information we don't want to loose (or can't afford to).

Companies have devised backup plans for a long time but the concept is now entering homes though cloud storage (and other means). When your data is lost, it is too late: you need to devise a plan now!

I will focus on the needs of individuals and deal with 3 different data types:
- photos
- videos
- documents (excel, doc, text, pdf...)

Also, there are diffrent risks to take into account when defining a backup plan:
- hardware failure (crashed hard drive)
- physical destruction of data at a physical location (think fire, theft ...)
- human error (oops! I deleted the file)


1. Put your data in the Cloud

The cloud will shield you from hardware failure, physical destruction but might not protect you against human error.... Added bonus is that you can share your data with other people :)
You also take on additional risks: like the risk of you online account being hacked or the risk of your data becoming visible to everybody because of a misconfiguration on your side.

The good news is that there usually are free allocations for each service but you might have to pay for a feature you really need.

For pictures, you have:
- Picasa: 1GB free + free unlimited storage of pictures up to 800x800 pixels (additional storage available: cost of 20GB is 5USD/year, see all prices here)
- Google plus: free unlimited storage of pictures up to 2048x2048 pixels
- Flickr: free upload of 300MB worth of pictures every month, paid option unlimited storage (original quality) & bandwith for (25 USD/year or 45 SD/2y. see all prices here)


For movies, you have:
- Youtube: Videos can be uploaded for free (up to 20GB per video)
The problem is that the videos are automatically edited (and reduced in quality) and it is not easy to download them once they are in the cloud!


For documents, you have:
- Google Docs: storage of documents, presentation and spreadsheets in google format is unlimited, you get 1GB for other type of files. Additional space can be bought (and shared with picasa). See the above link in Picasa for pricing details. The problem is there is no easy way to synchronize a local folder with Google Documents...
- Dropbox: 2B free storage. Local folders can be synchronized with dropbox.

2. Use a backup Service

This will shield you from hardware failure (but it might be slow to recover the data), physical destruction and human error.

The idea is to send your compressed and encrypted data to a remote server where it is stored. You can usually access our backups from a website and from a specific software.
The problem is that all your data goes through the internet and it can be very slow to do the initial upload if you have a lot of data. For example, if you have 1TB of data to backup, it can take months to do the initial backup!
Same problem when you try to do a full backup: it will be usually an order of magnitude faster than doing the initial backup but it can still take a few days.


If you also need to recover fast from an hard drive failure (the most common hardware failure) you can use a local redundant RAID configuration (like RAID 1 or 5). Please note that RAID alone will not prevent data loss: you are still vulnerable to other hardware failures (like RAID controller failure, destruction of the device and human error).

Let's compare the different plans out there. I will focus on 3 providers: Mozy, Carbonite and Crashplan.

Plan Name	+10GB	Unlimited	Family unlimited	Home	HomePlus	HomePremier	50GB	125GB
Yearly Price	25 USD	50 USD	120 USD	59 USD	99 USD	149 USD	72 USD	120 USD
Space	10 GB	Unlimited	Unlimited	Unlimited			50 GB	125 GB
Number of computers	1	1	2-10	1	1	1	1 (add computer +2USD/month)
Suported Os	Windows, Mac, Linux, Solaris			Windows, Mac		Windows	Windows, Mac
Automated backup	All files			All Except video		All files	All files

Whatever your usage, Crashplan seems always cheeper and has more features. I use it myself and I am very satisfied with it....


The free crasplan software also allows you to backup on a friend computer (running crashplan as well). This means that you can backup your data without paying anything, provided a friend is ready to allocate you some data for backup.


3. Case studies

We just need to find the most cost effective combination of the above:

Profile A:10GB of pictures and a few documents: 5USD/year
Pay 5USD/year for Picasa storage (20GB)
Use DropBox free allocation to store the documents.
Problem: the backup process is manual: if you forget to upload your pictures to Picasa, they are not uploaded, unless you use the software I wrote to automatically upload to Picasa: see my post here). You are still vulnerable to Human error.

Profile B:100GB of pictures, 200GB of movies and a 10GB of documents: 50USD/year
Cheapest alternative is Crashplan Unlimited (50USD / year)
The backup process is now automatic: no need to worry about forgetting to backup something. On top of that, you are protected against human error as you can retrieve former versions of a file.

If your data is spread across different computer, you can buy a NAS and run crasplan on the NAS (see my post on how to install crasplan on an Iomega NAS here. Alternatively, you have the simpler option to buy Crashplan unlimited Family.


4. Conclusion

If you care about your data: take the time to devise your backup/data recovery plan now! You can always find a way that fits your budget.

You can get a reduced quality backup for your pictures and videos for free. Truct me, it is better to have a reduced quality backup than nothing!

How to install Crashplan on an IOMEGA Storcenter ix4-200d NAS

This tutorial uses unsupported features of the IOMEGA Storcenter ix4-200d and Crashplan does not support a client for ARM processors. The steps I described worked for me but use at your own risk!
Tutorial tested on IOMEGA Storcenter ix4-200d firmware 3.1.14.995
It should be fairly easy to adapt it to the Iomega ix2 NAS and with a little bit more work, even other devices using the ARM processor Marvell Kirkwood 6281 CPU (aka Feroceon).

First of all, it was much more complex than I anticipated. Crashplan is running on java, I was hoping for a quick java install and go! Actually it is much more complex because nothing is supported by either Iomega or Crashplan, some libraries are missing from the NAS and the custom linux (EMC Lifeline OS) does not behave as expected (read only file system and strange daemon startup mechanism)...
This tutorial assumes some basic linux and vi knowledge...

EDIT: This tutorial was used as a base for tutorials for Synology on powerpc at http://chreggy.fr/thegeek/2012/01/10/crashplan-sur-un-synology-powerpc-ds109/ and also for a packaged form of the installer for Synology on a Marvell Kirkwood or Intel CPU at http://pcloadletter.co.uk/2012/01/30/crashplan-syno-package/. If this tutorial helps you to install crashplan for another architecture/device, please link to this page and leave a comment. I am always glad to know that the hours I spent have served the community!


1. Enable SSH on the NAS
Go to
http://your-nas-adress/diagnostics.html
click "enable SSH"
on older firware versions, I understand an equivalent page could be found at:
http://your-nas-adress/support.html

If the admin password of the NAS is "pass" the root password to use in ssh is sohopass.(thanks http://planetkris.com/2010/05/iomega-storcenter-ix2-ssh-email-notifications-and-busybox-init-d)


2. Create a share where we will install our stuff
Create a share called NAS_Extension using the web interface
EDIT: now that I understand better the way the OS is organized, It would have been better to install everthing in /opt/ (/opt/ejre1.7.0 and /opt/crasplan3.0.3) the partition /opt/ is on is 15GB, so no risk to run out of space (type df to see the mount points and free space)


3. Install java
Because of oracle licensing, you can't download the file directly. Go there:
http://www.oracle.com/technetwork/java/embedded/downloads/javase/index.html
and download the ejre corresponding to "ARMv5 Linux - Headless" under "Java SE for Embedded 7"
The file should be called ejre-7-fcs-b147-linux-arm-sflt-headless-27_jun_2011.tar.gz
Store it into the NAS_Extension partition you just created using your regular way of accessing the NAS.

TODO: Everything should work with openJDK as well. I'll try to update the tutorial when I get a chance.

configure ssh on you computer and access the NAS (windows users can use putty.exe available here http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html)
In the NAS command line, type:

cd /mnt/pools/A/A0/NAS_Extension
tar -xvf ejre-7-fcs-b147-linux-arm-sflt-headless-27_jun_2011.tar.gz

That's it!


4. Install crashplan

cd /mnt/pools/A/A0/NAS_Extension/
wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.0.3_Linux.tgz
tar -xvf CrashPlan_3.0.3_Linux.tgz
cd CrashPlan-install
./install.sh

and leave the default setting (press enter) for all the questions except:
What directory do you wish to install CrashPlan to? /mnt/pools/A/A0/NAS_Extension/crashplan/

vi /etc/init.d/crashplan

And add at the beginning of the script (line 18)

PATH=$PATH:/mnt/pools/A/A0/NAS_Extension/ejre1.7.0/bin
export PATH

Now, some shared objects (native libraries) need to be replaced in order for Crashplan to work (precompiled versions are available).
Replace libjtux.so in /mnt/pools/A/A0/NAS_Extension/crashplan/:

cd ../crashplan/
cp libjtux.so libjtux.so.bak
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/libjtux.so

(credit to https://crashplan.zendesk.com/entries/390250-crashplan-on-sheevaplug)

Replace jna.jar in /mnt/pools/A/A0/NAS_Extension/crashplan/lib/:

cd lib/
cp jna-3.2.5.jar jna-3.2.5.jar.bak
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/jna-3.2.5.jar

(shared object extracted from the debian package http://packages.debian.org/sid/armel/libjna-java/download and reinjected into Crashplan jna jar. Thanks to http://www.opticality.com/blog/2011/07/16/installing-crashplan-on-a-pogoplug-pro/ for the idea)

Replace libffi.so (required for jna and not available by default):

cd /mnt/pools/A/A0/NAS_Extension/
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/libffi.so.5
vi /etc/ld.so.conf

add

/mnt/pools/A/A0/NAS_Extension/

at the end of /etc/ld.so.conf

Start crasplan

/etc/init.d/crashplan start

Remember to give it some time (like 5 minutes) before connecting. Crashplan can be slow to start up.

Connect to crashplan using ssh tunnel
See how to do that here:
http://support.crashplan.com/doku.php/how_to/configure_a_headless_client

log in, configure your backup sets etc....

It seems like it is working, but you are not done yet!


5. resolve idendity problem
The OS seems to wipe clean /var/lib at each reboot. The problem is that's where crashplan stores your login information!
To solve the problem:

cp -p /var/lib/crashplan/.identity /mnt/pools/A/A0/NAS_Extension/crashplan/
vi /mnt/pools/A/A0/NAS_Extension/crashplan/bin/CrashPlanEngine

and add:

mkdir -p /var/lib/crashplan
cp -p .identity /var/lib/crashplan/

after:

echo "Using standard startup"
cd $TARGETDIR

6. Get crashplan to start automatically on reboot
You can see how to do that in this post How to run a program at boot on Iomega Storcenter
You need to add the command:

/etc/init.d/crashplan start >> /opt/init-opt.log

to the script /opt/init-opt.sh
The script will look like this:

#!/bin/sh
# modified from http://techmonks.net/installing-transmission-and-dnsmasq-on-a-nas/
rm /opt/init-opt.log
echo "Last bootup:" >> /opt/init-opt.log
date >> /opt/init-opt.log
/etc/init.d/crashplan start >> /opt/init-opt.log
while true; do
sleep 1d
done

afterwards.

You are all done!

Please note that it takes quite some time for crashplan to be available after a reboot. Because of the 'nice' level, the process basically waits for all the other boot processes (mostly twonkymediaserv) to finish before being available and sarting to listen on the socket. This takes a good 12 minutes for me.

Let me know if you manage to get the tutorial working for other NAS devices and what you had to change to do that. I'll try to compile your remarks into another post.
If this tutorial saved you 10 hours of messing around with your NAS, you can offer me a beer (a coffe or a tea would do as well)!

7. Faq:
Q1: The Crashplan engine starts but fails after a few seconds/minutes
Chances are you libjtux is not properly updated.... retry to download it.
You should see the following stack trace when you do

cat /mnt/pools/A/A0/NAS_Extension/crashplan/log/engine_error.log
java.lang.UnsatisfiedLinkError: /mnt/pools/A/A0/NAS_Extension/crashplan/libjtux.so: /mnt/pools/A/A0/NAS_Extension/crashplan/libjtux.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32-bit .so on a ARM-bit platform)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1928)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
        at java.lang.Runtime.loadLibrary0(Runtime.java:845)
        at java.lang.System.loadLibrary(System.java:1084)
        at jtux.UUtil.init(UUtil.java:14)
        at jtux.UProcess.(UProcess.java:30)
        at com.code42.os.posix.PosixProcessCommands.getuid(PosixProcessCommands.java:57)
        at com.backup42.service.CPService.logIds(CPService.java:1164)
        at com.backup42.service.CPService.start(CPService.java:396)
        at com.backup42.service.CPService.main(CPService.java:1552)

Q2: The Crashplan engine seems fine but it is never backing up anything
You should see the following stack trace when you do (it takes some time (like 5-10 minutes) before you get the stack trace, just be patient):

cat /mnt/pools/A/A0/NAS_Extension/crashplan/log/engine_error.log
Exception in thread "W10189988_ScanWrkr" java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
        at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.(JNAInotifyFileWatcherDriver.java:21)
        at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:376)
        at com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:314)
        at com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:64)
        at com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1007)
        at com.code42.utils.AWorker.run(AWorker.java:158)
        at java.lang.Thread.run(Thread.java:722)

In this case, it is likely jna-3.2.5.jar is not working. Try to install it again.
The same error can be caused by the fact that you miss libffi.so.5 or it is not found by ld. To make sure your problem is linked to libffi, you can try:

cd /mnt/pools/A/A0/NAS_Extension/crashplan/
mkdir tmp
cd tmp
unzip ../jna-3.2.5.jar
ld com/sun/jna/linux-arm/libjnidispatch.so

if ld tells you it can not find libffi.so.5, that's your problem and you should try to re install libffi.so.5 (don't forget to change /etc/ld.so.conf).


EDIT 28/10/2012
I am having this problem now:

root@IOMEGA_NAS:/mnt/pools/A/A0/NAS_Extension/crashplan/log# tail -40 engine_error.log
java.lang.UnsatisfiedLinkError: /mnt/pools/A/A0/NAS_Extension/crashplan/libmd5.so: /mnt/pools/A/A0/NAS_Extension/crashplan/libmd5.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32-bit .so on a ARM-bit platform)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1928)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
        at java.lang.Runtime.loadLibrary0(Runtime.java:845)
        at java.lang.System.loadLibrary(System.java:1084)
        at com.twmacinta.util.MD5._loadNativeLibrary(MD5.java:727)
        at com.twmacinta.util.MD5._initNativeLibrary(MD5.java:703)
        at com.twmacinta.util.MD5.(MD5.java:124)
        at com.code42.utils.OsPatternList.getChecksum(OsPatternList.java:142)
        at com.code42.utils.PatternList.equals(PatternList.java:350)
        at com.code42.config.ConfigItem.setValue(ConfigItem.java:66)
        at com.code42.config.ConfigXmlTransformer.fromXmlConfigItem(ConfigXmlTransformer.java:445)
        at com.code42.config.ConfigXmlTransformer.fromXmlAConfigItem(ConfigXmlTransformer.java:405)
        at com.code42.config.ConfigXmlTransformer.fromXml(ConfigXmlTransformer.java:306)
        at com.code42.config.ConfigXmlTransformer.fromXmlConfigComponent(ConfigXmlTransformer.java:487)
        at com.code42.config.ConfigXmlTransformer.fromXml(ConfigXmlTransformer.java:308)
        at com.code42.config.ConfigXmlTransformer.fromXml(ConfigXmlTransformer.java:74)
        at com.code42.xml.XmlTool.fromXml(XmlTool.java:176)
        at com.code42.xml.XmlTool.fromXml(XmlTool.java:155)
        at com.code42.config.AConfig.fromXml(AConfig.java:77)
        at com.backup42.common.config.AppConfig.load(AppConfig.java:106)
        at com.backup42.common.config.ServiceConfig.load(ServiceConfig.java:326)
        at com.backup42.service.CPService.loadConfig(CPService.java:675)
        at com.backup42.service.CPService.start(CPService.java:331)
        at com.backup42.service.CPService.main(CPService.java:1622)

to fix it, just do:

cd /mnt/pools/A/A0/NAS_Extension/crashplan/
mv libmd5.so libmd5.so.old
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/libmd5.so

EDIT 08/11/2015
it looks like you need a more recent version of jna version with Crashplan 4.4.1

cd /mnt/pools/A/A0/NAS_Extension/crashplan/lib
mv jna.jar jna.jar.old
wget https://dl.dropboxusercontent.com/u/50398581/crashplan%20on%20NAS/jna-4.2.1.jar
ln -s jna-4.2.1.jar jna.jar