Showing posts with label NAS. Show all posts
Showing posts with label NAS. Show all posts

Thursday, March 1, 2012

How to install Vuze on a NAS




The goal of this tutorial is to install vuze headless (as a command line application). Most of the tutorials found on the web suggest to do the configuration of vuze in the UI before starting it in headless mode. Unfortunately, this is not possible on a NAS where you have to X server and no screen...



Tutorial tested on IOMEGA Storcenter ix4-200d firmware 3.1.14.995 but uses unsupported features on the hardware. Please use at your own risk.


Since Vuze if a java program, the same steps should allow you to install vuze as an headless client on any hardware running java.




Unfortunately, I ran into a lot of JVM crashes with vuze headless and oracle jvm ejre1.7.0 (for ARM). On top of that, vuze is quite an heavy program in terms of CPU and memory usage, which is annoying for the type of hardware we are looking at (like a NAS). Therefore, I don't recommand to install vuze on a NAS. I suggest you look at rtorrent, which is much more reliable (see my tutorial How to install rtorrent with IP filtering).





1. SSH into your NAS

See my other post:
How to ssh into your Iomega StorCenter ix4-200d if you have an IOMEGA NAS




2. Download and install

Steps adapted from the Console_UI Vuze wiki
cd /opt/tmp
wget http://ftp.heanet.ie/mirrors/www.apache.org/dist//commons/cli/binaries/commons-cli-1.2-bin.tar.gz
tar 
wget http://sourceforge.net/projects/azureus/files/vuze/Vuze_4702/Vuze_4702_linux.tar.bz2
PATH=$PATH:/opt/bin/
tar -xvf Vuze_4702_linux.tar.bz2

wget http://ftp.heanet.ie/mirrors/www.apache.org/dist//commons/cli/binaries/commons-cli-1.2-bin.tar.gz
tar -xvf commons-cli-1.2-bin.tar.gz
mv commons-cli-1.2/commons-cli-1.2.jar vuze/

wget http://ftp.heanet.ie/mirrors/www.apache.org/dist//logging/log4j/1.2.16/apache-log4j-1.2.16.tar.gz
tar -xvf apache-log4j-1.2.16.tar.gz
mv apache-log4j-1.2.16/log4j-1.2.16.jar vuze

to install java, you can look at the java section of my other post: How to install crashplan on an Iomaga NAS.

The installation is pretty straight forward...



Now, install the webUI plugin:
cd vuze
cd plugins
mkdir webui
cd webui
wget http://azureus.sourceforge.net/plugins/webui_1.7.0.zip
ipkg-opt install zip
ipkg-opt install unzip
unzip webui_1.7.0.zip
mkdir /opt/var/log/vuze
If you don't have ipkg-opt, see my other post: How to install software into your Iomega StorCenter NAS



3. Configure the Vuze installation

cd /opt/tmp/
mv vuze /opt/
cd /opt/vuze/
/mnt/pools/A/A0/NAS_Extension/ejre1.7.0/bin/java -Xmx128m -Dazureus.config.path=/opt/vuze/.azureus/ -cp "Azureus2.jar:commons-cli-1.2.jar:log4j-1.2.16.jar" org.gudy.azureus2.ui.common.Main --ui=console
Vuze should now be running, we need to configure it now. Adapt the paths to suit your needs and type at the vuze cli:
set "Default save path" "/mnt/pools/A/A0/torrents/vuze/download" string
set "Use default data dir" true boolean
set "Logger.Enabled" true boolean
set "Logging Enable" true boolean
set "Logging Dir" "/opt/var/log/vuze/" string
set "Ip Filter Autoload File" "http://list.iblocklist.com/?list=bt_level1" string
set Plugin.azhtmlwebui.User myusername
set Plugin.azhtmlwebui.Password mypassword password
set "Plugin.azhtmlwebui.Password Enable" true boolean
This is installing IP filtering as well. If you don't want that, just skip the set "Ip Filter Autoload File" command.

You are good to go now. To have vuze automatically start at boot, you need to create the script in /etc/init.d (you can adapt the azureus script provided inside the install).
If you have an Iomega NAS, look at this tutorial to see how to have the program run at boot.





You can now connect to the web Vuze UI from your web browser at
http://ip_of_nas:6883/. Please note that the web UI is not as rich as the regular UI (most options are not available in the web UI)




Please comment to let me know how stable your install is!

Thanks

Sunday, February 26, 2012

How to backup your google docs documents




I am a fan of google docs: I often needs to access and edit my documents while I am away, and google docs offers a great way to do that.
The problem is: I have a lot of large pdfs there, and they can take a while to load: I would love to have a local copy when I am in the office...
On top of that, I always like to have a local copy of stuff... just in case! Call me paranoid but what happens if your account is hacked? or if google unilaterally closes your account because they consider you don't respect the terms of use? Better be safe than sorry...


I couldn't find anywhere an application that would do what I want (get a local backup of my google documents and update it regularly).
There is the google "takeout" application but you can not schedule regular downoads...
A project like google-docs-fs seems promissing but it does only support google documents (and not any other file you may have uploaded if you have -like me- a premium account). Plus, my analysis is that there are too many possible points of failures if you rsync this file system... I need something more robust.


I decided to code what I need myself: a java command line application that can be used to schedule regular downloads of all your google docs documents.


1. Presentation gdocsauploader.jar

The features implemented:
- re use data from a previous data to avoid re-downloading files that haven't changed
- rotating backups (for example, a maximum of 7 backups backup.zip being the most recent one and backup.7.zip being the oldest one)
- zip archive or just a folder archive (takes more space but easier to access)
- configurable document export mode (export google spreasheets as xls or as csv)
- download only once documents that are in multiple folders (gdocsbackup.removeduplicateddocuments)
- archive without folder structure (all documents in a zip, like google takeout) or with folder structure (much easier to navigate)
- support for any type of files.

TODO:
- use hard links on operating systems that support it (that would substantially reduce that amount of disk needed for multiple backups with a lot of unchanged documents)
- fix the bug that forces you to use a temp directory on the same partition as the destination directory

In my setup, I want to install pgdocsauploader.jar as a daily cron on my NAS, but you can install it anywhere.

The program is configured using the config file gdocsuploader.properties which reads as follows:
#use system defined proxy
gdocsbackup.usesystemproxy=true
#google account username and password
gdocsbackup.username=xxxx
gdocsbackup.password=xxx
#the path where we want to backup
gdocsbackup.backuppath=C:\\Users\\xxx\\Documents\\Data\\
#the name of the backup archive. 
#the zip archives will be named: backuprootname.zip backuprootname.1.zip
#the folder archives will be named: backuprootname/ backuprootname.1/
gdocsbackup.backuprootname=gdocs_backup
#the number of backup files to keep
gdocsbackup.nbbackupfiles=7
#TRUE is you want to stroe backup as zip file. 
gdocsbackup.usezip=FALSE
#zip compression level (0-9) with 9 being the most compressed (and most CPU intensive)
gdocsbackup.zipcompresslevel=6
#use hard links to link new data identical to older data. This does save a lot of space (you can't use this option with usezip)
#not supported yet!
gdocsbackup.usehardlinks=FALSE
#document export format: one of doc html odt pdf png rtf txt zip
gdocsbackup.documentexportformat=doc
#presentation export format: one of pdf png ppt txt
gdocsbackup.presentationexportformat=ppt
#spreadsheet export format: one of xls csv pdf ods tsv html (NB: first sheet export only for csv and tsv)
gdocsbackup.spreadsheetexportformat=xls
#try to replicate the directory structure in the zip
docsbackup.keepdirectorystructure=TRUE
#show documents that appear at different places in the folder tree only once (in the first folder where it is found)
gdocsbackup.removeduplicateddocuments=TRUE
#log file (for linux, good practice is to put it in /var/log/ or /opt/var/log (and make sur logrotate works correctly))
gdocsbackup.logfile=C:\\gdocsbackup.log

All options are self explanatory. You can customize it as required by your setup.

As the program is java, it can be run on any OS / Architecture supporting Java.

The jar is available for download at http://dl.dropbox.com/u/50398581/gdocsbackup/gdocsdownload.jar
sample properties files is available at http://dl.dropbox.com/u/50398581/gdocsbackup/gdocsdownload.properties
and source code is available at: http://dl.dropbox.com/u/50398581/gdocsbackup/gdocsdownload-src.zip


Please note that in order to "rotate" backups, the program will delete the oldest backup! Don't modify the backups or store anything there!
The program only gets information from the google server: it does not update or delet anything: you are safe there!


To determine if the file was already downloaded, the last_update tag given by google is checked. I suggest you do a full backup from time to time to avoid an error propagating from backup to backup (to do that, just add the option full download after the "properties" file launching the jar)


2. Steps to install the gdocsbackup on a linux based NAS
The setup is easy to adapt to any machine running linux. I didn't do a tutorial for Windows or Mac as I lack some knowledge to do it, but it can of course be done... feel free to adapt it and post your results and hints in the comments!
This tutorial assumes some vi ans linux knowledge...

This is how I installed the gdocsbackup.jar on my NAS (an Iomega Storcenter ix4-200d). Please note that the procedure is unsupported by Iomega! use at your own risk!

a. Download and setup of gdocsdownload
First, you need to ssh into your NAS (see my other post if you have am Iomega Storcenter)
Then:
mkdir /opt/usr/local
mkdir /opt/usr/local/gdocsdownload/
cd /opt/usr/local/gdocsdownload/
wget http://dl.dropbox.com/u/50398581/gdocsbackup/gdocsdownload.jar
wget http://dl.dropbox.com/u/50398581/gdocsbackup/gdocsdownload.properties
Don't forget to change the properties file to make it work for your setup (you at least need to change account information and paths):
vi gdocsdownload.properties

If you are concerned about security, you should put the properties files into you home folder...

If you haven't already done so, you need to install java on your NAS. See the java section of my previous post How to install Crashplan on an Iomega Storcenter to find out how to do it for an Iomega storcenter.

If you followed the java installation procedure of my other post, link java to a more usual location:
ln -s /mnt/pools/A/A0/NAS_Extension/ejre1.7.0/bin/java /opt/bin/java
The setup can already be tested by starting the command:
/opt/bin/java -jar /opt/usr/local/gdocsdownload/gdocsdownload.jar /opt/usr/local/gdocsdownload/gdocsdownload.properties
press Ctrl-C to stop the run


the program will need to be started from a script so that we can set correct folder permissions and TMP folder.
You need to make sure there is enough space in your temp folder (my /tmp/ folder is way to small, that's why I use /opt/tmp/
vi gdocsdownloader

and then type:
#!/bin/sh

#this is to have a backup that's readable by everybody
#but only writeable by the owner.
#change it to suit your needs
umask 022
#use a tmp file with enough space to fit all your docs
#NB: it seems like there is a bug somewhere and the tmp directory has to
#be on the same partition than the destination directory....
#please choose a tmp file respecting these conditions
#/opt/bin/java -Djava.io.tmpdir=/opt/tmp/ -jar /opt/usr/local/gdocsdownload/gdocsdownload.jar $@
/opt/bin/java -Djava.io.tmpdir=/mnt/pools/A/A0/data/perso/gdocs/ -jar /opt/usr/local/gdocsdownload/gdocsdownload.jar $@
make it an executable:
chmod a+x gdocsdownloader
And test with:
./gdocsdownloader /opt/usr/local/gdocsdownload/gdocsdownload.properties


b. Set up a cron job to backup google docs data
Create the gdocsdownloader cron (I don't use /etc/cron.daily/ because I want a full download once a week):
vi /etc/cron.d/gdocsdownload
and add:
# download google docs files at 3:45 AM

#full download on sunday
45 3    * * 0   root    /opt/usr/local/gdocsdownload/gdocsdownloader /opt/usr/local/gdocsdownload/gdocsdownload.properties fulldownload > /dev/null 2>&1
#regular download the other days
45 3    * * 1,2,3,4,5,6   root    /opt/usr/local/gdocsdownload/gdocsdownloader /opt/usr/local/gdocsdownload/gdocsdownload.properties > /dev/null 2>&1

The cron will run everyday!
you may want to run the first batch by starting:
/opt/usr/local/gdocsdownload/gdocsdownloader /opt/usr/local/gdocsdownload/gdocsdownload.properties

c. start the cron daemon

The cron daemon is not started at boot by default....

You can start it manually:
/etc/init.d/cron start

But to have it start up every time at boot, we need to add the line:
/etc/init.d/cron start >> /opt/init-opt.log
to our /opt/init-opt.sh script.

See my other post How to run a program at boot on the Iomega Storcenter NAS to see how it works!

d. set up logrotate
Logrotate is the process that compresses and delete old logs so that your logs don't eat all you disk space!
vi /etc/logrotate.d/gdocsdownload
and add:
/opt/var/log/gdocsdownload.log {
    rotate 4
    weekly
    compress
    delaycompress
    missingok
    notifempty
    prerotate
      while [ "`ps aux | grep gdocsdownloader.jar | grep -v grep | wc -l`" = "1" ]
        do
          sleep 10
        done
    endscript
}


This will rotate your gdocsdownload logs once a week and keep at least 4 weeks worth of logs. It is easy to modify these parameters in the config file above.

I try to make sure the gdocsdownload is done before rotating the logs to avoid conflict...

Don't forget to change the path if your log is somewhere else!

Saturday, February 11, 2012

How to automatically synchronize a picture folder with picasa (on a NAS or anywhere else)






I like to have a copy of my pictures on picasa to be able to share them with friends and family. I usually upload them in reduced resolution, to stay within the free storage space given by google.
The problem is that uploading them can be a pain: the picasa software can be very slow to upload them, especially if your are accessing the pictures on your NAS with a wireless network.

Instead of using the picasa software, I tried to use googlecl tools (http://code.google.com/p/googlecl/) to do that but it turns out I couldn't get it to do want I want (no sync folder option + no resize of picture on the fly).
There is an unsupported patch to synchronize folders with googlecl (http://code.google.com/p/googlecl/issues/detail?id=170) but that doesn't solve the problem of image resizing...I did not even test it...


1. Presentation of my solution: picasauploader.jar

To solve the problem, I wrote a small piece of java code (picasauploader.jar) that:
- creates any new album in picasa web when a new folder is created on the disk
- upload (and resizes if necessary) new pictures on the disk to picasa web

In my setup, I want to install picasauploader.jar as a daily cron on my NAS, but you can install it anywhere.

You just need to organize your pictures as
/path/albumname/picture.jpg
and use /path in the picasauploader.properties

The jar is configured using the config file picasauploader.properties which reads as follows:
#use system defined proxy
picasauploader.usesystemproxy=true
#picasa/google account username and password
picasauploader.username=xxxx
picasauploader.password=xxx
#semi column separated directories
picasauploader.diskpaths=/xxx/yyyy;/aaaa/bbbb
#can be either:
# private: accessible to anybody with the direct link but not without the direct link
# protected: not accessible except from your account
# public: available for everybody to see
picasauploader.albumcreationaccess=private
#if you want to resize images before uploading (aspect ratio is kept)
#Note: only JPEG images are resized...
#max Height in px
picasauploader.maxheigt=1600
#max Width in px
picasauploader.maxwidth=1600
#jpg quality when resizing
picasauploader.resizequality=85
#log file (for linux, good practice is to put it in /var/log/ or /opt/var/log (and make sur logrotate works correctly))
picasauploader.logfile=/opt/var/log/picasaupload.log

All options are self explanatory. You can customize it as required by your setup.

As the program is java, it can be run on any OS / Architecture supporting Java.

The jar is available for download at http://dl.dropbox.com/u/50398581/picasauploader/picasauploader.jar
sample properties files is available at http://dl.dropbox.com/u/50398581/picasauploader/picasauploader.properties
and source code is available at: http://dl.dropbox.com/u/50398581/picasauploader/PicasaUploader.java

Please note that for safety, the program does not delete anything on picasa web (nor on the disk, of course). Therefore, it is very safe to use.

Known Limitations:
- only suports JPG GIF PNG BMP image formats
- picture resizing is only supported for jpg images
- only the name is used to determine if a picture was already uploaded: if a picture was already uploaded and then changed on disk, it won't be uploaded again.


2. Steps to install the picasauploader on a linux based NAS
The setup is easy to adapt to any machine running linux. I didn't do a tutorial for Windows or Mac as I lack some knowledge to do it, but it can of course be done... feel free to adapt it and post your results and hints in the comments!
This tutorial assumes some vi ans linux knowledge...

This is how I installed the picasauploader.jar on my NAS (an Iomega Storcenter ix4-200d). Please note that the procedure is unsupported by Iomega! use at your own risk!

a. Download and setup of picasauploader
First, you need to ssh into your NAS (see my other post if you have am Iomega Storcenter)
Then:
mkdir /opt/usr/local
mkdir /opt/usr/local/picasauploader
cd /opt/usr/local/picasauploader
wget http://dl.dropbox.com/u/50398581/picasauploader/picasauploader.jar
wget http://dl.dropbox.com/u/50398581/picasauploader/picasauploader.properties
Don't forget to change the properties file to make it work for your setup (you at least need to change account information and paths):
vi picasauploader.properties

If you haven't already done so, you need to install java on your NAS. See the java section of my previous post How to install Crashplan on an Iomega Storcenter to find out how to do it for an Iomega storcenter.

If you followed the java installation procedure of my other post, link java to a more usual location:
ln -s /mnt/pools/A/A0/NAS_Extension/ejre1.7.0/bin/java /opt/bin/java
The setup can already be tested by starting the command:
/opt/bin/java -jar /opt/usr/local/picasauploader/picasauploader.jar /opt/usr/local/picasauploader/picasauploader.properties

b. Set up a cron job to synchronize image folders with picasa
Create the picasauploader cron:
cd /etc/cron.daily/
vi picasauploader
and add:
#!/bin/sh
/opt/bin/java -jar /opt/usr/local/picasauploader/picasauploader.jar /opt/usr/local/picasauploader/picasauploader.properties
Then:
chmod a+x picasauploader
And test with:
./picasauploader

c. start the cron daemon

The cron daemon is not started at boot by default....

You can start it manually:
/etc/init.d/cron start

But to have it start up every time at boot, we need to add the line:
/etc/init.d/cron start >> /opt/init-opt.log
to our /opt/init-opt.sh script.

See my other post How to run a program at boot on the Iomega Storcenter NAS to see how it works!

d. set up logrotate
Logrotate is the process that compresses and delete old logs so that your logs don't eat all you disk space!
vi /etc/logrotate.d/picasauploader
and add:
/opt/var/log/picasaupload.log {
    rotate 4
    weekly
    compress
    delaycompress
    missingok
    notifempty
    prerotate
      while [ "`ps aux | grep picasauploader.jar | grep -v grep | wc -l`" = "1" ]
        do
          sleep 10
        done
    endscript
}


This will rotate your picasauploader logs once a week and keep at least 4 weeks worth of logs. It is easy to modify these parameters in the config file above.

As logrotate is started by the same cron that starts the picasauploader (daily cron), you will notice that I try to make sure the picasauploader is done before rotating the logs...

Don't forget to change the path if your log is somewhere else!

Thursday, January 26, 2012

How to install rtorrent with ip filtering into your Iomega StorCenter ix4-200d


EDIT: this post is outdated now, please see this post instead with a much newer version of rtorrent that works with magnets and natively supports ip filtering

This tutorial uses unsupported features of the IOMEGA Storcenter ix4-200d. It worked for me but use it at your own risk! It should work (again, it is unsupported) on the ix2 Storcenter as well.
Tutorial tested on IOMEGA Storcenter ix4-200d firmware 3.1.14.995
The torrent software supplied with the Storcenter doesn't work well for me: some torrents never load, some disappear etc.. plus there is no ip filtering capability. The aim of the tutorial is to install rtorrent on the NAS which seems the most logical choice for a NAS (light weight and reliable) and explains how to enable ip filtering directly within rtorrent which is especially usefull since peerguardian/moblock can't be installed on the NAS because some kernel modules are missing...


1. SSH into your NAS
See my other post: How to ssh into your Iomega StorCenter ix4-200d


2. Install the software
See my other post here to setup at the minimum ikg and ipkg-opt. Then:
ipkg-opt install rtorrent
ipkg-opt install lighttpd
ipkg-opt install screen

If you don't want to connect remotely to rtorrent to manage it from you computer, you can skip the rest of this section...
Install nTorrent on your computer http://code.google.com/p/ntorrent/
Install xml-rpc on the NAS:
ipkg install optware-devel
ipkg install libcurl-dev
cd /opt/tmp/
svn checkout http://xmlrpc-c.svn.sourceforge.net/svnroot/xmlrpc-c/stable xmlrpc-c    
cd xmlrpc-c/
./configure --prefix=/opt
make
make install
Note: ou can choose something other that nTorrent. Please give me you feedback in the comments if you do.


3. Configure the software
Fix paths in the startup script:
vi /opt/etc/init.d/S99rtorrent
Some absolute paths need to be set. Replace screen with /opt/bin/screen in lines like:
su -c "/opt/bin/screen -ls | grep -sq "\.${srnname}[[:space:]]" " ${user} || su -c "/opt/bin/screen -dm -S ${srnname} 2>&1 1>/dev/null" ${user} | tee -a "$logfile" >&2
except:
su -c "/opt/bin/screen -S ${srnname} -X screen /opt/bin/rtorrent ${options} 2>&1 1>/dev/null" ${user} | tee -a "$logfile" >&2
replace:
if ps | grep -sq ${pid}.*rtorrent ; then # make sure the pid doesn't belong to another process
with
if ps auxxx | grep -sq ${pid}.*rtorrent ; then # make sure the pid doesn't belong to another process
Configure rtorrent (you should get a real rtorrent help for this, I am just trying to get you to a point where is works!):
vi /opt/etc/rtorrent.conf
set download and torrent directories:
instead of
directory = /opt/share/torrent/work/
set something like
directory = /mnt/pools/A/A0/torrents/rtorrent/download
instead of
schedule = watch_directory,5,5,load_start=/opt/share/torrent/dl/*.torrent
set something like
schedule = watch_directory,5,5,load_start=/mnt/pools/A/A0/torrents/rtorrent/torrents/*.torrent
comment out:
#schedule = untied_directory,5,5,stop_untied=
add at the end:
#files rwx for everybody
system.set_umask = 0000
Run:
mkdir /opt/share/torrent/session
Then, create the directories rtorrent/download and rtorrent/torrents inside the torrent share using regular NAS access (to have the right permissions)


4. Configure the software for remote access
This is only if you want to manage your rtorrent remotely:
Thanks to http://www.nslu2-linux.org/wiki/HowTo/RtorrentWithRemoteGUI for the setup.
Security warning: if you follow these steps, anybody that can access port 8081 of you NAS will be able to send commands to rtorrent! You want to make sure that this port is only accessible from your local network.
vi /opt/etc/lighttpdlighttpd.conf
between
#                               "mod_rrdtool",
and
"mod_accesslog" )
add
"mod_scgi",
and at the end add:
scgi.server = (
"/RPC2" => ( 
    "127.0.0.1" => (
        "host" => "127.0.0.1",
        "port" => 5000,
        "check-local" => "disable"
        )
    )
)
vi /opt/etc/rtorrent.conf
and at the end add:
scgi_port = localhost:5000


4. Test your setup
/opt/bin/rtorrent -n -o import=/opt/etc/rtorrent.conf
if you get:
rtorrent: Fault occured while inserting xmlrpc call.
did you install xmlprc correctly? is ld.so.conf updated correctly? did you run ldconfig?

to connect to the running instance:
/opt/bin/screen -r rtorrent
and kill the terminal (putty) to exit or press Ctrl-a d.

For remote access: you can start lighthttpd on the NAS
/opt/etc/init.d/S80lighttpd start
and then start nTorrent on your computer and connect to your NAS port 8081 (by default) on path /RPC2.



5. Get rtorrent to start automatically on reboot
Follow the tutorial How to run a program at boot on Iomage Strocenter You just need to add the following lines to the script:
/opt/etc/init.d/S80lighttpd start >> /opt/init-opt.log
/opt/etc/init.d/S99rtorrent start >> /opt/init-opt.log
If you have another brand of NAS (or a regular linux OS), just try to link the startup scripts to /etc/rc2.d/ like ou would normally do an a linux box:
ln -s /opt/etc/init.d/S80lighttpd /etc/rc2.d/S80lighttpd
ln -s /opt/etc/init.d/S99rtorrent /etc/rc2.d/S99rtorrent


6. Get a peerguardian like protection
First, I tried to install peerguardian linux but ran into a wall: the LifeLine OS on the Iomega Storcenter does not have the right kernel modules. I tried to recompute the kernel from the sources given by IOMEGA on their website (it is available for download in the support section) I got it to compile but insmod of the required module (x_tables.ko) does freeze the kernel (hard reboot required). Since I could not think of a safe way to push further in this direction (without risking to brick the NAS), I investigated other possibilities...(post a comment if you want more details on the kernel compilation)
I thought about abandoning rtorrent altogether and try Vuze (which has ip filtering).I got it to run but it was pretty unstable (jre crash)...

Luckily, someboby wrote a patch for rtorrent so that it supports ip filtering. I got it to compile on my NAS (version 0.8.6) and here is the result.You just need to:
cd /opt/bin/
mv rtorrent rtorrent.sav
wget http://dl.dropbox.com/u/50398581/rtorrent-0.8.6/rtorrent
that will give you a rtorrent with ip filtering supported.
Note: this only work if you previously installed version 0.8.6 of rtorrent!

The precompiled version will only work if you have an "armel" architecture. Otherwise, you need to recompile from source (see point 7 below)

The some config:
vi /opt/etc/rtorrent.conf
and add at the end
ip_filter=/mnt/pools/A/A0/torrents/rtorrent/ipfilter/level1
schedule = filter,18:30:00,24:00:00,reload_ip_filter=
thanks to http://bogdan.org.ua/2011/04/01/rtorrent-enhanced-with-ipfilter-and-geoip-debian-squeeze-amd64-package.html

Now, we need to download and update regularly the ip filter file:
vi /etc/cron.daily/rtorrent_ipfilter
#!/bin/sh
cd /mnt/pools/A/A0/torrents/rtorrent/ipfilter/
wget http://list.iblocklist.com/?list=bt_level1
mv index.html\?list\=bt_level1 level1new.gz
gunzip level1new.gz
rm level1
mv level1new level1
then:
mkdir /mnt/pools/A/A0/torrents/rtorrent/ipfilter/
cd /etc/cron.daily/
chmod a+x rtorrent_ipfilter
./rtorrent_ipfilter
That's it: you just need to restart rtorrent to enjoy ip filtering. The ip filter file will be update everyday thanks to the cron (and rtotorrent will reload it).




7. In case you want/need to compile rtorrent with ip filtering yourself!
This is usefull if you are compiling a different version or a different architecture (please comment and report your success if you do so).

First, you need the header files for libsigc++-2.0.
wget http://ftp.de.debian.org/debian/pool/main/libs/libsigc++-2.0/libsigc++-2.0-dev_2.0.18-2_armel.deb
dpkg --instdir=/opt/ --admindir=/opt/dpkg/ -i libsigc++-2.0-dev_2.0.18-2_armel.deb

Then, take care of litorrent. I recompile libtorrent to install the correct headers as I don't find them anywhere (I didn't look for a deb archive with the correct headers but that might have done the trick...):
wget http://libtorrent.rakshasa.no/downloads/libtorrent-0.12.6.tar.gz
tar -xvf libtorrent-0.12.6.tar.gz
cd libtorrent-0.12.6
PATH=$PATH:/opt/bin
Then:
vi configure
and add at the begining of the configure script:
OPENSSL_CFLAGS='-I/opt/include/'
OPENSSL_LIBS='-L/opt/lib/ -lopenssl'
STUFF_LIBS='-L/opt/lib/ -lsigc-2.0'
STUFF_CFLAGS='-I/opt/usr/include/sigc++-2.0/ -I/opt/usr/lib/sigc++-2.0/include'
Note that I do edit the configure because I don't manage to get ipkg-config to work correctly. Using /opt/bin/pkg-config works better than the /bin/pkg-config but still not good enough...
./configure --prefix=/opt/
make
make install

Then, the main thing: rtorrent
ipkg-opt install libcurl-dev
ipkg-opt install ncurses-dev
PATH=$PATH:/opt/bin
wget http://libtorrent.rakshasa.no/downloads/rtorrent-0.8.6.tar.gz
tar -xvf rtorrent-0.8.6.tar.gz
cd rtorrent-0.8.6
vi configure
and add at the begining of the configure script:
sigc_LIBS='-L/opt/lib/ -lsigc-2.0 -L/lib/'
sigc_CFLAGS='-I/opt/usr/include/sigc++-2.0/ -I/opt/usr/lib/sigc++-2.0/include -I/opt/include/ncurses'
libcurl_LIBS='-L/opt/lib/ -lcurl'
libcurl_CFLAGS='-I/opt/include/'
libtorrent_LIBS='-L/opt/lib/ -ltorrent'
libtorrent_CFLAGS='-I/opt/include/'
I did not know were to put the ncurses include, that's why you'll find it in sigc_CFLAGS...

Now, install the patch to have ip filtering (more details on the patch here http://libtorrent.rakshasa.no/ticket/239):
wget http://libtorrent.rakshasa.no/raw-attachment/ticket/239/rtorrent-0.8.6-ip_filter_no_boost-fast-bsd2.patch
/opt/bin/patch-patch -p1 < rtorrent-0.8.6-ip_filter_no_boost-fast-bsd2.patch
./configure --prefix=/opt/ --with-xmlrpc-c=/opt/bin/xmlrpc-c-config
Then, I got the following issue:
/opt/arm-none-linux-gnueabi/lib/libdl.so.2: undefined reference to `_dl_tls_get_addr_soft@GLIBC_PRIVATE'
My system is starting to be a mess. The problem comes from the fact that I have 2 libdb.so libs:
root@xxx:/opt/tmp/rtorrent-0.8.6# ls -l /mnt/apps/lib/libdl.so.2
lrwxrwxrwx 1 root root 12 Sep  9 20:46 /mnt/apps/lib/libdl.so.2 -> libdl-2.8.so
root@xxx:/opt/tmp/rtorrent-0.8.6# ls -l /mnt/system/opt/arm-none-linux-gnueabi/lib/libdl.so.2
lrwxrwxrwx 1 root root 12 Jan  6 22:47 /mnt/system/opt/arm-none-linux-gnueabi/lib/libdl.so.2 -> libdl-2.5.so
to fix the issue:
rm /mnt/system/opt/arm-none-linux-gnueabi/lib/libdl.so.2
Then, compile and install:
make
make install
final issue:
Could not compile XMLRPC-C test.
This one came from a mismatch between heders and libs. Recompiling the package xml-rpc from source did fix the issue. 8. What's next In a different post, I will detail how to install Vuze headless (without graphical interface)... I don't recommand installing vuze because I ran into stability issues while testing it (several jre crashes). However, a new Jre version might solve the issue...
On top of that, you can't do much from the Web UI: when you want to setup something, you ofen have to use the Vuze command line inteface and I did not find any proper documentation for it.
Note as well that rss feeds features don't work in headless mode.

Monday, January 23, 2012

How to run a program at boot on the Iomega Storcenter NAS


This tutorial uses unsupported features of the IOMEGA Storcenter ix4-200d. It worked for me but use it at your own risk! I undertand it works (but still isn't unsupported by IOMEGA) on the ix2 Storcenter as well.
Tutorial tested on IOMEGA Storcenter ix4-200d firmware 3.1.14.995
This post is extracted from a previous post (how to install crashplan on Iomega storcenter). I am planning more tutorials on the Iomega storcenter ix4 and wanted centralize this part in case it needs to evolve with future firmwares...


1. Enable SSH on the NAS
see my other post How to SSH into your Iomega Storcenter




2. Create a script that runs at boot
Iomega OS (EMC LifeLine) does not respect what is inside /etc/rcx.d/. If you have another brand of NAS, chances are that installing the script into /etc/rc2.d/ will result in the script running at boot...
If you already downloaded the scripts to have a command run at boot, you can just add the new command below the one you already have in /opt/init-opt.sh. Just make sure the commands all return immediately (or add a & at the end) so that the script does not get stuck before reaching the last command.

If that the first time you do this, download the scripts:
cd /opt/
wget http://dl.dropbox.com/u/50398581/Storcenter%20add%20on%20boot/editconfig.sh
chmod +x /opt/editconfig.sh
wget http://dl.dropbox.com/u/50398581/Storcenter%20add%20on%20boot/init-opt.sh
chmod +x /opt/init-opt.sh
Now we start editing the XML list of programs that will automatically be started. Run:
/opt/editconfig.sh
You will see lots of Groups. We are going to add one <program> to <group level="1">. We will add:
<program name="init-opt" path="/opt/init-opt.sh">
<sysoption restart="-1"/>
</program>
the scripts content for you reference:
/opt/editconfig.sh
#!/bin/sh
# edit the bootup config of the ix-2
# inspired by http://www.chrispont.co.uk/2010/10/allow-startup-daemons-on-storcenter-ix2-200-nas/
mknod -m0660 /dev/loop3 b 7 3
chown root.disk /dev/loop3
mkdir /tmp/apps
mount -o loop /boot/images/apps /tmp/apps
vi /tmp/apps/usr/local/cfg/sohoProcs.xml
sleep 1
umount /tmp/apps
rm /dev/loop3
/opt/init-opt.sh
#!/bin/sh
# modified from http://techmonks.net/installing-transmission-and-dnsmasq-on-a-nas/
rm /opt/init-opt.log
echo "Last bootup:" >> /opt/init-opt.log
date >> /opt/init-opt.log
#Add your command below
#/etc/init.d/xxxxxx start >> /opt/init-opt.log
while true; do
 sleep 1d
done
Then, you just need to edit editconfig.sh add the command(s) you wish to run after the
#/etc/init.d/xxxxxx start >> /opt/init-opt.log
line!

Tuesday, November 22, 2011

How to install Crashplan on an IOMEGA Storcenter ix4-200d NAS



This tutorial uses unsupported features of the IOMEGA Storcenter ix4-200d and Crashplan does not support a client for ARM processors. The steps I described worked for me but use at your own risk!
Tutorial tested on IOMEGA Storcenter ix4-200d firmware 3.1.14.995
It should be fairly easy to adapt it to the Iomega ix2 NAS and with a little bit more work, even other devices using the ARM processor Marvell Kirkwood 6281 CPU (aka Feroceon).

First of all, it was much more complex than I anticipated. Crashplan is running on java, I was hoping for a quick java install and go! Actually it is much more complex because nothing is supported by either Iomega or Crashplan, some libraries are missing from the NAS and the custom linux (EMC Lifeline OS) does not behave as expected (read only file system and strange daemon startup mechanism)...
This tutorial assumes some basic linux and vi knowledge...

EDIT: This tutorial was used as a base for tutorials for Synology on powerpc at http://chreggy.fr/thegeek/2012/01/10/crashplan-sur-un-synology-powerpc-ds109/ and also for a packaged form of the installer for Synology on a Marvell Kirkwood or Intel CPU at http://pcloadletter.co.uk/2012/01/30/crashplan-syno-package/. If this tutorial helps you to install crashplan for another architecture/device, please link to this page and leave a comment. I am always glad to know that the hours I spent have served the community!


1. Enable SSH on the NAS
Go to
http://your-nas-adress/diagnostics.html
click "enable SSH"
on older firware versions, I understand an equivalent page could be found at:
http://your-nas-adress/support.html

If the admin password of the NAS is "pass" the root password to use in ssh is sohopass.(thanks http://planetkris.com/2010/05/iomega-storcenter-ix2-ssh-email-notifications-and-busybox-init-d)


2. Create a share where we will install our stuff
Create a share called NAS_Extension using the web interface
EDIT: now that I understand better the way the OS is organized, It would have been better to install everthing in /opt/ (/opt/ejre1.7.0 and /opt/crasplan3.0.3) the partition /opt/ is on is 15GB, so no risk to run out of space (type df to see the mount points and free space)


3. Install java
Because of oracle licensing, you can't download the file directly. Go there:
http://www.oracle.com/technetwork/java/embedded/downloads/javase/index.html
and download the ejre corresponding to "ARMv5 Linux - Headless" under "Java SE for Embedded 7"
The file should be called ejre-7-fcs-b147-linux-arm-sflt-headless-27_jun_2011.tar.gz
Store it into the NAS_Extension partition you just created using your regular way of accessing the NAS.

TODO: Everything should work with openJDK as well. I'll try to update the tutorial when I get a chance.

configure ssh on you computer and access the NAS (windows users can use putty.exe available here http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html)
In the NAS command line, type:
cd /mnt/pools/A/A0/NAS_Extension
tar -xvf ejre-7-fcs-b147-linux-arm-sflt-headless-27_jun_2011.tar.gz
That's it!


4. Install crashplan
cd /mnt/pools/A/A0/NAS_Extension/
wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_3.0.3_Linux.tgz
tar -xvf CrashPlan_3.0.3_Linux.tgz
cd CrashPlan-install
./install.sh
and leave the default setting (press enter) for all the questions except:
What directory do you wish to install CrashPlan to? /mnt/pools/A/A0/NAS_Extension/crashplan/
vi /etc/init.d/crashplan
And add at the beginning of the script (line 18)
PATH=$PATH:/mnt/pools/A/A0/NAS_Extension/ejre1.7.0/bin
export PATH

Now, some shared objects (native libraries) need to be replaced in order for Crashplan to work (precompiled versions are available).
Replace libjtux.so in /mnt/pools/A/A0/NAS_Extension/crashplan/:
cd ../crashplan/
cp libjtux.so libjtux.so.bak
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/libjtux.so
(credit to https://crashplan.zendesk.com/entries/390250-crashplan-on-sheevaplug)

Replace jna.jar in /mnt/pools/A/A0/NAS_Extension/crashplan/lib/:
cd lib/
cp jna-3.2.5.jar jna-3.2.5.jar.bak
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/jna-3.2.5.jar
(shared object extracted from the debian package http://packages.debian.org/sid/armel/libjna-java/download and reinjected into Crashplan jna jar. Thanks to http://www.opticality.com/blog/2011/07/16/installing-crashplan-on-a-pogoplug-pro/ for the idea)

Replace libffi.so (required for jna and not available by default):
cd /mnt/pools/A/A0/NAS_Extension/
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/libffi.so.5
vi /etc/ld.so.conf
add
/mnt/pools/A/A0/NAS_Extension/
at the end of /etc/ld.so.conf

Start crasplan
/etc/init.d/crashplan start
Remember to give it some time (like 5 minutes) before connecting. Crashplan can be slow to start up.

Connect to crashplan using ssh tunnel
See how to do that here:
http://support.crashplan.com/doku.php/how_to/configure_a_headless_client

log in, configure your backup sets etc....

It seems like it is working, but you are not done yet!


5. resolve idendity problem
The OS seems to wipe clean /var/lib at each reboot. The problem is that's where crashplan stores your login information!
To solve the problem:
cp -p /var/lib/crashplan/.identity /mnt/pools/A/A0/NAS_Extension/crashplan/
vi /mnt/pools/A/A0/NAS_Extension/crashplan/bin/CrashPlanEngine
and add:
mkdir -p /var/lib/crashplan
cp -p .identity /var/lib/crashplan/
after:
echo "Using standard startup"
cd $TARGETDIR


6. Get crashplan to start automatically on reboot
You can see how to do that in this post How to run a program at boot on Iomega Storcenter
You need to add the command:
/etc/init.d/crashplan start >> /opt/init-opt.log
to the script /opt/init-opt.sh
The script will look like this:
#!/bin/sh
# modified from http://techmonks.net/installing-transmission-and-dnsmasq-on-a-nas/
rm /opt/init-opt.log
echo "Last bootup:" >> /opt/init-opt.log
date >> /opt/init-opt.log
/etc/init.d/crashplan start >> /opt/init-opt.log
while true; do
sleep 1d
done
afterwards.

You are all done!

Please note that it takes quite some time for crashplan to be available after a reboot. Because of the 'nice' level, the process basically waits for all the other boot processes (mostly twonkymediaserv) to finish before being available and sarting to listen on the socket. This takes a good 12 minutes for me.

Let me know if you manage to get the tutorial working for other NAS devices and what you had to change to do that. I'll try to compile your remarks into another post.
If this tutorial saved you 10 hours of messing around with your NAS, you can offer me a beer (a coffe or a tea would do as well)!


7. Faq:
Q1: The Crashplan engine starts but fails after a few seconds/minutes
Chances are you libjtux is not properly updated.... retry to download it.
You should see the following stack trace when you do
cat /mnt/pools/A/A0/NAS_Extension/crashplan/log/engine_error.log
java.lang.UnsatisfiedLinkError: /mnt/pools/A/A0/NAS_Extension/crashplan/libjtux.so: /mnt/pools/A/A0/NAS_Extension/crashplan/libjtux.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32-bit .so on a ARM-bit platform)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1928)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
        at java.lang.Runtime.loadLibrary0(Runtime.java:845)
        at java.lang.System.loadLibrary(System.java:1084)
        at jtux.UUtil.init(UUtil.java:14)
        at jtux.UProcess.(UProcess.java:30)
        at com.code42.os.posix.PosixProcessCommands.getuid(PosixProcessCommands.java:57)
        at com.backup42.service.CPService.logIds(CPService.java:1164)
        at com.backup42.service.CPService.start(CPService.java:396)
        at com.backup42.service.CPService.main(CPService.java:1552)

Q2: The Crashplan engine seems fine but it is never backing up anything
You should see the following stack trace when you do (it takes some time (like 5-10 minutes) before you get the stack trace, just be patient):
cat /mnt/pools/A/A0/NAS_Extension/crashplan/log/engine_error.log
Exception in thread "W10189988_ScanWrkr" java.lang.NoClassDefFoundError: Could not initialize class com.code42.jna.inotify.InotifyManager
        at com.code42.jna.inotify.JNAInotifyFileWatcherDriver.(JNAInotifyFileWatcherDriver.java:21)
        at com.code42.backup.path.BackupSetsManager.initFileWatcherDriver(BackupSetsManager.java:376)
        at com.code42.backup.path.BackupSetsManager.startScheduledFileQueue(BackupSetsManager.java:314)
        at com.code42.backup.path.BackupSetsManager.access$1600(BackupSetsManager.java:64)
        at com.code42.backup.path.BackupSetsManager$ScanWorker.delay(BackupSetsManager.java:1007)
        at com.code42.utils.AWorker.run(AWorker.java:158)
        at java.lang.Thread.run(Thread.java:722)
In this case, it is likely jna-3.2.5.jar is not working. Try to install it again.
The same error can be caused by the fact that you miss libffi.so.5 or it is not found by ld. To make sure your problem is linked to libffi, you can try:
cd /mnt/pools/A/A0/NAS_Extension/crashplan/
mkdir tmp
cd tmp
unzip ../jna-3.2.5.jar
ld com/sun/jna/linux-arm/libjnidispatch.so
if ld tells you it can not find libffi.so.5, that's your problem and you should try to re install libffi.so.5 (don't forget to change /etc/ld.so.conf).


EDIT 28/10/2012
I am having this problem now:
root@IOMEGA_NAS:/mnt/pools/A/A0/NAS_Extension/crashplan/log# tail -40 engine_error.log
java.lang.UnsatisfiedLinkError: /mnt/pools/A/A0/NAS_Extension/crashplan/libmd5.so: /mnt/pools/A/A0/NAS_Extension/crashplan/libmd5.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32-bit .so on a ARM-bit platform)
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1928)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1854)
        at java.lang.Runtime.loadLibrary0(Runtime.java:845)
        at java.lang.System.loadLibrary(System.java:1084)
        at com.twmacinta.util.MD5._loadNativeLibrary(MD5.java:727)
        at com.twmacinta.util.MD5._initNativeLibrary(MD5.java:703)
        at com.twmacinta.util.MD5.(MD5.java:124)
        at com.code42.utils.OsPatternList.getChecksum(OsPatternList.java:142)
        at com.code42.utils.PatternList.equals(PatternList.java:350)
        at com.code42.config.ConfigItem.setValue(ConfigItem.java:66)
        at com.code42.config.ConfigXmlTransformer.fromXmlConfigItem(ConfigXmlTransformer.java:445)
        at com.code42.config.ConfigXmlTransformer.fromXmlAConfigItem(ConfigXmlTransformer.java:405)
        at com.code42.config.ConfigXmlTransformer.fromXml(ConfigXmlTransformer.java:306)
        at com.code42.config.ConfigXmlTransformer.fromXmlConfigComponent(ConfigXmlTransformer.java:487)
        at com.code42.config.ConfigXmlTransformer.fromXml(ConfigXmlTransformer.java:308)
        at com.code42.config.ConfigXmlTransformer.fromXml(ConfigXmlTransformer.java:74)
        at com.code42.xml.XmlTool.fromXml(XmlTool.java:176)
        at com.code42.xml.XmlTool.fromXml(XmlTool.java:155)
        at com.code42.config.AConfig.fromXml(AConfig.java:77)
        at com.backup42.common.config.AppConfig.load(AppConfig.java:106)
        at com.backup42.common.config.ServiceConfig.load(ServiceConfig.java:326)
        at com.backup42.service.CPService.loadConfig(CPService.java:675)
        at com.backup42.service.CPService.start(CPService.java:331)
        at com.backup42.service.CPService.main(CPService.java:1622)

to fix it, just do:
cd /mnt/pools/A/A0/NAS_Extension/crashplan/
mv libmd5.so libmd5.so.old
wget http://dl.dropbox.com/u/50398581/crashplan%20on%20NAS/libmd5.so


EDIT 08/11/2015
it looks like you need a more recent version of jna version with Crashplan 4.4.1
cd /mnt/pools/A/A0/NAS_Extension/crashplan/lib
mv jna.jar jna.jar.old
wget https://dl.dropboxusercontent.com/u/50398581/crashplan%20on%20NAS/jna-4.2.1.jar
ln -s jna-4.2.1.jar jna.jar