Sunday, April 15, 2012

How to install rtorrent 0.9.1 and use magnet links on an Iomega Storcenter ix4-200d


This tutorial uses unsupported features of the IOMEGA Storcenter ix4-200d. It worked for me but use it at your own risk! It should work (again, it is unsupported) on the ix2 Storcenter as well.
Tutorial tested on IOMEGA Storcenter ix4-200d firmware 3.1.14.995

I explained in a previous post why I wanted to use rtorrent instead of the torrent client supplied with the storcenter.
There is a new developpment: thepiratebay.se switched to magnets only for file sharing: the version of rtorrent previously installed did not support magnets....
The good news is that the new rtorrent (0.9.1) does support IP filtering natively!

The problem is it was difficult to compile for the storcenter as the gcc toolchain available on the storcenter is very old... but no worries, I compiled it for you!

1. SSH into your NAS
See my other post: How to ssh into your Iomega StorCenter ix4-200d

2. Install the software
See my other post here to setup at the minimum ikg and ipkg-opt. Then:
ipkg-opt install lighttpd
ipkg-opt install screen

Then, get my pre-compiled version of rtorrent-0.9.1 (works on Iomega Storcenter ix4-200d):
if you want to compile it yourself for a strange architecture, you might want to look at section 3 of my other post How to solve the "undefined reference to '__sync_sub_and_fetch_4'" compilation problem
Warning: this is going to override the following files:
/opt/etc/rtorrent.conf
/opt/etc/init.d/S99rtorrent
/opt/lib/libtorrent.14.0.3
/opt/lib/libtorrent.14
/opt/bin/rtorrent
Make sure you saved everything that needed to be saved before running it!
cd /opt/tmp/
wget http://dl.dropbox.com/u/50398581/rtorrent-0.9.1/rtorrent-0.9.1-package.tar.gz
cd /
tar -xvf /opt/tmp/rtorrent-0.9.1-package.tar.gz


If you don't want to connect remotely to rtorrent to manage it from you computer, you can skip the rest of this section...
Install nTorrent on your computer http://code.google.com/p/ntorrent/
Install xml-rpc on the NAS:
ipkg install optware-devel
ipkg install libcurl-dev
cd /opt/tmp/
svn checkout http://xmlrpc-c.svn.sourceforge.net/svnroot/xmlrpc-c/stable xmlrpc-c    
cd xmlrpc-c/
./configure --prefix=/opt
make
make install
Note: ou can choose something other that nTorrent. Please give me you feedback in the comments if you do.



3. Configure the software

Fix path and other info in rtorrent.conf.
This is also where you want to disable remote access if you don't want it, comment the line:
scgi_port = localhost:5000

If you want to use remote acces, you need to:
vi /opt/etc/lighttpdlighttpd.conf
between
#                               "mod_rrdtool",
and
"mod_accesslog" )
add
"mod_scgi",
and at the end add:
scgi.server = (
"/RPC2" => ( 
    "127.0.0.1" => (
        "host" => "127.0.0.1",
        "port" => 5000,
        "check-local" => "disable"
        )
    )
)
Security warning: if you follow these steps, anybody that can access port 8081 of you NAS will be able to send commands to rtorrent! You want to make sure that this port is only accessible from your local network.

4. Ip filtering
a. download the file
Ip filtering support is build into rtorrent-0.9.1, but you still need to configure the download of the filter files:
vi /etc/cron.daily/rtorrent_ipfilter
#!/bin/sh
cd /mnt/pools/A/A0/torrents/rtorrent/ipfilter/
wget http://list.iblocklist.com/?list=bt_level1
mv index.html\?list\=bt_level1 level1new.gz
gunzip level1new.gz
sed 's/^.*:\([^:]*\)$/\1/g' level1new | grep -v '^#' > level1new_2
rm level1
mv level1new level1
rm level1_2
mv level1new_2 level1_2
then:
mkdir /mnt/pools/A/A0/torrents/rtorrent/ipfilter/
cd /etc/cron.daily/
chmod a+x rtorrent_ipfilter
./rtorrent_ipfilter

b. if not already done, make sure the cron daemon is started at boot
The cron daemon is not started at boot by default....

You can start it manually:
/etc/init.d/cron start

But to have it start up every time at boot, we need to add the line:
/etc/init.d/cron start >> /opt/init-opt.log
to our /opt/init-opt.sh script.

See my other post How to run a program at boot on the Iomega Storcenter NAS to see how it works!


5. Test your setup
/opt/bin/rtorrent -n -o import=/opt/etc/rtorrent.conf
if you get:
rtorrent: Fault occured while inserting xmlrpc call.
did you install xmlprc correctly? is ld.so.conf updated correctly? did you run ldconfig?

to connect to the running instance:
/opt/bin/screen -r rtorrent
and kill the terminal (putty) to exit or press Ctrl-a d.

For remote access: you can start lighthttpd on the NAS
/opt/etc/init.d/S80lighttpd start
and then start nTorrent on your computer and connect to your NAS port 8081 (by default) on path /RPC2.




6. Get rtorrent to start automatically on reboot
Follow the tutorial How to run a program at boot on Iomage Strocenter You just need to add the following lines to the script:
/opt/etc/init.d/S80lighttpd start >> /opt/init-opt.log
/opt/etc/init.d/S99rtorrent start >> /opt/init-opt.log
If you have another brand of NAS (or a regular linux OS), just try to link the startup scripts to /etc/rc2.d/ like ou would normally do an a linux box:
ln -s /opt/etc/init.d/S80lighttpd /etc/rc2.d/S80lighttpd
ln -s /opt/etc/init.d/S99rtorrent /etc/rc2.d/S99rtorrent


7. How to deal with magnet links
I suggest to create a /whereever/rtorrent/magnets like /whereever/rtorrent/torrents and /whereever/rtorrent/download.
And then:
cd /whereever/rtorrent
vi allmagnets.sh 
and add:
#!/bin/bash

for f in magnets/*
do
 echo "Processing $f"
 CONTENT=`cat "$f" | sed s/^URL=// | grep -v '\[InternetShortcut\]' | tr -d '\r'`
 [[ "$CONTENT" =~ xt=urn:btih:([^&/]+) ]] && echo "d10:magnet-uri${#CONTENT}:${CONTENT}e" > "torrents/meta-${BASH_REMATCH[1]}.torrent" && rm "$f"
done
chmod a+x allmagnet.sh
And then, add a cron to run this program every 5 minutes:
vi /etc/cron.d/magnets
and add:
# convert magnets to torrents every 5 minutes
1,6,11,16,21,26,31,36,41,46,51,56 *    * * *   root    cd /mnt/pools/A/A0/data/rtorrent/ && /mnt/pools/A/A0/data/rtorrent/allmagnets.sh 

Make sure the cron daemon is running!!! (see point 4.b above)

Enjoy!

How to solve the "undefined reference to '__sync_sub_and_fetch_4'" compilation problem

If you ran into the following compilation problems:
undefined reference to '__sync_sub_and_fetch_4' problem
or with any of the following functions:
__sync_fetch_and_add, __sync_fetch_and_sub, __sync_fetch_and_or, __sync_fetch_and_and, __sync_fetch_and_xor, __sync_fetch_and_nand,
 __sync_add_and_fetch, __sync_sub_and_fetch, __sync_sub_or_fetch, __sync_and_and_fetch, __sync_xor_and_fetch, __sync_nand_and_fetch
__sync_val_compare_and_swap, 
__sync_bool_compare_and_swap, 
__sync_lock_test_and_set,
__sync_lock_release

Chances are that you are trying to compile for ARM (or an exotic architecture) and your GCC version is too old compared to the source code you are trying to compile!
There is an Easy fix: upgrade your GCC.

If you can't upgrade your GCC for any reason (for example you are on an embedded hardware you don't have full control on), follow the steps below!

1. Find the source code file that's right for the architecture you are trying to compile on
You are going to find it inside a GCC source tarball.
To find it, go into your gcc source gcc/config and do
grep '__sync_fetch' */*
to find the right file.
For ARM, it is:
gcc/config/arm/linux-atomic.c

2. Compile the source code file and link in to the program you are compiling
libtool --tag=CC -mode=compile gcc -g -O2 -MT linux-atomic.lo -MD -MP -MF linux-atomic.Tpo -c -o linux-atmoic.lo linux-atmoic.c
libtool --tag=CC -mode=link gcc -g -O2 -o liblinux-atmoic.la linux-atmoic.lo
And add liblinux-atomic.la in the Makefile so it is linked to the other .la files (into a .so or a program).

3. Example to compile libtorrent 13.1 and rtorrent 0.9.1 for ARM with GCC 4.2.3
If you wonder, this is to compile rtorrent for my Iomega ix4-200d storcenter NAS.

Compile libtorrent:
PATH=$PATH:/opt/bin
wget http://libtorrent.rakshasa.no/downloads/libtorrent-0.13.1.tar.gz
tar -xvf libtorrent-0.13.1.tar.gz
cd libtorrent-0.13.1
vi configure
OPENSSL_CFLAGS='-I/opt/include/'
OPENSSL_LIBS='-L/opt/lib/ -lssl'
STUFF_LIBS='-L/opt/lib/ -lsigc-2.0'
STUFF_CFLAGS='-I/opt/usr/include/sigc++-2.0/ -I/opt/usr/lib/sigc++-2.0/include'

./configure --prefix=/opt/

Add linux-atomic:
cd src
wget http://dl.dropbox.com/u/50398581/rtorrent-0.9.1/linux_atomic.c
libtool --tag=CC --mode=compile gcc -g -O2 -MT linux_atomic.lo -MD -MP -MF linux_atomic.Tpo -c -o linux_atomic.lo linux_atomic.c
vi /opt/bin/libtool

And if necessary, modify libtool for the follwoing entries:
AR="ar"
RANLIB="ranlib"
CC="g++"

libtool --tag=CC   --mode=link gcc  -g -O2  -o liblinux_atomic.la linux_atomic.lo
vi Makefile

add
liblinux_atomic.la
at the end of libtorrent_la_LIBADD

cd ..
make
strip .libs/libtorrent.so
make install


Compile rtorrent:
wget http://libtorrent.rakshasa.no/downloads/rtorrent-0.9.1.tar.gz
tar -xvf rtorrent-0.9.1.tar.gz
cd rtorrent-0.9.1
vi configure
And add:
sigc_LIBS='-L/opt/lib/ -lsigc-2.0 -L/lib/'
sigc_CFLAGS='-I/opt/usr/include/sigc++-2.0/ -I/opt/usr/lib/sigc++-2.0/include -I/opt/include/ncurses'
libcurl_LIBS='-L/opt/lib/ -lcurl'
libcurl_CFLAGS='-I/opt/include/'
libtorrent_LIBS='-L/opt/lib/ -ltorrent'
libtorrent_CFLAGS='-I/opt/include/'

Then:
./configure --prefix=/opt/ --with-xmlrpc-c=/opt/bin/xmlrpc-c-config  --with-ncurses=yes LDFLAGS='-L/opt/lib/' CPPFLAGS='-I/opt/include -I/opt/include/ncurses/' 
cd src
cp ../../libtorrent-0.13.1/.libs/liblinux_atomic.a .
vi Makefile
at the end of rtorrent_LDADD, add
liblinux_atomic.a

Then:
make
strip rtorrent
cd ..
make install

You are done!