Saturday, 19 November 2016

Raspberry Pi Downloader

So I now have a Pi powered newsgroup indexer and a Pi powered newsgroup server, time for a Pi powered downloader, I'm going to set this up for Torrents and Usenet use, I've added a USB drive to it and mounted it at:

/home/majortom/Downloads

By editing the fstab file with:

sudo nano /etc/fstab

And adding this line to the bottom:

UUID=01669871-a2a3-406e-84a9-f63a9c5fffd0       /home/usernameDownloads        ext4    errors=remount-ro       0       1

You'll have to change the UUID to your drives and username to the user you have on the Pi.

I'm also going to make the downloads folder visible on the network by installing a Samba server.

sudo apt-get install samba samba-common-bin

Make some changes with 

sudo nano /etc/samba/smb.conf

Change the network name and workgroup settings at the top to suit your own use.



And add this to the bottom:
[Downloads]
        path = /home/username/Downloads
        browseable = yes
        public = yes
        writeable = yes
        create mask = 0777
        directory mask = 0777

Let's install SabNZB now and get it working:

sudo apt-get install sabnzbdplus

Edit a config file with:

sudo nano /etc/default/sabnzbdplus

It should look something like this when it's finished.

# This file is sourced by /etc/init.d/sabnzbdplus
#
# When SABnzbd+ is started using the init script, the
# --daemon option is always used, and the program is
# started under the account of $USER, as set below.
#
# Each setting is marked either "required" or "optional";
# leaving any required setting unconfigured will cause
# the service to not start.

# [required] user or uid of account to run the program as:
USER=username

# [optional] full path to the configuration file of your choice;
#            otherwise, the default location (in $USER's home
#            directory) is used:
CONFIG=

# [optional] hostname/ip and port number to listen on:
HOST=0.0.0.0
PORT=8082

# [optional] extra command line options, if any:
EXTRAOPTS=

Now we make it executable with:

sudo service smbd restart

And then it's time to finish off the configuration by pointing a web browser at:

http://machinename:8082


That's newsgroups sorted, let's move onto torrents:

sudo apt-get install deluged deluge-console deluge-web

Now we're going make deluge create a configuration file and then shut it down afterwards.

deluged

sudo pkill deluged

We need to change a files but let's back then up first:

cp ~/.config/deluge/auth ~/.config/deluge/auth.old

At the bottom of the file add the following:

user:password:level

User is the username you're running deluge under, password is the password for accessing the interface and level is 10, so something like:

username:password:10

Now we can run deluge by typing:

delugee

And connect to it with the console by typing:

deluge-console

We need to enter the following into the console to allow remote connections:

config -s allow_remote True

config allow_remote

exit

Time now to install the Deluge client on your desktop, you can get it from here. Once it's installed run it and you'll see something like this, depending on your operating system.




Go to preferences, interface and remove the tick from classic mode


Click Ok and then close down the client and reopen it. You should now get the connection manager popping up, click on Add.




And enter the network name of the Pi or it's IP address along with the username and password you added earlier.





You should now see the console where you can control your torrent client









There you go, you now have a nice low powered download computer.

Leafnode News Server

Following on with my setting up of a Raspberry Pi as a Newsgroup Indexer, I've realised that the next big issue is with news servers themselves. I don't really want a load of newsgroup just a few with a reasonable retention. Some of the paid for servers are suffering from missing posts so I've decided to use a Raspberry Pi with a 500GB drive in it and set up my own. If it works well, I'll eventually add on a bigger drive.

I've installed Raspbian Lite and mounted the external drive at /var/news, one problem is that the drive doesn't seem to mount at boot, this has been fixed by modifying the cmdline.txt file, run:

sudo nano /boot/cmdline.txt

And add rootdelay=5 to the end.

Once you've done all that let's bring the machine up to date with:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo apt-get autoremove

and then restart it with:

sudo shutdown -r now

Once it's rebooted, installation is fairly straightforward with:

sudo apt-get install leafnode

You get asked to enter the details of a news server, I've got Virgin Media here so it's news.virginmedia.com and a permanent connection.

Then the fun begins, you're going to need to edit your hosts file with:

sudo nano /etc/hosts

Edit the line that says

127.0.1.1 machinename

so that it shows

127.0.1.1 nachinename.domain machinename

You can't use localdomain here, so just put your Samba workgroup domain in, then we need to edit another two files so we can connect on the local network:

sudo nano /etc/hosts.allow


#-- leafnode begin
leafnode: 127.0.0.1
#-- leafnode end

Edit this so it looks like

#-- leafnode begin
leafnode: 192.168.0.1
#-- leafnode end

Changing the 192.168.0.1 to whatever your IP address range is and then:

sudo nano /etc/hosts.deny


#-- leafnode begin
leafnode: ALL
#-- leafnode end

Comment out the middle line so it looks like this:

#-- leafnode begin
# leafnode: ALL
#-- leafnode end

Now it's time to fetch a list of newsgroups:

sudo fetchnews -vvv

You can now connect to the server with news reader, I'm going to use Pan but there are loads out there, this is how you do it with Pan





Subscribe to a newsgroup and then go and request the latest headers, this will only give one leafnode entry at the moment, now we want to change some settings before we go any further, just enter:

sudo nano /etc/news/leafnode/config

I'm going to change the expire entry at the top to 400, that's just over a year

Going down the screen I'm changing the initial fetch to 1000, keep going down the screen and set maxfetch to 2000, maxcrosspost to 10 and maxage to 1000. If you're on a slow connection it may be worth you dropping some of these settings

We then run:

sudo fetchnews -vvv

This time it will populate the group(s)

You may find that your current news server doesn't hold all the groups you want so you can edit the leafnode config file again and add some other servers in, here's some you can add in;;


server = free.xsusenet.com
username = username
password = password

You can sign up for a free account with XSUsenet by following this link.


# Free Servers

server = blaine.gmane.org

server = news2.neva.ru

server = news2.informatik.uni-stuttgart.de

server = news-archive.icm.edu.pl


Now run sudo fetchnews -vvv again and it will pull down a list of groups from these servers, make sure that you ask your usenet client to refresh the groups list.

Last thing we have to do is make it update the groups by itself, there is a file in the leafnode folder that you can use but I've done it this way and it works for me:

sudo nano /bin/getnews

Paste the following into it:


texpire -v
fetchnews -vvv
echo


Let#s make it executable with:

sudo chmod +x /bin/getnews

And we'll get cron to run this every 15 minutes with:

sudo crontab -e

Paste this into it:

*/15 * * * * /bin/getnews

Then save it, give it a final reboot and you're all sorted, you're very own news server. You can't add a username and password for access as far as I'm aware so it's not a good idea to make it accessible from the Interwebs.

If you reboot at some point the lockfile may stop leafnode updating, I've created another file called clearlock with:

sudo /bin/clearlock

This goes into it:

rm /var/spool/news/leaf.node/lock.file 

Again, we make it executable with:

sudo chmod +x /bin/clearlock

And we make it run on boot with:

sudo crontab -e

Paste this in.

@reboot /bin/mtclearlock


And we're all done






Sunday, 6 November 2016

Newznab Pi

A lot of the newsgroup indexing sites are either being shut down or are becoming invite only which is a shame as there's a lot of useful stuff in the groups that isn't only illegal downloads. There's quite a few Linux, freeware and shareware groups out there. 

It's time to convert a Raspberry Pi into a usenet indexer, I tried this with an original Pi and it was far too slow but hopefully the model 2 will do the job. All configuration is done from a Linux desktop, if you haven't got an ssh command then you'll need to get something like Putty.

So what do we need, obviously a Raspberry Pi model 2 in a box, something like this will do.



An external hard drive of some sort, I've used an old 320Gb and got hold of a case from Amazon, it's
nothing exciting but it works, I ordered the case on Saturday evening and it arrived on Sunday lunchtime, good going there Amazon.

A USB hub also makes life easier, nice cheap one from Maplin. You're also going to need an ethernet lead to connect it to your router, £1 in Poundworld.

Some coffee and a sausage inna bun will help.

It's time to make a start, put it all together but don't worry about connecting the external drive yet, we need to get hold of Raspbian Jessie Lite from here. Extract it and then copy it to a micro SD card, easiest way is to use a cross platform app called Etcher

Once it's finished, put the card into the Pi and give it some power. Have a look at the connected devices in the router config and ssh into the pi with ssh pi@192.168.0.253 or whatever yours says, password to connect is raspberry. We now run:

sudo raspi-config

You should see this:


Scroll down to advanced, then down to update, once that's completed go back down to advanced options and put in your own hostname, mine is called Orac as the Pi is in a clear case just like Orac in Blakes 7, under advanced we want to select boot to console requiring user to login and enable ssh. Once last thing under advanced is memory split, set this to 16 to give as much RAM to the running of the Pi as possible. Last thing we do is select Expand Filesystem at the top.

Select finish and wait for a reboot, you should now be able to connect with 

ssh pi@orac

 or whatever you've called your pi.

Now we create a normal user with

sudo adduser username

then add them to the superuser group with

sudo adduser username sudo

Reboot with 

sudo shutdown -r now 

Now you should be able to login with ssh orac or ssh username@orac.

Let's remove the default Pi login with

sudo deluser pi

We need some helpful utilities now, so let's enter:

sudo apt-get install rsync mc

And now it's time to connect up the external drive.

Type dmesg at the command line and it should come up as /dev/sda or something similar, we now need to partition it so we enter:

sudo fdisk /dev/sda or whatever yours came up as.

Then D to delete any existing partitions, then N to create a new one, followed by P for primary partition, then enter 3 times followed by w to write the changes. There may be other options here depending on how many partitions there were on the drive when you started.

Next we have to format the drive with:

sudo mkfs.ext4 /dev/sda1

Now create a temporary mount point with 

sudo mkdir /mnt/temp

Then copy the filesystem to the external drive with:

sudo rsync -axv / /mnt/temp

Let's back some config files with:

sudo cp -R /boot /boot.orig

Time to edit some  stuff to allow us to boot from the external drive:

sudo nano /boot/cmdline.txt


The original looks like this:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

It's all on one line, we need to change the root=/dev/... bit to read root=/dev/sda1 or whatever your USB stick shows, we also need to add a delay to the end with rootdelay=5, it should look like the one below in the end.

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait rootdelay=5

Save it and then we need to edit fstab with:

sudo nano /etc/fstab

The following entry needs to be added:

/dev/sda1    /   ext4    defaults,noatime  0       1

And the line referring to the sd card needs to be commented out, mine looks like this now



proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
# /dev/mmcblk0p2  /               ext4    defaults,noatime  0       1

/dev/sda1    /   ext4    defaults,noatime  0       1


We now should be running from the external drive, type df -h to check the space, so now we move onto getting Newznab installed, there's a few utilities we need that don't seem to be in the raspbian repository, I've precompiled these so you can get them with these commands:

wget https://dl.dropboxusercontent.com/u/27964370/ffmpeg_3.1.1-1_armhf.deb

and 

wget https://dl.dropboxusercontent.com/u/27964370/unrar_5.2.7-0.1_armhf.deb

Let's get them installed and the system updated now

sudo apt-get update

sudo apt-get upgrade -y

sudo dpkg -i *.deb

Let's make some folders and set permissions:

sudo mkdir /var/www
sudo mkdir /var/www/newznab
sudo chmod 0777 /var/www/newznab

And now install some stuff

sudo apt-get install php5 php5-dev php-pear php5-gd php5-mysql php5-curl mysql-client-5.5 libmysqlclient-dev apache2 mysql-server-5.5 lame mediainfo

Make sure you put in a good strong mysql passqword if you're going to get access to this from the Interwebs.

Now we make some changes to the php configuration file:

sudo nano /etc/php5/cli/php.ini

Under the resource limits section we need to change the maximum execution time to 120:

max_execution_time = 120

Then under Module Settings we need to change the time zone settings, in my case I'm the UK so I'm going to change it to:

date,timezone = Europe/London

Save that and then time to edit another one:

sudo nano /etc/php5/apache2/php.ini

Under the resource limits section we need to change the maximum execution time to 120:

max_execution_time = 120

and

memory_limit = -1  

Then under Module Settings we need to change the time zone settings, in my case I'm the UK so I'm going to change it to:

date,timezone = Europe/London

Now we create the Apache config file for Newznab

sudo nano /etc/apache2/sites-available/newznab.conf

And paste the following into it:

<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName localhost

DocumentRoot /var/www/newznab/www
ErrorLog /var/log/apache2/error.log
LogLevel warn
</VirtualHost>
Change the port if you want it to run on a different port.

One more change to the Apache configuration file before we go on:

sudo nano /etc/apache2/apache2.conf

Find this bit

<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
</Directory>

And change it to this:

<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
</Directory>



We now need to get hold of Newznab itself, you can get it from here. Newznab classic works but there are some limitations, if you are serious just pay the money for a better product and you get free upgrades too.

Get the file downloaded and then we need to extract it and copy it to the right place, 

As I have a paid for version I'm going to use a slightly different approach, for this you need to install subversion with the following command:

sudo apt-get install subversion

Then to get Newznab you enter the following command:

sudo svn co svn://svn.newznab.com/nn/branches/nnplus /var/www/newznab

At this point you will be asked for the root password for your system and the SVN login details that have been emailed to you by the Newznab team.

Now it's time to activate the changes:

sudo a2ensite newznab
sudo a2dissite 000-default
sudo a2enmod rewrite
sudo service apache2 restart

Change some folder permissions with:

sudo chmod 777 /var/www/newznab/www/lib/smarty/templates_c
sudo chmod 777 /var/www/newznab/www/covers/movies
sudo chmod 777 /var/www/newznab/www/covers/music
sudo chmod 777 /var/www/newznab/www/covers/tv
sudo chmod 777 /var/www/newznab/www
sudo chmod 777 /var/www/newznab/www/install
sudo chmod 777 /var/www/newznab/nzbfiles/
sudo chmod 777 /var/www/newznab/www/covers/anime
sudo chmod 777 /var/www/newznab/www/covers/tv

And that's it for the moment, you now need to finish setting up Newznab from a browser with:

http://servername/install 

Once you've done all that, run

sudo chmod 0777 /var/www/newznab/nzbfiles/tmpunrar

And install phpmyadmin with:

sudo apt-get install phpmyadmin

Go to the site admin options and edit site, it works much better if you put in your own Rotten Tomatoes api, your own Amazon settings and your own Newznab ID, the section that asks for the path to some files should show this:

/usr/bin/unrar
/usr/bin/mediainfo
/usr/bin/ffmpeg
/usr/bin/lame

Once that's all completed, go to the View Groups option and make some groups active.
 Then it's back to the command line with

sudo -i

cd  /var/www/newznab/misc/update_scripts

Then run 

php update_binaries_threaded.php

after that it's

php update_releases.php 

That should start populating the indexer, it's up to you how you want to run this, you can it manually but I run a loop with a 10 minute pause each time it restarts.

I've made a file with 

sudo nano /bin/newznabrun

Put this into it


while :
do


# Beginning Update
cd /var/www/newznab/misc/update_scripts/
php update_binaries_threaded.php
php update_releases.php

        echo "Press [CTRL+C] to stop.."
        sleep 360

done

Save the file and then run

sudo chmod +x /bin/newznabrun 

You can then type sudo newznabrun to get it going.

That's it all done, hope it all works for you.


Big thanks to the howtogeek website for help with these instructions, I've added a few bits and pieces to get to this stage.