Archive for the ‘Linux’ Category

Linux Backup Strategy Using Backupninja

Monday, August 30th, 2010

No-one likes to loose digital data, especially precious memories in photos that can’t be replaced. I’ve had two pretty major hard drive failures in the last year and there’s nothing like the thought of having lost photos or other important information to give you the kick in the arse you need to make sure you have a backup strategy that is being followed.

About 9 months ago I had a 1TB hard drive start to fail. It had 360+ identified reallocated sectors and had numerous I/O errors while trying to access specific areas of the drive. That drive had all our photos on it. Luckily I got most of the data off it and, combined with a simple manual backup I was doing to an external drive, I didn’t loose a thing. It was a painful and somewhat stressful process restoring and checking the recovered data and could have easily resulted in significant data loss.

I needed a more robust backup process. Our entire home computing system is based on Linux - Ubuntu Lucid Lynx 10.04. This includes a desktop computer and a HTPC that triples as NAS, VDR using MythTV, and media centre using XBMC Media Centre. After review of the backup options available I finally settled on using backupninja. It’s a piece of cake to install as it’s available in the Ubuntu repositories, is easy to setup, has good documentation, and has just the right amount of notification related bells and whistles to let me know that things are running when they should, and when things don’t go as they should.

The backup strategy I use now is pretty straight forward, with all the important information on those 2 machines being backed up to 3 places using backupninja:

  1. Automatically to the other computer on a daily basis
  2. Manually to an external drive stored in our detached garage (in case the house burns down or we are burgled) on a weekly basis
  3. Manually to an external drive stored at work (in case both the house and garage burn down) on a weekly (turning out to be more like monthly) basis
  4. Future: Automatically to an Amazon S3 account in the cloud (only for the very most important information since bandwidth is an issue here in New Zealand)

Backupninja provides built in support for all the types of information I needed backing up and includes support for setting them up using the curses based ninjahelper interface. Configuring the necessary files manually is also ridiculously easy.

  • System Information dump - e.g. hard drive partition information, installed packages
  • MySQL database dump - for all of the MythTV recorded TV show information
  • File backup using rdiff-backup over SSH or directly to a locally mounted drive - for all our photos and other multimedia
  • Future: File backup using duplicity to Amazon S3 - for all our photos

To install backupninja and the other utilities that it can leverage, use the following command line:
$ sudo apt-get install backupninja, debconf-utils, hwinfo, rdiff-backup

Below is part of a sample configuration file for an rdiff-backup configuration to show how easy it is to configure:
include = /etc
include = /home
include = /root
include = /mnt/drive1/multimedia
include = /usr/local/*bin
include = /var/backups
include = /var/lib/dpkg/status*
include = /var/spool/cron/crontabs
exclude = **/.ccache
exclude = **/.local/share/Trash
exclude = **/.thumbnails
exclude = **/.Trash*

There is oodles of other useful information in the Backupninja Wiki so go and have a look at that if you want to know more.

I’m also using smartmontools to monitor the SMART status of the hard drives in all our machines totalling ~3TB of data across 5 drives. Getting a heads up that you might need to move data off a drive onto a new one before it fails is sometimes the easier option than restoring from a backup.

Putting a backup strategy in place is only something most people consider once they’ve already lost data - the same way that most people only get a house alarm after they have been burgled. If you need to go into your house to access any important digital information, then you are at risk of loosing something. Be smart, do something before you lose something.

Switch from Kubuntu to Ubuntu

Tuesday, December 25th, 2007

I’ve just made the move to Ubuntu from Kubuntu.

I started Linux life with an Kubuntu Live CD borrowed off a MythTV user from work over a year ago. Having been frustrated with how much configuration was necessary to do what seemed like sensible things I installed the ubunutu-desktop package about a week ago. I don’t think I’ll be going back.

Ubuntu works fantastically well with the default configuration. It has lots of the things I commonly do all out in the open

  • The Applications menu works well and uses Fitt’s law to good effect in the top left hand corner
  • The Places menu is way better than the similar analogy in KDE
  • The System menu is also much better on it’s own than buried in the KDE menu
  • Accessing Logout from the top right works well (again Fitt’s law comes into play)
  • Configuration of the desktop just seems easier - a good choice of sensible defaults helps
  • It has a great look and feel out of the box - after months of trying to get something consistently decent for KDE I gave up
  • Putting shortcuts to commonly used applications and other useful things in the top bar is easy - also a great default setup
  • A much easier way of switching users out-of-the-box
  • Many others…

It’s a shame it was so hard getting rid of all the KDE related applications. A few hours tidying with Add/Remove Applications and Synaptic Package Manager seems to have cleaned out most things.

Overall it seems like a great improvement to the usability of the common things that I use Linux for. Awesome.

And, since Shelly hasn’t had any significant problems yet I guess it’s a thumbs up from her as well.

Getting direct rendering to work with Kubuntu Gutsy (7.10) and Intel GMA 950 chipset

Saturday, October 20th, 2007

I spent eons this afternoon trying to get direct rendering to work with Kubuntu Gutsy (7.10) so the recently installed Compiz Fusion visual effects actually look remotely impressive rather than clunky and old skool. I am using AIGLX as opposed to XGL as my Shuttle SD32G2 server has an integrated Intel GMA 950 chipset which AIGLX supports well in Linux.

After much chagrin and reading of numerous bugs and waffly semi-related threads I finally found one page that actually resolved the problem (xserver-xgl on gutsy doesn’t work). I removed the xserver-xgl package from the server and double checked that everything for AIGLX was setup correctly. HOWTO AIGLX contains instructions for configuring AIGLX targeted for Gentoo but should work fine for most Linux distros.

glxinfo | grep direct now returns the much desired direct rendering: Yes output. Wohoo! No longer is scrolling in Firefox and other applications painfully slow and jumpy. The visual effects in Compiz Fusion are also now crisp and smooth as opposed to jagged and choppy. Excellent. Not sure it was worth the hours of what felt like banging of my head against a brick wall though.

It’s unfortunate that the problem may have arisen as I may have installed the xserver-xgl when I first started configuring the server to use XGL for Compiz. Rats!

KDE doesn’t start after upgrading to Kubuntu Gutsy 7.10

Saturday, October 20th, 2007

I can’t believe that the latest release of Kubuntu Gutsy (7.10) has been in beta for so long. The upgrade rendered my system all but useless because KDE wouldn’t start.

When I logged in the only thing that was presented to me was the new Kubuntu background. Nothing more, nothing less.

Starting Kubuntu with the Session in Failsafe mode presented a terminal window with the new Kubuntu background behind it. This was all it did, but I presume this is all that Failsafe mode does.

Starting the X server using startx threw an error about the user not being authorized. I found a few links like X: user not authorized to run the X server, aborting with similar looking problems. This error was resolved using dpkg-reconfigure x11-common and changing Allowed Users to “Anybody” instead of “Console”.

Running startkde then had KDE fire up but then bail back to the login screen. The last thing I saw in the terminal window was

No battery found.
This is not a laptop, quitting…

This had me puzzled for a while since it looked like this might have been the problem. However, it turns out that is simply the output of the guidance-power-manager. Nothing to be concerned with it would seem, just an unfortunate piece of wording in that output.


Configuring a Shuttle SD32G2 to wakeup using ACPI from Linux

Friday, June 8th, 2007

I recently configured my Shuttle SD32G2 MythTV PVR server to automatically wakeup when it needed to record a scheduled TV recording. Many of the examples on the web for doing this from MythTV used NVRAM. However, after installing NVRAM it didn’t recognise the BIOS. Given how NVRAM splashes about in sensitive memory in the BIOS I wasn’t too keen to try my luck getting that to work.

Some other web pages such as ACPI Wakeup mentioned being able to use the Linux Advanced Configuration and Power Interface (ACPI) instead. This was a much better option and I soon had everything working using simple command line scripts.

To configure your Shuttle SD32G2 server to work with ACPI all you need to do is:

  1. Start the machine up and go into the BIOS configuration (hold down Delete while the machine boots)
  2. Select Power Management Setup
  3. Change the Resume By Alarm option to Enabled
  4. Save the BIOS configuration change and reboot the machine

This configures the BIOS so it will indeed wake up when the alarm is set.

The following are notes that relate to configuring the use of the ACPI alarm in Linux (based on Kubuntu Feisty Fawn 7.04)

  • The date format for the ACPI alarm for the SD32G2 appears to be yyyy-MM-dd hh:mm:ss. The date format can be confirmed by executing cat /proc/acpi/alarm and looking at the alarm time that is already set in the BIOS.
  • The SD32G2 doesn’t appear to support setting the day of month for the wakeup alarm. This should be fine for TV recording with my configuration since the machine will simply boot up on the specified time on the next day and then shut down 2 minutes later if it doesn’t start recording.
  • If the day is set to 0 (which always seems to appear), then than means start up at the specified time on every day.
  • The wakeup alarm can be set using echo "2007-06-11 16:23:00" > /proc/acpi/alarm if run as root or inside a script executed using sudo
  • The wakeup alarm can be set using sudo sh -c "echo '2007-06-11 16:23:00' > /proc/acpi/alarm" if using sudo. Trying to simply do a direct sudo echo always gave me a permission problem.
  • Content located at ACPI Wakeup is quite useful. The Shuttle BIOS did not need any modifications to the script though and don’t bother with the clumsy script to alter the time from local time to UTC. Just alter the BIOS clock to use the local time.
  • The BIOS time can be changed from UTC to local time in Kubuntu by opening the /etc/default/rcS file and changing UTC=yes to UTC=no. Much easier all round.
  • The BIOS clock can be set from the system clock using sudo hwclock -systohc. I’m reasonably sure the Kubuntu does this on shutdown.

MythTV server now starts up and shuts down automatically

Friday, June 8th, 2007

I’ve spent the last few nights playing with the wonderful world of Linux Advanced Configuration and Power Interface (ACPI) so that the MythTV server will now automatically start up to record a scheduled TV show, and automatically shut down afterwards if there are no logged in users.

This is pretty cool because it means I won’t miss recording any more of the Americas Cup Highlights because I didn’t get out of bed early enough to turn on the server.

It was a pretty arduous task with a number of steps based on content from a number of different web-sites to get things to work. After all the research though it’s not actually that complicated to setup. At a high level the following was done (subsequent blog posts will cover off some of the more tricky bits in more detail):

  1. Configure Power Management in the BIOS of your machine to allow being resumed by alarm
  2. Install appropriate bits for Linux to support ACPI
  3. Alter the Linux configuration so the BIOS uses local time rather than UTC time (since MythTV works using local time)
  4. Ensure that the ACPI alarm interface works as expected when set
  5. Create a small script to set the ACPI alarm given a date
  6. Create a small script to check to see if it is OK to shutdown the server
  7. Configure MythTV to use the two scripts created above
  8. Alter the sudo permissions for the MythTV user so the scripts can be executed without permission problems
  9. Alter the login permissions so only root can shut the server down (to prevent accidental shutting down when the server is recording

Getting XBMC MythTV to work with Kubuntu Feisty Fawn (7.04)

Monday, April 30th, 2007

After upgrading to Feisty Fawn (7.04) of Kubuntu I thought I had made all the necessary changes for my XBMC (XBox Media Center) MythTV frontend to talk to the Kubuntu server running the MythTV backend. Part of the Kubuntu upgrade altered the /etc/mysql/my.cnf configuration file for MySQL. This alteration required me to change the bind-address from localhost to the static IP addess of the server.

I thought that this change would be sufficient but it turns out that it wasn’t. The MythTV frontend on my XBox failed to connect to the MySQL database on the MythTV backend for completely non-obvious reasons. If the error output in the MythTV frontend was better I probably would of identified the problem much more quickly. After some hours of looking closely at the MySQL configuration changes and various other network related settings in Kubuntu it looked like it might have something to do with the MySQL password handling.

The old /etc/mysql/my.cnf had a old_passwords = 1 setting in it. The new configuration file did not. In addition, within the newly added configuration import directory /etc/mysql/conf.d there was an old_passwords.cnf file that explicitely set old_passwords = false. This gave me enough clues to see if the MythTV front end for the XBMC was in fact using the old password authentication mechanism (no longer supported by the server with this configuration). I found plenty of references to versions of PHP needing to use the old password authentication mechanism for MySQL and wondered if there was a similar issue with Python (the language used for the MythTV frontend on the XBMC). That information was enough to convince me to try and find out how to configure the server to continue to use the old password mechanism to see if that changed anything.

Based on instructions in the MySQL reference documentation at Client does not support authentication protocol I first tried to get the MySQL daemon mysqld to start with the --old-passwords parameter so it worked like it would have prior to the upgrade. I couldn’t get this to work (the MySQL documentation is a bit sparse on how to set some of the configuration options) so I tried to set the password for the mythtv user using the OLD_PASSWORD() function instead of the PASSWORD() function. By a stroke of luck that actually worked. That’s where I’m leaving it too.

If it aint broke, don’t fix it.

So, at the moment the MySQL configuration is exactly the same way as it was after upgrading to Kubuntu Feisty Fawn (7.04) apart from the bind-address being set to the static IP address of the server. In addition, the MySQL password for the mythtv user has been reset using the OLD_PASSWORD() function.

  1. Login to MySQL as the root user
  2. mysql -u root -p
  3. SET PASSWORD for mythtv = OLD_PASSWORD('mythtv'); - where ‘mythtv’ is the password that is configured in the MythTV settings on the XBox.
  4. Test the settings in MythTV on the XBox
  5. Pat yourself on the back when it works and then go to bed…

This was pretty frustrating to have to deal with. If the error getting spat out of the MythTV frontend on XBMC was better this problem would have been much easier to find with a bit of searching on Google.

Ripping DVDs on Linux using AcidRip

Sunday, April 29th, 2007

One of the main reasons for moving to Linux from than Windoze was to be able to use the Linux based MythTV PVR (Personal Video Recorder). Since the Linux machine will hold lots of recorded TV it might as well hold a bunch of ripped DVDs as well. Finding a good Linux DVD ripper that gives you just enough control to get a great rip was harder than I thought it should be.

I’ve been playing around with AcidRip for a while now - playing with lots of the settings after reading about all the ins and outs of video encoding. AcidRip does some really useful stuff for you like automagically doing crop detection so you don’t end up wasting oodles of disk and time encoding the black bars at the top and bottom. It also lets you do a preview as well, which is mighty useful for checking that you’ve got the right crop settings and that you’ve actually chosen the correct sound track. Another bonus is that it encodes directly from the DVD. Under the hood it uses mencoder which is the other half of the popular video player mplayer. This means you can play with your hearts content with the huge array of settings that mencoder supports.

If you are happy using the LAVC or XVID encoding algorithms then AcidRip should work great for you. There are several articles floating on the internet about how to use it which were useful for the high level bits.

There are some down sides though. It doesn’t look like it’s been updated for a while and support on the AcidRip forum at Source Forge was almost non-existent. Also, it doesn’t really support any of the others encoders very well at all. I tried to use the newer X264 encoding but AcidRip doesn’t understand how to do anything more than a single pass encoding, which is by all means useless if you want a good quality encoding. Fortunately my XBox Media Center doesn’t fully support X264 encoding yet so I’m ripping using XVID, which AcidRip supports pretty well.

Upgrading Kubuntu from Edgy Eft (6.10) to Feisty Fawn (7.04)

Sunday, April 29th, 2007

Last night I finally got around to upgrading from the Edgy Eft version of Kubuntu to the recently released Feisty Fawn version. The process was amazingly simple. I updated all of the existing packages that needed upgrading and then the system very nicely told me that there was a completely new version of Kubuntu that I could upgrade to. So off I went and started the upgrade process.

The only place that I had any problems was with the MySQL configuration. A number of modified configuration files needed to be replaced as part of the installation. I had knowingly modified the /etc/mysql/my.cnf file so it had a properly networked bind-address. I needed to do this so my Xbox Media Center (XBMC) Myth TV client can connect to the MythTV backed database.

The entire process took just over an hour and overall I think it downloaded around 700Mb of data. This put a pretty big dent in the monthly download quota but it came down at approximately 350kbps so didn’t actually take that long. Who knows how much I’ve downloaded via Windows updates over the course of a year. So far everythings working as expected and fingers crossed it stays that way.

Overall I’m really impressed at how easy and relatively painless the upgrade was. Well done to the guys at Kubuntu who put the 7.04 release together.

Fixed video output device problem with MPlayer when opening files

Monday, April 16th, 2007

For a while I haven’t been able to play video’s using Mplayer from the KDE GUI in Kubuntu. Playing files through the command line worked fine though. It only fails when using the "Open With" command from the Konqueror GUI and choosing "MPlayer Movie Player" I’d always get a Fatal Error box popping up saying:

Error opening/initializing the selected video_output (-vo) device.

I’ve been using MPlayer quite a lot from the command line since I’ve been doing a bit of DVD encoding of late using AcidRip so it hasn’t been too problematic.

Anyway, the problem is now resolved and what follows are some simple instructions that worked for me.

  1. When the problem occurs the MPlayer window should still open (it just doesn’t play what you asked it to).
  2. In the MPlayer window, right-click and choose Preferences
  3. Choose the Video tab.
  4. Change the highlighted Video Driver to something else. Mine was set to xmga, Matrox G200/… I don’t have a Matrox video card so that is probably why MPlayer was barfing. I changed it to xv, X11/xv and now it is working fine.
  5. Click the OK button