Archive for the ‘Tips and Tricks’ Category

Linux Backup Strategy Using Backupninja

Monday, August 30th, 2010

No-one likes to loose digital data, especially precious memories in photos that can’t be replaced. I’ve had two pretty major hard drive failures in the last year and there’s nothing like the thought of having lost photos or other important information to give you the kick in the arse you need to make sure you have a backup strategy that is being followed.

About 9 months ago I had a 1TB hard drive start to fail. It had 360+ identified reallocated sectors and had numerous I/O errors while trying to access specific areas of the drive. That drive had all our photos on it. Luckily I got most of the data off it and, combined with a simple manual backup I was doing to an external drive, I didn’t loose a thing. It was a painful and somewhat stressful process restoring and checking the recovered data and could have easily resulted in significant data loss.

I needed a more robust backup process. Our entire home computing system is based on Linux - Ubuntu Lucid Lynx 10.04. This includes a desktop computer and a HTPC that triples as NAS, VDR using MythTV, and media centre using XBMC Media Centre. After review of the backup options available I finally settled on using backupninja. It’s a piece of cake to install as it’s available in the Ubuntu repositories, is easy to setup, has good documentation, and has just the right amount of notification related bells and whistles to let me know that things are running when they should, and when things don’t go as they should.

The backup strategy I use now is pretty straight forward, with all the important information on those 2 machines being backed up to 3 places using backupninja:

  1. Automatically to the other computer on a daily basis
  2. Manually to an external drive stored in our detached garage (in case the house burns down or we are burgled) on a weekly basis
  3. Manually to an external drive stored at work (in case both the house and garage burn down) on a weekly (turning out to be more like monthly) basis
  4. Future: Automatically to an Amazon S3 account in the cloud (only for the very most important information since bandwidth is an issue here in New Zealand)

Backupninja provides built in support for all the types of information I needed backing up and includes support for setting them up using the curses based ninjahelper interface. Configuring the necessary files manually is also ridiculously easy.

  • System Information dump - e.g. hard drive partition information, installed packages
  • MySQL database dump - for all of the MythTV recorded TV show information
  • File backup using rdiff-backup over SSH or directly to a locally mounted drive - for all our photos and other multimedia
  • Future: File backup using duplicity to Amazon S3 - for all our photos

To install backupninja and the other utilities that it can leverage, use the following command line:
$ sudo apt-get install backupninja, debconf-utils, hwinfo, rdiff-backup

Below is part of a sample configuration file for an rdiff-backup configuration to show how easy it is to configure:
include = /etc
include = /home
include = /root
include = /mnt/drive1/multimedia
include = /usr/local/*bin
include = /var/backups
include = /var/lib/dpkg/status*
include = /var/spool/cron/crontabs
exclude = **/.ccache
exclude = **/.local/share/Trash
exclude = **/.thumbnails
exclude = **/.Trash*

There is oodles of other useful information in the Backupninja Wiki so go and have a look at that if you want to know more.

I’m also using smartmontools to monitor the SMART status of the hard drives in all our machines totalling ~3TB of data across 5 drives. Getting a heads up that you might need to move data off a drive onto a new one before it fails is sometimes the easier option than restoring from a backup.

Putting a backup strategy in place is only something most people consider once they’ve already lost data - the same way that most people only get a house alarm after they have been burgled. If you need to go into your house to access any important digital information, then you are at risk of loosing something. Be smart, do something before you lose something.

New Zealand Freeview|HD running on XBMC from Myth TV

Monday, June 8th, 2009

UPDATE (20-Aug-2009): The bulk of the patch has now been committed to XBMC in revision 22285. A few extra goodies added to the patch now. Might cause some conflicts once these are committed, but that might not be for a week or more.

UPDATE (18-Aug-2009): A recent update of ffmpeg in XBMC (revision 21993 on 04-Aug-2009) meant that the original patch and instructions no longer worked. The instructions below have been updated and a new patch released. Much of the original patch is now directly in XBMC Media Center due to the ffmpeg update so it is much smaller.

I maintain the Myth TV page in the XBMC wiki. Some people in New Zealand have contacted me to get instructions on what needs to be done to get recorded Freeview|HD content playing in XBMC from a Myth TV backend.

Well, here those instructions are. Luckily there isn’t as much voodoo magic as there used to be. There will be even less voodoo once some of my other patches make it into XBMC proper, and Paul Kendall’s patches for LATM encapsulated AAC decoding finally wiggle their way into ffmpeg.

These instructions assume:

  • You already have Myth TV working as expected with the digital terrestrial Freeview|HD broadcast. Getting this working isn’t covered here. Go to Myth TV NZ as a starting point or join the mythtvnz mailing list for more information.
  • You have seen the light and are using Linux and not Windoze
  • You have a level of confidence with Linux
  • You are happy to compile XBMC Media Center from source code (and have some concept of what compiling is)
  • You have a smoking fast CPU. A dual core 3.0GHz or faster CPU is needed or an nVidia video card / chipset that supports VDPAU. Decoding 1080i H264 content from TV3 is super CPU intensive or needs to be offloaded to the GPU.

The New Zealand Freeview|HD digital free-to-air transmission DVB-T video stream uses the H264 codec (along with PAFF interlacing for TV3). The DVB-T audio stream uses either AC3 or LATM (Low-overhead MPEG-4 Audio Transport Multiplex) encapsulated HE-AAC. The audio and video streams are transmitted within an MPEGTS container.
 (more…)

Fix to receive Vodafone voice message text messages on the iPhone

Saturday, December 1st, 2007

I stumbled upon a fix that allows text messages from Vodafone New Zealand to appear when you miss a call and someone leaves a voice message.

The fix has been tested with iPhone firmware version 1.1.1.

  1. Open Installer, and add the following source if not already there - http://i.unlock.no/
  2. Choose the Unlocking Tools category.
  3. Install SMS Fix

This fix was a frequently asked questions titled SMS received from internet is scrambled/not working? at iphone.unlock.no.

Fix for XBMC MythTV script database connection failure after MythTV upgrade on (K)Ubuntu

Saturday, November 24th, 2007

I don’t understand why the password for the mythtv MySQL database user is reset whenever the MythTV package is updated in Ubuntu (and it’s derivatives like Kubuntu).

When this happens the Xbox Media Center (XBMC) fails to connect to the database. It’s somehow related to the password authentication mechanism that the MySQL libraries use on the XBox within XBMC.

Here are the instructions from one of my older posts to get the XBMC MythTV script to be able to connect to the MythTV database again. These steps are performed on the MythTV server.

  1. Login to MySQL as the root user - mysql -u root -p
  2. SET PASSWORD for mythtv = OLD_PASSWORD('mythtv');

Use the appropriate password for the mythtv user - the default is mythtv.

Apple iPhone 1.1.1 update instructions for New Zealand

Sunday, November 18th, 2007

I finally decided to update the iPhone from 1.0.2 to 1.1.1. I’ll still be a version behind as version 1.1.2 was released by Apple earlier in November.

The update process was reasonably straight forward. No problems like Ryan Maxwell of Red Ink Scribbles had - Dreaded 1.1.1 Update. I found and used a different method though. There are lots of upgrade methods to choose from, but the one used below seemed to be the easiest. You don’t even need any software on your computer.

  1. Virginze the 1.0.2 phone - How to virginize an unlocked phone before update to 1.1.1
  2. Upgrade to 1.1.1 - How should I upgrade or restore my phone? Be careful not to let iTunes do the upgrade automatically as it will use 1.1.2 instead of 1.1.1.
  3. Activate and unlock the SIMhttp://iphone.unlock.no
  4. Apply New Zealand specific tweaks
    • Open Installer and add the XK72 repository - http://xk72.com/iphone/repos/
    • Open the Tweaks (XK72) category and install all non 1.0.2 packages - Caller ID update (1.1.1), NZ Phone Formats (1.1.1) and NZDT 2007 update.
    • Update: Open Settings > General > International > Region Format and choose New Zealand
    • Open the Unlocking Tools category and install Edge Settings fix
    • Open Settings > General > Network > Edge and set the APN to direct.vodafone.net.nz
    • Go to Phone and manually dial *5005*86*707#. This will change the Voicemail button so it calls 707 rather than not doing anything.
  5. Apply SMS Patch - Open Installer and then open the Unlocking Tools category and install SMS Fix

Allow about an hour to get through everything. Leave a comment if you have any questions.

Really quietening down a Sundance Spa SunZone ozone generator

Monday, October 29th, 2007

I thought I had got lucky with my last attempt at stopping the intermittent humming noise coming from the ozone generator in the Sundance Spa. The last attempt was a bit unscientific and can be read at an earlier post.

I didn’t work long term. The noise came back last week.

While in the spa on Friday evening I noticed that the bubbles coming out of the circulation jet outlet were also intermittent. Every now and then some larger bubbles would come out. So I started playing with the outlet of the circulation pump with my foot. If part of the outlet was covered off the noise from ozone generator stopped and so did the larger bubbles. Interesting.

I popped the spa panels off at the weekend and looked more closely at the line running from the mazzi jet to the SunZone ozone generator. Sure enough I could see very small air bubbles getting sucked into the air line where it connected to the mazzi jet. A few zip ties later and that connection was sealed. As soon as it sealed the noise stopped as well. Voila!

I zip tied all of the other connectors in that same air line and checked the connections for all of the other components as well. I also zip tied the air line well out of the way of the other components. Now the spa really is quiet with a consistent stream of small bubbles coming out of the circulation pump. Excellent! Hopefully this problem is sorted for good now.

Quietening down a noisy SunZone ozone generator in a Sundance Spa

Sunday, October 21st, 2007

The Sundance Spa has been making a low humming/buzzing noise on and off for the past few weeks. This noise is quiet in comparison to the gurgling noise the spa used to make before the circulation pump was replaced, but it’s still annoying.

The low frequency humming/buzzing noise was intermittent which made it all the more obvious as it came and went. It sounded like something was loose or vibrating within the spa.

I took off the front panels of the spa to isolate the noise, which turned out to be coming from the SunZone ozone generator. This device uses high intensity UV light to create ozone that is sucked into the circulation water supply to help with sanitation.

To try and get rid of the noise, I thought I’d see if anything was obviously loose in the generator. After turning off the power to the spa I removed the SunZone ozone generator which was only held on by 4 wood screws. On the back though, there were at least 12 metal screws, 3 bolts and 2 rivets. Too many things to make pulling it apart simple.

Onto plan B. I held the unit at various angles and gently thumped the unit with the base of my hand in the hope that if something were loose it would move enough that it wouldn’t vibrate any more. It was a bit of a long shot but it was worth a try.

After reinstalling and turning the power back on I was pleasantly surprised to not be hearing that annoying buzz anymore. Magic! Sometimes I get lucky.

While I was there I also zip-tied the hartford loop in the ozone air line up above water level. Previously it was just sitting on the top of the circuit board housing. Not sure if that had any affect on the noise.

Yet another iPhone running in New Zealand

Monday, October 8th, 2007

I am lucky enough to be one of the many New Zealanders using an iPhone. Many have gone before and many more will follow. The first hacked iPhone may belong to John Ballinger of Bluespark Interactive as described at Andrew James Sommervell’s blog.

The Red Ink Scribbles iPhone Review Part II: The Unlocked Phone blog post contains the best instructions I’ve seen to configure the iPhone correctly for New Zealand after it has been unlocked. It contains instructions to:

  • Configure GPRS access for Vodafone (not full blown EDGE, just it’s baby sister equivalent)
  • Configure the Voicemail button so it dials 707 to access Vodafone voicemail
  • Change the phone number formating to suit New Zealand phone numbers. I’ve further updated the ABPhoneFormats.plist phone number formats to also handle 8 digit prepay phone numbers (the weird ones that look like 021 027xxxxx).

Other changes are also needed so Caller ID matches the locally stored contact phone numbers correctly. The Fix International Caller ID thread at ModMyiPhone contains instructions to fix this.

Configuring a Shuttle SD32G2 to wakeup using ACPI from Linux

Friday, June 8th, 2007

I recently configured my Shuttle SD32G2 MythTV PVR server to automatically wakeup when it needed to record a scheduled TV recording. Many of the examples on the web for doing this from MythTV used NVRAM. However, after installing NVRAM it didn’t recognise the BIOS. Given how NVRAM splashes about in sensitive memory in the BIOS I wasn’t too keen to try my luck getting that to work.

Some other web pages such as ACPI Wakeup mentioned being able to use the Linux Advanced Configuration and Power Interface (ACPI) instead. This was a much better option and I soon had everything working using simple command line scripts.

To configure your Shuttle SD32G2 server to work with ACPI all you need to do is:

  1. Start the machine up and go into the BIOS configuration (hold down Delete while the machine boots)
  2. Select Power Management Setup
  3. Change the Resume By Alarm option to Enabled
  4. Save the BIOS configuration change and reboot the machine

This configures the BIOS so it will indeed wake up when the alarm is set.

The following are notes that relate to configuring the use of the ACPI alarm in Linux (based on Kubuntu Feisty Fawn 7.04)

  • The date format for the ACPI alarm for the SD32G2 appears to be yyyy-MM-dd hh:mm:ss. The date format can be confirmed by executing cat /proc/acpi/alarm and looking at the alarm time that is already set in the BIOS.
  • The SD32G2 doesn’t appear to support setting the day of month for the wakeup alarm. This should be fine for TV recording with my configuration since the machine will simply boot up on the specified time on the next day and then shut down 2 minutes later if it doesn’t start recording.
  • If the day is set to 0 (which always seems to appear), then than means start up at the specified time on every day.
  • The wakeup alarm can be set using echo "2007-06-11 16:23:00" > /proc/acpi/alarm if run as root or inside a script executed using sudo
  • The wakeup alarm can be set using sudo sh -c "echo '2007-06-11 16:23:00' > /proc/acpi/alarm" if using sudo. Trying to simply do a direct sudo echo always gave me a permission problem.
  • Content located at ACPI Wakeup is quite useful. The Shuttle BIOS did not need any modifications to the hwclock.sh script though and don’t bother with the clumsy script to alter the time from local time to UTC. Just alter the BIOS clock to use the local time.
  • The BIOS time can be changed from UTC to local time in Kubuntu by opening the /etc/default/rcS file and changing UTC=yes to UTC=no. Much easier all round.
  • The BIOS clock can be set from the system clock using sudo hwclock -systohc. I’m reasonably sure the Kubuntu does this on shutdown.

MythTV server now starts up and shuts down automatically

Friday, June 8th, 2007

I’ve spent the last few nights playing with the wonderful world of Linux Advanced Configuration and Power Interface (ACPI) so that the MythTV server will now automatically start up to record a scheduled TV show, and automatically shut down afterwards if there are no logged in users.

This is pretty cool because it means I won’t miss recording any more of the Americas Cup Highlights because I didn’t get out of bed early enough to turn on the server.

It was a pretty arduous task with a number of steps based on content from a number of different web-sites to get things to work. After all the research though it’s not actually that complicated to setup. At a high level the following was done (subsequent blog posts will cover off some of the more tricky bits in more detail):

  1. Configure Power Management in the BIOS of your machine to allow being resumed by alarm
  2. Install appropriate bits for Linux to support ACPI
  3. Alter the Linux configuration so the BIOS uses local time rather than UTC time (since MythTV works using local time)
  4. Ensure that the ACPI alarm interface works as expected when set
  5. Create a small script to set the ACPI alarm given a date
  6. Create a small script to check to see if it is OK to shutdown the server
  7. Configure MythTV to use the two scripts created above
  8. Alter the sudo permissions for the MythTV user so the scripts can be executed without permission problems
  9. Alter the login permissions so only root can shut the server down (to prevent accidental shutting down when the server is recording