Ian Lewis' Blog

I just learned about the Linux Hater's Blog via Miguel De Icaza's blog. Super funny and super true. Users need software that works. I guess that's why the only companies run by programmers make software for programmers.

Recently I've been doing a decent amount of debugging a database conversion process and looking at log files on the Red Hat servers at work. This has meant looking at some rather big (10 or so megabytes) log files. Normally I just fire up vim when looking at text files but opening a text file in a text editor that is a number of megabytes is a no-no since pretty much any text editor will load the whole file.

Text viewers like more and less, however, however can skip this little bit since you aren't going to be changing an arbitrary part of the file. So you can skip through the file with relative ease. While I know my way around a Linux system, I am unfortunately woefully lacking in knowledge (sed-fu, grep-fu, find-fu or whatever) more than *very* basic usage of rather common *nix tools.

Anyway, I picked up a little bit about how to use less and found that I used a few commands often. Less normally opens a file starting at the beginning of the file but normally you want to look towards the end of logs and tail is not terribly useful for looking backwards through a log file.

less +G <file>

will open the file from the end and allow you to scroll backwards through it. In this context, like vi, using a slash '/' followed by a string can help you search forward through a file but going backward, I learned, is achieved with the question mark '?' character.

Another command that is pretty useful is grep, and when searching log files you might want to find a particular part of the file, like an error message but also some text before or after the error. You can do this with the -A and -B commands, meaning 'After' and 'Before' respectively.

grep -A 5 -B 10 error <file>

will give print the lines matching the string 'error' in the given file, but also give 5 lines of context after and 10 lines of context before the matched line. This is pretty useful when searching log files but can get confusing sorting out what is context and what is a match when it matches many lines.

Many people who are new to Debian might be thinking that Debian stable releases are slow. You are right. Many do look at this and turn to Ubuntu because of their relatively quick releases. It's true that Ubuntu does release "stable" versions more often but I would encourage people to sit back down and give Debian another try. Especially with the cool feature of Debian's packaging system, apt, called apt pinning.

Apt pinning allows you to get the best of both worlds. You get the stability of stable packages from stable, with the new software of testing and unstable as needed. Of course you will need to be a bit careful because some packages from testing or unstable can include packages that are unstable or otherwise broken. Installing packages from multiple distributions makes your system inherently more unstable since that kind of setup isn't tested as well as a normal setup.

Apt pinning requires that you modify two files. These files are /etc/apt/sources.list and /etc/apt/preferences. These files control where you get packages from and the priority of each repository respectively. First in your sources.list file you will want to add references to the testing and unstable repositories. Those references will look pretty similar to your existing references to the stable repository but will simply have references to testing and unstable rather than stable. Your sources.list might look like the following.

#Stable
deb http://ftp.us.debian.org/debian stable main non-free contrib
deb http://non-us.debian.org/debian-non-US stable/non-US main contrib non-free

#Testing
deb http://ftp.us.debian.org/debian testing main non-free contrib
deb http://non-us.debian.org/debian-non-US testing/non-US main contrib non-free

#Unstable
deb http://ftp.us.debian.org/debian unstable main non-free contrib
deb http://non-us.debian.org/debian-non-US unstable/non-US main contrib non-free 

Mine looks like the following because I live in Japan. I also run testing as my base system and include packages from unstable when needed. I also include the references to the package source repositories.

#Testing
deb ftp://ftp.jp.debian.org/debian/ lenny main
deb-src ftp://ftp.jp.debian.org/debian/ lenny main

deb http://security.debian.org/ lenny/updates main contrib
deb-src http://security.debian.org/ lenny/updates main contrib

deb http://http.us.debian.org/debian/ lenny main

deb ftp://ftp.debian.or.jp/debian/ lenny main contrib non-free


#Unstable
deb ftp://ftp.jp.debian.org/debian/ sid main
deb-src ftp://ftp.jp.debian.org/debian/ sid main

deb http://http.us.debian.org/debian/ sid main

deb ftp://ftp.debian.or.jp/debian/ sid main contrib non-free

Once you have that set up you will need to create your preferences file to set up the priority for getting packages. We are going to set a priority of stable -> testing -> unstable which means that if a package is found is more than one repository it will take the one from stable first, then the one from testing if the package doesn't exist in stable, then from unstable if the package doesn't exist in either stable or testing. This will have the effect of giving us the most stable package while allowing us to install newer packages.

Package: *
Pin: release a=stable
Pin-Priority: 700

Package: *
Pin: release a=testing
Pin-Priority: 650

Package: *
Pin: release a=unstable
Pin-Priority: 600

Once that is set up you can open up synaptic. One thing that is nice about synaptic is that it can support pinning. So the packages list in synaptic will show us the list of packages as we can install based on the priority we set up. According to this page one of the problems you might run into is this error:

E: Dynamic MMap ran out of room E: Error occured while processing sqlrelay-sqlite (NewPackage) E: Problem with MergeList /var/lib/apt/lists/ftp.us.debian.org_debian_dists_woody_contrib_binary-i386_Packages E: The package lists or status file could not be parsed or opened.

This is caused because apt's cache is too small to handle all of the packages that are included with stable, testing, and unstable. This is also very easy to fix. Add the following line to /etc/apt/apt.conf

APT::Cache-Limit "8388608";

Now that that is taken care of, lets say you have a particular package you want to install the newest version of, but the newest version of that packages isn't in stable yet. Synaptic will show you the version that's in stable because we set that as the highest priority. But you know there is a newer version out there. You can check packages.debian.org to see if the package you want is included in either testing or unstable but the quicker way is to check in synaptic itself.

If there is a package that has a newer version in testing or unstable that you would like to install then you can do so by opening the properties for the package. Then click on the versions tab. There it will show you what versions you can install from different repositories. You can then select which version to install by choosing Package -> Force version... or Package -> Versions... The name of the option might be slightly different as it changes a bit in different versions of synaptic.

I wanted to have a simple incremental backup system I could use on my machine to backup to an external drive so I came across this post on Benno's blog. Basically it involves using rsync with the --link-dest option to compare files you are backing up against an previous backup and only create new copies when the files have been modified since the previous backup. The rsync command would look something like the following:

rsync -a --link-dest=backup.0/ /dir.to.backup/ backups.1/. 

If the files have not been modified then it creates a hard link to the files so it doesn't have to create a copy in order for the backup folder to have references to all the files in the backup. Without something like that, restoring is hard because in order to get all the files during a restore because the latest backup would only contain files that have been updated.

The problem is that if you have an external drive that you want to be able to use with windows you need to use the vfat file system which doesn't support hard or soft links. I have looked around for ideas about how do get around this problem and it might just be that I need to split the drive into two partitions, vfat and ext2/3, and and use the ext part for backups. There at least seems to be some ext2/3 support for windows which seems promising but using it would mean that not every windows machine would be able to read it, just the ones with support installed. Also, the software does not support permissions or soft links in any way so using it might be difficult in some situations.

Both ways have downsides but I don't use windows much at all so I'm leaning towards just making a ext3 partition. Maybe I'll do that this weekend.

I have been reading the OLPC News blog for a while now and frankly I'm kind of tired of hearing about how American geeks are complaining that it's hard to get an XO or that it has taken a while for them to get their XO laptop or that the box was empty when it arrived. Here's a clue for those who are wondering why OLPCs service is sub-par. The XO laptop is was not made for American gadget addicts.

Nothing irks me more than seeing staged photos of playful American geeks fighting over their new gadget that they have no business having in the first place. And then at the same time they complain that OLPC's service is not good. Boo hoo. My shiny toy is broken. I can't believe the silliness. The last person who needs an OLPC laptop is an American computer geek and yet they complain about OLPC and having to wait a long time on the phone to get service. Think about it. OLPC is not DELL. Grow a brain.

If you will be doing OSS development (seriously, not just pretend) on the XO that will be used in developing countries, fine. But most will not. This guy even says that it's a toy and he doesn't know how he'll even use it. Most recieving an XO laptop would say that they are OLPC enthusiasts and that they think that OLPC is a great idea and that it would help the developing world a lot. I agree. But I am not itching to get an OLPC laptop. I will not do any development on one and I'm not a child in a developing country so I have no business having one. OLPC may have instigated the Give 1 Get 1 program themselves but think about it, Nicholas Negroponte said that having an XO if you aren't a child should be a badge of shame. I think that's true, whether you stole it or got it from the Give 1 Get 1 program.

Those recieving an XO laptop might say they are donating to OLPC out of the goodness of their hearts and recieving an XO is just icing on the cake. BULLSHIT. If they cared about children in the developing world they would donate the the laptop they were going to recieve too. Forget the Give 1 Get 1. How about the Give 2 program? American geeks would drop OLPC like every other passing fad that crosses their impatient minds. THEY ONLY CARE ABOUT OLPC BECAUSE THE LAPTOP IS COOL. Period. No other reason.

Is it just me or has the behavior of cp in linux distributions changed recently? cp is non-interactive by default so a lot of people, myself included, set an alias to include the -i flag so that cp was interactive by default.

alias cp=cp -i

But I used to enjoy the fact that if I set this alias it would prompt me when overwriting files but if there was a situation where I wanted it to be non-interactive I could do that by specifying -f. Basically, the last -i or -f would win. I want interative by default but the ability to specify non-interactive at my discretion.

However, recently or so, I noticed that several linux distributions include a cp that if you specify a cp -i alias you cannot specify -f to use non-interactive mode. The -f is ignored or at least doesn't cancel out the -i. Perhaps there is another way do to what I want but I'm unaware of it at the moment. Currently I have to remove the alias, do the non-interactive copy, and then reinstate the alias. Super annoying.

Update: You can bypass an alias by putting quotes around the command like so,

"cp" source destination

I just read this post over at the One Laptop Per Child News site on the user interface that they use with the OLPC called Sugar. It's a GUI designed to work on the OLPC X0-1, which is what they call the laptop, and also be user friendly for 3rd world children. No crazy complicated explanations about why you don't have enough memory or swap space or whatever, just a nice little circular bar that shows what applications are using your memory and how much. You can also load and close applications "like toys" by pulling them out of the toybox and putting them back in from the play area, so to speak.

Update: By the way, this is not the real UI for the OLPC. It's just Sugar. The video is "translated" into a Mac UI using a theme (or it's actually running on a Mac) for those who don't like to look at the real UI for the OLPC. Basically I think the author wanted to show off the ideas behind the UI without having to deal with comments about the UI being ugly and plain. If you want to see what people are talking about when they say they don't like the interface for the OLPC check out this video.