Here is the video of the initial announcement along with some juicy technical details:

If you are interested in seeing how your PHP project fares when run through HipHop, check out the official HipHop project page.

With tools like HipHop, scripting languages like PHP are no longer subject to the charges of gross computational inefficiency. In short, it’s a great day to be a web developer.

Query SVN log history and filter by username

svn log | sed -n '/username/,/-----$/ p'

Run the last command as root

sudo !!

Save a file you edited in vim without the needed permissions

:w !sudo tee %

Why is this command so awesome? Peter described it quite well:

This happens to me way too often. I open a system config file in vim and edit it just to find out that I don’t have permissions to save it. This one-liner saves the day. Instead of writing the while to a temporary file :w /tmp/foobar and then moving the temporary file to the right destination mv /tmp/foobar /etc/service.conf, you now just type the one-liner above in vim and it will save the file.

Change to the previous working directory

cd -

Run the previous shell command but replace string “foo” with “bar”

^foo^bar^

Find the last command that begins with “whatever,” but avoid running it

!whatever:p

Copy your public-key to remote-machine for public-key authentication

ssh-copy-id remote-machine

Capture video of a linux desktop

ffmpeg -f x11grab -s wxga -r 25 -i :0.0 -sameq /tmp/out.mpg

Whatever storage solution you decide to use it needs to be unobtrusive and largely automated because, if not, then you’ll find out at the worst possible time (usually in a crisis) that actually recovering your data is nearly impossible and often times, incomplete.

The most unobtrusive solution I’ve found so far is to use a Network Attached Storage solution. The one I use and highly recommend is the D-Link DNS-321 which accepts standard SATA drives (which means they are fast and reliable) in a RAID-1 configuration. RAID-1 means the drives are mirrored, meaning the data is automatically duplicated to two internal drives. Just about any NAS system will work but make sure it includes RAID (most don’t) and isn’t simply a more fancy external hard drive.

Being attached a network attached device also gives you the benefit of not having to rely on too many additional moving parts. For a long time I used to use spare computer systems as storage units but what I quickly found out is that the individual parts in them posed as multiple unnecessary points of failure. Motherboards, RAM, even graphics cards can cause significant headaches when all you care about is the hard drives and the data they contain.

In fact, since Google’s high powered cloud computing infrastructure runs on common hardware like the kind you and I use, it is significant to note the hardware failure rate they discovered from constantly pushing common hardware to it’s limits over long periods of time. This simply means that when you are planning a computational strategy (in this case, storage of sensitive data) you need to plan for failure instead of hoping for the best.

In contrast, having a system that only consists of a minimal operating system and two drives should be able to give you enough time to replace one drive if/when the other one fails and the NAS unit itself is cheap enough that you could easily have a spare mothballed for the rainy day when you’ll need it.

It’s also a good idea to keep a copy of your data in an offsite location. The principle being that if one place storing your data were flattened then the you should be able to recover from the offsite location. The best way to achieve this is through a continuous online storage solution. I personally don’t use an online storage solution but some things to look for in one would be the backing company’s reliability, whether they back your data up to a cloud or a single server, and how well put together their interface software is. Try the free services first, chances are that if they are really as good as they claim to be (and they all claim to be good) you’ll quickly find out during the trial period (which often is a certain amount of allowed data storage). Here are a few free ones, I have used box.net before (for random file storage, not for regular automated backups) and can say it is pretty good.

I’ve also adopted the strategy of using as many online solutions (such as Gmail for email) which allow me to leverage reliable 3rd party clouds which provide inherent protection from data loss and provide the added benefit of allowing me to access my data from a wide variety of computers without having to sync data between every system I want to use.

Finally, focus on only backing up the files you know you will need. There is no reason to back up the entire computer in terms of applications, operating system, etc. Backing up unnecessary data will only serve to max out your storage capacity and quickly overtax your backup solution. Instead, plan on replacing your whole PC (and the operating system it uses, but keep a copy of the applications you use) in the event of catastrophic data loss. If you stick with reasonably reliable hardware your failure rate should be much higher than Google’s (3-4 years). Average costs of new and decent systems are low enough now that treating a computer as a disposable device (like a cell phone) isn’t all too uncommon or that bad of an idea.

Microsoft initially tried to claim that this exploit was trivial but has since issued an out-of-cycle emergency patch for all versions of Internet Explorer.

Looks like now is the perfect time to switch to one of the more superior browsers like Chrome or Firefox.

Here’s a video detailing how this hack works in action in case you are like me and interested in the juicy technical details:

• The shortness and simplicity of passwords means many users select credentials that will make them susceptible to basic forms of cyber attacks known as “brute force attacks.”
• Nearly 50% of users used names, slang words, dictionary words or trivial passwords (consecutive digits, adjacent keyboard keys, and so on). The most common password is “123456”.
• Recommendations for users and administrators for choosing strong passwords.

Also, here are the top 10 most commonly used passwords they found:

1. 123456
2. 12345
3. 123456789
4. Password
5. iloveyou
6. princess
7. rockyou
8. 1234567
9. 12345678
10. abc123

I’ve said it before, the first step in computer security is having a strong password policy.

Many of you are wondering what the big deal is with blogs. Well here is a short video on blogs and why they are important/useful:

What’s so great about blogs?

Aside from being able to access specialized information put out on a regular basis, there is one other reason I enjoy reading blogs and consider them to be an essential element in our modern forms of communication.

Blogs help you connect with people.

You learn a lot about someone’s character, thoughts, and passions if you follow what they say on their blog. The trouble is that since blogs are generally authored by one person on individual website it can become time consuming and cumbersome to visit each blog you’re interested in to check for and read any new posts.

How can I keep up with blogs?

The easiest tool I’ve found to help bring a variety of different blogs together into one place is by utilizing the RSS feed offered by most blogs.

Google Reader is a web-based RSS reader which requires a Google account and a little bit of setup, but once you get it going its pretty much automated and will allow you to check a number of blogs without having to spend time visiting each and every website to get updates.

Here is a short video to help you get started with Google Reader:

Working in the information technology sector, one of the most common questions I get asked by parents is about monitoring internet access of their children.¹

Most parents want to know what their children are doing online but also recognize that most off-the-shelf products are just as easy to disable or circumvent (or are far more restrictive/bloated than they want) as they are to install or operate. And sadly, enterprise solutions that capture and control network traffic at the most basic level (making circumvention next to impossible) is still very expensive and therefore out of reach for the average family.

What I needed was a cheap and hackable router that I could modify to send captured URLs to a central source for storage and processing.

The What: WRT54G

After studying my options I remembered reading a lot about the Linksys WRT54G-series routers and how they were originally based on a heavily modified version of Linux and how Linksys made headlines when it lost a court case regarding the GPLed code it used in their router’s firmware.

So I did a little digging.

What I found was a whole router-hacking subculture built around the WRT54G. While it seems that much of the initial fervor has subsided, many of the packages show a last update time of 2007 or so, the documentation is still valid for the most part. The most popular projects which provide custom firmware are the OpenWRT and DD-WRT. While OpenWRT is the original, I found DD-WRT to be a lot more polished and (as we’ll see later) configurable without much headache.

It’s important to note here that the WRT54G has many variants and its easy to fall into the trap of thinking that any old WRT54G will do but a little diligence and study of the differences between the hardware revisions will certainly save you time and money.

After buying a few different routers and bricking one (a Buffalo AirStation WHR-HP-54G² ) and a false start with a newer WRT54G v7 (anyone need a highly configurable, albeit not-very-hackable router?) I discovered that the best router for hacking is the WRT54GL (which was designed by Linksys to allow for user modifications).

The How: URLSnarf and custom shell scripts

Space on a router is very limited. On the WRT54GL model I eventually ended up using I had 4Megs of space to work with.

The first order of business was to find a package that could monitor all of the network connections (wired and wireless) on the router and capture requested URLs. For this task I discovered  that URLSnarf, part of the dsniff OpenWrt package, worked quite well.

To install packages I used DD-WRT’s firmware modification kit which allowed me to simply add the scripts and packages I wanted without having to recompile everything.

Next I needed to transform the captured URL into a URLencoded string in order to send it to my monitoring service via a simple wget request. Initially I tried using several variations of user-generated Python and PHP packages but they both took up far more space than I could afford so, instead, I searched for a pure command-line based solution.

After some more digging I found a handy sed substitution script that worked like a charm. The script worked in two parts, the first one being the substitution script (/usr/bin/urlencode.sed):

s/%/%25/g
s/ /%20/g
s/ /%09/g
s/!/%21/g
s/"/%22/g
s/#/%23/g
s/\$/%24/g
s/\&/%26/g
s/'\''/%27/g
s/(/%28/g
s/)/%29/g
s/\*/%2a/g
s/+/%2b/g
s/,/%2c/g
s/-/%2d/g
s/\./%2e/g
s/\//%2f/g
s/:/%3a/g
s/;/%3b/g
s//%3e/g
s/?/%3f/g
s/@/%40/g
s/\[/%5b/g
s/\\/%5c/g
s/\]/%5d/g
s/\^/%5e/g
s/_/%5f/g
s/`/%60/g
s/{/%7b/g
s/|/%7c/g
s/}/%7d/g
s/~/%7e/g
s/	/%09/g

and the command line to use it:

sed -f urlencode.sed

To tie it all together, we can pass captured URLs to it via pipes from the command-line with:

urlsnarf | sed -f urlencode.sed

At this point, the only missing link of the capture chain is a script to continually read from the command line and send the urlencoded capture data to our storage application (described in the next part). For this task I used the following script (/usr/bin/urlmon.sh):

HOSTNAME=`hostname`
while read url; do
    DATE=`date +%s`
    echo $(wget -q -O- "http://myapp.appspot.com/log?l=$url&h=$HOSTNAME&t=$DATE")
done

exit 0

Finally, we need to have the router start listening for URLs as soon as it is booted. In a Linux environment this is generally done by init scripts. Since our router has limited capabilities, we don’t need to write a full init script. Here is the slimmed down init script I used (/etc/init.d/S50urlmon):

#!/bin/sh

/usr/sbin/urlsnarf -v "/(192.168.1.1|https\://myapp\.appspot\.com)/" | sed -f /usr/bin/urlencode.sed | /usr/bin/urlmon.sh

The Where: Google App Engine

I’ve been itching to try out Google’s App Engine for a while now and this project seemed to be a great fit since I didn’t know how much data to expect and I needed my receiving/processing/display application to be highly available and scalable. Especially if this works well enough that others might want to use it.

Since my initial phase is to merely capture the URLs requested from devices behind the router, and since the capture process should be as efficient and lean as possible (I don’t want the router to take very long logging a URL when it’s primary job is to retrieve that URL for the initial requester) I decided to make a simple Java servlet which simply takes the URLencoded log line generated by URLSnarf.

Google App Engine uses Java Data Objects enhanced by DataNucleus to store data in Google’s massive cluster. Here is the annotated JDO (LogLine.java) I used to store the captured URL:

import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.IdentityType;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;

@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class LogLine {
	@PrimaryKey
	@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
	private Long id;

	@Persistent
	private String host;

	@Persistent
	private Long time;

	@Persistent
	private String line;

	public void setId(Long id) {
		this.id = id;
	}

	public Long getId() {
		return id;
	}

	public void setLine(String line) {
		this.line = line;
	}

	public String getLine() {
		return line;
	}

	public String getHost() {
		return host;
	}

	public void setHost(String host) {
		this.host = host;
	}

	public Long getTime() {
		return time;
	}

	public void setTime(Long time) {
		this.time = time;
	}
}

And here is the servlet that processes the GET request (containing the captured URL in Apache Common Log format)

import java.io.IOException;
import java.net.URLDecoder;

import javax.jdo.PersistenceManager;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.werxltd.webmon.data.LogLine;

public class Log extends HttpServlet {
	private final static long serialVersionUID = 3;

	public void doGet(HttpServletRequest req, HttpServletResponse resp)
    	throws IOException {
			try {
				resp.setContentType("text/plain");

				LogLine logline = new LogLine();
				String logStr = URLDecoder.decode(req.getParameter("l"));
				logline.setLine(logStr);
				logline.setHost(req.getParameter("h"));
				logline.setTime(Long.parseLong(req.getParameter("t")));

				PersistenceManager pm = PMF.get().getPersistenceManager();
				pm.makePersistent(logline);
				pm.close();	

				resp.getWriter().println("OK");
			} catch (Exception e) {
				e.printStackTrace();
				resp.getWriter().println("FAIL");
			} finally {

			}

	}
}

The future

This project is still in it’s early stages. There is no real way to view the captured data just yet, though I plan on incorporating Polliwog, and the router software hasn’t been tested as much as I would like. I’m also leery of any security holes I may have introduced.

So if you have any suggestions or would like to know more, feel free to leave a comment below!

1. Most actually ask about “controlling what their kids see online” but I generally argue for a observe-only approach as it helps open lines of communication with your child whereas silently blocking “bad” sites will only start a silent war which will only frustrate you once they do find a suitable workaround, such as a proxy.
2. I might have had better luck had I seen this helpful guide. Oh well, this gives me a future project in figuring out how to de-brick my WHR-HP-54G

After some intense research I decided to go with “The Grinder” which allows multiple tests to be run by multiple machines which can all funnel their collected statistics back up to a central “console”. Tests are written in Python which, in turn, gets fed through Jython and converted into native Java bytecode to be run by participating Grinder agent instances. Grinder works on a single, user-definable port, for both pushing scripts to listening agents as well as gathering statistics from tests.

Initially I decided to try and capture the results of each individual test in a MySQL database but abandoned that idea when the tests ended up overloading the MySQL database server before the web app we were primarily testing. Logging results also proved to be an interesting feat since it swampped the agent’s filesystems after less than an hour (we were running multiple processes and threads) as well.

We eventually settled on simply capturing the combined statistical data at the root console level (the way Grinder is designed) and displaying it via a plugin in Hudson.

Overall Grinder worked great for testing the load of our web app (which passed with flying colors). And since Grinder works natively in Java we are also planning on testing specific Java classes directly in the future as well as their overall performance through a web based front-end such as a servlet.

At first I thought this was merely a fluke. That some random feminist organization managed to persuade the organizers of the event into allowing them to come and try and persuade us that we are somehow all intolerant and misogynistic simply because fewer¹ women are members. However, after seeing an article to the same effect on Slashdot recently I started looking into the so-called feminist open-source movement and found it to be neither new nor isolated.

Now while I have no problem with people pursuing their own agendas per se I do have a big problem when it comes to pursuing an agenda that is purely fabricated and detrimental to what it is purporting to help. Feminism, for example, has been a corrosive cancer to every single thing it has touched and I hold no illusions to what sort of effects it will have in the realm of science and technology. I also have a big problem when, in the name of tolerance and equality, feminism manages to wield a giant club beating down any dissenters in an effort to make everyone get in line. Get in line with what, though? The isolated incidents of sexism and questionable jokes/comments are not enough to convince me that the issues that need to be address fall primarily within the domain of gender-equality. As one attendee of the Atlanta Linux Fest so poignantly put it: “Aren’t these problems really human problems as opposed to problems that only women face?”

Social justice seems to be a pretty effective means of “turning the tables” by being just as biased as those you are decrying without fear of reprisal. After all, who’s going to dare speak out against the supposed victims? Interestingly enough, this whole movement has also found it’s way into the current US administration’s agenda.

When they said they were going to champion science and technology they failed to mention they were only interested in the social aspect, as if that is what is harming the competitiveness of the US tech industry.

1. Which is still more than none.

Both of these are interesting areas of computer science, they do sometimes overlap, and they are both going to change computing in general in significant ways as time rolls on, but they are not the same.

Here’s are the differences to help  you can tell them apart.

Diskless computing

Diskless computing is best demonstrated in the Linux Terminal Server Project (excellent project, I’ve use it before to deploy over 150 diskless workstations in a company before) and Microsoft’s pathetic rival, Windows Terminal Services. Sun has their own solution as well and there are countless 3rd party utilities, but the basic idea behind them all is that you have one big computer (or series of computers) that all these “headless” computers connect to in order to retrieve an operating system, store files, etc. For large networks this network model is absolutely amazing.

Cloud Computing

Cloud computing, however, is the concept that you have a large problem that requires a lot of computing power to solve. Rather than buy bigger and bigger hardware, what we’ve found out (going back to Cray supercomputers) is that it is far better to split the problem down into iterative chunks and push those through multiple processors all at once rather than try to get a single processor to process everything. This is called distributed computing.

You might have heard of one of the major platforms for this type of computing, Beowulf, from the popular internet meme “imagine a beowulf cluster of…” Another very popular distributed computing platform (popular because it is far easier to install, operate, and write code for than the Beowulf project) is Hadoop. Hadoop is a project inspired by Google’s implementation of the MapReduce design paradigm written in Java which makes it a lot more portable.

Projects using Cloud Computing

Parallel processing is done today in a wide variety of settings including:

• 3D rendering farms for companies such as Disney’s Pixar
• indexing the web with Google, Yahoo, Microsoft, etc.
• data mining of all sorts with companies like Wal-Mart, etc.

Join in!

There are some very popular projects using distributed computing technologies that regular people with CPU cycles to spare are encouraged to join in on like:

• SETI@home where you can help process data that might help us identify extraterrestrial signals
• Folding@home where you can help search for cures to various diseases
• Genome@home where you can help map the human genome (again), this is tied closely to the folding@home project above
• Shrek@home which was a pioneer project that a few of us got to participate in
• others, including fightaids@home to help fight AIDS and lhc@home to process the massive amounts of data coming from the CERN’s Large Hadron Collider

So while diskless computing and cloud computing can have some areas of overlap (I configured the LTSP network I mentioned earlier to assist with the genome@home project when the systems were idle) they aren’t necessarily tied together.