I was recently tasked with writing a Hadoop map/reduce job. This job had the requirement of taking a list of regular expressions and scouring hundreds of gigs worth of log files for matches. Since I’ve been leaning more and more towards Scala I wanted to use it for my job but I also wanted to use Maven for my job’s package management to make the job easy to setup and extend. And finally, I wanted to have unit tests for my mapper and reducer and an overall job unit test. The result is this project I posted to GitHub as a template for future projects. I hope it proves as helpful for others as I’m sure it’ll be for me.
Simple Scala Map/Reduce Job
Oct 27
Select distinct for MongoDB
Oct 23
Here is a handy script I’ve been using for MongoDB to retrieve a list of all the fields used in a collection. This uses a map/reduce routine and has to comb over all the documents in a collection so you may want to exercise caution when using this script.
// usage:
// mongo localhost/foo --quiet --eval="var collection='bar';" getcollectionkeys.js
var mr = db.runCommand({
"mapreduce":collection,
"map":function() {
for (var key in this) { emit(key, null); }
},
"reduce":function(key, stuff) { return null; },
"out":collection + "_keys"
})
print(db[mr.result].distinct("_id"))
db[collection+"_keys"].drop()
Here is a simple web scraping script I wrote for PhantomJS, the immensely useful headless browser, to load a page, inject jQuery into it, and then scrape the page using a user-supplied jQuery selector.
page = require('webpage').create()
system = require 'system'
phantom.injectJs "static/js/underscore-min.js"
page.onConsoleMessage = (msg) ->
if not msg.match /^Unsafe/
console.log msg
scrapeEl = (elselector) ->
rows = $ elselector
for el in rows
if el.innerHTML
str = el.innerHTML.trim()
if str.length > 0
console.log str
page.open system.args[1], (status) ->
if status isnt 'success'
phantom.exit 1
else
page.injectJs "static/js/underscore-min.js"
page.injectJs "static/js/utils.js"
page.injectJs "static/js/jquery-1.8.2.min.js"
page.evaluate scrapeEl, system.args[2]
phantom.exit()
Run it with:
phantomjs scrape_element.coffee "http://www.moviefone.com/coming-soon" ".movieTitle span"
Here is a presentation I gave at MongoSF 2012 on unique security considerations for MongoDB.
And here are my slides.
How hackers see bugs
Apr 30
[Guest post by Ryan Bailey]
Earlier this year roughly 50,000 stolen iTunes accounts were posted to a Chinese online auction site with prices ranging from 15 cents to $30 each. Many forms of attacks can be leveraged in acquiring passwords such as these through covert means, but almost none provide such a straightforward plan of attack like Phishing. Phishing, like many other forms of modern day email spam, is a form of social engineering aimed at acquiring sensitive information by attempting to fool users into freely surrendering passwords, credit card information or other potentially valuable information. Most current day attacks come in the form of an email seeking users to verify their account or billing details. These social engineering attempts often utilize pixel perfect facsimiles of websites or newsletters in order to gain a user’s trust. That’s where this phishing proof of concept gets its cue.
The Pitch
The first step of the attack comes in the form of utilizing a botnet to send out a fake news article using the default styles of a legitimate shared article. These emails often use minimal styling and shortened URLs making a fake shared story almost impossible to distinguish from a real one.
The Hook
The user is then taken to a fake mobile version of the news article where the user is able to browse the page as they normally would. The only clue to the fake at this point is the URL in the address bar which can easily be spoofed with either a shortened URL or a misspelled domain name.
The Bait
After a certain amount of time, an iOS alert pops up to inform the user that the iTunes Terms of Service has changed. Normally users are only prompted to accept changes after attempting a download from the iTunes store. But as anyone who has owned an iPhone will attest to, these changes often stand in the way of desired actions and often are accepted without a second thought. By delaying the bait until after the user has begun to read the article, it stands to reason the users will simply accept the thought of accepting the new Terms of Service without second thought.
The Switch
The users, after having clicked through to accept the changes, are taken to a facsimile iTunes Store Terms of Service page where users are given bogus information about the update they are about to agree to. This view of removing the standard Safari title bar is easily accomplished through several frameworks designed to perfectly mimic the iOS user interface.
The Payoff
The user scroll to the bottom of the fake Terms of Service to the realistic “Accept” button where upon clicking pops up a copy of the iOS Username and Password Prompt. After entering their username and tapping the “OK” button, the user can be forwarded back to the article being none the wiser to their personal information now being in the hands of the highest bidder.
Conclusion
This exploit brings up the interesting thought of educating users to how interfaces conduct their primary routines. Potentially, the only way the users could be warned of the fact that this is a scam would be through the fact that iTunes never asks for users to accept a Terms of Service update through a Safari window. This proof of concept was done in only a few hours but could easily be made almost undetectable through the use of an app slipping through the approval process at Apple or just even more advanced and malicious Javascript and CSS. While this was simply a harmless proof of concept, this could easily be put into the wild and start showing up in spam messages almost overnight.
node.js at Facebook
Jan 23
Recently I found the need to create an init.d script and since I had a hard time finding an example elsewhere1, here’s the overly simple script I came up with to get the job done:
#!/bin/bash
# myapp daemon
# chkconfig: 345 20 80
# description: myapp daemon
# processname: myapp
DAEMON_PATH="/home/wes/Development/projects/myapp"
DAEMON=myapp
DAEMONOPTS="-my opts"
NAME=myapp
DESC="My daemon description"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
case "$1" in
start)
printf "%-50s" "Starting $NAME..."
cd $DAEMON_PATH
PID=`$DAEMON $DAEMONOPTS > /dev/null 2>&1 & echo $!`
#echo "Saving PID" $PID " to " $PIDFILE
if [ -z $PID ]; then
printf "%s\n" "Fail"
else
echo $PID > $PIDFILE
printf "%s\n" "Ok"
fi
;;
status)
printf "%-50s" "Checking $NAME..."
if [ -f $PIDFILE ]; then
PID=`cat $PIDFILE`
if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
printf "%s\n" "Process dead but pidfile exists"
else
echo "Running"
fi
else
printf "%s\n" "Service not running"
fi
;;
stop)
printf "%-50s" "Stopping $NAME"
PID=`cat $PIDFILE`
cd $DAEMON_PATH
if [ -f $PIDFILE ]; then
kill -HUP $PID
printf "%s\n" "Ok"
rm -f $PIDFILE
else
printf "%s\n" "pidfile not found"
fi
;;
restart)
$0 stop
$0 start
;;
*)
echo "Usage: $0 {status|start|stop|restart}"
exit 1
esac
This script will work in /etc/init.d on Xubuntu 11.10 (so most Debian-based systems) and CentOS 5.5 and you can control it via chkconfig.
- That said, if you know of such an example I’d love to hear from you. [↩]
