October 2012 – Manwe

I was recently tasked with writing a Hadoop map/reduce job. This job had the requirement of taking a list of regular expressions and scouring hundreds of gigs worth of log files for matches. Since I’ve been leaning more and more towards Scala I wanted to use it for my job […]

Simple Scala Map/Reduce Job

27 Oct, 2012 in hadoop / scala tagged hadoop / java / maven / scala / unit test by

Here is a handy script I’ve been using for MongoDB to retrieve a list of all the fields used in a collection. This uses a map/reduce routine and has to comb over all the documents in a collection so you may want to exercise caution when using this script. // […]

Select distinct for MongoDB

23 Oct, 2012 in javascript / mongodb tagged administration / javascript / mongodb / scripting by

Here is a simple web scraping script I wrote for PhantomJS, the immensely useful headless browser, to load a page, inject jQuery into it, and then scrape the page using a user-supplied jQuery selector. page = require('webpage').create() system = require 'system' phantom.injectJs "static/js/underscore-min.js" page.onConsoleMessage = (msg) -> if not msg.match […]

Simple PhantomJS web scraping script

22 Oct, 2012 in bash / coffeescript / it industry tagged javascript coffeescript data mining by