Slides

Share/Save

Increasingly JSON is being used as a transport medium for data and with the rise of NoSQL solutions, having a selector language for JSON makes a lot of sense when dealing with JSON documents whose structure isn’t deterministic.

JSONSelect provides a good implementation of just such a JSON selector language but after working with it on a project I found myself needing to do more than it allowed me to do. Namely, I wanted 1. to be able to perform a selection and receive matching paths instead of the data contained in those paths and I wanted 2. to be able to modify data specified at a path location in-place.

jsonselect.match(sel, obj, asPath); // Added the asPath flag to return a path instead of the values
jsonselect.forEach(sel, obj, fun, asPath); // Added the same flag to forEach, I use this to
jsonselect.get(path,obj); // For getting the value using a path
jsonselect.set(path, value, obj); // For setting the value of a path
jsonselect.del(path,root); // For deleting a path

Here is my modified version of jsonselect in case anyone needs help solving the same problems I mentioned above.

Share/Save

Color = function() {
};

Color.hexdec = function(hex_string) {
    hex_string = (hex_string + '').replace(/[^a-f0-9]/gi, '');
    return parseInt(hex_string, 16);
}

Color.dechex = function(number) {
    if (number < 0) {
        number = 0xFFFFFFFF + number + 1;
    }
    return parseInt(number, 10).toString(16);
}

Color.pad = function(number, length) {
    var str = '' + number;
    while (str.length < length) {
        str = '0' + str;
    }
    return str;
}

Color.calcgrad = function(val, color1, color2) {

    if(!color1.match(/^#[0-9a-f]{6}/) || !color2.match(/^#[0-9a-f]{6}/)) return 'match err!';

    if (val > 1) {
        val = 1;
    }
    if (val < 0) {
        val = 0;
    }
    val = parseFloat(val);

    c1 = [Color.hexdec(color1.substr(1,2)), Color.hexdec(color1.substr(3,2)), Color.hexdec(color1.substr(5,2))]; //b
    c2 = [Color.hexdec(color2.substr(1,2)), Color.hexdec(color2.substr(3,2)), Color.hexdec(color2.substr(5,2))]; //r

    if (val < .5) {
        delta = [(c2[0] - c1[0]), (c2[1] - c1[1]), (c1[2] - c2[2])];
        arrColor = [c1[0] + * 2), c1[1] - * 2), c1[2] - * 2)];
    }
    return '#'+Color.pad(Color.dechex(arrColor[0]),2)+Color.pad(Color.dechex(arrColor[1]),2)+Color.pad(Color.dechex(arrColor[2]),2);
}

1. delta[0] * val) * 2), c1[1] + ((delta[1] * val) * 2), c1[2] - ((delta[2] * val) * 2)]; } else { delta = [(c1[0] - c2[0]), (c1[1] - c2[1]), (c1[2] - c2[2])]; arrColor = [c1[0] - ((delta[0] * (val - .5
2. delta[1] * (val - .5
3. delta[2] * (val - .5

Share/Save

Ext.define('Ext.ux.JSONP', {
    extend: 'Ext.data.ScriptTagProxy',
    alias: 'ux.jsonp',

    createRequestCallback: function(request, operation, callback, scope) {
        var me = this;

        return function(response) {
            callback.apply(scope, [response]);

            operation.setCompleted();
            operation.setSuccessful();
            me.afterRequest(request, true);
        };
    }
});

Share/Save

Ext.define('Ext.ux.MessageBroker', {
    extend: 'Ext.util.Observable',

    statics: {
        instance: null,
        setInstance: function(i) {
            this.instance = i;
        },
        sendMessage: function(msg, data) {
            this.fireEvent('message',{"msg":msg,"data":data});
        }
    },

    constructor: function(config){
        this.addEvents({
            "message" : true
        });

        if(config && config.listeners) this.listeners = config.listeners;

        Ext.ux.MessageBroker.superclass.constructor.call(this, config)
    }

}, function() {
    Ext.ux.MessageBroker.setInstance(new Ext.ux.MessageBroker());
});

// Sending a message:
Ext.ux.MessageBroker.sendMessage("load_stock", {"symbol":"INTC"});

// Recieving a message:
Ext.util.Observable.observe(Ext.ux.MessageBroker);
Ext.ux.MessageBroker.on('message', function(msg) {
    console.log('message fired: ',msg);
});

Share/Save

Share/Save

Color = function() {
};

Color.hexdec = function(hex_string) {
    hex_string = (hex_string + '').replace(/[^a-f0-9]/gi, '');
    return parseInt(hex_string, 16);
}

Color.dechex = function(number) {
    if (number < 0) {
        number = 0xFFFFFFFF + number + 1;
    }
    return parseInt(number, 10).toString(16);
}

Color.pad = function(number, length) {
    var str = '' + number;
    while (str.length < length) {
        str = '0' + str;
    }
    return str;
}

Color.calcgrad = function(val, color1, color2) {

    if(!color1.match(/^#[0-9a-f]{6}/) || !color2.match(/^#[0-9a-f]{6}/)) return 'match err!';

    if (val > 1) {
        val = 1;
    }
    if (val < 0) {
        val = 0;
    }
    val = parseFloat(val);

    c1 = [Color.hexdec(color1.substr(1,2)), Color.hexdec(color1.substr(3,2)), Color.hexdec(color1.substr(5,2))];
    c2 = [Color.hexdec(color2.substr(1,2)), Color.hexdec(color2.substr(3,2)), Color.hexdec(color2.substr(5,2))];

    if (val < .5) {
        delta = [(c2[0] - c1[0]), (c2[1] - c1[1]), (c1[2] - c2[2])];
        arrColor = [c1[0] + * 2), c1[1] - * 2), c1[2] - * 2)];
    }
    return '#'+Color.pad(Color.dechex(arrColor[0]),2)+Color.pad(Color.dechex(arrColor[1]),2)+Color.pad(Color.dechex(arrColor[2]),2);
}

1. delta[0] * val) * 2), c1[1] + ((delta[1] * val) * 2), c1[2] - ((delta[2] * val) * 2)]; } else { delta = [(c1[0] - c2[0]), (c1[1] - c2[1]), (c1[2] - c2[2])]; arrColor = [c1[0] - ((delta[0] * (val - .5
2. delta[1] * (val - .5
3. delta[2] * (val - .5

Share/Save

So here are the various technologies I used in the creation of my app.

YQL

To get the data for this project I used a simple Python script and YQL to get a list of locations in each Zip code in Georgia with the term “Church” in them. This yielded approximately 10,000 results, including a few from South Carolina.

MongoDB

I stored the data I got from YQL in MongoDB. Specifically, I used MongoLab’s to host the data because they have a generous free storage limit (256Meg) and can be accessed from a RackSpace server without incurring an additional bandwidth charge.

MongoDB is a JavaScript-based NoSQL solution used by sites like Foursquare. It has built-in support for Geo queries and is built to scale.

node.js

For the application layer I decided to try out node.js. Node is also JavaScript-based, based on WebKit, the engine behind Google’s Chrome and Apple’s Safari browsers. Node is event-based which means it has the potential to be lightning fast.

Canvas

The biggest factor in how well a heatmap solution performs is the graphics package that’s used. After searching around I found a pretty decent PHP heatmap solution, but it used GD and was very slow. I also found a JavaScript solution that used the new HTML5 Canvas element, but it choked when given a significant amount of datapoints to render all at once. So I decided to refactor some of the utility functions from the PHP solution and combine them with the Canvas function.

The great thing about the resulting solution is that it has the ability to run on either the client-side or the server-side. And in the end the heatmap application I built uses both. If the number of data points in a tile is less than a preset threshold defined per browser¹ the raw data is sent back to the browser which renders the tiles client-side rather than consuming precious server resources.

Web Sockets

The usual way to serve tiles is by serving images to overlay the map. And while the heatmap solution I developed does serve static PNG files from cache, I decided to use the new HTML5 Web Sockets to make things a bit more interesting. What is great about web sockets is that it allows me to pass events between the server and client very easily. Socket.io made it easy to forget where the server ended and where the client began.

ZeroMQ

As applications scale to multiple threads, processes, and eventually across servers and networks, they need to have a way for each component to communicate with other components in an efficient manner. So add a bit of future-proofing to my solution I decided to use ZeroMQ to pass messages between the front-end web server component and the back-end tile generator component. This allows me to tune the performance of the application in both directions, up or down².

Raphaël

To add some extra pizzazz to the app I decided to add in the ability to display each individual data point along with some additional detailed information. I found that Google’s native Marker system was a bit slow when it came to displaying over 2,000 markers at a time so I decided to give the Raphaël graphics library a try. The results were impressive. Raphael was not only able to draw thousands of data points on the map seamlessly, but was able to do it with smooth animations. Look for gRaphaël to be employed in future renditions of this heatmap solution.

Conclusion

Every now and then I run across a programming challenge that reminds me why I love doing what I do. These technologies and this project have done that for me. Being able to throw together a large, complex project like this in a relatively quick manner reminded me of Fred Brooks’s comment on why programming is fun.

1. 1000 for WebKit-based browsers, 250 for Mozilla-based browsers, and 0 for IE because IE still sucks.
2. Tuning performance down comes in handy when you are on a shared server with limited resources.

Share/Save

His biggest point is “do not optimize prematurely”. This is a sentiment I’ve heard a few times. Fred Brooks emphasizes it in Mythical Man Month, and Eric Raymond harps on it a bit in Art of Unix Programming.

Another point he makes is how using “weird language syntax” like double bitwise not is sometimes faster than their functional equivalents like parseInt.

And one surprising revelation (for me anyway) is that unrolled loops can sometimes be faster.

Overall, a great presentation, well worth your time if you work with javascript a lot.

Share/Save

To aid in the transition, ICANN devised a micro-language of sorts to allow a smooth transition between ASCII-only domain names and the more robust Unicode domain names (which allows for non-Latin characters). This micro-language is known as Punycode.

To give you an example of how this language works, let’s take the recent top level domain (TLD) approval of .iran in their native language of Farsi. Click here to see what this domain looks like in both it’s native Unicode (in Arabic script) and it’s punycode equivalent (xn--mgba3a4f16a).

There are a number of punycode decoders and encoders like this one. There are also excellent libraries available to convert punycode, like this javascript one.

Here is my own rendition using the javascript library above.

So if you have a need to use punycode and/or non-Latin domain names. I hope this helps.

Share/Save