So, what have you been upto? at brentp.net

So, as you can no doubt tell, this is the first update for a while. I’ve pretty much been sitting here coding stuff for a couple of months, so here are my most recent creations:

The first one you will be able to view. It is a pastebin clone hosted on google app engine (my first dive into python, besides this facebook app that i made – also hosted on google app engine). I chose GAE because it’s a relatively new entrant into the whole “cloud” hosting game, and because its insanely scalable (although i doubt i’d need to worry about that). The pastebin supports highlighting in tonnes of languages (way more than paste2.org), and uses FBJS and jQuery stuff for its frontend ajax. It also supports submission via command line applications that users create (because it returns its output in very standard JSON). If you want to check it out, it’s located at http://pastebox.appspot.com/. Oh, and for its backend, it uses the django templating system (very, very handy – it’s a shame that the django Model’s system doesnt work on GAE otherwise you’d be able to use the admin interface).

Second item i’ve been working on is a facebook application, as mentioned above. It’s not public yet and i dont ever expect it to be because it was a practice in python, and it doesnt really utilise django’s template system as much as it could have.

Third item, and the most recent (and fun) is a PHP RRDtool grapher. You might think, “yes, but there are plenty of those out there already, nothing new here.”, but you would be wrong. This actually renders the graphs on the CLIENTS side, instead of getting RRDtool to generate an image every 10 minutes or whatever. This came about because the only linux box i’ve got sitting here is a WD Mybook World Edition II, and as anyone would tell you they have fairly limited specifications (32mb of ram, 64 if you’re game to screw around with the bootloader), and a really, really crappy cpu. Because of this, i used the Mybook in conjunction with MRTG/RRDTool to generate images of the traffic going through my dlink (*cough*) router.

There were obvious performance concerns from the get-go. To alleviate the strain on the mybook, when i didn’t look at the graphs, i setup a ramdisk on it so the hard drives wouldnt have to spin up, and i also got a script that generated images whenever the user requested them (so there were no images pre-stored). I tried a PHP version, and a Perl version with little success, both being as bad as each other (probably took around 5 seconds to render a page with 5 graphs on it).

To alleviate this, i thought it would be easier to render the images on the client side and just pass the client some javascript/json datapoints in an array which would tell the browser what to graph. The first concept of this worked excellently. I simply ran “rrdtool fetch blah.rrd….” whenever a user loaded the page, and then sent that in an array to the web browser. To graph, i used the new “canvas” feature of HTML5 (more specifically, the flot graphing library). Those of you that are in-the-know, might know what challenges HTML5 presents – primarily, lack of compatability. The web-browsers i tested the graphing app in were: Firefox 3.5, and Chrome 2.0, however, a friend using the latest safari (i assume) reported that it worked fine on Mac OSX. Pretty much any web browser that supports HTML5 can view the page, any that cant – tough luck. This also has some advantages as well, because we know the browser is recent, we don’t have to worry about supporting older browsers like IE6/7 (don’t know if 8 chokes on the graphing app, havent tested with it).

If you want to skip the next paragraph, screenshots are at the bottom of the page.

The next features I added to it were the ability to switch between different periods (daily, hourly, monthly, etc etc). That was relatively easy, and was accomplished in a few short hours using jQueryUI for the tabbing library. There was another advantage to using jQueryUI, it handled fetching the graph page transparently via ajax which meant less work for me. Next features i added were the ability to graph multiple RRD’s. I should note that every new feature added didn’t decrease the performance of the application overall by much, compared to the regular static ones.

Once the code for graphing multiple RRD’s was done, i rewrote the way the graph data was sent to the browser so i could enable/disable certain datasets on the fly (note: all of this is supported out of the box with the ‘flot’ graphing library. Truly the best i’ve seen). It wasn’t long until i decided to jQueryUI everything (so you change one stylesheet, or make your own using the jQueryUI ThemeRoller and the looks of the tab ui change completely).

Anyway, i’m running out of ideas of what to add next. One idea is to fetch data from the server every time a client zooms on a graph (at the moment, it only redraws that section – and due to performance constraints, i can’t send every datapoint to the user initially, otherwise for daily graphs you’d have 40 datapoints or so, depending on how often your MRTG cronjob runs).

If you are wondering the obvious: “When/where can I download this?” the answer at the moment is: “You can’t”. If you would like a copy, email me (brent.pickup [-] gmail [.] com) and I’ll usually get back to you within 24hrs. I’ll give it out on a case-by-case basis depending on your use for it (if it’s commercial a fee might be attached).

By the way, if you’re curious: yes, this can be adapted for things other than data going in/out of a router. These could be memusage, cpu usage, etc etc. However, i haven’t gotten around to testing that yet because i haven’t got a cpu usage script for MRTG.