Vysnu

Hpricot/LibXML Benchmark

Tuesday, April 29, 2008

I’m off to release an internal XML conversion script. I began using Hpricot, but after hearing good things about LibXML, decided to switch to that. Here’s a benchmark:


real	0m11.692s
user	0m11.497s
sys	0m0.188s

… for Hpricot versus…


real	0m6.441s
user	0m6.258s
sys	0m0.177s

… for LibXML

So that’s almost twice as fast. Nice to know.

Comment! | Permalink

Symfony 0.6.2 Sandbox Script

Thursday, April 10, 2008

This is for people who still use the old Symfony 0.6.2.

Get the script from here and execute it so:


sh ./symfony_sandbox.sh

This is largely plagiarized from the create quickstart script from the Symfony SVN but changed to make the pake install work.

Comment! | Permalink

kup.in: URL redirection off Google’s App Engine

Wednesday, April 9, 2008

First the URL: Kup.in is a fast URL redirection service run off Google’s new App Engine.

This was mostly a test project than anything else, just to gauge how easy App Engine is to start up and get going (very easy!). I’ll post in detail later, but here are bulleted observations:

I’ve been sitting on this project for ~6 months. The dilemma was that if I did this in PHP, I’d be slow coding it (and also ugly) but fast to deploy. If I did this in Python/Ruby, it’d be much easier, but deployment’s a bitch. With the App Engine, deployment is a simple appcfg.py update . which is uber cool.
You’ve got to have py25-openssl installed if you’re on Macports.
I couldn’t get Django templates to work (some import _md5 error which I suspect again is my Macports). I didn’t need templates here.
The code is opensource on Github. Fork it and do what you will! I’ll decide on a license after I wake up!

Speaking of which, it’s 3 in the morn here and I’ve got office tomorrow!

Comment! | Permalink

file_get_contents_with_timeout

Sunday, April 6, 2008

So here’s a piece of code which I think you guys will like. It’s a URL call with timeout function in PHP.


/* This works only for simple URIs
  Returns:

  array(headers, body): if everything's fine.
  TIMED OUT if it times out.
  UNABLE TO OPEN if we can't connect to host.
  */
private function file_get_contents_with_timeout($url, $read_timeout = 5, $connection_timeout = 5) {
  $url_parts = parse_url($url);

  $host = $url_parts['host'];
  $get = $url_parts['path'] . '?' . $url_parts['query'];

  $fp = fsockopen($host, 80, $errno, $errstr, $connection_timeout);
  if (!$fp) {
    return "UNABLE TO OPEN";
  } else {
    $out = "GET $get HTTP/1.1\r\n";
    $out .= "Host: $host\r\n";
    $out .= "Connection: Close\r\n\r\n";

    fwrite($fp, $out);
    stream_set_timeout($fp, $read_timeout);

    $result = stream_get_contents($fp);
    $divider = strpos($result, "\r\n\r\n");
    $headers = substr($result, 0, $divider);

    $body = substr($result, $divider, strlen($result));

    $info = stream_get_meta_data($fp);

    fclose($fp);

    if ($info['timed_out']) {
      return 'TIMED OUT';
    } else {
      return array('headers' => $headers, 'body' => $body);
    }
  }
}

Yup, a bit crude, but hey it works for me! (Oh, and it probably needs PHP5). Feel free to take the core logic out and wrap it up in exceptions and what have you.

Why did I have to write this contraption? Ah, the joys of the Indian SMS scene. Let me take you through a tour: Fastalerts is a bulk SMS solution for end users and resellers. The web frontend is written in PHP using Symfony (an older version, 0.6). The API is written in plain PHP and it connects to a SOAP-based (using nuSOAP) SMS sending solution.

This is a comment I have on top of the new backend code which sums up all the complexity:


/*
  A note on message sending.

  These actions have to happen as transactions:
  * Calling the GATEWAY.
  * Entering data into LOGS.
  * Reducing CREDITS.

  ONLY if:
  * User has "enough" credits.
  * Input is valid (numbers and mesage)
  * User has valid credentials

  AND we have to handle:
  * Gateway timeouts
  * Gateway errors
  * CDMA senderid correction.

  ALSO:
  If the user doesn't have enough credits to send the entirety of numbers,
  messages are sent until his credits are exhausted.

  AND:
  A separate status message is returned for each of these conditions.

  BUT WE ENSURE:
  That everything is logged appropriately:
  
  * All successfully sent messages are logged and credits reduced.
  * In every other case, credits are NOT reduced (we are customer friendly).
  * When gateway times out or errors out, the messages are logged.

*/

Interested people should note that there is a lot more you can add on to this: dynamic gateway switches (automatic failover), regular gateway tests, more backend support etc. but this is the bare minimum that’s needed for the backend to work. If you spend some time thinking about the problem, you’ll come to the realization that this bit:


 AND we have to handle:
  * Gateway timeouts
  * Gateway errors

means that we have to handle gateways that time out and never return a response. Most of the SMPP providers in India use something called NowSMS to manage their backend connectivity to the operator. In short: we connect to a vendor via his common gateway, he routes it (depending on destination) to multiple operators. NowSMS exposes a simple HTTP service (a URL call in other words) to add SMS to the vendor queue and it’s this service that’s preferred by a majority of the good vendors. However, HTTP traffic being what it is, a monolithic backend in PHP will stall if the request doesn’t return a response. Hence that file_get_contents_with_timeout.

Note: this is hardly an ideal solution (that would be a separate daemon). But without adding moving parts, this simple function should increase the reliability of our backend.

Comments (2) | Permalink

Mobshare.in Broadcast Widget

Tuesday, March 25, 2008

Here’s a sneak peek of the Mobshare broadcast widget I’ve been working on. If you look to the right and below, you can see that in action live. You’ll see the last ten pics I’ve uploaded and direct from the widget you can subscribe to my feed or send me an SMS.

The widget is the first released code which uses our internal API. While the API itself is undocumented [which hopefully will soon change], it’s public and it’s JSON (which makes implementing Javascript widgets incredibly easy).

How do you get your own widget? You can’t do that easily right now (the pleasures of cutting-edge code eh?), but paste this code somewhere:


<iframe width="150" height="150" src="http://mobshare.in/user/:mobshare_id:/broadcast/widget"></iframe>

…and change :mobshare_id: to your Mobshare ID.

Try it out and let me know if you like it!

Tags: broadcast • cool • javascript • json • mobshare.in • widget

Comment! | Permalink

Loving Git

Tuesday, March 18, 2008

I suppose this must be the zillionth post about how Git is so cool, but I’ll tell you what I love about it:

Ultra fast commits. Coming from subversion, typing commit and getting a prompt back instantly is so surprising that you double check the first few times.
Branching! God, you don’t realize how much you miss it until you have fast branching and merging. Without the pain of creating cps, just a simple git checkout -b branch_name and git merge branch_name and everything just works.
The above was the reason I tried git in the first place. mobME’s an SVN shop and we do the usual trunk, tags, branches dance. When the trunk shapes up to be stable though I can’t seem to do anything on it. I can fork off a new branch in SVN but that’s too painful to even think about. What do I do now? git clone it, create a branch, and do regular git svn rebases and dcommits.
Oh did I mention bidirectional SVN support? Without which I wouldn’t/couldn’t have switched no matter how much I wanted to try this cool new thing.

But it’s great and it really changes the way you think about VCS. Notice something? I didn’t even talk about distributed source control, and that’s coz even when I use git like I use SVN - committing to a central repo at the end and pulling changes from it, it’s brilliant. I’d definitely want to explore cool stuff like github soon for personal use (some gracious soul gave me an invite some time back).

Tags: development • git • source control • subversion • sysadmin • vcs

Comments (3) | Permalink

An Orkut Application via a JSON API

Saturday, March 8, 2008

I talked about delegating rendering in Symfony for creating a JSON API. Now here’s a consumer: an Orkut opensocial gadget:


MobshareOrkutAPI = {
	
	api_url: 'http://api.mobshare/api.php',
	cache_time: 0, //0 to disable
	
	makeCachedRequest: function(url, callback, params, refreshInterval) {
	  var ts = new Date().getTime();
	  var sep = "?";
	  if (refreshInterval && refreshInterval > 0) {
	    ts = Math.floor(ts / (refreshInterval * 1000));
	  }
	  if (url.indexOf("?") > -1) {
	    sep = "&";
	  }
	  url = [ url, sep, "nocache=", ts ].join("");
	  gadgets.io.makeRequest(url, callback, params);
	},
	
	call: function(module, action, params, callback) {
		var options = {};
		options[gadgets.io.RequestParameters.CONTENT_TYPE] = gadgets.io.ContentType.JSON;
		this.makeCachedRequest(this.api_url + '/' + module + '/' + action + '?' + params, callback, options, this.cache_time);
	}

};

makeCachedRequest has been plaigarized from the Opensocial documentation and it’s very useful to bust the cache for requests. Also, notice this line for setting the content-type to JSON:


options[gadgets.io.RequestParameters.CONTENT_TYPE] = gadgets.io.ContentType.JSON;

This is how we access that API, a code fragment:


...
authenticate: function(alias, mobile_no, password) {
	MobshareOrkutAPI.call('user', 'authenticate', 
		'alias=' + encodeURIComponent(alias) + '&mobile_no=' + encodeURIComponent(mobile_no) + 
			'&password=' + encodeURIComponent(password),
 		MyOrkutApp.login
	);
}
...
login: function(orkut_response) {
	response = orkut_response.data;
	data_success = response['success'];
	data_error = response['error'];
	
	if(data_success) {
		html = '<h2>Login Successful!</h2>';
		html += '<p>Welcome: ' + data_success.name + '!</p>';
	} else if(data_error) {
		html = '<h2>Login Unsuccessful!</h2>';
		html += '<p>' + data_error + '!</p>';
	}
	
	document.getElementById('mobshare_login').innerHTML += html;
},
...

Note that orkut_response.data is automatically set by Orkut because you passed in the JSON content type; it parses the data received and creates a javascript object. Cool, ain’t it? It’s very easy to create a proper interactive Opensocial app this way.

Tags: api • application • development • javascript • json • opensocial • orkut • web

Comment! | Permalink

Delegate Rendering in Symfony

Thursday, March 6, 2008

Warning: pretty advanced Symfony ahead: if you’re not familiar with the framework, this wouldn’t make sense.

I recently developed a bare-bones API for Mobshare (it’s not yet live), and to keep everything clean, I abstracted away the rendering bit from the controller to an external class. It ended up being a sweet solution, so here it is!

I wanted this to be a JSONP API (the major use case would be a JS client, and parsing XML etc. via JS is a pain. Besides JSON is much shorter over the wire). I didn’t want to rewrite a lot of code: checking for a callback parameter and wrapping the returned string around the JSON output was just begging to be refactored away. So here it is, a generic Symfony JSON API wrapper class:


<?php

class JSONPAPI {
	
	const CALLBACK_PARAMETER = 'callback';
	
	var $status;
	var $data;
	var $callback;
	
	
	public function __construct($status, $data) {
		$this->status = $status;
		$this->data = $data;
		
		$callback_parameter_value = $this->getCurrentAction()->getRequestParameter(self::CALLBACK_PARAMETER);
		if($callback_parameter_value) 
			$this->callback = $callback_parameter_value;
	}
	
	public function render() {
		$render_text = json_encode(array($this->status => $this->data));
		
		if($this->callback)
			$render_text = $this->callback . '(' . $render_text . ');';
		
		$this->setJavascriptHeaders();
		return $this->getCurrentAction()->renderText($render_text);
	}
	
	//hack to get the current action
	private function getCurrentAction() {
		return sfContext::getInstance()->getActionStack()->getLastEntry()->getActionInstance();
	}
	
	private function setJavascriptHeaders() {
		sfContext::getInstance()->getResponse()->setParameter('Content-Type', 'application/javascript', 'symfony/response/http/headers');
	}
}

The bits of magic here are the getCurrentAction() function and the render() call. It works on one very simple idea: everything in Symfony can be accessed from the sfContext::getInstance() object, you just need to dig deep enough.

Once you write this boiler-plate code, using it is very elegant. First, subclass it for your API:


<?php

require_once('JSONPAPI.class.php');

class MobshareAPI extends JSONPAPI {
	

}

And then, use it like so within your controller:


<?php
class userActions extends sfActions
{
	
	public function executeAuthenticate() {
		$valid_user = UserPeer::authenticate($this->getRequestParameter('alias'), 
			$this->getRequestParameter('mobile_no'), $this->getRequestParameter('password'));
		if($valid_user instanceOf User) {
			$success = new MobshareAPI('success', $valid_user->toArray());
			return $success->render();
		} else {
			$error = new MobshareAPI('error', 'Authentication failed: Alias, mobile number or password invalid.');
			return $error->render();
		}
	}
	
}

Note: the rendering has been delegated to the $success and $error MobshareAPI objects. This allows for a really maintainable API. Adding functionality is much simpler since you don’t have to worry about the boilerplate.

You end up calling the API like this:


http://api.mobshare/user/authenticate?alias=vish&password=xxx&callback=handler

and you get back data which looks like this:


handler({"success":{"alias":"vish","name":"Vishnu Gopal","photo_mini": ...);

Note that callback handling is done entirely by the API and the controller needn’t worry about this parameter at all!

Tags: api • cool • development • json • jsonp • PHP • symfony • web

Comments (4) | Permalink

dtach, a brilliant daemonizer

Tuesday, March 4, 2008

Most people use GNU screen (or equivalent) to daemonize processes. There is a better alternative called dtach which should become the definitive way to run a process in the background as a daemon.

Installing dtach is easy. If you’re on gentoo, it is a simple:


$ emerge dtach

dtach --help provides the following output:


Usage: dtach -a <socket> <options>
       dtach -A <socket> <options> <command...>
       dtach -c <socket> <options> <command...>
       dtach -n <socket> <options> <command...>
Modes:
  -a		Attach to the specified socket.
  -A		Attach to the specified socket, or create it if it
		  does not exist, running the specified command.
  -c		Create a new socket and run the specified command.
  -n		Create a new socket and run the specified command detached.

This basically means that you start dtach like so:


$ dtach -c ~/rtorrent/.socket/dtach rtorrent

The command-shortcut Ctrl+\ detaches the running command and runs it in the background. To re-attach, just enter:


$ dtach -a ~/rtorrent/.socket/dtach

dtach hasn’t got much more functionality, but that’s what makes it superb. Tiny and functional. As you can see, I’m using dtach to run a Seedbox

Tags: *nix • applications • sysadmin

Comment! | Permalink

EC2-like instances with Slicehost

Sunday, February 24, 2008

Virtual hosting and infrastructure is coming up in a big way, and Amazon’s EC2 is leading the pack. However, there doesn’t exist a comprehensive & provably scalable system for deploying applications to such a stack yet: the components are in place or being developed (SimpleDB or equivalent, EC2 and S3) but the crucial connecting layer is missing. Several attempts are on to fill the gap.

Until it becomes a reality (and perhaps even then), a great intermediate tool is Slicehost’s xen-based virtual machines: “slices”, that allow for a quick snapshot from any particular slice and the creation of a new slice from that snapshot.

The way it works is a bit cumbersome now (and they really should expose this feature more), but the idea is simple:

Take a snapshot of a slice, currently you have to go to the backups tab to take a snapshot backup.
When you choose to create a new slice, you have the option to create from that backup snapshot (see picture).
Change a few variables specific to the host (maybe add a puppet instance to reconfigure automatically), and wham! EC2-like virtual infrastructure at half the pain.

I plan to use this for factory-like instances across MobME’s infrastructure.

Tags: ec2 • hosting • puppet • slicehost • sysadmin • virtual

Comments (1) | Permalink

Earlier