Generating RSA keys is a characteristically CPU intensive operation. This presents problems when operating on weak devices (such as the iPhone 3G) in environments under strict computation restrictions (such as safari mobile’s 10 second javascript execution timeout).

What is needed is an RSA key generation library that operates asynchronously in order to chug through the 2+ minutes of computation time required to generate a 512 RSA key on a weak device without bumping against the computation restrictions enforced by the safari mobile execution environment.

Some nearly suitable libraries do exist, but all of them fall short in some fashion.

• Probably the closest is this asynchronous keygen from Atsushi Oka
  http://ats.oka.nu/titaniumcore/js/crypto/readme.txt
  However, the interface for Ats Oka’s library is not simple and the architecture of the code leaves something to be desired.

• Cryptico is another library featuring RSA key generation which is also touted as a sort of all-in-one solution
  http://code.google.com/p/cryptico/
  However, this library really just glues together a bunch of already-available libraries and packages them as a unit.

• jsbn is the underlying RSA key generator packaged with Cryptico and is available on its own
  http://www-cs-students.stanford.edu/~tjw/jsbn/
  This library has a fairly simple interface and is relatively fast and compact meeting most of my requirements.
  However, this library doesn’t do asynchronous key generation.

  But we can fix that.

jsbn RSA keygen times out after 11 seconds on the iPhone 3G for even a 256 bit key but with a little fenangling and a lot of setTimeouts, we can get it to handle a key of virtually any size for which the user has the patience to wait.

Here’s an example of the new async interface:

key = new RSAKey();
key.generateAsync(512, "03", function(){
    var pubKey = hex2b64(key.n.toString(16));
    alert(pubKey);
});

This was a great exercise in how to turn synchronous javascript into asynchronous javascript. Taking procedural code and breaking those for loops into recursive functions was a mind bender but once I figured out how it generally ought to work, each function became easier to port.

Originally I did it all inline, but later I ripped it all out into a separate file which extends Tom Wu’s jsbn.

// Copyright (c) 2011  Kevin M Burns Jr.
// All Rights Reserved.
// See "LICENSE" for details.
//
// Extension to jsbn which adds facilities for asynchronous RSA key generation
// Primarily created to avoid execution timeout on mobile devices
//
// http://www-cs-students.stanford.edu/~tjw/jsbn/
//
// ---
 
(function(){
 
// Generate a new random private key B bits long, using public expt E
var RSAGenerateAsync = function (B, E, callback) {
    //var rng = new SeededRandom();
    var rng = new SecureRandom();
    var qs = B >> 1;
    this.e = parseInt(E, 16);
    var ee = new BigInteger(E, 16);
    var rsa = this;
    // These functions have non-descript names because they were originally for(;;) loops.
    // I don't know about cryptography to give them better names than loop1-4.
    var loop1 = function() {
        var loop4 = function() {
            if (rsa.p.compareTo(rsa.q) <= 0) {
                var t = rsa.p;
                rsa.p = rsa.q;
                rsa.q = t;
            }
            var p1 = rsa.p.subtract(BigInteger.ONE);
            var q1 = rsa.q.subtract(BigInteger.ONE);
            var phi = p1.multiply(q1);
            if (phi.gcd(ee).compareTo(BigInteger.ONE) == 0) {
                rsa.n = rsa.p.multiply(rsa.q);
                rsa.d = ee.modInverse(phi);
                rsa.dmp1 = rsa.d.mod(p1);
                rsa.dmq1 = rsa.d.mod(q1);
                rsa.coeff = rsa.q.modInverse(rsa.p);
                setTimeout(function(){callback()},0); // escape
            } else {
                setTimeout(loop1,0);
            }
        };
        var loop3 = function() {
            rsa.q = nbi();
            rsa.q.fromNumberAsync(qs, 1, rng, function(){
                rsa.q.subtract(BigInteger.ONE).gcda(ee, function(r){
                    if (r.compareTo(BigInteger.ONE) == 0 && rsa.q.isProbablePrime(10)) {
                        setTimeout(loop4,0);
                    } else {
                        setTimeout(loop3,0);
                    }
                });
            });
        };
        var loop2 = function() {
            rsa.p = nbi();
            rsa.p.fromNumberAsync(B - qs, 1, rng, function(){
                rsa.p.subtract(BigInteger.ONE).gcda(ee, function(r){
                    if (r.compareTo(BigInteger.ONE) == 0 && rsa.p.isProbablePrime(10)) {
                        setTimeout(loop3,0);
                    } else {
                        setTimeout(loop2,0);
                    }
                });
            });
        };
        setTimeout(loop2,0);
    };
    setTimeout(loop1,0);
};
RSAKey.prototype.generateAsync = RSAGenerateAsync;
 
// Public API method
var bnGCDAsync = function (a, callback) {
    var x = (this.s < 0) ? this.negate() : this.clone();
    var y = (a.s < 0) ? a.negate() : a.clone();
    if (x.compareTo(y) < 0) {
        var t = x;
        x = y;
        y = t;
    }
    var i = x.getLowestSetBit(),
        g = y.getLowestSetBit();
    if (g < 0) {
        callback(x);
        return;
    }
    if (i < g) g = i;
    if (g > 0) {
        x.rShiftTo(g, x);
        y.rShiftTo(g, y);
    }
    // Workhorse of the algorithm, gets called 200 - 800 times per 512 bit keygen.
    var gcda1 = function() {
        if ((i = x.getLowestSetBit()) > 0){ x.rShiftTo(i, x); }
        if ((i = y.getLowestSetBit()) > 0){ y.rShiftTo(i, y); }
        if (x.compareTo(y) >= 0) {
            x.subTo(y, x);
            x.rShiftTo(1, x);
        } else {
            y.subTo(x, y);
            y.rShiftTo(1, y);
        }
        if(!(x.signum() > 0)) {
            if (g > 0) y.lShiftTo(g, y);
            setTimeout(function(){callback(y)},0); // escape
        } else {
            setTimeout(gcda1,0);
        }
    };
    setTimeout(gcda1,10);
};
BigInteger.prototype.gcda = bnGCDAsync;
 
// (protected) alternate constructor
var bnpFromNumberAsync = function (a,b,c,callback) {
  if("number" == typeof b) {
    if(a < 2) {
        this.fromInt(1);
    } else {
      this.fromNumber(a,c);
      if(!this.testBit(a-1)){
        this.bitwiseTo(BigInteger.ONE.shiftLeft(a-1),op_or,this);
      }
      if(this.isEven()) {
        this.dAddOffset(1,0);
      }
      var bnp = this;
      var bnpfn1 = function(){
        bnp.dAddOffset(2,0);
        if(bnp.bitLength() > a) bnp.subTo(BigInteger.ONE.shiftLeft(a-1),bnp);
        if(bnp.isProbablePrime(b)) {
            setTimeout(function(){callback()},0); // escape
        } else {
            setTimeout(bnpfn1,0);
        }
      };
      setTimeout(bnpfn1,0);
    }
  } else {
    var x = new Array(), t = a&7;
    x.length = (a>>3)+1;
    b.nextBytes(x);
    if(t > 0) x[0] &= ((1<<t)-1); else x[0] = 0;
    this.fromString(x,256);
  }
};
BigInteger.prototype.fromNumberAsync = bnpFromNumberAsync;
 
})();

I’m also adding a few functions for sleeping a public or private RSA key object to a simple JSON Transport Object (for storage) and waking it again.

(function(){
 
// Cast to private Transport Object
var RSAPrivTPO = function () {
    return {
        'n'     : hex2b64(this.n.toString(16)),
        'e'     : hex2b64(this.e.toString(16)),
        'd'     : hex2b64(this.d.toString(16)),
        'p'     : hex2b64(this.p.toString(16)),
        'q'     : hex2b64(this.q.toString(16)),
        'dmp1'  : hex2b64(this.dmp1.toString(16)),
        'dmq1'  : hex2b64(this.dmq1.toString(16)),
        'coeff' : hex2b64(this.coeff.toString(16))
    }
}
RSAKey.prototype.privTPO = RSAPrivTPO;
 
// Hydrate from private Transport Object
var RSAFromPrivTPO = function (tpo) {
    this.setPrivateEx(
        b64tohex(tpo.n), 
        b64tohex(tpo.e), 
        b64tohex(tpo.d), 
        b64tohex(tpo.p), 
        b64tohex(tpo.q), 
        b64tohex(tpo.dmp1), 
        b64tohex(tpo.dmq1), 
        b64tohex(tpo.coeff)
    );
    return this;
};
RSAKey.prototype.fromPrivTPO = RSAFromPrivTPO;
 
// Cast to public Transport Object
var RSAPubTPO = function () {
    return {
        'n' : hex2b64(this.n.toString(16)),
        'e' : hex2b64(this.e.toString(16))
    }
};
RSAKey.prototype.pubTPO = RSAPubTPO;
 
// Hydrate from public Transport Object
var RSAFromPubTPO = function (tpo) {
    this.setPublic(
        b64tohex(tpo.n), 
        b64tohex(tpo.e)
    );
    return this;
};
RSAKey.prototype.fromPubTPO = RSAFromPubTPO;
 
})();

Source on GitHub – https://github.com/KevBurnsJr/rsasync

Daniel Roussel posts:

Daniel, Here are 3 ways you might use self-descriptive messages in your API

1) Create many domain specific media types (one for each view)

Content-Type: application/rent-a-room+xml

2) Create one domain specific media type

Content-Type: application/vnd.hotels.com+xml

3) Create zero domain specific media types

Content-Type: application/json
Link: </schema/rent-a-room>; rel="describedBy"

All three of these approaches could be seen as satisfying the self-descriptive messages constraint.

If you create many DSMs (domain specific media types), your application might bind the media type to the view class via some sort of client-side configuration.

"application/rent-a-room+xml"   =>   RentARoomView

If you create one DSM, your media type might specify the semantics by which a representation specifies details about itself which could be used in rendering the representation in a GUI.

{"_type": "rent-a-room", ... }

… which you might then bind to a view …

"rent-a-room"   =>   RentARoomView

If you create zero DSMs, your application might bind the value of the describedBy link header to a view in the gui.

"/schema/rent-a-room"   =>   RentARoomView

–

An alternative approach would be to create one DSM with a richer semantics which would effectively allow you to compose the interface from the server side using code-on-demand and/or more granular views

{"_links":[
    {"rel":"view","type":"text/javascript","href":"/views/RentARoomView.js"}
    {"rel":"commentable","type":"text/javascript","href":"/attributes/commentable.js"}
]}

This Code-on-demand approach would take greatest advantage of the constraints of REST to create a highly evolvable service by never binding anything directly to a view class within the application. Instead, your application would become a user agent, parsing representations and fetching additional computational resources as necessary to render the view.

Code-on-demand may be significantly less feasible if your client is written in object C, but perhaps it’s something to think about. The embedded links might not be javascript or CSS, but perhaps some other language used for GUI composition, such as XUL or a simple DSL.

Finally I’m sure it goes without saying that whatever way you wind up rendering a representation for a view, the UI would contain links which you would click to navigate to new screens which are built using the data and metadata from the representation of the resource identified by the link.

And there you have a few takes on creating an engine of application state with self-descriptive messages and code-on-demand.

Lambdas used for sections should receive the raw section string.

This means that the inputs to your lambda will not be pre-rendered. The output of the lambda will be rendered.

So I had a lambda that looked like this…

function($timestamp) {
	return str_replace("T", " ", $timestamp);
}

But It was giving jibberish.
bobthecow (author of Mustache.php) helped me realize that I needed to render the inputs myself…

function($timestamp) use($view) {
	return str_replace("T", " ", $view->render($timestamp));
}

The lesson is this:
The next time I run into a problem, I’ll be sure to read the tests.

Thanks, Justin!

During the conversation, the product guy (who has some tech chops) said something that caught me off guard. His quip was something like: “Cloud is the way to go. We’re definitely going to do everything in the cloud.”

Below is my lengthy response.

Hi Chris,

You mentioned something in that first meeting which I’ve been wanting to elaborate on and partially refute.

I’d like to caution that The Cloud is not a silver bullet.
While AWS may be an excellent solution for the immediate future, there may come a point where moving to a datacenter proves to be more reliable and cost-effective.

There are a number of properties of the infrastructure involved in cloud computing which guarantee that some cloud resources will never match the performance and reliability of bare metal.

See for instance the recent Reddit outage in which AWS EBS volumes became unavailable for several hours, resulting in serious data problems.
http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html

In AWS, the hard drives (EBS) are connected to the machines (EC2) over a shared 1Gb/s ethernet connection.
In a real server, the hard drives are connected to the machine over a dedicated 6Gb/s SATA cable.

There are some ways to help mitigate these bottlenecks
http://victortrac.com/EC2_Ephemeral_Disks_vs_EBS_Volumes

1) you can set up multiple EBS volumes in a RAID configuration to increase throughput
2) you can load balance traffic across multiple availability zones to help mitigate network congestion and availability issues like those experienced by Reddit

But also keep in mind that there will come a point at which renting a cabinet (42U) in a datacenter will become far more cost effective than running a server farm in the cloud.
A full cabinet with 15Amps of power and a 100Mb/s connection runs $500/mo with a signed 3 year lease.
A quarter cabinet (7U) with 2Amps of power and a 100Mb/s connection runs $200/mo with a signed 3 year lease.

So if we took one of these $200 PE 1850s I have sitting here and plugged it in to a quarter cabinet at Hurricane Electric, we would already have more processing power and it would save us $40/mo over our current us-west m1.large instance.

But there are 2 advantages to the cloud:
1) If we need more capacity, we can provision it immediately without having to provision hardeware
2) If we need capacity in a different location, we can easily spin up a cluster in Japan or Europe or the East Coast without having to physically be there.

In the long term I hope we’ll come to a hybrid approach where we can
- leverage the best of real-world datacenters (for backups and heavy number crunching)
- and leverage the cloud for all its advantages (for geo dispersion, fault tolerance and early stage ramp-up)

Remember the words of Devops Borat
http://twitter.com/DEVOPS_BORAT/status/40816140632596480

“Formation of hybrid cloud: 1) roll in datacenter 2) SHIT! roll to cloud 3) SHIT! roll half back to datacenter.”

- Kev

Thanks, Chris for suggesting I post this to the blog.

One of Tumblr’s engineers (presumably) deployed a file to production which contained a critical flaw.
The first character of the file was replaced with an `i` instead of a `<`.

i?php
    require_once('chorus/Utils.php');
    require_once('chorus/Kestrel.php');
    require_once('chorus/DataService.php');
    require_once('chorus/Shard.php');
 
    Database::set_defaults(array(
        'user'     => 'tumblr3',
        'password' => 'm3MpH1C0Koh39AQD83TFhsBPlOM1Rx9eW55Z8YWStbgTmcgQWJvFt4',
        'database' => 'tumblr3',
    //    'write_lock_tables' => '*',
        'extended_log' => (idate('G') == 17 && intval(idate('i')) == 56 && trim(`hostname`) == 'web10.tumblr.com')
    ));
 
    if (__FILE__ == '/var/www/apps/tumblr/config/config.php' || __FILE__ == '/data/tumblr/config/config.php') {
        define('ENVIRONMENT',      'production');
        if (! defined('DEFAULT_DATABASE')) define('DEFAULT_DATABASE', 'primary');
        define('S3_BUCKET',        'data.tumblr.com');
        define('ENABLE_PANTHER',   true);
        define('ENABLE_MEDIA_CDN', true);
        define('ASSETS_URL',       (ENABLE_MEDIA_CDN && ! (isset($_SERVER['HTTPS']) && $_SERVER['HTTPS']) ? 'http://assets.tumblr.com' : ''));
        define('MEMCACHE_HOST',         '10.252.0.68');
        define('MEMCACHE_VERSION_HOST', '10.252.0.67');
        define('VALIDATION_FAILURE_LOG', BASE_PATH . '/validate.log');
        # <snip>

Yes, that is tumblr’s production database password.

Full source here

Once this error was introduced to production, people viewing any page would see a dump of the first 749 lines of the config file along with some PHP errors. THEN, GoogleBot came along and indexed the whole mess which is why you can still see it in Google’s search results

Learning from the mistakes of others

How can we keep this from happening to us?

First of all, go to any project on your local dev environment, replace the first `<` with `i` in any included file in your project (hint config.php) and see what happens. Chances are you'll see exactly the same thing that happened to Tumblr.

The only solution I see to this is pre-commit syntax checking for committed PHP files.

Here's a tutorial for php syntax checking in SVN and here’s a pre-commit hook script for php syntax checking in GIT.

Basically the way it works is that if you ever commit a PHP file which contains a syntax error, your commit will be blocked and you will have to amend it before you are allowed to commit it to the repository. If Tumblr had done this, they might never have leaked their config file.

[UDPATE] Apparently both PHP’s built-in Syntax Checking and the PEAR package PHP_CodeSniffer won’t pick up on errors such as this. Searching for a valid solution now…

[UDPATE] The best solution I’ve seen so far is to always return 500 responses to clients with a generic error message in production, thus preventing errors like this from bubbling up at the Apache level.

[UDPATE] Turns out PHP returns 200 when it encounters a fatal error. Inconceivable.

[UDPATE] Here’s how to force PHP to return a 500 when it encounters a fatal error. Add this to a prepend file or make it the first 5 lines of your bootstrap file.

function die_with_honor() {
	header("HTTP/1.1 500 Internal Server Error");
	ob_clean();
}
register_shutdown_function('die_with_honor');

Hi, its me again ;-).

I was thinking about what you said about focusing on your stuff in the US. I was driving along the road to Supetar today, passing the olive orchards and thinking of you.

I was thinking about how one’s 20s are the time for learning and exploring, honing skills and interests, finding out what you like and don’t like, learning what gives you satisfaction and a sense of purpose – for your professional and personal life. One’s 30s are an important time to focus what you have learned and build the foundation for life. I think that means choosing how you want to work and what you want to do as well as building long lasting professional, personal and family relationships.

In my 20s I worked in a lot of areas and learned as much as I could technically as well as about business, constantly challenging myself to try new things until I found what I was best at and what gave me the most sense of satisfaction and purpose. It took a long time ;-). I must have taken every type of class offered at school, just for the enjoyment of it. Once I discovered the computer industry, there was an endless supply of things to learn, which I had never dreamed of. I hadn’t even seen a computer ever when I started my first computer business. I was stunned by the possibility of it when I went to my first computer club meetings where they were talking about the Apple I computer and Local Area Networks (ethernet to be specific).

I also learned after having you that I wanted to have as much control and flexibility over my time so I could focus on you children and the family. This is why I went into consulting. I felt I could have rewarding work, would learn a lot from working with many companies, but also not be overwhelmed by the demands of a job where someone else decided when and where. I think I built a very good foundation in my 30s for this and was able in my 40s to have even more flexibility and further develop my business capabilities. Of course, I would have liked to have spent even more time at home because once I focused on business, the demands from clients made it increasingly difficult to manage my time. The conundrum of balancing work, getting new business and family life was much harder than I thought it would be.

The first time I tried consulting on my own (right after I had Deirdre), I was unsuccessful. I didn’t want to go back to work with the two of you at home. I was 30. But it turned out I didn’t have the contacts or ability to get clients, so I didn’t have much work – we had invented email on the PC and Macintosh, but nobody really used it. And the web had not been commercialized, so the only people that used it were universities and research companies like SRI.

I was really lucky that a friend of mine who worked at Apple called me and asked if I wanted to work on a project for Apple because one of the consultants that was working on it quit right in the middle and had not done any of the work. It turns out he didn’t know what to do. So, I said yes and wound up starting a consulting firm with the 3 people who were working on the project together. . . so many serendipitous events, which really shaped my life. I think what I learned from it was that in order to get to a point where I could do what I wanted, I had to have some foundations put into place. About 5 years later, I was able to start my own firm, quite successfully – I had learned how to run a consulting business and I had the network to get work.

We haven’t spoken in a long time about what you want and what you are thinking about, I am very interested in your thoughts and feelings about what you are doing.

Lots of love,

Mom

Lean Thinking	process engineering
Metaphors We Live By	role of language in psychology
Just Enough Software Architecture	software architecture
Drive	psychology of motivation
Web Operations	operations
Leading Geeks	management
Programming Erlang	programming

This gives me the opportunity to jump around to follow whatever impulse is fueling me at any given time. If I’m feeling burned after working around some bogus proprietary API, I will jump into a book on Software Architecture to remind me that there is still a path toward sanity. If I’ve been teching out for days and feel like I’ve lost the forest for trees, I’ll pick up a book on psychology to remind me that software does still have the potential to assist human beings in a large number of not-necessarily-obvious ways. If I’ve spent all day writing emails and haven’t had time to touch a piece of code, I’ll jump into a book on process or management to give me new ideas on how to talk less and do more. If I’m feeling adventurous, I’ll pick up a book on a new language and type along for a few hours to advance my understanding of the universe.

And I do the same thing with projects:

If I’m burnt on churning out HTML and CSS for days, I’ll step back and jump into something esoteric like adding a new transport protocol to an open source database client. If I’ve been crunching on a hard CS problem and start lacking steam, I’ll jump over to an application I’ve got on the back burner and knock out a few quick features to get the gears turning again. If I’m losing sleep over a cool new use case I’ve been turning over, I’ll flip on the lights and crank out a few test cases to put my ideas on paper as fodder for tomorrow’s session.

This sort of autonomy is crucial to remaining happy and productive.

Plant a thousand projects and let them flourish as they may.