Posted by & filed under Blog, Tech.

I just finished coding a new version of Mobidick. This is a sample Rogerthat service which our customers use to try out & test drive our messaging APIs. I use some elegant and new techniques to perform API calls in a very reliable way. I’m pretty pleased with the result, so felt like blogging about it.

My definition of performing API calls in a reliable way: retriable API calls to remote systems getting an idempotent result

It seems obvious, and all systems should interoperate like that, but actually it is not so obvious at all. This is mainly because client and server must cooperate to get this behavior.

For example, on the client side you cannot submit a retriable API call from within a synchronous HTTP request processing, because the client invoking the HTTP request expects a result within a few seconds. Hence making an API call to a remote system reliably, requires to execute it from an out-of-context system (e.g. a queueing server) which is preferably transactional.

On the other hand, on the server side you must be able to deal with receiving single API calls multiple times, preventing double execution of the invoked API, yet returning the correct return values to the client.

There are two options to solve this on the server:

  • An API result cache
  • Implementing all functionality in the server in an idempotent way

For the Rogerthat server we opted for the API result cache method. The reason why we choose this method is because it is almost impossible to write a messaging system idempotently. Also the API result cache approach results in much easier coding, which makes our developers more productive.

For example, when I send two identical messages to my colleague Carl, he must receive them both. An idempotent implementation of the messaging system would prevent that. On the other hand, if I have to submit a single message twice due to eg. networking troubles, I only want Carl to receive the message once.

Indeed we cache every API call invocation and its calculated result for some time. If we receive an API call twice, we do not process it again, thereby avoiding the side effects of the call invocation. Instead we just return the cached result.

That’s why we opted for JSON-RPC which adds call id to every single API invocation.

Executing the following call twice, results in 1 message sent to Carl, because the server cashes the result based on the id that the client put on the API call.

  'id': '8CC77A45-7898-4DC9-8D52-6D3B7871BF98',
  'method': 'send_message',
  'params': {
    'subject': 'long time no see',
    'message': 'Hi Carl,\n\nits been a while since we saw each other\n\nShall we grab dinner together this evening?'

That is ok, the server is protected to send a message only once, even if the API caller was unable to receive the result of the first invocation of the API call. So Carl’s inbox won’t be spammed in case of trouble, and I’m sure he will get the message, because I don’t have to be afraid to retry the API call in troublesome situations.

Now I only have to make sure that my client code also plays along nicely.

If you know Google App Engine, you will have noticed that Mobidick is a Google App Engine app. In fact it is created in the python version of Google App Engine with the python 2.7 runtime.

Google App Engine makes it quite easy to implement the desired client behavior to call a remote API reliably.

Eg: Submitting the send_message API call correctly.

class SendMessageHandler(webapp2.RequestHandler):

    def post(self):
        # Get post data
        subject = self.request.get("subject")
        message = self.request.get("message")
        # Store message locally, and then submit to remote transmitting system
        def trans():
            msg = Message()
            msg.subject = subject
            msg.message = message
            deferred.defer(_send_message_to_carl, msg, str(uuid.uuid4())
        xg_on = db.create_transaction_options(xg=True)
        db.run_in_transaction_options(xg_on, trans)

def _send_message_to_carl(msg, json_rpc_id):
    json_rpc_call = {'id': json_rpc_id,
                     'method': 'send_message',
                     'params': {'subject':msg.subject, 'message': msg.message} }
    json_rpc_result = perform_http_json_rpc_request(json_rpc_call)

As can you see, it does not require a lot of energy from the developer to make the API call idempotent and reliable.

So, where is the magic ? The magic lies in the transactional scheduling of a task. The Python version of GAE supplies us with the magical way of scheduling a task through the deferred library, which makes it very convenient to perform out-of-context actions. Also, GAE automatically retries tasks when they fail. When perform_http_json_rpc_request fails, it will be executed again by GAE until it succeeds.

Useful links:

Rogerthat product:
Rogerthat web application:
Google App Engine:
Transactions in GAE:
Background work with deferred library:

Leave a Reply