Google App Engine push task queues & the development environment
Today Bart and I spent 2 hours debugging in the python Google App Engine, trying to figure out why all our scheduled tasks start to fail in the development server.
It is no secret that the public Rogerthat cloud is running on Google’s App Engine infrastructure.
To make our messaging service reliable, we make a lot of use of the deferred library which relies on the push task queue mechanism. So we kick off quite a large number of tasks.
At some point during testing on the development server, all tasks executions start to fail, flooding the logs as follows:
WARNING 2012-03-27 12:40:36,447 taskqueue_stub.py:1936] Task task4 failed to execute. This task will retry in 0.016 seconds WARNING 2012-03-27 12:40:36,455 taskqueue_stub.py:1936] Task task3 failed to execute. This task will retry in 0.032 seconds WARNING 2012-03-27 12:40:36,468 taskqueue_stub.py:1936] Task task4 failed to execute. This task will retry in 0.032 seconds WARNING 2012-03-27 12:40:36,491 taskqueue_stub.py:1936] Task task3 failed to execute. This task will retry in 0.064 seconds WARNING 2012-03-27 12:40:36,502 taskqueue_stub.py:1936] Task task4 failed to execute. This task will retry in 0.064 seconds WARNING 2012-03-27 12:40:36,558 taskqueue_stub.py:1936] Task task3 failed to execute. This task will retry in 0.128 seconds WARNING 2012-03-27 12:40:36,570 taskqueue_stub.py:1936] Task task4 failed to execute. This task will retry in 0.128 seconds WARNING 2012-03-27 12:40:36,688 taskqueue_stub.py:1936] Task task3 failed to execute. This task will retry in 0.256 seconds WARNING 2012-03-27 12:40:36,701 taskqueue_stub.py:1936] Task task4 failed to execute. This task will retry in 0.256 seconds
As you can see, no stack traces.
First we tried searching the internet for a solution, but apart from many others looking for help, we did not find the fix we needed. So there was no other option, as to debug the Google App Engine dev_server stack ourselves.
After a while we found that for some reason the development server stopped launching the task urls on the correct location. For some reason it resolved its own hostname and tried to access the RequestHandler with the hostname instead of on the ip address it is serving our app. Which did not work because the hostname resolves locally to 127.0.0.1 instead of on the ip address on wich we have the development serving. On top of that it totally lost the configured port as well on which we configured the dev server.
We fixed it by adding an extra line at the following location:
root@dev:/root/testing# git diff /root/testing/google_appengine/google/appengine/api/taskqueue/taskqueue_stub_original.py /root/testing/google_appengine/google/appengine/api/taskqueue/taskqueue_stub.py
diff --git a/root/testing/google_appengine/google/appengine/api/taskqueue/taskqueue_stub_original.py b/root/testing/google_appengine/google/appengine/api/taskqueue/taskqueue_stub.py
index df3b6a7..305167a 100644
--- a/root/testing/google_appengine/google/appengine/api/taskqueue/taskqueue_stub_original.py
+++ b/root/testing/google_appengine/google/appengine/api/taskqueue/taskqueue_stub.py
@@ -1826,6 +1826,7 @@ class _TaskExecutor(object):
connection_host, = header_dict.get('host', [self._default_host])
+ connection_host = self._default_host
if connection_host is None:
logging.error('Could not determine where to send the task "%s" '
'(Url: "%s") in queue "%s". Treating as an error.',
Performing remote API calls in a reliable way.
I just finished coding a new version of Mobidick. This is a sample Rogerthat service which our customers use to try out & test drive our messaging APIs. I use some elegant and new techniques to perform API calls in a very reliable way. I’m pretty pleased with the result, so felt like blogging about it.
My definition of performing API calls in a reliable way: retriable API calls to remote systems getting an idempotent result
It seems obvious, and all systems should interoperate like that, but actually it is not so obvious at all. This is mainly because client and server must cooperate to get this behavior.
For example, on the client side you cannot submit a retriable API call from within a synchronous HTTP request processing, because the client invoking the HTTP request expects a result within a few seconds. Hence making an API call to a remote system reliably, requires to execute it from an out-of-context system (e.g. a queueing server) which is preferably transactional.
On the other hand, on the server side you must be able to deal with receiving single API calls multiple times, preventing double execution of the invoked API, yet returning the correct return values to the client.
There are two options to solve this on the server:
- An API result cache
- Implementing all functionality in the server in an idempotent way
For the Rogerthat server we opted for the API result cache method. The reason why we choose this method is because it is almost impossible to write a messaging system idempotently. Also the API result cache approach results in much easier coding, which makes our developers more productive.
For example, when I send two identical messages to my colleague Carl, he must receive them both. An idempotent implementation of the messaging system would prevent that. On the other hand, if I have to submit a single message twice due to eg. networking troubles, I only want Carl to receive the message once.
Indeed we cache every API call invocation and its calculated result for some time. If we receive an API call twice, we do not process it again, thereby avoiding the side effects of the call invocation. Instead we just return the cached result.
That’s why we opted for JSON-RPC which adds call id to every single API invocation.
Executing the following call twice, results in 1 message sent to Carl, because the server cashes the result based on the id that the client put on the API call.
{
'id': '8CC77A45-7898-4DC9-8D52-6D3B7871BF98',
'method': 'send_message',
'params': {
'subject': 'long time no see',
'message': 'Hi Carl,\n\nits been a while since we saw each other\n\nShall we grab dinner together this evening?'
}
}
That is ok, the server is protected to send a message only once, even if the API caller was unable to receive the result of the first invocation of the API call. So Carl’s inbox won’t be spammed in case of trouble, and I’m sure he will get the message, because I don’t have to be afraid to retry the API call in troublesome situations.
Now I only have to make sure that my client code also plays along nicely.
If you know Google App Engine, you will have noticed that Mobidick is a Google App Engine app. In fact it is created in the python version of Google App Engine with the python 2.7 runtime.
Google App Engine makes it quite easy to implement the desired client behavior to call a remote API reliably.
Eg: Submitting the send_message API call correctly.
class SendMessageHandler(webapp2.RequestHandler):
def post(self):
# Get post data
subject = self.request.get("subject")
message = self.request.get("message")
# Store message locally, and then submit to remote transmitting system
def trans():
msg = Message()
msg.subject = subject
msg.message = message
msg.put()
deferred.defer(_send_message_to_carl, msg, str(uuid.uuid4())
xg_on = db.create_transaction_options(xg=True)
db.run_in_transaction_options(xg_on, trans)
def _send_message_to_carl(msg, json_rpc_id):
json_rpc_call = {'id': json_rpc_id,
'method': 'send_message',
'params': {'subject':msg.subject, 'message': msg.message} }
json_rpc_result = perform_http_json_rpc_request(json_rpc_call)
do_something_with_the_result(json_rpc_result)
As can you see, it does not require a lot of energy from the developer to make the API call idempotent and reliable.
So, where is the magic ? The magic lies in the transactional scheduling of a task. The Python version of GAE supplies us with the magical way of scheduling a task through the deferred library, which makes it very convenient to perform out-of-context actions. Also, GAE automatically retries tasks when they fail. When perform_http_json_rpc_request fails, it will be executed again by GAE until it succeeds.
Useful links:
Rogerthat product: http://www.rogerthat.net
Rogerthat web application: https://rogerth.at
Mobidick: https://mobidick-cloud.appspot.com/
json-rpc: http://json-rpc.org/
Google App Engine: http://code.google.com/appengine/
Transactions in GAE: http://code.google.com/appengine/docs/java/datastore/transactions.html
Background work with deferred library: http://code.google.com/appengine/articles/deferred.html
Accessing the Rogerthat API from mono.
Peter Grypdonck from TechShore.EU today reported that he could not access the Rogerthat API through the opensource .NET API stubs from his ubuntu 11.10.
After some digging Peter discovered the root cause of the problem. The Rogerthat SSL certificate was not trusted by his mono installation.
He resolved the problem by executing
certmgr -ssl https://rogerth.at/api/1
and answering ‘y’ on every question. As a result the mono installation trusts the complete certificate chain used by the Rogerthat API.
How to mute Rogerthat sound on iPhone 4 / iOS 5
Due to a bug in Apple iOS 5, it is not possible for applications to retrieve the state of the hardware mute button on your iPhone 4.
It has been confirmed by Apple that this is indeed not supported.
A workaround to silence the Rogerthat application is to mute your Media Volume as follows:
- Mute your ringer sound by toggling the hardware mute switch
- Mute your media sound using the following steps
- Double click your home button
- Scroll the bottom pane completely to the left
- Minimize the volume. Check the screenshot
Retrieving the state of the hardware mute button was possible on an iPhone 4 having the iOS 4 operating system.
We asked Apple to reintroduce this functionality in iOS 5 and will include this in the Rogerthat application as soon as the Apple functionality becomes available.
Processing iPhone or iPad Crash Reports
When iPhone or iPad apps crash, you typically see crash information such as
Thread 5 name: Dispatch queue: com.apple.root.default-priority Thread 5 Crashed: 0 libobjc.A.dylib 0x333a6c98 objc_msgSend + 16 1 rogerthat 0x00028066 0x1000 + 159846 2 rogerthat 0x00056876 0x1000 + 350326 3 rogerthat 0x0006eaa2 0x1000 + 449186 4 rogerthat 0x000c8224 0x1000 + 815652 5 rogerthat 0x0002b69a 0x1000 + 173722 6 rogerthat 0x0002c052 0x1000 + 176210 7 rogerthat 0x0002b336 0x1000 + 172854 8 rogerthat 0x00025de8 0x1000 + 151016 ...
This crash information can be found in XCode – Organizer – Device Logs.
These crash reports are pretty unusable unless you “symbolicate” them.
Here is a small tutorial on how to do this:
- Build your iPhone app using Build and Archive in the XCode Build menu
- In XCode Organizer, select your app in Archived Applications
- Select the appropriate build and right click Show in Finder
- Apple-C to copy this folder
- In your terminal cd to that folder (Apple-V to paste)
- locate symbolicatecrash – On my Mac it is in
/Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources/symbolicatecrash - Execute the symbolicatecrash command and enjoy a readable crash report
$pwd
/Users/carl/Library/Application Support/Developer/Shared/Archived Applications/97868891-2237-46CA-A87E-66E6E7B7A2DF.apparchive
$ls
ArchiveInfo.plist rogerthat.app rogerthat.app.dSYM
$locate symbolicatecrash
/Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/ DTDeviceKit.framework/Versions/A/Resources/symbolicatecrash
$/Developer/Platforms/iPhoneOS.platform/Developer/Library/ PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources/ symbolicatecrash ~/Downloads/stack.txt
...
Thread 5 name: Dispatch queue: com.apple.root.default-priority
Thread 5 Crashed:
0 libobjc.A.dylib 0x333a6c98
objc_msgSend + 16
1 rogerthat 0x00028066
-[NSMutableDictionary(MCTTools2) setString:forKey:]
(MCTJSONUtils.m:168)
2 rogerthat 0x00056876
-[MCTIntent setString:forKey:]
(MCTIntent.m:123)
3 rogerthat 0x0006eaa2
-[MCTIdentityStore saveAvatarWithData:]
(MCTIdentityStore.m:219)
4 rogerthat 0x000c8224
-[MCTGetAvatarResponseHandler handleResult:]
(MCTGetAvatarResponseHandler.m:66)
5 rogerthat 0x0002b69a
-[MCTRPCProtocol processIncomingResponse:]
(MCTRPCProtocol.m:201)
6 rogerthat 0x0002c052
-[MCTRPCProtocol processIncomingMessagesDict:]
(MCTRPCProtocol.m:260)
7 rogerthat 0x0002b336
-[MCTRPCProtocol processIncomingMessagesStr:]
(MCTRPCProtocol.m:321)
8 rogerthat 0x00025de8
-[MCTCommunicationManager communicationFinishedWithLoop:]
(MCTCommunicationManager.m:368)
...
Problems visiting https:// sites on your Android mobile browser?
On a number of mobile data networks, and for certain types of Android phones, it is impossible to visit SSL sites using the builtin mobile web browser.
What you typically experience:
- When your phone is connected to Wi-Fi, you can visit https:// sites without problems
- When Wi-Fi is disabled or out of reach, your phone will use mobile data (typically 3G or Edge) to connect to the internet. When visiting a https:// site, the browser window stays empty, and the browser immediately stops loading the page.
My specific configuration:
- Phone: Google Nexus One (2010)
- Android version: 2.3.4 “Gingerbread”
- Country: Belgium
- Mobile data provider: Mobistar
- Network type: Edge or 3G
The same problem has been reported on many mobile data networks in various countries.
The solution is simple: disable the APN Proxy settings on your phone. On my Nexus One with Android 2.3.4 this is the menu sequence:
- Open menu
- Select Settings
- Select Wireless & networks
- Select Mobile networks
- Select Access Point Names
- Select your active access point
- Clear the values for Proxy and Port
- Press menu button and select Save