I’ve been looking to understand the Android OS better, so that I can answer some questions and create plans for getting Gecko/Firefox running on Android-based devices.  One of the first questions I asked was, “How do Android apps start?”  They’re clearly separate processes while they’re running, but it wasn’t clear how they were launched.  It turns out, there are a couple of pieces here.  I’m going to describe what I’ve discovered here, in case it’s useful for someone else; I haven’t been able to find much of this information, largely because I don’t think many people need to know any of this.

At boot time, a special instance of the Java VM is launched, called the Zygote.  This process loads a bunch of the core Java classes and performs initial processing of them, making it possible to avoid this step for each app launch.  Once the initial work is done, the process listens to a socket and waits for requests.

To launch an app using the Zygote process, a command-line tool called “dvz” can be used.  It sends its arguments to the Zygote, which will fork and then start executing the main method in a given class.

So, we have these steps, dealing with the Zygote process:

• the zygote process is executed at system boot; it does initialization, and then runs a select() loop listening to requests
• the zygote process is sent a message, which includes the start args, which will include the class name for main launch
• the zygote process reads connection args in ZygoteConnection.runOnce
• the zygote process forks in ZygoteConnection.runOnce (Zygote.forkAndSpecialize, native)
• ZygoteConnection.handleChildProc is called, which does some cleanup and eventually throws a MethodAndArgsCaller (via ZygoteInit.invokeStaticMain)
• … which takes us all the way back to ZygoteInit.main, which catches that trampoline exception, and calls run()

But, that’s not fully how a new Android Activity is started.  It’s a bit of a roundabout process.  To launch a new Activity, the ActivityManagerService is notified with an activity start request, including things like the name/class/etc. of the activity.  It puts that information in a list of activities to run.  Then, a new process is started with the main from ActivityThread.  This new process then contacts the ActivityManagerService and asks, essentially, “what app am I?”.  The service then gives it the name of its activity class and other info, which is then loaded, and a message is enqeued on the main thread to instantiate the new activity and send it an onCreate() message.

This is interesting because it means that apps are not launched directly, but instead somewhat indirectly through specializing a generic “Activity” process for a specific activity.  A side effect of this is that I couldn’t find a way to actually register an app with the ActivityManagerService if it wasn’t launched by it.  So, to be a full Android app, you have to go through this normal startup process.

JNI Bridging

One of the difficulties in porting Gecko to Android is that the Android platform is built around Java, whereas Gecko is very much all native C/C++.  However, there is a fairly good native bridge layer, JNI, which is fairly heavily optimized by Dalvik.  So, the simplest way to connect these two is to write a shell app in Java, which bridges events, messages, paint requests, etc. to the native code for handling.

As a proof of concept of this, I wrote a simple test app.  It’s fairly straightforward, with a few wrinkles.  Most of this stuff can be done with the stock Android SDK and NDK — except painting.  The only API that the NDK exposes for graphics is OpenGL ES.  This is fine, but in some cases you may want to access Skia directly from native code.  This is possible, but requires version-specific code to accomplish.  You can ship multiple versions of your JNI glue layer, optimized for each Android version (or even platform), and load the right one during your app startup on the Java side.

This is, of course, not very portable, robust, or guaranteed to continue to work by Google, but it’s possible.  There are some very rough hacks in the test app, but for the most part it demonstrates that this approach can work fine.

Next up, I’ll probably blog about porting issues for large native apps, including library compatibility, Bionic, and integrating into a non-ant-based build system.

WebGL introduces two interesting concepts that I think have application outside of WebGL, the CanvasArrayBuffer and CanvasArray WebGLArrayBuffer and WebGLArray. This is all subject to change, of course, though this is what the current Gecko (and others’) implementation looks like. In particular, the Canvas prefix in the names might change soon.

Edit 12/2: these types have changed names; they now have a WebGL prefix instead of a Canvas prefix.

It became clear that pure JS arrays are not a useful way of shoveling around lots of 3D data; their very flexibility makes them impractical for performance-critical uses. In particular, WebGL often wants to deal with arrays of a specific type — an array of integers, an array of floats, etc. Even more complicated is the need to manage multiple types within a single memory region; for performance, it’s often preferable to allocate one chunk of video memory, and place coordinates, colors, and other types in there, replacing them as necessary.

There are two portions to the solution: the WebGLArrayBuffer and a set of typed WebGLArray views. A WebGLArrayBuffer represents chunk of data. It can be allocated with a size in bytes, but it can’t be accessed in any way. To actually manipulate the data inside a WebGLArrayBuffer, a WebGLArray has to be created that references it. An example:

var buf = new WebGLArrayBuffer(3*4);
var floats = new WebGLFloatArray(buf);
floats[0] = 12.3;
floats[1] = 23.4;
floats[2] = 34.5;

The above chunk of code allocates a 12-byte WebGLArrayBuffer, and then creates a float-typed view onto the buffer which can then be manipulated (almost) like a normal array. Of course, the above is cumbersome to write, so there are shorthands that will allocate a WebGLArrayBuffer, and optionally fill it with data from a JS array:

var f1 = new WebGLFloatArray(3);
var f2 = new WebGLFloatArray([12.3, 23.4, 34.5]);

The size of each WebGLArrayBuffer is fixed; there is currently no way to change its size once allocated.

Multiple WebGLArrays can reference the same WebGLArrayBuffer. For example:

var buf = new WebGLArrayBuffer(12*3*4+12*4);
var points = new WebGLFloatArray(buf, 0, 12*3);
var colors = new WebGLUnsignedByteArray(buf, 12*3*4, 12*4);

This creates a buffer of 192 bytes, which is enough room for 12 3-coordinate float points followed by 12 RGBA colors, with each component represented as an unsigned byte. The arguments to the WebGLArray constructors are the offset from the start of the buffer (in bytes), and the length (in elements). The offset must always be a multiple of the element size (to preserve alignment), and the buffer must obviously be large enough for the given offset and length. If length is not given, the length is assumed to be “from offset until the end of the array buffer”; that value must be a multiple of the element size. If offset is not given, it’s assumed to be zero.

For extra complex use cases, WebGLArrays can reference overlapping regions of a WebGLArrayBuffer:

var buf = new WebGLArrayBuffer(192); // same value from above
var points = new WebGLFloatArray(buf);
var colors = new WebGLUnsignedByteArray(buf);
points[0] = 12.3;
points[1] = 23.4;
points[2] = 34.5;
colors[12] = 0xff;
colors[13] = 0xaa;
colors[14] = 0x00;
colors[15] = 0x00;

In the buffer, this writes 3 float values followed by 4 byte values. You’ll note that this use is significantly more complex, and requires the user to keep track of the current position in terms of whatever element they’re modifying (thus setting array elements 12, 13, 14, and 15 for the color).

If an attempt is made to store data in a WebGLArray that doesn’t fit within the right type, a C-style cast is performed. If the data is an entirely wrong type (e.g. trying to store a string or an object), Gecko currently throws an exception, but this might become a silent 0 or similar in the future.

Where does this fit in WebGL? Any API function that needs an array of data takes a WebGLArrayBuffer. This avoids placing costly JS array type conversion in a potential critical performance path, and simplifies a number of aspects of the API. So, VBOs, texture data (if not loaded from a DOM image element or from a CanvasImageData object), index array, etc. all use WebGLArrayBuffers/WebGLArrays for pulling data in and out. WebGLArrays also help manage memory usage — an array of byte color data now takes up exactly as much memory as needed, instead of getting expanded out to 4 bytes. Also, critically, floating point data can be stored as 32-bit single-precision floats instead of 64-bit doubles, taking up half as much space when the underlying graphics system can’t support 64-bit values.

This API is overall lacking in developer niceties, since the focus was on providing the necessary functionality. Higher level wrappers can be written in JS to simplify usage. In addition, by keeping it as bare-bones as it is, it allows for fast implementation on native hardware via the JITs in all the current-generation JS engines. The web currently fudges around the problem of binary data by passing it around either in strings (because JS strings are UCS2, therefore all 8-bit elements are valid, but with a performance and memeory cost), or often encoding as base64 (again going back to strings). I can see this type of dense/native type access being useful for both the File and WebSockets APIs as a way to exchange and deal with binary data.

Starting with Firefox 3.6, only well-known components shipped with Firefox will be loaded from the application components directory.  Any other components (both binary and script) will be ignored.  This work might also be backported to the 3.5 branch, but that decision has not been made yet.  There are a number of reasons why this decision was made.

Firefox, through Gecko, has always had a flexible component-based architecture.  In particular, creating binary components to interface with the OS or with other applications is fairly straightforward, though ultimately dangerous.  Binary components have full access to the application and OS, and so can impact stability, security, and performance.  What’s worse, in a binary component, the line between supported/frozen and completely unfrozen internal Gecko interfaces is blurred, making it easy to create a binary component that works well against one very specific version of Firefox (potentially as specific as a minor security release), but causes serious problems with any other version.

As we’ve seen the popularity of Firefox increase, more and more binary components have been written to interface between Firefox and other applications.  However, we haven’t provided great guidance about the appropriate way to do so.

The only supported way of adding functionality to Firefox (whether a binary component is required or not) is through an add-on. This has many advantages for users: they can see that additional functionality is installed in the Add-ons Manager, and from there they can easily enable or disable it, as well as check for and receive updates.  We’re working on improving the user experience when third-party addons are installed in system-wide locations.  Also, critically, add-ons include information indicating their compatibility with specific versions of Firefox.  Having this version information allows for safe upgrades, especially when binary components are present.

Currently, third-party applications can drop binary components into the Firefox application’s components directory and expect them to be loaded as part of our normal startup.  This causes a number of problems, not the least of which is the removal of the user control that add-ons provide.  Unfortunately, a number of third-party applications are using this approach to integration, and are currently causing Firefox 3.5 users stability problems.  Many of these components were written for Firefox 3.0, and have not been updated for Firefox 3.5; a situation that we have no way of detecting because of the lack of versioning information on these “bare” components.  Because a number of internal interfaces changed between the two versions, this leads to crashes or other problems when these components are used.

In order to simplify future integration with native code, Firefox 3.6 will include support for JSCtypes for add-on developers.  This approach is greatly preferred over writing binary components.  For example, a third-party component that would like to perform some action implemented in native code when an event is received can write the integration pieces in Javascript (capturing the event and so on), and use JSCtypes to make function calls to regular non-XPCOM component native code.  Keeping as much of add-on code in Javascript reduces the impact of internal changes to Firefox, and allows for much easier maintainability.

(If you are a third-party application developer and have questions about integration in Firefox, please contact me — I’d be happy to put you in touch with the right people who can answer questions and provide guidance.)

Since my post on Friday, we landed a few fixes to improve our WebGL implementation and to fix a couple of bugs we discovered on Friday.  I’m looking forward to seeing what people do with WebGL, and how it can be useful on the web right now.  For example, EA/Maxis recently added COLLADA export of Creature designs to their popular game Spore, and they have a Sporepedia where players can see others’ creations.  Right now, those previews are just as images.  With WebGL, they could be fully 3D, even animated.

Over the weekend I’ve put together this example, which uses WebGL to render an exported Spore creature, and let the user rotate the 3D model to view it from different angles.  For those who want to try it out, you’ll need a recent Firefox nightly (one from today, September 21, or newer), and with one preference flipped as described in this post.

I’ll be working to update the very basic “getting started” demos from the GL ES 2 book that I ported to Canvas 3D as well, so that those who are interested in experimenting can have some good basic code to look at.  They’re not updated yet, but they should be in the next day or two.

For those of you on Windows who don’t have an up to date OpenGL driver, or don’t have the possibility of getting one (e.g. many common Intel graphics cards doesn’t have OpenGL drivers), you can enable software rendering by downloading a Windows build of the Mesa software OpenGL implementation.  It won’t be fast, but it should be enough for you to get an idea of what’s going on, and to play with some of the demos.  To use it, download webgl-mesa-751.zip and extract it somewhere on your computer.  It has a single file, OSMESA32.DLL, that you need to tell Firefox where to find:  open up about:config, and set the preference webgl.osmesalib to the path of OSMESA32.DLL.  If you extracted it to “C:\temp”, you would put in “C:\temp\osmesa32.dll” in the pref.  Then, flip webgl.software_rendering to true, and you should be good to go.

(Mac OS X users shouldn’t need to bother with software rendering, since Apple already provides a high quality OpenGL implementation, and Linux users should be ok as long as they have recent OpenGL drivers installed.)

Last night, I checked in some more work from Mark Steele (who’s focusing on the Firefox WebGL implementation), and along with that, enabled WebGL in trunk nightlies.  (Finally!)

If you’re not familiar with WebGL, it’s the evolution of work that Mozilla started a few years ago with experiments called Canvas 3D — essentially a way of accessing OpenGL from within the browser, through the HTML5 canvas tag.  Earlier this year, we started a standardization effort within the Khronos Group which oversees OpenGL, and have made very rapid progress.

Along with the Firefox implementation, a WebGL implementation landed in WebKit fairly recently.  All of these implementations are going to have some interoperability issues for the next little while, as the spec is still in flux and we’re tracking it at different rates, but will hopefully start to stabilize over the next few months.

If you’d like to experiment with WebGL with a trunk nightly build (starting from Friday, September 18th), all you have to do is flip a pref: load about:config, search for “webgl“, and double-click “webgl.enabled_for_all_sites” to change the value from false to true.  You’ll currently have the most luck on MacOS X machines or Windows machines with up-to-date OpenGL drivers.

We still have some ways to go, as there are issues in shader security and portability, not to mention figuring out what to do on platforms where OpenGL is not available.  (The latter is an interesting problem; we’re trying to ensure that the API can be implementable on top of a non-GL native 3D API, such as Direct3D, so that might be one option.)  But progress is being quickly made.

When paired with high-performance JavaScript, such as what we’ve seen come from both Firefox and other browsers, should allow for some exciting fully 3D-enabled web applications.  We’ll have some simple demos linked for you soon, both here and on Mark’s blog.

Edit: new post up, with some samples!