One of the problems with Sylvester is that while it's fully featured (arbitrary NxN matrices and vectors can be created and manipulated), it suffers in performance because of it.  Since this is such a crucial part of a successful WebGL program, I've put together a small package that I'm calling mjs.

mjs is designed around speed and simplicity.  For example, it doesn't attempt to stuff vectors and matrices into JavaScript objects.  Because the language offers no operator overloading, there's very little benefit in treating these types as discrete objects, and lots of performance and memory usage downsides.  Instead, it provides a set of functions for performing operations on vectors and matrices, which can be any array-like object.  For any function that returns a vector or matrix, an existing array can be passed in to take the result, or the function can create a new one.  Array reuse ends up being important because of the potential for expensive garbage collection churn eating away at performance.

Here's a sample of the API:

var r = M4x4.rotate(Math.PI/2, V3.$(0, 1, 0),  M4x4.I);

Note that V3.$ and M4x4.$ are shorthand for creating a new V3 or M4x4 (I wanted to use V3() and M4x4(), but that didn't work out too well since functions have a length property).  However, because all they return are just new array-like objects, you could also write:

var r = M4x4.rotate(Math.PI/2, [0, 1, 0], M4x4.I);

If the WebGL types are available, those will be used for newly created vectors/matrices.  They are a significant performance boost especially for repeated operations; but for specifying one-off vectors such as the above, literal array syntax is fine.

The rotate function internally makes a rotation matrix, and then multiplies it by the given matrix.  So the above could also be written as:

var rotation = M4x4.makeRotate(Math.PI/2, [0, 1, 0]);
var r = M4x4.mul(M4x4.I, rotation);

(The last line being redundant given that we're multiplying by the identity matrix.)

All methods that return a vector or matrix take an optional final argument, that of an existing object to reuse.  For example:

var m0 = M4x4.$();
r = M4x4.mul(someMatrixA, someMatrixB, m0);
// r == m0, so the assignment isn't necessary, but it's handy for chaining
// .... do something with r ...
r = M4x4.mul(someMatrixB, someMatrixC, m0);
// r == m0 still
// ... do something else with new results ...

Without allocating any additional temporary objects.

As mentioned before, one of the goals of mjs is performance.  Matrix multiplication is one of the most common tasks, so here are some numbers comparing mjs, Sylvester, and native C code.  This was run on a Core i7 desktop using a local build of Spidermonkey, which included one patch that's about to go into the tree that fixes the no-reuse tracing case.  (Without it, the no-reuse tracing case is much larger because it's never actually jitted.) The test is simple: it multiplies two matrices together in a loop 1,000,000 times.

(I also have numbers for MSVC without the SSE2 compile flag, but the numbers vary greatly depending on whether the values eventually go to infinity or not; if the values end up trending towards 0, the non-SSE2 code tends to win at around 52ms vs. 71ms; if the values trend to infinity, the non-SSE2 code takes around 11,000ms!)

Those numbers are pretty encouraging -- having native code be only 2x as slow for something like this is pretty nice to see. Granted, this is only a very isolated test, and I'm sure there are some tricks to optimizing the native code case (it's currently just a fully unrolled set of multiplies and adds). The "no JIT" case is less nice, but I'm sure that our Jaegermonkey folks will be all over this testcase (right, guys?). In any case, ideally most WebGL rendering loops will be fully traced in Firefox, so it would be less of an issue.

mjs is still very much a work in progress; it's missing a test suite and a whole bunch of features. You can find it hosted at Google Code, at webgl-mjs. (Side note: I couldn't just call the project mjs because a project called mjs was abandoned on Sourceforget 5 years ago, and Google Code complained.) There's also some documentation, viewable online here.

Bugs and contributions welcome!

Mouse events sort of work, toplevel windows sort of work, keyboard doesn't work yet but shouldn't be hard to hook up.  This is running in an emulator at the moment for ease of debugging, but it's working just fine on physical hardware as well.

You'll note that this is the full Firefox interface, and not the Fennec/Firefox Mobile UI; we're testing with the full interface because it's significantly more complex than the mobile UI and stresses Gecko much more.  So, if the full UI works, then Fennec should work fine as well.  Given the interest in Android on netbook and tablet devices, an updated version of the full Firefox UI might find a home on some of these.  Android has been pretty great to work with so far; it's a bit unusual platform for us due to its Java core, but with the NDK we're able to bridge things together without many problems.

We're still a ways  to go before any kind of usable alpha release, but we're certainly one step closer.  We'll also be able to accelerate our progress now that we have some of the basic scaffolding in place.  I know I'm looking forward to running Fennec on my Droid, and there are tons of Android devices coming out that should be great platforms for Fennec.

We're actively working through remaining warts and edge cases (and they are a lot of them!).  Take a look at the official Khronos WebGL landing page and Arun's blog post for more information, including where to go to sign up for the public mailing list and for a set of resources about WebGL.

We've had support for WebGL in Firefox desktop nightlies for a few weeks now, but soon I hope to have it enabled on Firefox for mobile devices (code name Fennec) as well.  Jay demonstrated some simple demos working on Nokia's N900 at the ARM techcon3 today a few weeks ago (took a while to get the video done!), and I've grabbed a quick video showing things in action.

This should be showing up in Firefox for mobile devices nightly builds, so we'll see some more interesting things running on phones soon!

Today's Android hacking notes

Bionic doesn't support dlopen(NULL, ...) to get access to a handle to the currently running program.  In fact, it really doesn't support it, and crashes should you try to do that perfectly normal thing.  Should you really need that, though, you can potentially hack it as seen here.  I ended up not needing to do this, but it's there should we need it in the future.

Also, to get all the happy debugging symbols, you can set gdb's solib-search-path to the symbol-rich versions of the binaries built as part of an android build, at source/out/target/product/generic/symbols/system/lib:source/out/target/product/generic/symbols/system/bin .. this will let you debug all the way from initial program entry in the linker down into whatever libraries you call.  If you don't have symbols, gdb will often lie about $pc when using gdbservers.

The feel of the OS is pretty nice, although some things are more sluggish than they really should be on an OMAP3 device.  Stuart keeps telling me that Fennec has smoother panning in the browser, and I think he's right.  It's not a deal breaker though; I find myself using the browser a lot to do all sorts of things that I never would have considered on my Blackberry (because, wow, the web browser situation there is awful), but that was a frustrating experience on my iPod Touch as well.  I've spent a while "browsing the web" on my phone, which I've never been able to say I've done before.

But, it's still a phone, and while the voice portion isn't all that important to me, the overall communication package is.  Coming from a Blackberry, the overall messaging situation on the Droid is simply horrible.  Email, whether Exchange or IMAP, is a disaster.  The email client seems designed for simple "lol r u there" type of messages, and even the message lists don't seem intended for people who get more than 5 messages a day -- turning a message list into landscape mode is worthless as you only get to see about 3-4 messages in the list (same view as in portrait mode, just along the much smaller axis of the display), no IMAP IDLE support etc. are all very strange on a top-end phone.  Exchange support works ok for Calendar sync, but for email sync it would only download the first 1000 bytes or so of a message, including headers; this meant that I often only got to see the first sentence or two of an email.  I don't know whether this is a problem with the Droid or our Zimbra Exchange connector, but switching to IMAP for work mail fixed that problem.

An recently-released version of the open-source K9 Email Client that works on the Droid resolves many of these issues, though it needs some polish.  I might write some code there, since it's close to becoming a pretty good email solution.

The Gtalk client is probably in worse shape than email.  It's almost as if Google entirely ignored Gtalk on this device (and I can't believe that would be Verizon's fault, since things like Google Voice work just fine).  First, it's in general buggy -- it's crashed on me multiple times, often freezes when returning to it from another app (after clicking a link to the browser, for example), and often shows contacts as offline with a big red message despite the contact clearly having a green dot next to their name and responding to my messages.

In the browser and in other apps, you can share a web page with someone using a "Share with" button.  The list you're presented is conspicuously missing Gtalk, despite having Facebook, Email, Messaging (SMS) and a random Twitter client I installed on there.  What gives?  All of these features are available on the Blackberry; I'm not sure if it was RIM that did the Gtalk app there, but can we get whoever it was to rewrite the Android one?

One of the best things about the Blackberry is the unified messaging; there's a single view where I can go to see all my emails, my gtalk conversations, my SMS messages, app updates, and whatever else.  No such thing exists on Android.  The closest thing is the notification bar, which requires a swipe down to use, and then only shows things that have come in since the last time you looked.  I'd prefer a more time-based list that contains both old and unread items.  Sounds like the Sony-Ericsson X10 might be doing some interesting things there, and I hope that someone figures out how to create an app like this.  What it comes down to is that anything to do with communication is faster and simpler on my Blackberry, which is really strange; you'd think Google would have spent some time working this out, as everything else about the device is far superior to my 8900.  I understand that more "enterprise oriented" customers (which apparently means those that like to use email a lot?) aren't necessarily the target market here, but they could've really attacked that market with some simple work that wouldn't have affected anything else.

The good news is that all of these are fairly straightforward software issues.  The hardware is solid, and Google has shown that they'll do frequent upgrades of the OS.  Given that the Droid is a "Google Experience" device, those updates should find their way to the device quickly.  Some fixes, combined with getting Firefox Mobile on the Droid and other Android devices, will make this a great phone.

At boot time, a special instance of the Java VM is launched, called the Zygote.  This process loads a bunch of the core Java classes and performs initial processing of them, making it possible to avoid this step for each app launch.  Once the initial work is done, the process listens to a socket and waits for requests.

To launch an app using the Zygote process, a command-line tool called "dvz" can be used.  It sends its arguments to the Zygote, which will fork and then start executing the main method in a given class.

So, we have these steps, dealing with the Zygote process:

• the zygote process is executed at system boot; it does initialization, and then runs a select() loop listening to requests
• the zygote process is sent a message, which includes the start args, which will include the class name for main launch
• the zygote process reads connection args in ZygoteConnection.runOnce
• the zygote process forks in ZygoteConnection.runOnce (Zygote.forkAndSpecialize, native)
• ZygoteConnection.handleChildProc is called, which does some cleanup and eventually throws a MethodAndArgsCaller (via ZygoteInit.invokeStaticMain)
• ... which takes us all the way back to ZygoteInit.main, which catches that trampoline exception, and calls run()

But, that's not fully how a new Android Activity is started.  It's a bit of a roundabout process.  To launch a new Activity, the ActivityManagerService is notified with an activity start request, including things like the name/class/etc. of the activity.  It puts that information in a list of activities to run.  Then, a new process is started with the main from ActivityThread.  This new process then contacts the ActivityManagerService and asks, essentially, "what app am I?".  The service then gives it the name of its activity class and other info, which is then loaded, and a message is enqeued on the main thread to instantiate the new activity and send it an onCreate() message.

This is interesting because it means that apps are not launched directly, but instead somewhat indirectly through specializing a generic "Activity" process for a specific activity.  A side effect of this is that I couldn't find a way to actually register an app with the ActivityManagerService if it wasn't launched by it.  So, to be a full Android app, you have to go through this normal startup process.

JNI Bridging

One of the difficulties in porting Gecko to Android is that the Android platform is built around Java, whereas Gecko is very much all native C/C++.  However, there is a fairly good native bridge layer, JNI, which is fairly heavily optimized by Dalvik.  So, the simplest way to connect these two is to write a shell app in Java, which bridges events, messages, paint requests, etc. to the native code for handling.

As a proof of concept of this, I wrote a simple test app.  It's fairly straightforward, with a few wrinkles.  Most of this stuff can be done with the stock Android SDK and NDK -- except painting.  The only API that the NDK exposes for graphics is OpenGL ES.  This is fine, but in some cases you may want to access Skia directly from native code.  This is possible, but requires version-specific code to accomplish.  You can ship multiple versions of your JNI glue layer, optimized for each Android version (or even platform), and load the right one during your app startup on the Java side.

This is, of course, not very portable, robust, or guaranteed to continue to work by Google, but it's possible.  There are some very rough hacks in the test app, but for the most part it demonstrates that this approach can work fine.

Next up, I'll probably blog about porting issues for large native apps, including library compatibility, Bionic, and integrating into a non-ant-based build system.

Edit 12/2: these types have changed names; they now have a WebGL prefix instead of a Canvas prefix.

It became clear that pure JS arrays are not a useful way of shoveling around lots of 3D data; their very flexibility makes them impractical for performance-critical uses. In particular, WebGL often wants to deal with arrays of a specific type -- an array of integers, an array of floats, etc. Even more complicated is the need to manage multiple types within a single memory region; for performance, it's often preferable to allocate one chunk of video memory, and place coordinates, colors, and other types in there, replacing them as necessary.

There are two portions to the solution: the WebGLArrayBuffer and a set of typed WebGLArray views. A WebGLArrayBuffer represents chunk of data. It can be allocated with a size in bytes, but it can't be accessed in any way. To actually manipulate the data inside a WebGLArrayBuffer, a WebGLArray has to be created that references it. An example:

var buf = new WebGLArrayBuffer(3*4);
var floats = new WebGLFloatArray(buf);
floats[0] = 12.3;
floats[1] = 23.4;
floats[2] = 34.5;

The above chunk of code allocates a 12-byte WebGLArrayBuffer, and then creates a float-typed view onto the buffer which can then be manipulated (almost) like a normal array. Of course, the above is cumbersome to write, so there are shorthands that will allocate a WebGLArrayBuffer, and optionally fill it with data from a JS array:

var f1 = new WebGLFloatArray(3);
var f2 = new WebGLFloatArray([12.3, 23.4, 34.5]);

The size of each WebGLArrayBuffer is fixed; there is currently no way to change its size once allocated.

Multiple WebGLArrays can reference the same WebGLArrayBuffer. For example:

var buf = new WebGLArrayBuffer(12*3*4+12*4);
var points = new WebGLFloatArray(buf, 0, 12*3);
var colors = new WebGLUnsignedByteArray(buf, 12*3*4, 12*4);

This creates a buffer of 192 bytes, which is enough room for 12 3-coordinate float points followed by 12 RGBA colors, with each component represented as an unsigned byte. The arguments to the WebGLArray constructors are the offset from the start of the buffer (in bytes), and the length (in elements). The offset must always be a multiple of the element size (to preserve alignment), and the buffer must obviously be large enough for the given offset and length. If length is not given, the length is assumed to be "from offset until the end of the array buffer"; that value must be a multiple of the element size. If offset is not given, it's assumed to be zero.

For extra complex use cases, WebGLArrays can reference overlapping regions of a WebGLArrayBuffer:

var buf = new WebGLArrayBuffer(192); // same value from above
var points = new WebGLFloatArray(buf);
var colors = new WebGLUnsignedByteArray(buf);
points[0] = 12.3;
points[1] = 23.4;
points[2] = 34.5;
colors[12] = 0xff;
colors[13] = 0xaa;
colors[14] = 0x00;
colors[15] = 0x00;

In the buffer, this writes 3 float values followed by 4 byte values. You'll note that this use is significantly more complex, and requires the user to keep track of the current position in terms of whatever element they're modifying (thus setting array elements 12, 13, 14, and 15 for the color).

If an attempt is made to store data in a WebGLArray that doesn't fit within the right type, a C-style cast is performed. If the data is an entirely wrong type (e.g. trying to store a string or an object), Gecko currently throws an exception, but this might become a silent 0 or similar in the future.

Where does this fit in WebGL? Any API function that needs an array of data takes a WebGLArrayBuffer. This avoids placing costly JS array type conversion in a potential critical performance path, and simplifies a number of aspects of the API. So, VBOs, texture data (if not loaded from a DOM image element or from a CanvasImageData object), index array, etc. all use WebGLArrayBuffers/WebGLArrays for pulling data in and out. WebGLArrays also help manage memory usage -- an array of byte color data now takes up exactly as much memory as needed, instead of getting expanded out to 4 bytes. Also, critically, floating point data can be stored as 32-bit single-precision floats instead of 64-bit doubles, taking up half as much space when the underlying graphics system can't support 64-bit values.

This API is overall lacking in developer niceties, since the focus was on providing the necessary functionality. Higher level wrappers can be written in JS to simplify usage. In addition, by keeping it as bare-bones as it is, it allows for fast implementation on native hardware via the JITs in all the current-generation JS engines. The web currently fudges around the problem of binary data by passing it around either in strings (because JS strings are UCS2, therefore all 8-bit elements are valid, but with a performance and memeory cost), or often encoding as base64 (again going back to strings). I can see this type of dense/native type access being useful for both the File and WebSockets APIs as a way to exchange and deal with binary data.