One of the issues with the Gecko port to Android is that, early on, I used some internals to tie in to the Android graphics system from native code.  This worked fine, but it complicated the build: you needed to pull in a bunch of headers and some libraries from the actual Android source to be able to complete a build.

The solution for this was pretty easy: move to OpenGL for rendering.  However, there are some interesting quirks here.  I’m targeting Android 2.x only: specifically the Motorola Droid, HTC Nexus One, and a NVIDIA Tegra 250 devkit I have here.  For this initial step, all I need is to just draw a textured quad.  We’ve got full OpenGL compositing, rendering, fancy video decoding and all that stuff coming later, but for now we’re just hooking into our software rasterizer, uploading the result as a texture, and drawing a textured quad.  Easy, right?  Here are some random issues I ran into while doing this over the past day or two.

First Attempt: OpenGL ES 1.1

Well, there are two wrinkles.  First, Cairo’s software rasterizer uses a 32-bit ARGB pixel format and layout.  In little-endian per-byte terms, that’s B G R A.  OpenGL ES supports A R G B.  There is an EXT_bgra extension that adds support for GL_BGRA as another byte format, and this extension is one that’s potentially available on GL ES.  The second wrinkle is that this quad is display-sized, so the texture is display sized; it’s not going to be power-of-two dimensions.  While OpenGL ES 2.0 supports non power of two textures in the base (with some limitations, which are not relevant for my use case), ES 1.1 does not, and I figured given that all I was drawing is a textured quad, I may as well use ES 1.1.

Unfortunately, the tree devices I mentioned above support different combinations of these.  The NVIDIA device supports both EXT_bgra and ARB_texture_non_power_of_two.  This is perfect; no workarounds are needed here, though for some reason it doesn’t like TexSubImage2D with BGRA data, but that’s not a big deal.  The Droid (OMAP3, PowerVR SGX) supports EXT_texture_format_BGRA8888 (note: different name, similar functionality), so that’s good, but it doesn’t support non power of two textures with ES1.1.  The Nexus One, on the other hand, supports neither BGRA nor NPOT textures.

I was about to start using OES_draw_texture as well, because that seemed like a potentially faster way to get what I want to happen — but the lack of BGRA support on the Nexus One made me turn to ES2, where I can do the RGBA->BGRA swizzle in the fragment shader.

Undefined Symbols in GLESv2 Import Library

More fun!  The Android r3 NDK includes GLESv2 support, yay!  The bad news is that libGLESv2.so includes an external reference to _ZN7android33egl_get_image_for_current_contextEPv (android::egl_get_image_for_current_context), which means you’ll get linker errors (or at least undefined symbol errors) if you try to link anything that’s not a shared library.  Conveniently, that’s what you need to produce with the NDK anyway, but if you have some helper command line tools along the way, they’ll fail.  The solution is to add -Wl,–allow-shlib-undefined to your binary compile/link step.

After that, this was fairly straightforward, though the SDK only grudgingly allows you to specify the necessary EGL tokens for GLES2; the code samples in the NDK all just provide explicit integer values for them inside the code.

Choosing an EGLConfig

This applies to both OpenGL ES 1 and OpenGL ES 2 on Android.  When creating an EGLSurface for a SurfaceView (take a look at how GLSurfaceView does it for the details), you have to get an EGLConfig that has an exact match for the number of red/green/blue/alpha bits as your surface.  There’s a format parameter to surfaceChanged that’s supposed to tell you the format of the surface.  However, it seems to always show up as ‘-1′, which according to PixelFormat.java, is “OPAQUE”.  That’s not very helpful.  Reading GLSurfaceView, it can show up as -2, which is TRANSPARENT.  So — you have to assume that if you have an OPAQUE surface format, it’ll be 5650, and if you have a TRANSPARENT format it’ll be 8888.  This is pretty silly, as there are PixelFormat constants for handy things like RGBA_8888, RGB_565, RGB_888, RGBA_5551, etc.  Why doesn’t SurfaceView send the actual format down?

The devices that I have seem pretty consistent at least with 565 for OPAQUE, so it works OK, but it’s not pretty, and will likely blow up spectacularly if anyone introduces, say, a large-display Android device that uses 24bpp color.

Another config issue is that some GPUs have some odd requirements for getting the most preformance; for example, as discovered via searching, the PowerVR SGX in the Droid really wants 24-bit depth, as it’s faster than 0 and 32.  The Tegra, on the other hand, doesn’t have 24bpp depth at all, only 0 or 16 (and I don’t think it cares one way or the other).  Not sure whether the GPU in the Nexus One cares or has a preference.  So, you have to search for a 24-bit depth config first, use it if it’s found, and then try 0 if not found.  I suppose an alternate approach might be to search for 16-bit depth, but that might give you 32-bit if that happens to be supported somewhere.

At the end of all of this though, I have an app that uses OpenGL ES 2 on three different Android devices (with three different GPUs).

A while ago, Rob Arnold wrote a simple python caching proxy server for use with our Talos tests — the idea was that you’d run your test once against the proxy server in “record” mode, and then after that you can use the server for consistent local playback.

I was giving some WebGL demos recently, and needed a way to have all the content from the web-hosted demos locally.  As anyone who’s tried to create a local cache of any “Web 2.0″ app knows, it’s painful, given all the server requests, XMLHttpRequests, etc. that go on.  However, with the proxy server, this was actually ridiculously easy.

You can grab the proxy server here — it still lives in Mozilla CVS — proxyserver.py.  It works fine on Win32, OS X, and Linux.  On Win32, the python that’s part of mozilla-build works well.  Run it like this:

python proxyserver.py

and then in Firefox’s proxy settings (or the system proxy settings), set your HTTP proxy to localhost:8000.  You can change the port via a command line option.  Then, visit all the pages/sites that you want cached (don’t forget to shift-reload or clear Firefox’s cache beforehand to ensure that Firefox actually goes out to the network!).  After you’ve got everything going, restart the proxy server in local-only mode:

python proxyserver.py -l

… and make sure that your demos work.  You can also run without -l live, especially if you will have a network connection (even a slow one) to give you the option of going out to the network if necessary.  Also, if you want to copy the proxy cache to another machine, just copy proxy_cache.db that gets created in the same directory as proxyserver.py.

The proxy server currently supports HEAD and GET requests.  It doesn’t support POST, so if you have something that depends on POST, you’re out of luck.  It wouldn’t be too hard to add though; patches accepted if someone wants to tackle that.

With the proxy server, I was able to give a bunch of demos that made heavy use of XHR, including some that loaded video, without having to rely on a network or spend time downloading and fixing up URLs.  It really made demo prep much easier.

One common thread running through the many different and interesting WebGL projects out there is that they all need to do vector and matrix math, do it quickly, and do it in JavaScript.  To date, developers have either rolled their own, or they’ve used Sylvester, a fairly featureful vector and matrix JavaScript library.

One of the problems with Sylvester is that while it’s fully featured (arbitrary NxN matrices and vectors can be created and manipulated), it suffers in performance because of it.  Since this is such a crucial part of a successful WebGL program, I’ve put together a small package that I’m calling mjs.

mjs is designed around speed and simplicity.  For example, it doesn’t attempt to stuff vectors and matrices into JavaScript objects.  Because the language offers no operator overloading, there’s very little benefit in treating these types as discrete objects, and lots of performance and memory usage downsides.  Instead, it provides a set of functions for performing operations on vectors and matrices, which can be any array-like object.  For any function that returns a vector or matrix, an existing array can be passed in to take the result, or the function can create a new one.  Array reuse ends up being important because of the potential for expensive garbage collection churn eating away at performance.

Here’s a sample of the API:

var r = M4x4.rotate(Math.PI/2, V3.$(0, 1, 0),  M4x4.I);

Note that V3.$ and M4x4.$ are shorthand for creating a new V3 or M4x4 (I wanted to use V3() and M4x4(), but that didn’t work out too well since functions have a length property).  However, because all they return are just new array-like objects, you could also write:

var r = M4x4.rotate(Math.PI/2, [0, 1, 0], M4x4.I);

If the WebGL types are available, those will be used for newly created vectors/matrices.  They are a significant performance boost especially for repeated operations; but for specifying one-off vectors such as the above, literal array syntax is fine.

The rotate function internally makes a rotation matrix, and then multiplies it by the given matrix.  So the above could also be written as:

var rotation = M4x4.makeRotate(Math.PI/2, [0, 1, 0]);
var r = M4x4.mul(M4x4.I, rotation);

(The last line being redundant given that we’re multiplying by the identity matrix.)

All methods that return a vector or matrix take an optional final argument, that of an existing object to reuse.  For example:

var m0 = M4x4.$();
r = M4x4.mul(someMatrixA, someMatrixB, m0);
// r == m0, so the assignment isn't necessary, but it's handy for chaining
// .... do something with r ...
r = M4x4.mul(someMatrixB, someMatrixC, m0);
// r == m0 still
// ... do something else with new results ...

Without allocating any additional temporary objects.

As mentioned before, one of the goals of mjs is performance.  Matrix multiplication is one of the most common tasks, so here are some numbers comparing mjs, Sylvester, and native C code.  This was run on a Core i7 desktop using a local build of Spidermonkey, which included one patch that’s about to go into the tree that fixes the no-reuse tracing case.  (Without it, the no-reuse tracing case is much larger because it’s never actually jitted.) The test is simple: it multiplies two matrices together in a loop 1,000,000 times.

(I also have numbers for MSVC without the SSE2 compile flag, but the numbers vary greatly depending on whether the values eventually go to infinity or not; if the values end up trending towards 0, the non-SSE2 code tends to win at around 52ms vs. 71ms; if the values trend to infinity, the non-SSE2 code takes around 11,000ms!)

Those numbers are pretty encouraging — having native code be only 2x as slow for something like this is pretty nice to see. Granted, this is only a very isolated test, and I’m sure there are some tricks to optimizing the native code case (it’s currently just a fully unrolled set of multiplies and adds). The “no JIT” case is less nice, but I’m sure that our Jaegermonkey folks will be all over this testcase (right, guys?). In any case, ideally most WebGL rendering loops will be fully traced in Firefox, so it would be less of an issue.

mjs is still very much a work in progress; it’s missing a test suite and a whole bunch of features. You can find it hosted at Google Code, at webgl-mjs. (Side note: I couldn’t just call the project mjs because a project called mjs was abandoned on Sourceforget 5 years ago, and Google Code complained.) There’s also some documentation, viewable online here.

Bugs and contributions welcome!

It’s been a while since I posted a progress update (or really any blog post, ahem), but porting Firefox/Fennec to Android is progressing at a good clip. After working out a few kinks (and setting the all-important “you’re allowed to touch the network” permission), I just got our first page load:

Mouse events sort of work, toplevel windows sort of work, keyboard doesn’t work yet but shouldn’t be hard to hook up.  This is running in an emulator at the moment for ease of debugging, but it’s working just fine on physical hardware as well.

You’ll note that this is the full Firefox interface, and not the Fennec/Firefox Mobile UI; we’re testing with the full interface because it’s significantly more complex than the mobile UI and stresses Gecko much more.  So, if the full UI works, then Fennec should work fine as well.  Given the interest in Android on netbook and tablet devices, an updated version of the full Firefox UI might find a home on some of these.  Android has been pretty great to work with so far; it’s a bit unusual platform for us due to its Java core, but with the NDK we’re able to bridge things together without many problems.

We’re still a ways  to go before any kind of usable alpha release, but we’re certainly one step closer.  We’ll also be able to accelerate our progress now that we have some of the basic scaffolding in place.  I know I’m looking forward to running Fennec on my Droid, and there are tons of Android devices coming out that should be great platforms for Fennec.

I’m pretty excited to have the WebGL draft spec available for review and comments today.  There’s still plenty of time for feedback, but we’re far enough along to be able to solicit meaningful feedback.  There are multiple implementations, which is a much better state than the early Canvas 3D work where things only worked during a full moon after saying “OpenGL” three times backwards into your monitor!

We’re actively working through remaining warts and edge cases (and they are a lot of them!).  Take a look at the official Khronos WebGL landing page and Arun’s blog post for more information, including where to go to sign up for the public mailing list and for a set of resources about WebGL.