Let’s get one thing out of the way up front.  Today’s web browser is in many ways acting like a miniature full operating system.  It runs multiple applications at once (whether in multiple windows or tabs).  It might do a lot of background processing.  It can work with large data sets, for example large images on flickr or large spreadsheets on Google Documents.  But, the final memory usage number that the user sees when they open up the Task Manager or Process Viewer is the aggregate memory usage of the entire system.  So, the goal of improving our memory usage is not to get that number to the lowest possible — doing that would be an unacceptable tradeoff in performance for users — but instead to understand where memory is being used, and then use that data to improve in those areas as possible.

One comment that I’ve heard is that Firefox 3.6 seems to use more memory than Firefox 3.0.  My initial tests show this to not be true; specifically, I looked at the “Private Bytes” value in the Windows 7 task manager shortly after startup with about:blank, and also after opening a number of tabs (gmail, google docs, cnn.com, front page of the boston.com big picture blog, engadget, and a few others).  Here are the results of a typical run:

The next question is figuring out where all the memory goes. I’ve been adding some instrumentation to Firefox to figure out in more detail where memory is being used. For a sample run with the multiple tabs shown above, here’s what some of that reporter data looks like:

Or, graphically:

There’s some overlap in those numbers — for example, the jemalloc commit size is a subset of the Windows Private Bytes number, and most of the rest is a subset of the jemalloc commit size. Likewise, the uncompressed images number is a subset of the Win32 graphics surfaces number; that is, ~53MB is in use by win32 surfaces, and almost all of that is due to live images in pages (remember that we’ve got some image heavy sites in that tab set, including the Big Picture blog which has around 10-11 large images on it… those should account for about 20-25MB just by themselves).

There are some things that don’t make sense in the above, which mean that my instrumentation isn’t quite correct… for example, adding up the JS numbers, the Images number, and the PresShell arenas number brings us beyond the jemalloc commit size, which shouldn’t be true. However, some of the Image data is allocated by GDI, likely bypassing jemalloc, so we have to take that into account. There’s also some large other chunks of the browser that have yet to be instrumented, which should provide additional insight.

Two initial observations: one, keeping images compressed in memory and only decompressing them briefly when we need to draw them is a potential huge memory win. We have the infrastructure and code to do this in place; it was disabled recently while some of the internals changed, and it needs to be reenabled.

Two, the 30MB or so in the “js_malloc Other” bucket is also pretty curious. We need to do some more work to figure out what exactly is in here. (This contains things like data structures for tracking array contents and — potentally a big one — string data.)

I’ll be blogging more as the instrumentation takes shape, and as it gets landed into trunk nightly builds. Much of this information will be visible in about:memory, and eventually we’ll be able to give some per-tab memory information as well.

• We’ve only really tested this on the Motorola Droid and the Nexus One.
• It will likely not eat your phone, but bugs might cause your phone to stop responding, requiring a reboot.
• Memory usage of this build isn’t great — in many ways it’s a debug build, and we haven’t really done a lot of optimization yet.  This could cause some problems with large pages, especially on low memory devices like the Droid.
• You’ll see the app exit and relaunch on first start, as well as on add-on installs; this is a quirk of our install process, and we’re working to get rid of it.
• You can’t open links from other apps using Fennec; we should have this for the next build.
• This build requires Android 2.0 or above, and likely an OpenGL ES 2.0 capable device.
• Edit: This build must be installed to internal memory, not to a SD card.

There also aren’t yet any automated nightly developer builds or automated updates to this build; it’s even more of a pre-nightly build (even earlier than pre-alpha).  But, it’s usable enough that we wanted to get some feedback on it as we continue to develop.

Weave Sync

There is an experimental version of Weave that is compatible with this build: from within Fennec on your phone, open the Mozilla Labs weave page at https://mozillalabs.com/weave/ and click on “Experimental Version”.  (It’s to the right of the big Download Weave now! link — don’t click on that one though, it’s an older version.)  Install the add-on, then you’ll need to restart Fennec (swipe the screen left and then click on the “gear” icon to open the browser tools panel, then click on addons and click the Restart button at the top).  Follow the instructions when Fennec restarts.

Troubleshooting

Should you run into problems, such as the app not responding or just giving you a black screen, you can force it to quit by going into the Android Settings, selecting Applications, selecting Manage Applications, then selecting Fennec, and tapping Force Stop.  (A utility called White Killer, available from the Market, can do the same job with fewer clicks.)  Worst case, uninstalling and reinstalling would clear out your profile and any saved data.

Installation & Feedback

So, now that you’ve read all that, you can download the build here — the easiest way is to download it using your phone’s browser, and then click on it in the downloads list to install it.  If you’re reading this on your desktop, you can scan the QR code here on your phone, or type in the following address in your phone’s browser: bit.ly/fennec-android.  You may need to enable installation of non-Market applications by going to Settings, Applications, and checking “Unknown Sources”.

We’ve created a temporary Google Group for feedback about this pre-alpha build.  In the future we’ll have a more permanent way for user feedback and comments, but for now, please use the group to let us know what you think!

We’re still working on getting the basic blocks in place to where it’s “dogfoodable”, that is, usable by the developers.  The good news is that while the builds are already pretty fast, we’re seeing that we have a lot of headroom for performance… especially for visual things like rendering and panning.  Most of the work we’ve done so far has just been quick work to unblock getting the basic port running; I’m looking forward to being able to dig deeper into a bunch of these issues!

The solution for this was pretty easy: move to OpenGL for rendering.  However, there are some interesting quirks here.  I’m targeting Android 2.x only: specifically the Motorola Droid, HTC Nexus One, and a NVIDIA Tegra 250 devkit I have here.  For this initial step, all I need is to just draw a textured quad.  We’ve got full OpenGL compositing, rendering, fancy video decoding and all that stuff coming later, but for now we’re just hooking into our software rasterizer, uploading the result as a texture, and drawing a textured quad.  Easy, right?  Here are some random issues I ran into while doing this over the past day or two.

First Attempt: OpenGL ES 1.1

Well, there are two wrinkles.  First, Cairo’s software rasterizer uses a 32-bit ARGB pixel format and layout.  In little-endian per-byte terms, that’s B G R A.  OpenGL ES supports A R G B.  There is an EXT_bgra extension that adds support for GL_BGRA as another byte format, and this extension is one that’s potentially available on GL ES.  The second wrinkle is that this quad is display-sized, so the texture is display sized; it’s not going to be power-of-two dimensions.  While OpenGL ES 2.0 supports non power of two textures in the base (with some limitations, which are not relevant for my use case), ES 1.1 does not, and I figured given that all I was drawing is a textured quad, I may as well use ES 1.1.

Unfortunately, the tree devices I mentioned above support different combinations of these.  The NVIDIA device supports both EXT_bgra and ARB_texture_non_power_of_two.  This is perfect; no workarounds are needed here, though for some reason it doesn’t like TexSubImage2D with BGRA data, but that’s not a big deal.  The Droid (OMAP3, PowerVR SGX) supports EXT_texture_format_BGRA8888 (note: different name, similar functionality), so that’s good, but it doesn’t support non power of two textures with ES1.1.  The Nexus One, on the other hand, supports neither BGRA nor NPOT textures.

I was about to start using OES_draw_texture as well, because that seemed like a potentially faster way to get what I want to happen — but the lack of BGRA support on the Nexus One made me turn to ES2, where I can do the RGBA->BGRA swizzle in the fragment shader.

Undefined Symbols in GLESv2 Import Library

More fun!  The Android r3 NDK includes GLESv2 support, yay!  The bad news is that libGLESv2.so includes an external reference to _ZN7android33egl_get_image_for_current_contextEPv (android::egl_get_image_for_current_context), which means you’ll get linker errors (or at least undefined symbol errors) if you try to link anything that’s not a shared library.  Conveniently, that’s what you need to produce with the NDK anyway, but if you have some helper command line tools along the way, they’ll fail.  The solution is to add -Wl,–allow-shlib-undefined to your binary compile/link step.

After that, this was fairly straightforward, though the SDK only grudgingly allows you to specify the necessary EGL tokens for GLES2; the code samples in the NDK all just provide explicit integer values for them inside the code.

Choosing an EGLConfig

This applies to both OpenGL ES 1 and OpenGL ES 2 on Android.  When creating an EGLSurface for a SurfaceView (take a look at how GLSurfaceView does it for the details), you have to get an EGLConfig that has an exact match for the number of red/green/blue/alpha bits as your surface.  There’s a format parameter to surfaceChanged that’s supposed to tell you the format of the surface.  However, it seems to always show up as ‘-1′, which according to PixelFormat.java, is “OPAQUE”.  That’s not very helpful.  Reading GLSurfaceView, it can show up as -2, which is TRANSPARENT.  So — you have to assume that if you have an OPAQUE surface format, it’ll be 5650, and if you have a TRANSPARENT format it’ll be 8888.  This is pretty silly, as there are PixelFormat constants for handy things like RGBA_8888, RGB_565, RGB_888, RGBA_5551, etc.  Why doesn’t SurfaceView send the actual format down?

The devices that I have seem pretty consistent at least with 565 for OPAQUE, so it works OK, but it’s not pretty, and will likely blow up spectacularly if anyone introduces, say, a large-display Android device that uses 24bpp color.

Another config issue is that some GPUs have some odd requirements for getting the most preformance; for example, as discovered via searching, the PowerVR SGX in the Droid really wants 24-bit depth, as it’s faster than 0 and 32.  The Tegra, on the other hand, doesn’t have 24bpp depth at all, only 0 or 16 (and I don’t think it cares one way or the other).  Not sure whether the GPU in the Nexus One cares or has a preference.  So, you have to search for a 24-bit depth config first, use it if it’s found, and then try 0 if not found.  I suppose an alternate approach might be to search for 16-bit depth, but that might give you 32-bit if that happens to be supported somewhere.

At the end of all of this though, I have an app that uses OpenGL ES 2 on three different Android devices (with three different GPUs).

I was giving some WebGL demos recently, and needed a way to have all the content from the web-hosted demos locally.  As anyone who’s tried to create a local cache of any “Web 2.0″ app knows, it’s painful, given all the server requests, XMLHttpRequests, etc. that go on.  However, with the proxy server, this was actually ridiculously easy.

You can grab the proxy server here — it still lives in Mozilla CVS — proxyserver.py.  It works fine on Win32, OS X, and Linux.  On Win32, the python that’s part of mozilla-build works well.  Run it like this:

python proxyserver.py

and then in Firefox’s proxy settings (or the system proxy settings), set your HTTP proxy to localhost:8000.  You can change the port via a command line option.  Then, visit all the pages/sites that you want cached (don’t forget to shift-reload or clear Firefox’s cache beforehand to ensure that Firefox actually goes out to the network!).  After you’ve got everything going, restart the proxy server in local-only mode:

python proxyserver.py -l

… and make sure that your demos work.  You can also run without -l live, especially if you will have a network connection (even a slow one) to give you the option of going out to the network if necessary.  Also, if you want to copy the proxy cache to another machine, just copy proxy_cache.db that gets created in the same directory as proxyserver.py.

The proxy server currently supports HEAD and GET requests.  It doesn’t support POST, so if you have something that depends on POST, you’re out of luck.  It wouldn’t be too hard to add though; patches accepted if someone wants to tackle that.

With the proxy server, I was able to give a bunch of demos that made heavy use of XHR, including some that loaded video, without having to rely on a network or spend time downloading and fixing up URLs.  It really made demo prep much easier.

One of the problems with Sylvester is that while it’s fully featured (arbitrary NxN matrices and vectors can be created and manipulated), it suffers in performance because of it.  Since this is such a crucial part of a successful WebGL program, I’ve put together a small package that I’m calling mjs.

mjs is designed around speed and simplicity.  For example, it doesn’t attempt to stuff vectors and matrices into JavaScript objects.  Because the language offers no operator overloading, there’s very little benefit in treating these types as discrete objects, and lots of performance and memory usage downsides.  Instead, it provides a set of functions for performing operations on vectors and matrices, which can be any array-like object.  For any function that returns a vector or matrix, an existing array can be passed in to take the result, or the function can create a new one.  Array reuse ends up being important because of the potential for expensive garbage collection churn eating away at performance.

Here’s a sample of the API:

var r = M4x4.rotate(Math.PI/2, V3.$(0, 1, 0),  M4x4.I);

Note that V3.$ and M4x4.$ are shorthand for creating a new V3 or M4x4 (I wanted to use V3() and M4x4(), but that didn’t work out too well since functions have a length property).  However, because all they return are just new array-like objects, you could also write:

var r = M4x4.rotate(Math.PI/2, [0, 1, 0], M4x4.I);

If the WebGL types are available, those will be used for newly created vectors/matrices.  They are a significant performance boost especially for repeated operations; but for specifying one-off vectors such as the above, literal array syntax is fine.

The rotate function internally makes a rotation matrix, and then multiplies it by the given matrix.  So the above could also be written as:

var rotation = M4x4.makeRotate(Math.PI/2, [0, 1, 0]);
var r = M4x4.mul(M4x4.I, rotation);

(The last line being redundant given that we’re multiplying by the identity matrix.)

All methods that return a vector or matrix take an optional final argument, that of an existing object to reuse.  For example:

var m0 = M4x4.$();
r = M4x4.mul(someMatrixA, someMatrixB, m0);
// r == m0, so the assignment isn't necessary, but it's handy for chaining
// .... do something with r ...
r = M4x4.mul(someMatrixB, someMatrixC, m0);
// r == m0 still
// ... do something else with new results ...

Without allocating any additional temporary objects.

As mentioned before, one of the goals of mjs is performance.  Matrix multiplication is one of the most common tasks, so here are some numbers comparing mjs, Sylvester, and native C code.  This was run on a Core i7 desktop using a local build of Spidermonkey, which included one patch that’s about to go into the tree that fixes the no-reuse tracing case.  (Without it, the no-reuse tracing case is much larger because it’s never actually jitted.) The test is simple: it multiplies two matrices together in a loop 1,000,000 times.

(I also have numbers for MSVC without the SSE2 compile flag, but the numbers vary greatly depending on whether the values eventually go to infinity or not; if the values end up trending towards 0, the non-SSE2 code tends to win at around 52ms vs. 71ms; if the values trend to infinity, the non-SSE2 code takes around 11,000ms!)

Those numbers are pretty encouraging — having native code be only 2x as slow for something like this is pretty nice to see. Granted, this is only a very isolated test, and I’m sure there are some tricks to optimizing the native code case (it’s currently just a fully unrolled set of multiplies and adds). The “no JIT” case is less nice, but I’m sure that our Jaegermonkey folks will be all over this testcase (right, guys?). In any case, ideally most WebGL rendering loops will be fully traced in Firefox, so it would be less of an issue.

mjs is still very much a work in progress; it’s missing a test suite and a whole bunch of features. You can find it hosted at Google Code, at webgl-mjs. (Side note: I couldn’t just call the project mjs because a project called mjs was abandoned on Sourceforget 5 years ago, and Google Code complained.) There’s also some documentation, viewable online here.

Bugs and contributions welcome!

Mouse events sort of work, toplevel windows sort of work, keyboard doesn’t work yet but shouldn’t be hard to hook up.  This is running in an emulator at the moment for ease of debugging, but it’s working just fine on physical hardware as well.

You’ll note that this is the full Firefox interface, and not the Fennec/Firefox Mobile UI; we’re testing with the full interface because it’s significantly more complex than the mobile UI and stresses Gecko much more.  So, if the full UI works, then Fennec should work fine as well.  Given the interest in Android on netbook and tablet devices, an updated version of the full Firefox UI might find a home on some of these.  Android has been pretty great to work with so far; it’s a bit unusual platform for us due to its Java core, but with the NDK we’re able to bridge things together without many problems.

We’re still a ways  to go before any kind of usable alpha release, but we’re certainly one step closer.  We’ll also be able to accelerate our progress now that we have some of the basic scaffolding in place.  I know I’m looking forward to running Fennec on my Droid, and there are tons of Android devices coming out that should be great platforms for Fennec.

We’re actively working through remaining warts and edge cases (and they are a lot of them!).  Take a look at the official Khronos WebGL landing page and Arun’s blog post for more information, including where to go to sign up for the public mailing list and for a set of resources about WebGL.

We’ve had support for WebGL in Firefox desktop nightlies for a few weeks now, but soon I hope to have it enabled on Firefox for mobile devices (code name Fennec) as well.  Jay demonstrated some simple demos working on Nokia’s N900 at the ARM techcon3 today a few weeks ago (took a while to get the video done!), and I’ve grabbed a quick video showing things in action.

This should be showing up in Firefox for mobile devices nightly builds, so we’ll see some more interesting things running on phones soon!