I became involved in Firefox and the Mozilla community when I was suckeredtalked into fixing a bug — a bug that involved RDF, bookmarks, and the template builder. (Thankfully, all three of those things — RDF, old bookmarks, template builder — have almost been completely excised from Firefox.) Somehow I ended up sticking around to fix more bugs and get involved in bigger projects, from working with Stuart to rework our graphics engine up to getting the Android port going.

One of the things that I had as a goal when I started at Mozilla was to bring 3D graphics to the web. It took a bit longer than I expected (as usual), but with WebGL, we now have a solid foundation for doing all sorts of 3D apps via the web. Many things had to come together to enable that: fast JavaScript, improvements to overall rendering architecture, emphasis on mobile, etc. all contributed to the viability of WebGL. Over the years, I’ve had the opportunity to work on many projects in those areas with many talented people, and the experiences will certainly help in future efforts.

So, what am I doing next? Something entirely different. I’ll be doing software in a totally different industry, joining some friends in bringing some disruptive innovation there. I’ll still be writing software, though with a much smaller team — something that I’ve come to enjoy, as being small and scrappy has a lot of advantages and is a lot of fun. There’s a lot of interesting technical challenges, mainly related to dealing with large volumes of data (multiple terabytes not being uncommon) — processing, visualization, analysis. You’ll also likely see me hacking various bits in my own spare time, whether in Mozilla, web apps, or mobile apps. I plan on continuing to blog about these topics.

For WebGL in particular, I’ll be around to launch the initial version of the spec, and plan on continuing to be involved in the standards group. I might not be hacking on Mozilla’s implementation as frequently, but it’ll be in good hands.

Thank you to all the people that I’ve had a chance to work with and learn from over the past 5 (almost 6!) years. I’ll still be around irc and other forums so won’t be pulling a disappearing act any time soon, and I’m looking forward to seeing Firefox 4 out there!

What is WebGL?

WebGL allows web developers to take advantage of the 3D capabilities of modern video cards to add 3D displays to their web applications. Apps that would have been possible only on the desktop or with plugins become possible in any modern browser that supports WebGL: 3D games, interactive product displays, scientific and medical visualization, shared virtual environments, and 3D content creation all become possible on the web.

For users, this means a more interactive and visually interesting web. It means having access to a wider range of applications and experiences on any device that supports the full web, instead of being limited to specific devices or platforms.

WebGL is being developed within the Khronos Group, the same group responsible for OpenGL and OpenGL ES.  Members of the WebGL group include Mozilla, Google, Opera, and Apple, as well as a number of hardware vendors who are interested in making sure that WebGL content can run well on both desktop and mobile hardware.  There’s a lot of support for WebGL!

I recently gave a talk at NVidia’s GPU Technology Conference about WebGL. The video stream is available (though sadly not using HTML5 video!), and it’s a good (though technical) overview of WebGL.

Let me check out some demos!

Here’s a couple of demos showcasing WebGL technology by Mozilla, Google, and others.

Some platform-specific notes: WebGL is currently disabled on Linux due to some build issues; it should be getting re-enabled in beta 9. For Windows users, you may need to install the DirectX Runtime in case these demos don’t work for you or if they have glitches — this allows Firefox to use an alternate rendering path that might be better supported on your system, especially on systems with Intel GPUs. We’re working on removing the need for this separate install in a future build.

Flight of the Navigator

Body Browser by Google

WebGL Aquarium

Check out Dave’s post for more details about the Flight of the Navigator demo.  You can find many more projects and demos using WebGL on the web and linked from webgl.org — for example, some other great examples are a web-based 3D editor called 3DTin and a vortex/anti-vortex annihilation simulation,

Give me more technical details!

WebGL brings the OpenGL ES 2.0 API to the HTML5 Canvas element. 3D content is confined to the canvas, but the canvas follows normal HTML compositing rules. For example, a 2D UI can be layered on top of the 3D scene using normal CSS mechanisms, and content underneath the canvas will show through transparent portions. In addition, CSS properties can be applied to the canvas itself, for effects like fading the entire scene in or out.

The WebGL API interacts well with the rest of the web platform; specifically, support is provided for loading 3D textures from HTML images or videos, and keyboard and mouse input is handled using familiar DOM events.

As a developer, how do I learn more about WebGL?

WebGL is based on OpenGL ES 2.0, which just so happens to be the same 3D API used for Android and iOS development, as well as being based on the desktop OpenGL API. Many resources available for ES 2.0 development translate almost directly to WebGL development.

Unlike desktop or mobile OpenGL development, it’s very easy to get started with WebGL. Some simple HTML and JS content lets you immediately start writing WebGL code. A number of tutorials already exist that focus on WebGL; you can take a look at Learning WebGL’s lessons to help you get started.

Here are some web resources with more information:

• webgl.org — official WebGL page, including specification and resource links
• learningwebgl.com — blog with regular updates on WebGL happenings

Future Work

WebGL focuses on OpenGL ES 2.0 feature compatibility to ensure content compatibility with mobile devices. However, ES 2.0 is behind the latest advances on the desktop today. In the future, various desktop features may become available in WebGL in the form of extensions.

There’s still some work to do on the Firefox side as well, in particular removing some performance bottlenecks on Windows when we’re using ANGLE for Direct3D compatibility.

Wrap up

WebGL will enable web developers to create new experiences for their users. As with any new technology, initial experimentation will lead to developers understanding better how to fully leverage WebGL. There’s already tremendous interest in WebGL, as can be seen by the wealth of frameworks and samples, even before WebGL has been released as part of any final shipping browser version! By including WebGL in Firefox, and along with our work on HTML5 video and audio support (including direct audio data access), Firefox supports a full set of web
technologies for building rich and compelling applications on the web.

However, Google provides the cross-compilers for Windows as well; since the external drive that had my VM on it died a week ago, it seemed like a good opportunity to get the build working under Windows.  Because the native compilers are built using cygwin, this is a bit convoluted, but seems to work.  You’ll need some msys and cygwin tools, as well as the standard Android build prereqs (Java, SDK, NDK):

• mozilla-build installed
• the mingw moztools, unpacked somewhere like c:\mozilla-build\moztools-mingw (see bug comment as to why this is required)
• cygwin installed, including gcc/g++ and the mingw 4.5.1 compilers
• a JDK (note: I don’t know if OpenJDK supports Windows; if it does, that’s also an option)
• the Android SDK (with at least the platform level 8 SDK)
• the Crystax r4 NDK for Win32
• the patches from bug 607975

You do the build inside the standard mozilla-build shell prompt (not a cygwin shell).  However, you’ll need to copy *.dll from c:/cygwin/bin (assuming you installed cygwin in c:/cygwin) into a new directory, such as c:/cygwin/bindll.  Then add that bindll directory to your msys path.  Adding the cygwin bin directory to your msys path will get you cygwin versions of various tools, and will greatly confuse things.  You just need the DLLs so that the compilers can execute; the compiler binary paths are explicitly specified.

There’s a sample mozconfig inside bug 607975.  You’ll need to use a relative path to run configure, e.g. “MOZCONFIG=../../fennec-config ../mozilla-central/configure”, and you must use mingw make and not pymake to build.  pymake has some weird issues, where it complains that it doesn’t know how to rebuild dependencies like “-lzlib”.  This is likely fixable, because I think this is a magic thing that gnu make special-cases, and something that our build system shouldn’t do in any case.

Some remaining problems with the build:

• It’s not clear why jemalloc hangs without the TLS patch; mwu says he has seen this in other cases, but it’s consistent here.
• Opt builds seem to just draw white; not sure why that is, debug builds work fine.  This should be fixable.
• Disabling automatic dependencies means you have to be extra careful if you modify a header file or something that might not get picked up; they have to be disabled because the cygwin compiler will generate paths like “/cygwin/c/…” in the deps file, which the msys make won’t ever be able to find — thus causing it to rebuild everything on every build.  This is also fixable, perhaps with a postprocessing step after each compilation.
• Would be nice if pymake worked for building speed.

This is obviously not a full solution, and VMs are still a lot more stable where builds are concerned, but it’s at least possible to build Fennec for Android under Win32.  GDB also seems to work fine, though as always, you’ll want to use the NVIDIA-provided gdb/gdbserver pair, because the one that’s part of the NDK seems to continue to have issues.

1. Did Firefox recover?

Immediately after the driver reset, did Firefox recover?  If so, skip this step.  If, however, it’s not repainting and otherwise seems hung, forcing a crash report can help give us data that we need to figure out why recovery didn’t happen properly.  To do this, I wrote a tool that forces a hung Firefox to crash in a way that will pop up the crash reporter.  You can download it here: force-firefox-crash.zip.  Just unzip and double click on the .exe, and the current running firefox.exe process will be killed, and you’ll be presented with the crash reporter.

Submit the report, and after restarting Firefox, type in about:crashes in the location bar.  You’ll want the ID of the top entry in this list for when you file a bug or contact support later.  (You can right click the top link and select “Copy Link Location” to grab a link for easy pasting.)

2. Grab the Driver Crash information

Select the start menu, and type in “problem report” in the search box.  Select View all problem reports from the list:

In the problem reports list, scroll down to the Windows section if it’s not visible, and sort by Date — you want the latest “Video hardware error” entry:

Double-click on the latest “Video hardware error” entry.  In the details screen, click “View a temporary copy of these files”:

You’ll get a new explorer window with three files.  Select all three, and compress them into a ZIP file, by right-clicking and selecting “Send To -> Compressed Folder”:

Copy the resulting zip file to your desktop, so that you can include it with your report.

3. Create a bug

Almost done! In bugzilla.mozilla.org, create a new bug in the Core product and the Graphics component.  Attach the zip file you created in step two to the report.  If you also created a Firefox crash dump in step one, include a link to it in your bug report.

This should give us and the graphics hardware vendors enough information to track down what the original problem was, so that we can fix it for Firefox 4!

Here are two examples that show off what can be done with the improved JS engine and capabilities that will be present in Firefox 4. The first example shows a simple web-based Darkroom that allows you to perform color correction on an image. The HTML+JS is around 700 lines of code, not counting jQuery. This is based on a demo that’s included with Google’s Native Client (NaCl) SDK; in that demo, the color correction work is done inside native code going through NaCl. That demo (originally presented as “too slow to run in JavaScript”) is a few thousand lines of code, and involves downloading and installing platform-specific compilers, multiple steps to test/deploy code, and installing a plugin on the browser side.

I get about 15-16 frames per second with the default zoomed out image (around 5 million pixels per second — that number won’t be affected by image size) on my MacBook Pro, which is definitely fast enough for live manipulation. The algorithm could be tightened up to make this faster still. Further optimizations to the JS engine could help here as well; for example, I noticed that we spend a lot of time doing floating point to integer conversions for writing the computed pixels back to the display canvas, due to how the canvas API specifies image data handling.

The Web Darkroom tool also supports drag & drop, so you can take any image from your computer and drop it onto the canvas to load it. A long (long!) time ago, back in 2006, I wrote an addon called “Croppr!”. It was intended to be used with Flickr, allowing users to play around with custom crops of any image, and then leave crop suggestions in comments to be viewed using Croppr. It almost certainly doesn’t work any more, but it would be neat to update it: this time with both cropping and color correction. Someone with the addon (perhaps a Jetpack now!) could then visit a Flickr photo and experiment, and leave suggestions for the photographer.

The second example is based on some work that Dave Humphrey and others have been doing to bring audio manipulation to the web platform. Originally, their spec included a pre-computed FFT with each audio frame delivered to the web app. I suggested that there’s no need for this — while a FFT is useful for some applications, for others it would be wasted work. Those apps that want a FFT could implement one in JS. Some benchmark numbers backed this up — using the typed arrays originally created for WebGL, computing an FFT in JS was approaching the speed of native code. Again, both could be sped up (perhaps using SSE2 or something like Mono.Simd on the JS side), but it’s fast enough to be useful already.

The demo shows this in action. A numeric benchmark isn’t really all that interesting, so instead I take a video clip, and as it’s playing, I extract a portion of the green channel of each frame and compute its 2D FFT, which is then displayed. The original clip plays at 24 frames per second, so that’s the upper bound of this demo. Using Float32 typed arrays, the computation and playback proceeds at around 22-24fps for me.

You can grab the video controls and scrub to a specific frame. (The frame rate calculation is only correct while the video is playing normally, not while you’re scrubbing.) The video source uses Theora, so you’ll need a browser that can play Theora content. (I didn’t have a similar clip that uses WebM, or I could have used that.)

These examples are demonstrating the strength of the trace-based JIT technique that Firefox has used for accelerating JavaScript since Firefox 3.5. However, not all code can see such dramatic speedups from that type of acceleration. Because of that, we’ll be including a full method-based JIT for Firefox 4 (for more details, see David Anderson’s blog, as well as David Mandelin’s blog). This will provide significantly faster baseline JS performance, with the trace JIT becoming a turbocharger for code that it would naturally apply to.

Combining fast JavaScript performance alongside new web platform technologies such as WebGL and Audio will make for some pretty exciting web apps, and I’m looking forward to seeing what developers do with them!

Edit: Made some last-minute changes to the demos, which ended up pulling in a slightly broken version of jQuery UI that wasn’t all that happy with Safari. Should be fixed now!

Edit #2: This post is available in Belorussian, courtesy of Jason Fragoso!

One of the most challenging things in getting Firefox working with WebGL and hardware graphics acceleration has been dealing with platform-specific pieces to get access to OpenGL. In many cases similar functionality works differently (often in subtle ways), requiring both lots of testing and lots of very specific codepaths. EGL replaces all of these with a modern system designed with portability in mind. Until now, however, EGL has only been adopted in the mobile space. On the desktop, the older GLX, CGL, and WGL subsystems have held this role; in the case of GLX and WGL in particular, they bring along years of accumulated cruft.

Having a native EGL driver will allow us to ship one particular hardware acceleration provider that will work and be tested across various desktop and mobile platforms. Additionally, the same provider can connect to the ANGLE project, which implements EGL and OpenGL ES on top of Direct3D 9. Having OpenGL ES will allow us to test and develop truly identical code across desktop and mobile. As mobile graphics development has become important (not just to Mozilla, but in general!), having the same API implemented on the desktop will make it easier to catch problems and portability issues in an environment that’s much more conducive to development and debugging.

Native OpenGL ES on the desktop will also mean that we can tie our WebGL implementation directly to it, instead of going through the desktop OpenGL driver. Because WebGL follows the OpenGL ES specification, the native ES driver on the desktop will allow us to make a more efficient binding between WebGL and the underlying platform, potentially leading to higher performance.

As with any such change, it will be a while before we can depend on the presence of these APIs on the desktop. These first steps are important to making that change happen. I’m looking forward to seeing other vendors following AMD here, both on Windows and on other platforms.

So, I finally wrote the (very simple) debug extension I wish I’d had.  You can find the source for imext here (it’s a hg repo), and a binary DLL built for x86 here.  Rename it to imext.dll and drop it alongside your x86 WinDbg.  Use “!imext.help” to get some basic instructions.  It’s pretty rough code and only does exactly what I needed yesterday, but it can become more useful pretty quickly.  There are some weird bugs with WinDbg’s expression evaluators that make it difficult to use with actual expressions; passing addresses directly works better. Also remember that WinDbg’s default expression evaluator is MASM, not C++ — the main thing is that this means that any bare numbers are interpreted as hex (use 0n123 to get decimal).

I’ll probably extend this quite a bit over time, including teaching it about Mozilla-specific things like gfxImageSurfaces.

(Somewhat technical OpenGL post follows; you’ve been warned.)

So over the past week, I’ve been trying to make our OpenGL stuff in Firefox be a little more coherent and easy to use, especially when it comes to doing things like rendering to a texture. This is something really for WebGL, and is a key to making WebGL fast… we want it all to live in video hardware, and we want to render it straight from there, instead of doing the horrible readback that it does currently.

Our current implementation uses PBuffers for each WebGL context. PBuffers are old and crufty, but they mostly work ok. But, everyone kept telling me, “PBuffers are old and crufty! FBOs are the new hotness! Why do you hate unicorns?” Problem is, WebGL really wants its own context — one of the advantages of FBOs over PBuffers is that they don’t need a separate context.  That’s great, if what you need is to render to a texture in your game or visualization app or whatever.  But, WebGL needs a context.

Ok, I thought, I can just use OpenGL’s (technically, WGL, EGL, GLX, and CGL’s) context sharing feature, have everything live in one shared global namespace, and things will just work.  This has one somewhat scary problem — if you don’t clean up any resources, they stick around forever, or at least until all the contexts in the sharing group are gone.  But, I figured we can keep track and clean up.

So I did all this work, set up a global context to use as a “shared” context.  Everything worked great, creating a FBO is certainly easier than creating a PBuffer and all that.

And then I decided to test it on Android.  One of the reasons why I wanted to do this was that on a number of mobile devices, among them Nokia’s N900 and Nvidia’s Android port, PBuffers cannot be bound as a texture; they just don’t support that.  The shared-context FBO approach worked great.  Then I plugged in a Nexus One.  Failure.

What gives?  It turns out, a bunch of current Android devices simply don’t support GL context sharing.  My plan?  Ruined.  The full table of tears actually looks like this:

The “Other” category means that the platform has an alternate approach that doesn’t involve either PBuffers or context sharing. These other approaches are:

Desktop – GLX: X11 Pixmaps can be rendered to and used as textures via texture_from_pixmap

Desktop – CGL: I think there’s some CoreAnimation thing that can be used here?

Maemo: X11 Pixmaps, as on GLX; potentially EGL_KHR_gl_texture_2D_image

Android – Tegra: EGL_KHR_gl_texture_2D_image, which allows a texture to be exported as an EGLImage.  This is ideal, since it gives all the benefit of FBOs without any of the downsides of context sharing.  Unfortunately, this is the only place where it’s supported, and as best I can tell nothing like this exists on the desktop.

The * next to some of the Android entries indicates that while they do support pbuffers, they only support power-of-two dimensions, at least for pbuffers that can be bound as textures.  This is annoying and caused me a bunch of grief until I realized that quirk.

So, with that information, the new plan is to attempt to share all window contexts’ resources — there are advantages here in cleanup operations and being able to do texture uploads and other things on different threads.  For WebGL and other offscreen contexts though, we want to avoid sharing if we can, so the order in which we’ll try things goes like this:

1. EGL_KHR_texture_2d_image + GL_OES_EGL_image.  Even though it’s only supported on one target, I still want to make sure that we use this where we can — it really is exactly what we want.
2. PBuffers.  Yes, they may be old and busted and difficult to create and all that, but if supported, they still do basically exactly what’s needed.
3. X11 Pixmaps + texture_from_pixmap.  Basically like PBuffers, but even more annoying and actually supported on X11.
4. Dummy window or other drawable and a FBO, plus full sharing with the windowed contexts.  This is only possible if sharing is possible, and is a little risky from a resource management perspective, but it works.
5. Can’t share, can’t texture from any renderable target?  Then we call glReadPixels and take the slow boat through system memory.

So, where I was hoping to just write one path — #4 in the list above — I now have to write 5.  On the plus side, 1-3 all don’t have the sharing problem.  On the minus side, it’s 5 separate code paths instead of just 1.

Hopefully the above information saves someone some pain while trying to do offscreen GL rendering on various platforms, especially mobile ones.  I wish the Android EGL implementations were higher quality; the non-Nvidia ones seem to support an identical set of extensions and report identical version strings, which makes me wonder if they’re just based on some generic code that Google provides.  If so, it would be nice to see that updated with support for texture_2d_image/EGL_image.

Let’s get one thing out of the way up front.  Today’s web browser is in many ways acting like a miniature full operating system.  It runs multiple applications at once (whether in multiple windows or tabs).  It might do a lot of background processing.  It can work with large data sets, for example large images on flickr or large spreadsheets on Google Documents.  But, the final memory usage number that the user sees when they open up the Task Manager or Process Viewer is the aggregate memory usage of the entire system.  So, the goal of improving our memory usage is not to get that number to the lowest possible — doing that would be an unacceptable tradeoff in performance for users — but instead to understand where memory is being used, and then use that data to improve in those areas as possible.

One comment that I’ve heard is that Firefox 3.6 seems to use more memory than Firefox 3.0.  My initial tests show this to not be true; specifically, I looked at the “Private Bytes” value in the Windows 7 task manager shortly after startup with about:blank, and also after opening a number of tabs (gmail, google docs, cnn.com, front page of the boston.com big picture blog, engadget, and a few others).  Here are the results of a typical run:

The next question is figuring out where all the memory goes. I’ve been adding some instrumentation to Firefox to figure out in more detail where memory is being used. For a sample run with the multiple tabs shown above, here’s what some of that reporter data looks like:

Or, graphically:

There’s some overlap in those numbers — for example, the jemalloc commit size is a subset of the Windows Private Bytes number, and most of the rest is a subset of the jemalloc commit size. Likewise, the uncompressed images number is a subset of the Win32 graphics surfaces number; that is, ~53MB is in use by win32 surfaces, and almost all of that is due to live images in pages (remember that we’ve got some image heavy sites in that tab set, including the Big Picture blog which has around 10-11 large images on it… those should account for about 20-25MB just by themselves).

There are some things that don’t make sense in the above, which mean that my instrumentation isn’t quite correct… for example, adding up the JS numbers, the Images number, and the PresShell arenas number brings us beyond the jemalloc commit size, which shouldn’t be true. However, some of the Image data is allocated by GDI, likely bypassing jemalloc, so we have to take that into account. There’s also some large other chunks of the browser that have yet to be instrumented, which should provide additional insight.

Two initial observations: one, keeping images compressed in memory and only decompressing them briefly when we need to draw them is a potential huge memory win. We have the infrastructure and code to do this in place; it was disabled recently while some of the internals changed, and it needs to be reenabled.

Two, the 30MB or so in the “js_malloc Other” bucket is also pretty curious. We need to do some more work to figure out what exactly is in here. (This contains things like data structures for tracking array contents and — potentally a big one — string data.)

I’ll be blogging more as the instrumentation takes shape, and as it gets landed into trunk nightly builds. Much of this information will be visible in about:memory, and eventually we’ll be able to give some per-tab memory information as well.

• We’ve only really tested this on the Motorola Droid and the Nexus One.
• It will likely not eat your phone, but bugs might cause your phone to stop responding, requiring a reboot.
• Memory usage of this build isn’t great — in many ways it’s a debug build, and we haven’t really done a lot of optimization yet.  This could cause some problems with large pages, especially on low memory devices like the Droid.
• You’ll see the app exit and relaunch on first start, as well as on add-on installs; this is a quirk of our install process, and we’re working to get rid of it.
• You can’t open links from other apps using Fennec; we should have this for the next build.
• This build requires Android 2.0 or above, and likely an OpenGL ES 2.0 capable device.
• Edit: This build must be installed to internal memory, not to a SD card.

There also aren’t yet any automated nightly developer builds or automated updates to this build; it’s even more of a pre-nightly build (even earlier than pre-alpha).  But, it’s usable enough that we wanted to get some feedback on it as we continue to develop.

Weave Sync

There is an experimental version of Weave that is compatible with this build: from within Fennec on your phone, open the Mozilla Labs weave page at https://mozillalabs.com/weave/ and click on “Experimental Version”.  (It’s to the right of the big Download Weave now! link — don’t click on that one though, it’s an older version.)  Install the add-on, then you’ll need to restart Fennec (swipe the screen left and then click on the “gear” icon to open the browser tools panel, then click on addons and click the Restart button at the top).  Follow the instructions when Fennec restarts.

Troubleshooting

Should you run into problems, such as the app not responding or just giving you a black screen, you can force it to quit by going into the Android Settings, selecting Applications, selecting Manage Applications, then selecting Fennec, and tapping Force Stop.  (A utility called White Killer, available from the Market, can do the same job with fewer clicks.)  Worst case, uninstalling and reinstalling would clear out your profile and any saved data.

Installation & Feedback

So, now that you’ve read all that, you can download the build here — the easiest way is to download it using your phone’s browser, and then click on it in the downloads list to install it.  If you’re reading this on your desktop, you can scan the QR code here on your phone, or type in the following address in your phone’s browser: bit.ly/fennec-android.  You may need to enable installation of non-Market applications by going to Settings, Applications, and checking “Unknown Sources”.

We’ve created a temporary Google Group for feedback about this pre-alpha build.  In the future we’ll have a more permanent way for user feedback and comments, but for now, please use the group to let us know what you think!