What is WebGL?

WebGL allows web developers to take advantage of the 3D capabilities of modern video cards to add 3D displays to their web applications. Apps that would have been possible only on the desktop or with plugins become possible in any modern browser that supports WebGL: 3D games, interactive product displays, scientific and medical visualization, shared virtual environments, and 3D content creation all become possible on the web.

For users, this means a more interactive and visually interesting web. It means having access to a wider range of applications and experiences on any device that supports the full web, instead of being limited to specific devices or platforms.

WebGL is being developed within the Khronos Group, the same group responsible for OpenGL and OpenGL ES.  Members of the WebGL group include Mozilla, Google, Opera, and Apple, as well as a number of hardware vendors who are interested in making sure that WebGL content can run well on both desktop and mobile hardware.  There’s a lot of support for WebGL!

I recently gave a talk at NVidia’s GPU Technology Conference about WebGL. The video stream is available (though sadly not using HTML5 video!), and it’s a good (though technical) overview of WebGL.

Let me check out some demos!

Here’s a couple of demos showcasing WebGL technology by Mozilla, Google, and others.

Some platform-specific notes: WebGL is currently disabled on Linux due to some build issues; it should be getting re-enabled in beta 9. For Windows users, you may need to install the DirectX Runtime in case these demos don’t work for you or if they have glitches — this allows Firefox to use an alternate rendering path that might be better supported on your system, especially on systems with Intel GPUs. We’re working on removing the need for this separate install in a future build.

Flight of the Navigator

Body Browser by Google

WebGL Aquarium

Check out Dave’s post for more details about the Flight of the Navigator demo.  You can find many more projects and demos using WebGL on the web and linked from webgl.org — for example, some other great examples are a web-based 3D editor called 3DTin and a vortex/anti-vortex annihilation simulation,

Give me more technical details!

WebGL brings the OpenGL ES 2.0 API to the HTML5 Canvas element. 3D content is confined to the canvas, but the canvas follows normal HTML compositing rules. For example, a 2D UI can be layered on top of the 3D scene using normal CSS mechanisms, and content underneath the canvas will show through transparent portions. In addition, CSS properties can be applied to the canvas itself, for effects like fading the entire scene in or out.

The WebGL API interacts well with the rest of the web platform; specifically, support is provided for loading 3D textures from HTML images or videos, and keyboard and mouse input is handled using familiar DOM events.

As a developer, how do I learn more about WebGL?

WebGL is based on OpenGL ES 2.0, which just so happens to be the same 3D API used for Android and iOS development, as well as being based on the desktop OpenGL API. Many resources available for ES 2.0 development translate almost directly to WebGL development.

Unlike desktop or mobile OpenGL development, it’s very easy to get started with WebGL. Some simple HTML and JS content lets you immediately start writing WebGL code. A number of tutorials already exist that focus on WebGL; you can take a look at Learning WebGL’s lessons to help you get started.

Here are some web resources with more information:

• webgl.org — official WebGL page, including specification and resource links
• learningwebgl.com — blog with regular updates on WebGL happenings

Future Work

WebGL focuses on OpenGL ES 2.0 feature compatibility to ensure content compatibility with mobile devices. However, ES 2.0 is behind the latest advances on the desktop today. In the future, various desktop features may become available in WebGL in the form of extensions.

There’s still some work to do on the Firefox side as well, in particular removing some performance bottlenecks on Windows when we’re using ANGLE for Direct3D compatibility.

Wrap up

WebGL will enable web developers to create new experiences for their users. As with any new technology, initial experimentation will lead to developers understanding better how to fully leverage WebGL. There’s already tremendous interest in WebGL, as can be seen by the wealth of frameworks and samples, even before WebGL has been released as part of any final shipping browser version! By including WebGL in Firefox, and along with our work on HTML5 video and audio support (including direct audio data access), Firefox supports a full set of web
technologies for building rich and compelling applications on the web.

One of the most challenging things in getting Firefox working with WebGL and hardware graphics acceleration has been dealing with platform-specific pieces to get access to OpenGL. In many cases similar functionality works differently (often in subtle ways), requiring both lots of testing and lots of very specific codepaths. EGL replaces all of these with a modern system designed with portability in mind. Until now, however, EGL has only been adopted in the mobile space. On the desktop, the older GLX, CGL, and WGL subsystems have held this role; in the case of GLX and WGL in particular, they bring along years of accumulated cruft.

Having a native EGL driver will allow us to ship one particular hardware acceleration provider that will work and be tested across various desktop and mobile platforms. Additionally, the same provider can connect to the ANGLE project, which implements EGL and OpenGL ES on top of Direct3D 9. Having OpenGL ES will allow us to test and develop truly identical code across desktop and mobile. As mobile graphics development has become important (not just to Mozilla, but in general!), having the same API implemented on the desktop will make it easier to catch problems and portability issues in an environment that’s much more conducive to development and debugging.

Native OpenGL ES on the desktop will also mean that we can tie our WebGL implementation directly to it, instead of going through the desktop OpenGL driver. Because WebGL follows the OpenGL ES specification, the native ES driver on the desktop will allow us to make a more efficient binding between WebGL and the underlying platform, potentially leading to higher performance.

As with any such change, it will be a while before we can depend on the presence of these APIs on the desktop. These first steps are important to making that change happen. I’m looking forward to seeing other vendors following AMD here, both on Windows and on other platforms.

(Somewhat technical OpenGL post follows; you’ve been warned.)

So over the past week, I’ve been trying to make our OpenGL stuff in Firefox be a little more coherent and easy to use, especially when it comes to doing things like rendering to a texture. This is something really for WebGL, and is a key to making WebGL fast… we want it all to live in video hardware, and we want to render it straight from there, instead of doing the horrible readback that it does currently.

Our current implementation uses PBuffers for each WebGL context. PBuffers are old and crufty, but they mostly work ok. But, everyone kept telling me, “PBuffers are old and crufty! FBOs are the new hotness! Why do you hate unicorns?” Problem is, WebGL really wants its own context — one of the advantages of FBOs over PBuffers is that they don’t need a separate context.  That’s great, if what you need is to render to a texture in your game or visualization app or whatever.  But, WebGL needs a context.

Ok, I thought, I can just use OpenGL’s (technically, WGL, EGL, GLX, and CGL’s) context sharing feature, have everything live in one shared global namespace, and things will just work.  This has one somewhat scary problem — if you don’t clean up any resources, they stick around forever, or at least until all the contexts in the sharing group are gone.  But, I figured we can keep track and clean up.

So I did all this work, set up a global context to use as a “shared” context.  Everything worked great, creating a FBO is certainly easier than creating a PBuffer and all that.

And then I decided to test it on Android.  One of the reasons why I wanted to do this was that on a number of mobile devices, among them Nokia’s N900 and Nvidia’s Android port, PBuffers cannot be bound as a texture; they just don’t support that.  The shared-context FBO approach worked great.  Then I plugged in a Nexus One.  Failure.

What gives?  It turns out, a bunch of current Android devices simply don’t support GL context sharing.  My plan?  Ruined.  The full table of tears actually looks like this:

The “Other” category means that the platform has an alternate approach that doesn’t involve either PBuffers or context sharing. These other approaches are:

Desktop – GLX: X11 Pixmaps can be rendered to and used as textures via texture_from_pixmap

Desktop – CGL: I think there’s some CoreAnimation thing that can be used here?

Maemo: X11 Pixmaps, as on GLX; potentially EGL_KHR_gl_texture_2D_image

Android – Tegra: EGL_KHR_gl_texture_2D_image, which allows a texture to be exported as an EGLImage.  This is ideal, since it gives all the benefit of FBOs without any of the downsides of context sharing.  Unfortunately, this is the only place where it’s supported, and as best I can tell nothing like this exists on the desktop.

The * next to some of the Android entries indicates that while they do support pbuffers, they only support power-of-two dimensions, at least for pbuffers that can be bound as textures.  This is annoying and caused me a bunch of grief until I realized that quirk.

So, with that information, the new plan is to attempt to share all window contexts’ resources — there are advantages here in cleanup operations and being able to do texture uploads and other things on different threads.  For WebGL and other offscreen contexts though, we want to avoid sharing if we can, so the order in which we’ll try things goes like this:

1. EGL_KHR_texture_2d_image + GL_OES_EGL_image.  Even though it’s only supported on one target, I still want to make sure that we use this where we can — it really is exactly what we want.
2. PBuffers.  Yes, they may be old and busted and difficult to create and all that, but if supported, they still do basically exactly what’s needed.
3. X11 Pixmaps + texture_from_pixmap.  Basically like PBuffers, but even more annoying and actually supported on X11.
4. Dummy window or other drawable and a FBO, plus full sharing with the windowed contexts.  This is only possible if sharing is possible, and is a little risky from a resource management perspective, but it works.
5. Can’t share, can’t texture from any renderable target?  Then we call glReadPixels and take the slow boat through system memory.

So, where I was hoping to just write one path — #4 in the list above — I now have to write 5.  On the plus side, 1-3 all don’t have the sharing problem.  On the minus side, it’s 5 separate code paths instead of just 1.

Hopefully the above information saves someone some pain while trying to do offscreen GL rendering on various platforms, especially mobile ones.  I wish the Android EGL implementations were higher quality; the non-Nvidia ones seem to support an identical set of extensions and report identical version strings, which makes me wonder if they’re just based on some generic code that Google provides.  If so, it would be nice to see that updated with support for texture_2d_image/EGL_image.

One of the problems with Sylvester is that while it’s fully featured (arbitrary NxN matrices and vectors can be created and manipulated), it suffers in performance because of it.  Since this is such a crucial part of a successful WebGL program, I’ve put together a small package that I’m calling mjs.

mjs is designed around speed and simplicity.  For example, it doesn’t attempt to stuff vectors and matrices into JavaScript objects.  Because the language offers no operator overloading, there’s very little benefit in treating these types as discrete objects, and lots of performance and memory usage downsides.  Instead, it provides a set of functions for performing operations on vectors and matrices, which can be any array-like object.  For any function that returns a vector or matrix, an existing array can be passed in to take the result, or the function can create a new one.  Array reuse ends up being important because of the potential for expensive garbage collection churn eating away at performance.

Here’s a sample of the API:

var r = M4x4.rotate(Math.PI/2, V3.$(0, 1, 0),  M4x4.I);

Note that V3.$ and M4x4.$ are shorthand for creating a new V3 or M4x4 (I wanted to use V3() and M4x4(), but that didn’t work out too well since functions have a length property).  However, because all they return are just new array-like objects, you could also write:

var r = M4x4.rotate(Math.PI/2, [0, 1, 0], M4x4.I);

If the WebGL types are available, those will be used for newly created vectors/matrices.  They are a significant performance boost especially for repeated operations; but for specifying one-off vectors such as the above, literal array syntax is fine.

The rotate function internally makes a rotation matrix, and then multiplies it by the given matrix.  So the above could also be written as:

var rotation = M4x4.makeRotate(Math.PI/2, [0, 1, 0]);
var r = M4x4.mul(M4x4.I, rotation);

(The last line being redundant given that we’re multiplying by the identity matrix.)

All methods that return a vector or matrix take an optional final argument, that of an existing object to reuse.  For example:

var m0 = M4x4.$();
r = M4x4.mul(someMatrixA, someMatrixB, m0);
// r == m0, so the assignment isn't necessary, but it's handy for chaining
// .... do something with r ...
r = M4x4.mul(someMatrixB, someMatrixC, m0);
// r == m0 still
// ... do something else with new results ...

Without allocating any additional temporary objects.

As mentioned before, one of the goals of mjs is performance.  Matrix multiplication is one of the most common tasks, so here are some numbers comparing mjs, Sylvester, and native C code.  This was run on a Core i7 desktop using a local build of Spidermonkey, which included one patch that’s about to go into the tree that fixes the no-reuse tracing case.  (Without it, the no-reuse tracing case is much larger because it’s never actually jitted.) The test is simple: it multiplies two matrices together in a loop 1,000,000 times.

(I also have numbers for MSVC without the SSE2 compile flag, but the numbers vary greatly depending on whether the values eventually go to infinity or not; if the values end up trending towards 0, the non-SSE2 code tends to win at around 52ms vs. 71ms; if the values trend to infinity, the non-SSE2 code takes around 11,000ms!)

Those numbers are pretty encouraging — having native code be only 2x as slow for something like this is pretty nice to see. Granted, this is only a very isolated test, and I’m sure there are some tricks to optimizing the native code case (it’s currently just a fully unrolled set of multiplies and adds). The “no JIT” case is less nice, but I’m sure that our Jaegermonkey folks will be all over this testcase (right, guys?). In any case, ideally most WebGL rendering loops will be fully traced in Firefox, so it would be less of an issue.

mjs is still very much a work in progress; it’s missing a test suite and a whole bunch of features. You can find it hosted at Google Code, at webgl-mjs. (Side note: I couldn’t just call the project mjs because a project called mjs was abandoned on Sourceforget 5 years ago, and Google Code complained.) There’s also some documentation, viewable online here.

Bugs and contributions welcome!

We’re actively working through remaining warts and edge cases (and they are a lot of them!).  Take a look at the official Khronos WebGL landing page and Arun’s blog post for more information, including where to go to sign up for the public mailing list and for a set of resources about WebGL.

We’ve had support for WebGL in Firefox desktop nightlies for a few weeks now, but soon I hope to have it enabled on Firefox for mobile devices (code name Fennec) as well.  Jay demonstrated some simple demos working on Nokia’s N900 at the ARM techcon3 today a few weeks ago (took a while to get the video done!), and I’ve grabbed a quick video showing things in action.

This should be showing up in Firefox for mobile devices nightly builds, so we’ll see some more interesting things running on phones soon!

Edit 12/2: these types have changed names; they now have a WebGL prefix instead of a Canvas prefix.

It became clear that pure JS arrays are not a useful way of shoveling around lots of 3D data; their very flexibility makes them impractical for performance-critical uses. In particular, WebGL often wants to deal with arrays of a specific type — an array of integers, an array of floats, etc. Even more complicated is the need to manage multiple types within a single memory region; for performance, it’s often preferable to allocate one chunk of video memory, and place coordinates, colors, and other types in there, replacing them as necessary.

There are two portions to the solution: the WebGLArrayBuffer and a set of typed WebGLArray views. A WebGLArrayBuffer represents chunk of data. It can be allocated with a size in bytes, but it can’t be accessed in any way. To actually manipulate the data inside a WebGLArrayBuffer, a WebGLArray has to be created that references it. An example:

var buf = new WebGLArrayBuffer(3*4);
var floats = new WebGLFloatArray(buf);
floats[0] = 12.3;
floats[1] = 23.4;
floats[2] = 34.5;

The above chunk of code allocates a 12-byte WebGLArrayBuffer, and then creates a float-typed view onto the buffer which can then be manipulated (almost) like a normal array. Of course, the above is cumbersome to write, so there are shorthands that will allocate a WebGLArrayBuffer, and optionally fill it with data from a JS array:

var f1 = new WebGLFloatArray(3);
var f2 = new WebGLFloatArray([12.3, 23.4, 34.5]);

The size of each WebGLArrayBuffer is fixed; there is currently no way to change its size once allocated.

Multiple WebGLArrays can reference the same WebGLArrayBuffer. For example:

var buf = new WebGLArrayBuffer(12*3*4+12*4);
var points = new WebGLFloatArray(buf, 0, 12*3);
var colors = new WebGLUnsignedByteArray(buf, 12*3*4, 12*4);

This creates a buffer of 192 bytes, which is enough room for 12 3-coordinate float points followed by 12 RGBA colors, with each component represented as an unsigned byte. The arguments to the WebGLArray constructors are the offset from the start of the buffer (in bytes), and the length (in elements). The offset must always be a multiple of the element size (to preserve alignment), and the buffer must obviously be large enough for the given offset and length. If length is not given, the length is assumed to be “from offset until the end of the array buffer”; that value must be a multiple of the element size. If offset is not given, it’s assumed to be zero.

For extra complex use cases, WebGLArrays can reference overlapping regions of a WebGLArrayBuffer:

var buf = new WebGLArrayBuffer(192); // same value from above
var points = new WebGLFloatArray(buf);
var colors = new WebGLUnsignedByteArray(buf);
points[0] = 12.3;
points[1] = 23.4;
points[2] = 34.5;
colors[12] = 0xff;
colors[13] = 0xaa;
colors[14] = 0x00;
colors[15] = 0x00;

In the buffer, this writes 3 float values followed by 4 byte values. You’ll note that this use is significantly more complex, and requires the user to keep track of the current position in terms of whatever element they’re modifying (thus setting array elements 12, 13, 14, and 15 for the color).

If an attempt is made to store data in a WebGLArray that doesn’t fit within the right type, a C-style cast is performed. If the data is an entirely wrong type (e.g. trying to store a string or an object), Gecko currently throws an exception, but this might become a silent 0 or similar in the future.

Where does this fit in WebGL? Any API function that needs an array of data takes a WebGLArrayBuffer. This avoids placing costly JS array type conversion in a potential critical performance path, and simplifies a number of aspects of the API. So, VBOs, texture data (if not loaded from a DOM image element or from a CanvasImageData object), index array, etc. all use WebGLArrayBuffers/WebGLArrays for pulling data in and out. WebGLArrays also help manage memory usage — an array of byte color data now takes up exactly as much memory as needed, instead of getting expanded out to 4 bytes. Also, critically, floating point data can be stored as 32-bit single-precision floats instead of 64-bit doubles, taking up half as much space when the underlying graphics system can’t support 64-bit values.

This API is overall lacking in developer niceties, since the focus was on providing the necessary functionality. Higher level wrappers can be written in JS to simplify usage. In addition, by keeping it as bare-bones as it is, it allows for fast implementation on native hardware via the JITs in all the current-generation JS engines. The web currently fudges around the problem of binary data by passing it around either in strings (because JS strings are UCS2, therefore all 8-bit elements are valid, but with a performance and memeory cost), or often encoding as base64 (again going back to strings). I can see this type of dense/native type access being useful for both the File and WebSockets APIs as a way to exchange and deal with binary data.

Over the weekend I’ve put together this example, which uses WebGL to render an exported Spore creature, and let the user rotate the 3D model to view it from different angles.  For those who want to try it out, you’ll need a recent Firefox nightly (one from today, September 21, or newer), and with one preference flipped as described in this post.

I’ll be working to update the very basic “getting started” demos from the GL ES 2 book that I ported to Canvas 3D as well, so that those who are interested in experimenting can have some good basic code to look at.  They’re not updated yet, but they should be in the next day or two.

For those of you on Windows who don’t have an up to date OpenGL driver, or don’t have the possibility of getting one (e.g. many common Intel graphics cards doesn’t have OpenGL drivers), you can enable software rendering by downloading a Windows build of the Mesa software OpenGL implementation.  It won’t be fast, but it should be enough for you to get an idea of what’s going on, and to play with some of the demos.  To use it, download webgl-mesa-751.zip and extract it somewhere on your computer.  It has a single file, OSMESA32.DLL, that you need to tell Firefox where to find:  open up about:config, and set the preference webgl.osmesalib to the path of OSMESA32.DLL.  If you extracted it to “C:\temp”, you would put in “C:\temp\osmesa32.dll” in the pref.  Then, flip webgl.software_rendering to true, and you should be good to go.

(Mac OS X users shouldn’t need to bother with software rendering, since Apple already provides a high quality OpenGL implementation, and Linux users should be ok as long as they have recent OpenGL drivers installed.)

If you’re not familiar with WebGL, it’s the evolution of work that Mozilla started a few years ago with experiments called Canvas 3D — essentially a way of accessing OpenGL from within the browser, through the HTML5 canvas tag.  Earlier this year, we started a standardization effort within the Khronos Group which oversees OpenGL, and have made very rapid progress.

Along with the Firefox implementation, a WebGL implementation landed in WebKit fairly recently.  All of these implementations are going to have some interoperability issues for the next little while, as the spec is still in flux and we’re tracking it at different rates, but will hopefully start to stabilize over the next few months.

If you’d like to experiment with WebGL with a trunk nightly build (starting from Friday, September 18th), all you have to do is flip a pref: load about:config, search for “webgl“, and double-click “webgl.enabled_for_all_sites” to change the value from false to true.  You’ll currently have the most luck on MacOS X machines or Windows machines with up-to-date OpenGL drivers.

We still have some ways to go, as there are issues in shader security and portability, not to mention figuring out what to do on platforms where OpenGL is not available.  (The latter is an interesting problem; we’re trying to ensure that the API can be implementable on top of a non-GL native 3D API, such as Direct3D, so that might be one option.)  But progress is being quickly made.

When paired with high-performance JavaScript, such as what we’ve seen come from both Firefox and other browsers, should allow for some exciting fully 3D-enabled web applications.  We’ll have some simple demos linked for you soon, both here and on Mark’s blog.

Edit: new post up, with some samples!