Updated Add-On installation site, now hosted on addons.mozilla.org.  Compatible with post-3.0b5 Firefox nightlies.

About a year and a half ago, I demoed an early prototype of an OpenGL 3D context for the canvas element at XTech. There was a lot of interest, but due to time constraints I wasn't able to spend as much time working on it as I would have liked. In addition, Gecko 1.9/Firefox 3 wasn't in a state where the extension could run against an unpatched build. Over the past few months I've been able to do some much-needed code cleanup and put together a preview of what an OpenGL context for canvas could look like.

There are two contexts provided by the extension. "moz-gles11" follows the OpenGL ES 1.1 spec very closely, providing an almost identical API and feature set, and "moz-glweb20" follows OpenGL ES 2.0 closely. Both are implemented directly on top of desktop OpenGL, so you must have support for at least OpenGL 1.5 on the desktop for the "moz-gles11" context and OpenGL 2.0 for the "moz-glweb20" context.

OpenGL was chosen as the base API for a number of reasons. It is a standardized, proven cross-platform API. 3D is a complex subject, and providing a fully-featured API will allow developers to create libraries and toolkits to abstract away the complexity in ways that are specific to their application, without being tied into a specific approach. Using a standard API also allows us to leverage the vast developer base out there who have OpenGL knowledge. OpenGL ES is also the standard for 3D on mobile devices, ensuring that any 3D content will be able to migrate to mobile devices as the web becomes more and more featureful on such devices.

The difficulty with using OpenGL is that the API is not very approachable for web developers; it's a huge jump going from the 2D canvas API to a full 3D API. My hope is that those developers who are familiar with OpenGL will be able to create helper libraries to make 3D content and manipulation more accessible, much like there are tools such as photo montage scripts that authors can use without having to understand the implementation details. Scene graphs and other frameworks would also provide great fodder for the performance improvement work going on as part of ES4.

I hope to propose the new API to an appropriate standards group after people have had an opportunity to play with the features and provide feedback on the initial implementation. As the core of the API is already a standard, a spec would only need to be written for the specific GL-Web integration points.

Installation

You must be running Firefox 3 Beta 1 (or a later nightly, though compatibility with the nightlies is not guaranteed). Install the extension here (Windows, MacOS X x86, and Linux x86 only). Then, to allow web content to access the 3D canvas contexts, go into the Add-ons Manager and select Options under the Canvas 3D Extension, and make sure that "Enable 3D canvas in web content" is checked. Because the extension exposes a new set of native system APIs, only enable it on sites that you trust as there could be security concerns that have not yet been addressed. In the future, enabling 3D canvas contexts will be a site-specific preference (like page zoom is now), so that you'll be able to set it on a per-site basis instead of globally. The 3D contexts are always available to extensions; combined with the drawWindow() API, extensions could use web pages as textures for interesting UI experiments.

You must also have a full implementation of desktop OpenGL 1.5 (for the GL ES 1.1 context) and, optionally, OpenGL 2.0 (for the 2.0 context) available. (A version of the canvas 3d extension will be available soon that supports software rendering using the Mesa GL library.) If support is not available, an error message will be printed in the JavaScript Console.

Documentation

Not a whole lot, at the moment. The interface files for the GL ES 1.1 context and for the GL ES 2.0 context should give you an idea of which functions are available; refer to the documentation for the equivalent OpenGL function on opengl.org. A few web-specific additions exist, such as "texImage2DHTML" for loading texture data from an HTML image element.

Examples

I've written a few examples:

Simple --- just the basics; shows how to create a moz-gles11 context and to use glClearColor/glClear to clear the viewport to a specific color.

Simple 2.0 --- just the basics again, but this time with moz-glweb20. Uses a simple fragment and vertex shader, and shows one possible way to include shaders in web pages (avoiding the need to specify long multiline string constants in javascript, which JS doesn't handle all that well).

kmzview --- Based on an earlier example by Philip Taylor, shows how to use the moz-gles11 context in conjuntion with E4X and the jar protocol handler to load Google Earth/Sketchup format kmz files (such as those available from the Google 3D Warehouse) and parse a (very limited) COLLADA model file and display it. This example uses the moz-gles11 context. It's a bit of a hack, and could be very much improved.

modelview --- Uses moz-glweb20 for displaying simple models that are specified as javascript objects, consisting of an array of vertices, faces, and normals. It uses the excellent Sylvester matrix math library, as well as a couple of tools that I've written that may be generally useful, such as glUtils.js and OrbitManipulator.js. I've also written a quick-and-dirty hack for converting OFF format files to js structures for faster loading.

Discussion

There is a 3D Canvas forums on the Mozilla Labs discussion site. I'll be watching this forum closely, and I encourage people to discuss their experiments and experiences with this canvas 3d implementation there. This is also the best place for general bugs, though eventually Bugzilla will be the right place for these.

Technical implementation notes

The current implementation is directly tied to your platform native OpenGL implementation. If there are bugs or other issues, these will appear in the canvas as well. In addition, for the 'moz-gles11' context, some parameter checking is performed; for 'moz-glweb20', currently very little is performed. In the future, checks will be added to ensure that the number of available elements in bound attribute arrays are sufficient to cover the consumption of each shader and the number of elements being rendered.

The current implementation requires pbuffers; all rendering is done to a pbuffer. When swapBuffers() is called, the contents of the pbuffer are placed into the canvas buffer. This is the most straightforward way to obtain cross-platform integration with web content; that is, a canvas element that's being drawn to using OpenGL will have the correct transparency blending and z-axis positioning occur with the rest of the HTML content on the page. The copy will often be the performance bottleneck; in future versions, OpenGL rendering will potentially happen directly to a native OpenGL view, removing that performance hit.

There is also very little control over the type of pixel format chosen by the canvas; in particular, there is no control over antialiasing or anisotropic filtering. In the future, these will be options provided by the Canvas 3D extension itself, and will apply to all 3D contexts.

Technical notes on the "moz-glweb20" context

GL ES 2.0 removes a number of convenience/compatibility aspects of OpenGL 2.0. In particular, all aspects of the fixed function pipeline are completely removed. For example, desktop OpenGL 2.0 provides a gl_ModelViewMatrix uniform that corresponds to the current MODELVIEW matrix. GL ES 2.0 provides no matrix stacks nor is the gl_ModelViewMatrix available.

Because the moz-glweb20 context is built on top of desktop OpenGL, a vertex program attempting to use gl_ModelViewMatrix will compile fine; but, because the matrix manipulation methods are not exposed, no matrix except the default can be used. Instead, the developer should use a custom matrix attribute to pass the required matrix to the shader. For matrix math and management, I've found the Sylvester matrix math library to be excellent. With a few additional methods, such as those found in glUtils.js in the ModelView example, Sylvester integrates well with the moz-glweb20 context.

So the machine where my blog used to live suffered a painful death --- I was even logged in as I watched the hard drive self-implode.  Of course, I haven't been all that good at keeping backups, so may have lost a post or two.  I did manage to also lose my theme, all of which is painful and annoying to recreate.  Hopefully I'll be able to yank the hard drive out and do some recovery in the next few days.

In the meantime, enjoy the default WordPress theme; I may even have something interesting to say in the next few days.