Update: the latest extension is now available for Firefox 3.1b3; view this post for more details.

One of the projects that I've been working on sporadically over the past two years is figuring out how to bring some baseline 3D capability to the web.  I wanted to bring a 3D API to the web which would provide access to a set of forward-looking 3D capabilities.  Web toolkit developers would then be able to build on top of these to provide a nicer and domain-specific interface for web application developers.

From looking at the available 3D APIs, I settled on exposing the OpenGL ES APIs through an HTML5 Canvas context, enabling access to OpenGL from within Javascript.  This has a number of advantages: there already exists a large group of people who have experience with OpenGL, and removes the need to go through a potentially long and drawn out API design phase.  This project is going to gain some steam soon, and I wanted to go through and describe some of the design decisions and challenges in the current prototype implementation.

OpenGL ES was designed to provide a subset of the full OpenGL API to embedded systems.  The subset chosen removes much functionality available in duplicate form, and removes infrequently-used features of OpenGL.  OpenGL 2.0 takes this a step further, and removes the entire fixed-function pipeline, resulting in a purely shader-driven API.

The original version of the Canvas 3D extension provided both an OpenGL ES 1.1-based and 2.0-based context.  Recently, I've scrapped the 1.1 context for a few reasons.  Both desktop and mobile hardware has moved quickly to having OpenGL 2.0 capabilities, especially on devices where a full web experience is possible.  (One notable exception to this is Apple's iPhone, which supports OpenGL ES 1.1 only in its current incarnation.)  It also doesn't make sense to introduce a forward-looking capability to the web, and then force developers to choose between two very different APIs, one of which is already heading out the door.  In the future, only the OpenGL ES 2.0-based context will be available.

The core problem is twofold:

1. How do we expose the OpenGL ES API, which is heavily tied to C and to C conventions, to Javascript in a way that feels natural?
2. How do we create a security model around access to fairly low-level graphics primitives to make them suitable for exposing on the web?

API Binding Principles

Edit: since writing this, it's become clear that much of the "convenience" functionality is actually both a performance hit (on the C++ side) and a hindrance to those who are familiar with the OpenGL APIs.  So, most of the convenience functions have been removed, as described in this post.  However, they can still be reimplemented in pure Javascript, calling the correct native functions underneath.

It's important to note that we're not designing a new 3D API here; the concepts, methods, and behaviours are taken directly from OpenGL.  However, the OpenGL API is constrained by the capabilities of the C language.  For example, there are over a dozen methods in the glUniform family, with suffixes like "3f", "4iv", etc. to indicate whether the function takes scalar arguments or arrays, how many elements to set, etc.  In Javascript, this would most naturally be expressed as a single "Uniform" call, with a varying number of arguments.

The current proposed binding can be seen in table form here (updated for 0.4.1), with the base OpenGL ES 2.0 function on the left and the Javascript equivalent on the right.  Where possible, additional functions have been introduced that remove type and size suffixes to provide an optional natural fit, and getters return the most natural type for the requested information.  Some examples:

• Removed. VertexAttrib[1234]f{v} becomes a single vertexAttrib call, taking the attrib index and between 1 and 4 arguments, or an array of 1 to 4 elements.
• GetIntegerv, GetFloatv, GetBooleanv, and GetString become a single getParameter call (there might be a better name here, but we can't use "get"); based on the given parameter, it will return the most appropriate type that represents that parameter.
• Complex getters, such as getActiveAttrib and getActiveUniform return their data as a Javascript object: getActiveAttrib(prog, index) -> { name: "foo", type: 0x8B52 /* GL_FLOAT_VEC4 */, size: 1 }.
• GenTextures and GenBuffers return an array of texture/buffer names directly.
• DeleteTextures and DeleteBuffers take just an array; the length of the array is implicit.

However, some complications arise, largely around the (few) uses of void pointers in the API:

• BufferData (GLenum target, GLsizeiptr size, const void *data, GLenum usage) doesn't translate well to Javascript as-is, because of its reliance on a generic void pointer.  The actual type of data isn't known until the buffer is actually used.  Instead, in the Javascript binding, this function loses the size parameter (because it's implicit), but gains an elementType parameter to specify the type of each element in the buffer: bufferData(target, array, elementType, usage).
• GetVertexAttribPointerv is currently unsupported; we have no way to return a useful reference to the pointer value to Javascript.  However, with an additional array wrapper that could be shared between C++ and Javascript, we'd be able to return an object here.

There are also some HTML-specific additions:

• texImage2DHTML (target, DOMImageElement element), and texImage2DHTML (target, level, DOMImageElement element) both take the DOM Image element and use it as the source for a TexImage2D operation.  The dimensions and pixel format of the image are used.

Security Considerations

More to come in this section as we gain experience with users of the API.

There is no bounds checking in the core OpenGL API.  If you bind a vertex array with only 4 elements in it, and then attempt to draw a few hundred triangles, OpenGL will happily follow you off that cliff.  This is, of course, a problem when these capabilities are exposed to untrusted web content.  An OpenGL for the Web implementation must not allow these types of overruns to happen.

The Canvas 3D extension takes the most straightforward approach; by examining the vertex attributes of the currently active shader, it can deduce the minimum number of values that must be present in each of the currently bound buffers to draw a given number of primitives.  Drawing with an index buffer complicates this somewhat, but at its simplest, each buffer must have enough values to fulfill the highest index value present in the index buffer (or rather, in the range of the part of the index buffer that's being drawn).  If these conditions are not met, a Javascript exception is thrown.

Performance Considerations

More to come in this section as we gain experience with users of the API.

Performance of a binding at this level is tightly coupled with the performance of Javascript.  Luckily, this has recently been an area of focus for browsers, and continues to be so.  There are many areas in which this can improve to better support 3D.  For example, in the current Mozilla implementation, conversion between Javascript arrays and native arrays is expensive; as such, buffers should be used as much as possible for any data that is expected to be reused.  This won't always be the case, and we're working to figure out how to keep data in native form even in JS, to simplify the transfer between JS and native code.

Also, because of Mozilla's focus on tracing and JIT compilation, some optimizations will become possible while on trace — for example, embedding direct calls to native OpenGL functions will become possible on trace, to avoid all the function dispatch machinery and get even closer to native code performance.

Future Directions

The current effort only solves the most core part of bringing 3D to the web — access to 3D rendering capabilities.  There are, however, other important aspects: asset loading, dynamic streaming, sound, etc.  All of these can be currently implemented in Javascript (for example, using COLLADA and AJAX, or using a custom ASSET format; using <audio> for sound, etc.), and we expect that some of these capabilities would become standardized and make it into the browser core based on their usage and popularity.

Extension Availability

A prototype extension implementing the current take on Canvas 3D will be available shortly; it will work with Firefox 3.1b3, and a later version may work with Firefox 3.0.  I'll update this page and make a post on my blog when this is ready.