« My Blog | Main | JOGL -> Java OpenGL »
February 16, 2005
<Media>Units
When I bought my G5 workstation this fall, I also signed up for a membership with Apple's Developer Connection that gives me access to pre-release versions of their next major OS release, Tiger. One of Tiger's most interesting new features is a framework called CoreImage, which allows developers to write image transform components that work in a similar way as Apple's AudioUnits work with audio. They are generic media transform components/plugins that are designed to easily plug in to OS X applications.
For example, Apple's Garage Band (an app that lets musicians play/record/loop songs and samples) has support for AudioUnit plugins built in. When you're playing a virtual instrument, you can add a chain of AudioUnits that add echoes or reverb, adjust the EQ levels of the audio, make the tone sound watery, just about anything you desire. The genius part is that AUs aren't an Apple-only thing, any developer can write AUs for use with OS X. This means I can write an AU that reads samples and graphs them in a pop-up Cocoa window, and then use that same AU when I'm using any product that supports AU filtering of audio streams.
Now that I've given you a taste of AUs, let me get back to CoreImage and ImageUnits. My developer token includes a pretty strict NDA, but I can talk about things that are public knowledge. One of the interesting aspects of CoreImage components is that the transformation kernels (the chunks of code that do the raw pixel manipulation) use a subset of the OpenGL Shading Language. Shaders have been used for a good while in the game industry, but only recently did a cross-platform language spec get finalized for writing generic shaders that work across multiple GPU platforms and operating systems.
A shader is basically a shim into the OpenGL rendering pipeline that lets application developers specify how to transform a canvas of OpenGL pixels. Shaders can do things like apply texture maps, generate shadows based on the defined light sources, transform the geometry of the image, and many other actions. Different video card drivers and rendering programs had their own shader implementations in the past, but using OGLSL (OpenGL Shading Language) makes the IUs more pluggable than if they used proprietary shading languages.
Another great thing about CoreImage is that ImageUnits that you write for transforming images can also be applied to transforming video using the CoreVideo framework. And because the shaders and the pixel processing pipelines that feed data into and out of the units are all based on OpenGL and optimized to move pixels around quickly, they are blazing fast.
I just started working my way through a book on OGLSL that will help me get bootstrapped with IUs more quickly. I've built some sample code and seen what it can do, but now I need to get dirty and implement the transforms that I require. BTW, CoreImage itself comes with 95+ ImageUnits that can be used to perform common image transforms, so unless you have some twisted requirements, they will suffice for most developers of image/video apps. Unfortunately, I have a few IUs I need to write that haven't been implemented by Apple, but this should be a fun learning experience.
Posted by djb at February 16, 2005 09:46 PM