Introducing Scene Kit

Session 504 WWDC 2012

OS X Mountain Lion introduces Scene Kit, a high-level graphics framework enabling your apps to efficiently load, manipulate, and render 3D scenes. Gain an understanding of the Scene Kit architecture and see how easily its capabilities can be applied to a variety of application genres. Learn the objects that define the 3D scene, see how to apply animation, and discover how to use Scene Kit with related OS X technologies such as AppKit, Core Animation, and OpenGL.

Thomas: Good afternoon everyone and thanks for coming. My name is Thomas Goossens and I'm going to talk about Scene Kit. What's Scene Kit? Scene Kit is a new framework on Mountain Lion and it's role is to simplify the integration of 3D in applications. There are many use cases for 3D applications for instance, sometimes it is used in 3D user interfaces like the Book Template chooser in iPhoto or Cover flow in iTunes and Finder or [inaudible 00:45] etc. Slide shows and presentations are another use case for 3D to make the animation nicer and more impressive. Data visualization for scientist applications or for the 3D chart in keynotes which is another use case for 3D. Games obviously where 3D is commonly used there.

For these use cases, OS 10 provides some low-level APIs which are OpenGL and its associated technologies. These APIs are very highly performant, they have a very strong foundation, you can do a lot of things with it. However, since they're low-level, they require a lot of advanced skills in computer graphics. If you want to do some relatively simple things, you will have to write quite a lot of complex code. That's why we introduce Scene Kit today.

Scene Kit is a high-level API that is built on top of OpenGL and that operates on a scene graph. This is the key difference with OpenGL. Open GL is a poor-schedule API where for every frame, you call some draw commands, set some states. You do this for each object frame per frame to handle your scene. With Scene Kit you describe a scene once. You change some properties of your scene when you want but then you let Scene Kit manages everything and manages rendering for you.

Scene Kit is an objective-C API. You can use it for all the use cases I mentioned before. It is not tied to a particular use case. It's integrated with Cocoa and Core Animation for its seamless and simple integration into applications. What can you do with Scene Kit? Scene Kit essentially allows you to do three things. You can load a scene from a file, you can manipulate the objects of your scene problematically, and finally you can render that scene onto the screen. Let's focus on the first part for now, loading a 3D file, and see why it's important to work with three files.

The first reason is because it's easier to create complex scenes using 3D software. I mean complex geometries or complex trajectories are usually created with dedicated tools. The second reason is that the more information you will put into your 3D files, the less code you will have to write and so this will minimize the number of bugs in your code. The last reason is that working with your 3D design allows you to work with artists. The artists work to build the nicest 3D scene on the site when you can focus on your code so work in parallel with artists.

Scene Kit allows you load 3D Scenes from DAE documents. DAE stands for Digital Asset Exchange. It's an external-based file format. It is supported by most of the major 3D software like 3S, Max, Maya, Modo, [Cinema, Fody 04:29] etc. It is adopted in the industry for instance it is used [Sketchup 04:36] or more recently it was adopted by iBooks and iBooks also for their 3D widgets. A DAE file can describe a lot of things in your 3D Scene. It can of course describe all the geometry and formation of your 3D objects, but it can also describe some animations or [inaudible 05:01] references to images or the lighting of your scene. Different point of views to render your scene and some advanced features like skinning and morphine.

Loading such DAE file is straightforward with Scene Kit. It can be done with just one single line of code. Typically we'll start by getting the URL to your DAE document using NSBundle. Once you have your URL, you can load your scene with URL options error on the SCNScene class. This will load the DAE file and [wait on 05:41] the new instance of SCNScene. Once you have your scene loaded, you may want to render it for instance, in a view. To do that Scene Kit provides a SCNView object. A SCNView is a subclass of NSView that has a capability to render a 3D Scene. Just allocate a SCNView and then set your scene previously loaded to your view using the scene property of the SCNView.

Another way to create a scene view is to use Interface Builder. To do this just drag and drop a SCNView object from the Interface Builder library into your user interface and then you add on.

Let me show you those two first things in a quick demo. For this first demo I will start with a very basic example. Let's say I want to build a simple slideshow to show my images from the iPhoto library or whatever source into a nice 3D animation for my slideshow. I asked my artist to build a nice animation in his 3D tool and he sent me a DAE document. Here so it's my slideshow the DAE and since [inaudible 07:08] you cannot preview DAE files using Quick look or Preview. If I DoubleClick on the DAE file, I can open it in Preview and I can even play the animations to have an idea of what the designer made. Here I'm just having a preview of the DAE document.

You can see that the DAE file can describe a lot of things. It can describe the 3D objects that are my photo frames. It describes all the lighting information, the movement of the cameras. Everything here is described in the DAE file. As an engineer I understand that I want to integrate that in my slideshow application and that I will have later to replace the two pictures by some photos that come from my iPhoto library for instance.

Let's see how to integrate that into an application. Here is the new ... make it slightly bigger. Here is a new application, a new project that I created from scratch so there is no color tool yet. The only thing I did was to link with Scene Kit in the build phases here. I added my DAE document to my resources so here you can see myslideshow.dae in the sidebar fixed code. The first thing I would want to do is to edit the Interface Builder file. Here is my window. From the Interface Builder library I can search for my SCNView that is here. I can drag and drop it into my window, make it slightly bigger.

The SCNView in Interface Builder comes with an inspector here that allows you to set some options. For instance, I can generically type the name of the file I want to load at launch. Here I want to launch myslideshow.dae. It autocompletes. I can also set out a few steps like I want the animations to play at launch and loop. Here if I run my application I see the 3D application playing in my application. This was done without having to write any line of code. Let's switch back to the slide.

You can render a scene into SCNView if you want to render that in a window with views, but it also provides SCNLayers or SCNLayer if you want to render a 3D scene into a layer tree. SCNRenderer if you want to render the 3D scene into an arbitrary OpenGL context. That's for loading and rendering a scene. You may want to manipulate the object in your scene towards some interactivity towards that. That's what I'm going to show now.

First, Scene Kit allows you to access all the objects and all the property of your scene so you can of course move, scale and rotate the objects in your scene. You can also add some animations, change the colors or change the textures, adjust the lighting etc. To do this, you need to know what are the objects, the data structures that describe a 3d Scene and that is what I'm going to show now.

A 3D scene is represented by the SCNC class and the SCNC class as a tree of nodes. A node is represented by the SCNNode class. The tree of node is similar to a tree of layers and animation or a tree of views in the [inaudible 11:12]. There's a position, rotation and scale that define a 3D coordinate system that is relative to its paramount node. It's exactly like a layer that has a frame that represents a 2D coordinate system that is relative to its super layer.

If you want to, for instance, move a node in the 3D space, you just need to set its position to a given vector. For instance, here I move my node to the origin. I set this position to the vector 0, 0, 0. Managing the [yacky 11:50] of nodes is very similar to managing a [yacky 11:54] of layers of use with the usual addChild and removeFromParent message.

A node doesn't represent anything in the 3D space. I mean things that can be rendered on the screen. Adjust a location in 3D space on which you can attach some attributes. The attributes you can attach to a node are the following: You can attach a camera, you can attach a light or a geometry. I will explain each of them but you can already notice that you share one attribute to multiple nodes. This can be useful if you want to give an attribute to be visible at different location in the 3D Scene at the same time. Let's see the first one camera.

A camera is a point of view that can be ... that is used by the renderers. By the SCNView, SCNLayer or SCNRenderer. The situation here shows a single scene on the Scene Kit that's rendered from different point of views. A camera also has a couple of parameters like the field of view for instance if you want to increase or decrease the perspective effect.

Second attribute you can attach to a node are the lights. There are different kind of lights. For instance, the ambient light is a light that illuminates in every direction and so it illuminates equally all the objects in the scene. An Omni light originates from a single point. A directional light illuminates in a given direction and a spotlight illuminates ... originates from a single point but spreads outward in a calm direction. Lights have a couple of parameters like the itineration distances, the intensity, the color, etc. One thing important about light is that it illuminates the entire scene and not only the node it is attached to.

The last attribute we can attach to a node is the geometry and the geometry represents a surface that can be rendered on the screen. The surface is made of triangles that are connected together to build a surface. The triangles are connected to vertices. Vertices are the normal that indicate the direction of the surface. Geometry may have some UVs sometimes named Texture Coordinates that tells how an image should be mapped onto the 3D object. Last the geometry as an airway of materials that defines the finer appearance of the surface. Not that it is an airway of materials and not just a material because sometime a geometry is split into several elements and each element can have a separate material.

Now the materials. The materials also defines the appearance of the surface. It is represented by the SCNMaterial class. The material is made of eight properties. A property is represented by the SCNMaterial property class. Each property can have color or an image. The eight properties of the material are the following. The combination of the settings of these eight properties produce the final appearance of your surface. For instance, the diffuse here can be considered as the base color of your material. More precisely it's the color that the material will reflect when it is hit by light. It can be an image like here or it can be a simple color if you set an NSColor or CGColor red.

The ambient is the color that the material reflects when it is hit by ambient light only. It can be again an image or a color. The specular and shininess controls the specular highlight of shiny materials. The shininess controls the shape of the specular highlight. Reflective is an image that is used as a reflective environment for your material. Transparent property allows you to set ... to define the transparent areas of your material. Normal property allows you to set the normal map sometime called bump map which is a popular technique to add some details to your surface without having to add more polygons.

The multiple property allows you, for instance, to set some light maps or shadow maps which is a popular technique that designers are familiar with. Last, the emission is an emissive color that means that it can be visible even when the material doesn't receive any light. For instance here, the circle image is visible even on the sides of the cube that doesn't receive any light and it's because it is emissive.

With materials we can control how it react to lights and this can be done with some lighting models that Scene Kit report. With the constant lighting model, the constant lighting model doesn't react to any light, it's just an uniform color. Lambert react to light but has no specular highlight and Blinn and Phong have a different specular highlight.

Now to sum up and give an example of all these data structures. Let's say I want to change the material of one of my object. I will first get the geometry from my node then I will create a new material let's say red material by setting the diffuse contents to red color. Then I can see this material to my geometry as the first material by setting it to the first material property of my geometry.

That was the overview of all the data structures in Scene Kit. Now I hand over to Amaury to talk about the steps to create an application using Scene Kit. Thank you.

Amaury: Thanks Thomas. Hi, my name is Amaury and I think it's time for you to build your first app using Scene Kit. I said before you will use DAE documents as I described in [inaudible 18:40] these files and I said before Preview and Quick Look have allowed you to open and place [inaudible 18:49]. For many reasons, viewers are not sufficient so today we're introducing a new Scene Kit editor.

The new Scene Kit editor is built into Xcode. It allows you to have a deep comprehension of your 3D Scene. This is really useful when reading with files you don't know either downloaded from an online library or simply handed by our artist. First, the editor allows you to have an idea of your Scene graph. Checks the name of your nodes, check whether attributes are and how they are positioned relative to each other. It allows you to preview animations and test performances.

We wanted to make it really useful, really easy for you to collaborate with your artist. We think the editor will be at the heart of your typical workflow. Artists will design objects in the [inaudible 19:45] authoring tool, infuse them with great materials and lights and cameras to your scene. Using the editors they will be able to have a preview of how they scene will be rendered in your app. Since they are artists and since they know what looks good, we wanted to provide them with the tool to make fine digital adjustments. The editor allows them to tweak materials and by interacting between the editor and the authoring tool, they will be able to craft the perfect-looking scene for your app and then you as a developer take over.

You will have the scene right within all your other application files. By interacting between the editor and your code, you will be able to efficiently add the programmatically here that will make the scene dynamic and your app great. Let's have a demo.

Starting from Thomas demo let's have a look at the scene. The first thing you see is a view part. It gives you a preview of what you will get in your app. On the left side we have the Scene graph. Let's take a look at this node, the [third 21:02] node. I can tell it has geometry attached to it because of this little [21:08] icon. The node just below has a camera attached to it and somewhere in [inaudible 21:14] it has a light. Let's have a look. There are three nodes and each of them has a light attached to it but let's get back to the frame.

In addition to the inspector you're familiar with, Scene Kit has three new of them: the node inspector, the node attributes inspector and the material inspector. The node inspector allows us to inspect the many properties of the node. [Inaudible 21:46] I could change this name to 41. I could also change its rotation to whatever I think that suits. Let's go back to the lights. I want to input the scene because I don't like this part. Using the nodes attributes inspector, I can see that this one is a spot. I'm happy with its color but not with its illumination code. I can easily tweak its outer angle to make it. Thank you.

Now I have the scene that I think looks perfect. What I want to do is add some programmatically here to change the pictures that are displayed. To do this we'll probably have to tweak materials. Let's take a look. Oh, this one is three materials: one for the wood, one for the paper frame and one for the picture. We have a reflective map for the glossy effect and the actual picture is setting are different property. That's all we need to know towards the programmatically here. First, we have the name of our node, then second material and then diffuse property. I also have to change the ... We also have to change the picture displayed at the back so let's fill in this [guy 23:48].

It's good to AppDelegate. The first thing we will want to do is have an entry point to our scene. This will be done by accessing the root node of our scene self.sceneView.scene.rootnode. We have idle outlet set through the scene which is named sceneView. On this full node we will called, the childNode we've named recursively selector to retrieve our node Photo1 and we will do the same for the horizontal frame.

The next thing we want to do is retrieve the materials. We'll take the second material for the vertical frame. We'll do the same for the horizontal frame. Let's add some debug code and let's display a blue picture and a green picture to see if it works. In My App Resources, I have these pictures. Now I want to show this one, this one, this one and this one. Let's get with of this [inaudible 25:20]. The way we will display the new images is by having a simple index, build and name for the image, and set the diffuse property of our material to be that image. Of course we do the same for the horizontal picture. We want to loop, that is to mean we want to display pictures and when they are at the back of the scene we will change them so that the user doesn't notice anything.

Let's go back to our scene and check for its duration, 7:33. Our animation direction will be 7:33 and we will set up a timer [inaudible 26:29] that swaps images. Because this is for the front most frame, we will have to add a slight delay to change the images when they are at the back. With this at the middle of the animation duration. Of course, with this for the second picture as well. The pictures we're used to seeing, and the new pictures. Thank you.

That's how easy it is to build a dynamic app and dynamics [inaudible 27:23]. Let's have a quick recap. You will start by retrieving nodes using the names that you can check in the editor. You will retrieve the materials and finally you'll tweak them to make them look great. With that I'm calling Aymeric to talk about more advanced features in Scene Kit. Thank you.

Aymeric: Thanks Amaury. Hi everyone, I'm Aymeric and I'm going to show you how to go further with Scene Kit. First, I will talk about one of the many feature of Scene Kit, animations. Then I will present several ways to customize its rendering using OpenGL or GLSL. I'll browse through the various parametric geometries Scene Kit has to offer for constructing programmatically your Scene. Finally, I will review the integration between Scene Kit ... the possible interactions between Scene Kit and Core Animation. It is crucial for [inaudible 28:35] to provide a solid animation system.

People expect to see thing move and change fluidly in 3D and Scene Kit does provide an amazing animation system. Almost every property of every object are animatable. This will be done in two ways. Either using implicit animation. By implicit I mean that no actual animation object are created. As you will see in the core example later, you just have to change the animation ... the property you want to animate.

The second way is using explicit animations. This time an actual animation object is created and played when the time is right. I want to stress out that Scene Kit uses the same programming model as Core Animation using the adoption for those of who that already knows it.

Here is an example of an implicit animation. First, you have to begin a transaction. Please note that this is an NCNTransaction and not a CATransaction but it has exactly the same API. For example, if I want to set its duration, I use the setAnimationDuration selector. Change any property on any object you need and finally, commit the transaction. The animation will start right away in its own stride and its own pace. You can also add a completion block to be notified when your animation is complete or has been interrupted.

Scene Kit also supports explicit animations. This time explicit animations are actual objects that are created upfront and played at your convenience. For example, animations created with [inaudible 30:31] tools such as Maya or Max and exported to DAE files, are loaded as explicit animations. Scene Kit supports three kind of explicit animations. The simplest one is CABasicAnimation. It will allow you to animate one property of one object at a time. We also have KeyframeAnimation. This time instead of just providing a finer target, you will provide intermittent values along with key times.

The last type of animation supported is the AnimationGroup. It will allow you to synchronize multiple animation on multiple properties. You can see an animation controlling the position along with another controlling the opacity. Let's see an example of an explicit animation creation [on usage 31:31] it was a CABasicAnimation. First you have to create your animation targeting the property you want to animate. You have the opacity again. Configure it by setting its duration but you could also set its begin time, its speed or timing function if you want. Set the destination value for your property.

Finally, set your animation to the node you want to animate. The animation will start right away but always in its own thread. You've seen several ways to animate your objects on their properties. Let's now talk about rendering.

Thomas talked earlier about all the different rendering aspects of Scene Kit covering different types of lights and the large number of material properties. While we made the Scene Kit Renderer as complete as possible, we know that it won't suit everyone's needs. Therefore, we provided three different ways to you guys to inject custom OpenGL on GLSL Code inside Scene Kit's Renderer at the scene, at the node, at all material level.

First of all on the constraints level, you will be allowed to inject code before and/or after the scene rendering using a scene delegate renderer. It will allow you for example to have an animated background outer layer 2D interface over your scene. Let's see how this will be done. The effect you just saw is a full screen [crowd 33:26] render with a custom GLSL shadow. In a setup phase, we created our [inaudible 33:32] and stored it in a vertexArray object along with vertex [inaudible 33:37] shadow that we also wrote. This is a selector that is called before [inaudible 33:44].

When it is [inaudible 33:46] we ensure that no vertex buffer object, no texture or no program [inaudible 33:53] so we just have to bind what we need. Here I bind my vertexArray object on my program and I don't forget to unbind it at the end of the delegate. Concerning over GL states, you will have to set them to your convenience but restore them to sync its default after a while. Here I don't want my background to alter the depths before so I disable the depths testing and restore it at the end.

Next, if you want to render something in the scene among others objects, you may want to use the Node Renderer Delegate. It allows you, for example, to create position special effect like particle system that similar to fire or smoke. Let's see an example of that. In this scene, the torch is rendered with such a delegate and everything else is rendered using Scene Kit's own renderer and the too cohabit nicely.

This is a selector that is called on each node that are using your delegate. The node is provided obviously along with the necessary transform needed to render the scene as the views projection matrices that are given in the arguments dictionary. Lastly, if you want to have a specific appearance for your materials, you can always provide custom programs to Scene Kit materials. This is done by writing your own vertex and fragment shadow. Here is an example of such an effect, a simple light funnel effect that could not be achieved using standard Scene Kit materials. Let's see how that can be done.

First, you have to create an SCNProgram instance. It is programmer's responsibility to load its vertex on fragment shadow code as an SString usually from the application bundle. Assign it to your custom programs and finally assign your program to the material you want to customize. This has the shadow codes of the effect you just saw. I won't go into the details of the shadow but I want you to notice that there are two special kind of inputs.

The first kind of parameters are the ones that are automatically set by Scene Kit. These are the geometry attributes like position, normals and texture coordinates and the transforms like the models of u and prediction matrices that are given as uniforms. To make this work, you have to bind your own GLSL variable with Scene Kit's own semantic. This is how it is done. For example, here we are binding the SCNGeometrySourceSemanticVertex with our GLSL variable a position.

The other kind of special inputs are the one that are not bound to Scene Kit's semantic. These are usually the custom parameter for your programs. This is the selector that is called for each one of them. Here we only have one the u_Fresnel factors that we set to some arbitrary values.

We've seen several ways to customize the rendering using OpenGL on GLSL. I will now talk about popular [inaudible 37:37] with parametric geometries. If you're not really at ease with the [inaudible mothering 37:43] tool or if you don't have one, you'll be happy to know that Scene Kit does provide ready-to-use and easy-to-use parametric geometries. It starts with a bunch of parametric primitives such as planes, cubes, cylinders, tubes, spheres and so on and are that really easy to use. Here's an example of them. I just create an SCNCylinder giving it a radius and a height adjust its desolation factor and then set it on the geometry in an [inaudible 38:18]. That's it.

Scene Kit also provide dynamic 3D text with the same text capabilities as OS 10. Here again really simple, just create an SCNText instance. Set the font using a standard NSFont object. Assign it, type in a text as a string or you can even use an NSAttributed string if you want to have different styles rendered like italic or bold. Thank you.

Adjust it [inaudible 38:58] and their materials because we have several materials for the front, back and extrusion and [tampering 39:05] part of the text. All of this is animatable and insert it in your scene and voila! You will also be happy to have ready-to-use reflective floor, infinite floor. Here again real simple just create an SCNFloor object, adjust its reflectivity on its falloff factor. Assign any kind of material you want. Here we will just want a black color floor and set it in your scene. Your whole scene will reflect on it with nothing else to do.

Lastly, you can create your own meshes by providing position, normals and texture coordinates. It can come in handy if you have your own mesh ladder if you want to do some data visualization application.

One last thing about Scene Kit I wanted to talk about is its tight integration with Core Animation and it works both ways. That means that you can render a scene inside a core animation layer and the other way around. That you can have a Core Animation layer used as a texture. Let's see that through some samples. Here I create an instance on an SCNLayer and assign it a scene and then insert it in an existing Core Animation layer tree. As SNLayer is a subclass of a Core Animation layer, I can use any effect on it. Here you can see a white border layer on a drab shadow but I could have also use any core image filter.

The other way, here I have an animation of multiple Core Animation layers and I set this layer tree as a diffuse content of my material. Scene Kit will automatically create a texture and update it only when it has to.

I will now illustrate all of these features we've seen so far in a demo. For this demo we wanted to create an example of a 3D user interface displaying photo albums in 3D photo frames. You've already seen the photo frames in the previous demos but this time we cloned it multiple times to reflect the events in the iPhoto library. The frames are arranged in a radial layout among a reflective floor and there's lit by several lights one of them is casting shadow. I can browse through them by swiping or by clicking on it using the Scene Kit picking feature.

On the top of the window there is a search tool that will allow me to filter for my friends. For example, there. Using implicit animation, the visible set of frames have been reduced in a smaller circle with just a few line of codes. Let's filter some more. There again using implicit animation, the entire layout has changed into a coverflow-like manner and I can still swipe through it smoothly. Okay, one more filtering. Yet another layout.

Here you can see that the lighting and the camera position also change always using implicit animation. I can switch between my front and back frames using explicit animation because of a more complex movement. This is the photo album I wanted to show you. As we are using a Core Animation layer as a [inaudible 43:28] it enables transition like fade and swipe and it also enables movie-playing with just a few line of code. By the way, this is my son and boom.

One last thing I wanted to show you with this demo is the custom rendering facility. Not this, sorry. There, here you go. Here you can see some fireflies that are rendered using the non-rendered delegate that I just talked about. I also have lights attached to these nodes. You can see that it impacts the lighting of the scene. That's it for the demo.

Let's do a quick recap. You've seen how to give life to your scene using animation. It's really easy, they're really fast so use them extensively. We've also seen how to customize your rendering using delegates and custom programs. Use them wisely. We've seen how to populate easily your scene by using the provided parametric geometries. Finally, how to get the best of both worlds by using the tight integration of Core Animation.

For more information, please contact our evangelist, Allan Schaffer and Mike Jurewitz. You can find online our reference API and we have set up a Scene Kit dedicated forum on our devforums website.

Apple, Inc. AAPL
1 Infinite Loop Cupertino CA 95014 US