The panelists discuss WebGL with Sean Griffin.

Special Guest: Sean Griffin.

 

  Dude, you really are at a well house. 

  Yeah!

  Hey everybody and welcome to episode 125 of the JavaScript Jabber Show. This week on our panel, we have Tim Caswell. AJ O’Neal. Jamison Dance.

  I’m here. [Laughs]

  Joe Eames.

  I feel like I shouldn’t be speaking now.

  [Chuckles] I know, right?

  [Laughs] Yeah, the silence is my acknowledgement that I’m here.

  I’m Charles Max Wood from DevChat.TV. And this week we have a special guest and that’s Sean

Griffin.

  I’m just really uncomfortable saying anything now. The creepy silence worked much better.

  Yeah.

  The one and only Sean Griffin. Do you want to introduce yourself?

  Yeah. My name’s Sean Griffin. I’m a developer for thoughtbot in Denver. And for the last year or so, I’ve been working on a very large scale WebGL application.

  Awesome. 

  What does that mean?

  So… [Chuckles]

  It’s enterprise. 

  It’s very enterprise. We’re…

  Is it cloud?

  It is cloud-based and it’s real-time. And I’ve never actually been able to use both of those buzzwords before.

  And he uses NoSQL and Node and Go.

  No. [Chuckles]

  No, none of those. That’s a little bit too web-scale for me. We want to scale back just a little bit.

  It’s actually the cutting…

  Is it Web 2.0?

  It’s NoHTML. That’s the cutting edge.

  No, it’s Web 3.5.

  NoHTML?

  Yeah, the cutting edge.

  Oh, I like it, Jamison. You should coin that.

  NoHTML, I like it. [Laughs]

  I love NoHTML so much now.

  So, tell us about what it means to, you called it a large WebGL application. Is that what you called it?

  Yeah, large just being size of the codebase relative to most startups. So, the idea was a couple of martial artists who are very enthusiastic about it wanted to improve how lessons were done outside of the dojo. So, they got some of the masters from various schools of martial arts into a 3D motion capture studio. That goes off to an animation studio to get touched up. And things like fingers, motion capture can’t do really well. And when you’re talking about martial arts, the subtle movement of the fingers is really important. And so, those get animated by hand. And they send that off to us. 

And basically, we needed a way to display the 3D animations in the browser and have it be interactive. It used to be you’d go with something like Flash or Unity or something that required a plugin. But WebGL’s really starting to gain some traction and be well supported enough by all of the various browsers that it’s actually feasible to use. So, the benefit is you can do your 3D, obviously games are the more traditional use case for 3D, but yeah, you can do 3D stuff in the browser without having to have the users install a plugin. And it just works. 

  I love that. That’s super cool and very interesting. I do have a recommendation for you. Next time you tell that story, instead of starting off with a couple of martial artists, start off with a couple of ninjas.

  [Laughs]

  Code ninjas.

  Well, I can see them though. So, I’m not sure.

  Ah. Yeah, it doesn’t really qualify.

  I feel like most of the topics we’ve talked about on the show, I usually know something about. I know nothing about WebGL.

  [Chuckles] I know less than Jamison.

  I know nothing about OpenGL at all. Can you give us a brief introduction or explanation on it, just for some background?

  Sure. Yeah, absolutely. So, WebGL is actually a thin wrapper around OpenGL 2.0 ES, which is basically the newest version of OpenGL that is supported by almost every device. Newer devices are up to I think 4.5 now and have all kinds of really cool and helpful things that we don’t get to use in the browser. And basically it is a library for drawing 2D graphics on the GPU. So, basically you give it whatever numeric inputs you would like. It loops over those and calls a function that’s written in a language called GLSL which is a math language that runs on the GPU. That function gives it, essentially equates to the coordinate of a pixel. And then once you’ve got, that function’s been called three times, it will loop through every pixel that’s going to be inside of that triangle and call a function which is meant to return a color. So, it’s about as low level as you can possibly get for doing graphics. 

And the biggest thing that you have to do most of the time when you are working with OpenGL is you’re most likely these days not drawing 2D graphics. You’re drawing 3D graphics. But the GPU and the computer don’t actually know anything about 3D or how 3D works short of there’s something that you can use to your benefit which essentially works like the CSS z-index property. And so, you have to do a lot of math basically to convert all of this 3D data and an image file into 2D graphical data.

  That seems like a very different paradigm from the normal draw boxes inside boxes inside more boxes of the web.

  Yeah.

  So, my Node/Knockout intro a few years ago was I added WebGL to Node, which coding-wise is very straightforward because the WebGL API’s dirt simple. But yeah, it makes canvas look like cotton candy at an API level versus WebGL. If you want to draw a line or write some text, it’s a whole other world.

  Well, and it’s so stateful as well. It takes, assuming you have all of the data that you need and it’s already in the right format and you’ve written all of the code, it takes between seven and nine API calls just to populate a single variable. It really is reminiscent of doing a system call in X86 Assembly where it’s just, you put some bits in a certain buffer and then you just go and call the single generic function and hopefully it does the right thing with those bits in that place.

  Kind of crazy. 

  Did you have a background in OpenGL before this or did you just learn it to use WebGL?

  Not a whole lot, passing knowledge from years and years and years ago. And my wife went to school for 3D animation. But that was more from the production side of things than the technical side. So, a lot of it was just picked up for this project, specifically. We tried out various libraries and had this weird bizarro world where every time we tried to use any sort of abstractions or libraries to help make this easier, we would have to drop down lower and lower and lower, because it just turned out to have something being broken that we couldn’t figure out how to fix or work around, or it just didn’t provide the abstractions for what we were trying to do. Because we’re displaying basically like a video player, almost, 3D data, as opposed to your video games where normally just things are a lot more dynamic, whereas we want controls that are closer to Windows Media Player.

  So, this is all in JavaScript, right?

  Yes.

  Okay. I was thinking, “Wow. This is crazy.” Do you do other types? Do you do regular web stuff in JavaScript as well?

  Oh yeah, absolutely.

  Does it feel like you’re in an entirely different world even if you’re using the same language when you’re doing this?

  Yeah. It happens to be in JavaScript, but the APIs are almost exactly one-to-one with C API. 

  Really?

  And while you may be writing the WebGL code in JavaScript, the actual shader code is a C-like language that runs on the GPU. So, the actual hardcore math is not running through your JavaScript interpreter. That would be way too slow. 

  And integers are helpful.

  Yeah, right.

  You mean, [inaudible] it’s not all floats?

  I hear we’re going to get those one of these days.

  What kind of abstractions do you use to draw stuff with OpenGL? I’m just struggling to understand it in my brain. So, you went through the step of drawing a bunch of triangles. I’m sure that’s not the level that you always deal with [at that], right?

  It’s the level we ended up dealing with more often than we’d like. Yeah, it gets built up. The whole thing starts with basically an animation file. And that’s where we get all of the triangle data. Since we’re not doing anything with video games, there’s not a ton for us to do other than do the calculations for where the skeleton of the model’s going to be for a given point in time, and then lots of math around that to get a matrix that we can use to translate the position of… Have you’ve ever seen in videos where they’re doing 3D production and you see the shot of the screen and it’s got the guy standing out with his arms in a T?

  Mmhmm.

  That’s where all of the data that comes in actually has at a position. And then that gets applied to various joints on the skeleton with varying weights. And basically with lots of linear algebra that I just found from textbooks and Wikipedia and don’t really understand [chuckles] how any of it works…

  [Chuckles]

  Because I didn’t take linear algebra in college. So, basically that’s the biggest math. 

  So, I was going to say basically you did the engineering form of linear form of linear algebra? I ended up taking both in college. And in engineering school, basically they’re like, “Here’s the formulas. Go learn how to use them.” And it’s all calculators and formulas. And then I switched majors to CS. And in the CS version, they’re like, “Alright. Go learn all the theorems and your test will be on memorizing theorems.” So, you learned the engineering version of linear algebra.

  Yeah, well…

  The very practical, get it done.

  And that was actually the frustrating part of digging into this, is that there’s not as many resources as you would like. And especially when you come at something with the mindset of, “I want to understand how this works and how to really dig into it and how to be able to know how to tweak things or play around with it, or especially be able to test it.” If you go look at how people implement matrices, they do all kinds of clever little things that are a fewer object allocations or are superefficient, as opposed to representing in the code what actually happens mathematically, which is what we ended up doing just because we wanted to make sure we really understood everything that we were going to be able to. 

And it was actually surprising to us. We wrote the entire codebase incredibly naively, didn’t really care about performance and just figured we’d profile and find our bottlenecks later. And the number of things that actually were performance bottlenecks was surprising, especially given looking at other libraries like three.js or GLGE, just seeing how many micro-optimizations they have across their entire codebase. And it was just shocking how many of those didn’t end up affecting us. We obviously don’t have every use case, and I’m sure there are cases where these performance optimizations they made were bottlenecks in other people’s codebases. But it really was for us a big example of just premature optimization is really a waste of time.

  Why is it triangles? I never understood that nor had anybody explain that to me.

  I actually don’t know either. My guess would be just because it’s the polygon with the fewest number of sides and every other shape can be expressed in terms of triangles.

  If I understand correctly, it’s because with a triangle, you can have any three points in threedimensional space and it’s still a plane. Whereas if you had four points, you could make these shapes that are no longer a plane. And so, it makes the math so much easier if they’re triangles, because any triangle is guaranteed to be flat, no matter where you put the points.

  Hmm.

  That would make sense.

  That does make sense. [Hums] That’s that ‘The More You Know’ song. I’m trying to do that. 

  [Laughs]

  So, you mentioned three.js. Why did you decide to write your own from scratch instead of just using that?

  We were actually using three.js until about three weeks before we launched. We got the final textures and lighting and flooring and everything from the animators. And we started putting that in. And our characters looked like they had gone and taken a dive in some canola oil and then decided to go wrestle around. And so, I assumed we were doing something wrong. And we started going through and verifying. Is this exporting wrong from this or when we’re converting from this file format to here? Is something getting messed up there? And eventually, we went all the way down. We traced everything through to the shaders. And lo and behold, all of our inputs were definitely correct, but the shaders that they were using turned out to be wrong. 

So, we started looking at the shader code trying to figure out what was wrong. And we’re using a pretty well-known lighting model called Phong shading, which is basically anytime you have something that looks kind of shiny, it’s what gets used. And it’s a pretty well-known formula and we started looking. We found where the Phong shading was happening. But we didn’t actually see anything that looked even remotely like what that formula is supposed to be like. 

So, I went off and tried to basically learn more about, at the lower level, how WebGL works and how OpenGL works and how 3D at the GPU level works. And by the time we had, I had the prerequisite knowledge to try and figure out what was broken in those shaders, at that point I had rewritten so much of the code, so much of the functionality we’re using from them, it just made sense to go off with our own, because we had a lot of frustrations with that library. It’s heavily focused on mutability. It makes it very difficult for us to have a good test suite for our codebase. 

  Perfectly valid reason. I would probably do the same.

  Is that something you’ve open sourced or would open source?

  Yeah. We’re still deciding exactly how much we’re going to open source. But I’m working on extracting several libraries actually. I’m a big fan of the tiers of abstraction. So, right now I’m working on something that just takes the basic OpenGL APIs, or the WebGL APIs, and wraps them in something that feels a little bit more JavaScript-y and has objects to represent your data and what goes in there. And nothing as high-level as a vertex or a triangle, but just to make the APIs a little bit more pleasant to work with, and then start building shape libraries on top of that.

  Okay, so here’s the premature optimizer in me. Wouldn’t that make your GC go crazy, having all these objects instead of numbers?

  Well, the thing is where you do or don’t need objects can vary. So yeah, definitely having an object represent a vector is going to make your GC go absolutely nuts. That’s one of the few places we don’t actually have objects. But you can have, if you get away from, if you stick to methods or [computive] properties and get away from using arrays to represent stuff, you can have objects that internally store things in raw float32 arrays just as a flattened data structure, which is what OpenGL’s going to expect, and then lazily instantiate objects with more meaning if and when you need them in your CPU code.

  Nice.

  And the big thing is just trying to have lazily instantiating things that are more useful as developers but then in terms of how you store things internally at the low level, trying to have a straight pipeline from when you parse the file to when it gets to the GPU.

  So, I’ve always wanted to make a 2D canvas-ish sprite library using WebGL. But for some reason, I never could figure out how to set it up properly. That’s the hard part for me. I understand the pipeline and the vertices and the shaders. I love the shader language. I just never could figure out how to just say, take this PNG, draw them to screen here scaled and rotated. Have you done much at that level or is it always just 3D stuff?

  It’s always 3D stuff. But at that point, it’s actually very similar in terms of what goes into it. And most of the API calls you’d make are the same. You treat it as a texture. And then as long as you don’t actually do any of the math that transforms it into 3D space, you can apply the same sort of rotations. In fact, I think for image data, there might actually be something specific but I don’t know off the top of my head.

  Okay, so you just have a whole bunch of triangle pairs for your squares?

  Yeah. Essentially, you would think of it the same way you’d think of 3D but just without a z-axis. But all of the other API calls and math and basically high-level thinking that you would do are essentially the same, if it was just all on one z-plane.

  So, if I wanted to get started with WebGL, where would I go? Do I have to go learn OpenGL specifically or yeah, how do I start?

  So, there’s a site that was good for just getting the boilerplate out of the way, which I think it was called WebGL Fundamentals, which didn’t have a ton of information on it. But it did have a good, “Here is the absolute boilerplate you need to get your canvas tag and start drawing a triangle on the screen.” And once you get to that point, then you can just start looking at any, you don’t have to look for WebGL tutorials. You can also look for OpenGL tutorials using C++ or C which are much, much, much more numerous, because the APIs are almost exactly one-to-one. 

There’s a great, I guess it’s not really an article, kind of a site/very, very long blog article, called ‘Render Hell’ which I’ll post a link to that we can put in the show notes that had a great nontechnical overview of how everything works and what things are going to go through. And I highly recommend reading that if you’re interested in getting started in 3D. Or if you’re not, just because it’s a very fascinating look into how your GPU works, and it has cute little animated images. 

And then beyond that, I found the best source of information was just straight up looking for OpenGL tutorials. There is a Wikibooks site on 3D with OpenGL. And I just found that those tended to be more available than WebGL. Not a lot of people are writing about WebGL unfortunately.

  This looks great. This is super cool.

  And I’m also working on a series of blog posts, which is basically just, “Here’s all of the information I wish I had available to me when I got started with this in the format that I would have found it most useful,” and that’s coming out over a couple of weeks on the thoughtbot blog. 

  That’ll be super helpful.

  Yeah, it sounds like there’s a big hole that that could help fill.

  When I started on this stuff a couple of years ago, I had a real hard time connecting the dots between the WebGL tutorials and what was available in, or in the OpenGL tutorials and what’s available in WebGL, because a lot of the tutorials were OpenGL 1.0. 

  Yeah.

  And can you explain that, the difference?

  Well, yes. So, 1.0 to 2.0 was one of the most significant shifts. 1.0 used, it was called the fixed rendering pipeline, which the APIs were just completely and totally different. You would send instructions much more individually from the CPU to the GPU. As opposed to now, the APIs are much more focused on you populate buffers with large amounts of data, send them all off in a single go, and let the GPU do its thing without having to inform the CPU when it’s done an awful lot. 

That said, there’s actually a lot of stuff that was possible in OpenGL 1 that isn’t possible in OpenGL 2 and didn’t become possible again until OpenGL 3 or 4. For example, one of those that we recently had a lot of trouble with was transparency does not work as well as you’d expect in OpenGL. Because to make sure that your objects, your 3D models display properly, you enable what’s called a depth test which is where every time you try and draw a pixel, it looks to see if something’s already drawn in front of it, because you do give it a z-axis which works like I said, the z-index in CSS. But it doesn’t actually take into account whether or not what you drew there was transparent or not. 

So, if you draw a transparent thing that’s in front of your opaque things and you draw the transparent thing first, whatever was behind it won’t get drawn. And it used to be that you could set really easy functions to have it throw away anything that has an opacity of less than 1 and then draw everything, and then change that function to throw away anything that has an opacity of greater than 1 and draw everything. 

And you could still technically do that in OpenGL but it’s a lot more difficult and can have a lot worse performance, because you basically have to do it with if statements in your shaders, which surprisingly if is actually a really big bottleneck in shader code.

  Really?

  Yeah. It forces you to use some best practices. 

  [Chuckles] So, you mentioned at the beginning that you use OpenGL 2.0 instead of I think you said it was on 4.5 or something like that.

  Yeah, that’s right. 

  So, on the web, it seems like the strategy is implement cutting edge stuff in newer browsers and then polyfill that to older stuff. Do you have any understanding of why they only support the oldest, most common standard instead of supporting the newer standards for browsers that support it? Or I guess it would be GPUs. I don’t even know what it would be. 

  It’d be GPUs. 

  Okay.

  The reason is because we’re still not quite there [chuckles] with consistent support for 2.0. And if there’s only going to be one to target, the one with the best support’s the way to go. There is talk of WebGL 2, which I don’t know which version of OpenGL that’s going to be targeting. But that’s the main reason, is just we’re still missing a lot of support for OpenGL 2. So, starting on OpenGL 3 or OpenGL 4 is just going to take resources away from getting up to speed.

  That makes sense. 

  Also, the WebGL lines up with what mobile has, the OpenGL 2.0 ES. And so, you’re also coding to the same thing that Android has and iOS has, or at least had. They may have newer stuff now.

But that’s definitely the broadest support, was the 2.0 ES with the shaders.

  Yeah. And that’s definitely still the case. 3.0 supported the, some of them, in some places on Android. I think the next iPhone’s going to support 3.0. But 2.0 ES is definitely still very dominant. 

  So, you mentioned a little bit about some of the, well, it sounds like some of this is stuff you’re going to talk about in your blog post. But you mentioned a little bit some of the things you wish you had known already when you started. Can you talk a little bit more about that? So, I feel like most of the people that listen are going to be in the brand new, beginner, experimenting with drawing a little black triangle on the page or something. 

  No, absolutely. And that’s part of why I’m looking to get more resources out there just because [chuckles] it is so much work. So, one of the big things for me was that going through all of the resources out there, they were very, very pragmatic and very focused on, “Just write this code and then it’s going to work,” and didn’t necessarily start explaining why buffers worked the way they did, or why the math works the way it does. 

So, I think one of the biggest things was just looking at it from a perspective of, these tutorials are great if I’m going to write a few triangles or I want to draw a cube on the screen, but how do I make this effective and scalable for an application that has to do a little bit more than that? Especially when you start getting into some of the topics that have a lot less written about them, like doing skeletal animation or how to set up code that it can draw an arbitrary 3D object and you don’t necessarily know ahead of time all of the details about how you’re going to be drawing it and doing that in a reasonably composable way. I think monadic data structures work really, really well there. 

I guess a lot of it’s just going over things that there are already resources for but with an emphasis on how it works and how to test it.

  That makes sense, especially if it’s something unfamiliar like this. Just someone saying, “Do this and it works,” leaves you totally at a loss when it doesn’t work, which happens pretty often with those kinds of things [chuckles] I feel like. So, you have the idea of [inaudible].

  Yeah. I think a lot of it’s because most of this is written for video games. And just the game world tends to be very, very pragmatic. A lot of the people, not to rag on game developers and this is certainly not always the case, but a lot of game developers I’ve met have been, they wanted to make video games and programming was a means to an end. 

  So, my question is, can you bring in third-party artwork into this? Or do you just render it all with WebGL or OpenGL and draw it with triangles?

  I’m sorry, so bring in third-party artwork into the application that we built?

  Yeah, mmhmm.

  Yeah, so…

  Yeah, so you mentioned you had artists earlier. How do you use other people’s work to render it?

  We used a final format for a while that was based off of three.js’s JSON file format. Or actually, going all the way back, the original first version of the engine used a file format called COLLADA, which is an XML-based universal file format that’s spec’d by the same people who make the OpenGL spec. Unfortunately, it’s just a very large file format. And if you’ve ever tried parsing XML in JavaScript before, it’s not fun and it’s certainly not fast. So, we switched to a JSON-based format that was a fork of three’s format, and then eventually switched from that to a binary format. 

And basically, the big thing was we had to write an exporter from Maya, which is the 3D editing application that the animators use. And that exports to the JSON-based format. And then when that goes into the admin panel on our Rails app, we parse that and then write it out to this new binary format that we did for the application, and then read that in, in the JavaScript. And at the end of it, what we end up getting is just these arrays of numbers which are exactly in the format that we need to send them off to the GPU. So, we do very little processing there. 

Unfortunately, well both fortunately and unfortunately, when you have animators at all and they’re writing or making animations in some program and there’s not necessarily a format that already exists so that it just universally works everywhere, you have to make, you’re forced to make something that just works with any arbitrary animation because it has to work with anything that they throw at you. So, I guess yes. [Chuckles] That’s a very long-winded way of saying yes to your question.

  [Chuckles]

  Yeah. It’s interesting though, because I’ve played around a little bit with moving stuff around a canvas and stuff and that’s all 2D. And so, you can just basically import JPEG or something and then you just change its position and stuff. Where it sounds like this, you really need something that’s more in the format of, “Here’s how to render the shape of whatever it is that you need to put into WebGL.”

  Yeah. That said, you do still get to use one of the APIs that we’re actually used to, which is image tags. That happens to actually be the easiest way to load up your text or data.

  Oh, really? So, that’s just a repeatable JPEG or PNG or something?

  Yup. And then there’s just one of the APIs that’s overloaded a lot. It’s called, I think, data/text2d, something along those lines. It definitely has 2D at the end of it. And it’s got several overloaded signatures and one of them, it just takes an image tag along with a couple of other newer HTML5 file API formats. Though one of the things that we learned when we were trying to get a progress bar going for loading was that image tags don’t actually emit on progress events as they’re loading, which makes having a loading bar when you have a lot of image data basically impossible. [Chuckles] And so, what we ended up having to do was do a normal XML HTTP request with the array buffer data type. And then once we get that in, there’s an API that I think is pretty new, if it’s older I just hadn’t seen it before, that lets you take an array buffer and put it at a URL that you can access anywhere. It’s window.url.fromdatasource I believe. [Chuckles] And then we create an image tag and point it at that, which is then just a local file system URL. But the whole thing was a gigantic runaround just to be able to get progress event on an image file.

  I’m like, “Wow.”

  It’s a whole history of web development, right there. Just hacks upon hacks upon hacks to get things that you think should work.

  That’s so the truth, Jamison.

  Right. And then they get codified into the language.

  Yeah. [Chuckles]

  Yeah, it doesn’t matter how hacky it is as long as it’s standardized.

  Exactly.

  [Laughs] That’s true. And people like you write nice wrappers for them.

  Welcome to package management on the frontend.

  So, what kind of mistakes do people make?

  Oh, just Bower. [Laughter]

  Wait, what? I’m sorry. It isn’t a [inaudible].

  What kind of mistakes do people make when they’re getting into this, into WebGL?

  What kind of mistakes do people make? I think that…

  I’m assuming that there are some things that are easy to get tripped up on.

  Yeah. Well, so there are a couple of things that are definitely easy to get tripped up on. The biggest one is not telling WebGL the correct data type for whatever data you’re giving it. For example, if it’s a float and you say, “Actually, these are a bunch of shorts,” nothing’s ever going to blow up on you. It’s going to just do the wrong thing. And that can be very difficult to debug.

Another one is if you wrote your shaders wrong. There is no print statement. There’s no debugger. [Chuckles] When I actually did have to drop down to really try to figure out what was going on, it basically came down to replacing a bunch of work with constants and setting the color global variable to the float I was trying to work on and taking a screenshot of what came out of it and then looking at the RG and B values for that to be able to debug it. So, just yeah, I guess you have to be very methodical and consider a lot, especially when you’re working with shaders or when you’re working with the API calls. Because if you do it wrong, it’s unlikely to blow up on you and it’s really hard to track down exactly where it went wrong.

  When I was doing WebGL for Node, the actual C part’s very straightforward, because I’m just binding the C API to JavaScript and it’s a one-to-one like you said. But it took me 24 hours to write the library and the first 20 hours in, my screen was still black. Not a single pixel was where I wanted it. And I’ve been hacking at this for 20 hours and still didn’t get anything on the screen. And then I got a pixel and I was able to debug a little more because there was some output and it went much faster. But that first bit was really painful, because you get nothing.

  For me when I was learning, it was this progression where, starting with a triangle and then moving on to a cube and adding textures and adding lights and things would start to make more and more and more sense. And it kept on continuing to progress in that direction and then I got to skeletal animation. And everything just stopped making sense again, because just none of the math that was having to be done would make sense. It took, I think three or four days to get the guys to stop looking like what we now refer to as accidental dead space, fingers going through…

  [Laughs]

  Ten feet long, going through the mouth.

  Yeah, that’s always fun. Or you’re playing the video game and you walk through the tree and so you see the outside of the tree, and then you see the inside of the tree, and then you see the other side of the tree as you walk through it.

  Yeah, exactly, all kinds of weird glitches like that. 

  That’s just your [z-bug] for clipping the wrong place. When your matrices are misaligned, you get really, really weird stuff. But yeah, it looks similar.

  There are still places in our shader code where we’ve got a number that we had to multiply times 0.5 and none of us have been able to figure out at all why on earth we would ever, why on earth we’re having to do that, because none of the formulas specify, but everything’s broken if we don’t.

  Magic. I’m wondering. How in WebGL do you handle things like physics and lighting and stuff? Do you have a separate engine for that?

  Physics tends to happen on the CPU actually, not on the GPU. And you tend to do things like collision detection or how fast something’s moving, you do that on the CPU. And it would affect the whole mesh or the whole object in the world, and not necessarily individual triangles on it. 

Lighting is a really interesting one, because that actually does happen on the GPU. And it happens basically in the place where you’re calculating the color. And what you end up doing is determining how much of your types of light are coming from all kinds of directions. And you do math on that to come up with a numeric representation between zero and one of how intense the light is. And you add all of the different light sources together. And then what you do is you just, when you read the pixel from the texture file that you’re about to apply before you return it, you multiply that color times this number of how much it’s going to be lit. And essentially, that will become close to black as it reaches zero. And the end result is lights.

  So, the only thing that I know about lighting I’ve learned from just reading stuff about the history of id software and Doom and 3D rendering engines and stuff. I don’t know the technical details. But it sounds like all of the advances have been just these incredible breakthroughs and stuff. Are you breaking new ground every time you figure out how to light stuff or is it pretty formulaic and you just do what you already know how to do? Does that make sense?

  Yeah, that does make sense. It’s definitely very formulaic. We’re using, the two shaders that we use are called, or the two lighting reflection whatever you want to call it, formulas that we use, one’s called Lambert and the other one’s called Phong reflection. And they’re both very, very wellknown, well-established formulas that have been around for ages. Most of the big breakthroughs that happen on the technical side are less discovering new formulas on how to light things and more how to do some of the more expensive lighting calculations  efficiently. One of the biggest ones…

  That makes sense.

  That is always, always very expensive is shadows, because you have to render the entire scene again to do shadows.

  You mentioned shaders as couple of times as well. Can you explain what those are again?

  Yeah. It’s as little bit of an overloaded term. Sometimes, it just refers to, well I guess actually these are the same thing. But it can refer to how you’re reflecting light off of something. But then from a technical point of view, they are the code that you write that runs on the GPU in this GLSL language. 

  So, the shader is where you implement the Phong or the Lambert. And then everything else, you just stuff lots of numbers into it and then it does all the math.

  Yup.

  Okay, makes sense.

  And that’s ultimately what OpenGL is, is just a thing to let you pass some bits to the GPU and run some code there. 

  Awesome. Anything else that we should talk about here before we get into the picks?

  Well, my experience is mostly 2D stuff. I’ve actually avoided 3D. But as Sean said, WebGL is actually 2D at its core. It’s just used for 3D because it’s an efficient way to render 3D.

  Yeah.

  So, is there a real difference between thinking about the different 2D versus 3D?

  Well, the process for drawing is identical. You have a pixel and you have some colors and it’s just math. I try to imagine sometimes how on earth people originally came up with, “Hey, if I take this data and I run it through this really intense linear algebra, the end result is 3D.” It’s mind-boggling how people came up with that. But yeah, it’s really quite similar I think. And that was the biggest I think mind-blowing realization for me, was just that the computer didn’t know how to 3D. 

  Nope. It’s not built in. I did have a lot of 3D experience before with ray tracers where it’s entirely on the CPU but it doesn’t matter because it’s not real-time. And I’m amazed that modern GPUs can basically ray trace in real-time. There’s some pretty crazy stuff in there nowadays. 

  If you look there, there are some really crazy hacks to do all kinds of things that you would expect to take obscene amounts of time, exponential grown algorithms, and they do it in fractions of that. People come up with really clever ways to do this stuff really efficiently with good enough approximation all the time. It’s amazing. 

  Very cool. Well, I don’t know if we have any other questions. Should we get to the picks?

  Yeah, let’s do the picks.

  Alright. Jamison, what are your picks?

  I have two. One is called SentimentalVersioning.org. There is some debate about what semantic versioning is and whether it’s good or not. And Dominic Tarr wrote this satire on the whole thing called Sentimental Versioning where your version is a freshman’s English paper where you have to write about how it makes you feel. And it’s pretty good.

The second pick is Meatspace. It’s a GIF-based chat. It’s super fun, but it’s fun because of the people that are on there. So, you go on there and every time you send a chat basically, it also uses your webcam to take a GIF screen chat of you. So, you’re using words but also there’s visual medium to communicate. It makes people goofier, which is fun. So, those are my picks.

  Awesome. Joe, what are your picks?

  Alright. So, I’m going to pick first off, the TV show the Arrow. There’s about, I think two seasons of it. Awesome show about Green Arrow and there’s a spinoff coming up on the Flash this year from the show. And I’ve been trying to basically catch up. I’m way behind. I’m trying to catch up so that I could be ready when the new season starts up and the Flash spins off from it. And watching it, I’m really enjoying it. It’s a great show. So, I’m going to pick the Arrow. 

I’m also going to pick a show coming up that I’ve not seen anything. So, I don’t have a direct experience with it. But it’s the only show coming out this year that I’m actually interested and excited about, and that’s Star Wars Rebels, the cartoon on Disney XD. 

And in conjunction with that, a book was just released on Amazon called Star Wars ‘A New Dawn’, which is written about the characters in the Rebels show. So, I’m going to pick that book alongside with that. If you’re excited about it and you didn’t know that they’ve just published a book, there is a book out there. And you can go pick it up on Kindle or on Audible. And check it out. And those are my picks.

  Cool. Tim, what are your picks?

  Alright. So, I mentioned in the show that I had a hard time learning 2D and WebGL. Well, I stumbled upon this great library called Pixi.js which basically implements a canvas-like API for moving around sprites and scaling them. It comes with a full scene graph. But it has a WebGL backend if your platform supports it and falls back to canvas. And using that, I’ve been able to write 2D games that run blazing fast on virtually any hardware. Even my old Firefox OS Keon can run thousands of sprites moving on the screen at once and stay smooth. So, if you don’t want to learn about shaders and the WebGL pipeline and just want to do fast 2D graphics, Pixi.js is awesome. And they do actually let you write custom shaders if you want to do a blur or something on top.

My other pick is I am working again on my Luvit project. And so, I just want to mention it. Go to lovit.IO. And it’s basically Node.js but in Lua. I recently got hired by Rackspace and they’re paying me to work on that. So, I think that’s fun.

  Awesome. Rackspace is based in Texas, aren’t they?

  They are. My team’s in San Francisco. But yeah, the main office is San Antonio. And thanks to a stable job, I now have time to come on the show again. 

  Yay. Awesome. I’ve got a couple of picks. The first one is The Last Ship. It’s a show on TNT. And basically, there’s this global pandemic that wipes out a ton of the population and this ship was isolated from everything else. And so, they haven’t gotten sick and they have a scientist on board who can hopefully find a cure. And so, they’re racing to find a cure and to get back on and find their families if they’re still alive. And anyway, it was really, really good. The season finale was last week.

I really enjoyed that. 

  Chuck?

  Yeah?

  I have to say, I watched the very first episode, the pilot of that, and thought it was terrible. Is it just not for me or did it get better?

  It got better. Oh, yeah.

My second pick is Doctor Who is back. Though I have to say, the first episode of the new season wasn’t terrific. So, I’m hoping for better things. The second episode was a little bit better. So, I’m excited for that.

  My daughter’s watched it six times each episode now.

  [Laughs] Nice.

  I think they’re less discriminating than you are.

  Yeah, they may be. So yeah, those are my picks. Sean, what are your picks?

  Yeah, so I’ve got two. One of them, I mentioned earlier in the show but I really just want to call it out. This article called ‘Render Hell’ was just a really great overview of how all of the GPU pipeline works. And it’s from a non-technical point of view so it doesn’t assume a lot of knowledge about APIs or anything like that. It’s just really a great, great, great read. 

My other pick is I realize I didn’t get a chance to mention it earlier, so I’m going to do the shameless plug. And the project that I was talking about earlier that prompted all of this is called Martial Codex. So, if you want to check out some cool martial artists, we’ve got lots of free content. You should go check it out. 

  Very nice.

  And that’s my picks.

  Alright. Well, thanks for coming, Sean.

  Yeah, thanks for having me. It was great to talk to you guys about it.

  And if people want to get a hold of you and ask you about this stuff, what’s the best way to do that?

  The best way to reach me would probably be on Twitter. I’m @sgrif, S-G-R-I-F, or sean@thoughtbot.com is my email.

  Alright, cool. Well, thanks again for coming. And we’ll catch everyone next week.

[Hosting and bandwidth provided by the Blue Box Group. Check them out at Bluebox.net.]