The panelists discuss Broccoli.js with its creator, Jo Liss.

Special Guest: Jo Liss.

 

  [Hums ‘Pop Goes the Weasel’]

  Hey everybody and welcome to episode 121 of the JavaScript Jabber Show. This week on our panel, we have Jamison Dance.

  Hello.

  I’m Charles Max Wood from DevChat.TV. And this week we have a special guest, and that’s Jo Liss.

  Hi. Thanks for having me.

  Do you want to introduce yourself?

  I’m Jo. I’m an entrepreneur. And I do a lot of open source in the JavaScript ecosystem.

  Entrepreneur. I like you better already.

  [Chuckles]

  Don’t you do Sudoku? Is that what it is? Sudoku apps?

  Solitaire, solitaire. 

  Solitaire. Okay. 

  Yeah.

  I knew it was a game.

  A solitaire web app.

  Oh, really?

  You should plug that. 

  Yeah. Well, it’s not… I’m not [chuckles] I need gobs and gobs of traffic for that. [Chuckles]

  I’m not trying to get visitors one at a time, but more like a thousand at a time.

  Sure.

  It’s an HTML5 solitaire and it’s completely ad monetized. 

  Oh, cool.

  And the traffic comes from Google. So, I basically need lots and lots of organic traffic to make it work.

  Oh, I see. So, you don’t want us to put a link to it in our show notes so that our thousands and thousands of users can go check it out?

  No, we should. We should definitely put a link. It’s solitr.com. We’ll put a link.

  Awesome. Well, cool. We brought you on because you put together this awesome tool called Broccoli. And I have to admit, I was getting ready to go try it out, but most of my apps are Rails and I’m not really sure where to get started. So, before we get into that, I do want to ask what is Broccoli, for those that don’t know what it is and how to use it?

  When you make JavaScript MVC apps, what you typically do is you have a lot of JavaScript and a lot of CSS in a lot of files. And you’re going to have several compile steps to process all these files and to concatenate them together in some way or another. And so, Broccoli is a build tool that lets you define these chains of compile steps and then gives you fast rebuilds. So, it’s conceptually similar to the Rails Asset Pipeline, except that it’s not tied to Ruby or Rails.

  I like it better already, and I’m a Rails developer. 

  Yeah, yeah. I come from Rails as well. And I noticed way before I wrote Broccoli, I noticed that we were doing this weird thing where we have JavaScript and the JavaScript has nothing to do with Rails whatsoever. But what we would do a lot was putting the JavaScript into a gem and then packaging it up for the Rails Asset Pipeline, because the RubyGems package manager is so great. And then the Asset Pipeline is so great. So, we would use that. We would piggy back on this existing Rails infrastructure to distribute JavaScript. Of course, that’s completely broken because it’s tied to one specific backend and it shouldn’t be.

  Yeah. Usually what you’re talking about is you pull in a library and the Ruby library has a vendor folder where you can put other things and other assets, basically. And that’s all it is.

  Exactly.

  Is the entire library is just putting it where Rails knows to look for it.

  Exactly, yeah. 

  And yeah, that does seem kind of broken to me, as well. So, how does Broccoli go about solving these problems?

  So, I think it’s probably useful to think of it as two separate problems. One of them is package management, which is not really where we want to have it on the JavaScript side. And the other part is building. And so, Broccoli solves the building part. So, when you have the Rails Asset Pipeline, what you might have is the //=require calls that tell you how to concatenate the files. And you might also have Sass which has import statements. And so, implicitly from that, you get the concatenation order. And so, the Rails Asset Pipeline puts these compilers together and makes them run really fast. And I wanted to have something like that, except backend agnostic. 

So, what a lot of people were doing at the time when I wrote Broccoli, what a lot of people are still doing is writing our build steps with Grunt. And so, the problem with that is, what tends to happens is in practice, as you add more and more files, the build gets slower and slower. So, it’s not unusual at all to see 15, 20-second rebuild times. And so, then you want to be, you go, “Okay, this is too slow. Every time I edit something, I need to wait 20 seconds until I reload it in the browser.” And you try to do partial rebuilds where you are smart about what you rebuild. But Grunt was never designed to do this reliably. 

So for example, if you edit one CSS file, one Sass file, you might have to rebuild another Sass file because it includes the first Sass file. And so, you have to know which parts you need to rebuild and which parts you can reuse from last time. And if you wire together a build definition with Grunt, it’s not going to do that. Grunt definitions are basically glorified shell scripts. And so, what I think we need is a dedicated build tool that plugs into something like Grunt, into a task runner. And Broccoli is one such tool that provides the core build pipeline for the part of your process where you have a bunch of input files and libraries, and then you run compilers. And then you get a directory full of output files. So, for this kind of frequent rebuild process, that’s what Broccoli is for.

  So, it’s explicitly only for the build process. It’s not a generic task runner.

  Not at all. 

  Is that correct?

  Yeah, exactly.

  Okay.

  So, something like deployment, or generators, or kicking off tests, that kind of stuff, that definitely lives in Grunt.

  So, I know when Broccoli first came out, it was compared a lot to Grunt and also to Gulp. And you’re saying that it’s not really a fair comparison because it could work alongside either of those tools, or a totally different tool? 

  So, it competes with Gulp, actually. Gulp is another build tool. And I’ve written a blog post about why I think Broccoli has the right “architecture”. But it definitely is complementary to Grunt. So, it doesn‘t seek to replace it.

  Okay, that makes sense.

  Awesome. So, you did mention the different frameworks, for example you have Angular and Ember and Knockout and Backbone. Does it take anything special to run your build steps? Or does it just work out of the box with those, because they’re just JavaScript?

  Typically if you just take Broccoli core, what you find is you have a completely empty canvas. And you then go, “Okay. How do I assemble all these compilers that will put my files together?” So, what you need for a complete build stack is actually not just Broccoli. Broccoli is just the framework sitting at the bottom. But you need a set of compilers like something that concatenates JavaScript, something that compiles Sass, maybe a CoffeeScript compiler, a template compiler, that kind of stuff. So, if you write a build definition for an average app, it actually turns out to be quite a lot of steps. So, it’s not unusual to have, let’s say 100, 150 lines of code in your build definition file. And that’s asking people to write these build definitions from scratch every time they want to make an app. It seems a bit excessive. It’s too complex. The learning curve is too steep. 

And so, something that we are trying in the Ember ecosystem is to make a default stack of compilers on top of Broccoli so that you can say ‘ember new to-do-mvc’ and you get a to-do-mvc app with an existing build definition that you then can tweak, but you don’t have to. There’s no need to reinvent the complete process from scratch. So, that’s conceptually quite similar to how Rails works. A ‘rails new myapp’ works. It just gives you all this code that you can use to get started immediately. And the Rails Asset Pipeline is immediately functional. It has several compilers. It comes preloaded with several compilers. And so, what I think we’re going to see now, what I would like to see more of is these default stacks like Ember CLI that give you a complete stack, a complete stack of compile steps without having to do anything.

  Very nice. I want to change gears back to something we were talking about before. You said that if you just change one file here or one file there, you don’t have to rebuild all of your assets. Is that what I heard? Did I hear that right? 

  Yeah.

  So, how does that work, exactly?

  Traditionally, the way this has been done is with tools like make. And most build tools since then, they all tried to do the same thing. The way it works is you specify your output files and then you specify which files these output files depend on. And if you have multiple build steps in sequence, you’re goingto specify these intermediate files and then specify what those depend on. So, you’re building this graph of file dependencies. And then when you change an input file, you know exactly which part of the graph to rebuild. 

And make is completely based around comparing which file is newer. So, whenever an input file is newer than an output file, it will rebuild it. But this turns out to be hard, getting these dependency graphs right, because they are very complex and it’s very easy to have a subtle bug in there. And so, then if you get the graph slightly wrong, your build will become unreliable. And this is what a lot of the time, people just type ‘make clean’ all the time to force a complete rebuild.

  [Chuckles] Yup. Wipe it all out. I don’t know what’s going to happen. 

  Exactly, exactly.

  I’ve done that. 

  Because you don’t trust the system, right?

  Yup.

  And another problem is that the dependencies are not known ahead of time oftentimes. So for example, in a C file you might have include statements at the top, include directives. Or in a CSS file, to stick with the web, you might have import statements. And actually, you only find out about the dependencies at build time. And so, with the make type tools, people resort to all kinds of strange hacks to make that work. And the third problem is it doesn’t really take care of cleaning up intermediate files. And when you delete an input file, the corresponding output file doesn’t get deleted. And that’s not an academic edge case. That’s really, really important, because every time you switch branches, probably some file is going to be deleted or moved around. And so, tools like make are notoriously bad at cleaning up the leftover output files from previous builds.

  It seems like a lot of the tools in the web, like you mentioned Sass or Compass or Stylus, a lot of these have their own require so to speak, their own way of locating dependencies.

  Correct.

  And consequently, I guess I’m wondering. Can Broccoli do anything about those tools forcing themselves on you, in terms of how dependencies are located? Because I don’t know how you could do a partial build if you’re not in control of the compile step. Does Broccoli have a solution for that? Or is that just something that needs to change in the ecosystem?

  Yes and yes. For some compilers like CoffeeScript, you can obviously express the dependency very easily. So, there’s nothing complicated about that. You know exactly when to rebuild. But like you said, for something like Sass or Compass, it’s harder. And there have been, let me tell you, there’s a set of solutions to this and each of them has its own drawbacks. But all of them are possible with Broccoli.

So, what the Rails Asset Pipeline does is with Sass, every time you include, you import a file, Sass will give the asset pipeline a callback. And so, the asset pipeline as a result knows what the Sass file depends on. Of course, that only works because the asset pipeline and Sass are both written in Ruby. So, Broccoli is written in JavaScript. It runs on Node. And a lot of compilers are written in JavaScript these days. But it seems that the approach is a bit unsatisfactory.

Another way the Rails Asset Pipeline does it is, so for example, to compile something like LESS which is the LESS CSS compiler is written in JavaScript, and so you cannot get the callback. And so, the LESS Rails Plugin for the asset pipeline, what it does is it uses a regex to parse all the imports ahead of time. And of course, it seems like a really terrible…

  It seems crazy.

  Yes, it seems like a really terrible hack, right? And I think that should tell us something: that there is a problem with the asset pipeline’s requirement to know all the dependencies. And so, with Broccoli the way it works is a compiler plugin just takes a set of input files, like a tree of files, a directory, and it emits an output directory. And oftentimes the output directory will just contain one single output file. And the plugin itself is then responsible for figuring out when to rebuild. It should be smart about whether to rebuild at all and when it rebuilds, how much to rebuild. And you could easily think about if you have something like Sass or LESS, you could easily think about wrapping it in some code that does the same thing that the Rails Asset Pipeline does. It figures out the dependencies through a callback or through a regex. And then whenever those change, it will rebuild. And if they don’t change, if they haven’t changed since the last build, it will just reuse the output directory from last time. 

But I think there would be a better approach to do this. And that requires buy in from compiler authors. Let me back up a step. When you recompile a Sass project, a Sass file, generally the performance, the time it takes is going to be linear with the number of input files. So, if you have ten input files, it might take one second. And if you have 50 input files, it might take five seconds. So, on a typical-sized project, you very, very quickly get into performance territory where it’s just too slow for the kind of continuous rebuilding that you want as a developer. So, I think what we’ll probably have to happen is that compilers like Sass start caching parts of their compile steps. 

So, if you have 50 input files and you run Sass once, then it will compile everything. But when you run it a second time, and only one of these files has changed, it will detect that 49 files still have the same timestamp and it will reuse a lot of the intermediate data structures like the parse tree, the AST from last time. So, if we do something like that, we get the rebuild time from O(n) from linear with the number of input files to constant time. Basically, if only one file has changed, then we’re going to take a constant time to rebuild the entire project, no matter how big it is. And I think that’s something that from a performance perspective and from a usability perspective, is really, really crucial. And once we have that, it extends to the case where no file has changed. 

So, if the compiler were to do caching anyway, then it could just detect that no file has changed and reuse the complete build output from last time. And that way, we don’t have to try and replicate the dependency graph outside the compiler. But all the complexity of figuring out which files depend on which other files is contained to within the compiler. So, I’ve tried to evangelize about this a little bit with compiler authors. And what I generally seem to get is a vague, “Yeah, that seems like it might actually be necessary, but it’s not completely clear that we want it.” So, of course we are going to need to see code and to figure out if this is really the best approach and what kind of pitfalls we’re going to encounter. But I haven’t gotten terrible pushback on that idea.

  So, just from you talking about that, it sounds like a lot of it depends on other people to implement features in the compilers. Is that a good summary of it?

  In a way, yes. What we can always do, of course, if we just want a compiler, getting a compiler running today, is put in these kinds of stopgap measures where we parse out the imports or some other approach that Robert Jackson has implanted in several plugins is, and there is a common base class for that that takes away all the complexity behind it, is if you pass a tree of files into the compiler, if none of the files have changed, then we just do not kick off the compiler again. So, if you are careful to only pass your Sass files into the Sass compiler, and no JavaScript files, then if you change the JavaScript files which live in a separate directory, the Sass compiler will not run again. 

So, we can always replicate the cluges that other systems implement. You can always express these on top of the Broccoli architecture. But I think importantly, Broccoli allows you to push all of this logic down into the compiler, which I think in the end is where we want it to live. That’s my hypothesis anyway.

  Sure, that makes sense.

  Does this work with, when you’re doing things with say Sass where the compiler’s written in Ruby and not in JavaScript? Does that affect things at all?

  Broccoli was intentionally written to just use the file system. So, you can just call into Ruby and you can just kick off a Ruby process.

  Okay.

  It’s obviously going to be a little bit slow. So, there are always performance problems when you hand off to a completely different language. But this has actually come in very useful for native libraries like libSass. I’m a huge fan of libSass. It’s a C reimplementation, a C++ reimplementation of Sass. And so, in my benchmark, it was basically ten times faster than Ruby-Sass. So, on a dataset where Ruby-Sass would take two to five seconds, on a typical app-sized file where RubySass would take two to five seconds, libSass would take 200 milliseconds. 

  It’s just shocking, honestly. [Chuckles]

  Yes, it’s huge. It’s absolutely huge. And so, seeing that kind of difference made me appreciate that it may not always be sensible to implement stuff in Ruby or JavaScript, because having delays like two to five seconds in your rebuild is clearly quite painful. And 200 milliseconds is just barely noticeable. So, libSass seems really, really awesome. It’s unfortunately not quite a replacement for Ruby-Sass yet. And we also don’t have a native replacement for Compass yet. But just from the performance difference alone, I think it made me appreciate how important it is to support native libraries. And Broccoli has no problem at all calling into libSass through a Node wrapper and just having that run against the files and the file system.

  So, I want to talk a little bit about implementation here. If I wanted to get started with Broccoli, say on a project with Express or Rails or something, how would I go about doing that? Do I just put all my assets in the same folder and install the plugins and then say go do your thing with those? 

  Sort of, yes. But the problem is that you would have to assemble the entire build pipeline yourself. So, let’s say CoffeeScript and Sass and then the //=require things and stuff like that. And then you have a vendor directory. And then maybe a public directory with an HTML file in it. So, there’s a bunch of things that come together. And to replace Rails projects in particular, we are missing a Broccoli plugin for a Sprockets type require directive parser. And it’s on my list of things that I would really like to write and it’s not terribly hard. But I haven’t gotten around to it and I don’t think anyone else has gotten around to it so far. 

So, there’s no way to just parse out the //=require calls with Broccoli at the moment, which is kind of a bummer. But once we have that kind of compiler, what I would like to maybe do is provide at least a sample broc file, a sample build definition to show, “This is how you would take a Rails project and then basically just build the same thing using Broccoli without calling Rails.”

  What about something that’s not Rails, that doesn’t have all the weird directives and stuff for the asset pipeline?

  So, you’re typically going to have something, right? How are you going to concatenate your JavaScript files?

  Yeah. So usually Chuck, you might pick CommonJS or you might pick AMD.

  Right.

  Or something like those. And then you use the corresponding Broccoli plugin to do the tracing and concatenation. 

  Yeah.

  Makes sense to me.

  Is it painful at all to hook that up to use the correct plugin?

  She probably is biased. But I would say it’s no harder than it is in Grunt or Gulp or any of the other ones. 

  That’s good to hear.

  Is that fair to say?

  I’m not a good judge because I know Broccoli. I know it so well that it’s…

  Right. 

  It’s hard for me to judge how…

  Sure.

  Hard the onboarding is.

  Have you talked about the server versus the output directory context thing that Broccoli does? Because that’s something that makes it rather unique.

  Does it actually? So, the way I think about it is it’s just the equivalent of rails serve and rails precompile.

  So, I suppose if you’re not coming from Rails and you’re coming from Grunt or Gulp, it’s an interesting piece, or even make. But what it is, is Broccoli has the ability to either take its input and its output and put it directly to an output file like a build, or it can actually serve it up as a web server that will compile on the fly that you could just link to.

  Yeah.

  Which is really cool.

  And this might seem like a small detail, but something that I was very careful about was making sure that the way Broccoli built and Broccoli served, meaning the one-off build, the one-off precompilation, and the continuous serving, these two builds use exactly the same code path. So, when you serve it’s just that the same process gets called over and over again. But it’s still the same thing. And the reason why that is important is if you have different code paths for serving and for pre-compilation, what’s going to happen is that your app works great locally. And then you deploy and stuff breaks. And that actually happens all the time on Heroku. And apparently 50% of their support requests are about the Rails Asset Pipeline.

  Wow.

  About pre-compilation issues.

  Oh my goodness.

  Wow.

  Yeah. 

  I’m sure they want that.

  That’s crazy. So, you talked a little bit how make leverages timestamps to do the incremental build and how make is notoriously bad for switching branches because it can’t handle file deletes and additions properly. How does Broccoli solve that problem?

  The way Broccoli thinks about it is if you have one plugin, let’s say a CoffeeScript compiler that takes a directory full of files of whatever file type, including some .coffee files. So, the CoffeeScript compiler is going to pass through all the files. And the .coffee files, it’s going to compile and turn into .js files. So, what’s going to happen is the CoffeeScript plugin remembers the timestamps and the sizes of the input coffee files. And then while it runs, it also puts the output of the CoffeeScript compiler into a cache. And next time it runs, it just checks the last modified times and the sizes of each file. And if it exists in the cache, it gets pulled out of the cache. 

And so, the actual CoffeeScript compiler doesn’t get invoked. It just is an instantaneous operation where it pulls everything out of the cache. And so, that works around a lot of the issues. Like if a file gets deleted, of course it’s not going to appear in the output again because we don’t try to rebuild it. If a file, if a timestamp jumps back in time, then we’re not going to get confused but we’re going to detect that the timestamp is older now and we still need to rebuild. 

So, this approach is a lot more robust against a lot of these edge cases that cause make to have problems. And in particular, one way to think about it is that we just infer file identity from the combination of path, last modified time, and file size. But we’re not doing any other kind of time comparison logic like make does. It’s just a shortcut. We could also just MD5 hash the files and get the same result.

Got it, very cool.

  I’m curious. Do you have a future direction you intend to take Broccoli? Are there things you want to add to it?

  So, there are some changes to the plugin API coming up. There are actually two refactorings. One of them is going to come very soon and the other one is maybe two or three months off, or something like that. And that comes with the being pre-1.0 I guess. So basically, what’s happening right now is fixing some performance issues and cleaning up some of the API that we are exposing, while we still can, while it’s still easy, before there is a huge number of applications being built on top of it. 

And then beyond that, like I said earlier, I’m hoping that a decent amount of improvements in terms of how we are smart about rebuilding is going to happen in the compilers, actually, and not in Broccoli. But I’m sure we’ll also have some more base classes for Broccoli plugin that implement all kinds of caching. And I think beyond that, Broccoli is basically done. It’s a very simple piece of software. It’s less than a thousand lines of code. And what I really want to see is default stacks like Ember CLI that are based on Broccoli, but so you don’t as a user, you don’t have to think about Broccoli anymore. It just works.

  One thing that just occurred to me too, does Broccoli play nicely with package managers with Bower and CDNs?

  So, that’s actually, playing together with package managers is one of the original use cases, because something that was very obviously missing from the JavaScript ecosystem and that is still missing is the ability to say, “This piece of code depends on another piece of code on another package.” So, I’m going to add this package to my package.json or something, and it gets pulled in. And then I can just call require at the top of my file or import and I can use it in my code immediately. And what we’re doing at the moment is just dropping stuff into vendor directories, which is pretty terrible. So, there are two parts to the problem. One of them is we have to have a distribution mechanism for browser-side JavaScript and CSS as well and images maybe. So, that’s a package manager. 

So, Bower is I think the most promising one at the moment. Some people are talking about using npm for browser-side JavaScript. And that’s maybe an option as well, but I’m feeling fairly positive about Bower. So, the problem with Bower is, or not problem but just where the scope of Bower ends is it just gives you a set of files. When you depend on a package, it just says, “Okay, here are your files. Now do whatever you want with them,” and you have to figure it out yourself, how you’re going to include them. 

And JavaScript until now hasn’t had a way to require other files. And so, we built these systems like AMD or CommonJS on top of it to add require calls after the fact. And this is the part where the build step comes in because being able to distribute packages and express dependencies between packages is useless until you can also in your JavaScript code express which files you depend on specifically and import things from other files. 

And so, the one thing that I’m really, really excited about is the ES6 module syntax. And my general idea of what I want to happen is I want there to be a default stack based on Broccoli that uses the ES6 module system and parses out the ES6 import statements. And then that is going to be backed by Bower. So, if you import something, it will automatically search through all the libraries that you have pulled in via Bower. And that is my long-term idea of where I would like things to be. 

And I’m not set on the specific tools. But I think this is something we definitely want, to make JavaScript great. It’s basically, we want to have the same experience that we have with something like Ruby where adding a dependency on another package is just I’m going to add this line to my gem file, I’m going to reinstall the gems, and then I’m going to say require some module in my code, and it’s going to be available. And I want that to happen in JavaScript land.

  Beautiful dream.

  So, I think to summarize that, what I want is something sitting on top of Broccoli, like a default build stack, that automatically pulls in files from Bower, et cetera, that uses the ES6 module system specifically.

  Cool. 

  Do you know anything about the ETA for that? The spec is still, it’s in the final stages of being worked on, if I understand correctly.

  Yeah.

  Is there something you can actually start building towards right now or are you waiting for things to settle down a little bit?

  Yes and yes. So, there was a recent change in the ES6 module syntax. And I think it is probably not completely stable. I suspect that there might be still a few changes coming in, minor changes maybe. So, it’s definitely still a draft. And the transpiler from ES6 modules into ES5 is still, I don’t know if it’s alpha or beta, but it was just rewritten. I think Brian Donovan did it. And so, it looks much better now. It covers edge cases much better. But it is probably, the transpilation step probably still has problems here and there where the semantics don’t map completely, correctly. So, that is something that definitely still needs to stabilize. 

The reason why I’m so excited though, about the ES6 module syntax, is that it’s static. So, unlike AMD or CommonJS, you can do static analysis on JavaScript files and you know which files they import. It’s not possible to import a variable module or something. And so, being able to statically parse out imports, and then based on that being able to figure out which files you need to concatenate, that’s a hugely important feature. And so, I think the ES6 module syntax is definitely… and it has other features as well, but in terms of building, that is one of the really important ones, the fact that it’s completely static. But still, that part is still, I think it’s basically alpha. 

And another part to make that happen is Bower needs to get better. I think there are a bunch of cases that Bower just doesn’t cover very well and that we are used from something like Bundler on Ruby. And that’s because Bundler is a pretty complex piece of software and it does a lot of things really great. And no one has gotten around to implementing them for Bower. And something that is missing from Bower is the ability to say, “Okay, so here’s my package. But if you include an ES6 JavaScript file, this is the directory that you should be looking in,” like the lib directory it’s usually. 

So, there’s nothing in Bower or on top of Bower that gives us the ability to communicate, “When you want to include files, JavaScript files from this package, please look inside this subdirectory.” So, we are still going to need to add these kinds of things. It seems very trivial, but it all needs to be spec’d out. We always need to think about all kinds of edge cases and how to implement it the right way. So, those pieces are missing as well. And I think those are the two big things that really need to be worked on. 

So, the Bower part, because we don’t have a way to say, “Look inside the lib directory to pick up source files,” the Bower part is basically unusable right now. And we still have to resort to manually linking up files rather than having them automatically picked up when you import them. And we’re not terribly far away from it. But I don’t know that anyone is working on it at all right now. So, I would really like that to happen. But I don’t have an ETA, simply because I’m not working on it. I

don’t know if anyone is. And the ES6 module stuff, I think that’s going to shape up over the coming year, maybe. It’s just going to incrementally stabilize as people start using Ember CLI, which uses the ES6 module system. People are going to discover bugs. And we’re going to iterate on the spec.

  So, I wanted to ask, go on a tangent and ask about the uptake of Broccoli in the community. It seems like it’s been embraced as the default build tool by the Ember community. Is that true? And then how is it being used in other places?

  Yeah. I think right now the uptake is mostly in the, most of it seems to have been in the Ember community via Ember CLI specifically. There are some people using it for their custom build steps for their apps. And some libraries are being built with it, which I think is another very important use case. But most of the existing users seem to come from Ember CLI. They use Broccoli through Ember CLI. And that is really the way I envisioned Broccoli being picked up. As I said earlier, I don’t want people to write build definitions from scratch. I think the learning curve for that is way too steep. And the same way the Rails Asset Pipeline, nobody ever uses it on its own. The Sprockets gem I think, you radically could.

  [Chuckles] That’s true, yeah.

  You could use it.

  Nobody would gem install sprockets in [inaudible] Ruby project.

  No, exactly, right? Yeah. So, the only way that ever gets used is through Rails. And so, what I would like to see is to have an Ember default stack based on Broccoli, and an Angular default stack based on Broccoli, and a Knockout default stack, and maybe a Rails Asset Pipeline replacement stack. And so, to have these sets of compilers where you just say, “Okay, I want, this is the general format of my app. Now give me a set of compilers.” And then I think that’s how Broccoli is really going to get adopted, not directly but through these stacks. And then nobody even cares anymore that they’re using Broccoli. That’s just what happens under the hood.

  So, once you have these build steps set up and you’ve got things figured out that way, what are the next steps? What are the other things or are there other things that you need to be doing or thinking about related to this?

  You mean, when you write an app?

  Yeah.

  Not really. I don’t think so. As far as Broccoli is concerned, that’s all it is. Obviously, something, all the other parts to writing an app like connecting it with a backend and maybe serving both of them from the same server and deployment and scaffolding, like you want to generate new controllers or stuff like that, all of these things, they still need to happen. And those are all things that people are thinking about with Ember CLI. Those are being added there. And I think those are generally useful for other projects as well. So, as far as Broccoli’s concerned, Broccoli doesn’t care about anything beyond the build, beyond the core build process. But these default stacks are definitely going to have more functionality for deployment, scaffolding, testing, et cetera.

  Are you accepting contributions to Broccoli?

  Yeah, I am. I haven’t been terribly good about keeping up with GitHub because I mostly just, when I spend time on Broccoli, I think the most important thing right now that I can do is pushing the refactorings forward and making sure that happens and that the APIs look nice, and then most of the time, what I want people to do is just to plug into the APIs. So, I’ve been saying no to a lot of contributions. Basically, the standard response is, “Please turn this into an external plugin or an external library. I don’t want this in Broccoli core.” I don’t want Broccoli core to be terribly configurable at all, because it’s really just, Broccoli core is really just, the Broccoli command line tool is just one way to kick off Broccoli. 

And in general, people are going to be using Broccoli as a library from their own tools. And they’re going to provide their own servers and their own watchers, et cetera. So, Broccoli itself is an incredibly thin piece of software. It basically just provides the general architecture, the opinionatedness about the, not even that, just the way the compilers go together. And then everything else happens outside of Broccoli. So, I guess the answer is yes and no. Definitely, I take patches for Broccoli and for a lot of the plugins that I maintain. But also, a lot of the time, many things can be done outside of Broccoli.

  So, if I wanted to write a Broccoli plugin, say for Rails so that I could get the //=require or whatever to work, where would I get started?

  I think you would pick a base class. In this case, you would pick the Broccoli writer base class and then subclass that to write your plugin. And I think what people do most of the time is they look at an existing plugin. So, in your case if you wanted to write something that parses out //=require, you would look at the Sass plugin because that’s conceptually similar, it takes multiple files and outputs one single file, and look at how that is implemented. And the API that Broccoli or that the base class you inherit from then exposes to you, is very small, is basically just, “Here’s your input directory. Here’s your output directory. Now, do whatever you want.” And then you just do file system operations. 

And so, the tricky part is usually, when you write a plugin, is usually getting it to be fast. But that’s up to you. And if you don’t want it to be fast, if you just care about getting it working at all, you don’t have to be. You can just make a plugin that naively recompiles everything every time. And it’s going to slow down your build, but Broccoli is not going to stop you from doing that. So, if you are willing to sacrifice performance just to get started, it degrades very nicely into an incredibly simple subcase.

  Cool. I’m excited to go play with it now.

  So, this isn’t related to Broccoli. I just wanted to say I think you’re super cool. I follow you on Twitter and it’s fun to see you just start wondering about a field. I saw the tweets that were like, “Build tools aren’t very good. I wonder what a good one would look like,” and then you start talking about an API. And then suddenly, you have this awesome tool called Broccoli.

  [Chuckles]

  So, I don’t know.

  Thank you.

  It’s really inspiring to see you create amazing things in the public eye.

  Yeah. As I write software, I try to be mindful about improving the tools that I use to write software. So, I try to spend about, let’s say 50% of my time just iterating on underlying tooling, like the libraries, the build tool in this case, whatever is involved in the development process, whatever makes me more productive. So, Broccoli fits into that theme. And actually, I ended up taking several months off to write Broccoli. It took longer than I thought. I thought I would be done, I thought I would have the first beta out after a month and it took four. [Chuckles] But it was still worth it. It was still okay. And so, when I allocate time, I try to very actively make time to work on tooling, to work on libraries.

  That is such a good idea.

  That’s a really cool philosophy.

  That is also very selfless of you.

  It’s awesome.

  The only other question I can see here is do you see this tool being used to build things other than web assets? So for example, mobile apps, or other things that have a build step to them, C programs. Do you see it calling out to the GCC compiler?

  I think the architecture is, if I were to write a build tool for anything, for any other environment, I would reuse the architecture.

  The architecture that exists in that environment? 

  No, sorry, the Broccoli architecture.

  Okay.

  So, the idea that a build process is just a series of tree transformations. And I’m actually going to link a slide that I had in one of my talks that illustrates the graph nature of these tree transformations. That’s really the core piece, the core inside of Broccoli. And the notion that everything that makes the build fast needs to happen in each individual step. You don’t have any global dependency graph that spans the individual steps. And so, I would probably take this kind of insight and try to apply it to other environments. Now, Broccoli itself is written in Node. And there is no reason why you would write something in another, let’s say to compile iOS apps, you wouldn’t use Node for that. It’s maybe a little bit too slow. And also, people just don’t have it installed. So, you wouldn’t use Node. 

And there are probably some things about Broccoli that are specific to the web, like I’m making some assumptions about the numbers of files and the file sizes that we’re moving around. In a web app, you have an inherent limit on the size of your app, because you can only serve out so much. You have a 400, 500 kilobyte budget for your app, or maybe a megabyte with images or two for a page, and that’s all you can do. And for C++ projects, you can have much, much more input files, much more intermediate files. And so, you have these hundreds of megabytes of build products flying around. 

And I’m not 100% sure if Broccoli would hold up well with something like that, or if you would run into issues where things are too slow or things take up too much space. So, in particular while I wrote Broccoli, I always had a worst case scenario in mind of I need to be able to process a few thousand files, I need to be able to process tens of megabytes. And with other environments, you might have an order of magnitude more data. And so, Broccoli might not be up to that. But I think the general architecture probably would.

  Cool. Yeah.

  My mind is spinning.

  Yeah, it’s been awesome.

  Joe actually asked an interesting question in the chat. Why did you call it Broccoli?

  So, the way I came up with it was it’s an acronym for Browser Compilation Library. And I felt very smart about that. [Chuckles] But…

  No one could spell Broccoli? Oh no. [Chuckles]

  I found out that no one can spell it. [Chuckles] I wasn’t sure. It didn’t seem hard to spell to me. But I can see that the double C and the single L, it totally trips people up. And so, I think in retrospect, I would have picked another name if I had known that. When it got posted on Hacker News, so many people complained that they cannot spell the name. [Laughter]

  That’s hilarious.

  I would be lying if I said I didn’t fall into that group of people.

  Well no, look at it this way. Broccoli has taught you how to spell better, in addition to building your code.

  Boosh!

  Good point, dude.

  [Laughs]

  With the next tool, we’re going to practice cauliflower.

  [Inaudible]. Let’s get it now.

  Oh, I love it. Jamison, what are your picks?

  I got two. One is an article by Raquel Vélez. She’s been on the show to talk about robotics. And it’s talking about diversity and how some people talk about trying to hire diverse candidates as a needle in a haystack problem where they can’t find anyone. All the people that apply or that they have in their networks just look like everyone else. And she talks about it more as a network problem, where you don’t need to find the one mythical diverse person that will apply to your job.

You need to form connections with people, and then that will introduce you to a network of people. And it’s just an idea about it I haven’t heard before. It’s a really good read and it’s pretty short.

My next pick is a website called hexascii.com. And it’s my one-stop shop for all the best Japanese emoticons. That’s all you need to know.

  Nice, nice.

  Yeah. If you need to express surprise that a bear is behind you, just in 100% text as an emoticon, they’ve got you covered.

  Vital.

  Very nice.

  Including the bear.

  Just in case.

  Yup.

  Alright. Merrick, what are your picks?

  So, I have two picks. The first is an iOS app called Reporter. And all it does is it lets you set up taxonomies of questions. So, you can set up a question and say which type of taxonomy the answer is going to be in. And you can just answer these questions. It will notify you randomly the number of times you request over a timeframe. So for example, I’ve been charting things that are typically hard to chart, things that are more subjective, like my emotions or who I’m spending time with. And it’s been pretty cool. It’s been pretty interesting just to see my life, or these things that are historically hard to measure over time.

And the other thing is a rapper named Trip Lee and his album ‘The Good Life’. I highly recommend it. That’s it. Those are my two.

  Awesome. Joe, what are your picks.

  So, I’ve got three picks. The first one is big O notation. Since Jo was talking about big O notation this episode, I can’t remember the last time anybody discussed big O notation or mentioned it at all in one of these podcasts. And I think it’s awesome that it was mentioned. It made me feel a lot smarter to be in a podcast that had big O notation mentioned. So, I’m going to pick big O notation.

  [Chuckles]

  If you don’t know what it is, google it. It’s awesome.

  Or ask Joe. That’s how I learned about it.

  Right, right. Well, I’ll be giving a master class on big O notation later on this week. [Chuckles]

  The week after, we’ll explain what recursion is, right?

  Right. [Chuckles]

  My next pick is going to be the company Pluralsight’s new office. They just had the ribbon-cutting ceremony for the new office that they built here locally in Utah. And I went out to the opening for it. And it is an amazing office. It’s way cool. And the best part about it is they have an entire shelf that’s just stocked with candy. And so, when you bring your kids over to some work function with dad, the fact that there’s candy there makes things so much better.

  You sound like you’re speaking from experience.

  [Laughs] Yes. Yeah, I am. The other thing that was really cool that they did is they installed this sound-deadening system so that they could have a mostly open office with not cube farms but just desks out. But there’s a sound-deadening system that makes it so that you don’t hear all. It’s not echo-y and chatter-y. People can be having conversations just a desk away and they don’t interrupt you in your conversations. So, I thought that was really pretty smart as well, and very cool. In the center, they have these little rooms where you can go in and just chill out. They’re small. They only fit one person. There’s a nice chair so you can go in there and take a nap. They’re dimmed and you can go take a nap or just chill out if you need some time, because you’re a programmer and you’re an introvert, which I thought was totally awesome as well. So, cool office. I’m going to pick their office as my second pick.

We Need You to Lead Us’. And it’s just about being a leader, even if you’re not necessarily a manager. And whether that’s at work or elsewhere in your life, very motivational book, really great, awesome read. And I really enjoyed it. So, I’m going to pick ‘Tribes’ as my final pick.

  Awesome. I’ve got a handful of picks here. Actually, I have more picks than I have time for. So, I’ll have to save them up and share them for next time. But a couple that I’ve got here, one of them is a command line tool. It’s called cowsay. If you don’t know what it is, you can go install it on most systems. I’m on a Mac, so I just did brew install cowsay. And what it does is it puts out a little cow that says whatever you told it to say.

My second pick is a command line utility. And basically, it’s called Xiki. And what it is, is it… I use it inside of Emacs and so it’s all interactive and stuff. And it does have its own shell that makes it interactive. And so, you can run commands from it just inside of the shell. And it’s basically like a menu system on the command line. It’s really hard to explain without showing people. If you go to Xiki.org, you can check it out. There are some videos there that explain what it is and how it works. And that’s X-I-K-I.

Another pick I have, and this is something I’ve been doing recently, it’s part of ‘The Miracle Morning’, which I picked before. It’s just keeping a journal. It really helps me think through things and figure out what I want to be doing and how I want to be involved with things and stuff like that. So, I wind up writing down goals and things. And it helps me process the stuff that’s going on in my life. And so, I really, really like it. 

And finally, I’m going to pick something that Aaron picked a few weeks ago. It’s ‘The Martian’. It’s a novel by Andy Weir. I listened to it on Audible. It was really good. I really enjoyed it. It’s this, like MacGyver on Mars. He gets stranded on Mars and so he’s got to figure out how to survive long enough to get rescued. So anyway, pretty cool stuff. So, those are my picks.

Jo, the other Jo, the guest Jo, what are your picks?

  So, I have two picks for today. The first one is a book called ‘Creativity, Inc.’ by the Pixar guy, yes. He was one of the founders of Pixar and is I think now the president of Disney. 

  John…

  Catmull.

  Ed Catmull. There you go. [Chuckles]

  It’s a management book as well, about, “Here are the things that they discovered that are important to make people work creatively, how to eliminate barriers to creativity.” And in the process, they also tell the story of Pixar and then Disney, which is super, super exciting. So, that’s a lot of fun to read. So, that’s what I read at night at the moment. 

And then the book that I read during the day is called ‘Doing Bayesian Data Analysis’. I’m sorry this is super pretentious. But I love the book. [Chuckles] So, humor me. Basically one of the problems that I’m having right now is with my Solitaire game. I’m getting a lot of analytics data and I would like to do all kinds of, create all kinds of estimates of things and run A/B tests and see if two [inaudible] hearts are statistically different, if I believe they are, they are different, and what the probability is that they are. And so, for these kinds of questions where you’re estimating stuff and where you’re asking questions about the state of the world where you’re trying to estimate a belief about something, that’s basically the whole field of Bayesian statistics. 

And I took some statistics classes in university. But they all fell short for various reasons, for practical stuff that I need as a programmer. And this is the book that, this is the stuff that I wish they had had in university. It has both the theoretical background as well as the practical, “Here is way you model your problems. You model your parameters. And here’s how you estimate them.” So, that’s really cool and it’s really accessible. It’s not like most math textbooks where you’re just working line by line through huge equations. But it’s very well-explained. 

  What kind of prior knowledge would one have to have to get something out of the book?

  I would say college-level math or maybe even high school math, depending on what kind of courses you took.

  Alright. I’m holding out on my high school promise.

  [Chuckles] I think with most of the math that you would know as a programmer, you should probably be able to get by. 

  Awesome.

  So yeah, ‘Doing Bayesian Data Analysis’. It’s really great.

  Cool. I did some Bayesian stuff in college for my AI class. 

  Yeah, and it’s such a natural way to model a lot of practical problems.

  Yeah.

  Practical AI kind of problems.

  Yeah, definitely. And it’s definitely an interesting way of thinking about some of those problems. Alright. Well, thanks for coming on the show, Jo.

  Yeah, this was great.

  Yeah, it was a lot of fun to talk about.

  Thanks for having me.

  It was awesome.

  It’s been fun, yeah.

  Alright.

  Interesting.

  We’ll wrap up the show and we’ll catch you all next week.

[Hosting and bandwidth provided by the Blue Box Group. Check them out at Bluebox.net.]