The panelists talk to Matt DeBergalis about Meteor.js.

Special Guest: Matt DeBergalis.

 

Hey everybody and welcome to episode 145 of the JavaScript Jabber Show. This week on our panel, we have AJ O’Neal.

Yo, yo, yo, coming at you live from an office bedroom in Provo, Utah.

Dave Smith.

Hi-lo.

I’m Charles Max Wood from DevChat.TV with a quick reminder to go sign up for JS Remote Conf. By the time you get this, we’ll be about a day into it. But I will be watching sign-ups and getting people access to the conference as we go. So, go check it out. We are also recording the talks. And so, they’ll be available to attendees. So, you can get caught up pretty quick. We also have a special guest this week and that’s Matt DeBergalis.

How you guys doing?

Doing alright. You want to introduce yourself really quickly?

Oh sure. I’m Matt DeBergalis. I’m one of the authors of Meteor and founder of Meteor Development Group here in San Francisco.

So, what all does that entail?

Well, we write software and we get people to use it.

[Laughs] Sounds like what I do. Do you want to give us a quick overview on what Meteor is?

Sure. We’re a full-stack framework for building apps in JavaScript. So, Meteor is a bunch of stuff, some of them we’ve written, some of them we’ve pulled in from other parts of the community. Node, for example, or a lot of the packages in the npm ecosystem. And it all adds up to a single stack that lets you write normal-looking JavaScript that runs both in the cloud for your secure business logic and also on mobile phones and inside web browsers for you client-side interface and screen rendering. And it all includes an end-to-end reactive data model.

So, unlike a lot of the older generation things like Rails or PHP or whatnot that have been retrofitted with a live updating screen, in Meteor the data all the way from the database on one end to your screen on the other end automatically updates as things change and different parts of the system react. So, it’s a really elegant and fun way to write modern apps that have that kind of live updating character to them.

So, how is it different from something like MEAN stack or something like that?

Oh, MEAN’s a, what is it, a design pattern I guess we could say, right? So, what we think about when we look at an app, and maybe I can go back a step. We’ve for a long time had the era of the website. So, the technology to build a modern website is your classic three-tier architecture. So, you’ve got a database, you’ve got a server-side stack of some kind that runs on typically some servers (originally actual machines, these days of course in the cloud). It’s stateless. It’s built around this HTTP protocol. So, when a client wants to request a URL, a request comes in, there’s no state attached to that. It just gets routed to one of these application processes in the server ultimately. There’s a lot of work of course, but ultimately the server’s job is to send down a page, a document. And the typical language of that is either HTML or if it’s an API maybe JSON or XML, something like that.

Here’s what’s interesting. So, if you want to build a modern app, something that has maybe a collaborative element to it like Uber where you can see where all the drivers are or Slack where you are talking in different chatrooms at the same time. It’s sharing data. You end up needing a

way to push data from the server to the client because you want to see other users’ changes as they happen. And you need a way to update the screen on your client once that data comes from the server. So, you can do that by assembling a bunch of pieces. That’s what MEAN is all about. That’s also what all of the really great consumer-facing companies have done over the last three or four years. So Facebook, Twitter, Quora, Asana, all of these companies have invested in some sort of technology stack that lets them create this live updating application experience.

What we realized is that the guts of that, the internal structure of those things, number one isn’t anything like the stack that we used in the web era. It’s stateful. It has a bidirectional protocol that’s not HTTP. It’s just built around a very different kind of box diagram if you were to draw it out.

And the second thing we realized is it’s pretty much the same in all of these apps. So, if you look at what’s inside any of these modern user experiences you end up seeing the same kinds of stuff that gets built. You have some kind of way of watching a database for changes. You’ve got some protocol that you’ve invented that runs on top of WebSockets to push that data to the client. You’ve got some kind of cache on the client because you can’t afford the roundtrip back to the server when you want to redraw your screen. So, you’re storing some of this data locally and keeping it up-to-date. And then you’ve got some story for how as that data changes you’re going to actually change the DOM. So, that’s where your React or your Angular or maybe in an older app Backbone would be. It’s all the same.

It’s a lot like how [chuckles] in the 70s applications had databases. But that meant you found the database guy. You hired the database PhD and they came in and they very carefully laid out all your B-trees and your indexes and your tables. And if you wanted to change the app down the road so that you sorted everybody by first name instead of last name, you had to call the guy back and he’d make you a new B-tree. And it was a really complicated slow process. And it turns out, once you have a standard way of expressing the idea of a database and you end up with these standard open source things like MySQL and Postgres, it’s a lot easier to build applications on top of that. And so, we think there is a similar set of basic building blocks for a modern app that’s all about stateful, live updating, real-time APIs that should be open source and that everybody should share.

And that’s really what Meteor’s all about.

So, is it fair to say that Meteor does not include a view layer? You mentioned a minute ago that you would have React or Angular or something else for that? Is that the case or does it ship with a bunch of view stuff, too?

We have a view equivalent called Blaze. The biggest difference between Angular and Blaze for example is that Blaze is based around templating. And we picked that because we think a lot of developers are really comfortable with similar to what you would see in a Rails app or a PHP app where you can write a normal, plain, HTML-looking template that just has a couple of little things inside that show where the data should go. So, we model ours after Handlebars.

And that’s really about creating an intuitive developer experience that works really well for people when they get started but that’s also based on really good implementation so that as your app gets more complex, it scales up with you. So, you can swap Blaze out for Angular. People do that. You can also… Pete Hunt gave a talk here where he showed React on top of Meteor. So, these parts of the stack are interchangeable.

The one thing we try to do with the standard stack which would include Blaze is that we’ve just tried to harmonize all the different pieces so they work really well together. And I think anybody who’s built an app with the MEAN stack has run into this problem where it’s doable. It’s just that a lot of the time ends up getting spent on these details of connecting one element to another, right?

So, in Meteor it’s really nice just because the system we use for tracking changes, for watching dependencies, and seeing when a piece of data has changed, that system is consistent across the different parts. So, you don’t have to write any explicit dependency management in Meteor. That’s not because it’s magical. It’s just because Blaze and our cache layer inside the client and all the other parts of the stack all use this behind the scenes. If you’re going to build something on React or Angular or whatnot, what you end up having to do is making a decision and writing some code just to attach that to whatever strategy you’re using for WebSockets or long polling or all these other parts of the system. And that’s really half the name of the game here.

So, as far as connecting backend to frontend, there’s going to be this division, right, in a web app?

For sure.

You’re going to have a backend and you’re going to have a frontend. And it seems like Meteor is there to line those up as much as you can. And one of the things that I keep hearing from people who want to use Meteor, I haven’t talked to too many people who have actually used Meteor, is that these are so well-aligned that you can share a lot of your code between the frontend and the backend. Is this the case? And what kinds of things can and can’t you share between the frontend and the backend?

Yeah. I [chuckles], looking back I wonder if the way we showed that off was maybe a little too confusing in some ways. So, here’s what’s actually going on. First of all you’re exactly right. There is a clean separation between the frontend and the backend in any good system. So, ours is built around a simple JSON protocol that runs on top of WebSockets called DDP. We could talk a little bit about that. But the API in the Meteor app is the subscriptions that run over DDP. That’s the data flowing from the server to the client. And the method calls that a client can make, those are the equivalent of a POST in REST, the things the client can ask the server to do. And there’s no other connection. So, you could swap out the backend for a different system or you could do the same with the frontend.

So, here’s the other really neat thing. If you’re careful about it, and thanks to JavaScript which now runs really well in all of these different environments, right? It runs well on the cloud with V8. It runs well inside Chrome. It runs actually really well now on a mobile device. You can end up having the same APIs in all of those settings. So, as an example I mentioned that there’s always going to be this cache that runs on the client just to minimize the roundtrip time.

So, in most apps the cache is this special thing. So, you might, I remember hearing a talk from the guys that built SoundCloud and they have some kind of a reference counting system. And of course Backbone has a very simple cache which is just the Backbone collection. And there are different strategies for that. Normally you think of that totally separately from the way you think of your actual database in the cloud, your SQL database, your Mongo database or whatever. In Meteor we just use the Mongo API for our cache. So, if you’re writing server code, you type a find statement and it’s going to connect to the Node driver and then ultimately fetch data out of the Mongo database. That same query, that same line of code on a client is actually drawing data out of an in-memory cache that uses the same Mongo API. So, they’re built very differently inside but they look the same when you’re calling the API.

The really nice thing about that is that you just don’t have to shift mental gears all the time. You can think in one API for doing database access, one API for doing XHRs, one API for all these different things you might want to do. In some cases and in this really slick demo we showed a while back, you can even share the code. so, your validators, a lot of your model code, you may actually have the same exact function that tests to make sure the attributes of a document fit whatever requirements you have and does some database access along the way to pull related objects and so forth. And there are cases where if you’re really thoughtful about it, you can share that stuff.

I just think that as apps get more complex, the story is less and less about actually sharing the code and more about having a consistent platform and a consistent API so that you’re not juggling four or five different strategies for doing what’s ultimately the same thing.

Yeah, and that makes a lot of sense. Things like the validators and some of the other examples you gave, it makes sense to be able to share that information, certain behaviors, other behaviors in basically your model layer.

Yeah.

If you’re talking about MVC. Yeah, you want all those things to be consistent. And since you’re writing it all in the same language anyway, why not make it so that you can just share it back and forth between the frontend and the backend so that you change it in one place and then it acts the same way on both?

Yeah, that’s right. And I think a lot of application development is stuff like that. We’re always struggling with this friction just because you end up having to write a couple of different variants of something. And just think about the downstream effects of that. You’ve got two different implementations of the same validation. Now you need to have two different test suites around it. Now you’ve got to have two different strategies for updating that code when it changes. And you have this whole, do I cross-reference each of these things in comments or do I just keep it all in my head? This stuff adds up.

Oh, honestly, you’re lucky if it’s only two.

No kidding, right?

Like if you’ve got a web app and a mobile app.

Yeah.

Two different mobile apps.

Yeah, it’s nuts, right? A modern application just table stakes today, it’s 2015. You want to build something people like, it’s got to run on iOS, it’s got to run on Android, it’s got to run on the browser obviously, and you have your backend, your cloud code. This is the least agile thing I can imagine. And it’s just insane. So, the really neat thing about, there’s this term floating around, isomorphic that some people use for this. The neat thing about having isomorphic APIs isn’t some magical thing about writing one function that runs everywhere. It’s that you can be really intelligent as a software architect about not repeating yourself, especially across the model layer of your app.

Earlier you described traditional web applications as being three tiers, database, backend, and frontend. Would you describe a Meteor application as being two-tiered?

Well, there’s still a database. There’s still a backend in the sense of code that’s running in a privileged environment, some server that you control. I actually think I may have gotten turned around with my words. Normally that third tier is your proxy layer. It’s some kind of frontend that isolates the application processes from the actual internet. And you can have a similar sort of thing in Meteor. I think the bigger difference is what’s inside the tiers more than deciding where to draw the lines. All these things are built around good systems design where you have clear protocols between the different layers. It’s more this shift to stateful protocols is a really big shift. And if you try to build stateful apps on top of stateless systems, yeah it hurts.

You end up building a lot of extra stuff.

So, help me understand what a Meteor stack would look like. And specifically, which parts I control and which parts I have abstracted to the framework.

Let’s see. So, one of the cool things about this is there’s a lot of unpaved territory here. And it’s not just Meteor, right? It’s everything that’s happening I think in the JavaScript ecosystem, with npm, with all the reactive frontend. So, we’re still sorting out what to call some of these modules and how exactly the boundaries between them work. But here’s one way to look at it. So, if I’m building an app in Meteor, the fundamental thing I need is to be able to watch a database for changes and have those changes appear on my screen. So, we can talk about the different parts of the stack that are involved in that. And we can talk about what it would look like to swap some of them out.

So, the first big thing is in a traditional web framework, when you do a database query you get back a result. You get back an array or some static result set. In Meteor it’s different. When you do a query in Meteor you can actually observe the query. So, you can register a callback that gets triggered any time the contents of the database changed. And that’s really interesting because it turns out databases don’t actually support that API. You can’t go to MySQL and register for changes on a query. You can’t go to Mongo and do that. So, one layer of the stack which we call live data is a translation essentially between the traditional database structure and this real-time or what we call observable query. And you could implement that in any database you like. We ship with a binding for Mongo and we have a binding for Redis that’s in the system.

On top of that you have a protocol. So, we’re going to have to push those changes to the client. So, that’s where this DDP protocol comes in that I mentioned. And that’s another isolation point because you don’t have to use the same system on both sides of the wire. You can implement either of them separately if you like.

On the client typically where that data lands is inside this cache of your data. So, we call that MiniMongo. So, it’s the same Mongo API more or less. But it’s all in-memory. In-memory is fine because you don’t typically store that much data on the client. This is just a cache. So, no indexes, no complex structure around planning queries. It’s just a really simple in-memory cache that you access using Mongo queries.

There’s a very simple lightweight piece of code called Tracker. And I think if you only want to read one piece of Meteor code just to understand how we think, check out the source to Tracker. It’s a rendezvous between sources of data that can change, so for example this cache, and things that want to watch changes, so for example the view layer, in our case Blaze. And it’s just a really simple way of registering that you want to have a dependency on the result of something and have that rerun if it changes

And then on top of Tracker you have Blaze or Angular or React or whatnot. So, that’s your view layer. And you write your templates using, we support Handlebars and we support Jade out of the box. But in theory you could use any template. You just have to bind it to the Blaze APIs.

Okay.

So, the lifecycle of your data changing as the system recognizes a change in Mongo sends a message over DDP to the client. That gets written into the cache on the client. Tracker has registered a dependency on that because that piece of data’s actually on your screen right now.

So, that triggers Blaze to re-compute that little piece of the DOM and your screen updates.

Okay.

So, there’s a bunch of pieces of the story like that. But I think that’s a pretty good sense of what’s going on. And you’re writing your templates, your business logic, your models, your RPC endpoints. You’re deciding what data’s visible to the client, and so on and so forth.

So, it sounds like in a nutshell the Meteor data model is like this distributed data model that everybody can monitor. And everyone acts as a client, frontend clients and backend clients. Is that fair to say?

Yeah. The model is pub/sub plus RPC. So, it’s actually based in a lot of ways on the client/server architecture of the 90s back when we had Unix servers and Novell and Windows and things like that. And we can compare it to REST. So, in REST you do a GET and you get a resource. So, you get some set of things. The equivalent in DDP is a subscription. So, it has a name. You subscribe to an endpoint and you get back not a single result but you get back this flow, this publication of data. So, it can change over time.

And then the other half, in REST you have POST or PUT or DELETE. So, you have modifiers where you’re asking the server to do something for you. In DDP you have RPC. So, it’s a general function call across the wire. It takes arguments. The server can do whatever it wants based those. And that’s it. It’s a very simple, easy to understand security model. We’re not doing any kind of multi-master replication. This isn’t OT. This isn’t some new way of thinking about data and how it should be done. This is bread and butter. The clients can ask the server to do something and the client’s going to ask the server for some data.

I think it’s really interesting. So, most frameworks that I’m familiar with, you’ve got Angular and Ember and Knockout and Backbone that live on the frontend. And so, they care about everything above the layer of, “I’m going to ask the API for something.” And then you’ve got the other frameworks like Rails or some of the PHP frameworks or Django or any of these that care about everything basically from the database up to fulfilling requests and answering calls from the API. And it seems like they all have their conventions and their methods for doing things.

And Meteor has just built itself around, “Okay, well we’ve got conventions for the database up to the boundary where the API is. And we’ve got conventions on the other end coming the other way. And then we’ve got conventions about how it all talks and connects.” And so, I don’t really have to think about how all that stuff works. I can just take advantage of what you’ve already done and use the APIs that are at my disposal to make all the work get done.

Yeah, that’s right. The other thing is some of the conventions in the old system are wrong. So, it’s not just about agreeing on them. It’s about changing some of them.

Right.

Rails for example has a really great story for request/response. Though admittedly in these modern apps, Rails has turned into an RPC server. And a lot of the Rails stack starts to look extraneous. But there’s no story for pushing data. You can certainly [plum] a WebSocket and you can send data down the WebSocket, but where do they come from? You don’t have a clean way in Rails of watching a database. There’s nothing like Active Record for this modern live updating real-time stuff. That turns out to be a really basic building block for any of these apps. So, in some cases it’s about building those new blocks, something like live data. In other cases it’s about, just yeah exactly, agreeing on some conventions so that different parts of the system plug into each other nicely so that you don’t have to think about that stuff at first if you don’t want to.

So, I’ve got a question to go back maybe a step or two. So, when I think of building a Node app I think Connect and most people think Express which are mainly the same thing. Connect’s just a little more barebones, right?

Right.

So, is Meteor built on Connect or Express and I can build it in a similar way? Or is it more like a Node backend where I’m just focused on the API and I’m not actually focusing on the endpoints?

So, Meteor’s server sits on top of Node plus Connect. You can write Connect middleware if you want. And that’s actually the story for supporting legacy REST endpoints from a Meteor server. It’s a funny question though, because I think the answer is yeah, you can. But it’s like, why would you want to? The problem is a lot of that stuff again is based around request/response.

And the biggest crazy thing in Meteor and the thing that, if you come at it from this perspective of the way we used to build these apps it can be really jarring, is we’re walking away from that strictly stateless request/response model. And that turns out to be baked into so much of the software we used to use when building this stuff that it’s sort of jarring. So, it’s not that you don’t want to think about your server or your backend when you’re using Meteor. It’s just that we think the place to structure your code looks like a publication or a method call. Those are the equivalent of what would go in a controller in a Rails app or inside one of these…

Right, so…

Plugins in a Node stack.

In my mind it sounds like Meteor is a Node backend with a backend. Like, the concept of, you’re programming to an API, you’re not programming to endpoints. But you also are taking some of that API and you actually do implement it. But Meteor handles the endpoints and the transport layer. Right? Is that what you’re saying?

Oh, I see what you’re getting at. We handle the transport layer the way Rails does, in the sense of we will parse the stream of bytes coming over the WebSocket and turn it into function calls on the server. You write the functions, but we will call your function when a message for a particular RPC comes over the wire. If you want to play at that layer you can. You can plug into that part. But there’s not a lot of value in there, right? And it’s similar for building a RESTful server on Express, right? Generally you don’t care about parsing the individual HTTP stream. You just want to get your callbacks called when particular requests come in. I just think of it the same way.

So, to me that kind of, well I guess the reason that I’ve heard Meteor a lot but I haven’t wanted to play with it is because when people have explained it, they haven’t explained it in a way that connects with the way that I map things in my mind. And now, this is becoming a little bit clear. But I’m concerned. If I wanted to test things in Meteor, it sounds like I wouldn’t just be able to write a quick and dirty curl command in bash and be able to test things that way. I’d be using specific frameworks that are tailored to Meteor, that are part of the Meteor framework. What’s the testing story like? And I’m not a big testing guy like Joe.

Yeah.

But I do occasionally want to make sure that these particular endpoints are really well worked out. Or I guess in this case we’re talking about this particular API is really well worked out.

So, that’s a really good way to look at it. Here’s how I think about it. You have a problem either way, right? Option one: you want to build a modern app which means you need to be able to push data to the client. So, option one is you can do it on top of REST. You can build a traditional backend that has REST endpoints. And yeah, you can test those using curl or a thousand other things that are out there because HTTP land is well built out. It’s been something we’ve worked on for 20 years. There’s all kinds of tooling that help support it. You’ve got proxy servers. You’ve got everything, right?

But the problem is if you do that, you’ve picked the wrong starting point in some sense. You’re a bit hamstrung. You’re dealing with long polling. You can’t use curl to test your WebSocket endpoint, because that doesn’t make any sense. The WebSocket’s going…

Yeah, yeah.

Going to be pushing data. So, the other way to look at it, which is the Meteor angle on it, is we’re using the wrong protocol. Let’s just fix that. I admit that there isn’t anywhere near in 2015 the tooling and the infrastructure and the ecosystem around testing or even inspecting what a modern WebSocket-based, what our protocol looks like. But that’s a problem.

So, I was working on a site for a client where one of the concerns was there was going to be, it was for auctions, there was going to be potentially thousands of people watching a couple of really rare items in an auction.

Mmhmm.

And so, the WebSockets were crucial because anything that happened needed to be published. But as I was building it what I found is that it was actually simplest to do all of the POSTs through REST.

Right.

And then just do the updates that didn’t require any sort of authorization through WebSockets. [Chuckles]

So, my architecture ended up being really simple and that all of my more complex logic was handled through my REST endpoints. And then it was just the data that was free to pretty much anyone and that didn’t have a lot of constraints on it or didn’t need a lot of validation. It was just going downstream and not upstream. And for me, that was a really testable and really good approach, plus of course I did have an endpoint that allowed me to GET and then get the same content of the WebSocket. And that made testing easier. So, what do you think about that kind of approach in contrast to this fully immersive approach?

Yeah, I think that’s real smart. I think that’s a good architecture for what you’re describing exactly for the reasons you explained. You want to build conservatively. You want to build on top of stuff that we can test, that we can inspect. We think the world is shifting rapidly, thanks to mobile, thanks to the rise of JavaScript, in this direction of apps where that data model is more and more complex, where the interaction is richer and richer. So, in your case you’ve got a couple of things that you can send over a WebSocket because it turns out the security implications are such that you don’t have to think about it too hard. It’s no big deal. But less and less is that true.

And maybe one way to think about Meteor is if we can build a new set of building blocks that are actually better fits for that kind of a data model, for that kind of an app… and yeah, Rome wasn’t built in a day. And it’s not just Meteor. This is all the real-time world. It’s nuts. It’s 2015 and I can’t from the command line plug into anybody’s real-time API, right? I can’t connect to Firehose. I can’t connect to any of the real-time APIs that any in the major systems provide, in any sort of repeatable or interoperable way, because they’re all different. They’re all made up. They’re all ad hoc. There’s no central agreement around how any of this stuff should work.

And you just think about all the benefits we’ve gotten from everybody agreeing on first HTTP and then REST and JSON and these elements of a reasonable approach. I think we can have all that for the modern app era where everything is real-time. But we’re going to have to do it, right?

Yeah.

I think longer and longer that we spend trying to retrofit request/response architectures for this stuff, yeah we’re the experts and we can obviously do it. It’s probably still a waste of our time. But can you imagine a beginning developer going through the exercise you just described? It’s like, it’s almost impossible. And I think that’s part of the story here, is it’s going to be some wrenching change just like it was when we went from writing software for Windows to writing web software. We didn’t really keep most of the old tools. And we think the long-term vision is to build some great stuff like that.

So, a modern web application I think you’re saying is not going to be based on request/response anymore, not going to be REST based, but is going to be more rich and stateful and stream-based. What about interoperability? One of the really great benefits of REST APIs is that I can just throw up some URLs in my company’s application and third parties can easily consume that data. And maybe you’re just going to say, “Well the tooling needs to evolve.” But today, what is Meteor’s answer to that? If I want to put up an API for third parties that’s REST and still have my rich web application or mobile application that uses Meteor?

Yeah, that’s exactly right. So, two things. One is for your REST endpoint you would build a REST endpoint. So, we have a really thin layer on top of Connect so that you can publish GET and POST endpoints that old clients can call. That’s no big deal. But yeah, I think you’re going to have to define a real-time API. And the protocol that was sketched out, and I can send you a link for one of my picks maybe, is a really simple, it’s like three pages of markdown that is exactly that. Let’s all agree on the structure of that protocol so that different apps can connect to each other.

And again, it’s not just Meteor apps. It’s anybody that’s trying to do a streaming or a live updating equivalent of these endpoints. I think the world needs it. I think the world is going to settle on some kind of standard, simple, open source based protocol for exchanging real-time data. It’s nuts not to have it.

It’s probably coming. Of course we have to remember that these kinds of agreements like HTTP are generally pre-seeded by a decade of fighting. [Chuckles]

Yeah.

Yeah.

Yeah.

You just said all that and I’m thinking, “Well gosh, ten years ago people were trying to build XML RPC endpoints.” And it was nuts.

Yeah. [Chuckles] yeah. And there is still SOAP out there. [Laughter]

I have a question regarding the, so you’re reliant right now on WebSockets. And the WebSocket support in browsers and just other related technologies have come a long way over the last few years.

Yeah.

So, is Meteor just not supported on older browsers or do you have some kind of polyfill that you’re pulling in? How do you handle that? Do you fall back to polling?

Yeah, we do. We use SockJS as our polyfill library for that. So, if you’re not able to use a WebSocket, and I forget. We’ve disabled some other transports. But it’ll fall back to long polling. We actually disabled WebSockets for the first year or two, because what we found is something really interesting. It’s different now. But WebSockets didn’t support compression for a long time in the browser. So, it was actually more efficient it turned out in apps, what we saw was using long polling and HTTP and by doing that gaining the benefit of hardware compression, native compression for the browser.

You know, and a lot of people…

And avoiding the overhead of the long polling. But we switched to WebSockets because that support as you said got better and better. And so, on a modern browser you’re on a real WebSocket. And I don’t think there’s any doubt that’s the future.

So, a lot of people were under the misconception that long polling was really inefficient because you’re opening up a new connection every time. And really, that’s not true. It’s just a little bit more data than a WebSocket, right?

That’s right. That’s exactly right. It depends a lot on the workload. But in many cases, long polling worked just fine, no problems at all.

And I noticed in some, and maybe SockJS is different and I hope so, but I noticed in a lot of the implementations of WebSocket wrappers, people would go through such great lengths to optimize every little thing. So, they’d have this Flash plugin and they’d have… [Chuckles]

If your browser doesn’t support WebSockets but it’s somehow a browser that supports progress events then it would use progress events. And all these layers and layers and layers. And to me, that always just, I wondered. Is it worth it to do all that? Or should you just stick to, if you’re targeting a platform that doesn’t support WebSockets proper, just use long polling and forget all of those six dozen other transports?

Yeah, I’m with you.

CHUCK: So…

Or just kill the old platforms. Ugh. [Chuckles]

So, one other place I want to go with this, and we’ve talked about the difference between the frontend and the backend. So, I tried to build a MEAN stack app. I’ll admit that I didn’t spend a lot of time doing it. But I figured out pretty fast that even thought I knew Angular I still had to learn Express in order to make it work. And I’m wondering how different does your frontend code look from your backend code in Meteor? Because some of it is going to be dealing with the frontend view, templating and stuff like that. And the backend is going to probably be dealing more with business rules and models?

Yeah. Here’s the deal. It ends up looking a lot like a Rails app, honestly. So…

That makes me happy. Now I have to try it.

Yeah. This isn’t rocket science. Most apps have business logic. The business logic is really simple stuff that involves hitting the database a few times, doing some, maybe you hit an external REST endpoint. So, maybe you do an HTTP request to a geocoder or a fraud scoring service or anything like that. You do writes to the database. That’s all there is. On the template side, our templates are a lot like ERB in Rails. It’s Handlebars so the syntax is a little different. But you have a helper inside your template that maps to a function call that you wrote in JavaScript. That function typically does a quick database fetch, which is coming out of that cache remember. You don’t have to think about any of this stuff. But it looks just like a controller action in Rails.

One interesting thing there which we took a lot of heat for but I think is proving to be the right call is most APIs in Meteor are synchronous. So, when you do a database read in Meteor, you run collection_name.find and you get back an array. It’s really nice. So, you don’t have this nested set of callbacks that people are used to when building inside the Express architecture. That’s…

So, how do you do that?

We use Fibers.

Okay, so you use Fibers. So, you still do get the benefit of being able to have multiple user interactions.

Oh yeah, yeah. Notice fast. The cool thing is JavaScript is a much better language than Ruby, right? And it has…

CHUCK: Ouch.

[inaudible] infrastructure. On a clean sheet of paper we can debate it. But in the real world, Google…

Yeah, guys…

And Apple and Microsoft have put so much into the JavaScript runtime that there’s not dynamic language that’s anywhere near as efficient or capable at this point.

Do you hear that, Chuck? Are you taking advice?

Alright, he’s right.

That [inaudible], right?

Believe it Chuck, believe it.

And yeah, I don’t think we explain this super well. But Fibers is a control flow system. It doesn’t mean we’ve lost all the advantages of Node being asynchronous and being able to run in a single process and serve multiple requests. None of that changes. It’s just that the asynchronous callback style in Node is really great if you’re building that, I think about the classic Node backend stuff that LinkedIn did early on and talked about or people like that where you’re really trying to build a very simple RPC server that sits between a database and a client. And what it has to do isn’t much more complicated than marshal data coming out of the database into JSON format for the REST client.

Once you start writing actual business logic, think about your classic Rails controller. It’s like, request comes in. First I’m going to fetch the user object out of the users table. Then I’m going to go fetch some other thing using eager association. So, I’m actually doing two or three database queries in a row. And then I’m going to look at it. And then maybe I’m going to do a write and then maybe something else. And if you do it in asynchronous style, you end up five or six levels deep in your callbacks. If you have a bug you don’t have a stack trace. So, it’s a nightmare to sort out what’s going on. It’s just, synchronous code is, well it’s the code you can read, right? So, it’s just worked out really well to be able to write software that way, in JavaScript.

Traditional request/response systems like Rails, or not really Rails specifically but REST, have two really valuable properties in my experience. One of them is that load balancing is very easy to do, because every request comes in and can be routed to a new resource. And the second one is it’s pretty easy to achieve downtime-less deploys. So, if you want to deploy new code you can do that without interrupting the service, because requests can be held and routed elsewhere while you upgrade. How do you do that in a stateful protocol-based system like Meteor?

This is a great question. So yeah, it’s hard. That’s just, them’s the breaks. It is a stateful system. That’s what client/server was like in the 90s and that’s what these modern apps are going to be like today. And that’s because there is actual state, right? There’s a WebSocket. There’s an open connection to the backend. And implicit in that is the fact that there’s state on that connection. Who’s logged in? Who is the user? What kind of security has been done across the wire for that particular socket? So, a couple of things emerge, right? Unlike the REST world, you need some sort of strategy for reconnecting and making sure that you don’t call methods multiple times.

Here’s another way to look at it. In the web era, it was mostly okay if the HTTP connection dropped or if you had some sort of weird problem, because if you hit a POST endpoint twice, it’s like no big deal, right? And that’s usually because what you’re asking of the server is pretty simple. Maybe you see an error or there’s some kind of exception that happens. But it’s okay if you call POST twice accidentally. As more and more of the complexity of the app moves into the client code and what the client is doing is actually calling more complex transactions back to the server, it’s more important that that system actually behave in an understandable way.

So, we have this idea in DDP that you can call a method twice. And if it’s a second call, it has a replay cache so it’ll remember and it won’t actually run the function a second time. You just need a bunch more infrastructure for it. And I don’t think that has to get super complicated. But it’s a good example of how when people build apps out of parts using MEAN stacks and all these other systems like that, this is the kind of stuff I think that doesn’t usually get done. So, that makes sense if you’re building a product. You got other things to worry about than handling the super weird corner case of a WebSocket disconnecting at exactly the wrong point in your authentication set of messages.

But it’s actually possible to get it right. And if we can centralize this into a standard protocol and a standard library that lets you use that protocol, we can just do it right. So, some of that stuff’s

baked into DDP. Some of that stuff is about DevOps for a modern app where you have to make sure the WebSockets or the long polling does get routed to the same backend each time. Some of this stuff is just around, I don’t meant to punt on it, but the ecosystem is going to have to build up around the right kind of DevOps and application architecture for doing this stuff exactly for the reasons you’ve laid out. You got to solve the problem one way or another.

So, I have the benefit of having done this same show for the Volt framework in Ruby this morning. One of the things that he kept talking about was that because they’re using WebSockets also, that if something changes, say I POST data back to the server, then the WebSocket can inform another client that something happened.

Yup.

How do you handle the case when two things post back to update the same thing at the same time?

Last writer wins. It’s basic RPC.

Okay. And…

And that’s the way, I think that’s the way it should be, because people understand that. That’s a programming model you can work with. And you can build an understandable, predictable system on.

Yeah.

It’s when you start getting into this weird, I think if you’re trying to build Lotus Notes, then a multimaster solution actually does make sense.

Mmhmm.

Very few applications actually need OT, right? It’s not the normal semantic you want when you’re trying to write data. And here’s an example. Suppose you’re building an app where you have business logic and you want to run some kind of transaction. So, the server’s going to check to see who you are and whether you have the right to do that. It’s really common where that’s easy to do in an RPC model like Rails has, or like Meteor has. It’s almost impossible to do in some sort of replication model where it’s just not clear how to govern these updates that are related to each other, if they’re being replicated from one database to another between clients.

Right. I do like though the idea of having it update things and stuff like that. It seems to me that when we talked about ShareJS a couple of weeks ago it was the same kind of thing where those updates got posted back to the server and then updated back on the other end. But they were using operational transforms and things like that.

Yeah. Let me be super clear. When a client wants to make a change in a Meteor app, it calls a method which means a message goes over the network to the server which runs a JavaScript function with the arguments that the client provided. That’s Meteor’s job, is to marshal that request across the wire, only run it once, things like that. What your method does is entirely up to you. Typically it turns into a database write. So, your options are either last writer wins because you don’t check, you just do the write. Or you could try something where you test to see what the current value is in the database and so on. But now we’re back in very comfortable, understandable territory that every systems developer or every web dev knows about. They got the same story from Rails or PHP or Node or whatever they’re using on the server. And it’s secure, right? We can reason about it.

Yup. So, one other question I want to ask about is mobile. So, you said you can write mobile apps with this. Is this on top of Cordova or is this a completely different approach?

Yeah, this is Cordova. So, the last big thing we wanted to get in for 1.0 is a really slick, there’s two parts of it. One part is Meteor lets you pull in Cordova packages. You can add the camera support, or GPS, or vibrate, or whatever, address book contact stuff that comes out of Cordova that you need.

And then the other thing is there’s a bunch of infrastructure for packaging up the app, not just for the app store initially but also so that you can do over the air updates. So, it’s really cool. You can build your app and then when you deploy a new version of the app to the server, mobile clients go get that update automatically. And they don’t have to go through the app store for an update. And if they’re offline, it’ll still work using whatever version has been saved on the phone. So, all of the plumbing and the security and just details around that are the other half of the story. And it’s pretty cool.

JavaScript’s gotten much, much better on the phone. I know that there’s this classic story about Facebook bailing. But that was actually a long time ago. Apple sign pages now in JavaScript. There’s I think fewer and fewer apps really need to be built using pure native APIs. If you care about every last pixel and really high framerate, yeah that makes sense. Certainly I would write a game in native. But for your classic app, the thing that most people want to be building, I think JavaScript and HTML is rapidly becoming in many ways the better choice just because the development tools around it are so great now. And of course the fact that you can use one codebase instead of four.

And as far as the native story goes, I know that it’s not quite there. If you look at what Mozilla’s doing with Firefox OS and their Flame phone, they’re making JavaScript native performant as best they can. And it will get there. I think they’re at, it’s half as fast as C with their asm compiler stuff right now.

Yeah, asm’s nuts. I love that stuff. Yeah. I think we can play this movie forward and predict where we’re going to land, right? There’s just so much force behind JavaScript that I don’t see how anyone stops it.

So, here’s something that I see as a trend. I think Golang is an excellent language. And the communities are finally starting to realize that. When it first came out, I was like, “Oh my gosh. Mind blown. This is the way that we should have been programming since the beginning of time.” And now, Go I think is starting to gain some of the traction, or some people are falling away from Node.js and defecting to Go. But the problem is you can’t run Go in the browser.

But there are also some projects that if they were to come about maybe could change things for how we view the web, like there is an LLVM compiler project for Golang. If that were to become complete or get some good sponsorship from Google, that would essentially open the door to be able to run any language, compile to JavaScript, and then gets compiled through asm, which means that every language would run half the speed of C in the browser. What do you think about that in terms of potential future derailing JavaScript? Or do you think no, that’s not going to happen?

Well, I think that is the future. But it’s not derailing JavaScript. I think it’s the opposite. I think JavaScript is the C of this generation. It’s the…

It’s the Assembly language, I think, of the future on the web.

Well, C is Assembly, right?

Exactly. No, what I mean is Unix was built in C. All of the standard libraries were built in C. That was the base language that everything was based off of. The calling conventions were always the C calling conventions, which yeah okay, it’s Assembly. It’s like, what register do you put the arguments in? On top of that you can build anything. So, source maps have gotten really good. I have to admit, I do almost all my work in JavaScript because I think that’s just required for the platform. But we’ve got people building in Coffee and it’s amazing. You can debug in Coffee on the client and on the server. It’s getting better and better.

And so, Volt’s another example. I think it’s a little weird. But if you want to write your whole app in Ruby, go for it. That’s cool. So, I think there’s a really bright future for alternative languages that do run. But I think we should think of that as yet more centralization around JavaScript as the root of that ecosystem. And then just really good tooling that allow us to pick whatever language is most convenient for us to think in, for the application that we’re building.

You know, the other thing is I think on the server, when you say a lot of the Node people have moved to Go, I think Go’s awesome. If you’re writing a server thing, a message bus, or any of the stuff that I think a lot of the original Node community was interested in, Go is a really good choice. And it does have a lot of advantages. And I think of that actually as quite separate from this web world, even though at one point we were using JavaScript for both of them, if that makes sense.

So, what are some of the cool projects that are using Meteor today?

Well, there’s a bunch of stuff out there. The thing I saw last week that I’m really excited about is, so we’ve been talking about DDP a little bit. So, there’s a fellow who built a DDP implementation in Objective C for iOS. And it’s amazing. It has all of the same features that the Meteor client stack has around resolving updates from the server and letting you do speculative updates on the client without waiting for the server. This cache layer I described, all of that’s there except it’s built in Objective-C. And it ties into Table Views and Core Data so that you can build your view layer in the normal stuff that any Cocoa developer would know and expect and connect to a DDP backend.

So, he’s got these really cool demos where he takes our multi-user to-do’s application that’s live updating and then alongside in the video you see the native iOS app updating with the same data at the same time. And I think that’s really interesting because DDP is not a Meteor thing. Like I said, we have to agree on a basic real-time protocol for the web. And I think seeing the emergence of some other clients and servers is part of that story because it means it’s useful for people that are doing something a little bit adjacent to what we’re talking about. So, that’s really exciting.

The other thing I love is just startups love this thing. It’s ludicrously fast to build an app in because you don’t have to worry about a lot of the technical underpinnings. So, we’re seeing a lot of excitement from some of our friends here in the Valley or startups elsewhere that have been able to build really great products with just a couple of people really quickly. And some of that’s on our website that I was excited by. But we keep seeing more stuff pop up over Twitter.

I want a world where writing an app isn’t such a big thing, where first of all you don’t need all the specialized knowledge to do it. Because I think there are some people with really cool ideas that just aren’t going to learn 25 different technologies before they want to get an app running. And partly because even the experts, if the app is going to take me two weeks to write, I’m never going to write it. I’m a busy person. I’ve got all kinds of stuff to do. But if I can, if Meteor’s the thing that lets me write this thing that I’ve had an itch for in a weekend or some late night, I’ll do it. And that’s why we built it. We just got tired of how long it took to build stuff.

So, here’s another question related to that. Sometimes people want to get started with programming and they want, you’re talking about Novices.

Yeah.

So many novices run Windows. And I am just completely at a loss because it’s like, they don’t know how to open up a terminal. And the concept is just completely foreign to them. They think it’s the 1980s. They don’t realize that it’s the 2015 and the terminal has all the latest hotness in it. So, does Meteor have good Windows support? Or if somebody is from the Windows world, is there a nice story that helps them get to Meteor?

We’re working on it. Meteor 1.1 is going to be Windows. So, we’ve got to do it. I admit I am not a Windows user.

Well, who is? Nobody on this podcast.

Actually, I’ll tell you who is.

Zing.

It turns out an enormous number of developers are.

Yes.

And we should care about them, too.

I ask because I just met a guy at church the other day and he was talking about how the job he’s doing now, he’s thinking that maybe development would be a better path for him.

Yeah.

But he’s only ever used Windows. And I was going to try to explain. [Chuckles]

And I just realized I have no foundation for teaching him what step one is.

Yeah. Look, the other thing is I think development in the cloud is coming. I don’t know if it’s quite there. I know for experts the text editor is so fundamentally part of the experience that you’re going to have to pry Emacs out of my cold, dead hands. And I’ve never used a web-based development environment that [inaudible]

Sorry, I have to get this in. [Inaudible] going to pry…

Emacs! Emacs! [Laughs]

going to pry Emacs out of your cold, dead arthritic hands.

Yeah, we’re [inaudible].

It was a Freudian slip. He meant vim.

No, I meant Emacs. But…

Whatever. The point is I think it’s getting closer and closer. We’ve seen some demos of some of this stuff. There’s, I think it’s Nitrous has been working on this. GitHub has Atom which I think is a really big part of it. So, there’s no doubt that sooner or later we’re going to get to a place where you can open up a tab in your browser and do all your work in the tab. And it’ll be great. But we’re not there yet. It’s really important to me that we find developers wherever they are. And that absolutely means Windows. I didn’t want to wait on that for 1.0. But it’s got to come just because there are so many people looking for a way to be able to build apps like that.

Yeah, popular ecosystem. You have to take care of those folks.

[Inaudible] got someone looking out for them.

And also, there’s god knows how many .NET developers out there and they have, they know they’re on a dead-end platform, right? They know that they need a way to get to mobile. They’ve heard about this cool JavaScript stuff. But they like the Microsoft, I think it’s not just Windows, I think it’s the environment as a whole. And I think there’s a way to serve that community while still having the same Unix-y good system design, good user interface design that a lot of us have grown to appreciate about OS X and Darwin.

Alright. Well, if people want to get started with Meteor, where do they go? What do they do?

www.Meteor.com. There’s an install link. There’s a tutorial which is the thing I would recommend going through. It’s about 12 steps. It’ll take an hour. And we’ve tried really hard. One of the things I’m most proud of is the first five minutes of Meteor are really fun. The first hour is really productive. You actually can get, as long as you know basic JavaScript and HTML, you can get a real app working on your phone, on the browser, that multiple people can use, that actually does something useful. I think that’s the mark of something exciting, because I think most people really want to see what it’s like right away.

I remember when I saw the Rails screencast 10 years ago and I just thought, “Oh, cool. I can get my stuff done now. This looks really exciting.” And we’re trying to create something similar to that for a modern app experience. So, the tutorial covers what’s involved in building a multi-user to-do application where you can have user accounts and login and you can have different-sized screens for the different devices. And you can actually even get it into the app store. And then you can go from there.

Alright. Well, let’s go ahead and do some picks. Dave, do you have some picks?

Oh, you know I do. Alright, first of all I want you to know that I don’t really like Sudoku. But basically I don’t like any game that I feel like I could write a computer program to solve for me. But I did find this really cool wooden color version of Sudoku where you have this wooden board and colored little wooden balls that you place in little divots on the 9 x 9 board. And it’s called Colorku. And it’s pretty fun. So, I’ve played it with my kids and wife. And I actually enjoyed it against my will at first. But I’ve come to enjoy it and I didn’t think I would. So, that’s Colorku.

The next thing I’d like to pick and this actually is really pertinent to our conversation today which I think has been really cool is a talk from April of last year by Gary Bernhardt, one of my favorite speakers in the industry, titled ‘The Birth and Death of JavaScript’. If you haven’t heard this talk, you really owe it to yourself for just the entertainment factor alone not to mention the enlightenment you’ll get, to go and take an hour and listen to this talk. It is a 40-year history of the JavaScript programming language starting in 1995 and ending in 2035. So, it’s really cool. His “history” (air quotes) is really interesting about what JavaScript became over the last 40 years ending in 2035.

So, that’s it. Those are my picks for this week.

Alright. AJ, picks.

Okay. I will pick anything that can get me from starting off every sentence with “K, so,” because I do that all the time and I have not figured out…

Cheese.

What is that?

Queso.

Oh [laughs] right. So, there’s a really good article explaining why music sucks today. And what that translates to is why you hear hipsters say that vinyl sounds better than CDs. And the truth of the matter is, is that vinyl does sound better than CDs but not for the reason that you think. And so, this article on The Loudness Wars takes you a little bit through history and why it is that we don’t have the fullness of sound in modern pop that we did in the older stuff that we would have bought on vinyl in the era that I was alive, as well as why vinyl that is produced today often does sound better than CDs. And it’s part of the mixing technology.

Also, I’m going to do a shameless self-pick. I talked I think about two weeks ago a little bit about how I was building a blog generator system. And I finally got that to the point where it’s not half bad. And you could reasonably start static blogging in it. You could host it on Google Drive or on GitHub pages or on Dropbox if you’re one of the lucky couple of thousand people that signed up before 2011 and they dropped public folder support. So, if you’ve been thinking about hosting your own blog, check it out. It’s in Angular. It also runs in Node but it’s built for the browser. So, everything runs in the browser. And I think I’ll just stick with those for today.

Alright. I’m going to jump in with some picks. The first one is if you’re interested in the mobile app, Meteor, Mongo, Cordova stuff, there will be a talk at JS Remote Conf by Josh Owens. And that will be on Friday the 12th. So, mark your calendar. It’s going to be in the evening US time. I’ve gotten a lot of feedback from people in Europe. Don’t worry, next time I do something like this it won’t be in the evening US time so that you can attend it. So, we’re going to be doing that.

And the other pick that I have is not a technical pick but it is something that’s pretty awesome. And it’s an email course by John Sonmez. It’s his blogging course. And so, if you’ve been thinking about getting into blogging this is just a terrific way to figure out how to get started and figure out how to build an audience so that you can better find jobs and things like that. So, I’ll put a link to that in the show notes. And Matt, what are your picks?

I’ve got two. We’re really into systems architecture here. A lot of us studied some of these classic things that got built in earlier generations of computing largely because we don’t want to make the same mistakes. But there’s’ one that’s just totally off the wall that came across our radar a while ago. It’s this thing called SAGE, the Semi-Automatic Ground Environment. It’s this massive, and I mean that in every sense of the word, system that IBM built for the US government back in the 50s that would automatically control every missile, airplane, air defense system, the whole US military infrastructure in the event of some sort of attack.

So, as I recall this thing employed something like a third of all programmers on the planet while it was running. It cost more than the Manhattan Project. It had 20 some odd buildings back when computers were the size of refrigerators. These things had hundreds of these things. And just an audacious attempt to orchestrate something that you could probably do with a couple of iPhones today. [Chuckles]

Really awesome. And of course all of the technical documentation and the design docs are all on the web now. Really cool to see how IBM thought about this stuff back then.

The other one is there’s a TV show actually that came out last year on AMC which I think is actually really good. Halt and Catch Fire. I don’t know if you guys have seen this.

No.

It’s cool. It’s a story of this renegade computer company in the 80s in Texas building a PC clone. And I don’t know actually if they’re doing another season or not. I haven’t followed it very closely. But it’s a really cool show. I think I’m a little younger than that. But they capture a lot of the craziness and the energy around the explosion of the PC and the computer world back in the 80s.

And not the most high-minded thing on TV but it’s pretty cool.

It has been renewed for season two.

Awesome.

Awesome.

I’ll definitely check that out.

Alright. Well, thanks for coming, Matt. It was super awesome to talk to you about this.

I don’t have time for it and I already went and downloaded Meteor and I’m playing with it. [Chuckles]

I appreciate it. This was fun. I enjoyed it.

Alright. Well, we’ll go ahead and wrap up the show and we’ll catch you all next week.