DAVE:
Cats are just like living with other adults.
[This episode is sponsored by Frontend Masters. They have a terrific lineup of live courses you can attend either online or in person. They also have a terrific backlog of courses you can watch including JavaScript the Good Parts, Build Web Applications with Node.js, AngularJS In-Depth, and Advanced JavaScript. You can go check them out at FrontEndMasters.com.]
[This episode is sponsored by Hired.com. Every week on Hired, they run an auction where over a thousand tech companies in San Francisco, New York, and L.A. bid on JavaScript developers, providing them with salary and equity upfront. The average JavaScript developer gets an average of 5 to 15 introductory offers and an average salary of $130,000 a year. Users can either accept an offer and go right into interviewing with the company or deny them without any continuing obligations. It’s totally free for users. And when you’re hired, they give you a $2,000 bonus as a thank you for using them. But if you use the JavaScript Jabber link, you’ll get a $4,000 bonus instead. Finally, if you’re not looking for a job but know someone who is, you can refer them to Hired and get a $1,337 bonus if they accept a job. Go sign up at Hired.com/JavaScriptJabber.]
[This episode is sponsored by Rackspace. Are you looking for a place to host your latest creation? Want terrific support and high performance all backed by the largest open source cloud? What if you could try it for free? Try out Rackspace at JavaScriptJabber.com/Rackspace and get a $300 credit over six months. That’s $50 per month at JavaScriptJabber.com/Rackspace.]
[This episode is sponsored by Wijmo 5, a brand new generation of JavaScript controls. A pretty amazing line of HTML5 and JavaScript products for enterprise application development in that Wijmo 5 leverages ECMAScript 5 and each control ships with AngularJS directives. Check out the faster, lighter, and more mobile Wijmo 5.]
CHUCK:
Hey, everybody, and welcome to episode 142 of the JavaScript Jabber Show. This week on our panel, we have Joe Eames.
JOE:
Hey, everybody.
CHUCK:
Dave Smith.
DAVE:
Hello, world.
CHUCK:
AJ O’Neal.
AJ:
Yo, yo, yo, still coming at you from Provo.
CHUCK:
Jamison Dance.
JAMISON:
Hi, friends.
CHUCK:
I’m Charles Max Wood from DevChat.TV. Quick reminder, go check out JS Remote Conf, JSRemoteConf.com. We also have a special guest this week, and that’s Joseph Gentle.
JOSEPH:
Hi, everybody.
CHUCK:
Do you want to introduce yourself real quick?
JOSEPH:
Sure. I’m Joseph. I work on ShareJS, and I’ve spent the last two years working at Lever, although I quit a few months ago. I used to work on Google Wave years ago before it got cancelled, and I’ve sort of spent the majority of my time since then programming, trying to rebuild a lot of the text stack that we had since then, that we had done, and put in the Node.jn world.
DAVE:
Google Wave? That’s awesome.
JOSEPH:
Yeah. I know. It’s totally my streak right now. Yeah. I know. It was great. I find it really interesting now because I talk in conferences and things sometimes, and I always ask people whether they had a Google Wave, and there seems to be like fewer and fewer people who know it, but…
DAVE:
Oh, no.
JOSEPH:
Yeah. Yeah. I don’t know. It was a great piece of tech. It was crazy. Like, the team was crazy. And it failed for all sorts of completely sensible reasons, but yeah, I’m really glad to have worked on it. It was fantastic watching a big company implode in and on itself and destroy one of its prized possessions. [Laughter]
JAMISON:
It’s like that obscure, indie band that goes on to inspire a lot?
AJ:
No.
DAVE:
Like a whole new genre of music or something?
AJ:
No. Google Wave is like web TV. It just came 15 years too early. [Laughter]
DAVE:
Wait. Are you saying that web TV…? Oh, wait. That is kind of thing though, isn’t it? [Laughter]
AJ:
Well, I mean Netflix, right? I mean…
JOSEPH:
Watching it. And like Justin.TV, a.k.a. Twitch, that’s just been sold to Google for some ridiculous… No. Yeah. Wave was super too early. We had a team of like 60 engineers working on it which was way too many engineers.
DAVE:
Wow.
JOSEPH:
It was 1.1 million lines of Java when all was said and done.
DAVE:
What could possibly go wrong? [Laughter]
JOSEPH:
Exactly, right? But then at the same time, all of the people who used to work on the Wave team, like one or two people ended up co-founding Firebase. Google Wave bought EtherPad, and then when EtherPad guys left Google, again, one of them went to… as one of the co-founders with David Greenspan, as one of the co-founders for Meteor, like I’ve gone and done ShareJS. I think there’s Google Wave people everywhere now and were just quietly incepting the world on a whole bunch of the IT code and stuff.
DAVE:
What year was it that Wave was finally cancelled?
JOSEPH:
I can’t even remember. It was several years ago.
DAVE:
Was it before the iPhone came out or so around?
JOSEPH:
No. No, the iPhone had come out like a year or two before then.
DAVE:
Okay.
JOSEPH:
It was sort of i9 days. The web browser was… Browsers were terrible back then.
DAVE:
[Laughs]
JOSEPH:
They got released in May of 2009 and got cancelled in August of the same year.
DAVE:
Okay.
JOSEPH:
So, yeah. The browsers were terrible. We had internal… We use Wave for everything on the Wave team, maybe unsurprisingly. But we have this giant Wave that scrolls from over the page, which is all of the open web browsers bugs that we found through the process of making Wave. Then we had open tickets and stuff on different browsers. Like we push the browser really hard, which is something that people didn’t really do back then at all and in all sorts of weird ways, including our own scroll bar, which was a terrible idea.
DAVE:
[Laughs]
JOSEPH:
But, yeah. I don’t know. It was what it was. Yeah.And the client took like… it was like several hours of CPU time to compile Google’s internal data centers. It only took about five minutes of clock time, but it compiled down to like 350K of JavaScripts compiled through GWT. And yeah, because that was probably, it was seen as a good idea back then.
It was launched in 2009, but I think it was started… work started about 2006, and Node.js was not a thing then. Node.js was barely a thing when I left Google and started ShareJS, which was right after. And I was working on ShareJS against Node.js 0.4. It was a crazy, weird, little thing that I just did for fun because, “Oh, hey. This Node.js thing looks cool.” Like this is before npm was around. Like the list of packages was a page on a wiki somewhere. It wasn’t wiki, it was on, I think it was a GitHub like markdown document that you could submit pull requests against so that you can have your package added to the view markdown file on Node.js itself. But, yeah. I mean like that’s the state of Node.js back then.
And yeah. Like, it’s easy to figure out how much time’s passed, how much better and faster it got in the last few years. Like now, we can just… you can take for granted the fact that we’ve got a modern version of Chrome, a modern version of Firefox. If it’s IE, then at least it’s going to be IE9 or 10, hopefully IE10, 11. Even IE is now semi-modern, and you can just assume that.
But back when Wave was around, we got a huge amount of crap for dropping old versions of IE support. That’s probably reasonable. Like, we just straight out ran out of man hours. We had a version of Wave working on I think IE7 for most of the development period, but it took about…
Yeah, but like it’s --
DAVE:
You masochists. [Laughter]
JOSEPH:
But people were using IE7 back then. So like, if we wanted to be compatible, we needed to, except for that IE7 loading 350 kilobytes of JavaScript, it took a long time. Like it looked like the browser had hung while it was loading all the JavaScript, and it was a terrible experience. Yeah.
CHUCK:
I can totally see that. “Have you gotten on Wave yet?” “Well, I opened it. I’m going to go get lunch.”
DAVE:
Just for the benefit of our listeners who may be wondering why we’re talking about Wave with a ShareJS topic today, Joseph can you tell us how ShareJS came to be and how it relates to the Google Wave back in the day?
JOSEPH:
So, Google Wave, for all of the people who haven’t played around with it, and unfortunately it’s not an experience you can have anymore, the original idea was that we should have this sort of like these things that are kind of like emails but also kind of like GoogleDocs where you can invite more people to them and then have a conversation in the document, and everyone can just see each other’s changes live, and everyone can make changes. Everyone can update each other’s comments, and then it just adds you as an author to that comment as well. Like, it’s this sort of weird technology that we didn’t really know what people would use it for, and it took us awhile to figure that out.
But when Wave got cancelled… I mean like… It’s a great idea. Like, this is something that we need in the world. I want to have the… someone called the glorious messaging bus in the sky.
But like, part of the other idea with Wave was that it should be federated, so you should be able to run your own Wave solo the same way you run your own email solo. And then if I, like, start a Wave with you, then my server and your server talk directly, and there’s no like… there’s no Facebook reading all of my messages. It just goes between respective servers in our companies.
JOE:
Mark Zuckerberg just rolled over in his grief. [Laughter]
DAVE:
He’s a –
CHUCK:
I was trying to figure that out to break that to you.
JOE:
It was a preemptive role. [Laughter]
JOSEPH:
Anyway, yeah. But when Wave got killed, I was like, “Oh, this code doesn’t need 1.1 million lines of java to work,” so I rewrote it in about 2,000 lines of JavaScript just on top of like one web stuff. And it worked great, and that’s what ShareJS originally was. It was just this like… it didn’t do federation. It didn’t do all sorts of things. It didn’t do rich text. It was just like this simple world for doing collaborative text documents and having that work as a simple library for people making web apps, which is something that it still is.
I've messed around with the idea of it, and it ended up being with like fabric. They’re not just text but also had a rich text to it and with added JSON OT support, so you can have obviously JSON documents and do collaborative operational transform operations. So you collaboratively edit a JSON document with other people. But this is made totally like giving a spiel about line and toy which is great.
JAMISON:
You said OT and operational transform. Can you tell us what that means?
JOSEPH:
Right. They both mean the same thing. OT just stands for operation transform. OT is a set of algorithms that we developed. I mean the OT algorithms that ShareJS uses are from 1995 or something. This stuff has been in academia for awhile.
The core algorithm, basically the way to think about it is real-time subversion. It’s like crappy, realtime Git. That’s what the algorithm is. That means that every change that you make is this sort of like special commit in a sense. They think it’s propagated immediately to everybody else. And in the same way that something like Git can handle merges. So if two people are collaboratively editing some code, when I pull your changes in, it’ll merge in your changes to match with my changes.
Operational transform has a facility to write a function that does that merge, and the way that you write that merge function, you often end up, and this is how ShareJS works. It’s conflict-free. So my individual characters will get merged in with your individual characters. And we just see each other typing.
JAMISON:
Hang on. How does that work? So if you’re sending… if you’re sharing some JSON, and I delete a field, and you modify the field, isn’t that a conflict?
JOSEPH:
Well, so you can decide what to do in that case. The way that ShareJS handles that is that if you delete the field while I’m editing the field, then the field gets deleted. All my edits get deleted too. So it’s not necessarily what you want in all cases. This is something… I’m looking at replacing the JSON type, and this is something that we’re looking at changing like that particular case. You have to be very careful with OT and specifically if you’re using the built-in types. It’s there’s this kind of interesting field of knowing what all the different operations do when they happen collaboratively.
Realistically, if you delete a field while I’m editing it, then tough luck for me. If you saw my changes before you hit the delete button, you’d probably still hit the delete button, so my changes get deleted anyway.
But it’s great for things like say if you have a JSON document that has a list in it, you and I can both be inserting into the list at the same time. And our respective inserts into that list are going to be automatically merged correctly so that they all appear in the right places in the list. If someone else deletes something from the start of the list, then when they see my change, they’re not going to think that it’s in the wrong place. It all just kind of like works, which is really amazing.
CHUCK:
Just to clarify a little bit, this isn’t federated. So it’s got to both point to the same server to do at least passing the data around, right?
JOSEPH:
Right, exactly. Yeah. So it’s designed for basically Node.js web apps. And there’s one server, and then your browser then connects to that server via, well, these days, via web sockets. And then the server manages all of that. ShareJS now supports multiple servers, so you can have multiple frontend servers, but ultimately, there’s still a set of servers that all the clients are connecting to.
That said though, basically every time I talk about this, someone asks me that question, and it’s a question of like, “Well, could we do this in a federated way? Could we make it so that we have multiple servers, and we have these documents that are long-lived that people can communicate in, like between the different servers?” This is something that I really care a lot about and actually to the point that, and this is really exciting news that I haven’t told anyone, I recently accepted a job offer in a company in London called Spatch, and we’re actually looking at doing this.
JAMISON:
That’s sweet.
JOSEPH:
So I’m going over there really soon, and we might… we’re applying this to start… obviously, this is early days, but we’re talking about building exactly this, a version of ShareJS sort of thing that also will be federated and distributed, and hopefully, would consider entering in encryption on it. So it will be a fantastic messaging platform that people can use to build applications.
DAVE:
Joseph, I’m not totally sure I understand what federation means in the context of an operational transformer share document. I understand what it means to have a couple of people editing the same document and having their edits appear real-time for them, but what do you mean with federation?
JOSEPH:
Well, say, take email. Email is quite different from Facebook because at Facebook, all of the messages go through Facebook. And with email, if I am at A.com and you’re at B.com, and I email you, then the email goes from me to my server to your server to you. There’s no centralized authority that gets to control the entire platform.
Federation in OT platform is kind of similar. The first thing that we need to do is move ShareJS from something that works kind of like subversion to something that works more like Git where anyone who’s on a document can push and pull their changes to everybody else who’s also looking at the same thing. So obviously, the first thing that would allow is if you wanted, you could build an OT-based source control system if you wanted that. But then, beyond that, it means that if people want to build their product, like products that are more like, say something like email, except email with collaborative editing. So it’s like a weird hybrid merge of email and Google Docs.
Then there’s no one true site that you have to go to to be able to access this product. You can host it yourself, and I can host my own version of it, and I can open up a Spatch Wave email, collaborative, editable thing with you, and we can both be talking about where we’re going to go on holidays with a blub at the top of the whole thing describing as with conversing, we just go back and edit the master plan. That’s the dream that we have this kind of system that will let us just be able to arbitrarily edit documents.
And this is… Sorry, I’m just getting into my regular rant. You guys should absolutely be interrupting me a lot more. One of the things I’ve been seeing with ShareJs –
JAMISON:
It’s so good though.
CHUCK:
Jamison’s just too nice.
JOSEPH:
Right, yeah.
AJ:
Let him speak. [Laughter]
JOSEPH:
One of the things I’ve been seeing with ShareJS, this is something I did a lot of work on at Lever, so it turns out that if you got like, obviously most web apps, like you got some JSON documents and you got some JSON stuff which is your model, and then you want to render that, so you have something like React, and then you render your JSON objects into HTML in the browser, and then as the user interacts or as the data gets modified, then in the browser, you’d say, “Hey React, my data was changed in this way, please re-render the DOM.”
With ShareJS, and this is really cool, with ShareJS, you kind of get all of the data side of that whole picture for free. So a web app can just synchronously make changes to the local data model, and as they make those changes, then everyone else can just see the changes live, and that all just works. And synchronously in the browser, you don’t even have to care that was anybody else in the world. You’re just making changes to this data model and keeping that data model visible in the local client, and then ShareJS takes care of keeping that data model in sync with all of the other clients that are also looking at the same data.
But, and this is the thing that gets really exciting when we get in the federation context, it means that with both of these kind of powers combined, we might end up with some sort of like awesome platform where people can actually build web apps… And I still have no idea how they’re going to be hosted or all sorts of different considerations. This is still very abstract, like, “Hey, we’re building this cool thing, only at work we make stage, but we can make something where people can build a web app that the web app isn’t like… like the data itself for the web app is hosted by the platform.
So you make a web app, and then everybody else who has like the Wave server can use your web app with their own data without any of the data going to your servers. All the data gets stored on their servers in their company. If messages don’t… if no one else outside the company is on the document, like if no one outside the company shares that conversation or that collaboration piece, then the data will never leave your company just like Git, which is really cool.
JAMISON:
So when you talk about these merging algorithms and operational transformation, it reminds me of distributed systems and CRDTs and like how… that seems like a concern that people have had a lot on the backend where you have to manage these concurrent updates and how do you resolve conflicts and things like that. Is there any overlap with distributed systems and this kind of research?
JOSEPH:
Well, absolutely. I mean distributed systems is a casual term for anything that’s distributed, so this is a distributed system because it’s distributed across multiple servers, multiple computers.
With CRDTs though, you’re absolutely right to bring that up. CRDTs are, in many ways, seen as like a newer version of OT.
JAMISON:
We should probably define that, sorry.
JOSEPH:
Yeah. Oh, sorry. CRDTs, I can’t remember what it stands for. I always think of it as I think commutative replicated data types.
DAVE:
That’s correct.
JAMISON:
That’s it! Ding, ding, ding.
DAVE:
That’s what it stands for.
JAMISON:
Winner.
DAVE:
There are people that disagree.
JOSEPH:
Yeah. I think the C cannot stand for commutative. I think it should stands for something else like collaborative and whatever.
DAVE:
Yeah.
JOSEPH:
CRDTs, the basic concept of it is the… Say, if you take something like Git, if you send me a commit, and I want to merge that commit, then I need to know that commit and all of the history, like all of the operations, all the other operations that you’ve done that I don’t know about. And I look at all the other operations that I’ve done since then, and I merge. I do a merge, and I look at all the operations together, and I say, “Hey, when you did that commit, I did this commit. Those two gets merged. They conflicted in this way, or they merged in this way.”
With CRDTs, you take the document, and the document kind of comes along with its entire history. So the document, in a sense, stores every single operation, every single change that’s ever happened to it so that you don’t have to worry about keeping all of these operations around. You don’t have to say, “Oh, well, but what if someone comes online, and they only got like a doc,” and they’ve known about the document that had operations since like two years ago.
With OT, you have to keep like every single operation for the last two years. With CRDTs, on your own, you have to keep them in a database somewhere. With CRDTs, all of those operations are like baked into the document itself, so you don’t have to keep them separately. So there’s lots of bookkeeping, but then your document ended and stopped growing without bounds. So that’s the tradeoff.
CRDTs are getting a lot of popularity lately. I still think there’s a lot of benefit in keeping it simple with operational transform. I mean like it’s something that we’re looking at, and we’re wondering if we should be using CRDTs instead. They’re basically in many ways the same technology. The implementation is slightly different. So if you see CRDT bantered around, you’ll often here OT bantered around as well because they’re basically like two technologies… well, two very similar algorithms that do very similar things.
AJ:
I’ve got some question for you.
JOSEPH:
Sure.
AJ:
What is Bitcoin?
JOSEPH:
Yeah.
AJ:
Can you… Well, maybe Bitcoin is just kind of often leftfield, and nobody knows what it is, but how does the way that you’re doing these shared updates relate to kind of like how Bitcoin does it?
JOSEPH:
Well, the whole point of the Bitcoin algorithm is that you end up getting through a distributed system. You host… like ShareJS relies on a, right now, it relies on a version of the increase, increases by one with every single change that happens.
AJ:
Okay.
JOSEPH:
And it’s just really, really simple. The way that ShareJS’s core algorithms work require that single number to be incrementing every time an operation gets applied in the server. Bitcoin, the algorithm of Bitcoin takes a distributed network, and through a ridiculous amount of work gives you a shared number that increases by one every time a new block is mined. So you could something like Bitcoin to actually run ShareJS in a distributed way, but no one would be able to merge their operations until they waited 10 minutes until the next block is mined.
So that’s what Bitcoin does, which is amazing because the whole point of Bitcoin is we can take algorithms like financial transactions which require like a shared block in a sense. And then Bitcoin can provide that shared block by doing all of this extra work, and then that means that now, we can suddenly have distributed currencies done and distributed everything else.
And this is why lots of distributed people get really excited by Bitcoin as a concept, like as an algorithm because it lets us do all these things. So the thing that people don’t talk about as much is that it does it really badly, but yeah.
AJ:
Well, I’d be interested to hear more about that, but there’s… I can’t remember the name of it. I’ve been searching just now, can’t find it. But there’s a new mail service that’s based on GPG encryption or maybe it’s not a service yet, but it’s like a project, and it’s kind of like Bitcoin in that everybody… like there’s a shared block chain type thing. And everybody sends their encrypted message to it, and then like everybody checks it all the time to see if there are any encrypted messages that they can decrypt which means that they’re for them. So like a network of people can be… like the laptop, in essence, could be “the server,” and every time a laptop comes online, it connects to other like friend laptops and friends store each other’s messages and that kind of thing. What do you think about that?
JOSEPH:
Well, that’s really cool. The big downside to it though, is that I mean how many emails are sent every day? It’s a lot, right? It’s in a terabyte range of bytes. I don’t have that much bandwidth. That system is really cool, but it scales limitlessly with the number of messages. And there’s probably ways that you can like shut it off and say, “No, this is just our little, private network for just our college or whatever,” but then you’re back to the same problem.
One of the nice things about federation is that we have exactly the opposite thing going on that actually your server only finds out about messages that are for someone in your company, and you only find out about messages that are for you. So the whole network scales as the internet scales, which is beautiful.
JAMISON:
So, we’ve talked a lot about a lot of different things. It seems like there’s kind of a broader trend right now of people realizing that client-side applications have more in common with distributed systems than they used to think, with like real-time stuff and especially with ShareJS where you’re collaborating with multiple people at the same time. Is that a broader trend do you think where we can’t just hide all that complexity on the server, but we have to deal with it in JavaScript on the client?
JOSEPH:
I sometimes imagine the world, like have you seen there’s like little videos of like stuff switching around inside the sun, and then the sun has like these little jets of fire that shoot out because there’s like some pressure that has to get released, and it gets shot out.
JAMISON:
Some science juice.
JOSEPH:
Yeah, some science juice. I feel like a lot of the things that go on in the open-source world are kind of like that, like… and there is something like this io.js and things like it. There’s these little communities of people that just push really hard for a short period of time, and then something kind of like shoots out the side.
And then sort of perpendicular to that is some other people who are working on something else that then shoots out a different way, and sometimes, they hit each other, and they swirl together, and then they end up interacting with each other positively or negatively or just making both of their projects different.
I feel like there’s this push at the moment like the re-decentralize push where people are saying, “Hey, I don’t want Facebook to be in the middle of me and the person I’m talking with because ultimately, there’s no way that we can, in the long run, provide security in a situation like that if the government can always just ask Facebook, I’m keeping on Facebook, asks Gmail for all of my emails. So if all of the communication that I have goes through Google, then the government can always just ask Google to see all my communication, and that’s not… I don’t want that, like I don’t want them reading all of my stuff.
JAMISON:
Yeah.
JOSEPH:
So, there’s this push now for like re-decentralizing everything. Hey, we’ve got the internet. That was cool. Then, we made a whole bunch of centralized applications, and that was pretty cool. You made a whole bunch of like new ways to communicate, and we have people in charge with those so they didn’t just like languish in the… in like, “Hey, we want to make this change, but we can’t because everyone’s gotten installed version and that would be with some compatibility.”
And now, there’s a new push for taking all of that stuff and saying, “Hey, let’s actually take a lot of the things that we’ve learned and then make a distributed system again,” like, “Let’s build something that works like the internet does, and let people actually communicate by sending message from my computer to my PC.” It’s crazy that when my friends in the next room, and I want to send them a link, I send them a link over G Chat, and I’m in Australia, right? So that message goes to the US and then all the way back from the US and gets to their room. That’s still faster than like reading out the message or any other way that I can send that message to them, but it’s also like dumb. It’s super dumb. I mean it doesn’t work offline. It doesn’t… it’s got all these problems. So, yeah. I mean there’s this new push for like at how these richer client web apps.
And there’s also a whole bunch of like crypter stuff that’s going on in the JavaScript world, and a lot of it is this sort of idea of like, “Oh, people actually have like rich applications.” Actually, I was talking to my uncle who’s been on the computing world for like, ah, god, since before I was born. I’m not that young. He’s just… I don’t know. He seems old to me, like he got a Logical Engineering degree, and he worked at Tandem who bought his old company, Borrowers, I think it was, and he was telling me about these first hard disks that was in Australia that they got, and it was this like giant, metal drum that took about a minute to spin up to speed, and like it’s this giant disk of copper that they had to install. Anyway, it cost them like over $1 million for 10 megabyte hard drive, which was huge.
But he was saying like there’s obviously like the whole computing history has been a history of people moving to think clients and do all the work of a server, and then rich clients that do all the work on the client, and then with dumb servers, and then back and forth, and back and forth. And in many ways, like as much as it’s glued to say so, the new push for like rich clients applications is the same thing all over again.
We have the web. Now, it’s dumb clients again. Okay. Now, we’ve got JavaScript and the web, and JavaScript can do all these amazing things. We have web sockets. We have ways for using web sockets for all your clients to connect the multiple servers, which now let’s us do a bunch of distributed stuff. We’ve got web RTC in the works. Okay, great. Now, we can go back to having rich clients again, which has kind of led kind of build these networks and systems.
AJ:
So, another question with the federated idea.
JOSEPH:
Yeah.
AJ:
I mean we pick on Facebook. We pick on Google. But really, if the NSA, like email is really federated, right?
JOSEPH:
Right.
AJ:
I mean it is stored on web servers, but the idea is that it’s a federated protocol that I can send from one provider to another provider, and it gets to me. But still, there’s only, in any realm of competition, there’s only like three main companies that generally rise to the top. There’s Microsoft, Apple, Google. There’s Gmail, Yahoo, Hotmail. There’s Blendtec, Vitamix, and Ninja. [Laughter] Right?
JOSEPH:
Right.
AJ:
And so, let’s say we have this awesome federated capability. How would we get that service out in a way that it’s not just federated between three or four companies, but it would be truly federated in this more private sense?
JOSEPH:
Right. I mean I guess I still use Gmail. For all that I complain about centralized services, I love Gmail. I think it’s a great product. I think the people that make Gmail do a great job. I really like it being, email being federate because it means that if Facebook goes evil and decides to sell all my data to companies, then there’s nothing I can do. But if email goes, if Google goes evil, I can remove all my data right away.
I’m sure there’s lots of people who really wish that everyone did actually run their own servers, but I’m actually not a lot worried about it. I think that if we have a federated system, a system that can federate and then all of a sudden like AUN where there’s only a few monopolistic providers or a few providers that actually, like Spatch, that will end up running our own servers that then people can use, and we will provide the system for lots and lots of users, I hope.
But we also base this on an algorithm and a system where you can actually host only yourself if you want to. I think you end up with something where people can then feel free to like choose your own adventure servers style where like, sure, I use GitHub, but then if GitHub turned into massive jerks, then we can just like have our own Git server, and that’s going to take some work to set up, but we can set that up ourselves. I personally really like that. I like giving people that sort of freedom even if they don’t actually exercise that freedom and just end up using whichever the big brands are.
What do you guys think? Do you think that it’s important that we actually do move away from things like Google and Facebook?
DAVE:
Oh, man. I feel like it’s important, but I don’t know if I have ever put my money where my mouth is and actually done anything about it.
JOE:
Yeah.
DAVE:
It feels nice to say I think it’s important though. That feels good.
AJ:
Well, I get more and more annoyed by them. But like with Google, you can reasonably replace some of their products with competing products or make something. If you wanted to build it yourself a Facebook, that’s where it means completely not federated. There’s no way you could build a competing Facebook because you wouldn’t be able to message anyone on Facebook.
JOSEPH:
Right.
AJ:
So I think Facebook is worse than Gmail is because if Gmail really gets sucky, well, I use Mailgun, and I do some stuff there, and I do a few things back and forth.
JOE:
I went to a conference up in… it was Open-Source Bridge in Portland, and they had kind of a keynote about getting out there, building distributed systems like you’re talking about, so that we could take back the power that we’re losing with DNSA and this lack of federation, and it was all well and good, but when it comes right down to it, very few people are building products anybody wants to use just like that.
And there’s just so much to be gained from having a company like Google behind who’s building your product. I mean nobody would have built Wave even though Wave is a big failure. It was a still good product. Nobody would have gotten together, six people wouldn’t have gotten together and built Wave. It was too much work, and I don’t see somebody… maybe somebody will replace Gmail, but it’s going to take a lot more people realizing that they can get together and build something really cool that’s really difficult or people building bigger projects.
DAVE:
Joe, are you saying that the financial incentives aren’t there for people to build federated things?
JOE:
No, no, no. I think it’s for the users. I think it’s the… the incentive isn’t there for the users. There’s just not enough products that are being built that are compelling compared to the non-federated products.
JOSEPH:
I guess straight out, it’s easier to build a non-federated product technically. And then, when you want to improve it, you don’t have to convince everybody who has an installed version of whatever the product is to upgrade, like you just upgrade your servers, and then everybody gets the latest and greatest version because it’s a website. You can see why it happens that way.
JOE:
Yeah, that’s true.
AJ:
I want to bring back up Bitcoin. If I understand correctly, they have something where they have a couple of blessed clients, and if more than 50% of the network upgrades to a new version of blessed clients, then everybody else just kind of this SOL, like you have to upgrade because 50% of people did, so the network demands it. Do you think that that kind of thing could take place on a federated system?
JOSEPH:
Well, it’s hard to… I mean, as I said, the whole point of Bitcoin is that you have a distributed system that creates and hosts a centralized system, and hilariously, when 50% of the clients do something, then we do X. It’s something you can never really do in a centralized way. In a distributed system, I don’t know who all the servers are. I don’t know where they are, so just by virtue of being a
distributed system, it’s hard to do something like that.
What you hope is that you hope that there’s no work of X, and when people start upgrading, then you hope that everybody else starts upgrading, and at some point, you can then break your backwards compatibility and say, “Well, you didn’t upgrade your client for a year. Sorry. You need to do that,” or, “You didn’t upgrade your server. You need to do that. That’s really important.” You can no longer talk to the rest of the network. Yeah, it’s an ongoing problem.
One of the interesting things with OT, which we’re talking about, this is very like pie in the sky prototype, like we have this kind of set of ideas that we’ve been talking about, is the idea of like having some versions of the applications. So I have some data on my… if you make an application, it sits on top of this platform, then your application says, “Hey, this is the schema that the data should look like, and this is the schema for what the data should look like if the application is version one,” schema version one or something. And then using OT, then we can say, “Oh, hey. The application now uses schema version two, so we’re going to use OT to be able to actually like apply an operation to all these documents to change them to the way the schema version.”
And as long as the application itself is backed with compatible with earlier versions of the data, then it should mean that we can kind of get this sort of like forwards compatibility across like effectively data migrations for free, which is really nice. Yeah, and maybe and then like the application could be stored there somewhere. I don’t know. We’re still trying to figure that out.
AJ:
Let’s say that it goes to version, like the protocol version two happens at document revision 57. Everyone has to converge on document version 57 before anybody moves on to 58 that’s in the new protocol version two, would that be correct?
JOSEPH:
Yeah. The data model, the data system will take care of that. That’s like separate from your application now. The data model will get you up to speed anyway. But the thing that you would still need is the client would still need to be able to understand the data at the previous like data version or whatever, so that it could still display that to you. Or like if you are hosting some content, and then I joined that, but I’ve got a newer version of the client, then it be kind of like a dick move if I went and just updated all the documents to the latest version, and then, now, you’re all like, “Ah, crap. I can’t read this.”
So it all have to be some… like some of my client will presumably need to be able to understand the previous version of the data, so he can say, “Well, I understand the new data version but also all these data of the old version.” So I’m just going to leave it as is and wait until the host, I don’t know, maybe someone is effectively the host for that particular document, and then maybe I’ll wait until that person upgrades their server version to the latest version of the software.” And then they can do the data migration. And then when that happens, then I’ll just be able to naturally understand the new data.
AJ:
I love this kind of algorithm idea of this, but I think Dave’s got a more practical question.
DAVE:
Hee hee hee hee. Ever the pragmatist. Can you tell us… let’s talk about ShareJS specifically for a few minutes. Can you tell us, let’s say I’ve got a requirement in my application I’m developing to have collaborative editing of something. How would I go about using ShareJS to do that?
JOSEPH:
As you point that currently ShareJS is 0.7, and I have a piece of work that I want to do, which is going to be bring it to under there, which is probably like regular open-source work as guilt of like, “Oh, god. I’m sorry. It’s still buggy, and there’s all these issues that I want to fix,” what you would do is you would install ShareJS, and you’d install ShareJS 0.7. You need a database, and currently, so rewrote all of ShareJS when I started Lever, which is virtually working for the last couple of years so that we could make ShareJS work in a cluster of servers.
I mean we at peak had about, when I left, we have 12 servers running. I think six instances each at our frontend server software, including ShareJS, and then any server could have a client connecting and any client could communicate and collaborate, collaboratively edit documents at any other clients on any, connected to any other servers we’re looking at. To make that work, you need Redis, but you can make ShareJS work in a single server context. And for that, you probably should install MongoDB which is the only database that ShareJS currently, officially supports, which I’ve said about. So I kind of hate Mongo. And you install ShareJS, and you then need to…
ShareJS has a server component and a client component. You need to install it in the server. The client then needs to be able to talk to the server, and probably, the best way to do that now is web sockets. ShareJS… Node people got mad at me because it was one big piece of software, and I was doing Node.js packages.
DAVE:
How uncool.
JOSEPH:
I know. I’m like the uncoolest. So I broke it up into a few different like packages. There’s like live DB that they handled all the database side. And then ShareJS itself, which mostly like is a wrap around the database bit and provides the network protocol and the client API so that your client application can actually use ShareJS.
So then, you need to put the client side JavaScript in your client, and you need to connect the client side to the server side via web sockets, best way to do that. Again, that’s like roll your own at the moment all this ShareJS web socket implantation that you can use. God, I should really make a good tutorial for all of the new stuff. [Laughter]
DAVE:
You got some more guilt.
JOSEPH:
I know.
DAVE:
You just saw the guilt grow. [Laughter]
JOSEPH:
I had all of these grand plans after I left Lever to like, “Yeah. I’ve a got few months before starting at Spatch. I should do all this change I’m wanting to make for a while, and clean up the document, and fix the website, and then bring out version wonder there.” And then I got back to Australia, and it’s really nice outside here, like I – [Laughter]
So if you look at my GitHub at the moment, it’s like all of this sporadic dots for the last two years as I’ve been maintaining all these different bits of open-source code. And in the last three months, there’s almost nothing. It’s just this big, wide-open space with some little dot when someone filed a poor request, but yeah.
I was chatting to the guys at Lever. ShareJS is not unmaintained, but the guys at Lever, since they’ve been in their business topped with software, are actually planning on like stepping up and doing a lot of the maintenance work of ShareJS going forward. So it won’t just be me, which is great because I’ve got all these crazy federated systems that I want to build. So don’t fear. But yeah.
Anyway, you have your client side that talks to the server side, and then you need to like have ShareJS. ShareJS needs to kind of own the documents, which is a little bit ownerist, but the reason why this is the case is because if a ShareJS client makes a change for the document, then all of the other ShareJS clients need to know about that change immediately. That’s the whole point. If you start willy-nilly making changes to your database independently of how your database being modeified in ShareJS, then the clients won’t know about that. And then when they go to make their own change, they’ll get very confused.
ShareJS kind of like takes on responsibility for at least some of the documents in your database, and then from the client, then you can execute commands like, “Hey, subscribe me to this document. Hey, run this query into the database, and tell me not just when any of the documents get changed but also like when this query gets changed, when these new documents get added to the results set of this query.”
And then you say, “Hey. On this document, at this path,” so like all the documents, I don’t know, maybe it’s got a shopping list, and you go like, “The document.shopping list at position zero, I want to insert a new item, and that item looks like this.” With ShareJS, you don’t just say like change the document wholesale through this whole thing. You can do that, but then ShareJS doesn’t do dispatch or anything. It just merges that blind in a sense and then call everyone else with changes. So you make your clients say like, “Hey, at this particular path, they make this change.”
And then all of the other clients who are looking at that same document are going to see that, so they can also subscribe to the changes on that document through the client side API and say, “Hey, subscribe me to this document, so subscribe me and tell me when any changes happen.” And then, that function will get a call saying like, “Hey, there was this operation.” It will tell you what the actual operation was, so if a new client, if you want to like do something really clever the way you like do piecewise re-rendering and insert a new item in your HTML list, you can do that, or you can just take a new and you look at the document snapshot, and by the time that function’s called in, presto. The document on that client side has already been updated, and everyone has that power. So all clients can make changes.
They can make those changes synchronously when they say insert this new item. By the time that function calls return, the document snapshot contains that new item, and you will like each client then in a sense sees a slightly different view of history because my change happened immediately, but everyone else’s change happened later. And the operations that you actually see if you took all the operations that you saw or on the ones that you made and applied them on that order, it would all work, and it would all make sense, which is cool.
So that’s if you want to do like JSON editing, and if you want to do text editing, then it’s a little bit easier. As an example, where you say like, “Just take this ShareJS document and bind it to this text area that’s on my page,” and then ShareJS will take care of doing all the text changes.
We also have rich text editing using Quilljs. Have you guys met Jason? He wrote Quill.
CHUCK:
M-hm.
JOE:
Nope.
CHUCK:
Yeah, we had him on the show.
JOE:
Oh, I wasn’t on that one, sorry.
JOSEPH:
He’s cool. Anyway, I worked with him on adding a rich text type to ShareJS. ShareJS, one of the big things that I changed from Wave, Wave had this idea of like this Wave data model, and then they wanted to have these cool, little gadgets, like a yes/no/maybe gadget. There was like three columns with a button at the top of each column, and you would say, someone would say like, “Hey. Does anyone want to go skydiving next week?” and then you would click on the yes column, and your face would appear in that column, and everyone could just do that, which is actually an amazing tool, super useful.
But to make that work for Wave, they had to do all these hacks to kind of add this JSON-like data on top of the Wave data model that was in the Wave data model is this kind of XML thing. With ShareJS, I said, alright, every document can have its own type and can have its own operational transform semantics. And so if you want, you can have text documents or you can have rich text documents, and rich text documents have all the semantics to support rich text. And Jason wrote that, and he wrote Quilljs which is the rich text editor.
So with ShareJS, you can have rich text documents, and for that, I think you end up putting Quill on your website and then you tell ShareJS about the Quill rich text type, and then you can just say, “Hey, bind this Quill document in my page to a ShareJS document,” and everyone who does that thing, they can all see each other’s changes. So, yeah. Easy, right? [Laughter]
CHUCK:
Ha ha ha.
JOSEPH:
I need a lot more example. A lot of the examples still haven’t been updated to ShareJS 0.7, which was the like, “Oh, god. Oh, god. I’m building a product with a startup. We just need something to work. Oh, god,” version that I started work on. I started last year when I started Lever, and I don’t know if you guys have worked at startups, but there’s not a lot of time to sit down, and breathe, and make your documentation for your stuff. At least that was my experience.
CHUCK:
If somebody wants to come in, figure all the stuff out, and contribute documentation, what’s the best way to do that?
JOSEPH:
You should submit a poor request to ShareJS. I love poor requests, or I love poor requests by people who are confident, and well-meaning, and very knowledge, like please look through the code and run the tests and stuff. Although ShareJS tests. God, it’s just big sea of guilt. [Laughter]. But yeah, please submit poor requests and help out. I’d really appreciate it.
Also as you pointed out, and this is a plate and plug, if you’re interested in working on this crazy OT-based distributed system thing that we got to build at Spatch in London, then you should absolutely email me because we’re looking for like smart, passionate, distributed systems engineers to help us build this crazy, new platform in the sky which I’m really excited about.
Yeah, we got funded by Techstars, and yeah. It’s all like we’re all in the like the planning phase in a sense of the moment where we’re like building things and chatting about architecture and stuff, and this is like absolutely the best time to get involved if you want to. And that would be really exciting to have more smart people online and onboard. So, yeah. That’s my plate and plug for our new company.
CHUCK:
Awesome.
JOSEPH:
I’ve been doing a lot of the talking. What do you guys think about all of this? I feel like I would love to hear your thoughts.
JOE:
You’re good at talking.
CHUCK:
Yeah. It’s really interesting, I mean just what it can do and how it works. Did we get into how it actually works on the backend?
JOSEPH:
No. I can explain some of that.
JOE:
Yeah. You mentioned Mongo and the Node server but not too much.
JOSEPH:
Yeah. So the way that it works in the backend, if you got a single server, then it’s reasonably simple, but it’s reasonably simple for me, and I’ve been thinking about this the last like five, six years. On the backend, the client sends a message on a web socket saying, “Hey, I’ve got this operation.” That goes back to the server. All of the operations are in Mongo at the moment.
You can make your own… other people have made PostgreSQL adapters and so on and different things, but as an API for what, like the actual messages for the database saying like, “Hey, if you want to implant your own database adapter, you have to implement a message for like ask the database if there’s an operation, get the set of range of operations from the database.
But the server can say like, the client says, “Hey, I have this operation.” It goes to the server. The server looks at the list, big list of everyone else who’s subscribed, and it applies the operation locally, so locally on the server, transforms it if it needs to, which is to say like does the Git merge and update, then resends it out to all the other clients who subscribed, and then they can all merge it locally with any changes that they’ve made in the meantime.
If you have multiple servers at the moment, the only way that happens is you need Redis instance, and then I’m using Redis Pub/Sub, which unfortunately adds a single point of failure. Although Redis is ridiculously reliable, but a single source of failure, point of failure in the sense that the servers broadcasts that that operation happened to Redis, and then Redis is responsible for rebroadcasting that out to all of the other servers that are interested in that document.
CHUCK:
And then the merge logic is all contained in the frontend code?
JOSEPH:
The merge logic is all in the OT type, which is both in the frontend code and the backend code. So all of the merge code is in the operational type, transform type itself.
My idea with this was that if anyone else wants to build an OT-based system, then you shouldn’t have to reimplement how JSON OT works because it’s a real pain in the ass. You shouldn’t have to reimplement text. You shouldn’t have to reimplement rich text.
So we’ve got a set of like standard operational transform types that are each in their own repair has its own GitHub organization which is GitHub.com/ottypes. They obey a standard API, and so rich text is just plugged in there, and then I know it’s like no changes in ShareJS to adverse check support. But that, inside those types is the, like those types then define what operations are. As far as ShareJS is concerned, operations are just these opaque JSON blobs that can JSON string apply and JSON parts.
The OT type itself defines a transform function that says, if you get two operations that happen at the same time against the same document, then what would operation A look like if it was chronologically after operation B, or if we swap them around, then what would B look like if it happened after A. And that’s the only hard function that you have to implement to get an operational transform type yourself. So actually, there’s no reason why OT can have conflicts. It just means that the transform function, I would have to say, if operation…
Like if two operations come in, and they both made changes to the same line of text, then operation A, which is the one that we care about right now, if operation A happened after operation B, which should actually look like insert a conflict marker at this position instead of just try and merge all the characters together anyway.
JAMISON:
Oh, sure.
CHUCK:
So what defines an operational transform is how it handles the merge.
JOSEPH:
Yeah. Yeah, in a sense, like, well, operations are, and like… people doing operational transform talk about your own COTs. They talk about this idea of intent preservation. The operation should preserve the intent of the user. So if the user’s intent was to insert a character at this particular position, then that’s great.
If their intent was, and this is how we think of it for like sales control systems, the intent was to change this line or delete this line of code and set this new line of code, which is suddenly different, right? If that’s the intent, then if two people try and make changes to the same line of code at once, then maybe the best way to preserve both of their intent is just to put in conflict markers. But that’s kind of the idea. So yes, the operation transform type is, yeah, defined by the operations and then defined by… well, yeah, exactly, defined by the transform itself.
The OT type as well has an apply function which is like what happens if you’re actually trying to apply this operation to the document, which are obviously important, so that the documents on both the client and server can be updated and then resaved back to the database or locally resend it back to the client application. But yeah.
DAVE:
Joseph, can you tell us some examples of applications you’ve known about that have been built with ShareJS?
JOSEPH:
I worked with Lever for the last two years, and Lever is an application for doing basically Gmail for hiring, so it’s Gmail for creators in a sense where you can watch candidates going through a job pipeline.
Like, what does it have anything to with like collaborative editing? Well, actually, if I’m going through a job candidate, and then one of my other co-workers messages them, I want to see that immediately. I love Nate Smith, who’s the CEO and is, as I said, taller than too tall Nate, which is very confusing for everybody, and he worked on DerbyJS. Have you guys… Do you know about the Derby thing?
DAVE:
Yeah, I do. The work tracker and for agile teams, no?
JOSEPH:
No.
JAMISON:
No. It’s kind of like a Meteorish thing, right?
JOSEPH:
Yes, it’s a Meteorish thing.
DAVE:
Oh, okay.
JOSEPH:
Derby is a web framework that’s build on top of ShareJS. It’s kind of like… I mean it’s got its own templating system inside. It’s like a full… In some ways, it’s similar to React except it’s got ShareJS bolted in in the backend.
React by the way works fantastically well with ShareJS, except that ShareJS could tell React a little bit more information. You could say like, “Hey, there’s this one item that I’ve been setting into this list.” React has to like regenerate the DOM then do a diff of the DOM, but with ShareJS, you can just actually make the change that you wanted.
Anyway, Nate worked on Derby, and when Nate and I put our minds together, we bolted Derby on top of ShareJS, so Derby supports complete, like full, real-time editing on everything. And Lever then built on top of Derby, then supports all of that. So the entire application is built on top of Derby, which itself is completely powered by ShareJS.
There was another small company. I’m not sure if they’re still around, called Rizzoma, and they were basically trying to remake Google Wave, the product, on top of ShareJS, which is really exciting. There was this team of six Russian guys who worked on that a few years ago.
There’s a bunch of like other random things. It’s used in a few different places for like collaborative job interview, editing, job interview, like pair programming sessions, and things like that. There hasn’t been, because I haven’t updated the website in awhile, and I haven’t written many examples and stuff, I haven’t been pushing it publicly a lot. But as I said, I had all these like grand ideas of like releasing wonderdom of ShareJS by fixing a lot of the… outset any bugs and refactoring some of the code and fixing up the tests, and then like yelling about it a lot.
There are still a couple of features that ShareJS is notably missing, including aggressive support. So you can actually, if you’re editing text documents, you can’t see where everyone else’s guesses are. Although that’s supported in the client. It’s just not supported by the distributed like Redis layer thing at the moment. So that’s the only part missing for that.
But yeah. I don’t know. There’s more examples of people using it. People are starting to use the rich text editing. Lever, I know, they want to use that for having like basically an email client built into the application itself.
It turns out people really want this. I think it’s crazy to give hiring company, well I think it’s crazy, to give a hiring company access to your corporate email. But recruiters love it that they can, if I’m a recruiter and I’m chatting to some candidate, then I go on holidays and someone else picks up the slack, then they can see the conversation that I’ve had with the candidate. So, yeah.
Lever has kind of ended up with this email client kind of bolted into the third client. Yeah, and they want to have rich text editing, which then obviously, then let’s people collaboratively edit an email to a candidate if they’re not sure how to word their email because they’re trying to hide or clean up something, and they want to make sure they get it right. Yeah, I don’t know. There’s lots of other random things.
This researcher in the Netherlands made a DOM, collaborative DOM editor using ShareJS, which is really cool.
JOE:
Whoa.
JOSEPH:
Yeah. I don’t know. Different things. There’s a bunch of researchers who’ll be making crazy things, which I really like. But, yeah.
What would you guys want to build and stuff? If we made this big like federated system, what would you want to make with it? Does anything spring to mind?
JOE:
I would build a to-do list because that’s what you make with JavaScript libraries. [Laughter]
DAVE:
Would it be a to-do MVC?
JOE:
No. It would be a to-do OT.
DAVE:
Federated to-do.
JOE:
Yeah. [Laughter]
CHUCK:
There you go. I go with federated wiki.
DAVE:
Yeah, federated wiki.
AJ:
Honestly, I would like to see an email version 2.0, kind of like somewhere between email and Wave, not two of OnGuard but like a stepping stone, something that’s like a graceful… Oh, nobody would call it that’s graceful degradation. The other one where –
JAMISON:
Progressive enhancement.
AJ:
Progressive enhancement, yes. Thank you. Like if I’m using Gmail that supports this V2, and you’re using Yahoo that just adopted V2, then we get these V2 features when we communicate together. That’s the thing that I really like to see.
DAVE:
Can we call it speed email? [Laughter]
AJ:
Yeah. Yeah, I think that would be a great name for it.
JOSEPH:
I’ve got to say this publicly because I think Spatch is a terrible name for our company. One of my Australian friends immediately was like, “Oh, Spatch like spatula,” and that’s all that’s stuck in my mind. [Laughter]
AJ:
That’s what I thought of.
JOSEPH:
Right. Okay, great. That’s another similar validation. We might end up changing the name, but it’s supposed to be short for dispatch. That was the original premise that the company was founded on, to make a better version of email because people spend like 20% of their time on email, and email’s kind of crappy.
AJ:
I just want something where… what I’d like to see for the future is notifications to become a different protocol, and then the new email to be for person-to-person communication. I want like 90% of what comes into my inbox to merge into the thing that shows up on my iPhone and on my Mac here on the right hand side with all the reminder crap, and this person just got on Skype. Oh, they just got off Skype. Like all that crap that feed, I want to have separated and just be able to focus on communication and some tool.
JOSEPH:
Yeah. The thing I’m really excited to build personally is… and I mean this is crazy. I have been talking about this for years, and it’s like this giant detour to ShareJS just so that I can have the infrastructure that I need for it.
I want to have a system where, when I’m editing code, instead of having like a big ID or something, like everything in one big application, I want to have my compiler at something… my editor should just be editing an OT document, and then my compiler just sits live watching the document be edited and then adds like highlighting and adds all the complete suggestions and all sorts of different things that the compiler knows about. And then my editor then could just see those edits and then mark that up inside the editor itself, so it can show me all the other complete suggestions and stuff.
And like different compilers if I’m editing different languages, different compilers will then do that for different languages. So I could just use BML, use one language, one editor that would have amazing built in language support for all these different languages.
And then have crazy development tools where my unit test runner could just run the unit tests and then annotate my source code with all of the errors and stuff that it found.
DAVE:
I think I would probably do something similar if I had a bunch of time and a bunch of shared JS expertise. The word I’ve thought about for this would be semantic version control where basically, you have something like Git but that’s aware of the language you’re working in.
One of the things that makes Git really weak is you say, if you move a function to a different file or to a different place in a file, Git sees that as subtract these lines and add these lines over here when, really, you’ve actually taken a unit of your programming language, a function, and relocated it and maybe tweaked it as you did so. And I think that a document aware, maybe even in the language itself, system could really help with something like that, make our lives a lot better.
JAMISON:
I know this a problem lots of people have tried to solve, but no one has solved it the right way or the way that I would like it solved. I mean it seems like –
AJ:
Which is the right way.
JAMISON:
Yeah, that’s the implicit thing. Just pair programming stuff, like it’s still… you see new ideas in there all the time because no one has made the solution that everyone likes yet, and it seems like you could do some really cool stuff with that, also really related to David and Joseph’s ideas too.
AJ:
Jamison, we have it. It’s called a desk. [Laughter]
JAMISON:
Remote pair programming. There we go.
CHUCK:
I have to admit that most of my backend work, I’m still doing in Ruby. So it’d be really interesting to see if there was some way to implement this in a way that I could build in to the apps that I build for my clients in Rails or Sinatra or something, or put this alongside that so that maybe updates get federated through to another system and still use that backend.
JOSEPH:
I would love that.
CHUCK:
I’m curious as to what your APIs and protocols are and if there’s a way for somebody to just set up ShareJS just the backend and then use it as needed where needed.
JOSEPH:
ShareJS, that’d be great. Actually, the very first time I’ve added to ShareJS, I had a native iOS… well, sorry. It wasn’t iOS. It was native MacOS client application, and I was doing collaborative editing between my native application and a web document, which is really cool because of course there’s no, there’s nothing that JavaScript’s so cynic about it except codes in JavaScript.
The thing that you need to do is you need to take the OT types, and then you need to bolt them to Derby, which is much easier than you would think. I ported one of them from JavaScript to C, like native C, a couple of years ago, and the C version was only like eight lines of code longer than the JavaScript version and around like 10 times faster, which is kind of nifty to see.
CHUCK:
Well, if you’ve written it in C, then you can extend Ruby with the C library.
JOSEPH:
Right. Yeah. I haven’t ported all of them across yet. I’m currently working on a replacement for the current JSON OT type. There’s a few features that we want to add support for. I want to add support for like arbitrarily moving items from like re-parenting items inside the JSON tree which a lot of people want, and that’s useful for a lot of different work cases. But, yeah.
So the protocols, there’s a protocol between the client and server, which is basically the web socket protocol of what messages are sending back and forth. If you had a pure Ruby server, then if you implemented that, and those server that talk that protocol, then the ShareJS client will just work.
Another option is if you want to go to ShareJS, if you wanted to have a Node.js server that manage the web socket connection coming in, there’s a protocol where they’re distributing across multiple ShareJS servers where basically it’s a pub/sub system where it’ll publish messages to Redis, and then all the servers listen to that. So if you wrote a Ruby program that could listen to these messages and send messages on its own, then your Ruby server could also talk to the same language and could communicate with that.
The trouble with a lot of this stuff is that, it goes back to a what we’ve talked about before, that if you got like a client… if one client is OT DOM, and it says, “Hey, just put this dry thing into this document,” like, “Replace this document wholesale with this other data,” if there’s any OT clients, they can’t merge that. They will just see it as the entire document got replaced, and if they got any like concurrent edits where they’re making some changes, and they want those changes to be merged intelligently, then you’d now just deleted all of their changes.
So you really want like some set of your documents or maybe all your documents to be in an OT system rather than like mixing and matching. You can read back out of the OT system, but if you’re doing any rights, you want those rights to be done with the operation, not with like just DOM, like kick everything off and then replace the entire document, which is a bit of a pain. So that’s the only thing. That’s what’s missing.
But I would absolutely love to see a Ruby port of all of the OT types or if we made a C version of all of them, which we probably will at some point, then expose that through Ruby. And then we could absolutely have a Ruby server.
The Y protocol, by the way, is well-documented, and it’s up-to-date and everything, and there’s a wiki page that I can give you a link to if you want, which is just documentation on what messages go back and forth and how that all works. So it’s just a case of implementing a server that does that.
CHUCK:
Awesome. If people want to know more about ShareJS or get involved in the project, what are the best ways to do that?
JOSEPH:
Well, you should either submit poor requests or email me. I actually really love, I find it really motivating speaking of emotions around projects and guilt. I find it really motivating when people email me, like, “Hey, your project’s cool,” like, “Do you want to chat?” just chatting to people over Skype or something, and yeah. So I actually love that.
If you want to get involved or at least if you find ShareJS really exciting, you should email me and say like, “Hey, do you want to just chat?” and then we can talk over Skype, and we can find the time that works for both of us. And you can tell me why you’re excited about it, and I can be like, “Yay, that’s cool. Now I’m more motivated to work on it.” And I can give you lots of advice on what is or isn’t hard with ShareJS right now. So, that’s what I would recommend.
But also mail me and check out the projects, submit poor requests, I love poor requests even though I’m sometimes bad at getting back to them, but I’ve been trying to get better lately. So, yeah.
CHUCK:
Alright. Let’s go to new picks. Jamison, do you have some picks for us?
JAMISON:
I do. I have three picks. First one is a blog post called Software Engineer Should Write, and it kind of makes the case for why it’s important for people in Engineering to write, and not just technical writing, any kind of writing. Its ground has been tried pretty well, but I just really like… this is a very well-written post about why you should write well. So it’s a good example of what it’s talking about.
The second pick is a Twine game. It’s a text game called My Father’s Long Legs, and it’s kind of this creepy, unsettling… there’s no jump scares or like gruesome things or anything. It’s just like a really weird, unsettling feeling that kind of grows the more you play it. So I thought that was cool. And they do some cool, just some cool effects that make the game more interesting.
And the last pick is an old paper called Can Programming Be Liberated from the von Neumann Style? It’s by this guy named John Backus, who is the guy who Backus Normal Form is named after. I think he was part of the creation of Fortran as well. He’s one of the old luminaries in computer science, and he talks about how computer architecture is very tied up with imperative programming and statements that modify locations and memory, and then you go onto the next statement. And it’s kind of presenting functional programming as an alternative to that but also an alternative at like the hardware level even. So…
And then it gets into some math that goes over my head, but just the pros arguments were pretty interesting to read. Those are my picks.
CHUCK:
Alright. Joe, what are your picks?
JOE:
Alright. My first pick is Brandon Sanderson’s new book. It just came out. It was yesterday, and I pre-purchased it. It’s the second book in his Reckoners series. The first one was Steelheart. Awesome books, just absolutely beautiful. Steelheart was probably my favorite book of 2014. So I was so excited when the new one came out, and I love it already. I only read a little bit last night for like five or 10 minutes, and I’m already just completely absorbed into it. I can’t wait for this podcast to get over so I can go read it some more. [Laughter]
JOSEPH:
Is it science fiction?
JOE:
Yeah. It’s kind of a dystopian, slightly dystopian future. Brandon Sanderson’s just completely amazing. He’s famous for inventing new magic systems, and it’s an unbelievable book, and I can’t wait for the second book. I’m sure it’ll be just as good as the first book. The first book completely blew my mind at the end of it.
DAVE:
So you picked it, and you haven’t read it.
JOE:
No, I haven’t. But all of it yet.
DAVE:
Shame. Shame. So you’re only picking the first 10 minutes.
JOE:
Yes. Picking the first 10 minutes.
DAVE:
Good. The next pick, I know what you’re going to be picking. [Laughter]
JOE:
I have read the first book, so I’m picking the series. How about that?
DAVE:
Ha ha ha, I got it.
CHUCK:
There you go. There’s a work around for you.
JOE:
I’m willing to bet your reputation, Dave, that it’s going to be awesome. [Laughter]
CHUCK:
Nice.
DAVE:
I’ll take that bet.
JOE:
So that’s my first pick, and my second pick, I just found out, I was checking out the Forbes 30 Under 30, and the guy under education was a guy who invented… I don’t know how to say it, Kano or Keno, Kano.me, and it’s this like nephew challenged him to make a computer that was as fun as Legos to build. And so he did, and he used raspberry pie underneath it, and you get this kit that you can plug in things just like Legos and build a computer, and then you can program Pong on it, or Snake, or Mine… it gets you into doing MineCraft mods, and so –
DAVE:
This is so cool.
JOE:
Yeah. It’s way, way awesome for like getting your kids into programming. I’ve been a big kick about that lately.
DAVE:
Yeah, this is really cool.
JOE:
Yeah. So I’m very excited about this. I’m really, really jounced. I haven’t ordered mine yet, so I’m picking it again before I’ve consumed it. Don’t hate me because I’m beautiful, Dave. [Laughter]
DAVE:
There goes my reputation. [Laughter]
JOE:
But, I mean there are companies like done awesome. They’ve been fundraising, so it’s got to be pretty cool, and I’m really excited. I think it’s a cool product. I want to support it, so I want to get the word out.
CHUCK:
Very nice.
JOE:
I have a second final pick.
CHUCK:
Alright, Dave, what are your picks?
DAVE:
Okay. Do you guys have guilty pleasures? [Laughter]
CHUCK:
A few.
DAVE:
I have one, and I’ve been sitting on it, and I’m finally going to pick it. Nobody knows this about me, but well, this is a two-part pick. The first part is not the guilty part. First part is a game, online game, it’s a multiplayer online battle arena called League of Legends. I know Jamison knows about it, even though I think he probably doesn’t like it. [Laughter] But anyway, it’s called League of Legends. It’s really fun, pretty addictive, habit-forming game, and it’s free-to-play.
But, here comes the guilty pleasure part, there is an online gamer of this game who records his games, and he talks about them while he does it, and he is hilarious, and his games are really entertaining to watch. I actually enjoy watching him play this game more than I enjoy playing it myself. His name is Anklespankin. [Laughter]
JOE:
Wait. That’s – [Laughter]
DAVE:
Yes.
JOE:
Please say that again. Please repeat that. I must have had something crazy in my ear.
DAVE:
I said it was a guilty pleasure, Joe. [Laughter] His name is Anklespankin. Anyway, if you just Google Anklespankin YouTube, you can watch any of his games. He does one every day, and he’s really entertaining to watch. And now you all know about my guilt. My shame.
JOE:
Your secret shame.
DAVE:
This is my secret.
JOE:
I’m actually excited about this because I love watching StarCraft 2 way more than I like playing it.
DAVE:
I watch a lot of Twitch TV, so I don’t think it’s shameful.
JOE:
I got a bunch of friends that are way into like the MOBIS, like League of Legends, and I just don’t get it. I play them… and even in the beta for this Blizzard one that just came out, and I play, and I’m just like, “Eh…” but I’m excited to watch somebody who’s actually really good and entertaining.
DAVE:
There you go. Well, this just proves that, on the internet, you can take any guilt and turn it into pride by having a group of people who tell you it’s okay.
JOE:
Well, and I didn’t understand the guilty part until you got to his name, then I understood. [Laughter]
CHUCK:
Yes. Speaking of his name, I really need to get a soundboard together for the show. And then I can have, Dave –
JOSEPH:
Speaking of a soundboard, you should really get a microphone. [Laughter]
CHUCK:
Do I really sound that bad to you guys? [Laughter]
DAVE:
Yeah. Super staticky today.
AJ:
It’s absolutely terrible.
JAMISON:
It’s usually much, much better than this.
CHUCK:
Comcast. I’m blaming Comcast, and the people listening to the show, they’re hearing me coming through the mic, through my microphone, into the… through the mixer, so it sounds terrific for everybody except for the people that I’m on this call with [laughter] so…
JOE:
Right.
CHUCK:
Alright. So are those your only picks, Dave?
DAVE:
Yes, and I just want to say once more, Anklespankin. [Laughter]
CHUCK:
Alright. AJ, what are your picks?
AJ:
Alright. So, to start us off, Captain Toad Treasure Tracker.
CHUCK:
What?
JOE:
Could you repeat that please?
AJ:
Captain Toad –
DAVE:
I think you said Anklespankin. [Laughter]
JOE:
Yeah.
AJ:
Almost exactly the same except for all of the syllables. [Laughter] There are word and letters in there, so you got that right. Captain Toad Treasure Tracker. It is the cutest game that Nintendo has ever made. That’s all I got to say, and if you know that you’re going to like it, you’re going to like it. And if you don’t know that you’re going to like it, and you don’t like the other games I like, you won’t like it, but if you… My favorite part of Mario 3D World was the treasure tracker mini games. I didn’t like the overall game quite that much, and the full game is a nice one.
After that, Endless Fantasy by Anamanaguchi. Don’t get distracted by the music video. Just listen to it. Then watch the music video afterwards.
Vagrant. It turns out that if you have the problem I do where every time you upgrade OSX and try to reinstall RVM, or Ruby, and Brew, and everything that… anything that you’re streaming on your system is just completely broken because it always is particularly static site blog generators, you can use Vagrant, and you can just keep it, and you can upgrade OSX from 10.5 to 10.6, all the way to 10.10, and you could just keep that blessed 1.8.6.
And so Vagrant’s like a… it’s a virtual box thingamabob. You specify config file. It downloads a virtual box image, and it runs virtual box, and so that’s why I’m talking about the Ruby thing. You can just have this virtual box. It’s like taken up not very much RAM on your system, and it’s just stable, and you can run your blog static site generator tool thing in that instead of on your OSX machine. So that’s that. [Laughter]
CHUCK:
I love how you blamed Ruby because people don’t have problem with their versions of Node.
AJ:
No, they do not. They do not. When was the last time you heard somebody complain about their version of Node? Node hasn’t even changed versions for two years. They got to create a fork because .12 is never happening. [Laughter]
CHUCK:
Okay. Biting my tongue. [Laughter] I’ve got a pick that I’m going to share. It’s called DeskTime, and I’ve decided that I really need to do better at keeping track of what I’m spending my time on.
And so what it does is it keeps track of what app you’re in. It also has a Chrome plugin and a Safari plugin, so if you’re on the Mac anyway, you can hook that into your browser, and then it’ll also tell you which websites you were on, and then you can say, “This website’s productive. This website’s not productive.” So if it’s YouTube or something, and you’re not watching a talk, then you can say, “Okay.” So it’s not productive, or you can just say, “Well, I spent an hour watching a talk on YouTube, so that hour is productive.” But the idea is, is then I can start saying, “Okay. I need to spend less time doing this or that,” or whatever.
This is especially relevant to me since I don’t have a boss that checks in on how I spend my time. And I track my time for my clients separately, but this is just for me, for my own productivity. So that’s my pick. Joseph, what are your picks?
JOSEPH:
So my picks, first of all, given that it’s the start of 2015, there’s a blog post by someone from the Less Wrong community who’s now… who is one of the directors of CFAR which is the Center for Applied Rationality.
I don’t know what anyone knows about Less Wrong, but it’s a really great blog press talking about how humans are not automatically strategic with the idea being that you will basically not accomplish your goals. Like if you ask someone what they actually want to accomplish in their life and then ask them what they’re actually doing with their life, and you ask them what specific actions will bring you closer to that goal, and then ask them what actions they’re actually doing, then often, those action aren’t actually the things that will get them closer to their goal.
I know this is true for me, and this is something that I’ve been thinking about a bit, and it’s this kind of like deep insight that like, “Oh, god. We’re like dumb monkeys.” We’re like just the… just smart enough to be able to make civilization but only just. As soon as we got smart enough, we made civilization, and that’s all of the evolving that we’ve done.
So, yeah. It talks about this idea that we’re not automatically strategic, and we have to actually go out of our way to act strategically and be strategic in our lives. And speaking of 20… it’s the start of 2015. This is something that… this is my New Year’s resolution, just trying to get on this kind of stuff.
My second pick is another blog post called Ai Weiwei is Living in Our Future. I bring it up because we’re talking about some of the re-decentralized stuff and government surveillance. There’s this amazing artist called Ai Weiwei, who’s Chinese. I’m probably mispronouncing his name. He does this amazing art that he’s been living under like 24/7 government surveillance with people standing outside his house and all sorts of crazy things that have passed in awhile.
And the article is a little bit depressing but really well-written and really interesting, talking about how, the way that he’s living today is the way that we are going to be living soon if we’re not already with like a lot of different technologies and things like governments reading all of our emails and SMSs and stuff. It’s an interesting thought. I always bounce back and forth between being like dystopian and pessimistic about the future and being really optimistic about all the stuff we’re building.
And my last pick is the response that I use. It’s a little tool called tup. tup is a build system that is my response every time someone tells me to use Grunt or Gulp or whatever the new fad build system in the Node.js world is. tup is amazing. It is an incredible build system.
What it actually does is it fires up a… It’s written in C. It’s quite old. It fires up a fuse files system which it mounts over the top of your actual real file system in your build directory, and then when you run your build program, it actually just watches your compiler and checks every single file that the compiler looks at and remembers that.
So, next time you make any modifications on these files, you just rerun tup, and tup knows exactly what files are dependents of any other building steps that it runs because it watched the compiler do it, which is really amazing. It’s like the ultimate build system thing where you don’t have to do any of the work. tup does all the work in figuring out… or almost all of the work in figuring out which files need to be read and updated and recompiled, when and if you make any changes.
CHUCK:
So how do you build the first time?
JOSEPH:
Build the first time, you just say, “Hey, I’m going to build this thing. Okay, I’m running it now. I’m doing it,” and then it watches, and it records it in the little internal database file where all the files get modified. So, that’s it, which is kind of amazing. Like, so, presumably the first time –
CHUCK:
That’s really something.
JOSEPH:
Yeah, you’re building the whole thing anyway, so it can just watch that version or watch that run. So, yeah. It’s kind of like the ultimate in build systems, and it’s based on a whole bunch of like build system theory and build system papers which they link on their website, which I find it really satisfying in some like deep, technical way. So, yeah.
DAVE:
And it kind of compares itself to something called Mordor. That’s cool.
CHUCK:
So the Grunt, Gulp war –
DAVE:
I don’t know what that is, but –
CHUCK:
Yeah, you’re walking into the middle of the battlefield with a target painted on your back.
JOSEPH:
Oh, yeah, but I’m telling you exactly what I’ll do with some clients.
CHUCK:
It sounds cool.
JOSEPH:
There was this big poor request someone made to convert ShareJS to Grunt… yeah, well, that was still the build system de jure, and I’ve had for a long time the default, the accept policy on poor requests, so then I can’t think of a good reason why not, that I’d like the project to move in the direction and incur it, maintain this and so on. But oh my god, that was such a mess, and it took me a long time to curse Grunt crap out, like double the number of dependencies I had and everything else.
I think that gulp is a lot better, but tup is better still. So there you go. That’s my opinion on the whole build system war.
CHUCK:
Alright. Well, thanks for coming, Joseph. It was a lot of fun, and I think this is a really interesting system with really interesting ways of moving the web forward. So hopefully, we’ll see some things come out of it but pay off for the world.
JOSEPH:
Yeah, I’m really excited to be working on it. And as I said, if anyone else is really excited to work on this stuff, then we’re hiring. So we would love to get some more impassioned, smart engineers to come work with us to build some really cool stuff.
There are still some things that I don’t want to talk about yet. We’re still like in stealth mode in some ways for some stuff, but the platform is all going to be open-sourced as we build it or it’s going to be open-sourced at some point. And I’m really excited to… yeah, I’m really looking forward to reach that point where you can start putting it out in the world and getting people building cool technology on top of the systems that we’re working on. So, yeah.
Thanks for having me on. It’s been a blast.
CHUCK:
Alright. Well, we’ll wrap up the show. We’ll catch you all next week.
[Have you noticed that a lot of developers always land the job they interview for? Are you worried that someone else just landed your dream job? John Sonmez can show you how to do this with the course ‘How to Market Yourself as a Software Developer’. Go to DevCareerBoost.com and sign up using the code JJABBER to get
$100 off.]
[This episode is sponsored by MadGlory. You’ve been building software for a long time and sometimes it’s get a little overwhelming. Work piles up, hiring sucks, and it’s hard to get projects out the door. Check out MadGlory. They’re a small shop with experience shipping big products. They’re smart, dedicated, will augment your team and work as hard as you do. Find them online at MadGlory.com or at Twitter @MadGlory.] [Hosting and bandwidth provided by the Blue Box Group. Check them out at Bluebox.net.]
[Bandwidth for this segment is provided by CacheFly, the world’s fastest CDN. Deliver your content fast with CacheFly. Visit CacheFly.com to learn more.]
[Do you wish you could be part of the discussion on JavaScript Jabber? Do you have a burning question for one of our guests? Now you can join the action at our membership forum. You can sign up at
JavaScriptJabber.com/jabber and there you can join discussions with the regular panelists and our guests.]