Check out JS Remote Conf!

 

02:29 - Nolan Lawson Introduction

04:19 - PouchDB (vs CouchDB)

05:25 - CouchDB Emulation

06:45 - How CouchDB Works

08:26 - Syncing and Replication

10:43 - PouchDB vs Other Paradigms for Building Client-side Apps and Managing Data

13:58 - AP Databases / CP Databases / CA Databases

17:25 - Ignoring Merge Conflicts

20:08 - Mutability vs Immutability

21:29 - Offline First

24:59 - Client-to-client Syncing

25:54 - IndexDB and Local Storage

28:50 - Authentication and Authorization

30:30 - Mobile Support

31:42 - Resource Usage When Syncing

33:06 - Use Cases

34:53 - Partitioning Data

36:22 - Getting Started

37:09 - Contribution

38:53 - Upcoming Features
On YouTube
185 JSJ PouchDB with Nolan Lawson

Picks

source-map-explorer (Jamison)
Facebook: Managing Bias Videos (Jamison)
Computers Are Fast (Jamison)
86 Mac Plus Vs. 07 AMD DualCore. You Won't Believe Who Wins (Jamison)
Authy App (AJ)
Chip Network Channel on YouTube (AJ)
Oregon (AJ)
Browser Authenticator (AJ)
Node Authenticator (AJ)
AngularConnect (Aimee)
Kevin Old (@kevinold) (Aimee)
Jordan Kasper (@jakerella) (Aimee)
Highrise (Chuck)
Streak (Chuck)
The Accursed Kings Series by Maurice Druon (Nolan)
The Smash Brothers (Nolan)
Super Smash Bros. Melee (Nolan)

Special Guest: Nolan Lawson.

 

Sorry, I was late.

Darn it. [Chuckles]

Three minutes late. That’s three demerits.

Hey everybody and welcome to episode 185 of the JavaScript Jabber Show. This week on our panel we have Jamison Dance.

Hello, friends.

Aimee Knight.

Hello.

AJ O'Neal .

Yo, yo, yo, I'm coming at you live from the beautiful forests of Troutdale, Oregon.

I don't even know where that is. I'm Charles Max Wood from DevChat.TV. Before we get started I just want to point you to JS Remote Conf, JSRemoteConf.com. We have a special guest this week, and that is Nolan Lawson.

Hi, everybody. How's it going?

Going well. I got somebody on the show whose name I can pronounce.

[Chuckles] Yeah. That might be a nice change of pace.

So, you want to introduce yourself?

Sure. So, I'm Nolan Lawson. And I'm an Android developer and web developer at Squarespace.

And I also co-maintain a JavaScript library called PouchDB.

So, how does it feel to know that your company is singlehandedly supporting the podcast industry? [Laughter]

I swear that you sponsor, your company sponsors every single podcast I listen to.

It feels really good. Although oddly, I don't think we've ever sponsored this one…

Nope. Slackers.

If I recall correctly.

Every single good podcast, you sponsor.

Oh, that's not fair. No, you guys, you have an excellent podcast. I'm actually a long-time listener. So, it's really exciting for me to be on the show. A long-time listener, first-time caller kind of situation.

I want to add to your list of things that you also write awesome blog posts.

Oh, thank you so much. Yeah, I do try to write interesting blog posts on the PouchDB blog. I know that one of them was really popular. I think it was the one on promises.

Yes.

We have a problem with promises.

Yes.

Yeah, that one was, that was a really passionate blog post that I just wrote because I think we were finding that a lot of people were learning promises through PouchDB. Because sometimes they hadn't really experienced promises before until they started with our library. That was actually my experience. Before I started working on PouchDB, I had barely used promises. And there are lots of ways to trip yourself up. So yeah, it was a fun blog post to write.

That was a great post. I always point people to that now that are not familiar with them. And for me, it just really solidified everything.

So, I know somebody is screaming at their phone and they're saying, “It's not PouchDB. It's CouchDB!” Do you want to explain what PouchDB is?

Yeah, sure. So, first off, the name yes, is a pun on CouchDB. So, the joke is it's CouchDB in your browser. But for the purposes of this podcast, I will be careful to enunciate clearly, so it's clear what I'm talking about. So, in short, PouchDB is a NoSQL JavaScript database. It runs in the browser or in Node.js. And it can also act as a straight up client to CouchDB, with the goal being that you can then sync PouchDB to a CouchDB running on the server.

Oh, interesting.

Yeah.

So, it is related in some way to Couch. It's not just the name?

Oh, it is very deeply tied to CouchDB. We interact pretty frequently with the CouchDB community, the Apache CouchDB community. Both projects are open source, by the way, and Apache licensed. So, we are not an official Apache project, but we are definitely a sister project in many ways. And we interact with them frequently.

So, what made you want to emulate the CouchDB stuff in PouchDB?

Well, that's a really good question. I think, so first off it wasn't me who wrote the library. I think it's useful to maybe give a little history because the library is now over five years old. This year was the five-year anniversary.

That's ancient in JavaScript [land].

[Chuckles] I know, right? So, when it started out it was this interesting experiment, actually started by Mikeal Rogers, of all people, of Request and io.js fame. He started it I believe back when he was working for Couch 1 with the idea being that he had just heard about IndexedDB which was this new database API for the web that was improving upon the local storage API. And I think at the time it was only available behind a feature flag in Firefox. But he got really fired up about the idea of this API. And so he thought, “Why don't I try to build CouchDB on top of this?” So, he built this thing. He called it IDBCouch, I think. Then it kind of sat for a while. And then Dale Harvey picked it up, Dale Harvey who works in Mozilla doing Firefox OS. He picked it up and renamed it PouchDB. And since then, it's just been a constant slew of new features and bug fixes in order to basically emulate CouchDB in the browser, in any modern browser.

So, I've never used Couch. I have some friends that have and I've heard a little bit about it. But do you think it'd be helpful to give an overview of how it works so we know. When you say it's like CouchDB, if people don't know what CouchDB is, that doesn't help them very much.

Yeah, that's a good point. So yeah, it's definitely useful for me to mention that PouchDB is a straight up reimplementation of CouchDB in JavaScript over IndexedDB or Web SQL Database as the DOM support or in browsers that don't support IndexedDB. And then in Node, it uses LevelDB. It also has adapters for in-memory and local storage, and for the SQLite plugin in Cordova. But anyway, the point is that it is a straight up reimplementation, method for method, option for option. So, it is just CouchDB running in your browser. And the reason that we did that with CouchDB is that CouchDB has some nice properties when it comes to syncing.

So, CouchDB has two major claims to fame that I think motivated everyone to make this crazy [chuckles] browser Couch thing. And the first one is that it's a database that speaks HTTP. So, that is the primary way of interacting with CouchDB. It is just HTTP and REST. It wholeheartedly embraces REST. If you want to PUT a document, you do an HTTP PUT. If you want to DELETE, that's an HTTP DELETE. It bills itself as a database for the web. And so, it very much achieves that.

And the second thing that it does really well is it syncs. That's the number one thing that it's really good at and it excels at over a lot of other databases. It's the reason that npm is built on top of CouchDB. When you download modules or when you publish modules, that's all CouchDB under the hood.

So, what do you mean when you say it syncs. You mean like syncing between different replicas, like primary and secondary, or what?

Yeah, so that's a good question. I think a lot of databases are modeled after this primary/secondary, or leader/follower architecture. CouchDB is unique in that it has a multi-master replication strategy, which basically means that… so, like with Mongo for instance you have a primary and a secondary right? Which means that you can write to the primary and then read from the secondaries. And that's the way that its replication model works. In CouchDB it's quite different because it's an AP database rather than a CP database. So, it [prices] availability over anything else. So, the idea is that you have all these different nodes. And every node is its own master, hence the name multi-master.

So, you can spin up three nodes and you can write to whatever node you want and just sync them all together. And it's not going to be consistent necessarily immediately. But it is eventually consistent. If you set up the three nodes to sync to each other, then eventually they'll all agree on something. But in the meantime, they might not agree. But the point is that when you query one of them, you immediately get results back.

And the innovation of PouchDB and the thing that makes it so compelling in my opinion is that it takes that idea of CouchDB, of multi-master, you can write to any node and read from any node, and then extends it to the client. It says, “Why don't we just make the client its own node as well?” And then you can write to the client and just sync it to the server in the background.

Related to that is, does it sync all of the data out of the server-side database into the client? And couldn't you cause issues that way?

You can actually do a filtered replication…

Okay.

Which means that you don't, yeah you don't replicate everything. But sometimes, in some use cases, it actually is really valuable to just replicate the entire database. And oh, and in CouchDB terms, a database is actually a very, it can be a very small unit. You can have a single CouchDB server. And you can have tens of thousands of databases on it. Databases are cheap in CouchDB. So, for lots of scenarios, people do just store data into a server-side “database” and then just completely sync all of that to and from the client.

So, that changes the model of building a client-side web application then. If you can talk to your database directly from the client instead of using REST or something to go through an API server, you just update your database and then it magically syncs. Do you see any parallels between things like Relay or Falcor or GraphQL? Like, these new paradigms that try and get around some of the pain points in building client-side applications and managing data.

So, I think GraphQL and Relay are really interesting. I think they have a very different approach to PouchDB's. Theirs is more based on the idea I believe of having lots and lots of data on the server and then just lightly caching some of that on the client. The strategy that PouchDB employs is actually, it's more similar to projects like Hoodie, if you're familiar with that, which actually Hoodie is in fact merging with PouchDB. And so, they're slowly re-writing their engine to use PouchDB which is, it's been really fun to work with them. They're a really cool, cool team.

I think my question wasn't clear enough. It's more… it's not, “Is it like Relay?” It's, “Is it trying to solve the same problem that Relay is trying to solve?” That you don't want to have to change an API server every time you need to change the data you need on your client. Whereas with, it seems like with Pouch, if you need different you just have it there already in the database.

I think one thing that might clarify this a little bit is that CouchDB is from what I understand, a document database. It stores in JSON, or at least communicates in JSON. So, that being said, with Falcor and GraphQL you can have any arrangement of your data on the backend, where this is much simpler. And it's not necessarily… you can think about it more as connecting to something like MongoDB where you're going to get a document back instead of getting formatted data back from GraphQL or Falcor.

Yeah, I know. Sorry, I think I understand your question a little bit better now, if I can try to provide [chuckles] a slightly better answer.

Sure, sure.

I think you're right that all of these projects are in the same family, in the same vein of saying, “Hey, we spend enough time writing custom REST servers that are different every time, for every single app that we write. And we always get it wrong and we always mess it up.” And it's always hard to do Ajax, right? How do you handle failed requests? How do you handle not re-syncing the entire database every time when all you need to do is just sync a little bit to get up-to-date with where you left off the last time that you stopped syncing? And all of these libraries solve that in different ways.

PouchDB's model is as I was saying closer to Hoodie, which is in this Offline First movement, which I provided a link to that. But if you don't know about Offline First, definitely go to OfflineFirst.org, I believe it is, and read the little manifesto. It's really interesting. The idea is that you prefer the local store because the local store is fast and it is reliable and it's right there. And if you go offline or if you have intermittent connectivity or if you have bad connectivity, then it's totally cool, because you just sync in the background and then changes will come in when they come in.

But you primarily interact with the local store, which is fast and it's right there.

So, I just wanted to clarify. Because earlier you said this is an AP database, not a CP database. And I know you're referencing CAP theorem, which is that you have consistency, meaning that the data is always accurate all the time; availability meaning that the data is always accessible all the time; and partition-tolerant, meaning that the data can be in multiple place all the time. And you have to choose two of the three. So, if you wanted to clarify what do you mean that it's an AP database and why is that better or worse or different than a CP database?

Yeah, that's a good question. You're right. I should take a step back and actually explain then. So, the CAP theorem as you said, yeah basically says that you have these three properties of distributed databases, whether they are consistent, available, or partition-tolerant. And the theorem says that you can only ever have two of them. And this doesn't mean that databases that choose a certain two and not another two are better or worse than other ones. They just have trade-offs and they are better in certain scenarios than other ones.

So, I mentioned that MongoDB is a CP database. There are also some CA databases. Like, I believe Postgres qualifies as a CA database. But CouchDB and PouchDB are more similar to React and Cassandra in that they are AP databases.

AP is also the eventual consistent model, right?

Yes, exactly. It doesn't mean, yeah. So, when it says that it's not consistent, that doesn't mean that it just tells you whatever and you can't [chuckles], you can't trust it at all. Just totally unreliable. It means that…

It means that an hour after your order on Amazon you get a notification saying, “Actually, that's out of stock. Sorry, buy something else for your niece for Christmas.”

[Laughs] Yes, that is a negative example of eventual consistency, absolutely. But I think actually…

But, at least you could place that order.

That is true, yeah.

I think, I think another…

Well, and the other thing is that for consistency, there are usually mechanisms put around the different trade-offs. So, if you have an AP database, a lot of times they require a certain level of consistency when they make the request. In other words, they'll ask more than one server or they'll ask all the servers and get a consensus. Or they'll force some kind of update when they do the request. I know Cassandra all of those things and React does several of them as well. And so, it really just depends.

Just because you're saying availability and partition-tolerance are the guarantees of the system, it doesn't mean that there aren't other mechanisms built around consistency. And so, just keep in mind that yeah, that can happen if you're only asking one server and you're okay getting old data back. In other words, in this case it's inventory data. But in a lot of cases you'll actually wind up making some kind of constraint on consistency. It's just that it's not the focus of the trade-offs in that particular database.

Yeah, well said. These categories are definitely fluid. And there's some wiggle room between them. I think another good way of thinking about how CouchDB works at least is that its replication model is very similar to Git. So, Git is decentralized and Git works very, very well offline. And CouchDB has a very similar model where you can sync the changes but then if there are merge conflicts you resolve the merge conflicts. And if you don't feel like resolving them right now, you can resolve them later and continue working locally and do whatever branching and forking locally that you want to do.

So, can you give me an example of that? Because when I think of ignoring merge conflicts in a database that has user data, I need an example to understand where is it appropriate to ignore it and where is it like, you must resolve it right away.

Sure. So, CouchDB by default doesn't have any built-in conflict resolution strategy. It basically lets you do whatever you want with the built-in default being that it will choose a winner that is arbitrary but agreed upon by all the nodes. So, that's a lot to take in. [Chuckles] So, let me actually try to provide a more concrete example.

So, let's say that you're building a collaborative document editor or something, like a wiki. And let's say you want it to work offline. So, let's say that I'm modifying the same document as my friend Joe. And I whip out my mobile phone and I start making a modification to this wiki page. And then I hop onto the subway. And anyone who lives in New York knows that as soon as you go down to the subway, you have zero bars and you're lucky if you get any internet connectivity at all while you're riding the subway.

So alright, I go on the subway. My changes have not been pushed yet. I was editing one version of the document. And then Joe simultaneously, he's on his desktop. He's got a super reliable fast connection. He starts editing the same old version of the document as me and he starts writing this long, series of rambling paragraphs or whatever. What you don't want, is you don't want that when he hits save, he pushes it to the server, what you don't want is that when I step off the subway, I

whip out my phone, and the change that I was going to push gets pushed and it overwrites whatever he was doing.

And so, what CouchDB and PouchDB will do is that they will recognize that the same version, the same revision of a document was modified by two different users. And it will actually just replicate those conflicts everywhere. So, in fact even people who weren't involved in this conflict, like someone else, Sarah let's say. So, Sarah syncs. She will actually see, in their database, “Oh, Nolan and Joe had a conflict.” And then at that point, the application developer, you the person writing an app on top of PouchDB and CouchDB can have your own conflict resolution strategy.

And you can imagine that depending on what kind of application you're writing, maybe the default is fine. Maybe the fact that CouchDB… so in this case, CouchDB will pick a winner and it's totally deterministic but effectively random. So, every node is going to agree on who the winner is but you can't predict in advance who the winner will be, really. But at this point the application developer can see that there are conflicts. And then you can resolve it using whatever conflict resolution strategy that you want.

So, you can write those strategies in code?

Yep, you just write it in JavaScript.

So, this also brings into question the idea of mutability versus immutability in the database. Like for some things like the wiki, it doesn't really matter who won. No, because when they check back and they realize that, oh, their copy wasn't the one that got saved, they go in and they edit it again. It's not the end of their life or anything, right? So, even if you accept the default randomness, that's fine. But then there's other cases where it seems like things should be immutable, like it should be append-only or something like that. Does Couch handle that case at all or is that, you just use something else if it should be append-only or immutable data?

Absolutely, yeah. So, for certain kinds of applications, what we typically advise is a systems that we call “Accountants don't use erasers” which is based on the idea that an accountant, when they're looking in their ledger and they're saying $50 in, $50 out, $100 in, $100 out, they don't erase the previous line. They just add whatever the change was. And you can totally do this in CouchDB and PouchDB by just adding documents that never get updated. So, you can add a document that just says 'plus 25' or 'minus 10' or whatever. And then when those get synced, there's no possibility of a conflict because documents never get modified. You can also call this append-only or a revisionless strategy.

I want to jump back a little bit to the offline-first idea, mainly because I think it provides a certain level of interesting functionality that you just can't get from other areas. One thing that I'm wondering is, if you're doing offline-first can you keep the same PouchDB database on your page and move to a different page, or does this only work with single-page apps if you're navigating offline?

So, per the IndexedDB and Web SQL spec, I believe that all databases are shared by the same domain. Or maybe it's the host. I'll have to double check. But I believe it's the host.

So, if I'm on MyAwesomeWebsite.com and I go from index to users list, then it can load that up and it'll actually pull the users out of that database even though I went to a different page, because I'm working under the same host name?

Yeah. That should be shared.

Okay.

Although I will admit that most of the use cases I've seen from most of our users are client-side apps. In fact, most of them I believe are even Cordova apps or PhoneGap apps or desktop apps using something like Node-Webkit or Electron. But you can definitely use it for server-side apps for clients, or for server-rendered pages and for client-side apps, whichever one you want.

So, I do want to talk a little bit more about the offline-first thing. So, offline-first has always seemed like a tricky thing to implement to me, especially if you implement your app in an online first way where all of your models for interacting with data are based on hitting an API server. So, when I heard offline-first, I was like, “Great. Add that to the list of things I feel guilty about for not doing.” But it seems like the way Pouch makes it easy is if you model all your interactions with data by talking to Pouch. And you're online, it'll just sync to the server. If you're offline, it'll still save it in IndexedDB. But you don't do different things if you're offline or online. Is that accurate?

Yeah. That's actually exactly the way you should use it. And you're right. Offline-first, Tom Dale mentioned this recently at a talk that he gave where he said, “There's so many firsts now. There's responsive-first and offline-first and mobile-first. There's so much that web developers are expected to do. It's overwhelming.” And that's exactly the bargain that PouchDB tries to offer, is that if you use PouchDB then it should be really effortless to make your website offline-first because that's right. You just interact with the local PouchDB. You tell it to sync and it syncs in the background. You put event listeners on the sync to know the current status. And if you go offline, it'll just continue retrying. And it'll pick up where it left off once it comes back online.

That being said, I do actually occasionally when I write apps with PouchDB, I will have a slightly different strategy which is interesting where if I'm syncing a large amount of data from the server to the client and I want to do it when the app first loads, then while it's downloading I'll actually just use a PouchDB that is a straight up proxy to CouchDB, because you can do that. And then use that while it's still syncing. And that's another benefit I find of PouchDB, the fact that it just implements the CouchDB API, is that you can say 'DB equals new PouchDB local' and that's your local database. Or you can say 'DB equals new PouchDB HTTP whatever', your path to your CouchDB. And then you interact with it exactly the same way as you would the local one. And it just works over Ajax.

So, does PouchDB support any client-to-client syncing without CouchDB servers? So, can I use it with WebRTC or something like that to sync directly to browsers? Or does it need a, I guess you need a WebRTC server to do that anyway. But does it need a CouchDB server in between?

Actually, it doesn't, which is another really cool aspect of this multi-master replication. In fact, if you have two local PouchDB's both running over IndexedDB, you can sync those. That's totally fine. Also you can sync between clients. There are a few different projects that are doing that. One of the most interesting ones is, I think you recently had Matt Podwysocki from Microsoft on. And he is part of the Thali Project which, they are building a decentralized peer-to-peer syncing system for iOS and Android. And they actually use PouchDB. And what they do is they sync over Bluetooth, I believe.

I had a few questions before we go much further. I feel like we're pretty clear now on the differences between CouchDB and PouchDB. So, you just mentioned IndexedDB and then also I

feel like for a lot of people who are not familiar with this, they're probably asking how PouchDB is different than local storage. So, if we could talk about first IndexedDB, also local storage, and how those are different from PouchDB.

Yeah, sure. Well, it's quite complicated.

And that's just an unfortunate part of working with these database APIs, sadly. So, there is local storage. And local storage is built-in to browsers. It's been there since about 2009. And it's supported by almost all browsers. And it's quite good. Local storage is pretty good. But it has a few problems which are well known. The first one is that it's synchronous. So, it can actually block the DOM if you're using it too much, because any synchronous operation blocks the DOM. Also, it's capped at 5 megabytes or 10 megabytes, depending on the browser. And also, you can only store strings. So, if you've worked with local storage you know that if you want to store objects you have to stringify them and if you want to store large binary data or something, you have to base64 encode it.

So, because of these limitations, there were some follow-up databases that were created by the W3C. The first one that they tried to standardize was Web SQL which is basically a straight up interface to SQLite, which is what is typically used to build native a mobile app. So, when I write a native Android application, I'm typically using SQLite. Of if you use iOS, that's Core Data which is also built on SQLite. And that was shipped in Chrome and Safari back in 2010. But it ended up getting deprecated because Mozilla and Microsoft didn't like the implementation. They didn't agree with the idea of just exposing SQLite itself and making that the standard. They didn't feel it was a very web-y solution.

So, IndexedDB [chuckles] followed hot on the heels of Web SQL in about 2010. And that is a NoSQL database that is now shipped in all the browsers in various forms and with various levels of support, which if you go to CanIUse.com/IndexedDB you can see the whole sad story about how not every browser implements IndexedDB perfectly. Notably Safari and IE. So, this is another thing that PouchDB does, is that there are all these underlying storage methods. And it's really confusing and you don't know in advance which browsers support which ones.

So, PouchDB acts like a jQuery for databases. You talk to the PouchDB API and it will use IndexedDB in most modern browsers. It will use Web SQL in things like older Android and Safari. And then if you really need to, you can even use, there's a local storage add-on adapter and an inmemory adapter as well, which you can use.

If your model of interacting with data is just write it to a data store, where do you put things like authentication and authorization? Usually one of the benefits of an API server is you have some layer to put all your logic that you don't want clients messing with. And if you're just letting them sync data to your database, it seems like you would lose that layer.

Yeah, absolutely. You absolutely need authentication. Well, the answer to that is very simple. We have a plugin called PouchDB Authentication. And the way that it works is that CouchDB actually offers its own authentication system. So, most databases that you work with probably have had authentication systems as well. I know that Mongo and MySQL, all of these have some kind of user system, privilege system.

The cool thing about CouchDB is because it's just HTTP and REST, you can just directly expose that database to your clients and then it has APIs for signing up, for logging in. You can on the server-side, you can give certain privileges to certain users. And you can say, “Hey, you can't write that document but you can read this one.” Or, “You can read and write everything in this database but not in this other database.” That's all doable with CouchDB. And CouchDB is missing some functionality there, which is what libraries like Hoodie step in and do. They offer some services on top of CouchDB. And all of this could be put in the umbrella of backend-as-a-service or databaseas-a-service.

So, if I'm using Couch, I would do the authentication in Couch. So, even if my client with Pouch is trying to sync stuff to Couch, it still might get rejected if it's invalid somehow.

Yeah. And that's just…

Okay.

That's just cookie authentication that CouchDB uses.

Okay, cool. One more question that popped in my head when you were talking about browser support. How does PouchDB work on mobile? Is it supported in the 8 trillion IE versions and the 5 trillion Safari versions now?

Oh, yes, yes.

Or not IE, sorry. Android.

Yeah. I know what you meant. [Chuckles] Yeah, so mobile has definitely been a primary target for us, at least for me working on the project. I've been very, very concerned since I started, about making sure that the performance was really good on mobile browsers and that support was really good. Right now we support IE 10 and above, because that's what supports IndexedDB. And then for mobile, we support basically all versions of iOS. I think it's something like iOS 5 or 6 and above. And for Android we support 4 and above. So, unfortunately Gingerbread users, sorry. Please upgrade and get an Ice Cream Sandwich phone.

But if you have 4.0 and above, you're good to go.

Yeah, don't make me rant about Android upgrades with the carriers and crap.

Oh, man. What the world does not need is another mobile tech podcast. So, let's stay away from that.

Yeah.

I'm just saying, yes please upgrade. But you can't always.

So, is there any kind of logic to manage resource usage when syncing? So, say you're offline for five days. You write 10 megabytes of to-do list to your offline to-do list app. Does it just try and send one giant HTTP request with those all or does it batch them somehow? Or how do you balance syncing versus number of requests and speed of syncing and all that stuff?

Yeah, so that's something that PouchDB handles for you. So, the built-in replicator has, yes it does do batch requests both for uploading and for downloading. You can also customize it. There are two options. I believe there's batch size and another one. I'd have to look at the API to see it. But you can configure how many documents it sends in a batch. There are also, because the CouchDB protocol, the replication protocol, is so well-defined there are also some plugins that do alternative kinds of replication. There's SocketPouch which I wrote which replicates over Socket.IO. There's PouchDB Replication Stream which replicates as a stream. So, what's neat about that is that you can just set up an Express server or something that then just streams the replication from the client to the server, or sorry, from the server to the client. But by default the built-in replicator, yes, batches requests with the configuration option.

Sweet. So, can you talk about some of the… have you used Pouch in your personal projects or client work or something like that?

Yeah. I've used it quite a bit.

Do you have some war stories about real world usage of it?

Oh, yeah. [Chuckles] Actually, probably the most interesting story is I sent you guys a link to, there was a talk at JSConf EU from someone who works at eHealth Africa, which they were using PouchDB to battle the Ebola epidemic. So, what they did was they were a company based in Berlin. And their goal was to write a mobile app, an Android app, that would be used by tracers, which are the people who go out and keep track of who is showing symptoms of the disease and what their status is. And they wanted to use PouchDB because in Western Africa where they were operating, mobile networks are notoriously bad. The network is just not reliable. And so, they honed in on PouchDB because they could use it to just store data on the phone and sync it back to the server when the tracer managed to get back to connectivity.

And they also had some interesting stories, if you go and watch this talk. And I'll send you guys a link. But one of the interesting stories that they had was that at some point, they ended up having a bottleneck because they had one database on the server which I believe was running in Berlin. And they were finding that the latency was just too bad and people were trying to replicate and it was taking too long. And then at some point they came up with this “aha!” moment where they realized, “Oh, since this is multi-master, we can just spin up another CouchDB at their call center somewhere. And then replicate from Berlin there.” And then every call center can just have its own CouchDB and they can all replicate to each other. They don't need to replicate all of them from Berlin, which was a really cool use case I think.

This brings up another point. When a CouchDB or PouchDB or something, at some point it gets too big, right? There's too much data to handle all at once. So, it's not just a matter of multiple servers or multiple clients but it's a matter of the data itself needs to be partitioned, right? How do you handle that?

Yeah, so that's a good question. So right now, that is supposed to be handled by the purge function. The idea of the purge function being that you would take some subset of your database and you would blow it away. And you wouldn't try to replicate those deletions back to the server, because it's not really deletions. It's just, “I'm not interested in this data anymore.” That has actually not been implemented yet. That's the last thing we have to do before we have 100% API parity with CouchDB.

But right now, what you can do is you can compact, which compaction is an operation that removes all the non-leaf nodes in the tree. So, if you imagine it, the CouchDB database being like a Git tree, like branches. And you imagine all the conflicts being leaves on those trees, then what compaction does is it just removes everything that's not a leaf. So, it removes all your history except for those things on the very fringes. So, that could save some space.

What you can also do is you can also just replicate between local stores. You can do a filtered replication between two local PouchDB's. But yeah, ultimately that will be solved by the purge function.

So, if people want to get started with PouchDB, what's the best way to do it? Just include the script on the page and go for it?

Yep. You can go to PouchDB.com and click on a button to download it or you can 'npm install pouchdb', 'bower install pouchdb'. There's lots of ways to try it out. And there's also a 'Getting Started' video and tutorial on the site, which stars me. So, I highly recommend that you check it out.

That's awesome.

Yeah. So, you would just put it on an HTML page, pull up the page, open Developer Tools, and then go on the JavaScript console and just hack away at it?

Yep. That's a great way to do it. There is also an inspector plugin or an inspector extension for Chrome and Firefox so that you can see what your database looks like as you're working with it.

Oh, nice. If people want to contribute to PouchDB, what do they do?

They should go to GitHub.com/PouchDB/PouchDB. And anybody who wants to get involved definitely should. We were recently very inspired by Kent C. Dodds' article about 'First Timers Only'. And we actually created a bunch of issues saying, “This issue is only for people who have never contributed before.” And we were just totally overwhelmed by the response because I think we got three or four issues fixed within 24 hours each of posting them.

Nice. And are there other ways that people can contribute besides code, if maybe they feel too new to even do that?

Oh, some of these weren't even code. Some of them were just documentation.

Oh, nice.

And we absolutely welcome that. Documentation is just as important as code. And we are actually very, very serious about that. If you go to the PouchDB website and look at our documentation, you'll see that every page actually has a little pencil symbol. And if you click on that, you will immediately start editing that page. Even our blog posts. So, if I make a mistake in any of my blog posts or there's a typo or I'm just wrong, then please feel free to go on and correct us. We love hearing from people.

Very nice. And I do want to commend you on the first-timers dealio. Because that's important that people feel like they can contribute to the community very early. And I think that's how we get more people in as we have more people coming in, into the area where they're contributing to open source.

Absolutely.

I love it. I just wrote in [my notes]: This proves you at least know Git. [Chuckles]

That is true.

So, if nothing else, if people [inaudible] at you doing a doc PR, you can tell them, “Well, I'm showing someone I know how to use Git.” [Chuckles]

Can you maybe talk about what's coming in Pouch next? Are there any exciting things coming down the pipeline?

Oh, yeah. So, there are some exciting things. So, besides the purge function which is much requested and people will be very happy when that's there, we are also working on custom builds. Because some people don't need to support Web SQL or some people just want to use PouchDB as a straight up client to CouchDB and they don't need the IndexedDB or local map/reduce or any of that. So, we are working on custom builds. And oh, and CouchDB 2.0 is coming out soon, which is really exciting. So, we're going to be really happy when that comes out.

Very cool. Well, I'm sure people are going to want to follow that. If they are interested in following what's coming and what is coming out with PouchDB, how do they do that?

You can follow us on Twitter. It's just PouchDB is the name of the account. You can also hang out in Slack or in the IRC channel. And all of those links are available on PouchDB.com if you just scroll to the bottom. There are links to the forum and the chat and all that good stuff.

Super awesome. Let's go ahead and get to some picks.

Before we get to the picks, I want to take some time to thank our silver sponsors.

Jamison, do you want to start us off with picks?

Do I ever. I have four picks today. The first pick is a cool npm…I guess it's a program. It's not really a module. It's called source-map-explorer. And I was just messing around with source maps this week. It gives you a tree view of all of the modules in your build. Like if you're using Webpack or Browserify or something and you have source maps, you point it at the source maps and the build file. And then it shows you with this series of nested rectangles how much physical space each package is taking up in your bundle. Which is really helpful and made me realize that some of the dependencies that we don't use that much are actually adding a lot to our load time. So, it's helpful for optimizing things.

My next pick is a series of videos that Facebook put out called 'Managing Bias'. They're about unconscious bias. And I think it's kind of related to hiring but it's not explicitly about hiring. It's just really well done. Cool stuff.

And then my next two are related to performance of computers in general. One is this game called Computers are Fast.  I guess game is in air quotes. It's this quiz where they show a program and then you guess how many iterations per second a program can do or how many bytes it can process or something. It kind of helps you get a feel for how fast computers can do things and how much data they can send and receive. It's usually more than you think if you haven't done a lot of that kind of stuff before.

And then my last pick is this article someone linked to from 2011 where they took a state of the art 1986 Mac and then a state of the art 2011 computer. They loaded word processing programs on them and compared how long they took to boot and to load the word processing program and to save. And it was the same speed for faster for the computer from 1986, which is terrifying [chuckles] and kind of fascinating. Basically computers get way faster and we write programs that are slower at the same rate that they get faster. Yeah, so those are my picks.

Alright. AJ, what are your picks?

Well, I'm going to follow up with the iPhone 3G there because I saw somebody using that in town the other day. And I was freaking amazed that they were flipping through and it was so fast and everything opened so quick. And it brought that to my mind of exactly what Jamison was talking about. But that's not really what I want to pick. What I really want to pick, I've got a couple of good ones.

Okay. One is the Authy App. So, I'm co-founding [Gaply] and one of our company policies, all two of us so far, is that we're going security-first which is like mobile-first or offline-first but security-first. And so, I've gone through and made sure that we've turned on two-factor authentication on everything. And then I've got a different phone. And it was a really sucky process of then having to redo that with eight different accounts. And the Authy app will allow you to transfer all of your authentication codes from one phone to the next phone. So, when you upgrade or side-grade, you can keep your two-factor authentications. So, that's way cool.

I discovered the Chip Network channel on YouTube which I'm always picking chip tunes. And so, if you like chip tunes and video game remixes and overclocked remixes, you will like the Chip Network channel.

I'm going to also pick the great state of Oregon. It's very nice here. It's very green. When I breathe in it's like I'm having a little bit of, like a light snack. There are just nutrients in the air. And I'm enjoying that.

And then I also wanted to put a shameful plug. We have made some libraries for two-factor, multifactor, one-time password authentication. One's called Browser Authenticator and the other's called Node Authenticator. So, if you want to check out the demo, we've got one where all of the basic two-factor, the six-digit number stuff that you see on other sites, all of it's being done in the browser. And then all of it's being done in Node as well.

Alright. Aimee, what are your picks?

Okay. So, I have a lot of picks around my trip last week to AngularConnect which will be now a couple of weeks when this comes out. But everyone at the conference was super, super nice. The Angular community was super nice. The organizers of the conference, Pete Bacon Darwin. He was just way, way, way nice. And then I don't know. I just want to say thank you to people because in case they're listening or in case their friends are listening.

There's a guy named Kevin Old who is in Nashville, Tennessee who I worked with to get the idea for my talk. So, he's just, I want to say nice things about him. And then someone named Jordan Kasper who is a Developer Evangelist for StrongLoop. He helped me really prepare for the talk. All in his own time and I just really appreciate both of their help. So, just a great week. And that was my pick. If you have a chance to go next year and you're an Angular developer, you should totally go. And then if you know those people, just be nice to them because they're nice people.

[Chuckles]

Alright. I'm going to throw a few picks out there. One that I've been fiddling with lately and I've been really liking is Highrise. I tried Salesforce. It took me about seven days to give up on them because I couldn't get it to do a few things that I wanted it to do. And once I got close, then what wound up happening was I wound up getting on their support forums and finding out that it didn't actually do what I needed it to. And that is basically I want to be able to tag people in my CRM and then I want to be able to email them all. And so, if they're all interested in the same kind of thing then I want to be able to reach out to all of them and say, “Hey, it's me. Here's what I'm doing.” And it didn't like that. So anyway, I moved over to Highrise and it is working terrific. It's a little bit cheaper. And it does exactly what I want. So that said, I am picking them.

Another one that I want to pick just in general, I've used Gmail for years and years and years. There is a plugin for Gmail called Streak. It is also kind of a CRM. It does CRM functionality but it also does, you can postpone emails like Boomerang does. But it also gives you the option if I remember right, I think it's Streak that's doing that for me, where it'll automatically BCC stuff over to Highrise for me. So anyway, liking that.

So anyway, those are my picks. Nolan, what are your picks?

So, I have three picks. My first one is a book by a French guy called Maurice Druon. And it's called 'The Iron King'. And if you've read 'A Game of Thrones' or 'A Song of Ice and Fire' and you're really into that stuff and you've finished reading all the books and you devoured them like I did, and if you're still hungry for more, this book is the book that inspired him. And it's really cool because it's historical fiction. It takes place in 14th century France. And all the stories are true. But when you read it, you see there are so many parallels between things that actually happened in actual real life France and things that he describes in his book. So, it's basically 'Game of Thrones' minus the dragons and magic and stuff.

And my second pick is a series of documentaries on YouTube called 'The Smash Brothers'. So, if you just search for 'the smash brothers' you'll find it. And this is… it's an e-sports documentary. But don't get scared. It is amazing. It is so fun to watch. It's a documentary about Super Smash Bros. Melee which was the version of Smash Brothers that came out for GameCube. And it is, it just follows the competitive scene over the past 10 years and how the game has continued to live on, even though it's been replaced by other games. People just continue playing it because it has really tight mechanics and it's really, really fun to watch and to play. And it's a really engrossing documentary.

And then my third pick is just Super Smash Bros. Melee itself because it is one of the greatest games of all time and I still love it. And we play it almost every single day here at Squarespace, believe it or not. [Chuckles] We have some people who are very, very serious about it. And we practice our [tech skill] with each other and have lots of fun. So, those are my picks.

Alright. Well, thank you for coming, Nolan.

Thanks for having me.

We'll go ahead and wrap up the show. We'll catch everyone next week.