When is it worthwhile to introduce a new language, tool, or database? And when will it likely bite you in the rearend?

02:43 - Episode Idea Background

04:28 - Implementing Standards and Comparisons

08:23 - “Dumb Code” and Developer Expectations

10:48 - Code Coverage and Regular Expressions

12:49 - Risk Impact/Probability Chart, Risk - Reward Matrix

24:01 - Collaboration, Communication => Constraint

30:36 - Bringing It In: Process

38:48 - Why would you want to switch databases and when is it worth it?

46:06 - Success Cases

Picks

OS4W: Open Source for Women (Coraline)
Contributor Covenant (Coraline)
Camille Fournier: Hopelessness and Confidence in Distributed Systems Design (Jessica)
Abby Bobé: From Protesting to Programming: Becoming a Tech Activist (Jessica)
Rails Remote Conf (Chuck)
TV Fool (Chuck)

 

And then every time I call these people up, I just wanted to pull my hair out so I was done.

Is it too late for that, Chuck?

Yeah, it is.

That's what my kids would say, too.

Hey, everybody and welcome to episode 229 of the Ruby Rogues Podcast. This week on our panel, we have David Brady.

Hey, everybody and welcome to episode 229 of the Ruby Rogues Podcast. Today on our panel, we have Charles Max Wood.

[Chuckles] Thanks. We also have Coraline Ada Ehmke.

Hi everybody.

And Jessica Kerr.

Good morning.

I'm Charles Max Wood from DevChat.TV. And this week, we're going to be talking about when is it worthwhile to introduce a new language, tool, or database and when will it likely bite you in the rear-end. So Jessica, you suggested this. Do you want to kindly get us going?

Sure. A few months ago, I went to PolyConf in Poland and it was a wonderful conference. We talked about all these different languages. Yet, I have this nagging question in the back of my mind of I love trying new languages but I don't always like it when other people at work do. [Laughter]

Ooh, you're the only one that happens to, so...

This actually happened to me right now too.

My rule of thumb for introducing new tools or learning new tools, languages, databases, whatever, it's okay if I do it. That's my rule.

It's also okay if we standardize as long as we standardize on what I want to use.

Yes.

Yeah.

Yes.

So I think some companies decide to be polyglot just for the sake of being polyglot. Just to say that, "Oh, we're really an exciting place. We should try these new technologies, all these new languages, all these new and latest tools without really looking at are these things going to add values to our stock or not?

I totally have this image in my head of two developers sitting in, facing cubicles and one of them leans over and says, "Pardon me, do you have any [Grey Poupon]?" It's just we're so good because we use all of these varied and different tools.

I think it is important as a developer to be polyglot because that expands your thinking and that expands your problem solving vocabulary. But there has to be some kind of criteria that a new language or tool meets before you introduce it into your work stack.

Because working at many languages makes development more exciting but it also makes protection more exciting.

Yeah, in the bad kind.

I want more introduction. Yeah.

In the bad kind of exciting.

Exactly. Right. So Coraline, you said there should be some standards. Why is that?

To make sure that it's not resume' driven development basically. To make sure that there's a clear case to be made for a new tool or technology or language that is going to solve an actual problem that actually exists. And not just it's something interesting and a diversion, it's something like you said to make production more complicated.

So I guess the thing is this, it may be a great tool but when do you know you have the right sales pitch for it? When do you know when you're describing it to your co-workers that you're saying the right things about it?

Again, I think it ties to we have this problem and our current tool set is not solving that problem very well. This other tool has promise for letting us solve the problem in a more effective or efficient way. I think that's the best possible case you could make for something like that.

I just realized I'm in the middle of writing a rant about good software like writing good code. I realized it absolutely applies to these tools. Can I rant for a second?

Rant. Please rant.

Okay.

Oh, yeah.

All right. Here's my rant for good code and good tools, okay? It should minimize the buy-in effort for the entry level. So the amount of effort learning needed, to be able to just get around in the tool. We've all sat down at a code base and we had to fix a bug and we have no clue where to even begin looking, right? Because I mean, it's in 47 directories that they're 15 levels deep. Who knows where anything is?

But it should also maximize pay-off for mastering the learning curve. So what I mean by this is there are two classic arguments for how we should write code. Should we write smart code or should we write dumb code? We always argue in the negative sense like we talk about writing code that everybody can understand. They usually mean programming for the lowest common denominator or programming for dumb people.

They think they're saying minimize buy-in but what they're really saying is cripple the pay-off for anybody bothering to master anything more advanced than the five operators for incrementing and documenting variables. On the other hand, when you have people who get angry because they've been crippled by this, they then turn around and they say, "No, we should expect our developers to be smart. We should expect people to step up their game. We have a minimum level and it's up there that we expect you."

It turns out that they think that they're maximizing pay-off for mastering the learning curve but what they end up doing is just enlarging the buy-in for the entry level, for the people that are trying to get around. So, that's my rant. If you're going to write good stuff, it should be easy or if you're going to introduce a new tool, if you're going to say, "I want to solve this problem in a different way than we're already solving it because the way we're solving it right now hurts."

If you can demonstrate that the new way of solving it is easier for somebody who's never seen the new solution to use at a simple rudimentary level and yet also has all the extensibility that if somebody takes the time to learn it, they can actually do more and more things with it. Then that's a good tool and that's a good case. That's worth making the argument to production of I'm going to make your life a little interesting for a little while. I know that's bad but hey, all change makes production interesting. We just have to vet the change.

That's my dog scratching herself by the way. That's the jingling in the background. And that was her shaking and now she's walking off. That was Bella. That's her place in the podcast.

But yeah so, anyway, that's the end of my rant. It's just minimize entry level and maximize the payoff at the other end. With production, all change makes things interesting for production. So when you want to introduce your change, all you have to do with production is you have to vet that change. You have to say, "It's going to be worth it."

David, I'm interested in a couple of things you said. First of all, about dumb code, do we want to write our code for the lowest common denominator? How do you think that's influenced by trying to level up developers that you're working with that maybe are more junior. You have changing expectations for them over time that their code is going to get smarter and smarter?

No, I have an expectation that their code will get clearer and clearer. The epiphany that I had on this is that you don't have to write dumb code in order to write good code. What I mean by this is that if you take the time to write an intention-revealing method instead of trying to figure out how to code golf it then that's going to be better code. That's going to be code that is...

Right.

Right? And it's arguably, okay if you're dumbing down, so if you got some code golf and you remove it, you replace it with 15 methods that are all supposedly intention-revealing, okay. You're probably dumbing things down.

But on the other hand, if you take a really gnarly regular expression thing and then you wrap it in a method call that tells you exactly what that gnarly thing is doing, you now have a bit of code that somebody can read and go, "Oh, this does that." Then they can look at the gnarly regex and go, "What on Earth is that?" How does this even… this is line noise, how does this even parse? Somebody who is at a more junior level that can kind of back away slowly from that weird line of code. They could just run the test suite and see that that line is covered and go, "Okay, it works. I'm just going to back away from this now." Then a senior person is going to look at that and go, "I totally know how that works and yes, that's a good name for it."

But that's the trick is that the people who are let's try it smart code, they'll just inject that gnarly regex in the middle of some methods somewhere without trying to reveal their intention. That was the… the second part of my rant is we should stop calling code, code because I'm sick and tired of people trying to encode what they mean with a sufficient level of intrepid that the Germans can't break the cipher.

[Laughing].

Stop trying to...

Stop trying to encrypt your software. Write software. Stop writing code. I don't want to learn your code

Nice.

That's hilarious. You mentioned code coverage on a regular expression? I want to see a code coverage tool that tells you whether every possible case in your regex has been hit.

So yeah, a code coverage tool is going to back away slowly in curve when it sees your regular expression. You're going to get a yes or no. Was this ever touched?

Right, right.

It could be a good case for generative testing, Jessica.

It could, it could. Just measuring that, oh gosh! I don't want to look at a regex account for different ways to do it. Ooh, but you're right. You're right. At least with the generative testing, you'd be more likely to hit one of those important cases.

Right.

Yeah.

But I believe...

Yeah. And it gets worse because if you're using Oniguruma, that's a turing-complete language and so you can prove...

Oniguruma?

Yeah, Oniguruma.

It's the regex engine, a regex engine in Ruby.

Okay.

Yeah, Ruby's regular expression. It allows you to embed call backs and recursive call backs into itself. I saw a person actually write the entire Fizz Buzz solution in a single regular expression using call backs. And it seems like about a year ago, we had Tom Stuart on the show and we went through understanding computation and he had this big rant about how you can't parse HTML with regular expressions because normal regular expressions are not turing-complete but HTML is a turing-complete language. So you'll never be able to parse all of HTML with regular expression. You just can't do it.

Well, it actually turns out that with Oniguruma, you can because it's also a turing-complete regular expression. But let me tell you, it's not going to be simple. It's not going to be angle bracket, find me stuff and then close angle bracket. You're going to have three pages of code barf in that regular expression.

Which goes back to your solution of yes, give it a name. Give it some tests. Layer the understanding on top of the cleverness.

Yeah.

And I agree with you. Part of code being clear is knowing your audience. If your audience is a bunch of Ruby developers then writing your program in another language is going to have an increased cost so there better be a good pay-off.

Yes, yes. There's a developer at CoverMyMeds named Mark Lawrence and he's one of the smartest human beings on the planet. I mean, he can shatter a human mind at 40 paces. It's just astonishing. He wrote some pairing tools that let you share, kind of like TeamMate does, it lets you show you your terminal and you can join and unjoin and it'll play back the session. He wrote it kind of on a dare. He wanted to learn go. And this is actually, you know what? I've named Mark. I'm going to praise Mark. I'm going to ding him at the same time just because I can. I love you, Mark.

This program is called go-pty-tunnel, right? It's like a tunnel start and tunnel stop and that sort of thing. It lets us connect over the VPN, over an encrypted connection and lets people that are out on sticks like me, it'll let us exchange PHI and PII as much as we need to as developers. I shouldn't see very much of it but when I do it's encrypted over this tunnel. It's a good thing. This is a perfect example of what I would call a good tool because I haven't learned go yet but I don't need to, to use this tool.

I just say, "go-pty-tunnel," and poof, ended up comes up and it runs. There was this weird hiccup that went wrong with it one day. I pinged him and he says, "Yeah, it does that and just stop doing the thing that makes it do that." It's like the oblinx thing of when I move my mouse to the left, it hangs the computer and the guru says, "Stop moving your mouse to the left." But it was like a really weird corner case. It was like yeah, I was trying to send command option something over gopty and it wasn't going through.

It was like, "Yeah, I don't do that." I'm like, "Yeah, yeah, I don't really need it." So that was fine. But if I wanted to learn go, I could add that feature or I could debug that problem. So this is a tool that would maximize investment in the learning curve but it also minimizes the minimum buy-in to use this tool. So comments on that and then I've got a counter case that I can also pull from Mark's code back.

Well, so the thing that comes to my mind is that - so the tool doesn't add any cognitive load to you because you can just use it to do whatever it is that you're doing, right? The issue becomes what if Mark gets hit by a bus then it becomes somebody's cognitive load. It's not in the core competency of anybody else in the company.

Right.

And so I mean go may be the right tool for the job and ultimately it may make things better all around to have used go to solve this problem but at the same time, there is that trade-off. If somebody comes in and says, "We'll give you a million dollars to come help us with our start-up." He's gone.

Yeah.

But it's just a tool.

Yes.

If you didn't have this tool or if you are unable to modify it and Mark wasn't there, you would probably use a different tool.

Yeah.

The risk of using this tool is very low.

Right.

Because you never get to a place where you're worse off without it than you were before you had it.

Right.

It's a drop-in.

Yeah, it's a drop-in because we have not maximized, because he hasn't maximized the minimum buy-in entry level for this tool. There's nobody at our company that has to maintain this tool.

Right.

It's just there and it has a known issue. Nobody including Mark cares to fix it and it’s fine. We just live with it. So now, what we're literally trading off is this problem that Chuck just mentioned, right, that if Mark leaves, okay yeah, we have a couple other golfers at work.

Right.

But let's say we didn't have any and okay, now this tool becomes a legacy tool, right? We don't have to throw it out right away. We can wait until we have a problem or there's a reason we can't use it. And then we'll go out and we'll look at TeamMate or we'll look at some other screen here or whatever and that kind of thing.

So this is still a really good example of the hit on maintaining this tool down the road. You said you nailed it just right, Chuck, which is the cognitive load, the number of things that people who use the tool have to keep in working memory just in order to buy-in and use the tool and be at the table is minimized.

And so in that particular case, having an environment where we actually tell our developers we want you to go learn new languages just for the sake of learning new languages is absolutely worth it. Because we were learning new things, we're learning interesting things. It's the kind of interesting that's exciting and encouraging and not the kind of interesting that's like our server burned down at 2am on a Saturday and production is down and that's a bad interesting.

There's a reason that this one is low risk and useful and not scary and it's because it is a tool. You didn't introduce a new programming language that's going into production.

Yeah, I also see that it's not mission critical for what you're doing.

Yeah.

I mean, you pointed out that you can find other viable replacements for it. So yeah, it's not high risk that way either. Because if something changes in the ecosystem and it just doesn't work anymore and Mark doesn't have time or doesn't have the inclination to fix it, then you just move on to something else.

Yeah. Are you guys familiar with the probability and risk matrix?

No.

Explain that.

It's from economics and I can't remember all of them. There are four quadrants. Basically there's how likely is this to happen, the low probability or high probability? And how much danger is there if it does happen? So low, is it low risk or is it high risk? So this tool hits the sweet spot because it's low probability of going off the Rails on us. And it's low risk, if it goes off the Rails, we don't really care.

The one thing I remember from economics is that if you have low probability, high risk - buy insurance. That's what you should insure. If you have something that's low probability, low risk you should not insure. If you can afford to replace that consumer item, you should not pay the insurance premiums. That's one of the things that I remember from it.

So yeah, so a bad tool or a bad coding decision is high probability, high risk. I'm going to start using binary pointers in Ruby. Why not? In fact, I'm going to recompile Ruby to add case and sensitivity to it. We're going to push that to production. What could possibly go wrong, right?

Anyway, that's the risk reward matrix.

I think the matrix is an interesting way to start categorizing tools that you want to add to the tool chain. That's sort of what I was talking about with having some kind of criteria even if it's [inaudible] economics, even if it's something that’s made up on the spur of the moment. Because having some sort of evaluation criteria so you're not just saying, "Oh, I'm interested in this and so everyone has to learn it."

Yeah and that's actually a really fair point, that this tool is something that the developers use and they don't have to work in. They're not soaking in it. They're just using it. It's a tunnel. It just appears and you can magically share your screen with somebody else whereas writing software, well now the rest of your team has to soak in that. That can make people crazy. But your end user doesn't have to soak in that.

And so, there are things that you can do in your software that are gross and gnarly. But as long as you all agree that yes, that's not the stupidest thing that could possibly work but it is pretty simple but we don't have time to optimize it, okay fine. Just push it out there. As long as the customer doesn't know this and doesn't care then for the customer, it's low risk, low probability whereas for the developers, it's high probability because they're soaking in it. They're touching it but how much risk is it? How seriously nasty is it? And that's the question you have to argue the merit of that.

Yeah, but you also have to take into account the pay-off, too.

Yeah.

And the probability of success.

Right.

There are both ends to that. So if it's a high probability that you're going to be successful but it's also high risk in the sense that it if it doesn't work out, it's high impact. You do have to account for both of those things.

Yeah, I said risk reward matrix earlier and that was incorrect. I meant the risk in probability or the impact in probability matrix. But yeah, you balance that against the risk reward matrix which is the same thing, the probability of gaining a reward and the amount of the reward. And yeah, you offset that.

The point that I was making about developers soaking in it is that the further down the delivery chain you are for a given decision usually and I could be completely wrong about this but the impact of that is usually ameliorated because you usually don't have to go up your food chain to...

What is this delivery chain?

I'm talking about things like the text editor that you use, something you use to write code. The codebase is something that you as a team put together and it makes a website or makes the thing that the IT team has to publish and put to production.

You have a separate production support team?

Yes. Oh, I shiny thinged you. Good, good.

Oh, no, no, no, no. I was making fun of you for not being DevOps.

Oh, okay. Yeah. Well, DevOps is easy. You just cd/ and then chmod-r777, right?

I don't want you on my team anymore. [Laughter]

Nobody knows after you say that.

Now, we know why you have a separate production support team.

Yes, yes, exactly. They made it just for me. I'm also the reason we have, actually we don't have an HR department but I'm kind of the standing joke about someday we're going to have to have an HR department and you're the reason why, Dave.

But that's what I mean. It's like the production chain, the food chain, right? It's like you could be using RubyMine as your text editor. I could be using Emacs as my text editor. Anybody else on the development team doesn't care because they are receiving text and you deliver. The IT team couldn't care less what text editor you're using. They also couldn't care about what code you're writing. They just have, does it work? They just have to push it out. And user doesn't care what production is doing. They just want to access the website. And so that's what I mean by the further down you get down the production tool chain, the lower probability and the lower well, you don't actually have ..

Less frightening?

Yeah, the lower the impact of a problem. I want to say I think probability just channels right through if I make a bad decision in a code. Yeah, if I make a bad decision in Emacs, it can end up writing all the files on the disc bad and then production pushes this. Yeah, I don't know what I'm talking about.

No, that's why we have version control.

Exactly.

So you can kill your own computer.

Yup.

So I'm curious with your idea of the delivery pipeline, there are some decisions that you make as a developer that do have downstream impact. For example if you choose...

Or sideways impact. Yeah.

Yeah, if you choose a particular language and now there's a new app framework that has to go with it and maybe all your doc or images have to change, somebody has to adjust to that. How important is it to collaborate on these sorts of changes with different people who are responsible for different parts of delivering that package to production?

Wow! I think I've actually turned out to be the guest on the show. That's weird. [Laughter]

It's kind of fun, actually. I'm actually making this up as I go along, by which what I mean is you're actually using the Socratic method on me and teaching me about my own idea that I was going to rant more heavily on this. This is actually really beautiful that if you have, of course I can turn it into a joke. So going back to the beginning of the call, right, as long as I'm the one learning and playing with the new tool, it's okay. But if anybody else does it, it's not okay.

That holds true for sufficiently large values of me, by which I mean if you're going to decide on an event-based web framework instead of a procedurally donor or threaded-based standard like Rails type web framework that impacts everyone sideways to you, all of your peers in the production chain. So a sufficiently large value of me is my entire team then you absolutely have to collaborate. That's kind of what I would define as the - how do you define sufficiently large values of me is, are you communicating with each other?

If you're not communicating with another vertical in your company, you can't ever really say that you're collaborating at a native level. You can collaborate further down. You can both be shooting into the same output bucket. You can synchronize that but one of you could be doing event based go or whatever or Elixir-based stuff. The other one of you is doing straight up Ruby on Rails and nobody really cares. But within your team collaboration, it's one of those things that were, there's stuff that’s important and then there's stuff that's required. But there's actually a higher level priority than requirement and that is a constraint.

And so collaboration, I would say in this case is a constraint. If you're not talking with the rest of your team, by definition you should not be rippling changes on to them.

You want to say, "Surprise! I just wrote this entire subsystem in go so good luck with that.”

Yeah.

Or even in Ruby, let's say that you're course stack is Rails. I had a boss at one job and he was the very definition of a cowboy coder. So he would stay up all night and he would write about half of the feature. He wasn't working on our team all the time so he didn't understand all of the constraints.

So when he'd get in and he'd write half a feature, he'd change half of the other features in the app. Then we'd show up the next day and he'd be like, "Well, I got it most away there. Go ahead and fix it or go ahead and finish it." And what would wind up happening unbeknownst to him was we would go in and we would rip out all the changes he made. Then we would put the feature in the way that it needed to go in.

Nice.

But in some cases, he would get upset because he'd go and look at it later and he'd realize that none of his code made it into the codebase.

Yeah.

Ultimately it's those kinds of things, too where you make these assumptions. You decide something needs to be done. And yes, without communicating with the rest of the team, you force them to deal with this pile of garbage you've left in the middle of the floor. Now, whether or not it's actual garbage doesn't really matter. Because when they come in and they see it and they don't understand it, it looks like this big ball of string that they've got to go unwind to figure out what you did so that they can actually make it function in a way that the team can accept.

Yeah. I laughed when you said your boss would stay up all night because I was literally about to cite that exact diurnal condition, if you will, or nocturnal condition that we've all come in to work in the morning. The crazy guy on the team has stayed up all night and has converted everything to XML-based python and you're like, "What?"

Oh, God! Please, no.

Yeah, right?

And so, sideways changes. I would be okay if he'd stayed up all night or if he had stayed up all night writing this crazy thing and put it in a branch. Then you show up, everybody shows up the next day and then this developer says, "Okay, real quick team meeting. I just did something completely crazy. But it takes our web server from 1200 ms per request to 12 ms per request." And now you've got the rest of the team going, "Oh, really? EML and python you say? No, no, python and XML. Hmm, tell me more."

Yeah, I know. My skin is crawling, too. But if you included this constraint of collaboration and communication on a sideways, moving sideways inside the production chain is what I'm saying. Basically the people that are in the same level of the production chain is you. That's what I'm talking about when I say moving sideways. If you aren't communicating, then by definition you are placing yourself upstream in the chain to everybody else. I guess that completely shoots my theory that downstream in the chain lowers the risk of change or impact, doesn't it?

You've talked about collaboration and coordination being necessary constraints but who's responsible for training in a scenario like that?

That's a really good question. I don't know that I have a good answer for that.

Someone tried just the other day. I don't remember who it was. Somebody at [inaudible]. That when you bring a new language into your work, you are taking on the second full time job which is being that expert for everyone in the company.

I like it.

Yup.

And that should be part of the cost of when you consider vetting the cost of change, you should include that, right? "Oh, yeah, I now am the full time go evangelist because I brought this is in."

"Lucky me," he said and huge on productivity so definitely something to consider at the very least, she should probably put together some training materials or do a brown-bag lunch or do something along those lines. But yeah, you have to be that expert moving forward. Yeah, you're going to be the person that people are going to go to and that should factor into your decision.

So I think we've talked a lot about impact. We've talked a lot about whether or not it's worth it. But yeah, let's talk about bringing. So let's say that you've convinced everybody, okay, to bring in this new technology, whatever it is. What is the process for bringing it in then? Do you just go after a whole hog or should you experiment with it for a while and see if it really delivers the way that you expect it to? Yeah, who does the training? Where do you go to get expertise? How far do you adopt it into your system? Do you move everything over to the database?

I think it depends a lot about what technology you're talking about. If it's a tool, you can just start using it with your team and maybe it’ll spread to other teams. If it's a language, you've got to prove out the production support story and the maintenance story and the upgrade story. If it's a database, run away.

Seriously, when you talk about bringing in a new database technology, this is really frightening because if you put code in production and it doesn't work out and you want to replace it, you just replace it. But once you start giving a database your data, it owns you.

I was talking to someone once about why I didn't like SQL queries embedded in Ruby cat and they just sync the PHP to me. One of the arguments I was making was database portability because if you are keying down to a feature of a specific database then that makes that code really fragile, really brittle, you can never change the database. The person stomped me when I said, "How many times have you actually changed your database in your production system?"

Right.

Right. It's really, really hard.

Yeah. I have worked in environments where we had more than one database type and I've seen it done well and I've seen it done poorly. What it usually boils down to is how well you understand the data, and the shape of the data, and how the data needs to be thought about.

You can have more than one database. Yesterday, I was talking to Aaron Bedra who works at Groupon. At Groupon, you could have any database you want as long as it's Postgres or Cassandra.

Right.

Because you can support multiple databases but when you as a company are supporting a database, you have to commit to that. You need a team of people who really understands the database and the data you're putting into it. You need to be able to upgrade it and add, configure it, and add space when you need to. It's a major commitment for the company.

I'd like to take the… there were two comments there. First of all, Coraline, you mentioned how many times have you changed the database? That's actually a self-fulfilling prophecy because nobody is writing portable layers. It's part of the reason why changing database is so hard, right?

So it's kind of like saying, "Well this is so hard to do. Let's just go ahead and make it even harder."

I disagree with that.

Okay. Well, I'm saying like this is so hard to do, let's not invest in making this easy because it's low probability, high risk. We'll just buy insurance against.

I disagree with that in this particular case. The code is much easier to change than moving the data to another database.

Oh, this will be fun. This will be fun because I was about to go there next which is that at CoverMyMeds, we have several different verticals, right? There are some core technologies that we all use across the entire company. Like there's the drug’s API for looking up prescriptions. That service is visible to everybody in the company; is used by every team. So it's a layer, it's a horizontal layer in the company stack.

Nice.

Seems nice on the face of it. It is an absolute freaking nightmare because you can't change anything because it breaks, like you literally can't even tell. I found a bug in the thing that handles prior authorization requests. I'm like, "No, this is… There's a comment on there saying that this is going away soon." And I get blamed. Yeah, it was 24 months old. And I went and checked and the bug was still there.

I tracked down one of the senior developers and I said, "This bug is still here. Can I just change the validation on this to require this field to exist?" He's like, "Oh, no, no, no, no, no. We still have people in production that are writing incorrect requests to the database.” We argued about it a little bit and basically, the conclusion we came to is that we're not writing incorrect request to the database. Because this is a horizontal layer, this is now the de facto standard.

It's a contact. It's an API.

Yes. Basically, it's like we can't, you can't break the drugs' API. You can only break yourself against it.

To abuse a quote by Cecil B. DeMille. But I would argue that a database is like a database technology frequently is just implicitly considered to be this horizontal layer underpinning everybody that's working on it. So when you try to introduce a different database technology, especially if it's a completely different kind, it's good that they're using Postgres or Cassandra because now you've got a relational database. You've got a document store. You can leverage whichever one works better for you.

But if you try to introduce Mongo or if you try to introduce a vertical store, column store or whatever, one of those databases, that's already served by the Cassandra people. You're establishing a new minimum barrier to entry for that database technology. I'm agreeing that changing the database, usually a bad idea. I've done it a couple of times in my career. You always, always, always run into, you start discovering that databases sort things differently and they always index things differently.

But the second half of the argument is that I would say it actually goes to code because of this contract; that a horizontal change or a horizontal layer is a nightmare because your API is very heavily depended on. You have a lot of dependents downstream and a lot of them are outside your vertical. They're not directly in your downstream. They're in your sideways stream.

Isn't that what APIs are? Isn't that what services are? I would personally, much rather have that layer in there. And then hypothetically, you could if you wanted to replace the whole drugs' API with something that uses a different database.

Yes, yes, and that is actually, the drugs' API, it's a repository service. It's intended to sit on top of a database technology that we all love to hate and we hate to love. We want to get away hermit a single server. [Laughter]

I was waiting for Bob or DB. Anyway,

We want to switch to Postgres and we can't because we've got a ton of store procedures written in T-SQL. So yeah, what we're moving towards in order to… there was this huge amount of friction between the DBA team and the development team. The only peace treaty that could be arrived at was repository services.

That works great right up until you call a service that has to call another service, that has to call a third service because that first service can't just go look in the database for the thing that it needs. Then the customer says, "Oh by the way, that first head service, I need it to return in under 200 ms." And all three of these services are HTTP-based services which can take up to half a sec at a

handshake.

Half a second?

Yeah, okay. HTTP is a high throughput but also high latency. It establishes an entire handshake frame before it's unstuffed across. It's a nightmare.

I want to raise my hand here because you've spoken a little bit about okay, here's how you would start to replace databases by moving things into services. You basically create a more malleable and more friendly horizontal layer, so to speak but why? Why would you want to switch databases and when is it actually worth it?

Because I heard a bunch of usuallys and almost alwayses as far as it's extremely painful to move your data from one database to another and keep the kind of consistency you need to be able to not interrupt service for your customer. So when is it worth it? I mean when is it worth that level of pain and grief? Why would you do it?

David, why do you want away from SQL server?

So every database move I have ever done and the current attempting to move away from SQL server to Postgres has been motivated by the same cause and that's money. And when I say money, I mean money with a capital M. I mean lots of money like six figures kind of money. We migrated away from MySQL to Postgres back in 2006 because Postgres...

I remember that.

Postgres is open source. Well so is MySQL, right?

Uhmn.

No, MySQL is not open source. You are allowed to run a copy of it on your server for free and that's great. But if you deliver the database server to someone and we were doing that, we were helping customers do self-hosting. You have to pay $1,000 site license each time. At least, you did back in 2006. So we switched to Postgres and it was a nightmare.

SQL server, I don't know what we're paying but I'm confident that the accumulated cost is well into six figures. I'm also confident that the annual cost is in five figures and not like small five figures. It's not like 10,000 or 15,000. I want to say it's like 50 or 60. I don't know. Our DBA is just going to laugh at me when he hears me trying to talk about what we're doing with our database.

I just know that we've been trying to get away from SQL server for a long time. There's a second reason which is that we're trying to standardize the development environment and just give everyone a vagrant box to develop on. You flamingly violate the SQL server license agreement if you put SQL server inside the vagrant box that you're giving out to each of your developers. You can't do that.

That's the one thing being driven not by cost. But it is also being driven by cost because we want to stop paying these huge licensing fees for SQL server because we've thrown a lot of iron and a lot of let us pay for extra performance kind of things at SQL server.

That's interesting from a different perspective. I think everything we've talked about so far has been developers wanting to bring in a new tool, developers wanting to bring in a new technology. But sometimes these pressures to change can come from outside of the development organization.

Yeah. Yeah.

So what do you do about those where you don't agree with them or they impact you in a negative way?

Well, you either change databases or you pull 100 grand out of your butt. [Laughter]

And if you're going to take the second option, you might want to stand up first.

We also have moved into a case of talking about eliminating a technology.

That's true.

That's the cost that you have to consider when you introduce one. What's it going to take to get it out? If it's a tool, you stop using it. If it's a language then you need to rewrite some portion of your system, of your code. And clearly if it's a database, you're in piercing pain.

There's a beautiful blog post that came out. I don't know. I just saw it today. Have you seen it by Peter Siebel about 1,000 flowers? Let 1,000 flowers bloom, pull up 999 of them.

Nice.

I like it.

It's beautiful and it is about okay let all these ideas happen. Now, go back and see which ones really work for your organization and start making those work so well that people want to use them.

Yeah. Yeah.

Lure people into standardization.

Yes. That's a good illustration of this trade-off like should we use Memcached or should we use Redis? And the answer is it's a trade-off. I've sorted it out in my mind. So Redis is a proper superset of Memcached. I think it's almost a drop it. Like you change the require and your done. The get and set still work and everything is just fine. But Redis has all of these other things that you can do with it that you can't do in Memcached.

So I almost want to tell people install Memcached first because if you want to go to Redis later on, you can. But if you install Redis first, you're going to end up leveraging one of these extra things. Then if you decide to go to Memcached later on, you're not going to be able to. Okay, show of hands, who switched, right? For who's ever gone from Redis down to Memcached for any reason other than like standardization or like 99 developers in the company are using Memcached and one person is using Redis and that sort of thing. So that might be a case where yeah, you might want to go back the other way.

But I've flipped on this. I've started arguing that you should just install Redis first. The reason why is because if you know Memcached, that minimum barrier to entry is met. You can use Redis exactly the same way. Rediss pays off investment into the learning curve where Memcached doesn't. Memcached is dumb code. It's a dumb key store.

And it's got a minimum barrier to entry but it also doesn't pay-off. If you're going to do any kind of cleverness, you have to do it in the code and that increases your cognitive load. You have to remember that you're building this index of sets in the code now. That means you have to keep it in your head where in Redis, you just say I need it in next of sets, go.

If you are using that Redis feature then apparently you got to settle that Redis feature.

Right.

Well and the other thing is just that in my experience, I haven't found that it's hurt me to be on Redis in the first place. I mean, I haven't found a pain that would make me want to basically downgrade to Memcached.

Right.

Unless there's some used case where it's like yeah, if you're managing these keys in this way on Redis it's going to be 10 times more performant on Memcached then I could see a serious tradeoff having to be made. But I haven't encountered that and I don't know anyone who has.

Yeah. If I did encounter it, I can imagine two cases where I would encounter and they both come from the outside. One is when IT pushes back and says, "We have vetted Memcached but we have not vetted Redis in which case you say, "Well vet Rediss." And then they'd say, 'Well, that's going to cost money."

And the second case is where, now I forgot the second case, crap. But it was actually coming from the business side of things where somebody's actually trying to save money or make money and I can't think of it. If I remember it, I'll mention it. But they're both external pressures. Neither one of these pressures is internal. Yeah, you don't see people saying, "Oh, my Gosh! Memcached, I hate it. You can increment these variables atomically, why? That's dumb. Why do I have to learn this, right? No, you just look at it, it just works and you’re just fine.

Yup.

Does anybody have success stories of when they brought in a new technology and it worked really well?

We only remember the failures. They stand out so much more.

Yeah.

That's true. That's common, isn't it?

I would hark back to Mark Lawrence bringing in go for a tool. I did actually have a success case bringing in Cucumber into a team. All of the developers hated Cucumber because it was so slow. These are the same people that hate RSpec because it's so slow compared to Minitest. Okay for a deuce, once I saw how fast Minitest was I was like, "Holy crap. You guys have a point." But that's [inaudible].

With Cucumber, I was able to sit down with the customer and write out features in a domain language that the customer could read. And that, all of a sudden became like a business reason. So the developers on the team didn't really like Cucumber but the customer absolutely freaking loved it. I mean, he flipped out. And so the developers were like, "Well, okay. I guess we're going to do Cucumber now." We pushed in Cucumber at that point. It made things interesting for everybody else. So maybe this is actually a bad idea, a bad example.

Well I mean, I can think of a few cases. I remember when I came in and worked with you at CrimeReports, one of the things that I came in and had experience with as far as tools that nobody else had really brought in to the system, I'm sure some people had used it before was Jenkins or it might have been Hudson at the time.

Yeah.

I came in and set up continuous integration for our applications. Then that moved into an area where we had a small monitor in our bullpen that showed the status of the projects and things like that. That paid off. Generally, I see the easiest wins in things like tools. Occasionally, I'll see it in libraries where the API is just so much simpler than something else.

Also testing.

Yeah. I haven't had to fight that battle, at least not much.

I just had a huge epiphany on how to take an existing thing and increase the pay-off for investment and reduce the barrier to entry. Can I share?

That's it for the time today.

Yeah, we're done.

Yeah, we're done. We're done.

Time for pick.

Yup. So Chuck actually, you reminded me of this because I remember you coming on board at CrimeReports. I specifically remember I want continuous integration and I want a monitor. We fought with the IT team for like six months.

Oh, yeah. [Laughing].

Before, they would just put a stupid monitor up but I wanted a TV up on the screen or up on the wall that showed us all of our repos and what the build status was on each of them, just green or red to show me that. That's all I care about. I remember Chuck coming in and I remember having the thought, I'm going to make CI Chuck's problem.

It was my problem. It worked.

Yeah and I said that to myself and I was actually thinking, I'm going to make it his problem so that it isn't anybody else's problem. So in other words, we put an API around and Chuck, you were the API.

Yeah.

We put an API around continuous integration so that there was no sideways impact on the other developers. There was one day when Eric and I decided that we wanted to change how the continuous integration was displayed and so we've dove in to the Jenkins code and we ripped into this gnarly XML rendering nightmare thing. It looked like Erubis, embedded Ruby ERB. It looked like that only it was Java in XML. It was a nightmare. We ended up just backing away and saying, "We don't want this feature bad enough to look at this."

But the thing is that after Chuck turned on the lights on this project, the rest of the team never had to learn CI. The only thing you had to learn was how to look at a screen on the wall that all of us could see in the bullpen. So by extension, what I'm saying is if you put an API around it, if you build the tool rather than building a project that has to be maintained by everyone in a new language, if you build a tool that can be used then the language that the tool is written in doesn't impact anybody. They're just using the API.

If you write a library that has all these hairy, weird regular expressions but all of the public methods are well-named and well-factored then you have decreased the minimum barrier to entry to use that piece of code while maximizing the pay-off for using that code. If somebody wants to dive in again but if they don't want to, they don't have to.

I think you've thought of something I've heard of before. I think it was called abstraction.

Abstraction? I thought that was [inaudible].

No,no.

I drove through some of that today.

Yeah.

Oh, wait, wait.

No, that's road construction that you're thinking of.

Yeah.

Yeah, you can abstract in a library in the code or you can abstract something organizationally. So then a person can be the API, can be the abstraction layer or Twitter and Netflix have entire teams for this.

I would point out that you can do dumb abstraction and it doesn't help anybody. It still makes baby Jesus cry if he could read and if he could type. But a good abstraction is like this. Yes, it reduces this barrier.

If you've got a function that takes three other function pointers as lambda or prox and the function is called do it, you probably have a bad abstraction is where I'm going with that. I'm good on you for extracting the three methods but then passing them as method pointers, too with third method to string them together, that's not the best way you could have implemented the inversion of control pattern.

Besides, what if you are really doing functional programming, it would be called F.

Yes. Yes.

Do it is four letters and that's a waste of space.

Yup. Do it is a smell or go or run is a smell in my opinion. The only place I don't get angry about that is if the class is called application and it had a class method called run. I'm okay with that. Even an instance method if you have to instantiate the application and then called that run, that's fine. But I call these methods punch press methods because the caller of the method has to set everything up, has to set all the instance. Then you hit the button and it goes kachunk! And now you reach into the class and you collect all of your output out of the punch press.

I got my team hooked on pronouncing do it because we did actually have some methods called do it. I got my team hooked on pronouncing it doit and just the funny sound of doit made people stop writing doit methods. That's a bad abstraction is what I'm saying. Don't do that, kids.

From now on I'm calling mine work.

Unless you're not getting paid for it in which case you should call it play.

[Laughing]. Thank you.

Unless it's a media player then it should be called work.

So many roles.

[Laughing].

I know. Now you know why I'm so messed up.

How many years have you worked with David?

Enough to be this messed up.

[Laughing]. Yeah.

[Laughing].

I think we worked together off and on, probably totalling like three or four years maybe.

Yeah, something like that.

All right. Well, any other aspects to this we should jump on before we get to the picks?

No, I think I got my rants out.

Yehey!

All right. Well, let's go ahead and do picks then. Before we get to picks, I want to take some time to thank our Silver sponsors.

Coraline, do you want to start with some picks?

Sure. Actually, I want to talk about a couple of projects that I have been working on lately. The first one is Open Source for Women. It's at OS4W.org. Basically, the problem I'm trying to solve with OS for Women is that only 110 Open Source contributors are women. I want to move that number. So I created this online community that lets other women, well it’s women find other women find other women to collaborate with, to pair with, to form mentoring relationships with. It also has a project finder that you can use to locate projects that for example have a code of conduct or that have women on the core team.

So these are in theory welcoming projects that you can feel confident contributing to. Ideally you find a pair partner and pair upon working on that big given project. It's an interesting resource. It's pretty new. We have about 200 users so far and it’s just becoming an active community. It is just for people who identify as women. If you're interested in supporting the project but do not identify as a woman, we have a be an ally page with some ideas on how you can help with that. That's OS for Women.

The other project is one that's then more long term. It's called Contributor Covenant. The problem that Contributor Covenant is trying to solve is that open source is not very welcoming and inclusive by its very nature. And that a lot of projects are hostile to newbies or hostile to people who don't look exactly like the core maintainers. So Contributor Covenant is a way of signalling that your project is welcoming, is inclusive by establishing some baselines for community behavior and what your community ideals are; what sort of things will be tolerated; what sort of things are encouraged; what sort of things are incredibly wrong to do or just kind of wrong to do or things that you frown upon. Basically establishing some community mores and enforcing that as a mechanism for enforcing this.

Contributor Covenant has been adopted by about 5,000 open source projects so far and most recently, Rails adopted it. So if you have an open source project and you want to be welcoming and inclusive to different types of contributors, you should consider that. And I'll put a link to that site in the show notes.

Yeah, I'd also got to shout out at Angular Remote Conf this last week.

Oh, awesome.

Yeah. Kent C. Dodds was doing a talk on how to open source your stuff. So he goes through, "Here's how you set up these different files." And he said, "And you should include..." I can't even remember the term anymore.

Code of conduct.

The code of conduct and he basically showed them how to pull that covenant into the project.

I'll ask them. That's critical.

All right. Jessica, what are your picks?

All right. Well, I just got home from Strange Loop which was an amazing conference yet again this year. I'm going to pick two of the key notes. One is Camille Fournier and she talked about Distributed Systems, a beautiful description of three distributed systems. What we can learn from them. How we can consider trade-offs and make the right choices for our particular problem. Spoiler alert, the answer is to be careful about how we design our data. Be conscious of the tradeoffs that matter to us. There's also a piece in there where she describes the lurking dangers behind a fragmented microservices architecture with lots of different programming languages.

My second pick is the key note from the first day, the afternoon key note by Idalin Bobe' and this one's very different. This one is about what it's like growing up in a poor neighborhood in North Philadelphia and somehow finding her way into tech and trying to bring technology to people who are in circumstances like she had growing up. It's about her life. It's about the life of people of color in the United States and the world today. I highly recommend it to any technologist who wants to expand their own perspective on what we're doing and what we can do. Those are my picks.

All right. I've got a couple of picks. So I had Angular Remote Conf last week, went really well. First one is that I am going to shout out about Rails Remote Conf which is going to be at the beginning of November. So, if you are interested in speaking or attending then you are welcome to. We also have users group tickets which I have made available to both corporations and users groups. So if you have a group of people that want to get together and participate in the conference then do that.

The other pick I have, TV Fool. TVFool.com, that's where I actually went and figured out where to get the antenna or which antenna I needed to get the channels at my house. Basically it showed different colors. So there were green which meant that I could get it without too much trouble. Yellow meant that again, I could get regular reception. And then the majority of the channels were actually red and it listed them as red because I needed a high-powered directional antenna on the top of my house.

So we put the antenna up. They told me what direction to point it in so I just opened up the compass app of my phone. We got to point it in the right direction and hooked it up to all the TVs in the house. So now all the TVs get about 40 channels. Several of them were Spanish which I don't speak or were shopping channels and so I just use the TV’s functions to cut those up.

But anyway, it was super awesome and I'm super happy with the results. So if you're looking at cutting your cable and you want to get an antenna, then that’s a terrific way to go. I think that's all we got. So thanks everyone for coming and look forward to talking next week.