02:34 - Mike Gehard Introduction

03:23 - Monoliths

04:18 - Microservices

09:47 - Migrating a Monolith to Microservices Via an Evolutionary Path

15:21 - The Steps Along the Continuum

19:25 - Bounded Context

21:52 - Evolution

30:40 - Dependency on the Database

35:15 - Testing

51:16 - Deployment

52:47 - Security

55:15 - Advice for those getting started

01:04:09 - Fault Tolerance

Picks

LOWA Boots (Avdi)
Pearl’s Peril (David)
He Never Died (David)
httplog (Coraline)
Toxic Workers by Michael Housman and Dylan Minor (Coraline)
The Unbeatable Squirrel Girl Vol. 1 by Ryan North (Jessica)
Hamilton (Original Broadway Cast Recording) (Jessica)
MyFitnessPal (Chuck)
Run 10k (Chuck)
Pebble Time Steel (Chuck)
Aftershokz Bluez 2 (Chuck)
FLUIDSTANCE (Mike)
The Elm Programming Language (Mike)
Kotlin (Mike)

Special Guest: Mike Gehard.

 

I lost 55 pounds in 2015, Chuck.

Oh, nice.

Wow.

Oh, congrats.

And the… so, ask me how I did it.

How did you do it?

How did you do it?

Poop more than you eat. [Laughter]

Hey everybody and welcome to episode 242 of the Ruby Rogues Podcast. This week on our panel we have Avdi Grimm.

I am a network of tiny people microservices.

David Brady.

I'm back from spending a quarter dead for tax purposes.

[Chuckles] Coraline Ada Ehmke.

Not in fact three dwarfs inside of trench coat. [Laughter]

Jessica Kerr.

Good morning, good morning, good morning.

I'm Charles Max Wood from DevChat.tv. Quick shout-out about JS Remote Conf, Ruby Remote Conf, and Freelance Remote Conf, all coming up very soon. We also have a special guest and that's Mike Gehard.

Greetings from Boulder.

Do you want to introduce yourself?

Yeah. My name's Mike. I work for Pivotal here in Boulder. And I've lately been writing a ton of Java and then talking to a lot of clients about microservices. So, I am kind of excited to chat with others outside of the sound chamber that tends to be the microservice world about what they think about microservices and monoliths and this all fun stuff.

Oh, I'm no help then.

I heart monoliths.

Yay, monoliths.

A bold statement.

I have felt the pain of both, so I'm curious.

Yeah, I am skeptical of large, black monoliths left over from previous space civilizations. I feel they hold danger.

And well you should. Hit it with the bone. Hit it with the bone again.

The microservices bone?

Small bone, very small.

Why don't we get some definitions? [Laughter]

Let's get a definition of a monolith and microservices and then we'll move from there. Anyone, anyone?

Coraline, you heart monoliths. Tell us: what's a monolith?

I consider a monolith a large, single-purpose application with bad division of single responsibility principles between its components. So, things get really muddled. It's a large application. It's very hard to maintain. It's very hard to extend. There's a lot of feature friction and it's not really clear. There's a lot of entanglement between object models, which I think is just a result of bit rot. So, I would say a monolith is a phase in the evolution of a healthy application and I think it's probably a necessary phase. But it's something you definitely will eventually probably need to move past into a microservices or some other alternate architecture.

Alright. If we're talking about microservices, what are those?

So, I would go along. It's another phase in the evolution of an application which I think demonstrates very strong boundaries. Because it has to, because now you've got processes involved and things like that. So, I think it's the evolution or maybe the next phase or a final phase where you can start to put things in really nice, small little boxes and not have that feature friction between the different bits of your application, the different domains of your application.

So, is there a phase after microservices then?

I don't know. I'm not sure we've found that yet.

Nano-services?

Customer service.

[Chuckles] There you go.

It's funny because I've worked on monoliths that we've broken up into microservices. I've worked on microservices that we eventually merged several parts of it back together, not necessarily to a monolith with the same problems that Coraline outlined, but just because we found that the division didn't really get us anything. And so, we put some of the responsibilities back together, even though we still had that neat division, which I guess circumvents Coraline's entire definition. I don't know that I found a perfect architecture for any of the apps that I've worked on. Do you find that microservices get you closer to that, since we're talking about moving from monoliths to microservices?

I think it just depends on the application. I think some applications are really good as microservices. So, one of the theories I've been thinking about lately is the more you know about your domain, the more quickly you can get to microservices. Because you know what the boxes look like. And I think Chuck, as you mentioned, you end up sometimes smashing them back together because you either get them wrong or you figure out that you didn't gain anything from splitting it up. Because at the end of the day, it's still a system. All those parts have to interact at some point to get stuff done. It just depends on how far away from each other they are.

Yeah. There's a…

I think one of the great things about doing microservices though, and you can of course do this in a monolithic app as well, but it sort of forces you to think about your service boundaries and forces you to put in place things that conform to SRP a little bit better.

Mmhmm. And have like…

Yeah.

An API in between the pieces.

Yeah. It's like kind of the ultimate evolution of Single Responsibility Principle, because we have APIs throughout our systems. Even in a monolith, you have interfaces. You have APIs. You have interactions. They're just muddled. You just can't see them very easily and that's where I think that feature friction comes from, is there's rubbing up against each other in funny ways. Now that said, it's a lot easier to smash those pieces around when you don't understand. You know, you're a startup, you're six months old, you don't understand what your business does. It's a lot easier to smash those around in one codebase than it is to be like, “Oh, that service running on that server over there, we need to bring some of that code into this other one.” So, I think people discount the costs of microservices. Because there is a cost. It's not something you should be taking lightly. It's not something that comes for free.

Yeah.

So Mike, does that mean that it might be a mistake to move to microservices too quickly or to start with microservices?

I think so. I think our software as you said, is about pieces and how they work together. And if you move too quickly and those boundaries aren't well-defined, now you have a bunch of poorly designed components that now live on different servers and have higher communication bandwidth and have more cost. The big thing now is cost of microservices. They go from running one application, maybe multiple instances of that application, to running multiple instances of multiple applications, which incurs huge ops overhead if you're not ready for it.

Netflix is kind of the poster child for microservices. Everybody wants to say, “Hey, look what Netflix did.” But not many of us understand the problems Netflix was solving. Like if you look at the Netflix model, the [inaudible] is like, “Why did Netflix need microservices? They serve movies and they take my money.” My guess is that architecture is a lot more complex than we see.

Mmhmm.

So, it's that's lack of understanding of why did they go to microservices that can bite you in the butt if you don't understand the reasons they did it.

Most of us don't have the quantity of customers, the service level requirements, the geographic challenges. Netflix has a lot of very large-scale problems that most businesses do not have.

Yeah. Yeah, they have to connect up to your machine and stream video to you. And they have to do it to all their customers. And if they can't stream video right now to you, [inaudible] go in and cancel your online subscription. So yeah, it's not just serve up a page in two seconds or less and get out.

And they have a lot of money.

Yeah.

Because there is a tremendous amount of overhead to doing things the way they do them. They have teams and teams devoted to tooling. They have teams devoted to writing tools to break things.

Yeah.

So that other teams can test that their tools don't break things.

Yup.

It's layers and layers and layers of stuff you don't want to worry about until your team is over a thousand people.

Yeah.

Yup.

It's got to be so awesome to work at Netflix and have your job just be to test the Chaos Monkey, right? [Laughter]

Like, I test the…

Chaos Monkey to Chaos Monkey?

Yeah, I test the software that breaks our software. Wait.

I can see you doing that, David. Not ridiculous enough yet!

Yes, yes.

[Laughs] So, I know that we're kind of having fun spit-balling about microservices. But our topic here is actually migrating a monolith to microservices via an evolutionary path.

Yeah.

And I'm curious. And I know somebody's already screaming at their phone or their iPod going, “Hey, but I want to know how do I move from monolith to microservices? Because I have all of the coupling and SRP violations and pain that comes with having a poorly put monolith.”

I think it's all about organization. I mean really, it's creating those clear boundaries. So, there's this article that got me started thinking about this. It's AppContinuum.io. It was written by one of our Directors of Engineering at Pivotal. You can literally start your app in a one directory. Like if you had seven files, put them all in one directory. And then move it down a path of structure. And then microservices become like the ultimate structure of your application. So, I think it's just taking time along the way to do some housecleaning and put your socks in the right drawer and put your underwear in the right drawer. It's not rocket science. And at the end of the day, if you can do that then you set yourself up to going to microservices if you need them.

So, somebody else is thinking, “Well, that's easy enough if you're using something simple like Sinatra. But what about Rails?”

So, I would say, and this is going to get me in trouble with the Rails community, that Rails actually gives us a pretty bad anti-pattern when it comes to organizing our applications. So, if you think of microservices, the definition from the Netflix folks is the bounded context. Their domain entities.

So, the way Rails break our app up is it breaks it along technological boundaries. I have controllers and they all sit in one drawer. I have models. They all sit in one drawer. But now if I want to organize that into a microservice, I got to go into four or five, maybe six drawers to figure out what socks go with which ones. Whereas if you actually broke your app down based on domain concepts, which I think Uncle Bob came up with this idea a couple of years back, where if I open up your app directory and I see people, I see users and I see products and I see catalogs, that's going to be a lot easier organization. So, I think it's, yes if you have Sinatra it's easy. But you can do this in Rails. We do it in Spring all the time. I just put my stuff in a drawer with all of the related stuff for that domain concept and not the related technology stuff.

So, is this like putting your pajamas in one place and your casual clothes in another place and your dress clothes in another place? As opposed to a socks drawer, a pant drawer, a shirt drawer?

I guess it depends on how you're organizing it. I think of it as having a sock drawer because when I want socks I go into that drawer as opposed to putting orange socks in one drawer that go with my orange shirts and my orange pants. Because rarely do I wear them altogether. And I guess that's kind of a bad analogy. But it's putting things together that are going to change together, which is one of the definitions of cohesion, is if you put things together that change together then you're good. Where, like my models might change more quickly than my controllers in Rails because I'm implementing more domain logic that doesn't necessarily change my HTTP interface. Whereas when I'm dealing with users I might be churning on that stuff, and that stuff might all be changing at the same pace.

Alright. So, heading down this path of maybe reorganizing a monolith into microservices then. Do I

want to be for example, let's say I decide to pull the user stuff out into its own service, do I then build some kind of API interface to that service? Or do I have it forward-facing so people can hit it from the web? So, they log in through the user service and then they wind up in the ‘everything else’ service?

I think that depends on the usage pattern of the service. Theoretically by the time you've gotten to microservices you understand what your usage patterns are. And then you…

Can you give us an example?

So, if you start out, let's say you start on a Rails app. And you've got everything up and you adhere to my suggestion that you scrap the controllers, models, and views directories and you put users and products in two different directories. You would naturally, you start to break up the domain. So, you wouldn't necessarily have… you wouldn't necessarily be able to ask a user how many products they've, well, purchased. Because those are two… they're in two different places. So, by the time you got to that you'd already be asking the products service, “service” being a little box inside of your app, “Hey, what products do I have for this user?”

So, in your example Chuck, I think you'd log in and you get a user ID. And then you just pass that user ID around to different services asking questions about that from that service what belongs to that user. This is one of the big problems with a lot of Rails apps is that you get this tight coupling. You actually get circular coupling, because I have a user that knows about products and I have products that know about a user. Whereas when I create that 'has many from user to products', I now start to create my god object. I just tie user to everything in my system.

Mmhmm.

And when I want to extract that, now I got to go sever all of those connections before I can extract user out into its own service. Which is usually the first one that goes, because you want single-sign on. And everybody wants single sign-on.

Mmhmm.

And that kind of mandates a user service, because I need to go to one place to manage my users.

You've mentioned one or twice the idea that by the time you get to the microservice stage, that you've hopefully sussed out these different functions. And you've mentioned a continuum. Can you talk about what are the steps along that path, along that continuum?

Sure. So, the first one is as I mentioned earlier I think is just completely unstructured. So, when you're using Sinatra, just dump it on the same directory. It's a lot easier to look at. The next one which we've used is a little bit more powerful in Java, not as powerful in Ruby but still there, is namespaces. I'm just going to start to put these things into boxes. The problem with Ruby is once I dump that namespace into the swimming pool of all of my stuff, now I don't have to explicitly define those dependencies between things, because I don't have to require that file again. It's already in the swimming pool. So, namespaces are typically the next bucket you want to drop stuff into. And the directories help with that. Because I've got a user namespace, I've got a product namespace, I've got a catalog namespace.

And then you can actually start to move them into gems. So, let's put another harder boundary between my system... I'm just going to move it into a gem and maybe I have a product gem and maybe I have a user gem. So, in Java we use JARs. In Ruby you can use gems. And then you can start to share that stuff across applications. So now, I might have two or three applications that need user. And then you just keep doing that. You're like, “Oh, now I have another application and a couple more libraries, and another application, and a couple more libraries.” And you kind of shuffle back and forth between this, putting logic in an application and then extracting it out into a library and then sharing that across multiple applications. You can shuffle back and forth there for a while.

So, we've got our Office Director in Boulder, Stefan Hagemann, is writing a book on componentbased Rails applications. He's actually starting to use Rails engines for this stuff, where you start to literally rip the stuff out into engines.

That was going to be my next question. Where, if anywhere, where do engines fit into this continuum?

Yeah. And [inaudible] I haven't read the book yet. But he and I have been talking about it. And that's, so you're moving harder and harder boundaries between the code. And then eventually when you want a microservice you're like, “Sweet. I have this engine. I'm just going to tear it out of this app, put another fronting Rails app on it,” and then you're done. And that's the benefit of this, is if I can start to create those boundaries, extracting that box is a lot easier, and moving it somewhere else. In Rails we don't necessarily do that, nor do we take the time. And then there are gigantic Java… I'm bashing on Ruby on Rails, but there are gigantic Java apps that do the same thing. We just don't take the time to do the housekeeping and the gardening to keep the paths clean between the rows of vegetables. And they just get all inter-tangled. And now we're like, “Oh, now I got to go out there with a weed-whacker and take it all down.”

So, I imagine through that process, like one of the things you've relied on in your explanation is a knowledge of where those service boundaries should be. But it seems to me like a lot of people, that's an area that's rife for experimentation but also really getting thing wrong. And how do you make those service boundary definitions something where it's safe to be wrong a couple of times?

That's the benefit of keeping them on the codebase, keeping them together closer longer. Because the closer I can keep them, the less cost there is to making that mistake. Because if I'm in the same repo, if I'm just shuffling files across directories, that's pretty, pretty low-cost. But if I go to microservices and [heaven forbid] I'm shuffling them across repositories, that starts to get really expensive and really hard to do. And you just don't want to do it anymore. You're like, “Ah, I'm just going to leave it here.”

Yeah.

And I think one of the tools that I don't quite… I've been focusing on a lot more lately personally is a lot of domain-driven design. So, going back and figuring out what is a bounded context and how to my bounded contexts come up? And what do I need? What questions do I need to ask of the product people, the actual domain experts, so I can start to get that information? Because it's not a technical solution. This is talking to the domain folks. We just end up putting it in small, technical boxes.

Can you describe what a bounded context means?

Oh, a bounded context… so, I'm going to paraphrase it because I don't have the definition in front of me. And it's kind of not really well-agreed on, is things that change together. So, business concepts that belong together when you talk to a business expert. So, I like to think of my bounded context, or the way I identify bounded context is if I'm talking to a business expert and they kind of shift their eyes funny at me and don't understand, then I'm not talking bounded contexts. I'm talking too much technology. [Inaudible] I'm going to have to get the definition. But it's more like, it's the domain objects that stay together and change together.

It almost sounds like Single Responsibility Principle for business logic.

It is. It's very similar to Single Responsibility for business logic.

I'm kind of wondering. You mentioned before with along these same lines that you would put for example your user model in your user's controller and possibly even your user's views into the same directory. So, instead of having like an app models, maybe it's app users. Do you just modify the load path for Rails to pull that in? Like you tell it load files from here too, in your application configuration? Or is there more to it than that?

I think that's it. I actually, to tell the truth, I have never done this in Rails. I used to use Rails as the whipping boy and I would just say, “Hey, Java does this better.”

Actually, the Rails [inaudible] tell me you can do this. And I'm mentoring a friend of mine who's trying to pick up Rails and his homework assignment was to actually implement this. So, when he gets the solution I'll make sure everybody hears about it. My guess is it's super easy, that you just modify the load path. Now I don't know how the class loader would handle that because Rails has this, kind of some of this magic that goes around when it comes to auto-loading classes and reloading classes when they change.

Mmhmm.

But I've been told, the word on the street that I've been told that I have yet to prove is that you can do this in Rails.

Okay. But the concept makes sense.

Yeah, yeah the concept does. And if you look at Lotus, so one of the relatively new Ruby frameworks for the web, Lotus is embracing this. They're really trying to kind of not follow the same path as Rails and try to forge a new path to see if we can write Ruby-based web applications a little bit differently.

Yeah, we did an episode on Lotus. People can go check that out. We'll put a link to that in the show notes.

So Mike, when you talk about evolutionary paths, when I think about evolution I think of several things. And the first one that comes to mind is that things evolve in response to external forces. So, what are some of the external forces that are brought to bear on code that makes it need to evolve?

So, one of the big ones is product requirements. So, Pivotal's a big Agile shop. We do things kind of… requirements coming from products. So, changing product requirements is one of the forces. The other one I think that some touched on was kind of technical requirements. So, at some point Netflix needs to scale their streaming service probably a lot more quickly than they need to scale their payments [inaudible]. Because they're streaming a lot more video than people are signing up. So, that's another reason you might go to microservices. I now have pressure to scale one part of the application.

And the way you traditionally do that on a monolith is you just horizontally scale the whole thing. But now you have waste. Now I've got a bunch of user code that's running in four different places when I don't really need all that user code floating around in the world because my users are, you know, I don't have a ton of sign-ups on a regular basis. So, there's both product and technical pressures that can come to bear on a codebase to force it to move. And that's why I love that point, is that you're not just pushing it down this evolutionary path to push it down the evolutionary path because microservices are cool. You're allowing things to force you down this path. I'm a big proponent of don't do it unless you have to. So, it really works really well for me personally because I'm always, I don't want to have any more stuff laying around than I need.

So basically, due to some of these external forces, it just gets to the point where the pain of dealing with it the way it is becomes greater than the pain of actually moving it off to its own service?

I think so. It's a cost/benefit analysis. So, the cost of going to a microservice has certain cost and the benefit is we can stream more videos.

Mmhmm.

And I think that's the hard piece of this is that you have to be very aware of what your code is telling you. Is the code pushing back when you try to scale it? Most people will be like, “Ah, I'm on Heroku. I'm just going to scale this thing horizontally with the click of a button and we're good.” Where if they had taken a little time they'd be like, “Wow, we've done this four times. Should we step back and say is there something else we can do to make this a little easier on ourselves so we don't have to run this app across multiple servers?” And with Heroku, it makes it easy. It's a click of a button. But if you're running your own servers, there's more cost to that.

So, the second thing that comes to mind with evolution is change over successive generations. Do you think that microservices lend themselves to rapid change more easily than a monolith does?

And I'm pretty sure I know the answer to this.

[Laughs] Serving up the soft ball. I think they do, because now you can start to have teams that work on the simple microservices. So, if I need a user change I can go into the user microservices and make that small change and not have the risk of other things changing, having to deploy other code. So, I think it does allow you to move faster at some point.

And that gem organization probably helps with that quite a bit, except now you have to manage versions of the gem across a suite of services.

Yeah, yeah. So, the first thing you do is you go to unbuilt gems. So, one of the things we've been doing on these big Rails and Ruby projects is just leaving them as unbuilt gems. So, you're never under versioning them because there's no need for sharing them. And then you pointed out, as I start to need to share those now I have a cost of managing versions. But if that cost is outweighed by the fact that I can have a whole team working on this or I can version it once and not have to think about it because it's not changing as quickly, that could be a benefit to going to those built gems and having different versions.

You talked about team structure needs to change in response to that desire for changing code quickly.

Yeah, it's the whole Conway's Law. So, [inaudible] familiar with Conway's Law it says that the systems you build will be a direct reflection of the communication patterns in your organization. So, if you've got a very tightly-coupled organization you're going to naturally tend to write tightlycoupled software. So, if you want to change your organization to be more flexible, you actually start writing software that's more flexible, which will then force your organization to become more flexible. But microservices yeah, because they're loosely coupled it allows you to put teams on them. But it also forces you to do that.

If you're thinking about microservices as products, that's a lot more healthy because now you have a team responsible for that service. And as long as that service is running, that team is responsible for keeping it running. If you think of microservices like projects where you have a team come in, write something, and then they hand it off to somebody else and go work on something else, that gets dangerous because now you've just offloaded that thing. And the team that wrote it's not responsible for that anymore. So, it definitely forces organizational structure and change and kind of can be stressful if people aren't ready for that.

Before I move on to the third part of my question, you talked about unbuilt gems. Can you define what you mean there?

So, unbuilt gems are simply gems that sit in the same repository and are required using the file path in Bundler as opposed to using a Git or an actual built gem on RubyGems. So, you can just… I think it's the file method where you get a path to that just in the project.

So, that would be under like vendor or lib?

Yeah. You'd typically put them in, we start them in lib and then we move them to vendor. Those are kind of semantics that you can hash out between teams. I don't really have a… there's no hard and fast rule of where those things should live.

Oh. So, the last thing that I think about with evolution is that evolutionary processes give rise to diversity at every level in a system. So, does that imply that when you move to a microservice path you're opening the way to for example polyglot architecture?

Yes, very much so. So, very much so. Not only in language the system is written in but also data stores. So, Facebook Friend Graph is kind of a perfect example of something that is pretty wellserved by a graph database where orders and products and things like that are much more wellserved in a just traditional relational database. So yeah, you open yourself up to be able to now hide the implementation details. Now with that come costs as well. And now somebody decides they want to write Haskell, now you're supporting Haskell services. I think, is it 6Wunderkinder did this approach? I think they've got multiple different languages in production because they've gone to this microservice-based architecture.

Yes. They encourage polyglot and Haskell in particular at 6Wunderkinder.

Yeah.

Do they also encourage like the person who's responsible for that service to be basically responsible end-to-end including deployment and administration and all that?

Absolutely, absolutely. I would argue that if you don't have that in place already, microservices are going to eat your gut.

Yeah, that makes sense.

Yeah. Yeah, having that frictionless deployment. Look at Netflix, they just open-sourced, what is it called, Spinnaker which is their continuous delivery pipeline. So, they have this super automated system that will take code from Git commit all the way through multiple environments. And they just open sourced it. If you don't have that kind of tooling, yeah forget about it. You're going to be managing these things and it's just going to be way too costly.

I do want to call something out rather explicitly and that is that the contract between objects in a monolith is the API, the public method list essentially, the public API for the class. Whereas when you move things out to a service, then your public API is a JSON API that runs over HTTP or some other protocol. And so, that's why you can diversify underneath. You can diversify the technologies that you're using including the database, et cetera, which is something that we've said before. But I don't know that we necessarily explicitly called out the fact that because you've moved the way that things talk to each other up a level, you can then freely move underneath it as long as you maintain the contract you have at the outside.

Yes, or XML. Let's not forget where we came from.

[Laughs] Says the Java developer. I mean [coughs]. [Laughter]

I have a question that's kind of related to that, having to do with some of the pain points that I've seen people run into when breaking services out. When you hit that transition between the components that are all still within the same application and then moving them out into services, one of the unwritten, unspoken dependencies that is often usually there is the dependency on the database. So, you have these two separate services that both want, are expecting, implicitly expecting to talk to the same database tables. How do you deal with that?

Yeah, so that's the hardest part of the split typically. Because when you have one app you have one database. And I haven't come up with a great way to figure out who owns the migration. So, if we're going back to Rails, who owns the migration when I need to change a database table? The one rule that we've used very successfully is one migration changes one database table. The reason being that if I have multiple database tables that are being changed in a migration, I now need to split that migration in half because one table goes to the right and the other table goes to the left. Now I have to figure out how to unwind that migration.

So, I don't have a great answer for that. I'm going to kind of punt on that. Because that is the hardest part, is I've got this coupling at the database layer that I can't have across microservices. I can't be coupled at the database layer because then I don't have different rates of change. And I

haven't come up with a foolproof way of doing that besides a lot of blood, sweat, and tears.

Okay. So, you scared me for a minute there.

I thought you were going to say it was okay for them to continue to share the same database. But no, I think what you just said was that one migration per database table because later you will need to split those so that each microservice can have its own database, right?

Yes, yes. You do…

Okay.

Do not, do not want to be sharing a database.

That is the most insidious type of coupling you could ever have in your system. [Laughs] Thank you for clarifying that.

[Chuckles] Yeah, in my opinion that's the biggest differences between microservices and SOA, the old Service-Oriented Architecture. It is not about [this]. Really, it's about the hard separation especially at the database. Because if you are sharing one database between two services, this is like two mouths going to the same stomach and it's disgusting. [Laughter]

The other thing is that…

I wanted to see where you were going to go with that. I was like…

I just threw up a little bit in both of my mouths.

Exactly. [Laughter]

I did want to point out though, the reason that it's so insidious is not just that you have two things talking to the same data store and therefore you don't know where the changes are coming from necessarily. And even if you have great auditing, the other thing is that if you have a clear separation on your service it's really easy to sneak something in that reaches into the part of the database that the other service is supposed to be in charge of. And so, you wind up having that same problem except now it's not supposed to do that strictly speaking but it does it anyway. And we're going to kind of hand-wave over it. And so, if you have that hard separation then you're not even tempted to go, “Yeah, but I only need this little bit of information out of the products table.”

Yeah.

I'm kind of crying inside because somebody I know who I won't name names because I wish to remain anonymous, but somebody I know right now is refactoring some code that was written admittedly, time to market was a factor. And so, they leveraged the technical debt. They incurred it like racking up a charge on a credit card. And yeah, there's SQL code in the test suite that reaches into another service's database and inserts data and manipulates it to set up the functional specs for the one repo. That's bad, right? The full-body convulsions that my friend is having, that's an appropriate response right?

Well, sometimes it will work. But in my opinion you're heading down the path to where you're going to do something that's going to be really painful. And your data is kind of canonical in your app. And so, if you goof it, it becomes that much harder to deal with because now you have three different things that you have to look at to figure out what goofed it.

Yup.

And those tests are going to break because somebody else changed their database structure which is frankly none of your business.

Yup.

That does bring up an interesting point though, and that's that testing in a microservice environment is really, really hard.

Yeah, yeah.

Yes.

Yeah. Let me throw a specific question out for Mike or for anybody on the team. Is it viable to create a route or an endpoint on the API of a service just so that another service can access it during the test suite?

So at Stripe we have a test environment. It's called production.

Yeah.

And I don't mean…

I've heard of that one.

I do not mean our tests. I mean our customer's tests. So, all of the Stripe APIs have a little flag, live mode, that says, “Is this production or is this you're testing?”

Mm.

So yeah, we totally have modified our APIs for other people to be able to test with them.

Okay, that makes sense.

I go back to the age-old question of do you test private methods? Because I look at those back doors as little bits of private method that are hanging out.

Yeah.

And if you… and sometimes it's okay to test private methods. If you've got… again if you have to test a private method I would probably say that you need to refactor it out. But sometimes you have to. You're like, “I need to get this thing under a test,” for some reason. You know, Kent Beck in a tweet once said the answer to any good engineering question always should start with “It depends.” You just never know. That's I think… for me lately as I work with kind of aspiring developers and clients, everyone wants the kind of cookbook. Like, what do I do when this happens? I'm like, “Well, it depends.” And then they're like, “Okay.” And like [chuckles]

And that's hard, it's really hard because we're all trying to move really quickly. But it takes a lot of awareness to be aware of what you're watching for and then be able to react to it. If you're not aware of what you're watching for, then you can't be faulted for ending up down the wrong path.

Can I poke this a little bit? Because, so what Jessica's talking about is that they expose all their APIs and then you just use… I've used Stripe. I'm assuming a lot of us have. But Stripe, you effectively use test keys instead of live keys and then they go into a test database. And they act just a little bit differently. I think that's different than actually adding endpoints to a service so that you can run specialized tests against it.

Right.

You can add testing behavior to a production system with its current APIs so that you access it and you tell it, “Hey, I'm doing test things and it does test things under the hood.” Adding extra endpoints, especially if they circumvent anything, is in my opinion again asking for trouble. If it's not something you're going to expose on your production system, then it's probably not something you want to expose on your service at all. And if you need to look under the covers for example to see into the database to see that something got stored properly, then you can do some kind of end-toend testing with the test harness that has access to the database in a testing environment where that happened. But adding extra endpoints just for testing just so that I can do whatever I'm going to do, it seems really iffy to me.

On the other hand Chuck, you brought up the example of making sure something got into the database. When I'm testing a microservice from the outside which in some ways is actually easier, testing individual microservices for themselves is sometimes easier than components in a monolith, if something got into the database my preference is strongly not to test did it get there but to test the consequences of it being there. And sometimes…

Mmhmm. Yeah, does the behavior change?

Yeah.

So, behavior-driven instead of data-driven or whatever?

But sometimes saying, “Okay, how do I know that it's there?” leads me to add an endpoint that lets me test whether it's there. And that test, that endpoint is not… yeah, okay, maybe I was driven there by testing. That is not for testing. That's for visibility.

Yeah.

That's for being able to see whether this is working and I'm going to use the heck out of that when I'm debugging in production. I do think those are worthwhile.

I'm generally suspicious of code that you add to support testing. But I think that in this case I would make an exception, Jessica.

Mmhmm.

Yeah. I think it's driven by testing but that's not really its purpose. Its purpose is visibility, is transparency, is the radical honesty of you microservice.

Yeah, but you're not…

Radical…

You're not adding any special behavior to it. You're just opening up some visibility so you can see what it's doing.

Right, give you a way to check its work in production as well as in tests.

Yeah. I mean, if you look at a test as just an automation of an exploratory test you might have done manually, then you can see that stuff is just revealing the points where you needed introspection, you needed visibility. You need to surface information as an explorer. And exportability is one of the things that any software artifact needs to have in order to be useful.

Yeah. I think you have to be careful though that you don't give visibility into something that doesn't mean anything to the business. Like if… the method that this bit of SQL could be removed to replace actually manipulates the state machine of an object, or manipulates an object through the state machine in the foreign service. And the way you would actually do that in the real business world is by performing actions on that object through the remote service, which yeah, you're starting to test something that's outside your application. But if you provide visibility into the state machine you are now almost making your state machine implementation public. And now if you want to get away from the state machine, you may have trouble because you're pinned down by extra visibility into that application, when the customers don't care.

Oh, absolutely. Let me give an example and see if you'll agree with me. Let's say it is something like Stripe. Let's say you're processing transactions and you want to test some transaction that you trigger. And the normal way that's currently supported to find out that a transaction succeeded is the system sends you an email. So, you could set up some sort of elaborate testing system where there was a test email account and it would sit there just waiting until that email arrived to verify that the test succeeded. Which if you were doing that manually that would drive you insane. That would drive you bonkers. And you would eventually go after the implementer of that service with a pitchfork.

Or you could have a service… you could say, “Hey, maybe we need to add a service that enables us to say what is the current status of this transaction?” And that's not… what you're not doing

there, you're not saying “Let's add a service that lets you execute an arbitrary SQL query.”

Yeah.

Mmhmm.

What you're really doing is you're making a process visible. And that process is part of the business domain that the customer is interested in, the user is interested in.

Yeah. Actually, this kind of touches into an interesting question that is definitely applicable to SOA. And I think microservices almost exacerbate it. But there's thing that you end up wanting to test. And then the stuff that I'm manipulating right now is our actual functional test suite, which you should be able to run it against production, against the whole thing, as a black box. Which means you can't stub out a service. You can't stub things out. You basically have to bring up the whole system and all the services at once. And what we're testing is that the system acts coherently with the services collaborating with each other. Where do we test collaboration?

I think you've kind of hit on it, is you're bringing the system up as a whole. So at some point, you need a test suite that exercises the whole system.

Mmhmm.

And that's very much, as you would with a single Rails app, you'd write Capybara, maybe a Selenium-based Capybara tests to do that. And that gets expensive. You don't want a ton of those. So, you want… yeah, I think the nice thing with microservices, there's a library called Pact, P-A-CT, which was developed at ThoughtWorks to test microservices-based architectures. And they actually define a contract between microservices in JavaScript, or in JSON. And then you can take that JSON document and run it in the test suites of either end, both producer and the consumer, to test that the consumer's expecting a contract and the producer is still producing that contract. Which may minimize the need for those kind of full-system tests, because you've now shored up the contracts. And one or another test suite will fail if that contract has been violated, which may reduce the need for that.

So, I think you could [try] to test those boundaries. And those boundaries become real, legitimate test suites that have to be agreed on by both sides and tested by both sides.

Yeah, yeah.

I do want to get into this a little bit and just point out, I love how David said, so do you just spin up everything? Like just spin up everything? Every microservices setup that I've set up?

[Chuckles] Just a [inaudible], isn't it?

There's no 'just' about spinning up everything. I mean, you know, I've written little scripts that start them all on different ports. And even that's not perfect. I'll kill this one, kill that one, start it up again. That's a mess. Okay, well maybe we'll use Docker or VirtualBox or whatever, or Vagrant. And sometimes, with Chef Solo, okay well that will spin it up. But you're adding all of these extra things onto the pile just to get it up and running.

And now you have a test that consumes too much memory to run on your CI server.

Right, or on your development machine. Gee, it'd sure be nice to spin this whole thing up in one place. Yeah, it just doesn't work anymore.

I don't know. If your crap doesn't fit into 16 gigabytes, you've got too much crap. [Chuckles]

I've heard more and more people take this in a direction where they say basically, we have to stop expecting that we can test the whole system integration on one machine. [Inaudible] at some point.

Okay. So, at that point then…

I don't know Mike if you agree with that or not.

Yeah, I think you're absolutely right. At some point, you have… and this is the benefit of microservices. If I can't change my microservice independent of other microservices, I'm not doing microservices. I'm doing SOA. So, the ability to have these interfaces change at different rates is super important, because you don't want that coupling.

Well, I think there's also a point of diminishing returns when it comes to testing these systems with all the services running together in some sort of sandbox environment. Because I mean, the kind of issues that you're most going to want to diagnose with those kinds of tests are the kind of issues that you're never going to be able to properly reproduce in a sandbox. They're the kind of issues that are just going to pop up running on your production systems with your production latency, your production net splits, your production who knows what?

Yeah. It's hard to automate testing of yanking the Ethernet cable out of the back of the cart.

Welcome to distributed systems. [Chuckles]

Because now you have a distributed system.

Yeah.

And those are hard.

See Jepsen.

Yeah.

I cut my teeth on distributed systems in a different context where they were all part of like an air traffic control system. So, they're all just little boxes talking to each other, not internet machines. But that was one of those situations where it was, you could not test them in isolation, the whole thing in some sort of isolation sandbox. You had to bring it up and not only that but they had periodic self-tests build into. And it wasn't just a matter of testing it once and then shipping it. They would, every part, a piece of every second that the system was up would be dedicated to testing itself, or every few minutes would be dedicated to testing itself.

I want to just jump back a little bit, because somebody mentioned something to the effect of if your services are talking to other services you're doing it wrong. To what degree is that true? I mean, if you have some centralized system that people are hitting that's kind of a web surface frontend kind of system that serves all the requests and it talks to the services, then does it just make the request say to the user system and then make the request to the products system and then make a request to some other system to get the information it needs? As opposed to having it kind of go ask the user system for information and have the user system pull it in from the products system. Because I've seen that too, where they have services and then the different services are talking to the other different services to get you all the information you need or do all the work that you need done. Do you have to coordinate that differently in order to avoid some of these issues?

I think that might have been me. And I think I may have misspoken. Your services can talk to other services. They just can't be dependent on the rate of change of those other services. So, if I want to release a new user service I should be able to do that pretty independently of the products service.

Okay.

Unless the products services, like I'd added an API endpoint or I'm getting ready to remove an API endpoint because I've deprecated it. Because now you have to communicate.

Mmhmm.

So, not only your systems have to communicate but now your teams have to communicate. “Hey, I'm getting ready to make this change.” If I can't make that change without getting four other teams in a room, I don't…

Then why did you split out the teams in the first place, right?

Right. My bounded context might be a leaky. I have a leaky abstraction. I've not abstracted away from those teams enough where I can change independently of them.

So yeah, to answer the question you asked, if we want to go that route now and stop me if we don't want to get down this path, is there are multiple ways of doing that. So, you can either have kind of an orchestration piece where you have one conductor that conducts the whole orchestra. Or you could have services do their own thing, they know what they need from other people. It just depends. We get really into that world of 'it depends'.

Alright. Coraline, what was your… you said you had another testing question. Let's go back to that.

Yeah. This may be because I'm bad at TDD but I tend to do testing first in a REPL and then I capture the results of that exploratory testing in a spec. What is the REPL of microservices?

Curl.

I wish I was…

That was my reaction, too.

It totally is. It totally is.

Postman. Postman if you like GUIs.

Yeah.

Postman. I don't think I've heard of that. Link in the show notes. We'll get it in there.

Yeah.

So, I mean is it just testing the service boundaries via HTTP? Is that just what you do?

Yeah.

It's what I do.

Yeah.

Yeah, totally. What happens if I send it this? What happens if I send it this? Curl.

Yeah. I've actually started fleshing out a serializer that will dump an object not to JSON but to a Curl query. [Chuckles]

So that I'm basically saying, okay this object. It has 75 fields on it. Let me just create one from this factory. Okay, cool. Now, just dump out the data blob that you would post from a form. [Laughter]

I do use Postman quite a bit, actually. So, it was kind of a leading question. I do like yeah, I do like the fact that I can parameterize everything in a form and I can save queries if I want to, reuse them later, and test them again later. So, that's a pretty cool tool. I've also seen, I forget the name of it, but I've seen an API testing tool that basically gives you a UI for each and every one of your endpoints in your defined API and lets you interact with them by filling out a form and hitting a button.

That's pretty cool. Yeah, I guess… so, not only… this actually speaks to story acceptance as well. So, now that we're in this world of Curl and APIs how do we do story acceptance? How do we allow a product owner to do that? And I think these approaches of giving tooling to people so they can more easily interact with those is super important. Because you want another set of eyes on that API.

It's I/O Docs. That's the system I was trying to think of.

So, what about deployment? We talked our way around it a little bit. Everything kind of at this point could have different dependencies. So, is each team responsible for their own deployment?

Yes.

You say that like somewhat resigned.

No, I think it has to be that way because I am a huge proponent of owning your own mess. So at some point, I have to own the ability to get this stuff into production. Because if I'm not feeling the pain of how hard it is to get into production, then like tests, I'm not getting the full picture to do that.

If you're not living the pain of deployment, there is somebody on your infrastructure team who wants to kill you.

Yeah. And you have to make it easy. So, commercial coming. We have a product called Pivotal Cloud Foundry which is on premise, Heroku. And our whole goal is to make that easier because you want the developers to be pushing and you want them to feel the pain. You want them to get the feedback. But if I have to hand this service to somebody else, not only am I not feeling the pain of that or the joy, it might be so easy that it's a joy, but I'm also now dependent on somebody else's schedule. So, if my ops team is super busy and I'm handing off services, how can I do continuous delivery if I have to wait on another team? That's kind of like external QA in Agile.

Yeah.

So, bringing those whole teams together where you've got design, you've got product, you've got development, you've got ops altogether, now they're like a little SEAL team. And they can just do whatever they want and get stuff out as quickly as possible and not depend. They don't have external dependencies on other teams.

The other question I have, Jessica asked a question similar to this. But this is something that I run into. And that is some kind of access control or authorization. She mentioned security, which is I guess somewhat of a different concern. But you have many services that handle many different types of data. And you've got some kind of overarching 'Can this user access this data?' or 'Can this user change this data?' And that's a concern that all of the services share. So, do you set up another service for that? And then is it kind of a dependency that no one sees except for admins?

How does that all flow together?

Inside of Cloud Foundry, both the open source and the closed source bits, we have a tool called the UAA, User Authorization and Authentication. It is another service. It's an OAuth2 provider.

Mmhmm.

So you have, you know, you have another spinning process that is going to handle that for you. And it kind of depends. I don't know how Netflix… I know Netflix, my understanding of the Netflix architecture is that once you're inside the titanium eggshell, so once you've made it past that first layer onto the network, their services don't care as much. There's still 'Can this user do this thing?' but the services don't have to say, “Hey, I'm a trusted service. I'm allowed to call you.” But if you're a bank, my guess is that is not how that happens. So, you have… I would think you'd have this other layer, which you have in a monolith. You have devise or you have cancan. You have these gems in there doing that for you. They're all kind of internal. Now you've got a distributed system that's going to… you know, somebody's got to handle that for you.

So, they just check in with your authorization service and make sure that you have the right credential?

Yeah. You get a JWT token. And you can kind of pass that around. And there are multiple ways of authorizing who gets to do what. It just depends on what you put in your data UT token. Again, it depends.

My clients hate it when I say that, by the way. [Laughter]

Especially when it comes to how much it's going to cost them, probably.

And a JWT token, is that an OAuth token or is that something else?

JavaScript or JSON Web Token I think is what JWT stands for. It's a standard that is independent of OAuth. It is something layered on top of OAuth.

I have a question about another potential pain point that these projects can run into. And that's sort of… we talked about the database and this is sort of on the other end, the UI end. Let's say we're just talking about web applications. I feel like I've seen these go down one of two roads where on one hand you have basically each service providing its own UI. And in that situation it seems like it's really easy for UI differences to creep in. And it starts to become obvious that you're dealing with a different service because it's idiosyncratic and slightly different.

And then there's the other way that they go, is the services are all just exposing JSON endpoints or something and then you have one thin UI application that talks to all of them. But there the issue that I see come up is now whenever you add a feature you have to modify the service that's responsible for the feature and the UI. So, you're modifying at multiple levels which is one of those things you try to avoid by splitting things across functional lines. I'm curious. Are those basically the only two routes or is there a third way that I'm missing?

So, when you say expose their own UI, do you mean actually serve up HTML?

Yeah, yeah.

Okay. Yeah, one way to avoid that is just have them all serve up JSON. So, that's kind of the easiest way. Because now, the contract is simple JSON. And I don't have to really worry about it as a backend developer what color that thing ends up being or how it's being presented on the screen.

Mmhmm.

It also makes it more flexible for mobile. So, iOS and Android are always the wild cards in these things because if you're doing native mobile you don't really want to be serving up a ton of HTML.

Okay. But let's say I'm on a team that's implementing a really exciting new feature and we're implementing it as a service because we've already got a fully-fledged microservice architecture and we can move fast because we're implementing our own service instead of having to coordinate with a big monolith. But we're supposed to expose just a JSON API. So now, when I want to quickly iterate on the user experience for this new, exciting feature, does that mean I'm not bottlenecked by the team that's responsible for that thin UI layer? Or how does that work?

I would say that you would probably put a UI developer with the backend folks. Because if you're treating that new feature as a product, you'd have a complete team. So, you'd have product, you'd have frontend, you'd have backend, you'd have ops.

Mm.

You'd hammer that through. So, maybe if you're pairing you could have the frontend developer sit with the backend developer and kind of bounce back and forth between what they work on. [That's hard]. Those vertical slices through the whole system become really costly because it requires everybody. Whereas if you're just putting a new UI over existing services, maybe you don't need it.

You don't need the whole team to do that.

But would you still prefer that architecture over the one where the new micro-app is actually exposing its own pages or no?

Yeah, I probably would. Just to keep it consistent, to keep that separation between backend and frontend. That comes with a cost, though. Because if nobody is every going to reuse that API then I now have created an artificial separation. And that's why if you can build these little monoliths, maybe you just build that as like a little monolith. So, you haven't fully adopted microservices architecture. [That team] serves up HTML for a little while. And then another team comes in and says, “Hey, I'd love to use your HTML but oh by the way I need that on an iOS device that I need JSON from.” And maybe the next iteration is to split that API out.

So Mike, if you have to give advice to someone who's just getting started on this path, what would it be?

Keep it all in one Git repo as long as you can.

[Chuckles] Yeah.

Leverage tools that the languages give until you're really clear as to what the bounded contexts are. Don't be so quick to force the evolution down. Let it happen and be aware that as you're feeling that friction, it's telling you something. And it's moving you down this continuum from everything in one directory to fully-blown microservices.

So, when you're talking about moving things down into… so, keeping it all in the same Git repo, do you move everything into one directory that's like core app and then you move the piece off that your… users maybe, so that goes into a users folder? Or does it go into users folder under the core app? Or how do you…?

I'd start with the users. So, I'd start… kind of the evolution I have is start in one directory, like that's the app directory. Now I try, I break out directories inside of that application directory as long as I can. I want to keep them as close as possible. And then when I feel some sort of need that is dependent to the situation, I'm going to move that somewhere. Typically it's reuse or something like that. I'm going to move it out and I might move it into a lib directory. I might move into a vendor directory. And then at some point I continue down that path, moving the pieces farther and further away as I see the need for those harder and harder boundaries.

Okay.

Yeah. That's the big trade-off, right? Is that by separating out microservices we gain freedom to manipulate things intra-service, right? Inside the service we've got all the freedom to do whatever we want. But we've done it at the cost of creating an output boundary on one service, stringing cabling over to the other service and creating an input boundary on that service. In other words, there's a much more expensive interface now between these two objects. And that's the trade-off, is that if we want to change the inter-service cost of things, or if we want to change the interservice behavior the cost of that is now much higher. And we've just leveraged that. We've just, we traded it in order to get intra-service speed. Okay, that's fine. We just have to remember that's what we traded.

Yeah. And being aware and conscious of that decision.

Yeah.

Yeah, because someone brought up SOA. I mean, what we're doing now is really not that different than SOA, the big difference being this idea of bounded context. I think that's where SOA got it wrong, is we put too much. We put… the cables were too smart.

Yeah.

The Enterprise Service Buses were too smart. Now we're moving it closer to that and we're stringing dumb CAT5 cable between them that has no intelligence. But this is no different than CORBA. This is no different than, conceptually no different than a lot of these technologies. We're just kind of reinventing it with more smarts.

That raises an interesting question. And maybe we should have asked this at the beginning of the show. But what do you see between SOA, Service-Oriented Architecture, what do you see the difference between SOA and microservices is? Because I'm sitting here thinking that SOA is just microservices that can't be rewritten in two weeks. [Laughter]

I think it's that intelligent pipe versus dumb pipe.

Yeah, my experience was definitely that. One of my first projects, we split everything up into Service-Oriented Architecture. And then the ESB, the Enterprise Service Bus immediately became the pipeline because whenever anything changed, something needed to change there at the pipeline. I meant the, what's the thing that makes you slow?

Bottleneck?

The bottleneck, yes.

Yeah, because then nobody could make a change until that team was available to change its smart [tweakiness].

To be fair, the bottleneck is also where the lovely liquid comes out. [Laughter]

Yes, yes.

Yes.

Yeah. The other mistake was we all talked to the same database so there was no freaking point in breaking it up. We just did that so it would look good and they would pay us by the hour.

Yeah. Mike, you said that with SOA the pipeline is smart. Do you literally mean that I can put something… I can make a remote asynchronous call with data X, Y, Z but the receiving service is going to get X2, Y2, Z2? It's actually going to get differently transformed data?

Jessica's nodding. Yeah. That's what happened in SOA is you had this Enterprise Service Bus where the services ended up being dumb CRUD interfaces to databases. They were kind of HTTP interfaces. And then all the logic was in this bus that was shared by everybody and understood by nobody. And you didn't have that, “I can deploy X service without affecting anything else in the system.”

Yeah.

And it was tons of money was made doing that. And that might be okay. If that works for you, that works for you.

Okay. So, [chuckles] Chuck is pinging us in the private chat that he needs to wrap soon. So, I have one more softball question for you, which is fault tolerance. How important is it? Is it vital or would you say it's more essential? [Laughter]

Do you need it or do you really need it?

Yeah, exactly.

I'm formulating my response.

Yeah. I admit to an agenda behind the question, yes. [Laughs]

Can I just clarify? When you say fault tolerance, what you mean is “User service. I need to know about this user. User service? User service?”

Actually, more than that. More than that. What I mean is, “Hey, product service, I need to know how many products I have on order.” And product service immediately comes back with, “It's cool man. But hey, I'm going to give you a 503 bad gateway because user service is down. I can't actually fulfill your request because this other service that I depend on is down.”

So timeouts or explosions or what have you.

Mmhmm. Yeah, and essentially what I'm saying is if I get a 500 in the user service, the product service shouldn't automatically start throwing 500s. Basically it's almost like a dependency on the error state, like? It's like “I'm now dependent on this thing throwing errors,” which is bad I think.

I am going to [say] absolutely necessarily. Just like functional programming where if I have an option type I'm forced explicitly to handle the error case, I have to handle that. Because the last thing I want is a 500 going back to my user. So, yeah.

Yeah.

Because you're in a distributed system so the CAP theory applies. But not part… you don't get to choose partition tolerance.

Yeah.

That's going to happen when you have microservice because the network will go down. You also, if you want five nine's, if you're running a hundred services think of how infrequently one of those would have to fail for the whole system to keep five nine's or four nine's, whatever you need.

Yeah.

So, you have to bulkhead this stuff. You have to stop the whole system from just cascading failing across the whole system. So, yeah.

I remember when we had Daniel Jacobson on from Netflix. He talked about that. If the recommendation system was down, you couldn't bring all of Netflix down. You had a fallback service that you could go back and use.

Yeah.

Mm.

Like well, what's popular?

Yup.

Yup. And Netflix in their open source, they have a think called Hystrix which is a circuit breaker pattern, which is this distributed systems pattern where, hey if I make a call and it doesn't respond, I can give it a backup response. And then it's just stuff… because as you said David, if one service… it might not be a service that you're talking to that's down, it might be four down the path.

Mmhmm.

You're just seeing it now because you just happen to talk to that service and not directly to the service that's down.

Yeah.

Plus, if you don't bulkhead that, the system will probably never come back up.

Yeah.

Because if you keep hammering those services they're just going to continue to fail. Or you have to build exponential back-off and all that fun stuff into all your services.

Yeah. But even if you have exponential back-off, one time in a thousand or one time in 10,000 the system goes down and doesn't come back up.

Yeah.

Right? I mean if you get the perfect storm, you just need a more perfect storm to knock the system down. But this is computer science, right? One in a million is next Tuesday.

Mmhmm.

Yeah.

Yup. If you're working at high scale, it might be tomorrow.

Yeah. I do want to nitpick a little bit that if a remote service returns a 500 and it just blows up, it just completely dies, and returns a call stack, it's a very… the service is actually misbehaving. Your service, it's okay to return a 500 class error. In fact I would think it's, I would say it's actually appropriate because a 400 error is usually, you're telling the user that they've screwed up and they need to change their request where a 500, a 5XX is a server-side error. And you're telling the user “You didn't do anything wrong. Just wait. There's nothing you can do to fix this problem on your end.” But I would like it to come back as a clean bit of JSON out of my application that says, “Hey, you know what? 501 not implemented or 502 bad gateway,” and my app is still up and I'm not going to blow my brains out and show you a call stack that I blew up because a dependent service died.

Mmhmm. I would even say that's even a little bit loosey-goosey. I would say the user would never know that that 500 happened, because it's not their fault.

Yeah. Well, the question is do you…

You know, you should return them like “Sorry, try later.”

Yeah. Oh yeah, that makes sense. And that actually would be a 200 OK, wouldn't it?

Mmhmm. As Coraline said, like, “Here, I can't get you your recommendations. But here is some canned list of recommendations of movies that I have for you because I know this is going to happen.”

Yeah.

Because the user is going to, they're not going to know what to do. Like you said, it's not their fault.

Why do they even need to know what happens? It's happened.

Yeah.

Alright. Well, let's go ahead and get to some picks. Avdi, what are your picks?

I don't have a lot this week. I think I'll just pick the hiking boots that I got a few weeks ago. I got a pair of LOWAs. And after trying on a few at REI. And I got to say, they're some of the most comfortable hiking boots I've ever had. And they've performed quite well, at least in the last few hikes I've taken them on. So, that's really all I have this week. Hopefully I'll have more next time.

Alright. David, what are your picks?

I've got two today. And I'll try to do them quickly rather than my traditional 20-minute pick sermon. The first one is an iPad game called Pearl's Peril. I love hidden object games. And I've really come to learn that hidden object games as misnamed. They shouldn't be called hidden objects. They should be called objects which are in fact in plain sight but that you cannot actually see. And the fun thing that comes out of these games that makes me crazy is when a hidden object game will cheat. They'll actually hide objects out of sight. Like they'll say 'Find the umbrella' and all you can see is the curved handle on the umbrella. And when that happens I'm like, “That's bull crap.” You can't actually hide the objects in a hidden object game.

And Pearl's Peril, it actually turns out, I didn't realize it but it's almost three years old. It's actually a fairly mature iPad game. And it does have in-app purchases and I had to go in and disable in-app purchases to save my marriage. Because I really, really enjoyed Pearl's Peril. Where the objects are makes sense. Like if it says 'Find the light switch' it's going to be on the wall next to the door, you know? If it says 'Find the boots' they're probably going to be on the floor. That kind of thing. And so, it's really fun. And when you can't find something, 'Find the bull', okay, well I don't see any cattle anywhere. And then finally you know, where is it? Where's the hint? And then it shows you. Oh, there's a cave painting on the wall that has a picture of a bull. When the game tricks you, you go, “Oh, that's clever,” instead of, “Oh, that's bull crap.” So, Pearl's Peril is my first pick.

My second pick…

David, David.

Yeah?

If you really enjoy finding boots, I would be happy to employ you every morning here. [Laughter]

Finding six pairs of boots.

Yeah, yeah. Well see, I'm naturally a hoarder. So, finding anything is like a hidden object game. Like I walk into my office and I'm like, “Okay, which stack is this going to be in?”

But what if you only see the umbrella handle, David? Do you get really angry?

Yeah, oh yeah I do. I totally do. It's like if I see just the end of the keyboard sticking out I'm like, “That's bull crap!”

“Who put that there?”

Yeah, exactly. Well, I know who put it there unfortunately.

So, my second pick just real quick is a movie called He Never Died. And I love this movie. It joins a short list which previously just contained The Sixth Sense and The Usual Suspects as movies that I watched and then immediately rewound and watched again and saw a completely different movie the second time through. I'm not giving anything away that isn't in the trailer for the movie, but the main character, somebody asks him how old are you? And he says I have no idea. But I'm in the bible if that means anything. And he's Cain from the bible. And he's tens of thousands of years old. And the first time you watch it he's depressed and neurotic and clueless and disconnected because he's immortal and he's just bored. And then you get to the end of the movie and you realize he's not clueless. He's totally clued in and he's trying to… he's not disconnected, he's distracted by continuously trying to restrain himself by all of the things that he's learned and come to know by being immortal.

And so, the second time you watch it he's this very powerful person who is dedicating all of his resources to restraining himself. And it just makes for a fantastic movie. So, I highly recommend it. He Never Died. I think it's free if you have Amazon Prime. I think it's free to rent. I could be wrong. But if not, it's definitely worth the five dollar rental. It's lots of fun. Kind of fun to have a main character who's pretty much indestructible. It is an intellectual movie. The ending is pretty anticlimactic. It's different from Sixth Sense and Usual Suspects. There's no twist at the end. You just get to the end and the meaning of the thing that they've been telling you the entire movie suddenly sinks in. And that's what makes it so beautiful to me, is that there's no twist. But it has the same slamming impact as an extremely twisty movie like The Usual Suspects or The Sixth Sense.

So, those are my picks.

Coraline, what are your picks?

I've got a couple of picks today. The first is a Ruby gem called httplog. And surprisingly it logs outgoing HTTP requests. It logs to the Rails consoles through standard out. It works with Net HTTP, faraday, open URI, httparty, all the usual suspects. When you integrate it, you get a log of the outbound requests, the response status, a benchmark, and the full response in raw format. It currently only works with GET and POST but that's usually enough for you needs. So, I was actually debugging interaction with a third-party API and found this gem. And it is amazing what that information that it provides can do in terms of helping you figure out what's going on when something's going wrong. So, httplog is my first pick.

My second pick is a paper out of Harvard Business School by Michael Housman and Dylan Minor. It's called 'Toxic Workers'. They point out that there's been a lot of research on developing and discovering top performers in the workplace but less attention's been paid to the question of how to manage workers on the opposite side of the spectrum, those who are harmful to organizational performance. So, they explore a large dataset of over 50,000 workers across 11 different companies and analyze what aspects of workers' characteristics and circumstances led them to engage in what we call toxic behavior. And they also explored the relationship between toxicity and productivity and the ripple effect that a toxic worker has on his or her peers. So, their findings, I'll just summarize. The value of a superstar is estimated between 2,000 and 5,000 dollars in cost savings. But the cost of a toxic worker is estimated at over 12,000 in lost productivity.

Mmhmm.

So, it's a really interesting paper. It's available for free and I'll post a link to it. And I definitely recommend to anyone who has to work with people.

My favorite quote in that line is a manager that I had years ago who would… he said “My definition of gross negligence, or gross incompetence is when I can look at somebody and say you know, if nobody was doing your job we'd be better off.”

[Chuckles] Oh, man. Alright. Jessica, what are your picks?

Okay. Still coming off of Christmas break. So, I've got two kid-related picks. One is a comic. It's from Marvel and it's called 'The Unbeatable Squirrel Girl'. And it's got a great heroine and it's really funny. And my favorite line so far is, “I'm Squirrel Girl. I'm not achieving consistency across database systems girl. Yet.”

Nice.

So, it's pretty awesome. I recommend that. My second pick is a musical. I haven't even seen this yet. I'm just listening to the soundtrack and it's pretty amazing. It's called Hamilton. It's about the autobiography of Alexander Hamilton. And it's a great way to listen to some cool music. It's mostly hip-hop. And also learn about American history. It was brought to my attention by my 10-year-old daughter who has learned one of the songs in one of her classes at school. And it's pretty awesome. The end.

Alright. I've got a handful of picks I'm going to throw out there. This is the time of year where we start making New Year’s resolutions. For me it's more kind of a planning process for the year. It's not something where I just sit down and go, “Yeah, it would sure be nice to lose some weight and make a little more money.” I really start thinking about how I'm going to get that done.

Last month in December 2015 I watched my dad go through some pretty horrific health issues. I don't know if I've mentioned them on the show or not. But I'm heading down the same path. He's 60. So, he's not that… he's 25 years older than I am or 24 years older than I am. Anyway, however that works out. Anyway, so it really kind of frightened me because I looked at it and I was like, “Man, I really don't want to have these problems.” He's got hip problems because he's been overweight his whole life. One of them resulted in a hip surgery that went bad so he kind of drags one leg when he walks. He had open-heart surgery last month. And then I've been taking… I had to take him to dialysis a few times because he's got kidney failure.

Wow.

He has type 2 diabetes, as do I. So, there are these things. He has osteoarthritis which is directly caused by his diabetes. And I just look at his quality of life at age 60 which really, if you think about it, isn't that old, especially where people are living well into their 100s, 110 in some cases. I just, I don't want to be there when I'm 60. I want to be healthy and things like that. So, one thing is just gaining that perspective. This year my main focus is actually going to be on being healthy. I've made some changes. I did a Periscope a while back on my standing desk and you can check that out. And I'm actually standing up while we record this. I stand up for all the podcasts and for several other calls that I do. However, the main things that I’m going to pick are the things that I'm using to focus on my health.

I've listened to several experts and read several sources. And it appears that at least for diabetes, one of the main things is how you eat. And losing weight, your diet is critical anyway. So, my first pick for tracking that is MyFitnessPal. That's MyFitnessPal.com. I've used it before.

Oh, yeah.

I'm sure I've picked it on the show before. But I'm using it and I'm deadly serious about it.

I've also been working out at the Lehigh Legacy Center since I live in Lehigh, Utah. And I've been using an app called 10k app for my iPhone. Now, I use this setup with my Pebble Time Steel Watch. And so, it actually pops up the notifications. It vibrates on my wrist when I'm running. And it doesn't interrupt my podcast or my music when I'm listening by telling me “Run now.” It just vibrates and I look at my watch and make sure that it looks like the right kind of message and I run or walk, as is appropriate. And I'm going to be running a 10k this year. I'm also going to be running a half marathon toward the end of the year.

Awesome.

Is where I'm heading with that. So, the 10k is going to happen by May and the half marathon by November. Anyway, so I'm going to pick the Pebble Time Steel watch. The Pebble Time Steel watch also just added health tracking. So, it tracks all my steps and everything else, tells me when I'm active and stuff like that. So, I'm loving that.

I also have Aftershokz Bluez Bluetooth headphones. They're bone-conduction headphones. And so, that's the other thing is half the time when I'm running, I don't know why but I guess I have these massive thumbs that like to catch the cord to the regular Apple ear pods. And so, when I run I wind up jerking them out about three times. So, it's nice to have it all wireless. And then if I need to control the music or anything, I can either tap the button on it or I can tap the button on my watch and stop or start music. So, that's kind of the setup that I've got going right there.

I want to start tracking some of the other information like my weight and things. I'll probably wind up doing that in MyFitnessPal. If you want to be my friend on MyFitnessPal, you can connect with people. My username on MyFitnessPal is woody2shoes. So anyway, I'm happy with my…

With the decimal number two or T-O?

Decimal number two, yeah.

Yeah.

Woody number two, shoes. And yeah, I'm in this to win it. Incidentally a few other decisions I made, because know some people are interested in these kinds of goals, I have quit caffeine about six times. And it's hard. But I just kind of quit with whatever. This time I'm going for a year. I've quit for three months is the longest I've gone. So, I'm going to do it for a year. So January 3rd next year, I can have a diet coke if I want one.

Nice.

So anyway, so that's kind of what I'm about. I know this is kind of a different flavor for picks. But that's what I'm doing. So yeah, so those are my picks. Mike, what are your picks?

So, I'm going to continue along with the fitness standing desk movement from a company called FLUIDSTANCE. They have a thing called the Level. A friend of mine started a company, was doing a lot of standing, noticed that his joints were getting really tight. So, while he wasn't going to die or whatever, sitting causes you to die. He felt like his joints were getting tight. So, he started a company, built this thing called the Level. It's kind of like a little surfboard. And you stand on it and it allows your body, it forces your body to move just enough where it doesn't disrupt your typing but at least gets you some movement in your joints. So, you're not standing with your knees locked or like I like to do, stand on one leg for long periods of time. So, that's FLUIDSTANCE.com. It's called the Level. They're kind of fun. I've been using one for about a week.

The last two I'm going to go back to the geek world, are two languages that I've been playing with. So, 2016 at the beginning of the year I usually pick up a couple of new languages that I haven't been writing. So, one of them is the Elm language which I love and I've been really enjoying writing some functional reactive programming in the browser.

Yay, Elm.

Because I'm not a huge fan of JavaScript. [Laughs] I was waiting for the shout-out.

Yeah, you'll find some fans here.

Yeah, totally enjoying what it gives me. The compiler is super friendly. It's actually compiled. So, I've been building some little toy apps for that.

Another language is called Kotlin. It's from the guys over at IntelliJ and RubyMine. And it is a kind of better Java. So, they're using it to write IntelliJ and RubyMine. It gives you some functional programming paradigms. It's type inferred. So, a lot of niceties and things that I want in Java these days but I don't get, even with Java 8 I wasn't getting them. So, I started playing with that. I'm moving some of my Spring Boot applications from Java 8 into Kotlin which has been very interesting. So yeah, Kotlin, if you're looking for Java that's not Java, pick it up. And that's all I have.

Awesome. If people want to know more about other things that you're working on or whatever you've got going on, what are the best places to do that, Mike?

Twitter is usually the best place to get a hold of me. So, my Twitter handle is just M-I-K-E-G-E-H-AR-D, mikegehard. Yeah, reach out to me there. I've got a lot of big thoughts going on. If you really enjoyed some of the things I mentioned about microservices, I have a talk that I give that I'm trying to shop around to a bunch of conferences, not only in the states but in Europe. Because I think we're at this point now where we want to do microservices, we want to have good apps. And this talk really gives you, talks more about this, actual steps you can do. So, if you have any conferences near you that you think, “Hey, a talk on microservices, an evolutionary approach would be great,” please reach out to me and let me know. I'm just kind of trying to figure out where I can go and give this talk.

Alright. Well, submit a talk to Ruby Remote Conf. Plug, plug, plug.

Alright. Well, we will go ahead and wrap up the show. And thank you again for coming, Mike. It was a fun discussion.

Yeah, this was fun.

Thank you. It's wonderful to be on the show.

Alright. We'll catch everyone next week.