RailsClips is officially launched!

03:11 - Michael Nygard Introduction

04:36 -  Release It!: Design and Deploy Production-Ready Software by Michael T. Nygard

07:15 - Upfront Architecture

14:29 - Architecture Without an End State (Definition)

18:42 - Beware Grandiosity

22:45 - Context

23:52 - Holding Teams to Standards

26:27 - Architecture Between Groups

29:16 - “It’s not my job” (Developer Responsibility)

31:45 - Design Artifacts

37:55 - Staying Humble

39:43 - Distributing Economic Decision Making

45:51 - Tools and Technologies That Are Helping

48:45 - Future Book Plans?

Picks

Michael Nygard: Architecture Without an End State (Avdi)
Daily Tech Video (Avdi)
October CincyRb - Jim Weirich on Decoupling from Rails (Avdi)
Rachel Shadoan: Why Algorithm Transparency is Vital to the Future of Thinking (Avdi)
Avdi Grimm: A review of news summary services (Avdi)
Systems Thinking, Third Edition: Managing Chaos and Complexity: A Platform for Designing Business Architecture by Jamshid Gharajedaghi (Jessica)
William Byrd: The Promise of Relational Programming @ PolyConf 15 (Jessica)
Again (Coraline)
Seveneves: A Novel by Neal Stephenson (Coraline)
remoteconfs.com (Chuck)
God (Chuck)
Jesus Christ (Chuck)
The Church of Jesus Christ of Latter-day Saints (Chuck)
Cory Doctorow: The Internet of Things That Do What You Tell Them (Michael)
The TCP/IP Guide: A Comprehensive, Illustrated Internet Protocols Reference by Charles M. Kozierok (Michael)
services-engineering (Michael)

Special Guest: Michael Nygard.

 
CORALINE:

East Coast storms are massive and destructive and have lots of thunder and lightning crashes and the sky turns a nasty green right before they come off.

Right.

Like politicians.

Hey everybody and welcome to episode 217 of the Ruby Rogues Podcast. This week on our panel, we have Coraline Ada Ehmke.

Hello from Chicago.

Jessica Kerr.

Good morning.

Avdi Grimm.

Hello from Tennessee.

I'm Charles Max Wood from DevChat.tv. Really quickly before we get going, I have two quick announcements. The first one is that I launched RailsClips so you can go sign up at RailsClips.com. The second thing is, is that I've done some in-person training for a few companies and I'm wondering if anybody else wants it. So if you are struggling getting started with your testing or you have a whole bunch of legacy code that you're not quite sure how to get test around then send me an email, chuck@devchat.tv and I would love to talk to you. We also have a special guest this week and that is Michael Nygard.

Hello. Glad to be here.

Can you give us an introduction really quickly?

Sure. I've been a developer and architect for more than 20 years now and took an odd turn into operations in the early 2000’s and lived in the apps world for a while guaranteeing the availability of applications I didn't write. So it sounds a little bit crazy. It was a crazy time but I learned a lot about operations and started trying to bring that back into the dev world most particularly with the book called ‘Release It!’ which I'm happy to say has been well received and a lot of people have taken to heart. So very happy to bring back that learning.

These days I work for Cognitect. We're the company behind Clojure, ClojureScript, Datomic, Pedestal, and a host of other interesting open source libraries. And our vision is all about once again changing the way we do development by using the sharpest tools possible to get a lot done with high leverage.

I think it's funny it's always been your fault. You were developer and it was your fault. Then you were in ops and it was your fault. And now you're writing open source tools so it's still your fault.

Absolutely our fault. Yes. It's true. Really interesting to be woken up at three in the morning because there's NullPointerException being thrown from some bit of code that you're not even allowed to see the source to.

Ouch.

And somehow having to get the system back online.

[Chuckles] That's not interesting. That's just infuriating.

I want to jump in before we go too far and just be a total fanboy for a second.

Thank you so much. Thank you so much for Release It! It is one of my all-time favorite programming books. It's one of those books that I tell people to go read all the time. It is one of those books that I refer back to regularly. I recently did a little miniseries on RubyTapas about the Circuit Breaker Pattern. Just incredibly important stuff in there. Thank you so much.

Oh, wonderful. I'm glad to hear that you've gotten use out of it and enjoyed it. I'm actually really gratified that some of the concepts have diffused enough that they're not even being spoken off in conjunction with the book anymore. You talk about circuit breakers and that pattern has now grown way beyond the book and is in pretty wide use. So I'm really gratified to see that.

Can someone summarize what the book is about and how this Circuit Breaker Pattern marks in.

Basically, the whole premise of the book is that software design has mostly been taught as a way of passing functional tests. That's to say we learn in school, in college and from other books how to make the software do what it's supposed to do. But we never really learn how to prevent it from doing the things that it's not supposed to do. We don't learn how to answer those non-functional requirements like availability, like the ability to upgrade.

So ‘Release It’ was really about how do we approach that? How do you design software so that you can make it past release 1.0? The analogy that I always go back to is having a child. I've got four kids and the nine months leading up to birth were an important time but it certainly wasn't the end of the effort. Most of the effort goes in after that. But we tend to treat software projects like the release date is the end. We move on after that. But the release date is actually the beginning of the software's life.

So in a nutshell, Release It is about how do you design software to survive after that 1.0 mar?

When you designed this software to get past 1.0, does that effort take place before the initial release?

Oh, yes. Many of the things that I talked about, they're not expensive or difficult to do but they are the type of thing where making some decisions early helps a lot. Otherwise you end up retrofitting a lot of things. And you're spending time and effort backfilling things that could've been decided differently earlier when your sponsors would probably rather be exploding the market and learning from feedback from users and incorporating new features and so on. So it's definitely better to get some of these things taken care of earlier.

It's really interesting because there's such a knee-jerk reaction against doing upfront architecture these days. Everyone thinks at Agile, an iteration is the way to make software but I've seen a lot of projects that could have benefited from some upfront planning especially with the architecture.

Yeah, I'm glad you brought that up because I'm an Agile developer since before we even have the term Agile. I remember when we called them lightweight methods. I don't think there is any inherent conflict between good design and Agile development.

When you're doing Agile dev, you're thinking about design all the time. You're doing design with every story, with every card you play. I consider that the same thing as architecture at different scales. So you can do agile architecture. You can introduce it incrementally and you're much more likely to succeed at it if you have a vision of what good looks like in that area. Just like understanding what good looks like at the method or function level.

Where do you get that vision of what good looks like? From your book?

Well, that's certainly one place. I think reading code is great. I think learning from existing systems is great. There's a series of books on the architecture of open source applications. I love those books. Even if you don't agree with the architectures, you at least learn a lot from how other people have solved problems.

I think that's something we don't do enough of as an industry and that's revisiting history, revisiting projects that are existing and are successful in trying to understand what makes them work. I think in our field, we have a lot of reinvention going on.

We need some of that reinvention in order to find the new stuff but it seems to spend a lot time.

I think of a jazz band where some of what they do is playing standards but every standard that they play has an improvisational aspect to it. I think that you can make progress in advance to feel that way by building off of the things that have come before and then innovating around them.

I think a lot of the innovation can occur when the context changes that makes something old relevant again or relevant in a new way.

When you think about the plummeting cost of memory and of storage space, it makes certain approaches more interesting that weren't before. Cloud computing makes certain things interesting. You can often find this sort of critical ratios where we pass a threshold. For example that you can stream a movie down the network in less time than it takes to watch the movie. That makes certain things possible that weren't possible before.

Oh, you're right that that it seems like a quantitative increase but at some point the quantitative change becomes the qualitative change and the whole game is different. Like with the streaming movies, that's a great example.

Michael, you pointed out that old ideas in new context result in innovation. I like that because sometimes in programming communities, there are complaints about, "Oh, that idea's been around forever," be it functional programming or languages like Erlang and Haskell. It's like yeah, sure, the idea is old but the way we're using it is new.

I had this moment a few years ago where I sat back and I said, "Wait a minute. Tons of money is being poured into batch processing systems - Hadoop® and big data. I'm working in a language that was invented more than 50 years ago and using an editor that's 30 years old. How did this happen in an industry that fetishizes novelty? It is because I think some ideas keep coming back in different forms like the way that Bohemian Rhapsody is popular every 10 years.

Then some ideas just never really went away but apply in certain context better than others.

What are some of these fundamentals you think that are persistent across, let's say, architecture?

So when I think about architecture, I really regard it as being all about time. We do a thought experiment and say, "What if you could answer any programming request in a day? Start from scratch, no code, nothing and you build the system that's needed in a day? Would we ever care about the architecture of that system?" Well, no, clearly not because you could simply throw it away and build a new one in a day.

The reason we care about architecture is because we expect things to live a long time and we expect them to change. So there are certain immutable values why simplicity and flexibility, they really don't go away regardless of what the programming tool is or how much is the code you run on the server versus the client.

Well, the other thing that's interesting is that as our applications live and change and go on for a long time, I mean, they increase in complexity and they change their concerns. So architecture is a way not just of thinking about the problems but a way of organizing all of that information so that we can either have a good way of organizing the complexity so we can understand it or to actually mediate some of it.

I think it's quite essential to have a good architecture upfront because it paves the way for future enhancements.

So by a good architecture upfront, now we're getting back to the - when do you do architecture? Do you do it as needed with a particular story or do you do it upfront? Michael's talk at Craft Conf this year was about Architecture Without an End State. There is no final architecture.

Earlier, Michael you mentioned that you can do this internet Agile style if you have that vision of what good looks like then you can move toward that vision but then that requires your team to have a shared vision. Does that establishment of a shared architecture vision happen upfront?

I think it's the type of thing that begins as seeds in the very early days of a system. You have some notions of the way that things should be. You communicate those and you talk about them and the ideas change as you discuss them and work through them with the team and the stakeholders and everyone else. As you progress, you learn more about what the system wants to be and what the environment is that it's going to be immersed in.

So when people talk about doing upfront architecture, it always sounds a little bit like, "We're going to have an architecture phase and once we're done with that, the architecture's done and it doesn't change again. That's a very dangerous idea.

But the notion that we start with a partially formed idea of what the architecture is and we gradually build more clarity around those ideas as we work forward, I think that's actually a healthy way to do it.

There's certainly not a time when you sit down and say, "All right, now I'm going to code the architecture and when I'm done coding that then the architecture's in place." I think the architecture exists as much as a set of ideas about patterns of interaction and communication ensured understanding within the team as it does artifacts and deployable units.

Now, I just want to clarify something because I haven't watched the talk. What exactly do you mean by architecture without an end state?

So I've been in quite a few large companies and there's this common recurring pattern where you'll get a change of leadership because IT leadership tends to turn over pretty quickly. The new leadership brings in a new vision for what the architecture should be. If we're going to start building everything in this way it comes along with diagrams and pictures and market value assessments and all of these ideas. You spend the first year in proof of concept or prototyping. Then the second year is like vendor selection. Then in year three, you're going to build it and after that you'll receive all the benefits because you reached the end state.

But it turns out the average life span of a CIO in a Fortune 500 company is like 18 to 24 months and the plans tend to be three to five years. So you can just look at that and see you're going to get a regime change before you reach that end state. So what we actually have in every organization that's been around for a while is not one architecture or even one architectural style. You have a sort of stucco layering of all the past efforts and the current efforts that are on their way.

So in my talk, Architecture Without an End State, what I'm really trying to convey is don't build this five-year plan that arrives at Nirvana at the end of it. Build something that's delivering value now that you can continue to build and continue to deliver value in the future. Continue to morph and change and mutate.

I've got a set of rules or guidelines that I go through in there to say "Okay, so it’s a great sounding idea to build so we can keep changing but how do we actually go about that?" That's the main body of the talk, is how do you go about it?

So I have to then jump in as the confused listener and say that it sounded like you and Coraline were both advocating that upfront design or upfront architecture is a good thing. So in this case, it sounds like you're saying, “Don't go do all the upfront design because you're never going to arrive.” So is it one or the other or is there some happy middle ground?

I think what I'm trying to get at is that architecture is an ongoing process and an evolving set of ideas that get reified into code and into systems. But I don't think you ever finish with it. Like when is design done in an application when the application is dead? When is architecture done in the organization when the organization is dead? Out of business or acquired.

But we also have the idea that we want to make decisions early on that leave us the greatest amount of optionality in the future. We don't want to make decisions that box us into a very high cost of change a couple of years down the road. So thinking early, implementing continuously, and rethinking continuously is really what I would say.

Make it about the journey because the destination's going to change as you're going?

That's definitely a lot of it. We're also kind of jumping scales back and forth here. Architecture Without an End State is aimed more at the organizational level or the idea of a system of systems. Then we also have to talk about architecture within an individual system. And you can even talk about architecture within subsystems or bounded contexts within a system.

So I would say Architecture Without an End State, it is aimed at those larger scales and I also think you need to think about architecture and design continually as you're implementing systems at every scale.

Within this cycle of innovation, there is certainly some consistency in the architectural vision over time, would you agree with that?

I do like some consistency, yes. But like everything else, there's a balance where at some point, the desire for consistency needs to bow to the requirement for change. So maybe it's like a tension building up in a fault line. Either you take a large number of small slips or you have a massive fracture when the tension gets too great.

That makes sense.

You said something in your talk that really struck a cord with me. You talked about how a lot of organizations have an ideal of coming up with a single unified architecture which includes things like a single system of record and a single model of their whole business. Like an enterprise dated definition dictionary or something, global model, some kind of thing like that. You said something about how maybe that shouldn't be the ideal. Can you expand on that?

Sure. I think I actually titled it Beware Grandiosity which is a bit of a grandiose title for that section anyway. I can make the center core remark that there're two types of programmers. There are the optimists and then there are people who've been on an enterprise dated dictionary project.

[Laughs].

The problem is that all of those projects tend to be deft to context. So I've been in tons of situations where I’ve got multiple groups all using a word like customer or order and it turns out they have very different ideas attached to those nouns. We're using the noun as a single symbol to represent a complex of ideas. But we can't actually make those complexes of ideas mesh and match between say a sales group and a customer service or product support group. One of them wants to keep tracking people as soon as you learn their name and the other one wants to only be aware of people who've actually not only paid for the product but paid for support in the product as well. That's just the life cycle mismatch. That's one you can paper over with sufficient numbers of state fields.

But you can also have cardinality mismatches. What's the cardinality between account and user? One part of your business may say that a single user is an account. Another part of your business may work with NGOs or B2B and say an account may have multiple users with different levels of privilege.

If you actually try to satisfy all of those desperate needs with a single model, you either end up with the least common denominator which satisfies no one or you end up with something that is so bloated and difficult to work with that nobody wants to use it. And you have to force them to use it by decreeing it as an enterprise mandate.

The other thing about those projects is they tend to go for a very long time in modeling before they deliver anything. And so, two things happen. One is some projects just stall waiting for the enterprise model to get done. Other projects race ahead and end up with something that then has to be retrofitted or just get an exemption or a waiver to continue to exist and not conform to the standard. I think we're often better off by decomposing those concepts, actually splitting the ideas up into smaller and smaller ideas.

So when we say a customer, what kinds of things come to mind? Supposed you're doing a retail operation or you're selling media items. Well, a customer may be someone who's providing payment and may be someone who has a contact address that you can send notices to; may be someone who consumes the media or downloads the media or gets the license key to a media. There's no reason that all of those things have to be bound together into the same database table or same model object or same controller. You can actually split those out and then recompose them in different ways for different contexts.

So if you're working with individual users, you have a one to one relationship between email address, account payment, and media. But if you've got group accounts or users groups or something like that, you can have a many to one relationship. Now by decomposing these concepts and pulling the relationships out of the individual entities and tying them together at a higher level, I get much more flexibility in the future. Each one is simpler to implement and it's less likely that I've got some grandiose project that's going to spin on for years without delivering anything.

One of the things I got out of one of your blog posts was it sort of got me thinking about contacts. You talk about things that are billable, things that are supportable, things that are marketable. It sounds like context is more about adjectives than nouns. Would you agree with that?

Yeah, in a way. I think that's the segregated interface principle. One of the solid principles says that the interface I am interested in is only based on what I need to do with your object. So don't give me an interface that has 98 methods on it. Give me an interface that has the five methods that allow me to do the thing I need to do.

And yeah, your object may end up implementing 10 or 12 of these tiny, little faceted interfaces. But that's way more flexible because in the future, I may receive a different instance that still lets me do the thing I need to do and you're free to change your implementation. You can apply exactly the same idea to services and micro services and create segregated interfaces on those. And when you look at what do I need to do with it - yes, those names tend to be more adjectival than nounlike.

You mentioned standards and the difficulty of holding teams to standards and how that can be destructive. But yet, if in a larger organization such as this one that I'm in right now, if we want to encourage teams to use a better architecture and we've got a lot of different experience levels across the IT organization and different backgrounds, and we want to encourage them to use a better architecture without ramming it down their throat, how do you do that?

Oh, that's a very tough question. Because now we're not talking about the technology, we're talking about teams and team interactions.

Architecture comes down to organizations, right?

Yes, it does. True. Fair point.

One of the things I've observed is enterprise architecture groups tend to be understaffed relative to the size of the dev groups that they’re meant to support. I do think of it in terms of support. What I would prefer to see is teams using the best tools and frameworks because there are people to show them how and because it's the easiest and best way to get things done.

So nobody has to decree that AWS is going to be the standard for startups doing websites. It just is because it's the easiest and fastest way to get things done. If you can make your architecture within the company such that the easiest and fastest way to get things done is also the way that you would most like people to do them then you've definitely won. But there's still that outreach part, that notion of going and sitting with the teams. Lean people call it a Gemba walk where you go down and you see what's happening at the point where the people are doing the work.

The trouble is most enterprise architecture teams aren't staffed large enough to be able to do that. And so, when you're overwhelmed and you can't go and work with each group individually, you end up retreating to writing standards documents to try and get leverage to scale enough. I think that takes you down a bad pathway.

Thank you. You just totally described exactly what we're trying to do - make it easy and provide outreach and individual coaching and learn from the different groups and exactly why we can't do it.

Well, I was hoping to offer more of a positive path than that but maybe have your management team listen to this podcast and then give me a call. [Laughter]

There you go.

Exactly.

I have a related question. You talked about an architecture team working with other groups. Does this include other groups like development teams? In other words, I've worked in organizations that have architects and then they have developers. So a lot of what we're talking about here gets assigned to the architect even though the developers are ultimately responsible for implementing the architecture, so to speak.

So where do the different jobs separate or where do they overlap, I guess is a better question. And then the other question is do you need an explicit architect in order to have good architecture?

Oh, boy. Complicated set of issues wrapped up in that question. I was embedded in a company about 10 years ago now, where the architects were explicitly told not to write code, that you're too expensive and too valuable to be spending your time writing code. I actually know someone who is formally reprimanded, had a letter in his file because he'd been caught coding.

Oh! [Laughs] Sorry, I couldn't help it.

That is amazing. It is amazing.

I'm mind blown, I mean really. We want you to design software but don't do software.

Now, I have encountered some architects that I might want to prevent from writing code. [Laughter]

That's an anti-pattern of its own. Neal Ford was fond of saying that in many companies, the title architect means post technical which is also not a compliment.

Sad but true.

You could turn it around and call them a technical post as well.

Ouch.

But anyway, so architects.

Obviously, I don't think much of those structures. I think architecture is a role not a title. Like a lot of roles on an individual project, you may have more than one person filling that role. Hopefully you don't have zero filling that role. You need at least one person with that skill set. And I do think that you need to have one person who's finally accountable for that skill set and that work getting done. So there may be multiple people on the project with architecture skills. They may all be contributing but eventually you need a way to resolve disputes and come to a consensus or to some coherent solution.

You also need a point of contact so that people outside the team who are talking about coordinating architecture know whom to call.

Yeah, I totally agree. So, yes, I think it's something that needs to be present on most teams to varying degrees. People who've done more of it hopefully, take an interest in coordinating architectures across multiple teams, multiple projects in groups.

So I want to kind of remove the excuse that I know that some people have and that is that it's not my job. So what responsibility do developers have for architecture especially if they have somebody who's assigned to think about those issues?

Well, first and foremost it’s the question of understanding but I want to be careful. Again, we are using this single word ‘architect’ or ‘architecture’ and we each have our own complex of ideas about what that means and so we have to be careful about it. We've been smoothly jumping between the ideas of enterprise architect, application architect, architecture as artifact, architecture as idea, architecture as interactions.

So to unpack it just a little bit, I think that developers on an individual project have a responsibility to understand the desired sets of interactions and how those interactions – and I mean interactions in the code and in the system – and how those interactions produce the emergent qualities of the non-functional requirements.

So to get concrete for a moment, if you're a developer on a project that makes outcalls to a whole bunch of services and you're meant to be using a circuit breaker and you're meant to be using say Netflix/Hystrix to make those calls, it's your responsibility to understand why and understand why it's important for you to use that so that you can produce the quality of availability. Okay, so that's the base level.

But I think the next most important responsibility is to feedback learning about the architecture to the people who were working on that so that it can be adapted and worked better. Just as humans, we all have a tendency to grumble about the things we don't like more often than we actually go back and see if they can be changed. Frankly, it's easier and a lot more fun to grumble and gripe than to go and present the case that says, "Hey, this isn't working out all that well." So I think those are probably the two fundamentals. I'll just pause there.

It's like on an open source project. If you find a bug and you don't submit an issue, well you get to keep griping about it but it's also never going to get fixed.

That's right.

So in that sense, a developer should be aware of the architecture, know why it's there, know what it's accomplishing and be able to evaluate whether it is accomplishing that and send that feedback around.

That was much more succinctly said than my rambling. Thank you. Yes.

A lot of teams especially in the startup world are comprised with a mix of junior and mid-level and if you're lucky, experienced developers. So in the old days, and you've been doing this for as long as I have, one of the nice things about big upfront design was that you had design artifacts. You had artifacts that were stating what the desired architecture would be and obviously you pointed out some problems with that but I miss the artifacts. So how do you communicate to mixed experienced team what you want that architecture to be and what those values, these core values, of the good architecture are?

One of the things that I use is a lightweight documentation format that I call an architecture decision record. I blogged about it on Cognitect blog. Actually, I think we were thinkrelevant or it was thinkrelevance.com. The company was relevance at the time.

The idea of an architecture decision record is to say almost in like an Alexandrian pattern form - we were faced with this context, we made this decision and here are the consequences of the decision. And I'm careful to phrase it in terms of context and consequences rather than pros and cons because sometimes your decisions have consequences that aren't clearly pro or con. They're just different. We have to do something differently because of this decision.

The reason I like capturing those is it's just a page or two for each one but I like capturing those because the thing people most wonder when they look at something is why is this like it is? Hopefully you can read the code and understand what the code is doing without too much difficulty. But understanding why it was created that way is much harder. You can't get that from the code as easily.

You can't read the emergent qualities in the code.

Right. Or maybe you can infer some of the historical struggles but having the ADRs there, it lets you first understand why something was done the way it was. But also look at the context in which the decision was made, so that you can evaluate whether the context is changed enough that you should challenge or revisit that decision.

This is something that I've found useful because I've been on projects where developers would rotate through and this one comes in and says, “We clearly need an active record adapter here. It's a different kind of database but we can turn it into something that looks like active record.” And then the next one comes in and says, “Well, the format they really want is CSV so I'm going write a CSV exporter on top of the active record thing.” And then somebody else takes the CSV and adopt it to a different file format and it turns out the underlying database could have emitted that file format to begin with, right?

This sequence of incremental decisions that each person thinks makes sense, it's because they lack the full context of the previous decisions. And so I think the ADRs help with that.

And then in lieu of ADRs, commit messages can help, too. As a developer, if I don't have time to write a page or two, I can at least try to paragraph in my commit message why the heck I did this.

I think this falls into the realm of good archaeology to some degree. Because an archaeologist isn't snatching the idol off of the altar of this ruined temple defining a potsherd and trying to establish the context in which it was used and why it's not placed as it is and what its use was. I think code can be a lot the same way. Without that context, we tend to make value judgments about code that are not informed by the context in which the code was created.

Totally agree.

Yes. When people ask ‘why is this here’? All reasons are historical reasons. Some of them should still apply.

Yes.

And a lot of times you won't even hear them ask ‘why is this here’? They'll just say something like ‘this is really dumb’.

And I hear ‘this is really dumb’ and I mentally translate it to ‘I don't understand why this is here’.

Exactly. The value judgments are quick and easy to make and context is hard to decipher.

That's interesting that the judgment of the code is really a statement of ignorance.

[Laughs] I can't tell you how many times I've heard ‘that's dumb’ though. Really, it doesn't come out as why is this here or I don't understand this. It comes out as this is stupid.

Well, it's that first knee-jerk reaction, right?

Yeah.

I mean like 80% of the time that I've felt that way. I explored a little bit more and I found out why.

But it's still the first reaction.

But then again, we can take that ‘this is stupid’ - that's a clue for us for there's something here worth investigating because one of these things is not like the other.

Yes, absolutely.

It doesn't make sense. It's an opportunity for learning.

Yes, well, then it goes back to the ‘well I could have written this in half the lines of code’. So you go and you start doing it then you figure out that the other half of the code is handling all the edge cases that make it actually work.

Exactly.

And this was Spolsky's case against rewrites. That you look at this incredibly complicated code and every line of it, every if statement, every early out represent some defect that got fixed or some corner cased work around.

Yes.

I think understanding context is a matter of experience.

Experience and wanting to. But as Michael points out, we can do that. We can leave the history textbooks for the people coming after us.

It does seem like there's some humility called for. And it's tricky because I feel like especially in this Ruby space where there's a lot of startup coding going on still.

You get hired to be a rock star, right? You get hired to be someone who's going to do more with less and do great things. It almost, it feels like that's setting you up at odds with this sense of humility in the face of the decisions that other people have already made in a code base.

Someone's in adversarial situation between you and the people who created "legacy cut".

Especially if you've been brought in to do the rewrite or to clean up the mess or do V2 in some sense.

V2 is always better than V1, right?

It's always…

So what do you do…

So what do you do when you don't have those decisions documented? What do you do when you don't even have good commit messages? Maybe the commit history got lost somewhere. Are there tricks for divining, staying humble and figuring out why things are the way they are?

I'm not sure I would call this a trick. It's a perspective change. I've worked under an ops manager who repeated this mantra. He said, "Assume positive intent." No matter how asinine someone's decisions seem or how much trouble it appears to be making for you, just assume that they are doing the best thing they know how to do and they have positive intent, maybe not for you, maybe positive intent for the company as a whole or the business. But if you do that then you know your first question is not, "Man, how did we ever hire such a moron?" Your question is, "Okay, what was their positive intent when they were doing this?”

It boils down to a fundamental attribution which means that when we do something stupid or – not stupid, stupid is the wrong word. When we do something pragmatic and this code works, leave it, move on to the next thing, then it's because of our context. It's because of our deadline. It's because the next thing was more important than cleaning up this code. But when someone else does it, it's because they're stupid. It’s like a quality of them when in reality in both cases, in our case, and in the case of someone else, our reasons are almost always contextual. I like to notice judgment and actively try to replace it with curiosity.

I don't have a good segue for this but something else that you pointed out in your talk was the idea of distributing economic decision making. You gave an example from Boeing on the 777 project of letting the engineers know that weight is important. Cost is important but weight is also really, really important. Making the plane lighter is important. So if they can find a way to make the plane one pound lighter, they're allowed to make the plane $300 more expensive using some fancier material or something. If it buys them a pound less weight which is a fascinating way of distributing the goals to the people, the engineers on the ground but I'm curious what does that look like in software?

Yes, that's a fantastic question and I wish I had a perfect answer for it. By the way the credit for that example goes to Donald Reinertsen. I learned about it in his book ‘The Principles of Product Development Flow’.

In some sense, Boeing was perhaps lucky to have such clear economic tradeoffs in front of them. If we look at where our major cost drivers are in software, it's usually development time or the opportunity cost of being late to the market. So maybe one of the tradeoffs we could look at is you're allowed to trade a week of future development time for a day of getting to the market sooner or something along those lines.

Another thing that Reinertsen talks about a lot is the cost of delay. How much does it cost us to have the product reach the market one week later than our current plan date? Sometimes it's just giving up the benefits that you expect to receive for a week. But sometimes slipping by a week means you miss industry convention or you miss a holiday or some crucial deadline. So the cost of delay isn't always linear.

If you start looking at cost of delay now versus cost of redevelopment in the future, you can actually start to quantify the metaphor of technical debt. And say you’re allowed to make a decision now that’s going to cost us in the future in the interest of getting something done now because we know that future time costs less than current time.

Interesting. Thank you.

Yes, future time also might never happen. The product might take off. We may never have to replace this database with the more high performance one.

Yeah, absolutely. I’ve had an issue with the metaphor of technical debt because if the company goes under, it’s not like somebody else has to assume the debt and pay it off. It just disappears. If you delete all the code, the technical debt just disappears. So you may never actually incur those costs. There’s some likelihood of it.

It’s also very ugly on the other hand. This gets back to a problem that we’re also trying to deal with here in your talks about earlier of people want the software to be done at release. And when the budget provides money until the release and then nothing afterward, then the costs of that technical debt are extremely high because there’s no one to pay them later.

Oh, boy. I can’t tell you how much I hate project thinking in enterprises. The product approach that you see in startups is so much healthier. When do you defund the product when you’re canceling the product?

When you take it out of production.

Right. Exactly. But yes, having a project in an enterprise with an end date, it not only has this problem of amplifying the cost of technical debt but it also guarantees that everything you want to change is more expensive than it needs to be because small changes don’t happen. You have to have enough to rise to the level of a project that you can go on charter and get budgeted for and get all your paperwork, whatever it is - the RFE or the AFR, or whatever three-letter acronym your company has for spending money. [Laughs] Whatever that is, you’ve got to have enough there to justify it. As soon as you do that, you’re going to get a project manager attached. You’re going to have kick-off phase. You’re going to have all these meetings. It may boil down to we really just needed a dev to sit with the users for a week and fix some little stuff. The little stuff doesn’t get fixed because it never rises to the scale that warrants a project. I’d much rather have product thinking where you’ve got a team that’s responsible for the product. They keep it alive. They keep it healthy.

Go ahead.

They keep improving it overtime.

In my experience, the product approach this to the concept of an MVP and the MVP architecture is usually the worst architecture you can imagine because there’s such a push to get a product out the door. And there’s no time or budget to think about the future.

[Inaudible] brings out that the MVP is not a product even though that’s in the name. It’s actually a test, a test to see if it’s useful.

I think that’s a very healthy approach but I don’t see that done very often.

It plays into that technical debt. Doesn’t count for anything if the product isn’t useful and you throw it away. But then you do still have that amplification of technical debt if the product is useful. So you better throw it away but I agree people don’t want to do that.

But they also make their internal products much too big.

Well, you’ve got to get everything in the initial release because there will never be a 2.0.

You know IT has trained users to say if we’re prioritizing, it’s always three phases. Phase one is 90% of everything. Phase two is the stuff that we grudgingly slip from phase one. And phase three is never going to happen.

At some level, it all boils down to accounting. The capital project of once and done is a capital expense and it goes in a different place on the balance sheet than something that’s going to be supported that is an ongoing expense and my gosh, that’s fused the incentives.

I have found some ways of dealing with that but I fear if we go into them, we’re going to put the listeners straight to sleep.

It is a programming podcast not accounting.

Speaking of programming, here is an open-ended question for you. I know that most of the stuff that we’re talking about is of very high architectural level, really more of an organizational level. I’m curious though. Are there any technologies that you’re working with or that are on your radar that you feel are particularly conducive to helping organizations to go in the right direction with this stuff or are particularly hurting them?

I want to talk more about the positive side of things that are helping. I definitely think that tools in the dev apps space in general - packaging, deployment, provisioning - all of those tools really help move toward smaller teams because they allow much smaller deployable artifacts to reach production. There’s actually a pretty high transaction cost to moving traditional software into production. Whether it’s a new rails monolith that’s going to sit on some number of servers and talk to a relational database or whatever, the traditional way of doing it involved a lot of paper, a lot of meetings, approval processes and so on.

And so you want to go through that as little as possible which means you make the artifacts as big as possible to minimize the transaction costs. With the idea that we can have exactly the same configuration running in our dev environment, QA environment, production environment and we don’t have to go through a large approval process because not only the code has been vetted but the precise configuration has been vetted. But it doesn’t hurt as much to make it through that production pass and you can do much smaller things.

That leads us to decomposing the teams into much smaller teams and it leads us to focus much more on the way they talk to each other. Now what are the representations that we’re exchanging over the wire? What are the protocols that we’re going to use? Those are all benefits. The fact that smaller things are more easily discarded or disposed of, those are benefits.

So I’m trying hard not to say the word but I’m just going take the plunge and say the move toward micro services is really a reflection both of the architecture change and the organizational change that’s enabled by the dev apps tools.

Where we can make more small things because the cost of producing and things are slower.

You mentioned the interactions between teams and then you talked about that in terms of the data passing through or between the software produced by the teams. What about the interaction between teams on a personal level like between the teams as groups of people? What changes there as we get to smaller teams?

I probably don’t have a whole lot to say towards that. I’d encourage you to invite Ester Derby on the podcast and ask her about some questions.

Oh, great idea.

All right. Well, anything else we should discuss before we get to picks?

When’s you’re next book coming out?

That’s a question that’s guaranteed to make an author run for the exit. [Laughter]

Do you have another book that you’re working on?

And the problem is I have about five.

I’m not taking my own advice. I need to pick one and get it done.

Awesome. Well, we will look forward to whatever it is. Let’s go ahead and do picks. Avdi, do you want to start us with picks?

Sure. I got several picks this time around. A lot of videos this time around. First off, if you haven’t seen the talk that we’ve been discussing, go see it. It is really, really good. I really enjoyed it. I got important stuff out of it especially if you’re in a large organization. But I think even if you’re in a smaller one.

Other than that, I’ve been really enjoying a website called Daily Tech Video from Reuven Lerner. He’s been curating a series of – like it says, daily tech videos. Basically just picking a video that’s particularly good, usually conference talks and it’s a really well curated collection. So if you’re not sure which good conference talks to watch because there are always so many, more than anybody could ever possibly watch, this is a good way of finding top notch ones.

Also more specifically, somebody clued me into a video of the talk that Jim Weirich did in 2013 at the Cincinnati Ruby users group. This is not a conference talk. In it, he talks basically he jumps off from where Bob Martin left off when Bob Martin did his famous architecture in last year’s talk. In 2011, he really encouraged Rails programmers to decouple their stuff, decouple their domain logic from Rails but it didn’t really give a lot of particulars about how to accomplish that. So, this talk shows Jim Weirich live coding and showing off some ways of actually making that happen in a Rails project. So it’s some classic Weirich stuff. I really enjoyed that.

I will also pick a blog article that I really, really liked this past week called Why Algorithm Transparency is Vital to the Future of Thinking by Rachel Shadoan. I don’t know if I’m pronouncing her name correctly. Just a terrific article about what the title says - why it’s important for algorithms to be transparent, why it’s important to understand that there is no such thing as a neutral algorithm. So it’s important to understand the algorithms that are feeding your own thinking and understand the blind spots that those can introduce.

I promise I’m getting close to the end of my list. On a less technical level, recently I’ve been looking around for a new summary service to replace the sadly now defunct circa and I tried a bunch of stuff out. I’m not going to go over my whole review here but I will say I wrote up a blog post listing the services that I tried out and what I wound up liking. So, if you’re looking for a way to get all of your world news all summed up in one place, there are some resources there and I’ll put a link to my blog post in the show notes. And that is it for me.

All right. Jessica what are your picks?

Oh, I have some picks. Okay one of them is totally related to what we were talking about today because when we think about architecture without an end state, the fact is life doesn’t have an end state. We get really caught up in states and destinations but all the interesting emergent properties of life come out of processes. This is a major topic in a book I’m reading right now which is called Systems Thinking: Managing Chaos and Complexity. It’s a book about people, businesses, plus the software. I highly recommend, super fascinating.

My second pick, I was just at PolyConf two weeks ago. The best talk at PolyConf was by Will Byrd and he talked about miniKanren and logic programming and running programs backwards. It was really cool and you should watch it and I’ll put a link in the show notes. That will do.

All right. Coraline, what are your picks?

I have a couple today. The first is in iOS and Androids to do list with a twist. It’s called Again. It’s made by my friend [inaudible]. He works in Madison. It’s basically a to-do list for things that you want to repeat or habits say you want to form so you can use it for things like a routine [inaudible]. For me, it’s like cleaning the house and practicing voice and practicing guitar. You create individual lists by topic. And when you check something off, it doesn’t go away. It goes to the bottom of the list and the next item pops up to the top. So it’s a rotating to-do list which I think is a pretty cool idea and I’ll link to the site in the show notes.

My second pick is a novel by Neil Stevenson. His new novel is called Seveneves. In the novel, something hits the moon and smashes it to bits. Scientists realized that debris from those [inaudible] moon will start falling to earth and set the sky on fire in about two years. So the human race has two years to build an ark to take a certain number of people into space to live and hope that the species survive and so we can be repopulated in about 5000 years. It’s a good mix of personal stories and hard sci-fi. It’s pretty fascinating if you’re interested in space travel at all. So I highly recommend that.

All right. Well, I’ve got a few picks myself.

The first one is if you missed JS Remote Conf or Ruby Remote Conf, the JS Remote Conf videos have been out for a while but I decided to put them into a podcast feed. So if people want to just consume, get one every week, you can do that. Go to remoteconfs.com and you can get those there.

I just started at the beginning of JS Remote Conf. So Jessica’s talk is in the feed. Brad Midgley’s went in today as we record. So when you get this, we’ll probably have another talk. I think Craig McKeachie’s talk about MV* Frameworks is up next. So go check that out.

One other thing – the picks we always tell people are things that make your life better. I feel a little bit remised that I haven’t picked this before. I don’t know how many of you know but I’m a person of faith. I feel like God is very involved in my life and in what I do. So, I’m going to pick God and I’m going to pick Jesus Christ because He’s our Savior. I feel like They have done a lot for us. We need to acknowledge Them in our lives.

I’m also if you’re curious about more details about my faith, you can go check it out at lds.org. If you want a book of Mormon or anything like that, you can put your information in there and they’ll send somebody out with one. But yes, those are my picks. Michael, what are your picks?

So I’ve got something new, something old and something continuous.

The new one is a talk by Cory Doctorow called the Internet of Things That Do What You Tell Them. He has a lot to say on the subject of DRM devices and devices with lock in, the current legal framework that basically puts us at the service of the devices that we’ve extensively purchased. He had some interesting humorous and chilling examples that I think we all need to be aware of plus a call to action. Doctorow has been involved with the EFF for some time. It’s about an 18-minute talk. I’ll send you the link. It’s really a good listen.

The older item is a book that was a fantastic reference for me and something that I think is still totally relevant and can be learned from. It’s called the TCP/IP guide by Charles Kozierok, the Illustrated Internet Protocols Reference. This is a massive book. It’s something like 1200 pages on basically every protocol that crosses the wire. So if you ever wanted to know what IP packets actually look like or how TCP nests inside IP, what BGP is, how UDP works, it’s all there. Everything is there and it’s a great reference. That plus Wireshark and you sit there on a wire in that coffee shop and you learn everything you want to know about the network.

The continuous item I’ve got is a curated reading list that Mark McGranaghan keeps up. He calls it the Services Engineering Reading List. It’s got a bunch of papers - some old classics, some new ones, explanations behind many of the distributed systems protocols that we rely on. It’s got some blog links, some book links and some conferences that are worth looking at. So he keeps adding things to the list and it’s really a valuable centralized place to go.

All right. Thanks for coming Michael. It was a fascinating discussion and hopefully some folks can get some ideas about how to think about architecture. If people want to follow up with you or check out what you’re doing, what are the best ways to do that?

I’m on Twitter at @mtnygard. I’m certainly available by email mtnygard@cognitect.com.

All right. I guess that’s it.