The Rogues talk complexity with Glenn Vanderburg.

Special Guest: Glenn Vanderburg.

 

All right. So what are we talking about today? Are we talking about complexity?

OK. So, do we have a drinking game for this? Like every time David uses the word “complecting”?

Hey everybody and welcome to episode 77 of the Ruby Rogues Podcast! This week on our panel we have James Edward Gray.

Who are you people and why do you keep calling me?

Avdi Grimm.

Hello from Pennsylvania!

David Brady.

I'm going to complect this podcast until Josh loses his liver.

Josh Susser.

I really don’t know how to follow that one. [laughs] [laughter]

Just drink man. Just drink. That's the rule!

I like that. David Brady wins again.

[laughs] That's a little frightening. I'm Charles Max Wood from devchat.tv and this week we have a special guest, that's Glenn Vanderburg.

I have nothing snappy to say.

That's OK. I'm sure you will come up with something. We have a few announcements before we get going; we will let Avdi start us off and then James will take it away from there.

OK. Book Club. We are reading… this will come as a complete surprise to anybody who has been a long time listener but we are reading Practical Object-Oriented Design in Ruby…

Nooooo!

Has anybody picked that?

Has anybody not picked that?

I think one of you.

…by the great Sandi Metz. We are having her on the show January 2nd, so if you haven’t already, pick up a copy of the book and read along with us, and then we will chat with her.

For the listeners, that means that episode will air on the 9th right?

Right. Unless we figure something out and start doing live episodes. But I don’t know, I really wanna--

Or we can loop the episodes and have them air before we record them.

[laughs] Josh has been watching Looper, for those of you who missed the pre-show.

No, no. I'm about to watch it. [laughs]

Oh, yeah right.

I'm actually cripping for my attendance at this show looped back in time.

Right. So, James you also have an announcement for us?

Yes, it’s time for Best of Parley. The best stuff on our mailing list. I was gone for two weeks, so I'm cheating and I get two picks. First is Josh Susser’s genius move; we invited all past speakers and we will invite all future speakers who have been on the show. They have been invited so, if you want one mailing list where you can get people like Steve Klabnik and Jim Weirich and Chad Fowler answering your questions, we now have that. We have seen exactly that on the Parley mailing list since they were added. So, that is crazy awesome.

The other cool pick was, there was a great thread by Benjamin Fleischer, (I think) just asking, “Hey, I'm going to Ruby Conf. It’s kind of the first time I've done this. What do we do?” And everybody chimed in with their conference going tips from what to bring to what see, what to miss, what not to miss and that’s absolutely awesome even for an experienced conference goer like myself. So, lots of awesome things going on the Parley mailing list. So, check it out.

I just wanna second that and say it will surprise you how many people said, picked what thing not to take to the conference.

That's right.

Just leave it in your hotel room.

It was awesome.

One last announcement.

Yes.

And that’s about the Ruby Nuby Project. So, if you are listening to this episode on Halloween, which I don’t know how likely that is, but if you are, today is the last day to get your video in to entering the listener challenge and potentially get picked to be a guest on the show and come talk to us about what it’s like to be new to Ruby. So, go check out the rubyrogues.com. There's a link in the sidebar about the contest and give us a couple minute video and then maybe come on the show.

And thanks to our supporters for offering some incentives to participants. So, rubymonk.com is offering a month of free subscription to their service and RailsCasts Pro is also doing that, and I believe we now have a 20% discount offer for the Pragmatic Studio’s Ruby Online Training as well.

OK, that's it.

Yeah, yeah enough stalling. Tell me about this Glen Vanderburg guy.

[laughs] I don’t know but whatever we are talking about, it sounds complicated.

I mean complex. Overly complected.

Oh, complected.

So, I guess as it’s obvious by now, we are going to talk about Glenn’s recent talk at GoGaRuCo 2012, which was called “Dealing with Complexity”. Is that right, Glenn?

“Grasping Complexity with Both Hands”.

Oh, yes Grasping Complexity with Both Hands. And Josh told me to go watch the thing and I did yesterday. And you know when I saw the title, I thought, “Hmm, I wonder what this is going to be.” And my initial reaction was that, you would show complicated programming scenarios and maybe give advice on how to get through them, and that was not at all what your talk was. So, do you wanna tell us what it was?

Well I've spent a whole lot of time, both in conference talks and sitting around tables and office and everything else, doing exactly what you just talked about. But, I've become aware and I have a mania for simplicity in code and I found that nearly always, if you try hard enough, you can take complicated code and make it simple. But I began to notice that there are some things in the world (not just in programming) problems that are genuinely complicated.

And people have a natural sort of bias toward trying to think in simple terms and think in black and white, and kind off recoil from real sort of dizzying complexity. Which is okay, but if we don’t face up to real complexity and situations that we deal with, we end up making stupid decisions. And so the talk is about how to recognize genuinely complex problem and situations and think clearly about them and try to make smart decisions instead of thinking that a complex situation is really simple.

Nice. So, do the simplest thing that could possibly work, does not mean you can do the simplest thing you can think of and then skip the last half of that?

Sure. Right. And we all know that, but sometimes in the thick of things, we forget.

Oh, I like that a lot. The current project I'm working on, the company has decided to… my current client, they have decided to take on to HIPAA compliance and PCI compliance at the same time (it’s medical billing) and the simplest thing that could possibly work is heinously, insanely complicated.

Yup, that happens.

I like how throughout you talk, I mean like I said, I was originally thinking it would be about different code scenarios or something. But actually, it didn’t even turn out to be about programming as much as it was about people and communication, and the silly things our brains do to us.

Which really shouldn't be surprising. It was surprise to me too as I wrote the talk and everything but it shouldn't be surprising because that's when things tend to get complicated is when people get involved.

Yeah, those darn people.

Glenn, have you read Peopleware?

Oh, yeah.

DAVE: OK.

That is one of my very favorite.

I love the fact… the original book was like 1978 or 1980 or something and---

I was working at the time. Where I was working at the time, that was passed around as subversive literature.

Oh, really?

Oh, yeah. This is about a book that everything they are doing wrong around this place.

Yeah. So they revised a newly updated… because it was written before the internet and they revised it, released a 2nd edition in 2000. And the preface to the… the preface to the preface or the preface to the 2nd edition says, “We went back and read the 1980 manuscript and we have decided to fix a couple of typos and add a chapter on email.” And that was it.

All of the problems that we described in 1980 are still alive and well, that they have nothing to do… you can’t solve a people problem with more technology and the internet has absolutely no effect on that truism.

Right.

Now we can just distribute our dysfunction.

Yup. Well, dysfunction is now scalable.

Yeah.

It’s Web Scale. Yay!

OK. So Glenn, back to the topic. [laughs] I thought your talk was very interesting. And I tend to… when I think about the complexity involved in a system, I look at it from sort of a thermodynamics perspective. Complexity isn’t… I don’t know, it seems like if you draw the circle around the stuff in your system in the right place and you have a good grasp of what are all the… what's the whole functionality of this system (maybe including the users or whatever other things are part of it), that past a certain point, you can’t really reduce the complexity of a system.

No.

There are just some things that are innately complicated. And this is what you were talking about and you did this whole whiff on over simplification. So the way that I like to look at it is that, there's like the 2nd law of thermodynamics for complexity where, you can’t remove the complexity from the system. The most you can hope for is arranging things, so that most of the complex things are convenient to ignore.

Right. Fred Brooks talked about this many years ago in… oh, I forgot the name of that paper now.

Mythical Man-Month?

No, that was the book.

Oh, OK.

There was a paper; the subtitle was “Essence and Accident in Software Design”. And he developed this concept that in every system, there is essential complexity and accidental complexity. And he chose those terms from Aristotle. I tend to think of it more as “incidental” rather than “accidental” complexity. The essential complexity is just inherent to the problem you are trying to solve in the system that you are trying to build. You can’t reduce it beyond that point. But yeah, “No Silver Bullet”. Thank you. That's the name of the paper.

Oh. I’ve read that.

Yeah. So, the incidental complexity or accidental complexity is kind of all the other stuff, that if you knew how, or had tools that were really well suited to the problem, you can get rid of that. And in typical software systems, the accidental complexity turns out to be… well, a lot of it is just extra stuff because you don’t understand things well. But a lot of it too is taking general purpose programming environment and building it up to be the special purpose programming environment that you need to solve that problem, right? So it’s pretty old concept of its important to try to understand what complexity is essential and inherent to the problem that you are trying to solve and get all the other stuff and compartmentalize it and keep it out of the way.

You know, I read about awesome article just like yesterday or the day before and I think it was called “Learnable Programming”.

Oh, yeah. Bret Victor’s thing.

Yeah. That guy who did “Inventing on Principle”, right. That famous video that everybody loves. And I was just struck with what you were saying and how much that is tied in to… he is trying to figure out, obviously trying to resolve a really hard problem of how can we make our editors take them to the next level, you know, or tools basically. And how can we make it where you can start with some simple idea, but then just do simple steps with the editor or interactions with the mouse or whatever. You know, it abstracts that to a point where it’s useful to you and those are very interesting.

I don’t necessarily agree with all the directions he is going with that, but it’s amazing work and you know, I have to admire him, he is really doing the hard work on things that a lot of us have just been kind of thinking about for a long time. One thing I've noticed is the best programmers I know all have some good techniques in their mind for visualizing or conceptualizing or modelling programs they work with. And it tends to be sort of a spatial visual model in your head but not always. And you know, what's going on is our brains and geared towards the physical world and dealing with our senses and integrating that sensory input, and sometimes sensual input I guess and dealing with that.

But the work we do as programmers is all abstract. And it makes perfect sense that you would wanna find techniques to rope the physical sensory parts of your brain into this task of dealing with abstractions. But we don’t ever teach anybody how to do that, or even that they should do that. And we do a really poor job of building programming tools to help with that. So, you know, the good programmers are the ones who kind of somehow managed to figure this out on their own with no help. And Bret Victor, it seems to me is trying to figure out ways to make the tools really work well with the majority of our brains.

Yeah, that's a great point. It seems like almost in learning programming, you hit this point where you are basically thrown to the wolves and you have to you know, just go off and suffer for a while. And in some people head it clicks and with others it doesn’t and they just go away. And you know, the ones who make it past that horrible stage, they come back and they were able to teach some more advanced stuff. But I think that you are right. I think we don’t do a good job of teaching how we fight the layers of abstraction and stuff.

So the talk I gave at GoGaRuCo sort of came out of the talk I gave last year at Lone Star Ruby Conf. I don’t know if you remember James. You opened that Conf with a keynote about how we need more science in our field. And I sheepishly stood up for the next talk and said, “My talk is about how we can’t really do good science in our field.” [laughs]

And so, you know, I certainly agree with everything you were saying. But in a broader scope beyond the code, it’s just too costly or impractical or in some cases even impossible to do what we would consider genuine scientific experiments with rigor on software development and the way teams work and things like that. So that talk was all about various kinds of soft evidence that are weaker than in scientific evidence and how you make smart decisions if that's all you got, right? So when I started planning for GoGaRuCo, I sort of zeroed in on one part of that. It’s just ways of dealing with complex situations and thinking about them clearly. And that turned out to be a fairly rich vain all by itself.

Yeah, I mean if you look at just you know, most of the modern interest, let’s say even just in our community, I mean, you've got things like SOA getting a lot of attention. Which is about dealing with complexity as your application size grows right? Object orientation has made a huge comeback in our community and that's about dealing with complexity. You know, it’s on almost every level that seems to be what we are interested in.

Right. For the listeners, I should make a point of saying something that I said in the talk, which is this is absolutely not a topic where I'm an expert. I chose to kind of give this talk and spend some time thinking about it because I realized it was important and I needed to learn about it. But I'm absolutely not an expert on it.

You made an interesting distinction in the talk between “bounded complexity”, which engineers actually tend to like even if they don’t always admit it and “unbounded complexity”.

Right and I made this snide joke about, if we didn’t actually like some kinds of complexity, we wouldn’t have languages like Scala.

…or Ruby for that matter. But it’s much less snarky to say that. You know, I think it comes down to… I don’t know if “bounded” is really the right word, but it’s “Do we have this intuition that it is within our grasp, right? Do you feel like if you rolled up your sleeves, you can conquer this?” And engineers in general I think really finds those kinds of puzzles really attractive and wanna dive in and deal with things. In the talk I gave this example of a couple of engineers in LivingSocial where I work said, “Hey let’s have a crappy code contest and see who can write the worst implementation of this.” And whole bunch of people dove in and came up with some just breathtakingly horrible complex solutions and it’s loads of fun.

You put yours up on the slide and I loved it. It was great.

I loved your comment of… what did you say… “It’s disturbing how proud I am of this code.” [laughter]

Yeah. It’s YAML Parser that’s basically built around two giant gnarly regular expressions.

And stabby lambdas.

I have to admit the stabby lambdas were sort of gratuitous. It was, OK it’s ugly and horrible with just the regular expressions as I was hoping, so I have to find-[laughter]

It sounds like the geek’s version of fake Hemingway Contest.

You are right.

I did a fizzbuzz contest years ago and somebody submitted a working solution, that was all regular expressions.

Oh wow.

Terrifying.

Oh, man.

Did you all see Jim Weirich’s version of that? It was all pure lambda calculus.

Yeah, that was awesome.

Yeah, I mean in Ruby but he limited himself so much.

Yeah, that was awesome.

Isn’t that kind of the idea behind the Coderetreats is you solve a problem. Not that you are writing horrible code, but you put constraints upon your code?

Exactly. You are trying to write the best code you can, but you put constraints on yourself or your pair to kind of see, “OK if I can’t do that what other techniques do I have to discover or invent or whatever in order to keep this code reasonable.”

And you had a cool thing in your talk that was a aha moment for me, where like you talked about how as programmers, we are always searching for that one simple solution that satisfies all he edge cases. And of course we are generally always disappointed right? Because it almost never exist. I knew that intuitively but it wasnt until you said it that it’s like, “Yeah. That's what we are after.”

So, I think some of the move to try to program without if statements you know, is driven by that. Every now and then, you can craft a solution to a problem that seemed to have a lot of special cases and it turns out that there's a way of solving it where the special case just fall out naturally from the logic. And sometimes it happens without you realizing it and you write the test for one of the special cases and it passes right away.

Right.

Now Glenn, you were talking about bounded versus unbounded complexity just a moment ago. One of the things that I find really fascinating is that the human brain is not wired for being able to deal with large numbers.

No.

Yeah we can handle a few numbers, like seven or ten or something like that but, when you get up to a trillion or something like that, we just have no concept of how these things work. And I think that a lot of the surprise reaction that we get to encountering coincidences is because we don’t really understand all of the large numbers in play.

For instance, if I'm walking down the street and I run into my college roommate or an old buddy from college. I go, “Oh my god! That was just amazing. I haven’t seen him in years. We just ran in to each other.” and someone will say, “Wow! What are the odds of that happening?” and my response is always, “Well, apparently pretty good because it happened.” [laughter]

When people say “it’s a small world”, they are talking about two people who have lived in to the same city for years. [laughter]

But if you think about all the people that I've met in my life or all the people I went to college with even (and you know, this may be you know, a couple of thousand people just in that pool) and how many of them I'm like I might run into any day in my whole and then you run the numbers on that and say, “Oh well, did I just happen to run in to somebody at some date.” It’s actually not very unpredictable or it’s not very unbelievable from a statistical point of view that that will eventually happen.

Right. “The Birthday Paradox” is an example of that.

Right.

Yeah, that's a great one. For people who don’t know what it is, tell us Glenn.

Oh, well so you know, I forget exactly what the threshold is, but you have a room full of say, 20 people and you say, you know, you make a bet with somebody, “I bet two people in this room have the same birthday.” And you know, generally in a room of 20 people, it’s easy for some to take that bet. “No way. What are the odds of that?” Well it turns out, if you do the math, that the odds of random group of 20 people two them will have the same birthday is pretty close to 100%. Just is totally unintuitive for the way most people think about those odds.

Well, it’s unintuitive to me. [laughs]

It’s combinatorics, and our brains don’t really have any hardware support for doing that.

All you have to do to change it well is say you know, “Pick one person in this room and what are the odds that another person will have the same birthday as that person?”

Yeah, that gets really hard.

Yeah. That's really astronomical but, if you can just pick any two of the 20 people in this room that have the same birthday, a pretty good bet that one pair will have the same birthday.

Yeah, so just to give the correct percent, if you have 23 people, that's a 50% chance that they will share the same birthday. Yeah, it’s very surprising as it turns out. And it’s so confusing, people understand this so badly, there a episode of I think it was The Tonight Show, where somebody comes on and they try to explain this to Johnny Carson and he didn’t buy it of course. And then the guy did such a bad job of explaining it and saying what it was, that John is like, “Ha! We got that many people in the audience. I’ll prove it to you.” And then he asked the audience the wrong question, which turns out to be and fails. So, yeah, it’s very difficult to understand and it’s like what Josh said, its combinatorics and we’re not good at that, intuitively.

Well, the other thing is, that I just looked it up and it’s explain that it’s the number of possible pairs as opposed to the number of individuals. And the interesting thing is that, it harkens back to another part of talk, where he talked about asking the right questions. So instead of asking you know, how many days in the year are there and how many people are in the room, that's not exactly y the right question or at least it’s not the complete right question.

This problem, yeah it goes even further than that because I know Glenn also talked about, can we do this thing that would get us most of the way the way there. That would be mostly right. And that's kind of the interesting thing with the birthday problem. If you need a room full of people, that were two of them have the same birthday and you have to guarantee it, then you need 366 people to cover every day plus one. That's how many you need. But if you are willing to accept the 50% chance, that number drops to 23, right? So, massive difference.

Yeah, there's a lot of cases where, people are asking the wrong question or looking for the wrong kind of solution. That's why in the talk, I talked about how we like compete solutions, right? And we are trained for this when we start programing. If we learn to program in school, when you turn in to program and you get marks counted off if it doesn't cover all the edge cases. But in real life, often, you can get by without covering the edge cases. Almost every sizable business has automated processes that just simply on some hard cases and send an email to somebody or stick something in a task queue or something for human to deal with.

And you know, if those edge cases are rare enough, that’s a perfectly reasonable thing to do because it will take a whole lot of work to automate it. The automation would probably be very brittle and you'd probably still only cover like 10 to 20 percent of the remaining weird cases that could happen. So, you know, once we get it down to where there's just a couple of these things a week, let’s just let a human being to take care.

Well, I think it’s an interesting thing too. Because ultimately, what we are talking about here is providing some kind of value to our customer or boss or whatever. And so by solving this edge cases, if we spend 8 hours or 16 hours and let’s say our time is worth, I don’t know, $50/hour and that's probably high for some and low for others but, let’s just say $50/hour and cost them $50/hour to make you work which is I think really low actually now that I think about it. So we are talking $400 to $800 where you could pay some minimum waged guy to handle it every once in a great while. So the ROI isn’t there either.

Sure.

I was just thinking about how, you know, just now Glenn you talked about get an email when something goes wrong or things like that. You also talked about that you talk quite a bit where you said “Don’t try to make things not go wrong. You can’t make things not go wrong.” You know, just make it where if you find out when things go wrong and you can do something about it. You know, or try to make it transparent and---

Right and this is kind of the… and again, most of the examples of this I think turn out in people processes rather than in programs. But, this is one of the things that distinguishes Agile work from older school style of developing software. The command and control kind of process only goes so far if you try to prevent bad things happening or always steer things in the right direction. You are going to miss opportunities to do clever things that might initially seem like bad idea. And, you are still going to have problems and be ill-equipped to deal with them instead of control, look for ways to make it easy for people to make progress, but also get feedback quickly when things go wrong.

That’s a really good point. I worked on a system one time that had a database that didn’t have racking for example. So, you can do something like uniqueness check. You didn’t have a transaction and you couldn’t do… only add this email address for this guy if it’s not already in the database. And it turned out, it sounds like a massive problem and we are thought that you have to have that. And the truth is that, it really wasn’t as crippling as you might think.

We looked at the problem really hard and it’s like, “OK, there's like 3 areas where this actually matters.” And most of them can be solved with this simple little trick or something so that we are reasonably sure. There was actually one which was actually new user sign up where, but if everybody is using their real email address, there shouldn’t be any duplicates anyway. But there wasn’t a way for us to technologically ensure that without accidentally creating the user. And we solved it with a really stupid simple process where, anytime we have to fetch a user from the database, instead of fetching just the first on that matched, we asked for the first two that matched so that if a duplicate had been created, we’d get more than one. And then at that point, if that ever happen we just you know, lock those accounts and say, “Something went wrong here.” and so we send an email to the admins. And to my knowledge, that was never triggered. You know, it never even came up. But the point is, we could have exhausted such a ridiculous amount of effort trying to get transactions because we just knew we had to have one. But, it turns out we didn’t really need them as much as we thought.

Yeah, it tells you how big of a problem it is by getting those emails. The other point I was going to drive at was more along the lines of development process. A lot of the times we tend to get paralyzed by the fact that we don’t know how to solve the entire problem. You know, “I can’t get this whole problem in to my head” or “I kind of get the whole problem in my head, but I can’t get a whole solution in my head.” And I have been in situations there where, you build the incomplete solution and by the time you are done building what you could formulate in your head, then you understand the problem well enough to move ahead and finish it.

And sometimes, you can finish it by integrating that into your solution you are building and sometimes, you have to build like two-part solution to recognize when the simple way or fast way or whatever can’t deal with this particular case and punt to a different way of dealing with it.

Plus just doing that bit that you can do, you have already reduced the complexity so much and make it easier on yourself.

Though it does happen sometimes that you load the guns in to the car and then you fall over on your back and break your arm while getting out of the car. I mean, that does happen.

That would never happen.

What are the odds honestly of that having happen to like say somebody on this podcast?

Yeah, exactly. We are the odds?

Astronomical. Oh, wait. [laughter]

The probability is 1.0.

Yeah, so what I'm saying is, building the incomplete solution isn’t always a guarantee that you are going to be able to finish the solution. But like we have been saying before, it may be good enough.

Yeah.

Exactly.

Start with what you know.

That was the great thing about all your talk; everything was down to earth. You know, you are going to run in these complex situations, you’re going to run into these problems, try to see the try to simplify as much as you can, try to ignore the parts you can’t. It was very practical.

Now, one of the canonically difficult problems with complexity is talking about it.

That's true.

And how you communicate about something that is really complex. You are sitting there programming and you run into this ridiculously complicated thing about the product feature that your product manager has given you. They wrote up a story and you start working on this. Now you estimate it as a three-point story and you run into it and you are like, “Oh, my god. This is like 25 point story. What am I going to do?” But you don’t share enough common language in the technical jargon to be able to talk about the complexity and explain things well enough so that they can really understand what's going on and then figure out some better course of action.

In general, programmers are really well equipped for that because we have a language that we share that lets you be really specific about things. But then when you are talking about people who aren’t programmers that can be very difficult.

it’s hard for me sometimes though, because even faced with the complicated problem, I sometimes feel sorry for the person I have to answer to. Because when I'm in the middle of it, I know that I have it, but I probably can’t explain it well enough that they know that I have it. [laughs] It’s like, “Yeah I got it and I'm not really sure what I'm doing, but I'm pretty sure it’s going to work. I’ll tell you when I'm done.”

[laughs] yes.

OK. But other than that, just in general, when you’re talking about complex things, is there some psychology of it or something about it that makes it easier to communicate about?

Not that I know of.

[laughs] OK.

Is there a magic bullet for it?

…Why it’s so hard and I was going to say, “Yes, absolutely.” You know, one thing is that the people you are talking to, being people, have such a short memory, right? And so, you start talking about, “Well OK, so x in case y, in case z. And if you mention a case that they have some opinion about, they immediately and I say “they”, I mean “we” it’s just a natural thing that people do. If you mention a case that they have some knowledge of or opinion about or suggest for, they immediately zero in on that and forget all the others. And its hard, Josh you mentioned people have problem with large numbers, when it comes to keeping multiple thoughts in your head at the same time, people have problem with very mall numbers.

[laughs] That's awesome.

You mean it’s easier to keep big numbers in your head?

Well yeah, because you don’t have to think about each one.

What I love is that Glenn Vanderburg just called all of our listeners stupid.

And he is right. It’s one of those things where we think we are really good at something but in reality, as human beings, we are really bad at it and we have an epic blind spot on it as well.

For a second there, that felt like you were saying, “Look Glenn, look down these stairs.” [laughter]

But if you read books like “Pragmatic Thinking and Learning”, they talked about that a lot; about how, as you get up on the experts scale, you’re actually much more humble right? Because you have developed all these defense mechanisms [laughs] you know, about everything you know you can’t trust yourself on or you can’t know. And so you have these checks and balances and your instincts of course that you have gained along the way that guide you in roughly the right direction. But at the same time, you better be double checking because you also know your instincts are very fallible.

This is why people make check lists and have GTD programs and all those things because we've learned how fallible and stupid we are.

Right.

Unit tests.

Yeah.

The best description of GTD that I ever heard was, “GTD is a Trojan Horse that you invite into your life and then basically, you'll become and agent for GTD running your life.”

[laughs] that is awesome.

Wow.

That is a great description.

Somebody mentioned unit testing. That's one of my favorite examples of sort of a complex thing that people have trouble grasping. And one of the patterns… I hate using the word “patterns” here.. but, one of the common patterns of people not dealing with complexity adequately is focusing on obvious benefits and costs and forgetting about second order things. And I've spent a lot of time over the years teaching unit testing and TDD and I've noticed this happened all the time.

Unit testing has some costs especially at times and it also has benefits. And the interesting thing is that, it has multiple different benefits that help you in different ways. And people who are just learning about have real trouble holding all or even more than one of those benefits in their mind at the same time. They focused in on the one they understand or the one that means the most to them or answers their current problem or whatever. But then, they reach the point of saying, “Well, if that's all it is, it’s not worth the cost. It doesn't seem.” And I kind of agree with them. But, it’s when you add all the different benefits together that it becomes a slam dunk and it’s kind of hard for a lot of people, when they are first learning about it, to understand all that.

I had a great chat with Corey Haines at Aloha Ruby Conf and we discussed exactly this. He said that had written a system recently no unit test because he wanted to see if it was still worth it. Like, you know, is it valuable to him. And he said, for the most part, he wrote the app and he says, “If you look at it, for the most part, you would think I would probably TDDed it.”

His instincts are pretty good at this point. He's done it long enough and he in the neighborhood but he also admitted that, there is some part so you can look and think, “Oh, that would have been a lot better if I had actually TDDed it.”

And that’s the slam dunk is that you unit test and suddenly you changed the way you write code. You write stuff that's more testable and more modular. And when you’re still just learning how to unit test stuff, you don’t see that. You have a blind spot. You cannot be made to see that. It’s not until you screw up and you write some code without putting test behind it and then you go back to write a unit test and you find out you can’t test it, because you have complected your code (take a drink Josh) yeah you have mixed everything up and now you can’t test it and you don’t see that.

Or the flipside is you go back through working on a project later on and you are dreading something and you go in and you realize you TDDed it and it either saved you because the test doesn't pass or you realized, “Hey that wasn’t so bad because I had to write the code in a different way.”

Yeah. I'm actually facing with multiple clients recently this resistance to unit testing. And there's just this innate need to think about code and to logic your way to the facts. And we talked earlier in the show about computer science needs more science, and I'm going to nit-pick and say that computer science is heaving with science, but its heaving with formal science as opposed to empirical science.

(Yeah!)

And formal science is a science in which you set up a set of a priori rules and you just logic your way through it. And, “Well of course it works, I thought about it and I'm sure it works just fine.” But empirical science is where you actually design an experiment to find out if it does in fact work. And if there is a unit test, that is an automatic empirical science test that will let you give this huge formal logic pile of nested mangled garbage or this huge pile of code, to someone else who doesn’t know the code who needs to sit down on one end or maybe sit all the way through it. Or they could just run the unit test and bam! they’ve got empirical evidence that says the code works. Now they can go look through it and understand how it works because they figured out first that it does in fact work.

With as it Knuth who said, “I haven’t tested this program to see if it works; I’ve merely proven it correct.”

So, be careful.

“Beware of bugs in the following code. I have merely proved it correct. I have not tested it.”

Yeah that was kind of a compromise I think Glenn and I came to when we both gave those sciency talks at Lone Star was that, it really depends on how you are looking at science. And I was going for the very straight up definition, that anyone can do… we don’t need teams, we don’t need massive projects or whatever it was we build the hypothesis and then try to prove it wrong, right?

You know, one of the things I said in that talk last year was we throw around this word “prove” and “proof”. And you know, our training in science and math makes us think we know that means. But if you want some fun, put a physicist and a mathematician and a historian and a lawyer in a room together and tell them to agree to and what the meaning of “prove” is.

And because you know, (well, law is arguably outside that altogether) but [laughs] you have different qualities of evidence. You can’t go back in time and repeat things. And the standard of proof that works in some situation is not going to work in other situations. And in software, we are trying to prove something to where we have confidence to act. And sometimes, a much lower standard proof is perfectly adequate for that, and sometimes it has to be because going beyond that would just be way to costly.

To me though, I think that's why science makes such a good example, because even things that we “know” you know, like the theory of gravity--

Right

Because we have enough evidence, we tried our best to disprove it and at this point, that's how we can explain it that's seems to fit the rules.

We have proven that that's the closest approximation to the truth that we come up with yet.

Exactly. It’s like Newton got close on relativity and then for some scales, he was effectively right. But, it wasn’t until Einstein came along and did some tweaks that it got even better.

You know, this reminds me and this kind of taking things in an odd direction, but you know, it took a while after Newton’s discoveries for the world at large to really come to grips with those laws. And I'm using the standard sort of average high school educated guy, you know who paid attention in class kind of understands the way the world works according to those laws. And that had as those ideas spread throughout society, that had a big impact on things, right?

Right.

Well, then the 20th century comes along and Einstein appends a lot of that. And it took a long time for people to sort of have an intuitive understanding of relativity. But eventually, and even when I was a kid, the average student who was interested in science and paid attention in class got that and it came partly though school and partly through reading of science fiction novels that feature it and everything. But you kind of have an intuitive grasp for how relativity affects you in different situations, hypothetically. Well, there are other scientific revolutions that I think haven’t penetrated into society that thoroughly yet, and one of them is Quantum Mechanics. But one of them that I'm really interested in is the idea of chaos and emergent phenomena and things like that. And I think that, a big impact on society as late people, (if you wanna use that term) come to really be comfortable with that concept of small, simple rules having large and apparently very complex effects at different scales.

And we talked earlier about how Agile processes are, basically they substitute facilitation and feedback for control. And that's an example of that. Its kind of an emergent process where you give people goals and put a few lightweight, simple, localized rules in place that govern how they act in different situations. And you watch that achieve, (in most cases) better results than you could if you planned it out and controlled every aspect of the process. And TDD is a very similar kind of thing. One of the reasons that people find it hard to grasp is that, the idea of an emergent process like that where you don’t think about this large, complex artifact you are trying to build at a large scale very much but you focus on the details and let the design grow, seems like it couldn’t possibly work.

In one sense what you are talking about is the evolutionary process.

Yeah.

Some form of mutation change and selection. Have you ever read anything about the work of Thomas Ray and genetic algorithm stuff that he did with this Darwin System?

Yes.

It is fascinating. It’s worth looking up for listeners. (Maybe I’ll put a link in if I can dig one up) but basically, he wrote this little virtual machine that had about the same order of complexity for the instruction set of the programs as the human genetic code. Meaning there are about the same number of instructions as there are groups of code that you find in the DNA. And the simplest possible program you can write that could copy itself and turn the system loose with a little bit of mutation allowed and some natural selection pressures on the system. And like immediately, (I don’t know how many thousands of generations, but in human it was immediately) they had all sorts of mutated creatures and the things that always developed was parasitism.

Wow.

it’s amazing how all this stuff works. And then there were like, organisms that develop defenses against kinds particular kinds of parasitism and then the parasites would evolve to do something else. So it’s just fascinating stuff. And it really speaks to what you are talking about with you can set up some simple mechanisms for providing feedback in to your process and that can that move what you are building in unexpected directions, but it can also take you a lot further than having to sit down and prognosticate what are all the things that you might do down the road.

That is awesome. Josh you need to put that in the show notes.

OK.

A lot of times, when you are face with the situation that seems just forbiddingly complex, part of what is happening is we are trying to think of a sort of command and control style solution to that problem where we sort of analyse it and one by one, deal with everything that can happen. And one technique and try to look for a more emergent style solution, where you do simple things and you can pick the right combination of simple things that together, produce and effective, if not perfect solution to this problem without you ever having to understand all of the details of what a solution looks like.

And just to kind of redirect a tiny bit before we leave this, to me, I’d look at genetic algorithms quite a bit in the past and played with them and you can almost always get to the solution that way and its almost always great, but it’s the amount and effort and energy it takes to push that all the way through or you can crack open any algorithm book and if you turn to the right kind of problem, you can probably get the heuristic solution that gets you 80 to 90% of the way there are almost no effort. You know, it’s very strange.

Awesome. Well, let’s go ahead and get into the picks. David, why don’t you start us off?

OK two really quick. We mentioned Peopleware earlier in the call. I don’t think that's actually ever been picked, so I'm going to pick it now. It’s called “Peopleware: Productive Projects and Teams” by Tom DeMarco and Timothy Lister. There's some software from the 90s called Peopleware, it’s not that. The is just a book about all the things that organization do wrong to make in order to try and control people’s productivity that ends up screwing up their productivity. If this statement resonate with you, you need this book. Do you have a manger who thinks that you are lazy and need to be driven to do work, and yet do you sneak off to a conference room and hide in a dark office so that you can get work done. Those statements do not jive. You are not lazy if you have to go hide from your work in order to get work done. If that resonates with you, you need to read Peopleware.

Adding technology to people problem makes it worst. It’s a fantastic, fantastic book.

Why We Do What We Do in Life and Business” by Charles Duhigg. This book is… I'm just going to flat out say that this book is in my top 5 all-time books I have ever read. If you want to know why you do habits the way you do, if you wanna know why marketers are totally controlling your life, if you want to know why Target can tell if you are pregnant before even you have told anyone. And so they can start mailing you coupons for baby products and why they do that and why that's really freaking evil, this book is absolutely incredible.

The reason I recommend it is because he breaks down what the elements of habit are. Habit is some queue that triggers a routine that you will go do that then gets you a reward and all three of those things are wrapped around a craving for something. And if you have those four elements; queue, routine, reward wrapped around craving, you will form a habit every single time. And it controls over about 40% of what you do during the day is a habit and not a conscious decision. When you go to the bathroom, (it wouldn’t be David Brady if I didn’t work a potty story into it) but when you go to the bathroom, you don’t think about taking your pants off. That's a conscious decision. You have habitualized this. This book is absofreakinglutely incredible. If you want to figure out how to make your bed in the morning or brush your teeth more often or exercise or quit smoking or any of those things, this book breaks down the math out of how to tear apart the chunks of your internalized habits, replace the pieces that you want and move forward. It’s absolutely incredible.

So, if I had a habit of like really long picks, what would I do? [laughter]

Basically, you’d really have to delve into that craving to hear yourself speak and being in love with the sound of your voice. And then consider the reward that you get of getting to hear yourself talk and talk and talk and talk…

Yeah very, very, very cool. That's my picks.

All right Avdi, what are your picks?

I think I just have one. This is kind of been all over the place on Twitter and other places, so a lot of listeners have probably heard this already, but on the Code Climate blog, there is an article on “7 Patterns to Refactor Fat ActiveRecord Models” and its really good.

That was by one of my co-workers and its awesome.

We are going to have an episode on that. It’s just we are kind of booked up right now so it’s a little ways out, but we are going to do it.

Great.

Good stuff. Lots of good advice there.

Yeah, thank you Brian.

OK. James what are your picks?

Two tech picks. The first is there's a podcast by the guys who did “Giant Robots Smashing into other Giant Robots”

Anyways, they have a podcast called “Giant Robots Smashing into other Giant Robots” and I don’t even know which episode this is but kind of hard to find podcast, there have been like individual pages for it, but there was an episode with Chris Wanstrath and somebody else I think. Anyways, it’s a really neat episode. They go through and talk about a lot of interesting things like, does tell don’t ask conflict with the MVC design pattern, (which is kind of interesting). They talk about class oriented design, they talk about cool things we can get from Smalltalk like categories actually, which you know maybe doesn’t get discussed a lot. So anyways, it’s just kind of obscure podcast that I enjoyed listening to. I found this episode from somebody’s recommendation and lot of cool titbits in it. So check that out.

Then, Avdi’s new series Ruby Tapas, where he is doing these Ruby videos all the time. They are crazy great. The thing you have to know about this is that the length is perfect. They are usually like 2 minutes, sometimes less. Very occasionally a little more, but they are just awesome short. Like I literally got my headphones on, I was waiting for this call and I was like, “Oh, I've got a couple of minutes. OK. I’ll watch todays Ruby Tapas episode.” And because they are basically free, time wise you never feel bad just about watching one. So, that’s awesome. They are excellent Rails cast for Ruby. Awesome. Go check that out

And finally, since we discussed all kinds of awesome science today, this is one of my favorite science videos. It’s in HD on YouTube and it’s totally worth it. So just blow it up full screen and watch. It’s a slinky video. They stretch slinkies out and then let them go. And you see in full HD what happens. They film it really slow motion and you see what happens to a slinky when you like stretch it out and let it go and if you do not know, that's why you have to go and watch this video because Physics and weird and this is an example of why Physics is weird. So, go check that out.

[laughs] Awesome. Alright Josh what are your picks?

Well, I was going to pick Ruby Tapas. Right, James?

[laughs] I know, I totally forgot. It’s so funny.

I told Josh…

Josh doesn't think it’s funny.

He’s laughing. [laughter]

Moving right along, it’s OK. [laughs]

[laughs] I'm sorry, I forgot and it was like I was half way through it I was like, “Oh yeah. I said Josh’s.”

OK so I don’t have a programming pick this week [laughs] but let’s see, I do have a life hacking pick. This passed around on Tumblr last week or so, there is a Tumblog called “shialabeowulf” who posted “99 Life Hacks to make your life easier!” And it’s a bunch of these like image macros, you know, meme things, pictures with text on top of them explaining… I counted 98 things but, it’s supposed to be 99 things. Which are things like, you know, using a comb to hold a nail when you are hammering, but like bore them up to everything which you can imagine. So, it’s pretty clever; A lot of it. I would say that the one life hack for putting your coaster on top of your drink in a bar to reserve your seat and keep them from taking the drink away, isn’t something that you should do anymore because there is too much that goes on with people getting in bars, so never let your drink out of your sight in the bar. So don’t do that. Other than that, the hacks all look pretty amusing, if not awesome. So, there's that.

Also, I saw the movie “Looper” last night. Or I'm about to see it last night. I'm not sure [laughs]. It was pretty awesome instead of the paradoxes of course. It’s basically impossible to do a modern time travel story without doing paradoxes because nobody writes time travel stories the right way. Because the only time travel stories you need to read are by Robert A. Heinlein and he wrote two stories. In 1940 he wrote “By His Own Bootstraps” and then in 1959 he wrote “All You Zombies” or that's a newer publish, I think. But those are the stories that you should go read and after you read them, you will pretty much be done with time travel stories.

Actually, there was one good time travel movie, “Primer”.

TIME COP! Oh, wait.

Yeah, I don't know if I saw that one.

Primer, yeah I didn’t see that.

Oh, you guys have to see it. It’s well done and completely baffling.

OK. Maybe I’ll check that out.

It’s one of those movies that’s very carefully put together and then people have like come along after the fact that and done elaborate flow charts to try to understand what happen.

[laughs] OK.

There's actually an XKCD about it. It’s freakin hilarious.

While I'm interjecting, James, slinky pick reminded me of I do have a non-programing pick and it’s a video about Hexaflexagons and that’s all I’ll say.

That video was awesome. That needs to be in our picks, yes. That was violet blue right?

Was it?

Is that? Oh, not violet blue. What's her name?

Violet Heart? Vihart?

Yes, Vihart. Yes, yes.

All right well, I'm going to go next. I just have one pick and that is Tweetbot for Mac. It was just released in to the Mac App Store. I think its $20. I have heard a few people complaining about price but it does basically everything that I wanted to, so I can’t complain. So you can go check it out. And you know, buy it if you like it. Just a lot of things you can set up multiple columns or you can have just one column or you can set it up to check multiple accounts, just super awesome. So that's my pick. It saved me a bunch of time and its made it easy for me to keep track of what is going on with the shows while we are going in tweets and stuff and things like that. So go check it out and we’ll let Glenn do his picks.

OK. Well first, I made the comment about Avdi’s pick about 7 Patterns to Refactor Fat ActiveRecord Models was written by a co-worker, I decided to check my work after I made that comment and I got confused somehow. So, it’s not but, now I wish he was.

So I have three picks. The first, so again my talk one of the things I talked about for ways to learn to deal with complex issues, was to study what are called “Wicked Problems”. And lot of people hadn’t heard of this idea. It’s something that comes out of sort of policy planning and things. And unfortunately, most of what you can go read about Wicked Problems tends to be fairly old. But recently, a guy named Karl Schroeder, who is Canadian science fiction writer and futurist wrote 3 blog posts about wicked problems. And sort of with the modern take on things we are facing today that fall in to that category and they are wonderful. And I’ll kind of sneak in another pick here, if you haven’t read Karl Schroeder’s books in particular his Virga Novels, they are well worth it.

The second pick I have, I mentioned when I gave my talk that the idea for this talk probably wouldn’t have occurred to me if it weren’t an election year in the US. Because I have spent a lot of time this year anguishing over both parties over simplifying complex issues and our terrible media helping them do that. And so, I'm going to recommend a book by a Princeton philosopher named Harry Frankfurt and the book is called “On Bullshit”.

And it’s an accessible but serious work of philosophy. And the friend who recommended this book to me, Tim Berglund in the email, he describes it like this, “If it does to you what it did to me, you'll never hear a politician in the same way again. The author argues that truth and false that are probably not right categories to use to evaluate political speech and other kinds of speech. And that this is a far worst thing than lying.” So it’s really terrific.

The Ghost Map”, where he tells the story of the origin of the German theory of disease with the famous cholera epidemic in London and everything sort of weaves them to a story about where those beliefs came from and what the new beliefs changed in society and just really tying past and future together at that one nexus moment. And finally where good ideas come from are all terrific and he's just a master at taking super complex issues and describing them very clearly without shying away from the complexity at all. That's it.

Awesome. Thank you. Thanks for coming on the show again Glenn. Real quick, I wanna remind everybody that today, (“today” being the day this is released, not the day we are recoding) is the last day to get your Ruby Nuby submissions in. So, if it’s still Halloween, you still have until midnight or so. So go do it.

Bonus points for recording your video in costume!

Yes absolutely. Also, the Book Club book and I forget because it’s a long name.

POODR!

POODR.

Just go to Amazon and look up “Sandi Metz”. You'll find it. And then once again, go sign up for RubyParley on the rubyrogues.com website. Beyond that, we will catch you all next week. What are we talking about next week? Who do we have next week?

“Hexagonal Rails” with Matt Wynne.

Awesome.

Very cool.

OK.

Is that related to Hexaflexagons?

No, I don’t think so.

[laughs] You should have saved it a week, whoever put that.

[laughs] OK.

Now I got to go find some adding machine tapes so that I can build one of those again. [laughter]

Awesome.

OK that's all folks!