The Rogues talk about object oriented programming with Jessica Kerr.

Special Guest: Jessica Kerr.

 

I was reviewing your video in chipmunk mode.

Watching you gesture with your arms was particularly funny.

Hey everybody and welcome to Episode 115 of the Ruby Rogues Podcast. This week on our panel, we have Avdi Grimm.

Hello from insomnia.

Josh Susser.

Hello from San Francisco, that suburb of insomnia.

I’m Charles Max Wood from DevChat.TV. And this week, we have a special guest and that is Jessica Kerr.

Hello from Saint Louis.

You want to introduce your self really quickly since you haven’t been on the show before?

Sure. I’m a Java Developer for 12 years and more recently, I do Scala in my day job. But at home, given the choice of what language to play around in, I like Ruby. Since I’m learning Scala and working in Scala these days, I’m very into functional programming. Ruby, I think, is really wellsuited to functional programming and the functional style has a lot to offer Ruby. And I really wanted to speak at Ruby conferences because, so when I went to CodeMash and some of the more general conferences, I learned that Ruby people are by far the most awesome. So, I learned Ruby and talked about functional programming in Ruby, and I think it’s resonated with the community, which is why James Edward Gray asked me to be on the show. So, I’m very excited.

And I know that James is extremely disappointed that his travel schedule didn’t let him be here while you were on the show. So, let me just whine on his behalf. Aaaaaahhhh! [Laughter]

You’re good at that, Josh. [Laughter]

Lots of practice. [Laughter]

That’s great. So Jessica, James and I and a couple of people saw your video from Ruby Midwest that you did earlier this year. And James is like, “Okay, we’ve got to get Jessica on the show.” And I’m like, “Okay, I’ve got to get Jessica speaking at GoGaRuCo.” But you’re at Strange Loop. So, oh well.

Yeah. I was really disappointed, it was the same day. And then, I told some of my Ruby friends and they were like, “Oh my god. You could be speaking at GoGaRuCo. What are you thinking?” [Laughter]

Yeah, so listen to your friends next time. Anyway, the video was great. I want to say that I’ve seen a fair number of talks that are about OOP and functional programming and comparing and contrasting. Yours is the first talk that I’ve seen that is less about who would win in a fight and more about what can we learn from functional programming that can make our object-oriented programs better.

Right, right. There’s nothing wrong with object-oriented programming.

So, great job.

Thanks.

Oh, we love all those talks where people go, “Immutable state. Aaaaaaaahhhh!” [Laughter]

You know, I totally wanted to do that. I totally wanted to have a, “This is how you should do immutable things in Ruby.” And then I tried it and it was so hard, I decided no. [Laughs] No, you shouldn’t try. By all means, don’t mutate things in Ruby. But trying to make it impossible to mutate them -- Ruby as a language, it’s like the whole thing. If you have an omnipotent god, can he make a rock that he cannot lift? Well, if Ruby is the god, no, you can’t make a rock that you cannot lift.

You just choose not to lift it.

That’s an excellent analogy.

Yes. But can you make a god object that is so big that it can’t contain itself? Sorry… [Laughter]

Yes, if you run out of memory.

Okay, now we’re going really meta.

You can run out of memory.

Well, if you think about when they invented object-oriented programming in, I guess, the 70’s, functional programming had been around for a decade or so at that point. They decided, “Oh, there’s something we need. We need to invent this new thing because the way we’re doing stuff in functional programming isn’t really working out for us in all the ways we want it to.”

Right. I think the world was not ready for functional programming back then.

I think that there were things that were problems with it that they solved in a particular way with object-oriented programming. Now, it’s decades later and everybody’s advanced more. So, I think that it’s definitely, there’s a lot to learn on both sides, I think.

Absolutely. OO is totally useful.

So, I like these synergy kinds of viewpoints and I like that you had just a small number of points that I thought made a really good story. So, do we want to talk about some of those points? You had data in/data out, functions are data, errors are data, but nil is not.

Sure.

Oh, the nil point was such a good point.

So, this data in/data out, I think that’s what most people think about when they think about functional programming.

I think that is the most important part. I think that is the meaning of functional programming, from what I’ve grasped of it so far, and I’m not an expert. But the idea that a function is evaluated to its return value and that’s all it does, I think of it as evaluation over execution. So, if you have a subroutine, meaning a method that has some side-effects that affects the outside world, it writes to a database or it prints to the console or it reads some input, that is executed. One statement after another is executed and things happened in particular order and that’s important, as opposed to a data in/data out function. It accepts its input, it produces its output, and it doesn’t do anything in between. So, you can call it once with the same input and output and just substitute that from then on, or you can call it a million times. From the outside world, you can’t tell a difference. That is evaluation. It’s like simplification in algebra sense. When you think of programs in those terms, as data flowing through functions that evaluate it into more and more interesting data, and eventually into information, you get a different view of the program than if you look at it from the outside and see the order of its effects on the outside world, which is an execution perspective. It all starts with those data in/data out functions that don’t affect anything around them.

Okay, great point.

Yeah, the one thing that really hit home for me when you were talking about the different kinds of functions where you have the functional data in/data out and no side-effects then you have the one where you call in and it changes the state of the world or the object somehow, then you have the other one where it effectively poops out some side-effect, I guess.

Yeah, yeah. [Laughter]

It’s too bad David’s not here, because he could have run with that in ways I can’t even imagine.

Yeah, that function crapped all over my data structures.

Well, that’s what it was supposed to do.

But yeah, I realize that I do blend a lot of those a lot of times in a method or what have you, a proc or whatever. Splitting it off like that so that I know that it’s a no side-effect, I put this in, I get this out, those are extremely easy to test. Then the rest of it is just instrumenting. But if you’re doing that, I think you said that 95% of the time, it’s just a data in/data out problem anyway. So, you can simplify 95% of your code by moving the side-effects into something that gets called somewhere in the chain, but only where you need it to occur. Then you can test the rest of it in isolation and make sure that it does exactly what it’s supposed to do and not worry about how it’s affecting the state of things.

Absolutely.

In my current software I’m working on right now, I’m doing exactly that. We’re writing a data converter pipeline and it’s really great writing all the little conversion steps as functional pieces.

Sweet.

Yeah. It makes them easy to test in isolation. It’s really easy because we’re converting stuff from one database to another database, which is all about side-effects, which would make it really hard to test all the pieces if we were always having to be coupled to the database. But we can test all these little pieces in isolation and it’s really fast. So, yey!

So, I guess I have a couple of questions here. One is a lot of times I’ll have objects where I’m calling whatever it is that transforms the input into the output, but the input is actually some state in my object. In the Rails example, you have your attributes. Sometimes, it’s like a phone number and I stripped all the non-numeric characters out. But when I want it back, I want it to put the parentheses, or the dashes or whatever, into it. So, I’m not actually changing the state of the attribute, but it is actually printing out the string that I need for the phone number. Does that qualify in your mind as the functional sort of thing? Or since it’s not directly taking in input as a parameter, but rather is pulling something off the state of the object, is that less functional?

Well, you can find a functional piece in there. In that case, I like to make a little method called format phone number or whatever and from within the object that contains the phone number attribute, pass the attribute into the format phone number method, and then get back the formatted string with the parentheses. That way, it’s like that method could be on the object, without passing any parameters into it. But if you put it on there and maybe you have a different property that wraps it, one way or another, you’ll have a functional method that accepts the phone number as a parameter and you explicitly pass it in. It makes it a little more clear what that function is doing, then you can test that function. You could put it in some other module, if you wanted, and test it without having the whole big object with the phone number attribute. So, I think there’s a functional component in there, but only if you call that out into a separate method into which you pass the phone number.

Okay. I guess my next question is, and you guys feel free to chime in on this too, but then that job or that function isn’t necessarily part of the job of the class that that object belongs to. Then should it be called out into its own module?

You can. If you ever use it anywhere else, then you totally should. Part of the value of making it functional in that way, of making it accept input is that you have the option of calling it out. If this is the only place you ever use it, then you don’t have to give yourself that overhead. But there is value in moving it out, as soon as you could use it from another place.

Yeah, that makes sense.

You see that a bit, or maybe a lot even, in functional object-oriented languages. I guess maybe even Scala has something like this. I’m not too familiar with Scala. But where you can grab a method implementation that’s just sitting around and then bind it to an object. In a functional world where functions are all just data in/data out and they don’t have to worry about external state, then binding a method into an object is something that’s totally reasonable because it’s never going to do anything to the internal state of the object. It’s just a convenient place to attach the method.

Oh, right. And you get the invisible parameter this when you have a method.

Yes. Right.

Or self in Ruby, no?

Yeah, self in Ruby. Given that perspective, that sort of binding makes sense. Avdi has, I think, done a lot of crazy stuff around how to find methods in method dictionaries and bind them into other objects and things, which always makes me cringe when I see it. [Chuckles]

Yeah, does it make the code more clear? It’s always a question to ask.

Yes.

I don’t really like to use the invisible parameter of self. I like to explicitly pass in the fields that I need. That way it’s clear which fields that method needs access to. If you ever split that class up into multiple classes, then you know where that method belongs and which fields are necessarily together.

Right. So, I want to ask about a concept in OOP where we talk about command versus query methods. Are you familiar with that?

Is that like ask versus tell?

Yeah, very much. We talked about this with Sandi Metz when she was on the show about how it’s useful to distinguish between methods that can modify state in the object and methods that you can call them any number of times and all they do is return the same value. They don’t have any side-effects.

Now, you have a distinction there. You have a distinction here. There are two kinds of methods that you can call any number of times. Ones that will return the same value at all times and one that will return whatever value is currently in [inaudible] object.

Yeah. So, you have the query methods, which all they do is return a value, they don’t side-effect anything. Anyway, one of the side-effects of is that they’re idempotent. You can just keep calling them forever. Sorry, keep going.

Okay. Yeah, they’re idempotent but they’re not necessarily referentially transparent, if it’s a mutable object.

Right. Can you define referential transparency?

Yes. That’s a great question, because I’m sorry I used that phrase, but you started it with idempotent. [Laughter]

Okay. I’ll define idempotent if you --

It was there, two of the words.

[Laughter] Okay.

I’ll define idempotent if you define referential transparency.

Referential transparency, deal, deal. But first, I’m going to complain about these words because one of the evils of functional programming, in my opinion, is it takes all these words like referentially transparent and monad and first class, and those are actually the pretty minor ones. Heaven forbid, you get into the bind and return methods inside of monad which do nothing like what we think of binding or returning. They’re in there. These terms exist for historical reasons that come out of math. If you don’t approach functional programming from a category theory perspective, then these words just get in our way. So, data in/data out functions are both idempotent and referentially transparent. What referentially transparent means is at any point where you see a function call, you could immediately substitute the output of that function for those inputs, right in the code there, and it wouldn’t make any difference. If the function had some effect like printing to the console or printing to the log, that would not be the case because it would matter whether you actually called the function or just got its return value. So, referentially transparent functions, it doesn’t care whether it actually calls the function or if it just happens to know the right return value for that input and substitutes it.

So, if you move the call to print to the log, for example, out to its own function, then what’s left could be referentially transparent because it has no side-effect at that point. All it does is return a value.

Yes. And the same value for the same input every time.

Right. Well, that’s idempotence, isn’t it? Where you put the same thing in, you get the same thing out, regardless of state?

That’s a question for Josh.

Oh, so idempotence.

Definition, Josh.

Yeah. So, idempotence is that you can keep doing the same operation on the same data and it won’t ever change. So, taking the absolute value of a positive number is an idempotent operation because it’s always the same positive number that gets returned. Having functional query methods on objects are by definition idempotent, because they’re always operating on the same input state. Therefore, they’ll always return the same output state. But probably they’re only idempotent from the perspective that they’re working on the same self, the same object because the data that’s flowing in, it’s not the data that’s flowing out for the most part.

So, Josh…

Sorry. I was just saying if you’re…

Oh, so in general, I don’t think it’s a terribly useful thing to keep in mind as you’re programming up productivity apps or websites. I know that the chef guys, they talk a lot about idempotence in their server configuration operations. They say, “Oh, I can run this configuration recipe,” or I guess that’s what they call them. It’s all cooking, right? I can run a configuration recipe in this cookbook and it will always have the same result. Even if nginx is already installed on the server, if I run the install nginx recipe, it will leave the system alone. It converges on one thing. It won’t mess it up once you’ve gotten to where you want to be.

So, I think from your understanding and mine, though I don’t guarantee mine is completely mathematically accurate, idempotency means you can call it over and over and over again and not worry about that, while referential transparency says, “You don’t have to call it over and over and over again if you cache the result.”

Right, yes. From an OOP perspective, the referential transparency thing is pretty cool because you can automatically memoize anything that’s referentially transparent.

I love the word memoize, except that it’s incredibly hard to type in anything with autocorrect. [Laughter]

Yes. There should be a whole website just devoted to memoize autocorrects. So, if people are not familiar with memoize, the easiest way to think of it in Ruby is you have an instance variable and then ||= the or equals operator and then some_calculation. @memoized ||= some_calculation. And that says okay.

That would be good, the Ruby implementation.

Yeah. So, I’m typing this into the chat room so it will show up in the notes.

In Java, I would memoize is most easily implemented with the guava cache builder, which is a map or dictionary that will calculate its values based on its keys, but it only calculates them once and then it caches them. So, memoize just saves the result for a particular input and then you never have to calculate it again.

Yeah. That’s the idea.

Okay. So, you had a bunch of other things in your talk as well. Functions are data, errors are data, nil is not data.

Yeah, can I lead into that with memoization? If you memoize something as false with that or or =, that ||=?

Well no, that’s not going to work. That’s not going to work in Ruby.

Yeah, that’s what I’m saying. Part of that is because we use nil and false interchangeably among nil being used as substitute for other meanings. I really like that point in your slides.

The point where nil means a million different things, therefore it means nothing?

Yeah, that one.

Or at least nothing in particular.

Right, right. You can’t be sure what it means because it means something different in every context and it could mean one of two different things in the same context a lot of times. That’s something I’ve really picked up from Scala and even when I go back and do Java and Ruby. Well, Ruby has those symbols which are awesome, but always try to return something meaningful instead of null.

Good for you.

Yey! [Laughter]

That’s basically the entire point of the confident Ruby book that I’m just finishing up, is stop using nil for everything because its meaning is entirely contextual.

If it has any at all.

Yeah, if it has any at all. It might not have any. It might be accidental. But if it has any meaning at all, you have to go read the documentation to find out what nil means in this context, because it means about 15 different things depending on where you’re at.

It’s so easy to accidentally propagate.

Yes.

And usually, the number one thing that using nil does is it introduces conditionals in the code that consumes it.

You think conditionals hurt people?

Well, they’re not necessarily evil, but they make a lot of code more difficult to work with in that it’s harder to refactor and harder to tease apart into separate responsibilities.

Oh, conditionals. We tend to replace those in pattern matching with functional programming, but that’s not really any different. It’s just a lot prettier.

In OOP, we replace conditionals with polymorphism.

Right.

That’s not really any different either. It’s just prettier in that it’s easier to separate the pieces that you want to use independently in maybe different classes or inherit separately. I was looking at some code like this yesterday, where we had a case statement that did a bunch of things depending on what was in a string, but the strings are very regular. So, it was like, “Oh, we can just take the last word in the string and send that as a message to self. Then what we have is a bunch of small methods that in subclasses are easy to override in different ways.” We could have done that by pulling all these little bits out of the case statement and turning them into methods even without getting rid of the case statement. But then, all the logic for how you do those things is buried in that case statement in the superclass and a subclass that wanted to do things a little might have a lot more trouble. But god, I hate talking about code over the air. [Laughter]

How to obscure understanding. The thing that I really liked is you said errors are data. That’s, I think, an uncommon perspective, so I’d love to hear more about that. CHUCK: I think I read a book on that once.

Yes. Only one?

[Laughs] Fair enough. Anyway, you were saying?

Okay, so errors are data. Right. There’s this tendency in programming, and in all of life really, to say, “We will prevent failures from happening. We will not let them happen. We will fix all the things and our program will be perfectly under our control.” That’s just not how the world works. It’s not even how computers work. In general, when we don’t have control, just let go and instead get visibility. Get reporting. Find out when things are happening in the way you don’t want. Once you have that metric, then you can influence the system that you don’t have control over. It totally works with people. Like in some issue tracking system, the management will say, “Oh, well you must not be able to enter an issue unless it has a sponsor and a customer and blah, blah, blah.” Well, okay.

Now, I’m just going to do the work without entering the issue. If instead, you let me track my work without jumping through a bunch of hoops for a sponsor that I’m not going to go get right now, and instead track that and measure how many people are working on something that doesn’t have a sponsor, then you can influence this. You can make it part of people’s reviews. You can go get the sponsor yourself. In a way, when we ignore errors or just blow up when they happen, we’re losing information that can help us make the system better. That was kind of a tangent. But it’s one I feel strongly about.

I think that’s a cool way of looking at that.

Given our last segment of conversation, I take it you think that rescue nil is evil?

What does rescue nil do?

Well, rescue catches an exception and then returns nil.

It throws away an exception.

Yeah.

Oh, that’s terrible.

And then reassigns the value as nil.

Yes, that’s evil. That’s horrible.

You must hate everyone who uses your code. In Java, it’s the empty catch block, is the same effect.

Yeah, it’s the equivalent.

And I get really pissed when I see those too.

It’s interesting. I think we need either some better practices or some better constructs for failure handling in Ruby. I wrote Exceptional Ruby and the more time that goes by since then, the more I want to revisit some of that stuff and talk more about these philosophies, what to do with errors.

Because one think I see a lot of, I don’t know where you fall on this, is I see a lot of either/or, especially in language design discussions, a lot of either/or. Errors should be return values and they should be, you should just be checking your return values and maybe the language should force you to check your return values, which I have yet to see a language that actually does that as opposed to claiming it does that. But anyway, versus errors should be exceptions, period. For me, it’s a much more contextual thing. If I’m not sure in a method what to do with an error, not absolutely sure what to do with one, maybe it’s going to depend on where my method’s being called, then I want to provide a way to return that information in some form about that condition without necessarily forcing an exception. But on the other hand, if I am at the level that knows exactly what to do in a particular situation, and I know that it’s potentially fatal, then I do want to raise an exception.

Right, I agree. Exceptions are for exceptional occurrences that we don’t expect anybody to be able to recover from.

Yeah, but it’s very, very important to make a distinction there. We all say that, but it’s important to make this distinction that it depends on your context. It depends on the level of the abstraction that you’re in. At one level, a low-level system call that opens a file, it is a perfectly ordinary circumstance for that file not to be found. That’s not an error. That’s just one of the possible outcomes of that system call. At a high level in your application where the application is starting up and there is a mandatory configuration file that it opens and reads, if that mandatory configuration file is not there, that is an exceptional circumstance. It’s worth killing the program over because it can’t go any further, let’s say, in this particular application. It’s like a missing file is both an ordinary circumstance and an exceptional circumstance, depending on the level of abstraction that you’re at.

Yes, and I think that the file opening routine there should be returning the piece of information, because when the program is trying to open its configuration file and it’s not found, what happens should not be letting an exception propagate from lower down that just says, “File not found, blah, blah, blah, blah, blah.” No. The information that the user needs to recover from that error is not in the file opening routine, it’s in the configuration file accessing routine. That needs to throw an exception because it’s not recoverable and that exception needs to say, “I am unable to run without a configuration file. This is where you should get it and it’s called blah, blah, blah,” actual information that the user can use.

Yeah. It’s tough though, because if that system call, let’s say there are several levels of call in between that top-level startup code and that system call. If the system call doesn’t raise an exception, then there’s a chance that important information about the error may be lost along the way as those intermediate methods don’t bother to pass it along.

This is where you get into the either idea. The default error handling method for me is either, is I’m going to return you something or I’m going to give you an error. The nice thing about those is, at least in Scala and of course it can be implemented in Ruby, it can propagate up very much like an exception with only very minor generic code in each of the methods in between. So if you get an either back from something and you return an either to your caller, then all you do is the functional term which I don’t like, is map on the data value in that either that continues your processing. But if you’ve got an error in that either, and oh my gosh you’re right I totally need this like this. If you’ve got an error, then it just passes right back up to the function that calls you. Not unlike exceptions, except it’s using the regular stack and every level in the layers of calls on the way down has the opportunity, not the obligation, but the opportunity to add information to the error that was returned.

Yeah, definitely.

So, I have a question that will change the topic. Is there any more you guys have to say on this before I ask my next question?

Go for it.

Errors are not evil. Embrace them.

Errors are not evil.

Right, right. Embrace them. Embrace failure.

There we go.

Learn from it.

So, my next question is in your talk, and I know you only had a half hour or so, you could really only give the one example with map/reduce. Are there other examples of good ways to use functional programming in Ruby, other than transforming data?

That’s a good question. The collections methods of the internal iterators of map and reduce and filter are the easy wins. They’re the easy, “Look, you can pass a function as a value and it does this.” There’s a lot more. I don’t where to point you to examples on Ruby yet, but passing functions as values, it’s like the strategy pattern in OO. It’s really painful in Java, until we get Java 8 and then we have lambdas. Then suddenly passing how to do something becomes really easy. In Ruby, it’s already pretty easy and you have the idiom of blocks and you do that some. So really, it’s a matter of once you really get your head around map and filter and the idea of passing the how, opportunities crop up. But no, I’m sorry. I don’t have any examples to point you to.

That’s okay. You’re leading into the other question that I have and that is in your talk you said that you could do functional programming in Ruby if you use lambdas and call rather than yield with blocks or procs. I’m wondering, what is the difference between lambdas and procs that make it so that one is more functional than the other? I understand some of the differences between lambdas and procs, but I can’t really point to one and say, “This has got to be what she’s talking about.”

I wrote this up in a blog post right before Ruby Midwest and then since then I’ve forgotten it to make room for other things. But in the end, shoot, now I’d have to reference my blog post.

I can jump in here if you want.

Sweet.

So there are only a few differences between what we call blocks, procs and lambdas. Blocks and procs are basically the same thing, although I think procs will check arity on the inputs and regular blocks won’t. But a proc is just basically a block turned into an object that you can point at. A lambda is very similar, but the big difference is that lambdas, you can return from them any number of times because a lambda always returns from itself to the point where it was called.

Sorry, a lambda gets a layer on the stack, right?

Well, okay, so a block or proc, they’re basically synonyms for each other, they carry along the execution context of the place where they were declared, the actual method where you wrote the curly braces or the do end for that block. That never really goes away when you’ve passed a block out of it. When you return from the block, you’re actually saying, “I want to return from the method where I was defined and here’s the return value.” So that’s what happens when you put a break or return or whatever in a block. It’s operating within the context of the method where it was defined. Whereas a lambda is more like a full-fledged method that just happens to be anonymous. When you return from it, it’s returning to the place where it got called, not the place where the method that defined it was called.

That’s a great description, thanks.

So, it’s execution context, but they’re both still closures, right? They both still contain the context of everything that’s outside of it. It’s just that where it’s executing, the proc returns for the method that called it, in the other case it returns to the method that called it? JOSH: Yes.

Okay.

Right. And blocks are kind of a hack. They’re not super functional. Like Jessica said, they break a lot of your ideas about, I guess referential transparency. You can’t substitute the return value of a block for a block because it forces another method to exit.

What you’re saying is that Ruby having blocks is not, strictly speaking, functional.

They don’t behave like a function. They don’t get their own level on the stack and then return back right back from where they were called, like Josh said.

So can I ask you another question then, and this is more along the lines of JavaScript. So you can pass functions. Functions are first-class citizens in JavaScript, so you can do a lot of functional things with it. But a lot of the way that the virtual machines are built is it’s all event-driven. So if you’re doing a map/reduce, you do the map and then you trigger a callback to do the reduce. Is that callback a side-effect, strictly speaking, with regards to functional programming?

That depends what happens in the side-effect. Then a callback is clearly passing a function. So it’s functional in the sense that you’re using functions as first-class values, which is cool. But what does your callback do? If it has side-effects, then in the end it’s not strictly functional.

But if it’s just a mechanism to trigger the functionality in that function that you passed in, then it is?

So, I don’t know if it’s really useful to pursue this level of detail in distinction.

Okay. Yeah, it’s just me being curious and trying to understand the deep nuances.

Yeah. In the end, it’s a religious argument. That the observer pattern which callbacks are part of where you’re like, “Okay, 15 minutes later after you do your thing, have this side-effect.” Yeah, it’s not strictly functional because it has side-effects. Whoopee. It’s still useful and it’s still useful to think of passing, in this case, a subroutine as a value.

I really like that point too where, “Is it useful?” as opposed to, “Is it functional?”

Right, right. In the case of map and filter where you’re passing a function in, there’s an expectation there that that function is data in/data out, because whatever you’re passing control to when you hand it a function, it should be able to expect to call that function as many times as it needs in ways that you might not expect. So if you pass a function that has side-effects, you don’t know what you’re going to get. In the case of a callback, you do know what you’re going to get, because it explicitly says in the documentation, I will call this when I have a successful result. So there, there is an expectation that you can have side-effects in that method. In that sense, you just need to make a distinction of what is the function you’re passing for? Is it okay to have side-effects?

Okay.

Cool, cool. We had Gary Bernhardt on the show. Wow, that was a while ago. That was episode 67 and we’re up to 115 now, wow. Okay, so that was like a year ago. Gary was on this kick where he talked about a functional core imperative shell. I’ve seen Michael Feathers talk about something similar with objects above, functions below. It seems like you have an evolving perspective on this that seems pretty similar, that there’s a place for functional methods or functions within objectoriented programs and then there’s a place for the state changing mutability stuff going on.

Absolutely, because in the end if our program never changes the outside world, what good is it?

Yeah. [Chuckles] Right, you could just use referential transparency to substitute the exited successfully return code.

You can make websites. You can totally make static websites, because a web request is some information coming in that gets transformed to HTML going out. But if you never record any other information in a database, then your website never changes, and that’s pretty boring.

Yes. Okay, we’ve talked a bit about the functional part of things, but we haven’t talked about wetting that stuff with the state changing OOP side of things. If I’m building something like a state machine to represent the flow of some piece of data, like you’re publishing and article and it needs to go through it’s a draft, it’s being reviewed, it’s being corrected, it’s being published, there’s a whole state that comes along with that. Then you get to make different decisions about what you can do based on its state. If it’s passed through editing and it’s approved, then you can publish it. So how do you like to integrate those perspectives where you have this functional data in/data out no sideeffects with dealing with changing the real world?

Well, in this case, it’s like that database program that you’re talking about where you’re transforming data from one database into another. At some point, you have to get data out of the original database, then you do all your transformations, then you put it in. Your transformations are the functional core and you have an in at the beginning and an out at the end. That’s what I like my programs to look like. If the document that you’re talking about, the publishing, is a website then you might have a request that says, “Update the state somehow,” and at the beginning of that request, I’m going to get all the information I need from the database and from you. Then I’m going to do the transformation of that document into the new state. You can do state transformations and state machines without mutating objects, if you choose. I usually do choose. Then at the end,

Can you say something about how you do that? I’m sorry to interrupt, but it’s just

Sure.

How do you represent that change in state?

Whenever you call a method on the object that would change its state, instead of altering that object it returns you a new object that’s a copy of the old one but slightly updated in the way you needed.

Okay, so if I had a document object, I could have a status attribute of that object and then if I’m moving it from draft to submitted I would just change the symbol there in that data structure and then pass that on to the next function? Something like that?

I would return a new data structure, a new instance of the data structure, with a different value in that status.

Great. Good clarification.

And I would leave the other one alone. That way, if somebody else has a reference to the document in the previous status, it doesn’t change out from under them. There’s a weird word for this. This is one of the worst functional programming terms, because it means something completely different to me, to imperative programmers. That’s persistence. When you have a data structure such that when you go to make a change on it, you actually get a new version back and the old one stays around in memory, that’s called persistent. That’s a persistent data structure, which is totally not what I thought the first 16 times I heard a functional programmer use that term, because to me persistent means it goes in a database, it goes on the file system, it stays around when the program exits. So watch out for that, if you’re reading functional programming literature. Persistence just means I didn’t mess with your memory. I just created a new object in a new place in memory that I’m going to use from now on.

So, you persist in using that word.

I do not think it means what you think it means. [Laughter]

Thank you.

It’s like all the processes in Elixir Erlang. Basically, they’re a top-level loop except there is no looping in the language. What they do is they return a new version of the whole process state. Well, they don’t return, they recurse. They call themselves at the end with an updated version of their whole process state. That’s the way they move forward on state. It’s worth noting though that a lot of functional languages are heavily optimized to make that stuff really fast, make that style really fast. Ruby is not. For better or for worse.

Is that something that works better in JRuby? Have you noticed?

I haven’t played with it enough. Instinctively, I would say that’s damn near impossible to optimize that kind of stuff in Ruby, just because there isn’t a level of analysis that you can do where the VM can say this is never going to change. The whole thing with persistent data structures, since it knows that any given iteration of a data structure is never going to change, then a VM can do all kinds of optimizations based on the idea, like deltas. The new state is really just a delta off the old state. It shares most of its state with the old state. So yeah, I don’t know.

Basically you’re talking about Git as the internals of the language.

Yes, exactly.

Yeah, Git has a lot of parallels to this, totally, with this object storage. On the other hand, you can do things in code in Ruby like there’s that hamster library of immutable collections, then you can write your classes to share state between instances. So it can be done, but yeah, it is not native. As far as JRuby, garbage collection is a big part of this, because you wind up with a lot of little objects being created over and over. Efficient garbage collection is going to help, whether JRuby has more efficient garbage collection, it sounds good. JOSH: Well it definitely does versus earlier versions of MRI. I think MRI’s GC is definitely getting better. I haven’t seen any good benchmarks though, and we all know how good benchmarks are. [Laughter]

So, do we have anything left on our list here? Be lazy. Good thing we left that one for the end.

Ah. Yeah, functional gives you the opportunity to do some things lazy, because when you’re passing around actions, you don’t have to get all excited and take every possible action just to pass in the parameters. But the interesting part about lazy, well there are two things. One is just a practical method of when you have a lazy stream, when you’re reading from a gigantic file one line at a time instead of reading the whole thing in, then you can do tasks that you just couldn’t do without lazily processing that file as you need it. The other interesting piece that’s in terms of abstraction is more interesting to me. It lets you separate what to do from when to stop. You can establish what you want to do but then you only, for instance if you’re processing the lines in a file and doing some transformation on each of them, you only pull as many of those lines as you need to and you decide where to stop in a different piece of your code than the piece that reads from the file.

Right. So, you’re data processing logic is separated from your control flow logic.

Yes.

I love that you can do infinite lazy lists. Haskell has those. That’s great for what you’re talking about. There’s I think a simpler example in Ruby, for people who are familiar with Ruby, of just blocks are lazy evaluation. It’s deferred execution. If you look at a hash, you’re specifying a block, you’re doing hash fetch and you pass in a block as, “Here’s the thing to do if you don’t have a value.” And you don’t have to calculate what that default value is until you need it.

Ah. Yeah, that makes a lot of sense. In Scala, we would call that a by name parameter, because you pass in a block of code of how to get the parameter instead of pre-calculating it.

I guess my question is if you have an exceptionally large file, you gave the example 10GB, 3GB memory, is there a good way to do that in Ruby? Get this chunk of the file?

The lazy enumerable does a lot for you. In terms of the actual file I/O, I don’t know, I haven’t tried it.

You should.

What exactly are you trying to do?

If you were trying to read in --

Read in a file.

Yeah, a file that’s too large for memory.

You can do all sorts of streaming processing in Ruby. The csv stuff, you can stream through a csv file row by row and just pass in a block for what to do on each row and then stream the output of that into a file.

There’s streaming and then I guess there’s laziness, which you can often derive from streaming, especially with enumerators.

Okay, so we solved that one. [Laughter]

It’s one of those things that it’s hard to talk about in the abstract. It’s hard to talk about without code in front of you and it’s also really hard to talk about unless you have a concrete problem in front of you. I’ve done a lot of streaming processing and lazy processing, but it’s hard to just hand wave them and talk through it. I’m not really convinced that the lazy enumerators are actually a big part of it. They’re good for chaining. I feel like that’s what they add, is easier chaining. But the ability to do lazy stuff has been around long before the lazy enumerators came along. An enumerator is a lazy construct, not to be confused with enumerable. An enumerator is a lazy construct, always has been.

Right. In Java, it’s an iterator versus an iterable and it’s equally confusing. [Laughter]

Okay, so can one of you explain the difference?

Ooh, ooh, I love this one.

Go for it.

So, an iterator on an enumerator is a source of data. You can call next on it, but it’s stateful. Every time you call next, you’re probably going to get something different back. You can never go backwards. You’re always going to get the next thing and you get each thing exactly once.

Although you can call rewind on the ones in Ruby.

Wow.

I don’t have that.

Results not guaranteed.

Exactly. JESSICA: Yeah.[Inaudible] that’s a pretty difficult implementation.

I thought iterable was what you feel like when you don’t get enough sleep.

I don’t know. We can ask Avdi that one. AVDI: Oh, har, har. [Laughter]

Moving right along.

Alright, so an enumerable or an iterable in Java or in Scala is something that can give you a stream of data over and over. You can get enumerators out of enumerables as many times as you want. The enumerable has all of its data and you can process it over and over. The enumerator, just once.

Yeah. Although, it’s even a little bit fuzzier in Ruby just because a lot of exhaustible data sources I guess you could say are enumerable in Ruby. For instance, I/O is enumerable. It’s got each on it. But if it’s plugged into a socket, well you’re only going to get, whatever comes over that socket, you’re only going to get it once. You’re not going to be able to go back. But now I’m just being mean and coming up with obscure exceptions.

That’s okay. I’m learning something.

The important part to know, the way this has bit me, especially in Scala, several times is the interface on iterator and iterable is very similar. I can filter, for instance, over either one. But if I filter over an enumerator and say I take the first ten of them, then that enumerator is corrupted, it’s gone. I can’t perform a different filter on the same enumerator and expect it to basically repeat all its results so that I can filter over them again. But in enumerable, that totally works on. You can reuse enumerables, with the exception of this crazy socket thing. Enumerables anyway that behave as giving the same enumerator over and over, you can reuse those. But never reuse an iterator or enumerator.

Yeah, that makes sense. I think Java and I guess Scala draws a slightly harder line between those two concepts, but yeah.

That would be consistent with the whole type safety.

[Laughter] Yeah. [Inaudible] language.

I’m a little curious, because you only had a half hour. It seemed like there could have been a little bit more that you could have talked about. If there was anything else you could have added to the talk, what would you have added?

At the end of the talk, I talked about how to perform the data transformation with chaining and completely immutable and functional style. But I didn’t show any of the implementation of that. I’ve since coded it up in my 0.1 version gem, which I called aqueductron. A few months ago, I presented this at the local user group with construction paper slides, which were popular. And aqueductron was really fun. This is an example of, as Avdi points out, this is probably not going to perform well in Ruby, but we can keep things completely immutable and lazy as we pipe data into information. The best part is, I wish I could show you, it does ASCII art to illustrate the pipes.

Do you have a video of that or a screen grab or anything that we could look at?

I’ll post the slides, probably to my blog I guess.

That sounds great.

But yeah, if I could go further, I would go into the different ways that we can combine laziness and immutability to process data and information. The really interesting thing is when you take the data flow and you start mutating the flow based on the data. Then the code is actually changing itself, in my view. And still, it’s immutable, without changing its state, it’s changing the flow of the data through the code. I’ll write that up for you.

Yeah, that’d be awesome.

So I’ve got one last thing and that’s, you opened your talk with a Sherlock Holmes reference. When I was a kid, I got a book of the complete Sherlock Holmes collection, everything. I think the first story in it is A Study in Scarlet, and I think that’s where that little interaction between Holmes and Watson happens, right? Was that A Study in Scarlet?

Oh, is it? Because I looked for it and I couldn’t find it, but I remembered that story. It stuck with me for years about how Holmes just wants these irrelevant facts out of his head.

Yeah, where Holmes says I don’t care whether the earth goes around the sun or vice versa.

Oh, yes. Something like I have a limited amount of space in my brain attic.

I shall promptly have to forget that.

Yes.

Yes, exactly, which is why I don’t feel bad about not remembering the results of my investigations into lambdas and blocks, because I did the investigation. I wrote it up. I promptly abstracted it to always use lambdas when I have a choice, and then I let the rest of it go so I can learn something new.

Very good. Well it’s just like you remembered the important quote from the story, but you didn’t care which story it was in.

Yeah.

So that works too. But yeah, I love that. I learned that when I was a kid and that’s stuck with me for

a long time. What do I care about remembering?

Yeah, good.

So Jessica, before we wrap up, where are you going to be speaking?

In September, I’m at WindyCityRails doing Functional Principles in Ruby again. That one’s going to be really awesome. It looks like it’s going to be a great conference. Other than that, in August I’m at PyCon Canada and then at Strange Loop, I’m doing at workshop on Git. But Strange Loop is sold out so that doesn’t help you. Finally, in November I get to go to Sweden again and speak at Øredev on both functional principles and the counterpoint to that, it’s flipside which is object-oriented development principles taken to other languages, including Ruby.

Oh, nice.

Awesome.

Looking forward to seeing that one. Are a lot of those going to show up on video, do you think?

I have no idea.

Well, we can keep our fingers crossed. Thanks.

It seems to be more and more of a thing to record the talks and publish them, so we can hope.

Cool. Are we ready for picks, Chuck?

Yeah. We’re definitely at that time. Avdi, why don’t you start us off with picks this week?

Uuuuh. Sleep. I think it’s sleep. I seem to recall it being good. [Laughter]

Once upon a time I enjoyed it.

Nice.

Well done, sir.

Josh, what are your picks?

Okay. So my first pick is, I’m going to pretend to be James to do this pick and I’m picking the regular expression crossword puzzle site, which I actually introduced James to yesterday. But this is great. It’s crossword puzzles and the clues for the rows and columns are regular expressions. Then it’s a logic puzzle to figure out how they interact. The rows and the columns interact at the squares. Then you figure out what the crossword puzzle solution is. I saw that and I was pretty astounded.

That is the best criticism of regular expressions I’ve ever heard. [Laughter]

Terrible or awesome? You decide.

It’s a logic puzzle to figure out how they interact. Yes, programming is all about solving puzzles. I prefer to [inaudible] puzzles.

Pretty cool. Okay. And then in keeping with our philosophy that picks are things that made your life awesome, I’m picking JRuby. It’s probably been picked before, but this is the first time I’m using it in anger on a project and it’s going really great. Doing multithreading on JRuby is awesome. So yay JRuby and thanks to the whole JRuby team for all that great work. Then I have a pick that’s relevant to our topic. Alan Kay, we all know him as the man who defined the term object-oriented programming and invented Smalltalk and the entire world of computing that we use today, what he’s up to these days is he’s at this Viewpoints Research Institute I think is what it’s called. Part of the work they’re doing is this system called COLA, which is combined object lambda architecture, I believe, is what it means. I haven’t had too much time to dig into this, but if you’re a language nerd, this is probably terribly exciting stuff. What COLA is about is that you have the object side of things which is the mutable state and you have the lambda side of things which is the functional programming. They’re duals of each other. They’re just building a language architecture where those things lean on each other successfully and interact constructively rather than get in each other’s way. It’s all academic research right now, but it’s interesting to read about. That’s it for me.

Awesome. Last week, I had a pick all ready to go and I forgot to pick it. It would have been much more appropriate last week than this week, but we did talk about functional programming. This week I’m going to pick the Mostly Erlang Podcast. It’s hosted by my friend Zach Kessin and a bunch of other people. It’s formatted like Ruby Rogues. We’ve actually inspired a few podcasts of this format and that’s one of them. It’s pretty good. They talk a lot about Erlang. They also did an episode on Elixir. So it’s interesting to see it coming from the Erlang side as opposed to the Ruby side and what their take on it is. So that’s the only pick I really have this week. I’ll throw it over to Jessica to do her picks.

Alright. I’ve got an article and a book and someone I think you should follow on Twitter. The article is on firstround.com and it’s Kris Gale talking about the hidden cost of every feature we add to an application. The title of the article is ‘The one cost engineers and product managers don’t consider’ and that cost is complexity. Because if you’ve ever noticed, as we write our applications, it starts off and we’re super-fast and we can do thing really quickly and then the bigger our application gets, the harder it is to add what seems like it used to be easy functionality. That’s because every feature that we put in increases the complexity of our application. So my new theory is whenever we estimate a feature, there’s the implementation cost, the upfront cost of getting it in there, which is all anybody thinks about now. I think we also need to consider a complexity cost that affects the multiplier that we will apply to every estimate in the future, because this feature has just made our lives that much harder forever and ever. So great article, there’s the link. My second pick. If you are serious about learning functional programming, the best book I found is by Rúnar Bjarnason and Paul Chiusano and it’s called Functional Programming in Scala. The Scala part is ancillary. Rúnar uses Scala to make the points, but he’s really teaching the philosophy of functional programming, of immutable state and of everything is an evaluation. I think it’s a fantastic book for that, the best I’ve seen. It’s in Early Access, but it’s great.

Awesome.

Finally, if I were going to follow one person on Twitter, it would be @flowchainsensei, because he posts his own articles and links to articles that are really about having a life within work, personal growth within your team, and this is why I think that our agile development is going to change the world. Because slowly, we’re coming to recognize work relationships as valuable relationships.

From there, we can only get better.

Awesome. There’s a scrum master I want to send that article to and just watch his head explode. [Laughter]

Anyway, well thanks for coming. It was an excellent conversation and Josh didn’t even have to defend OO from the advances of functional programming. JESSICA: I am not against OO.

That’s not [inaudible].

I came in here expecting a war. I’m disappointed. [Laughter]

We took away Josh’s launch codes.

Well, you should have told me. If you wanted to fight him. No, no, he makes too much sense.

Darn logic.

Alright. Well, we’ll go ahead and wrap up the show. Thanks again for coming and we’ll catch you all next week.

Yeah, thanks Jessica.

Thank you very much.

Thank you very much.