122 RR Daemons with Kenneth Kalmer

Kenneth Kalmer comes on the show and discusses the ins and outs of long running processes and when and how to manage daemons in Ruby.

Special Guest: Kenneth Kalmer.

 

Can you say, “I believe I broke his jaw”?

Who, me?

Yes, you. Of course, you.

I believe I broke his jaw.

Ah! It’s so brilliant.

Okay, now you have to explain that.

I think at the first Mission Impossible movie, there are some really bad guys from South Africa and one of them says, “I believe I broke his jaw,” and it’s so funny. [Laughter]

Okay.

Hey everybody and welcome to episode 122 of the Ruby Rogues podcast. This week on our panel, we have James Edward Gray.

Good morning everyone.

Josh Susser.

Everything done for the first time unleashes a daemon.

Katrina Owen.

Hello.

Avdi Grimm.

Hello from Pennsylvania.

I’m Charles Max Wood from DevChat.TV. And this week, we have a special guest and that is Kenneth Kalmer.

Hi from springy Johannesburg.

Wow! I don’t think we’ve had anyone from that far south before.

I’m glad to be here and to represent the whole southern hemisphere.

So, you’re new to the show. Do you want to introduce your self really quickly?

Sure. Like you said, my name’s Kenneth Kalmer. I’ve been doing Ruby for seven or eight years now. I got my first Rails app up before migrations actually ended in Rails. And I fell in love with it completely and from there just tried to apply it to everything. In relation to today’s talk, that included trying to run as much Ruby in our infrastructure doing all kinds of crazy stuff and learning about daemons and how to do all these long-running tasks and processes that need to be up for months on end without any human intervention. So, I’ve just been doing that forever and absolutely love it and can’t see myself using anything else.

So, we need to settle it right there. He said daemon. Is it [day-mon] or [dee-mon]?

Can I object? [Laughter]

Can you what?

Object. I object, your honor. It’s [dee-mon].

Oh.

Actually, I think in Latin, it’s actually [die-mon], is what we landed on.

Yeah, like maestro.

Yeah. But I think the official wording here, Kenneth actually gave it to us. But I say [day-mon] too, by the way. So, I’m sure you’ll hear me make that slip a bunch. The word demon is an alternative spelling of daemon and is pronounced [dee-mon].

So you say [day-mon], I say potato.

Let’s call the whole thing off.

[Chuckles] Let’s call the whole thing off.

Anyway, so this conversation came about because Kenneth listened to our episode that we recorded live at the retreat. And I think Josh, was it Josh that said that we should do an episode on daemons when we were talking about some of the other uses for Ruby that aren’t Rails?

Yeah, which is I think a huge topic, doing background processing and all the stuff that’s not directly connected to user.

Can we get a definition?

Of a user? [Chuckles]

No, I think I know what those are. How about of a daemon? Please do not relate it to elder gods this time.

Kenneth, please tell us, what is a daemon?

So, I’m taking this from a presentation from Thoughtbot’s Tammer Saleh. I hope I pronounced that right. Old presentation from 2007. He says daemons are less than gods, which is the OS and the kernel, but more than mortals, which is the programs. And then from the UNIX FAQ, it says a daemon process is usually defined as a background process that does not belong to a terminal session. I think that that’s the more accurate one. It’s a process that simply has no standard in and no standard out and it’s completely detached. It runs in the background. The only way to get to it is through ps or activity monitor. And you’ll usually see a machine is full of them doing all kinds of mundane things to keep your OS ticking along.

I have a question. Could you send a signal to a daemon?

Yes.

Yes, you can.

Yes, absolutely.

Actually, it’s very common. In something like Unicorn, for example, it often runs as a daemon and sending signals is the primary way to give daemons commands. Not to send them data, but to give them commands. So, in the case of Unicorn, you can send it a signal to stop or you can send it a signal to do its very clever no downtime reloading where it forks a new master and starts a new set of connections and then eventually cycles the old master out.

Another great use for signals is if you’ve got log rotation. Once one’s log rotates, finish rotating the logs, just signal the daemon so it closes the file handle to the log file and reopens a fresh log to get you rolling that way around. And you can even use signals to toggle between logging labels. So, you can have a normal logging label and give it a signal and switch it to a debug logging label to see what’s going on instead of having your daemons running production, just logging away endlessly.

Damn it, we’re a few minutes into this talk and I’m already learning awesome tricks. [Chuckles]

Okay. So, we’ve said what daemons are. They’re background processes that are not connected to a terminal session or things that sit in the background without user interaction, no user interface. And we’ve mentioned a couple of things that they get used for. What are some other kinds of things that daemons usually get used for?

Mostly, I’d say performing tasks. A daemon could probably be fully autonomous. So, they could be monitoring a system, gathering stats and reporting it elsewhere. Say like New Relic has got a server agent that you can install that just sits there, or like a chef client or puppet that will sit and at certain intervals, trigger off and do some kind of work. Another use would be a daemon that sits and waits for input over a messaging medium. So, over AMQP or over Jabber, for instance, and they’ll get a command. They’ll perform some kind of work and reply off and do something. But more often than not, they’re just sitting there listening. 100% idle. Either at a set interval or on some external stimulus, they’ll get a job done. Does that make sense?

Sure, yeah. I was looking for some particular examples of kinds of things that people use daemons for.

So, in the context of daemon-kit, which I’ll just introduce quickly. The elevator pitch would be its Rails for daemons. You guys can check it out or we’ll have it in the show notes. Then a lot of the topics we’ll cover in that context. Daemon-kit was developed in an ISP environment where people would buy hosting services or domains or email or internet connectivity. And in that context, we have different daemons running on different servers. So, one daemon would for instance, on receiving a message, it’d create a mailbox for a user. Another one would be setting up domains on the DNS server. Another one would be setting up hosting. So, configuring Apache, configuring FTP access, creating directories on the web servers that are necessary, and plenty more that these would be doing. Even migrating clients from server to server, daemons would start talking to each other to get all the files packaged up, moved across, making sure things are in good state. So, that’s kind of the environment where daemon-kit was developed and took form. So, we had a lot of them and they needed to be up. People would be ordering services day and night. We couldn’t afford human intervention.

Can we talk a little bit about the process of starting up a daemon, what it is that makes you want some kind of library around it rather than just doing it manually.

Yeah, actually that’s great.

Because that’s an involved process.

It is an involved process and it’s cool actually, because in the old days, one of the ways people would daemonize a process in Ruby is to load WEBrick because it had a WEBrick daemonize method. So, they would load WEBrick and call it. That makes it kind of interesting because if you just crack open the source code of WEBrick, you can look at what that method does in plain Ruby and it’s pretty interesting stuff. I’ll link to it in the show notes. But go ahead, who wants to explain that?

I’ll give it my best shot. So, what daemon-kit would do is do the initialization in two steps. There’d be a pre-daemonization round and a post-daemonization round. Just for a start, daemonizing itself, and that’s kind of what WEBrick does and Ruby 1.9 has got it built-in in a process on daemonized method, it would essentially fork. The process would fork itself and exit and then fork again and exit. That kind of detaches it completely from the parent terminal. That’s how it goes into the background. So, it’s a double fork. But in that process, there’s a lot of other stuff you want to do. You might want to drop privileges, so you might start the daemon as root but you want to drop it down to nobody or to a specific user. There’s also, if you’ve got any file descriptors or sockets open, you want to close them and reopen them once you’ve finished daemonizing. You might want to [inaudible] yourself into a directory for security reasons, if you’re taking untrusted inputs. You want to make sure that if somebody somehow gets your code to do the wrong thing that it’s at least isolated from the system. You also what to redirect your I/O’s, so standard in and standard out. You want to close and reopen to attach either to a log file or you want to attach it to dev null. And you want to set up signal handlers for rotating log files or any other kind of stuff you want to do. You also want to have good exception handling in there, especially a kind of safety net so that if your daemon ever dies on you and you’ve now lost standard in and standard out completely, you can’t get to a backtrace, that information somehow is captured and stored somewhere so that you can go back and see, “Why did my daemon die?” So, there’s quite a lot that’s involved in more than just the double fork.

One thing I didn’t hear you mention was the pid file management, which always seems to be super crazy to deal with.

Yes, that’s right. I forgot about that. That’s the[writing out a pid file and checking for stale pid files so that your code can look after itself, try to see if it’s running after if a box crashed. So, it’s got a hard reboot and comes back up again. Is this daemon able to, like it will find a pid file and then, is it still running? Is it not? Can it make that decision to delete it and replace it? How do you handle double ones? There was something else. Oh, it’ll come back later in the conversation. Oh, the umasks. That’s also another thing on Linux boxes. If your daemon is creating files, even if it’s just log artifacts, you need to be able to set a umask so that those files are created with the right permission to be able to be read later again, which is something where if you’re not daemonizing, that stuff usually comes from the environment that started, your terminal environment when you log in as a user. But on a cold boot from an init script, the environments look completely different. You need to be able to handle those discrepancies.

So, this is one of the major points, I think. A lot of what Kenneth just described, the whole thing this daemonizing process is doing is detaching you from any means to interactively communicate with a user. So, you have to take a lot of steps in order to still have some kind of communication with the outside world. So for example, Josh mentioned the pid file, the process ID file. And the idea there is in some well-known location, you want to create a file and write your process ID down in that file so that if you think about commands that give you ways to start something and stop something, that stop has to know how to find what got started. And the pid file’s how that happens. You have the process ID and you can send signals to it, as we talked about before. Then setting up all this information with the logs turns out to be really key because if something goes wrong, your only means to record that somehow is probably like a log file. So, you have to set that up. JOSH:

So, does a cron job count as a daemon?

No, not at all.

It’s not long-running, right?

Right.

Yes, exactly. It’s not long-running.

Cron itself though is a daemon.

That’s correct. Yes.

Case in point.

It’s the system that runs so that periodically, if you think of how cron works, it basically fires up some process, figures out when the next job needs to run, sleeps that long, then wakes up and runs the job. Then rinse, repeat. So it’s there all the time and that’s what allows it to run jobs on some kind of a schedule.

Yeah.

One of the things that I found really tricky when working with daemons is that all of my mistakes seem so much bigger in a daemon. If I have some sort of memory leak, if I have a cron job, it’ll run and within a seconds or minutes, it’ll exit and all of that memory is released. But whenever I’m working with daemons, that doesn’t happen.

Excellent.

So for that, you can choose to do a forking model as opposed to a threaded model. Every time you want to do your piece of work, you can just fork off a subprocess, let it do it and then die again. But it’s very true. It is a very real problem and it’s a really tricky one to monitor as well. We lack a lot of good tools. New Relic can give us memory monitoring over time, but daemons, it’s very raw. You’re left to figure a lot of this out on your own. So a forking pool, so to say, would probably be better than a thread pool in the very, very long run. And it’s a problem that plagues a lot of daemon projects. I know the shift client as well, they have an option to specifically, every shift run when it runs every half an hour if you use that as a daemon, to fork when it runs because a badly written cookbook can completely just balloon the memory on every run. You end up with your provisioning software taking over your entire server. So yes, that’s a big problem.

What Kenneth’s just explained there, that was one of the major things I eventually understood when I was working with daemons, is that anything you do in the daemon process is basically forever. If you perform some huge database query which causes it to fetch a whole bunch of memory from the operating system, then your process ballooned by that much and it’s forever. So you want that top level loop to be as lean as possible and do as little as possible. It’s basically like cron, right? All cron is doing is figuring out what to do next and then doing it. But that process, who cares what happens in that process? It’ll be dead in a few minutes. So if you use that same model where you figure out what you want to do and then when it’s time to do something, you split that off and do it, then who cares what happens in that process?

Yeah. And when you say split that off, what we’re talking about, and this is a good call-out to go listen to the episode we did with Jesse Storimer, but a lot of times what they do is they wind up forking off a process that goes and does whatever it is that has to be done. Then you get all the memory cleanup and stuff that Kenneth here is talking about avoiding.

Exit is the ultimate GC (GC == garbage collection). Never forget that.

Yeah, your operating system is really good at cleaning that stuff up.

Yeah.

So then just to add to that, I know with all the MRI’s, and I think it’s still probably the case for the MRI, it never gives the heap back to the OS. So what it’s allocated, it will just simply hold onto. Internally, it might free up some memory and then it might start climbing again. At one stage, [it might] decrease the stack again. But MRI will never give the heap back. The JVM for instance, which is not really good for running daemons, but you can run a blocking process but that still will eventually free up if your job gets tired, or if your job frees up the memory.

I’m curious why you say the JVM isn’t good for running daemons.

It’s good for running background processes, but it doesn’t do the forking model. So if you want to run a daemon through Java, you’ve got to, in the daemon-kit context, you can run your project in the console while you develop and you’ve got access to standard out, printing out log files, pretty much like a Rails server does. It logs it to the development log and to standard out. Daemon-kit gives you the same ability. But then when you want to run it into production, you actually need to run a script that blocks so your whole process supervision model changes. You want to run it under something like God or Bluepill or run it under Monit or systemd. And it just sits there. It will never detach from the terminal. It’s a slightly different thing. But if you’ve got daemons that would be data-intensive, that would fetch a lot of data and they need to give all that memory back, then that would work fantastically well.

Yeah, I’ve had a couple of problems with the JVM just not being very POSIX compliant. There are things that you tend to do when you’re writing daemons, things you take advantage of because of the way UNIX-like operating systems work. A good example of that is the exec call. We talked earlier about pid files and knowing where a process is. Sometimes what you need to do is you need to turn one process into another process and you don’t want its process id to change. That’s what exec does. It lets you basically say, “From now on, make me this.” But the JVM was built for it to run everywhere, so not just in UNIXy environments so a lot of times it doesn’t behave exactly as you would expect in some of these things. And I’ve filed bugs on them, JRuby, about things like this in the past. And I know they’ve corrected as much of this as they reasonably can with the limitations that the JVM gives them.

James, that’s spot on. The daemon-kit embodies a lot of the UNIX philosophies. The directory structure looks similar to what your OS would look like. Just like Rails, give you a place to stash your things. That was heavily inspired from just, “It should look like UNIX.” When you look at Ubuntu operating system from the root, a daemon-kit project looks something similar from the root as well. It makes sense where these things belong. And yeah, the JVM just doesn't fit in so nicely with all the UNIXisms that’s in there.

So UNIX features very heavily in all of this and forking and all of that. How about Windows? What does daemonizing look like on a Windows system?

Windows has the concept of services and they don’t work the same way. So if you want to run it on Windows, you’re either going to have to have some kind of emulator to run your daemons or you’re going to have to write a Windows service.

There is a Ruby gem, if I’m not mistaken. I haven’t checked it out in years, but there is a Ruby gem to help you create a win32 service. I’m not sure what infrastructure they give, but I think it should be completely different. Different signal handling semantics, the environments change. What does a Windows environment look like for a user? There’s this service managed code. You need to somehow plug into the service management console. I think it’s a completely different game.

Yeah. But if that’s your environment, it looks like there are some pretty handy tools to do that.

Yeah, it should be possible, definitely possible.

Yeah.

Are the [batches] very similar though in a Windows environment?

The ways they work are totally different, so probably not.

Okay, so it would be a different conversation then.

Yeah.

Okay.

Although some things would still apply. For example, if your service is going to be a long-running Ruby process, you’ve still got to think about that, “Anything you do is forever,” right?

Yes.

So no class variables.

No randomly generated symbols. [Chuckles]

No.

I’ve helped several people with their daemons. The one thing that people to run into the most or have questions about the most is getting it to start up when the server starts. Is there an approach that you like for that?

I’m used to using God to boot up. So I’ll run God under system boot [label]. It’s just always been part of my shift deployments. It’s just there. Then I have God monitoring the daemon. And in daemon-kit I’ve got just a little rake file to help you generate the stub God config that you can then edit and tweak and then load into a server. So God will be kept up by [SV, what’s it], I forgot the project now. God will start on startup and it will monitor and maintain the daemons. It will watch its memory usage as well and its CPU usage all through the pid file that gets created. That gets the whole chain going. But there’s just so many ways to get that done. I’m considering looking at borrowing some code from the Foreman project which would allow me to give rake tasks as well to generate upstart scripts for Ubuntu and System V scripts for initd, to just simplify the whole lot.

Yeah, I was going to say, are there tradeoffs to using God versus just adding it to your RC or some of the other options that you have there?

Well, if you just add it to your RC then all you really do, your resource config (Is that what it stands for?) file, it will be fired off. But that’s where the connection ends. It just makes sure something happens. But using something like God, God is going to continue to watch the process. Like Kenneth said, watch for memory overrun or whatever, you can set all that, then using the pid file it knows the commands has started up, it can kill the current version and start a new one fresh so that you don’t have to worry as much about mistakes made in the daemon process itself.

That makes sense. So it’s similar to using something like Monit that watches it and says this should be running and it should be behaving this way?

Exactly the same use.

Yes. It’s the same thing except a different version, yes. Bluepill, God, Monit, those all do the same thing. If you’re on Mac OS X, Mac OS X has the launchd service which is surprise, a daemon. It is how you start and stop things like at system start time. You basically write this xml file describing your process and you end up giving it the command and the arguments for it and stuff like that. It reads those files on startup and if they request that the process be run on startup, it’ll fire it up as it comes up.

Right. And on the typical UNIX system, that process is called initd. And years ago, Apple just changed it to do I guess something that was more suited to how they were dynamically loading code on Mac OS X. Hey, I have a slightly different direction to pursue here, and that’s how you actually go about writing the guts of your daemon. Many of us have done something simple like writing a rake task and found that the way that rake runs code makes it a little hard to TDD, to testdrive the guts of that task. So we’ll pull of that out into an ordinary Ruby class or set of classes and objects and test-drive that and then once we’re done we’ll just drop in a one-liner into a task and then we’re done. That seems like that would be a good way to go for building daemons. Is that a good guess on my part or are there other ways that are better for developing daemon code?

No, I think that’s fantastic. It’s absolutely spot on. By doing it through the classes you get the testability and you are disconnected from the daemonization process. In a daemon-kit project typically there’ll be a lib folder which is ideally there that you drop these classes in. it’s on the load path so you can get going. And these should ideally be little classes that you can just instantiate and almost like a method object and just call one method, let it do its thing and report somehow whatever’s required. Then there’ll be a libexec folder and this is the place where you would put the glue between the daemon-kit and your own code. For instance, if you create a daemon that’s based on rufus-scheduler, rufus-scheduler gives you cron abilities in your own code. You’re not dependent on the cron daemon. And one reason you might want to do that is if you want subsecond precision. Cron can’t go less than a second, oh actually I think cron can’t go less than a minute in between writes.

I think you’re right. I think you’re right.

So if you want sub-minute precision, rufus-scheduler uses EventMachine if it’s there. Daemon-kit by default uses EventMachine. So you can get up to 250 millisecond precision in your jobs. That libexec class, there are skeletons that get generated for you. It will show you how to basically configure just your scheduling. It’s a little block that gets called and ideally all you want to do in that block is instantiate and call a method. Or you want to do that wrapping it in a forked block, if you want to get your memory back once the job is that. So that way, there’s a little seam that you can use. And there’s a generator for an AMQP-based daemon and a Jabber daemon, say if you want to send commands in over those mediums to your daemon to have it work like that. Same thing, it generates a stub for you. It shows you, this is the block that’s going to accept the message. Ideally you just want to instantiate the object with that message and the initializer calls something and either fork or not and then just be done with it. So that way, you can neatly TDD it on the side. There are test generators as well so you get failing specs or failing unit tests when you start and you just [up the races].

So, basically what you don’t want to do is be forking processes in every single test and trying to keep track of all of that. Maybe you want an integration suite or something that works with the full system. But it can get complicated when you’re launching all those processes and having to wait on them until they get to a certain point and all that. So that’s why what Josh said is really wise. Just separate out the work codes so that you can verify that the work does what the work is supposed to do without having the complication of it being moved through different processes and stopping you having to follow it through the pipeline.

I went down a really bad path when I was doing daemons and I was using the threading model not the forking model, which was probably my first mistake. But I ended up writing tests that would use sleep heavily to even be able to verify anything. It was a nightmare. Don’t do that.

Yeah, and then randomly one of your tests will just fail.

Oh yeah, and then you just run it again, because then it will pass.

Right, but then it works.

[Chuckles] So I have a question related to when, and I think we touched on this, but when would you want to create your own daemon versus setting up a cron job that will run a rake task for you or using some other recurring system like a queuing system?

Some of the daemons that I worked on listen to the fire hose of Twitter or the fire hose of Instagram and once that data is gone, it’s gone. If you miss it, you miss it. So we actually listen to the fire hose and stuff those in a queue that could then be processed. Or I think we stuffed only the ones that we wanted into the queue for later processing.

I’ll buy that.

Another example might be handling inbound mail. So if you want to run a little SMTP server and people can email attachments in that you then continue taking an attachment, upload to s3, pass it onto the next thing for processing. You have got no idea when that mail’s going to come in. That socket needs to be open and ready and to do its job.

That’s a really great distinction right there. So there are definitely pros and cons to both approaches Chuck just mentioned. If you run a cron job, the great thing is that that process can probably just worry about itself. It’s not a big deal if it allocates a bunch of resources and gets killed by the operating system. Who cares? Whatever. It’s all going to go away and happen again later. Whereas as we’ve been talking about with daemons, they’re there forever or theoretically so have to be more careful. But that’s basically the distinction. If you can get by with just doing something periodically, then a cron job’s definitely easier. But if you need something that’s there all the time and when I turn to talk to it, it will be there to answer me, then you need a daemon.

Well said.

Yeah. It also seems like basically, I think the key to what James is saying is more or less, I need something there when I come to talk to it so that it can answer. And you never know when that talking to it is going to happen. A lot of these other things, if you’re just checking on something, then you can periodically pull it. So you can actually use cron. But yeah, if something’s going to be sending it messages, “Here’s this information. Handle it,” it can’t periodically wake up and go, “Did somebody talk to me?”

Daemons are used a lot nowadays in metrics gathering. So if you want to gather metrics on various things, you add a little line of code that’s like, “Send this metric now. New user signup plus one,” something that. Then there’s usually a daemon on the other side to catch that bit of data. A cron job wouldn’t really be great for that because you would have to stick that information somewhere, like in a file or whatever, then the cron job would have to wake up, read all that stuff out, put it somewhere but meanwhile zero out the file that’s being written to while it’s running or whatever. It gets complicated, right?

And the question is who’s writing the file?

Right, exactly. Yeah. Is it various processes and the provisions are going to be right that everybody can do that or you use some kind of queuing system? But that queue has to be there in order to accept the incoming thing.

Which is going to be a daemon, a server of some sort, or some process.

Exactly.

Okay, we talked about queues here. What are the typical ways that people or software communicates with daemons? We’ve mentioned message queues and we did a whole episode on that months or a year ago or something. But there are also UNIX sockets or ports or you can do signals or files. What are the sane ways of actually getting the communication going?

To me, I think it depends on the kind of communication. XMPP or Blather, Jabber is a perfect way

of getting it going. Daemon-kit uses the Blather gem underneath to give you an evented XMPP client. Part of that is you’ve got [poor man’s] monitoring as well. You can friend your daemon using Adium or some other Jabber program and you can literally use the status of your daemon whether to see if it’s running or not. You don’t even need to check emails or monitoring systems or anything. But that’s great for one to one channel kind of communication where a daemon process has a separate identity that you’re talking to. AMQP is fantastic where you might be broadcasting different command over channels and the same command might end up happening on different boxes, like heartbeat. Broadcast all your daemons and they respond back. So that would make that medium great. UNIX sockets are fantastic for local communication when you’re on the same box or you can use a TCP or UDP socket if you want to go over the network. For that, I definitely recommend using EventMachine. EventMachine for UNIX sockets as well is in daemon-kit. It’s ready for you to go. With all of this though, that’s just sending something to the daemon. You still need to develop your own little protocol. Are you sending it all snippets of JSON? Are you sending in plain text? Are you sending in some kind of binary representation? That’s something that’s left for the project owner to solve how they want. So there’s the transport medium and then what goes in and what goes out over that transport medium. Does that help?

Yeah, yeah. That’s a pretty good overview.

And then, oh sorry, just to add, like James said earlier with the signals and when Katrina asked for it when she asked. Signals is more of an operations concern than the daemon doing its job. So those you’ll use for shutdown safely, so start cleaning up after yourself before shutting down or log rotation or switching debug levels. Something like that is what you would use signals for. It’s not for just the running, it’s more ops concern where that falls in.

Okay. And what about things like pipes?

So that would be you can create a pipe and then you can just use EventMachine to read from the pipe in a non-blocking way so the moment data gets written to the pipe it works like a UNIX socket. So there’s the FIFO pipe. Or you can use a blocking Ruby can just read and it will block waiting for input to come in. Then you just need to make sure that the buffer size that you’re reading off this pipe is big enough or small enough that you actually get useable commands through. Those are quite easy to set up with the UNIX commands mkfifo. And it will create a special pointer on the disk that’s managed by the kernel. Your daemon can attach to it and you can write to it.

Yeah. Those are nice. It’s pretty low-tech but they’re nice because you can just interact with them with regular file tools. You can cat some data to it and give it a command that way.

Okay. So there’s also been a lot of, if you’re doing a regular web application, there’s a lot of infrastructure and tooling people have built around deploying these things. I’m curious what the deployment cycle is like for daemons.

Well, Capistrano. [Chuckles]

Really?

It’s pretty much the same thing.

Okay.

Capistrano. So daemon-kit’s got a built-in cap recipe. It looks and behaves very, very closely to a Rails one. It depends on the pid file in the end to actually stop the daemon and start up a fresh copy. That’s the only magic, is signaling when it is time to shut down, waiting for it to shut down, and starting it up again. Then in that shutdown I guess the only trick there really is you want your code to give it some grace, but you also want it to be able to die forcefully. So the default would be 30 seconds. If that application cannot finish doing what it’s doing, you want it really just to then die. Then you can inspect the backtrace logs to see what went wrong. But we’ve hooked it up very, very nicely with Capistrano and you just go. It’s as simple as that.

That’s a great description right there. Having done all of these things manually, if I could give anyone one piece of advice is use daemon-kit if at all possible. Because that process you just described is tricky. You have to read the pid file and go find that process and send, it is it killed? The one that can be caught? I always get kill and term backwards. Anyways, one of them is the peaceful, “I would like you to stop now.”

That’s the kill. [Inaudible] the term. Term.

Term? Okay. So you send the nice one first and then you wait. You go to sleep for a while. Then you scan the process table and see it that process is gone, if it listened to you. If it did, great, you start it up. If not, you need to send it the not nice one, the “Okay I gave you your chance. You didn’t listen to me. Now you’re just going away.” This is exactly what your operating system does when it shuts down. It tries to close everything out nicely and then if it can’t, it ends up killing things because it has to shut down. So then you kill it. But if you kill it, they don’t get a chance to respond to that, so there’s going to be consequences. For example, that pid file that didn’t get cleaned up because you killed it before it could clean up its own pid file. So you’ll need to clean that up after. So there’s just this complex set of interactions. Try this then do this, wait for this, and there’s really a lot of room to make mistakes in doing all of this. So if you get a chance, use something like daemon-kit so that you can take advantage of all of that thinking without having to do it yourself.

And everything James just mentioned, you have to decide what responsibilities lie in your deployment setup and what just lies in your normal running the daemon setup. Because being able to stop a daemon from the command line and giving it grace before killing it, that’s something you want to do outside of deployment as well. So it’s this whole, [it’s about a huge response]. I was just skimming through the daemon-kit cab file and it’s 484 lines. That’s not the code to wait for a process to die and signal it again and eventually forcefully kill it. That’s built into a different part of the framework. It’s quite a lot of plumbing that’s involved.

Interesting. Okay, so little different direction here. Are there issues with sharing code between your daemon and a web app? This seems to be a common thing that people do. They’re building a web app and then they start pushing pieces of it into services or background jobs. I’ve done this a bunch of times. I want to have something to send your mails in the background so that they’re not taking time out of the request/response cycle processing.

So to clarify, by sharing you mean by creating a gem for example, that’s used both places?

Well that would be one approach. I’m curious of what Kenneth thinks about good ways to share that code. Should they be in the same codebase? Should they be in different codebases? Does coupling the deployment of your web app and your daemon processes, is that a problem or is that an advantage?

That’s a tough one. I prefer to keep them apart. During the development, it’s a schlep to keep things in sync and to make sure each thing has the bit of code it has. But I think in the long term, allowing each snippet of the codebase to focus and specialize for me has been more rewarding. I think the only thing, really, that you end up sharing is your Active Record models. Sometimes you really want completely different methods and lookups and behaviors in a daemon process than what you would want on the Rails side. The same thing goes if you split, for instance, your management app that the company uses internally from the main app serving the public. You face exactly the same problem. The same database table, it might get represented through completely different models depending on the context where you’ve got the code. So I have yet to figure out the best way for myself as well. What I’ve got at the moment works. It’s just a lot of work in development. But in deployment, it works fine. I think if you’re running a daemon in a Rails project directly in the code and you’re sharing models, you would definitely need to have your deployments tweaked so that you restart the daemon process together with your main application. Because then the running code loaded in your daemon might be completely out of sync with what’s actually been deployed and what the team thinks is running live in production. So that’s a big risk. The other way you could do it is if your daemon talks to your app through a little REST API. That way, they can run, they can be even more disconnected from each other and somehow they can signal and chat to each other. The app can give the daemon commands over a different channel, AMQP. Or the daemon might have its own little Sinatra that’s just for local use. Then the daemon can talk back to the main app and they can bounce messages up and down. But I don’t know. I haven’t found the silver bullet yet.

I think what Kenneth just said is really important. Basically, the idea here is SOA again if you have a servers-oriented architecture. The thing I would stress there is try, try, try, try, try, “Don’t share the stuff.” Don’t make that gem and pull it out into both places. That way lies pain. Communicate. Treat them as separate. Treat them as services. So send a message, going down to basics like JSON. Here’s the data you need to know to do your job. Then it does that job and then sends back. I would even prefer not to have them share the database if that’s reasonable. I would rather I send a message down to it, it did its thing, then if it has to it sends a message back to me and I’ll save it in my database or whatever. But obviously, it depends on what the daemon is doing and how the communication goes. But I would rather not have that shared. I would rather communicate with it. And the reason is many. Kenneth hit on a couple of big ones. If you share it, okay, now you’ve got to make sure both of those running pieces are using the exact same version of everything. So kill one, you’ve got to kill the other, and that kind of thing, which sucks. If there’s this background service, the point of it is that it runs all the time. So you don’t want to have to kill it because you added some new feature to your Rails app and you change that model in a way that isn’t going to end up affecting the service or whatever. You want that to be separate so you can choose when that one goes down and when the other one goes down. You can treat the two independently. And just a million other reasons. Try and try not to share. That’s Erlang’s whole thing. Send messages everywhere. That’s how processes communicate.

I can add one more potential horror story to that. If you’re daemon’s using autoloading and it’s not been restarted the whole time with your main app and the codebase has changed underneath, I imagine at some point, it kicks in and it autoloads a new class and that class runs in a completely different environment as to when the daemon started. That can just be a nightmare to contend with.

That’s another point. If you share a significant portion of the Rails infrastructure, then your daemon just got a lot more complicated. The whole point is to keep them as simple as you possibly can because they’re running out there in the background detached from any way to, say help, that kind of thing. You want them to be as simple as possible. So it’s doing something like require rails infrastructure that violates the as simple as possible.

[Chuckles] Well, okay. But sometimes, daemons are doing really complicated things, right?

Yes.

They’re not always really simple things. I’m working on something now that’s doing basically data warehousing and it needs to have all these Active Record model classes around because that’s the implementation of the business logic that is being used to sort out all the warehousing and figuring out what information to put in the warehouse records. So that’s a pretty complicated amount of behavior in there.

Yes, but depending on the behavior you may want to just pull an Active Record instead of the entire Rails environment.

Sure, yeah. I think that’s a great idea. Yeah, bring in just Active Record. Bring in just the models you need. But then again, I would stress Kenneth’s points of a lot of times what happens in the daemon is not what happens in the app. I’ve found that it’s totally reasonable to keep two separate models on both sides sometimes because what this model is doing and what that model is doing may not be very related. Now if you have tons if inner relationships and stuff like that that are being used on both sides, that may not be reasonable. But definitely, you’re probably going to want to at the very minimum do a different mixin on each side because surely there are going to be some methods that just don’t make any sense from the other side and you don’t want to have to update the one application just because you changed those. And you’re right. That’s a great example. If you’re manipulating a bunch of things in a database then obviously you’re going to have to go down that path.

Yeah. And then you have to worry about things like how the daemon and the app, where their coupled as. Are they coupled at the schema level in the database or do they share Active Record class model definitions? This is probably way outside the scope of this conversation because there’s almost certainly not a general answer to this kind of questions.

Well that’s the thing that is so interesting about daemons in my opinion, is that we write them to do all kinds of things. The way that you talk to Rails or even a web application in general is different. Most of the time you’re doing things over the HTTP protocol and you follow a couple of general rules and with daemons it really varies depending on what you want to do and how you want to communicate with it. That’s why this problem is so hard.

Yeah. One of the, I think, typical questions that everybody who’s doing any sort of daemon that works with a Rails application is what you put in the payload of the job. If you have a message queue based daemon that’s pulling job tickets off of the queue and working on them and maybe it’s sending emails, user confirmation emails, something like that. I think a typical question people have to deal with is what you put in that payload. Are you putting in all of the data that you need to do all of the work to render the email and send it off to the recipient or are you just putting an object id or you have a user record, what’s the user id, and then let the demon load all of the data from the database using Active Record and do all of its work that way? That’s a pretty big difference.

My choice is definitely just the id. But that again depends. So if you’ve got your email sending code living somewhere completely outside the project, not in a Resque or Sidekick worker, then it might make sense to just fill in the variables and the mail can just get fired off later and you don’t need to worry about after commit semantics. Otherwise, if you just send the id and load from a database, you need to make sure that that job or that message just gets dispatched after commit, which is probably the biggest thing that trips people up. I think to some extent, Sidekick alleviates that with its automatic retry, so the job would fail and two and a half seconds later it would pass on the first rerun. It just happens for you. And people wonder, “What?” and that’s the transaction had time to commit. So I guess it’s also under the load of your system. There are so many things to consider here.

I take the other view in that I prefer to pass the details of the job if they’re not ridiculously large and stuff. My reasoning is that I can decouple from the database, hopefully. If I don’t have to have that connection to the database, then I consider that better. If I can treat that service as something separate entirely. So I’d rather just pass it down, what it needs to know, let it do its thing and then notify back or whatever, if it has to and decouple from that. But there are a lot of concerns there like Kenneth brought up. What if the payload is very large? Then that’s a problem hanging around in the queue. I try to write it where when I’m queuing things, there’s no danger of the queue going away. I try to write it where the queue is not the canonical source I’m relying on to stick around. So if it dies, I try to write it in such a way that I’ll be able to tell what hasn’t been done yet and I can just re-queue it the same way or something. But sometimes it’s not always possible if the task is an idempotent or something. In the case of sending emails, I don’t want somebody getting a second email. So there are definitely exceptions.

Disconnecting from the Active Record specifically is, yeah I’m sold James. The other thing, I was thinking with daemons, it’s hidden away in Rails for you, the Active Records connection management. If your daemon is doing any kind of database interactions, you need to take a connection from the Active Record connection pool and release it. Otherwise, at some indeterminate time in the future, you'll run out of active connections which is another thing that we just forget about that Rack middleware handles for us in our day to day. So that’s another thing, being able to disconnect from Active Record just makes a hell of a lot more sense. I need to rewrite some code.

Unicorn has an example of that. If you look at the basic default Unicorn config, there are a couple of lines in there in the after fork handler, I think, that Unicorn supports. That’s after it forks a process, it goes through and makes sure it reestablishes the Active Record connection correctly for exactly the reasons Kenneth just mentioned.

Alright. Well we’re getting pretty close to our hour. In fact, I think we’ve probably just hit an hour. Are there any other aspects of daemons that we haven’t talked about that are critical to understanding and not screwing things up?

Yeah, I have one more safety tip, because I never learned this until it was way too late. And it’s very helpful. If you’re going to write a daemon and it’s running in the background, the hardest thing to know is when something goes wrong, especially if a process dies because of an exception. You need to know that so that you can debug it. And the best to do that is to set an at_exit handler when the process starts and inside at_exit you can check Ruby’s special variable, the $! variable, that holds the latest exception. It will be nil if Ruby’s exiting without an exception and it will have the exception in it if Ruby is exiting because of an exception. So set an at_exit handler that checks that variable and if it’s got an exception in it, throw it to the log file on your way down. So [inaudible] have the exception in the stack trace.

If you want to see an example of doing that, one thing you could do is read my book ‘Exceptional Ruby’.

But a long time ago, I wrote a tiny gem called zero-zero, which does exactly that. Basically you include it and it installs an at_exit handler which tries to save some vital information before the program dies completely. If nothing else, it’s kind of a proof of concept. I would suggest just check out the gem and just take a look at the code to see what it does rather than just using it. I named it zero-zero because it’s named after the zero-zero ejection seat which is supposedly capable of saving a pilot from a crashing plane even at zero altitude and zero speed. [Chuckles]

Awesome.

Now just to add to that, to the exceptions, you guys mentioned it in the retreat’s show, is setting Thread.abort_on_exception to true. Otherwise, if some code in the thread dies, you’ll never know what happened. It’s just a thread that ends up dying and your daemon sits there happily spinning along and you’re completely blind as to what happened. And similar to what Avdi mentioned, daemon-kit, I extracted from it a gem called safely which has exactly the same thing. It’s got the error handling and then error break reporting. But what it also does is the at_exit hook and then I had help from James Tucker and he helped hook in basically traversing ObjectSpace. Through ObjectSpace you can look for multiple exceptions. So any unhandled exception or even other ones that might have been handled by something like if you’d have been on Blather or AMQP, so invalid protocol messages or connections dropping up and down. When the daemon exits, it writes a separate log file with all the exceptions it found and it tries to take a guess onto which the one was which could have ended the daemon, which made it die out. So you always have these backtraces that you can see all the unhandled exceptions and work with them and figure out what the error actually was.

I can’t stress again how important this step is. It’s one of those things you’ll think about, “Oh, I’ll handle it when things start to go wrong.” Don’t do that. When a daemon separates from its environment, that’s the whole point. So when things go wrong, it’s just something going wrong in this magic place that you have no visibility on and you’re not aware of what went wrong. Then think about if you’re daemon is deployed to someplace that’s different still, a different environment, then there are going to be things in that environment you didn’t think about. So you need that info captured in that log file, because that’s your only hope of being able to debug it. I actually had a project cancelled because it was a daemon that we became afraid of because we weren’t able to tell when it would die and why, because I hadn’t learned this trick yet.

And on that note, that happy note, let’s do picks.

Good idea. Alright. Katrina, why don’t you give us your picks?

Alright. I have two today. The first is a gem called servolux, which I think I’ve picked before but since it’s so on topic, I’m going to pick it again. Tim Pease wrote a gem to write daemons and he handles a lot of different types of daemons. So it’s set up so that it’s a more modular approach. With daemon-kit, you get a lot of things out of the box. With servolux, you have to do a little bit more work on your own, but it does give you some flexibility if the Rails approach to everything being the way daemon-kit wants it to be. Anyway, servolux is great. My other one is Aral Balkan is a programmer and UI designer person. He did a conference talk called Superheroes and Villains in Design. And it is jaw-droppingly awesome. It’s about how awesome people feel when you have done design right, when they’re using your product. I’m not going to say any more than that, but definitely go watch this talk.

Okay. Josh, what are your picks?

Okay. Let’s see. So there’s a UNIX utility that I discovered recently that I’ve been really itching to use for something real, but I haven’t had a chance yet. I’ll just throw it out there because it looks so super cool. That’s called Pipe Viewer. It’s basically like Ruby tap for pipes. So you can pipe the output of one UNIX command into Pipe Viewer and then onto wherever else it’s supposed to go.

So if you’re doing a ps and then piping that to grep or whatever, you can take the output of the first command, run it through Pipe Viewer, and it will output a whole bunch of essentially debugging information for you so that you can see what’s going through the pipe on its way to the next command. It looks really cool. I’m really dying to use it on a project, but I haven’t found a use for it quite yet.

That sounds awesome.

Yeah. And the project looks in good shape and they’ve put a lot of good work into it. So that’s pretty cool. Then my other pick is a classic science fiction novel. I did a long airplane flight recently and I always like to have a dead tree book with me because the airlines are so insanely paranoid about letting people use their phones and things on takeoff. So I read ‘The Mote in God’s Eye’ by Larry Niven and Jerry Pournelle and this is just super classic sci-fi. It’s almost 40 years old, I think. It’s in the reasonably far future, mankind encountering, or humankind I should say by that point, encountering an alien species that’s really different from us. And it’s one of the best sci-fi novels every written, in many, many people’s opinion. The odds are many people have already read it who are listening to this. But if you haven’t, it’s really worth checking out and reading. There’s a sequel to it as well that I also enjoyed. But in my opinion, nothing really can stand up and challenge this book. So that’s it for me.

Awesome. James, what are your picks?

I’m flushed with picks right now. So I’m going to go through several here real quick. First of all, I don’t usually like to pick my own stuff, but I’ve had several people ask me if I would explain what I’ve figured out about Emacs since I’ve been playing with it. So I did do that. I held a Google Hangout recently and I tried to explain a lot of the key points of my particular configuration and how I use it. If you missed that, it’s okay, because it was recorded. And it’s in two videos due to a hilarious incident where I killed my own Hangout. So there’s two parts to the video and I’ll link to both of those in the show notes. But check it out if you’re remotely curious about Emacs. The other thing I’m going to link to, also mine, is I did a different talk at Lone Star this year where I played Jeopardy on stage with three contestants, Avdi Grimm, Dave Thomas and Steve Klabnik. What I did was just nothing, basically, just provided some Ruby trivia questions. What they did was make an absolutely freaking hilarious section where Dave Thomas rage quits my game in the middle of it. Steve Klabnik purposefully blows some points to troll the entire room. It’s freaking hilarious. So definitely in the spirit of wacky fun more than learning. Though I did have a couple of questions in there that seemed to stumble them. So you may pick up a Ruby trick or two while you’re watching. Anyways, that video is also now online and I’ll put it in the show notes. And finally, I probably shouldn’t say this in front of Josh but while I was procrastinating on my GoGaRuCo talk this weekend, I have this great [Laughter]

What?!

Calm down Josh. I found this great game called Dungeon of Dredmor. I tweet about how I like FTL all the time which is one of those rogue like games and that it’s just this crazy hard game that you die to most of the time. And somebody responded to one of my tweets and said you should check out Dungeon of Dredmor. So I finally did that this weekend and oh my gosh, this game is awesome. It’s Diablo-esque in that you grab a character and take off into the dungeon. The things this one has going for it, first of all there’s just a massive amount of character abilities from vampirism, your typical swords and shields and armor, then magic and [inaudible] stuff, to piracy, to crazy stuff like rogue scientists and just all kinds of crazy abilities. So there’s lots of combinations and synergies that makes dying awesome, because then you just go back and you’re like, “Whoa. I want to try this.” And that’s cool. It has a really neat turn-based thing. The whole thing’s turn-based, but it feels real-time. You’re just walking around in it, but then when things start to go horrible bad, you realize that every step you take, every potion you drink, all that, that’s one turn. So you use something then the bad guys get to use something. So it lets you slow it down to a strategy game pace and think your way out of it, which I really enjoy. And finally, the humor in it is freaking hilarious. I cannot stress this enough. This game is a blast. It has tons of pop culture references in it. Indiana Jones, if you start off with the archeologist ability, you have a fedora hat that turns out to be one of the great items of armor for a starting level character. If you get your perception up high enough, you get lasers that shoot out of your eyes. Your eyes just get so good they get lasers. It’s totally hilarious. [Laughter]

Awesome game. You have to check it out. Those are my picks.

Awesome. Avdi, what are your picks?

First of all, I’ll second servolux as I always do and also Mote in God’s Eye. Great book. So I’ve been revamping my whole publishing tool chain lately, both for the launch of Confident Ruby and for the book that we, the Rogues, are working on. That’s involved throwing some old tools out, slotting some new tools in. Anybody who’s done eBook publishing, the tool chain usually winds up something like this. Usually you have part of the tool chain which whatever you start out with, you wind up with html and then the html goes into your epub version and into your Kindle version, possibly into a website version. So you have this whole html chain. But then you’ve got this parallel chain, because there are still lots of people who want pdf’s. Especially, you’re going to need pdf if you’re ever going to print it. So you’ve got this parallel chain for pdf which is a different universe, because usually you’re using either LaTeX or maybe you’re using DocBook to XS-LFO or something like that. Either way, it’s a completely different language from the html and CSS that you used to style all the other versions of the book. So you wind up doubling your time working on styling and layout for two different, these two parallel universes. And it’s a huge waste of effort. And some of these things just are not pleasant to work with. I haven’t done a lot with XS-LFO but LaTeX certainly is a gigantic learning curve and often makes easy things surprisingly difficult. So when I redid my tool chain I decided to see if I can get away from all this. I’ve been using html as the universal intermediate format and then using a tool called Prince XML for the pdf output this time. It is a commercial tool. It is not free. But it is pretty freaking amazing. Basically, what it does is it will take html and CSS and have really, really good CSS3 support, which includes all of the CSS3 print stuff. So you can actually do a really good print layout including stuff like where the page breaks would fall and if it should always put a particular kind of page, break the pages so that that page should be on the right hand side when you open the book. And the leaders and the footers along each page and automated generation of page number references, all that kind of stuff. Plus it does the Knuth [hyph] layout, text breaking and hyphenation algorithm, to make printed text look really, really good. And you can just take the same html source and the same CSS and add some print CSS and toss it into Prince and you get a great looking pdf. So it’s a really nice tool. Sure, there are some other tools out there, some open source tools out there that will do this as well, but they’re basically as good as hitting the print button from your browser, which as you know is not that good. So now the downside of Prince is that it is $500 for a personal license. But there’s another cool tool out there called DocRaptor which is the software as a service version of Prince XML. So you sign up with them and you can send them RESTful web requests with your document and they will send back the pdf formatted version. And that is very reasonably priced. So both of these tools together have been really incredibly helpful in redoing my tool chain. I guess I went a long time with that, so I’ll just one more. Another thing that I’ve been playing with as I’ve been doing this tool chain is a little tool called XMLStarlet, which is just a nice little command line tool for dealing with XML in many different ways. You can do validation, you can clean up your XML, you can select just a little part of it using an XPath. It’s like sed and awk, only for XML. Pretty cool.

Alright. That’s enough for now.

Awesome. Alright, well I’ll go ahead and do a couple of picks. So I’ve been getting my stuff organized. I keep changing the way I do this. I guess I’m just not happy with the way that I organize things. But a few things that I’ve been using lately to get things organized is, one of them is Evernote. I’ve recently started putting all of my, I’ve been trying to go paperless. And I guess that’s another pick. So there’s the MacSparky field guide. It’s called Paperless. It’s a book and videos and stuff and it talks about different techniques for basically moving away from keeping a whole bunch of papers in a file. So I’ve been scanning all of my papers using a ScanSnap S1300i which is actually a little portable scanner. It’s really, really nice to have and I can actually take it on trips, scan the receipts and just throw them away, which is also nice. And then I’ve been putting it into Evernote. So I’m going to pick the scanner and Evernote as well as the Paperless book. And a few other things that have come in handy, I use SaneBox for my email box and I’ve been trying to follow GTD so it’s nice to have something that just goes in and filters out all of the stuff that isn’t critically important. Then I can either move the stuff out of my inbox into action or follow-up. And then I just routinely go through the action stuff and do the stuff in the action stuff. In the action folder, sorry. Then I go through the follow-up folder and just make sure that I’m keeping up with the people that I need to keep up with for stuff. So I’m also going to pick Gmail and SaneBox. And those are my picks. Kenneth, what are your picks?

My first one is the presentation I mentioned in the beginning for the definition of what a daemon is, is the Angels and Daemons presentation by Tammer Saleh from Thoughtbot. That’s just a great glimpse of the history where we all started on this quest for figuring out how to daemonize your code. So it’s really nice to just get into the guts and see what’s going on. For the second pick, I’m going to pick Rubyfuza. It’s a Ruby conference in Cape Town in February, first Thursday and Friday of February. It’s a fantastic conference. It’s well worth the trip to come visit South Africa. For my third pick, I’m going to pick a music festival here called Oppikoppi. It’s also inside Africa. It’s in August, first weekend of August. It’s the only festival in the world where you can have your beer delivered by drone. It is really, really fantastic. Given that and five stages of amazing bands. Then further [off tech] is two other podcasts. 99% Invisible. They’re from San Francisco. And then Radiolab from New York. I find both those are just great for listening to something else, opening your mind, hearing about different stuff people do, different adventures people go on. It’s just a fantastic way to just be inspired. So those are my picks.

Plus one on those two podcasts. They’re great.

Absolutely.

Yeah, they are. I listen to them as well. Alright, well let’s go ahead and wrap up the book [Chuckles] the book. [Chuckles] Let’s go ahead and wrap up the show. I was reminding myself, mention the book club book. We’re going to be reading Confident Ruby by Avdi Grimm. That will be, I think the beginning of October. Anyway, go pick it up. It’s been a good read so far and we’ll catch you all next week.