The Rogues talk to ThoughtWorks' Sudhindra R. Rao about JRuby.

Special Guest: Sudhindra R. Rao.

 
CHUCK:

Here we go.

Meow. Meow.

Meow.

Hey everybody and welcome to episode 194 of the Ruby Rogues Podcast. This week on our panel, we have David Brady.

The pod call was coming from inside the house.

Avdi Grimm.

I am probably not a cat.

I’m Charles Max Wood from DevChat.TV. And this week we have a special guest and that’s Sudhindra Rao.

Hi, hi everybody. And I am certainly not a cat. [Chuckles]

Do you want to introduce yourself really quickly?

I have been doing Ruby for about seven, eight years I have done a few Ruby projects with ThoughtWorks, different sizes, different Ruby technologies, Sinatra or Rails, old Rails, new Rails, [inaudible] 1.0. And recently, like the last three years, I have been doing Ruby on the longest running JRuby project that we have at ThoughtWorks called Mingle. Mingle is the project management tool that ThoughtWorks sells under its umbrella, which is very, very configurable and suits your process, not the other way around.

I’ve used Mingle. I liked it.

Does that mean I can configure Mingle to ignore all of my tasks until the last minute and then suddenly panic?

[Laughs] You can. But that [inaudible] you.

You can. You can actually do all those things. [Laughter]

I haven’t found one yet that’ll stop me from doing that. [Laughter]

And my managers have looked.

I want a project management system that will panic right along with me. [Laughter]

I will put that in my notes.

[Laughs] Anyway.

I can totally see that. OhCrap.io. So, I’m curious. We brought you on to talk about using JRuby. And I just, whenever I hear JRuby I just shudder and remember pain. Has it gotten better?

It has gotten way, way, way, significantly better, especially since the time we started. And I have to go back in history, even before my time. I’ve got Mingle. And Mingle was basically being built to be this tool where it will not force users to follow its religion or its style, but the other way around where users can say, “I would like to do this in my process. I would like to do that in my process.” And it will allow people to skin their Mingle to the way they work, basically to fit their organization.

And for that at the time, Ruby was chosen because it allowed to deliver faster. It allowed to write code which would help you build such a configurable app without all the restrictions like Java and other things, and other languages have. Plus, given Rails was released around that time. Mingle was started around 2006. So, Rails was released at that time. And then JRuby was the alternative platform that it would release. JRuby did present a lot of promise. And what Mingle was looking at was being delivered as a simply manageable app so that anyone could download it, manage it themselves, and use the tool to help their organization and do the kind of agile or do a kind of process they would like to do.

And JRuby in those days, especially if you wanted to run the whole app on your local machine, would consume a lot of resources. And starting JRuby runtimes, and every time you need to start a test or stop a test was expensive. But over time it has become significantly faster. The performances that we see today are very close to MRI. Recently we just added another optimization which will reduce the startup time for all our tests from 20 seconds. It does reduce to five seconds. We use this thing [called as] [inaudible]. And these kinds of technologies are actually helping us build the same kind of performance that we would get with an MRI app.

That sounds really promising. The thing that I remember with pain was that A, yeah I would start the tests and then it would be like, okay, okay…

Okay, okay, oh, there it goes. And it says it ran in way less time than it did. [Chuckles]

And it was because it was starting up the JVM and then running everything. And the benchmark only covered the Ruby part of the process. And then yeah, that was really the biggest thing, was just the startup lag. But it does look very promising. And the JVM is very cross-platform. I’m not going to say completely cross-platform because I know that there are still edge cases and issues there. But for the most part, it works really well on all of the systems. And so, it’s one option I really like especially when I’m going to Windows.

Yeah. And actually Mingle was built so that we could deliver on Windows as well as on Unix machines. And the way we packaged it at that time was build an installer kind of a thing, so just one click. You download it and one click you install it on your machine. And just, it runs like a Java app. And you do things with it like a Java app. You send in environment variables or parameters to tune the Mingle. And it just behaves like a Java app. That was the killer feature that we liked, because we could write in Ruby. So, we had the performance, oh sorry, delivery benefits of Ruby. And we could still give the users an app where they could be agnostic of what the technology is. And they could just manage it very well.

Although something that we have done over and over and still continue to do is when we are doing development we still run MRI because it is so much faster to run our tests from your editor or run tests locally. And when we run in production work, then we do testing on JRuby. We run all our CI builds on JRuby because we want to get feedback about how the production environment will behave.

So, do you run into any gotchas between the two systems?

There are. Basically we have to be continuously aware or whether what we are testing has any Java aspects to it. If it does, then we need to ensure that those Java aspects are tested starting a JRuby server. And in that case the test becomes slower. But that is a once in a while case a not a very, very regular case. A few things we have to have, to connect with Oracle, is because Oracle’s drivers were available and we have JDBC drivers to interact with Oracle. So, we always have to run tests in Oracle using JRuby. So, whenever something fails weirdly in Oracle then we need to start up the server in JRuby, connect to Oracle, and see how it behaves and then fix it.

So, when you’re talking about stuff that’s using Java, are you talking about talking to Java-specific libraries directly?

Yes. Think of a Rails app which has frontend processing, background processing, and all of that. And Mingle does have all of this encapsulated into the Java app. So, we basically have servlets which start a web frontend and the background node and background processing, everything, in the same container. So, when we need to make sure that this interaction works, we need to start the app in JRuby mode. We also use a queueing system like ActiveMQ in Mingle. So, when we need to start such things then we have no choice but to use the Java mode.

Okay. So, it sounds like when you put it all together in integration you pretty much have to be testing it on the Java side.

Yeah. Although we do some optimizations given the size of the application and our need for fast feedback. We have [inaudible] stages in our build cycle. So we, let’s say I write a feature and then I test it locally. Then I push it to a Ruby build, just a plain MRI build which we call it as a pre-commit build, which runs all the unit, functionals, and database-agnostic tests. In the sense, they run on Postgres. And if they don’t have any specific Oracle functionality then we run those tests as part of the suite. So, we have faster feedback on that. And that runs significantly faster. It runs in about 10 minutes. And we know whether whatever we are checking in has passed that basic test. And then when we run the final acceptance phase which runs on the actual JRuby environment, that takes significantly longer than this pre-commit. So, in the final stage we do run everything on JRuby.

I’m a little nervous that you consider a 10-minute test to be faster.

Yes, and I was too when I started on Mingle.

And then you just went with it? You just like, Stockholm syndrome set in and you’re like, “10 minutes is pretty good”?

No. Actually, when I started it, I started there, I was used to, like yourselves used to a two-minute build because Ruby, when you get into that world you’re habits change.

Yeah.

And that’s when the realization hit that the expanse of Mingle, the number of features, the number of things that are put together, is significantly larger. So, if I just wrote down a few numbers, so that will give you an idea. So, Mingle consists of about 8,500 tests, units and functionals, and about 3,000 acceptance tests. And these 3,000 tests actually do user-level testing to some extent. Some are user-level tests and some are in-between tests where they verify the functionality at a higher level than just unit. So, we run the units and functionals, 8,500 of them, in 10 minutes. And that’s very, very fast.

The other constraint actually Mingle has is that it cannot be database agnostic. And that is because of the design of Mingle. And this was talked about way back when Mingle was released. Even in those days when there was no NoSQL, Mingle was built to do DDL on the fly, what you call transactional DDL. And a lot of things were written either for Active Record or in the driver. And we needed to ensure that all of that worked every time we exercise the code.

Right.

So, we could not actually use things that were database-agnostic. Or, even our unit tests had to hit the database once in a while. Out of these 8,500 I’m thinking more than a thousand tests do hit the database.

Okay.

And all that had a cost. And we had to live with that pain. But what we went for…

Yes.

Yes. So, we couldn’t ignore that. So, what we went for instead is massive, massive parallelization of these builds.

Nice.

And we, yeah, to put out a few more numbers. Our main build that tests against Oracle, against all the acceptance tests which includes running units and functionals in JRuby as well, because we run them in Ruby and then JRuby, it takes about 40 minutes. And we have 48 agents. And it’s about a hundred cores that run these tests. Yeah, and given the complexity of the app, getting feedback in 40 minutes is pretty good, because our check-in frequency as well as the rate at which we build features is not as fast as it would be in other cases.

Mmhmm.

Just because of the number of objects that interact with each other.

That makes sense. That makes sense. I’m from the school of thought that when I run my unit tests, I have a number that I call the David Brady Twitter limit, which is if my unit tests take longer than seven seconds to run, I’m going to go check Twitter. And that’s going to be a hit to my productivity. And when I come into a project and I see a two-minute test suite, I just start eating my hat because I’m just like, “I am so screwed.”

And one of the things that I have found on several different projects is that yes, if you cannot completely decouple from the database, you’re going to be running hundreds of tests per second at the most. That’s the highest speed that you’re going to get. If you can decouple from that, you can get RSpec to run a thousand tests a second. You can get minitest to run three or four thousand tests a second. But like you said, if you’ve got a custom driver and if you’ve got stored procedures that you actually have to interact with, you’re coupled to the database at that point. So yeah, I won’t give you too much crap about that.

I will ask you though. How long is your really long acceptance test? Just an informal war story, horror story for our listeners. If you were only running that on one machine, how long would that take?

Just one test, you’re saying?

No, like if you wanted to do the full end-to-end, all your functionals, all your units, all your acceptance tests in just one process linearly without 48 machines with a hundred cores, how many hours are you talking to run that?

It would be hours. In a sense, I would go home and come back and it would still be running. [Chuckles]

Yeah. I can actually give you a number for just the unit tests, units and functional, the pre-commit build. If we run, even that is massively parallelized. If we run that set, which runs under 10 minutes on our unit test system, oh sorry, pre-commit system, that itself takes an hour, 40 minutes, if you run them serially. And I did this when I joined Mingle, just find out how long it takes.

Yeah. Wow.

And all of those are just Ruby tests. They’re running on MRI. They don’t hit Oracle. But they do have to hit Postgres, because there are DDLs we need to exercise.

When you’re ready to commit code in your massively parallel system, your whole system takes how long to run? Did you say 40 minutes to test?

Yes. The whole system takes 40 minutes, yeah.

Okay.

Yeah. So usually, here is our way of doing things. Before pre-commit we figure out if everything is working fine. And then we pre-commit and that is a good Pomodoro time for us so that we can take a break.

Sure.

Come back, and by the time we are back we are ready to check into the main branch.

Why are you guys sword fighting? We’re running our unit tests.

Yeah. But this is not a main problem on Mingle. Yes, this does take so long. But this is also our safety net. And a massively, what do you call, very high-confidence level safety net because of the complexity of the app, because of how many years this has been in production and is being used and affects so many customers. A lot of features fall through the cracks or people don’t know because they never use that feature or it was built before their time. So, a lot of our tests catch these things. And we never imagined that this feature does have this kind of an effect.

Mmhmm. Mmhmm.

So, I want to change tactics a little bit. I’m just wondering. What is it about JRuby that makes you choose that over MRI or some of these other things? Rubinius, whatever.

So, main thing that people will push towards JRuby was the way it would be deployed, because when Mingle was designed it was supposed to be a downloadable app and you could put it as a WAR, although it never got released as a WAR. It was always an installer app. But it was sold as you just have a one-click deployment and you can manage it like a Java app. So, that was the big seller. When people run into performance issues, they know what to do and how to tune the Java app, how to change heap size, memory sizes, where to look for things when a Java app goes wrong. That was one big thing. The other thing that we had to build for was to integrate with databases like Oracle where there was no Ruby driver way back when, when this was being developed.

There was an attempt to figure out how this could be deployed on just an MRI stack. And at that time it was Mongrel. Actually they found with tests that performance in JRuby was actually better than Mongrel. And Mongrel had some process hang issues at the time when it was run on a higher load. So, JRuby did give much more confidence to us. Plus packaging them with those servlets like I said which would allow us to abstract out the application from the user where they don’t need to worry about how background jobs are run, how they are allocated, what is going on. That was a key to enable our users to not worry about how this app is built or what goes into this. Rather they could just start and use it and integrate as they would with a JRuby app. A Java app, I mean.

And from the developer point of view, of course it allowed us to do some nice things. For example, whenever we published an upgrade we actually now have a servlet which runs migrations automatically on start. And the user just sees a page where migrations are running and the user doesn’t have to do anything once the migrations are over. It switches over back to the app and the app is now, now suddenly has a few nice features in it.

Oh, that’s a good point, because everyone’s self-hosting Mingle. So, when you make as serious upgrade like changing the database, you’re not just running migrations on your production servers, you’re giving migrations out to your end users and saying, “Good luck with that.” [Chuckles]

Yeah.

What could possibly go wrong, right?

[Chuckles] Exactly, exactly. So, we wanted this to be as simple a process as possible without them having to worry about, what are migrations and why do I care? Yeah, they didn’t really. So, all they had to do was download their latest .tar file or the .exe file for their system and double-click it. And it does everything. It just upgrades you and switches.

So, do you host this for anybody yourself? And if you’re doing that, do you use the general Java stack or something else?

Yeah. So, we used to host this for people. And we used to do the same thing that the user would do, have a Java stack. You don’t need to have a Java stack per se. All you need to do is download the package, because the package itself includes everything you need on Windows. On Unix we expect you to have a particular, what do you call, compatible Java running. And that’s all. And once that is done, you just install the app and run it like any other Java app.

But in the last two and a half years we have moved on from hosting for people. And we have launched a SaaS solution for this. And we host it on the cloud. So, we have one system that just serves I think more than 10,000 organizations now, 10,000 customers.

That begs the question, what has it been like to scale that up?

And this is where [chuckles], where I had joined Mingle and this happened in my day. So, I’m way more comfortable talking about this. So, we started by saying that we really cannot support a lot of customers in hosting because the costs are going up. And we need to provide a SaaS solution. And then we started doing that.

And initially we started by using the most simple technique of building an instance of Mingle for each customer. And obviously, it became not very scalable soon, really, really fast. Like after about 20 customers we realized this is not going to work. And we had, in those days we had a simple LXC-based design where each LXC would host one particular customer. And our plan was to scale that based on load. And that was not going to work for too long.

And in about six months we added multitenancy. And we could do this because we were on Ruby, basically. And the way we added multitenancy was just, we just added a Rack middleware which just interrupts every request and checks the hostname or the site name as we call it. It’s sitename.mingle.thoughtworks.com. So, based on your site name it routes it to a different database schema and different user session, basically. And that’s what has helped us scale it immensely horizontally.

And right now I think we serve about 10,000 sites where we have about, every request is about… so it’s like 30 requests per minute, 30 to 45 requests per minute we serve through our SaaS system.

Hmm. That’s interesting that it was easy to switch to multitenancy just using a Rake middleware. Were there choices that had gone into the design of the app that made that easier? Had you consciously decided to keep a lot of stuff in the individual user sessions or something?

So, all along the app was built to serve one site at a time. And so, all along everything was dependent on the site. What we had to redesign though was how the database pool is used. Because initially when we did multitenancy as well we had individual database pools, database connection pools for each customer. And again, we couldn’t scale that because of restrictions on Oracle and how big the pool could be, and how much that taxes Oracle’s use of memory and cache. So, the major redesign came when we had to redesign the database pool so that the database pool, connection pool can be shared amongst all these users.

Mmhmm.

And since Mingle’s request rate is not so high as Facebook or something, we still can do it. And we still can have customers who have reasonable web response. A lot of the optimization to get a reasonable web response had already been done in the installer app as well, because there is a lot of background processing that Mingle does to ensure that the web requests seem to be really fast.

Hmm. In my experience, adding something like multitenancy usually involves a lot of… usually the headaches that you run into are lots of things that are talking to singleton methods, just assuming that there’s only ever going to be one of these things. And suddenly, there are more than one of those things. Did you run into a lot of that?

I think we ran into such things in our version before SaaS. And we ran into those because of thread safety issues in JRuby. We wanted to run multiple Rails, what do you call, Rails sessions in the same container and we couldn’t do that because JRuby 1.2 was not thread-safe.

Mm, okay.

Yes. And some of the monkey patches that we had done on Active Record and things that are essential for Mingle were not thread-safe. So, I think that was a decision that was taken before we went to SaaS a few generations before me, was to move to JRuby 1.5 that allowed to add threadsafety. And then there was a massive effort that can order all global things, like you said singleton things, to thread-safe local variables. And that actually was the longest delivery cycle that the team had to go through. And I think this was in 2008, 2009. A year almost was spent on doing this.

Okay, so…

[Inaudible], yeah.

So now, so after that change, when you went to get some singleton attribute you were actually talking to a thread-local version of it?

Yeah, yes, yes.

Okay.

And that simplified these things that we added later on, like the multitenancy.

Right, okay.

Yeah, yeah.

That makes sense.

Yes. And actually after we did multitenancy we could then use a lot of AWS scaling techniques to scale the number of EC2 units we were going to use to serve Mingle. And that is how we are doing it now.

Interesting.

So, you mentioned that you’re using Oracle. And I’m assuming Oracle database. I’m wondering. Which JVM are you using? Is it just a foregone conclusion that you’re also using Oracle? Or are you using OpenJDK, IBM, or the GNU compiler?

We used to use Sun JDK in the day. And then we did not depend in any way on a particular type of JDK. I think there were maybe a couple of things we had needed to optimize and we did that effort somewhere between when we went to SaaS and Mingle got a little mature. And since SaaS I think we are… yeah, SaaS was one reason and the other reason was it was so hard to get Oracle JDK for Linux installations and such. So, we made the move to OpenJDK in about 2010, yeah, three, four years ago.

Okay, okay.

So, we are running all our SaaS systems on OpenJDK. And what we package is also OpenJDK now.

Okay. Are you guys still offering the self-hosted version of Mingle?

We do. We do offer for cases where customers are not comfortable putting their systems and data on the cloud. But more and more customers are convinced that actually cloud does offer way more security than they need. And it also has a lot of compliance and they can easily host things on a SaaS. So, a lot of our customers are willingly moving.

That’s cool. I work at CoverMyMeds and we have, HIPAA is the ogre in the room. And so, HIPAA compliance is something that’s very, very essential to us. And so, right now it’s like nothing is allowed to be in the cloud. And we have a fantastic IT department that makes sure that we have all the resources that we need, that we’re using to getting from the cloud. But it’s all in-house and over the VPN. So, I’m actually curious about the self-hosting kind of stuff. Now, if you do the self-hosting stuff do you have to have an Oracle license and an Oracle database?

Yes. So, you have multiple choices actually when you do self-hosting. You can choose to have Oracle, which is what most of our larger customers had. And that’s why we support Oracle. But our database of choice and love is Postgres. So, if you have Postgres you can just host it on…

Yeah.

Excellent, excellent.

That’s awesome. I also want to say it’s awesome that it doesn’t do MySQL. But that’s my own bias.

It used to way back when. And then MySQL just was not good enough. And Postgres did it for us. So, we have been on Postgres since I think version 1.

Yeah.

So, we have made the cultural shift as well as the way people look at our app from being an installer to SaaS. A lot of our customers are now used to us delivering every day. We do daily deployments. So, they actually get a new feature a day. We also have features that are in beta, but you will see them. And then over time they’ll get improved and you will enjoy the [ride] as well. We also engage a lot with our customers because we are on SaaS and we can do daily deployments. So, we have moved over from a three-month, six-month delivery cycle to a daily delivery cycle, or a weekly delivery cycle where the feature, we hypothesize it, we build it, we measure what is going on with that feature, we tweak it, and then push it to the customers.

Nice.

We also use a lot of… so given that we were using JRuby, another challenge we faced was how to [inaudible] the deployment itself. It was an installer app. So, it was this whole monolithic one. We used to put it on AWS, explored it, and then run it. And even that was causing a lot of, what do you call, [inaudible] in terms of how things are managed, and how things are restarted, and the timing between things. Like, oh we need to start this particular node before this starts so that migrations can be run, and so on and so forth.

Right.

Over the course of time though, we have changed the way we deploy, from being the whole installer, one thing includes everything app, to just a plain simple WAR. And we just deploy it on Tomcat. And since we do that, we do not have to do any custom EC2, what do you call, construction when we deploy to Amazon. We just use Elastic Beanstalk. We have this tool that we developed called EbDeployer. I don’t know if you have heard of it. So, we just give it a WAR. It figures our where the WAR needs to go.

It does a blue-green deployment. So, the user doesn’t even know that anything has been deployed. And when the new thing is being deployed, after it’s deployed, we run a simple smoke test checking if we can log in and a couple more things, which is less than one minute. And then we switch. And then they suddenly have new features.

Nice.

So, because [inaudible]…

How do you handle…

We could do all this… sorry, yeah?

How do you deal with, say something breaks, gets through CI and turns out to be a problem in production, how do you deal with rollbacks?

So, since the time we went to SaaS and the way we were delivering deploys, we knew that rollback is not going to work for us because it’s going to be terribly hard to roll back any migration for example, and to roll back or ask a customer to roll back a certain feature. So, we have believed and used the strategy of roll-forward. Put the fix in. But we have drastically reduced the time of how long it takes to put the fix in for even serious issues.

And the same holds true for, like you said, if a feature slips through. Or maybe if there is a security for glibc or what is it, the one before, [inaudible] issue. We have tuned our team or the way it works to turn around in less than a day. So, the next deployment everything will be fixed or we work towards fixing that. To ensure that database migrations need not be rolled back, we only do additive migrations. We do not delete columns. We just do non-destructive migrations. And we also do, what do you call, multi-step deployment in case there are migrations on SaaS. So, we do the migration first. That gets deployed. And a day later the actual feature gets deployed. So, in the day we also find our whether the migration is a problem or not.

Mm.

Nice.

That makes a lot of sense. I like the two-step.

Yeah. We pretty much follow the process that is laid out in ‘Refactoring Databases’.

Oh, okay. I need to read that. If you, let’s say you’ve, I don’t know, dumb example. But let’s say long ago you had a full name column and then you migrated to last name and first name. And then much later you decide to go back to using full name but you’ve already got that old column there.

We haven’t had this particular situation. But if we had this, we would just give the new column a name like new_full_name or something.

Okay.

And then continue with that and let the old full name be there. Yeah, it just doesn’t hurt us anyway.

[Chuckles] Does that ever drive you crazy, just poking around amongst all the… when you’re poking around the database dealing with all the columns and not knowing which ones are still valid and which ones aren’t?

We tend not to poke around with the database.

[Laughs] Good answer. Good answer.

And try to do things in Ruby or through our Rails models and exercise them, because that is the way our app is going to interact with it. There are situations where we need to poke around with the database when a user comes back or a customer, install customer, comes back and says, “Hey, this is not working with us.” And then we go and look at the database. And then that is the only time when we look at the database but not poke around it, just to find out what…

Gotcha.

What could have gone wrong. Yeah.

So, then you’re not in a position to do the, “Hey Jim. Are we using new_first_name or new_new_first_name or new_new_new_first_name?

Have you ever done the experiment of bringing a DBA in and sticking them in front if this database and then backing away and watching their head explode?

We… I shouldn’t say that we have done this experiment and announced it that we have done it. But we do this every time we [inaudible] a customer. And then their head does explode and then they come back to us with a ticket saying, “How come you’re doing this thing?” [Laughter]

With my database. And we just say, “We just do it. It’s fast enough.”

Yeah.

It’s not bothering you. [Chuckles] But we did have such situations where a database was doing crazy stuff. And it was being overloaded and did have significant performance impact on Mingle. And this was way back when we were using [acts as] searchable, I think. This was Rails 1, right? So, we were using acts as searchable and we were dumping the whole index, the search index, into the database. And our solution was at the time, just do some research. And then we came across Elasticsearch. And since then, we have moved over to Elasticsearch. And they don’t see those crazy queries or one-million-row databases anymore.

Mmhmm.

And the customers are way happier. Mingle is happier. Mingle operates really fast. Searches are fast, yeah. And so, basically our motivation has been if a problem is recurring, then look for a solution that would solve it.

Yeah. I worked at a university that pretty heavily used Oracle. And they had a whole team of DBAs to manage it. And so, I’m wondering. Do you ever need that kind of expertise? Avdi kind of alluded to that. But do you ever have to bring somebody in that just really gets Oracle?

There have been instances. I don’t want to deny that there have. There have been instances where were needed serious Oracle help. And that was when we were learning, or we didn’t know a particular quirk or another. And usually we just fall back onto our in-house systems support guy, the IS guy, Barrow Kwan. He does it for us. And he typically has come across one of those problems, or knows where to tune an Oracle database. But over the course of time and over the course of teams, actually the team has built significant Oracle expertise, significant in the sense to tune it to suit Mingle.

One of the things that we have learned is that Oracle is not good at optimizing queries. And we do that every so often. Like, every six months or a year we’ll find a query that is terribly, terribly underoptimized because Oracle just doesn’t unroll and optimize the query to make it fast while Postgres does. And most of our tests are happy on Postgres. So, in those cases we just go ahead and optimize a query and write the prepared statement instead. And then that’s how we optimize it.

We have had… so, at the time when we were optimizing the database pool we used a lot of monitoring on the Oracle RDS in AWS. And that’s where we learned a lot about how Oracle uses its cache and memory management to support a number of connections. So, we learn as we go.

But we haven’t needed a serious Oracle guru on the team.

So, one other question I have is if you were going to be building Mingle now, would you go with JRuby and Oracle or would you use different technologies?

If we were building Mingle now, I think if you ask me personally I would go with Ruby because we are anyway going to be on SaaS. So, it doesn’t matter to be packaged as a Java or anything. And Ruby 1.9 is fast enough. And the process management is now mature enough and [inaudible] and other servers are better to launch a Ruby process. And we would not go with Oracle. We would choose a NoSQL database if we can, because Mingle is so much suited for the flexibility of the NoSQL databases. And it would scale as well.

Gotcha. So, is there a compelling reason then for people to go to JRuby at this point?

Yes, in the sense for things that Mingle is doing, where if you want to package an app and make sure you wanted a very easy way to deploy it, then JRuby is a very good option. Because the person managing that app doesn’t need to, or the DevOps person who needs to deploy that app doesn’t need to worry about how to manage this app and how to monitor this. Because all of that is just from the Java book. And the other thing that actually has helped us with JRuby is how to tune the JVM. Because every so often we see garbage collection issues or memory leaks that we can identify through our monitors. And there are already made or well-learned JVM techniques to tune the JVM so that we can overcome all these issues. Plus the JRuby community is much more mature now, where we do get a lot of, what do you call, Stack Overflow hits and the like. So, we can add those optimizations easily.

Awesome.

A few other things I just wanted to chat about or mention is the way we have changed the way we do testing. Since we moved on to SaaS we have also started using a lot of JavaScript tests using PhantomJS for our testing. We have started using the latest Capybara RSpec tests which run against our SaaS system, which run in a significantly smaller time which is, I think it takes about 10 minutes to run the entire suite, meaning bring up the app and run the entire suite in AWS. And it gives us good enough feedback off what is different between our installer features and SaaS, because there are some differences still.

Alright. Anything else that we should dig into? Dave, Avdi?

I don’t think I have any other questions today.

I’m good. I’m good.

It was really interesting though to dig into…

Yeah.

What you were after and…

Yeah.

The problems you had to solve. So, definitely big thanks for coming on and talking about it.

Yes.

If people want to…

No problem.

Follow you or check out Mingle, what do they do?

To follow me, follow me on @sudhindraRao. And if they want to check out Mingle we are on Twitter @thatsMingle.

Alright. Let’s go ahead and do some picks then. David, do you want to start us off with picks?

Sure. I’ve got three technical picks today. The first one is SizeUp. I may have picked this one in the past before. There are two really good window managers for OS X. The first one is called Divvy and that one lets you drag and describe the size that you want your window to be. And I came to Divvy after using SizeUp. And you can set up Divvy with macros to do everything SizeUp can, sort of. And when I got my new Mac I just got Divvy. I didn’t go get a new copy of SizeUp. And there was something hurting in the way I was working. And I finally realized what it was.

And that is that SizeUp, in addition to… it only lets you half and quarter size your windows and throw them to one edge or the other. It doesn’t give you the fine grain control. But the thing that I love about it is that there’s a hotkey to send this window to the next monitor or the previous monitor. And I absolutely love that. And it’s worth the price. Literally, if I needed fine-grained control of my windows I would turn Divvy back on and I would keep SizeUp and I would run them both. So, SizeUp is, it’s not free but it’s not going to break the bank.

The second pick that I have is called Curb. And this is a library. It’s Ruby bindings to libcurl. So, if you’ve ever written a Ruby script where you shell out to the system and call curl and do something, stop that. You should be using Curb which gives you a curl object that you can then set up posts and queries and set your headers. And it’s all in a very friendly Ruby syntax, so you don’t have to remember the –x and the –f and the –d and all of that fun stuff that you do when you’re playing with curl on the command line. And that’s at github.com/taf2/curb.

And the last technical pick I have is BlueJeans.com. This is a teleconferencing software that we’re using at CoverMyMeds. And it freaking rocks. The most amazing thing about it is that it’s, even on a very bad connection it has very little lag. As the connection goes down, the video starts to get more and more blurry. So, it’s got a progressive decoder that is smart enough to know, oh I don’t have enough bandwidth to download enough progressive frames on this to give you a really highresolution picture. And so, if you’re on Wi-Fi you get grainy video. It’s not bad. But it’s certainly not super crisp. And if you plug into a cable modem and directly into Ethernet with your laptop, all of a sudden you’ve got full HD crystal clear video. And it just works out of the box. And it’s just, it’s fantastic. So, BlueJeans.com is that one. And those are my picks.

Alright. Avdi, what are your picks?

I kind of slacked off on picking picks this week. So, I’m just going to pick something silly. Contractor bags. Contractor bags have been making my life very easier lately. What is a contractor bag? If you don’t know, it is simply a garbage bag on crack.

And now that I have been introduced to them I never want to go back to the regular kind because they’re just garbage bags. But they’re the kind that are made of some indestructible plastic where when you throw sharp things into them they don’t shred into smithereens and dump stuff all across the floor. And I’ve been doing a whole lot of housecleaning lately and clearing a whole bunch of junk out. And they have been making my life a lot easier. So, contractor bags. You can find them at your local hardware store. They’re great.

I can totally see that. We cut the plastic to make the bags, but we’re not going to tell you how. [Laughter]

I’ve seen a few things get through them. But it takes a really sharp wire or something, something thin and sharp.

I want to see Mythbusters throw dynamite in them. [Laughter]

I just want to see a family hold them out as a curtain and play red rover with their kids.

I don’t know what that is.

Red rover is where you link arms and you call somebody from the other line that they run and they try to break through the hands.

So, the mental image is your kid running full tilt straight into this bag and then going flinging backwards. I just find that a great image. [Chuckles]

Yes, yes. The image of children running into plastic bags, yes.

Yes.

Let’s leave our listeners with that. [Laughter]

Yes.

Here comes Johnny. I think we killed Johnny. [Laughter]

This bag is not a toy unless there is adult supervision. Come on. Let’s be responsible here. [Laughter]

Alright. I’ve got a couple of picks here. My first pick is I actually, I broke down and bought an iPad, which means I was super excited to get it. I got the iPad 3 Mini. And I got this OtherBox case that comes with it called the Defender. And it has the screen protector built into it. It’s completely indestructible I guess. And I have this awesome warranty on it. It protects against spills and drops and everything. So, if your iPad gets broken in it, they pretty much will fix it. But the other thing is that it has this cover that goes over it that has a stand on it. So, I can actually set the iPad in it three or four different ways. So, I’ve actually been using it with OmniFocus to get my to-do’s in. And I just have it sitting next to me on my desk. And so, I use it for that and I use it to play podcasts and things like that. And so, it’s been super awesome. And so, I’m going to pick both of those things.

And then the other thing I’m going to pick. I’m not a hundred percent happy with this, but I haven’t found anything better. For JSRemoteConf we’re actually using some software called ClickWebinar. The things that I really like about it are that it puts a video of the speaker, the slides, you can embed YouTube videos. You can share a screen on it and a bunch of other stuff. It also has a chat running on the side. And it has a Q and A feature which is awesome. So, I switch it to Q and A mode and then people can basically raise their hands. And when I pick them then they can actually enable their voice and talk to the speaker for a minute and, “Hi. This is my question,” and get clarification and stuff like that. It’s just really cool.

The downsides are it’s built in Flash. And I’m running Mac OS Yosemite I think is the latest version of Mac OS. Whatever the latest version is, that’s what I got. And their driver for screen sharing doesn’t work on my Mac. But everything else seems to work well. Also, uploading keynote files for some reason doesn’t seem to work. But you can get around that by uploading a PowerPoint so you just export it. It’s not perfect but it works. Anyway, it works way better in my opinion than things like Google Plus Hangouts or GoToMeeting. So, those are my picks. Sudhindra, what are your picks?

I’m going to go with ‘Working Effectively with Legacy Code’, Michael Feathers, the book. I think may people have read it.

Woo! Classic.

And it’s over and over and over every time I come to a project, this is the first thing I do is hold this book, restart with the process. And in a day or two, we are done with the refactoring. And we have learned a lot just by writing tests and following the techniques mentioned in that book. So, that is going to be my one book.

And the second pick is going to be actually a sequence of books. ‘Thinking Fast and Slow’. I learned a lot from that book just the way a brain processes information. Then the recommendation from that book was ‘Nudge’ about how people can be nudged to do things or make decisions. And the third thing, the third book is just follow-up from ‘Nudge’ is ‘Power of Habit’. It’s how to learn how people do things and tune your business or your apps to their behavior. And then make them adopt a new habit. And that’s what actually a lot of successful companies are doing, is trying to teach people new habits and in terms of getting to use them.

Is that the Charles Duhiq book?

Yes, that is.

There are actually two books called ‘The Power of Habit’ on Amazon. So, you have to specify the author.

Charles Duhiq, yeah.

The Duhiq book is awesome, yes.

Yes it is awesome, yes. I just recently finished it. It’s amazing.

And since I am what we call under the covers internet of things geek, I like to talk about these things. And one Kickstarter that I came across is called Blynk, B-L-Y-N-K. And their aim is to allow you to build an Arduino project in five minutes. And a project that will allow you to control your Arduino from your phone, iPhone or Android. So, that is something.

And then one last pick is a tool that will make embedded Linux really easy so that you can put embedded Linux on any processor of your choice. It’s called Buildroot. And that allows you to cross-compile it to any processor, like your Raspberry Pi. Or if a new processor comes up maybe from Colcom or something. And you can generate embedded Linux systems for that. So, this is what I’m going to do in my break this weekend, figure out how to put that on my Raspberry Pi.

It’s not fair for you to pick so many toys.

I am just full of toys.

Well, you’re fired.

Alright, well I don’t think we have any other announcements or anything. So, we’ll go ahead and wrap up and we’ll catch you all next week.