Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
In this episode of Top End Devs, host David Camira is joined by panelists Luke Stutters and John Epperson, along with special guest Jesse Spivak, a senior engineer at Ibotta. Jesse shares his experiences and insights from a challenging project at Ibotta, where he navigated through four critical mistakes. These included choosing the wrong technology, siloing work, falling into premature optimization, and making too many changes at once. Jesse explains how these mistakes jeopardized the project but ultimately led to valuable learning experiences.
Special Guests:
Jesse Spevack
Show Notes
In this episode of Top End Devs, host David Camira is joined by panelists Luke Stutters and John Epperson, along with special guest Jesse Spivak, a senior engineer at Ibotta. Jesse shares his experiences and insights from a challenging project at Ibotta, where he navigated through four critical mistakes. These included choosing the wrong technology, siloing work, falling into premature optimization, and making too many changes at once. Jesse explains how these mistakes jeopardized the project but ultimately led to valuable learning experiences. The conversation also touches on the importance of discussing and learning from mistakes openly, the complexities of transitioning to new technologies, and the significance of making systematic, verified changes. Additionally, they delve into the evolving landscape of developer interviews, aiming to create a more inclusive and positive experience. Join us as we explore the trials, lessons, and growth that come from navigating the highs and lows of software development.
Transcript
David Kimura [00:00:05]:
Hey, everyone. Welcome to another episode of Ruby Robes. I'm David Kamira. And today on our panel, we have Luke Suttters
Luke Stutters [00:00:11]:
Hi.
David Kimura [00:00:12]:
Stuttters. And we have John Epperson.
John Epperson [00:00:14]:
Hello, everybody.
David Kimura [00:00:15]:
And today, we have a special guest, Jesse Spivak.
Jesse Spevak [00:00:18]:
Great to be here.
David Kimura [00:00:19]:
So, Jesse, would you mind explaining just a bit about who you are, some of the things that you're doing, who you work for, and why you're famous, and all that good stuff?
Jesse Spevak [00:00:27]:
Yeah. Absolutely. So my name is Justin Spudak. I'm a senior engineer at a company called Ibotta, which is a cashback for shopping app based in Denver, Colorado. I've been working there for about three and a half years. We are doing some hiring, so check out our careers page. I guess I'm I'm famous as it were because I gave a talk at the first remote Rails comp this past May,
Luke Stutters [00:00:52]:
and
Jesse Spevak [00:00:53]:
I talked about kind of how crummy of a developer I am.
David Kimura [00:00:57]:
Yeah. I think we can all relate to that on a daily basis sometimes. So would you mind giving a bit of a, you know, highlight talk about what you covered at the conference and stuff so we can just kinda pick it up from there? We'll link to the in the show notes, a link to the conference, but just for those who maybe didn't see it.
Jesse Spevak [00:01:17]:
Sure. And and there there's no substitute for actually watching this fantastic talk. But in in more more seriously, the talk is really about my experience as a tech lead at Ibotta, working on a pretty critical project over the course of about 6, 6 months or so. And over that time, I made 4 very big mistakes that put the project in jeopardy. And hopefully they're mistakes that I will learn from and not make again as I continue to lead projects that I brought in, in the future. And my hope is that by sort of articulating these mistakes and what I learned from them, other folks can benefit. And so the 4 mistakes that I made are first, we picked the wrong technology. You can get more into that.
Jesse Spevak [00:02:09]:
We also, as a team, we siloed work. So work was divided up in not the best way. We fell into the premature optimization trap. And then, maybe worst of all, we made way too many changes at one time. So I can I can go into detail on any of those?
David Kimura [00:02:27]:
Yeah. And I think it's fair to say that I, our listeners, other members on the panel have been there before on
Luke Stutters [00:02:35]:
Yeah. I've I've worked with loads of people who've made those mistakes. Obviously, I've never made them myself, but I've oh, my words.
John Epperson [00:02:41]:
Oh, my words. Luke Luke is the perfect developer. Let's be fair. No. I I was actually going to jump in and say that, like, I feel like I I don't feel like I'm alone, but usually when you make a mistake, it cascades into more. For whatever reason, I generally feel like you either you either kind of a rolling along and you feel pretty good about stuff, or you you're just, like, in a bottomless pit. And there's very little in between time. You just suddenly notice that you're you're at the bottom.
John Epperson [00:03:11]:
So yeah. Just saying.
Jesse Spevak [00:03:13]:
And absolutely. And I just wanna kinda call out that it's there's, like, a certain amount of privilege that comes with being able to talk about our mistakes. Right? I'm not worried that my boss is gonna fire me. And I'm also not worried that folks won't take me seriously for giving this talk. If anything, it probably improves my reputation as evidenced by getting to talk to 3 of you gentlemen. So I just wanna call that out because I I I think it's important.
John Epperson [00:03:40]:
I I think that oh, okay. So so speaking of that, I think there's a give and take there too. So in I think this is one of this will get into a thing that I care a lot about this particular subject. Because for most of my shoot. I still feel it today or whatever. It doesn't really matter. I still always feel this pressure to not let people know about the mistakes that I made. Right? Because letting people know about the mistakes that I made makes me more vulnerable to them not wanting me to work for them.
John Epperson [00:04:08]:
Right? Like, fire me. Whatever whatever the bad thing is that it's a boogeyman. It's not real. It's just a pressure. Right? And so I guess what I'm trying to get at is, like, I kinda feel like one of the things that I always care a lot about with mistakes is telling people about your mistakes. I actually have discovered in reality actually makes people respect you more, but it doesn't change the fact that it's, like, really hard, and I still feel that pressure to this day.
Jesse Spevak [00:04:39]:
Yeah. John, yeah. I I agree with what you're saying. I think that for a lot of us, talking about mistakes openly and honestly, and what we learn from them actually builds our credibility. But that's not always the case depending on sort of how you present, what you look like. I think that, you know, you guys recently did an episode, I think, where you you talked about issues issues of hiring and getting equality in a workplace, and I think that that that plays in here for sure.
David Kimura [00:05:08]:
And I would go as far to say that it depends on what the mistake is, what kind of mistake it is. If it is just utter and complete negligence, then those aren't really the kind of mistakes I would really wanna be forthcoming and outright about. You know? But, like, if I went in and always had a VPN tunnel into my production environment, then we got malware in our local work environment, and that transferred over to production and encrypted all of our production data. I don't know if I would really say, like, oh, yeah. You know, that was a silly mistake of mine. No. It's like, okay. Not only have our customers affected, but now, you know, my job's in jeopardy because I decided to always take shortcuts.
David Kimura [00:05:56]:
But something like and what I'm really interested in is your technology, framework or the you used the wrong technology. Can you explain a bit more about the scenario?
Jesse Spevak [00:06:08]:
Sure. Absolutely. So this the high level, we at Ibotta, we have just a wonderful, majestic monolith, Rails application. Actually, originally, the application we've written in Scala. And then after about a month of that, they switched it over and rebuilt the whole thing in Rails, and it's been Rails ever since. So it's going on close almost 10 years at this point. So it's a large application, it serves millions of users. Very cool.
Jesse Spevak [00:06:34]:
And about 3 years ago when I joined, we started thinking we started growing the team and thinking about how we could, like many people have, pull pieces out of the monolith into microservices. So this project in particular was about taking a piece of billing logic from the system, from the monolith, and pulling it out into a microservice. The hope was to make it better encapsulated, easier to iterate on, not depend you know, isolate dependencies, every reason that you think to build a microservice. So we chose actually, before I say the technology, before I get trolled by all the lovers of this technology, I'm gonna preface this by saying, I don't think that this technology is wrong and I don't think it's bad in and of itself. I just think it was not the right technology for the problem and the team.
Luke Stutters [00:07:26]:
Hold on a second. Hold on a second, Jesse, because the slide I'm looking at says big mistakes. It doesn't say small mistakes. It says big mistakes. So they should keep this in mind before we reveal what this technology is, what a big mistake was.
Jesse Spevak [00:07:41]:
Absolutely. This whoever whoever uses this technology here is is definitely making a big mistake. So I know, spoiler, we weren't building like us, a side rails app microservice, which would probably would not have been as big of a mistake. But the issue really was that our team, the the small team of developers, and then the larger team of engineers in the company, really did not have a ton of experience with the framework that we chose. And as a result, we ended up having to do a lot of plumbing, and reinventing the wheel, and just not benefiting from the institutional experience that exists at Ibotta. And unfortunately, right, this, this could work if you're doing kind of like a proof of concept, like, let's show what this technology can do. Let's pick a pretty isolated use case. But the billing logic that we were pulling out about Monolith was basically like do or die, right? If it did not work, it costs 1,000,000 of dollars to fix, or, you know, it ends up costing the company 1,000,000 of dollars.
Jesse Spevak [00:08:46]:
So it really, we were walking on a tight rope and there was no net underneath us. And unfortunately, we decided to, I guess, like walk on our hands instead of, go across normally. So the framework that we used is called ACCA. And I think for a team that knows Akka, this probably would have been really a perfect tool for the job. But our team and our company really did not have a ton of experience with Akka. And so unfortunately, we weren't able to sort of take advantage of it and use it in a way that sort of professional Akka developers likely can.
Luke Stutters [00:09:20]:
So this is kind of like a functional programming thing?
Jesse Spevak [00:09:23]:
Right. It deals with data streams and passing data along in a functional paradigm, and it's meant to accommodate high volume data across highly parallelized system. So, you know, at the time, we went there, well, I'll talk about why we went there in a second, but it, in retrospect, it was, it was something that could handle basically 10000x really what we needed in terms of what it was designed to handle. So just sort of on, on paper, it probably wasn't the best, the best move in that respect. But I could also talk about the team as well and why it wasn't a great a great fit.
John Epperson [00:10:02]:
Yeah. Can you can you talk for just a minute about, like, what was the problem that you were trying to solve? Why did you choose like, what did you need ACCA for?
Jesse Spevak [00:10:11]:
Sure. Yeah. Perfect. So the problem that we're trying to solve, and you have to know a little bit about the Ibotta app. So I assume all of you all of you guys have, downloaded it and are actively using it.
Luke Stutters [00:10:23]:
Absolutely. It's the best app I've ever used.
John Epperson [00:10:25]:
I've definitely looked at it and I was like, nope.
Jesse Spevak [00:10:30]:
That's funny. We did a search. Well, I'll get to that later. But basically, Ibotta is a way for you to get digital coupons. You can get brands put offers in the app, you click on the offer, you show us evidence that you bought the thing that is on offer and then Ibotta will pay you cash back. They'll put, send it to your PayPal account, give you a gift card to Amazon, whatever you want. So the problem that we're trying to solve is how do we make sure that the offers in the app don't exceed the budget that is allocated to them by the brands that put those offers in the app? And that sounds maybe like an easy problem, like there's an easy way to just say, okay, there's $500,000 of budget for Oreo coupons. Just divide $500,000 by how many, how much money we're giving out per coupon, and then you know.
Jesse Spevak [00:11:19]:
There's actually much, like, it's obviously much harder than that. And in order to preserve a good user experience, we need to make sure that we're not yanking content and surprising our users. Like you would be really upset if you went to the store specifically to buy Oreos to get a coupon, and then by the time you checked out, the coupon is no longer in your application. So we have to learn some predictive algorithms to basically guess when we're gonna run out of money and kind of slow the coupon, slow the velocity down as we approach that point.
Luke Stutters [00:11:52]:
Oh, dear. That's that's a that's a that's a dangerous recipe. I must add that I I don't think the bottle is available in the UK at the moment. Is it available in Canada?
Jesse Spevak [00:12:03]:
We are only in the United States right now. Sorry, Luke.
Luke Stutters [00:12:07]:
So That's why it's in my defense. This looks like the perfect storm for me because you've both got the kind of strong financial components. So if you get it wrong, you lose money. And if you get it long, then people will have wasted their time going to the shop, you know. In Britain, it's so small, I can walk to the shop, you know. Sometimes I just open the window and shout at them and tell them what I want. But, you know, I know when I was living in the States and people maybe drive for 2 hours to go to Walmart. So this is this is this is quite a quite a problem you've got there.
Jesse Spevak [00:12:41]:
Yeah. It was it's interesting that you say that because right when I I joined the company, I was kinda put on the team. I was actually giving, you know, my tech buddy, my mentor when I joined the company. This was his project. And as someone new in the company, it was very, it was really overwhelming problem space. Because basically these campaigns and application almost have like a physical momentum to them. So if you imagine trying to stop a moving train, you can't just hit the brakes and expect it to stop on a dime. You have to apply the brakes over some distance to slow the train down.
Jesse Spevak [00:13:17]:
And that's really how the content in the application is modeled. And it's I'm not a physics person, and so it's really confusing.
John Epperson [00:13:26]:
So alright. So this seems a slightly I mean, it's a it's a twist, right, on your classical inventory problem, right, is what it kinda sounds like. So the way that Aka came into this is is what you were taking, like, what all these requests from users and trying to decide whether or not you're gonna, I guess, give them the coupon or not or something?
Jesse Spevak [00:13:50]:
Yeah. That's that's that's sort of right. So we have an event based architecture where our system and for the for folks who aren't familiar with that, that means that basically your system publishes events, which is data that signify that something has happened of interest in the system. So maybe like, you know, shopping cart loaded is an event that you might have in like a typical inventory space or something like that. So we have events that we're interested in, like content awarded, which means that John went to the store, submitted a receipt through the app, and got cash back. So the content has been awarded. So we listen for those events in order to keep track in real time of how much budget is being used, and we basically track that over time to make a rough prediction about how fast things are moving. And so, Akka, so sorry, John, to get back to your original question, Akka is really good at streaming data, at streaming large amounts, high volume, data.
Jesse Spevak [00:14:54]:
And Ibotta will get on the order of several 100,000 of these content awarded events per day, which seems like a lot, but it's actually much lower than I think what Akka can kinda deal with or is designed to deal with out of the box.
John Epperson [00:15:09]:
Yeah. Did you so did you consider alternatives? Did you reject them for various reasons? And and I guess so, obviously, ACCA didn't work out for you, so you must have picked something else that you like better. And how did you arrive at that new thing, and what was that, if that makes some sense?
Jesse Spevak [00:15:28]:
Yeah. Perfect. So yeah. We we basically this this comes down to some some some team issues again and not not an issue with Okta. So the team issue was basically that at the start of this project, we scaled up our team. We're like, this is a lot of work. We need, we need to bring in some artillery. And we brought in a new developer from outside the company who is awesome.
Jesse Spevak [00:15:57]:
She's a rock star and had it was coming from the ad product space. And with dealing with volume, much volume of streaming data at scale much higher than what we needed or we're going to be dealing with in any near near future. And, you know, she was coming from, I believe in Akasha. And so she joins the team. We're excited to have her. We think she's a rock star. I mean, she is a rock star. And she's like, this is a perfect use case for Akka.
Jesse Spevak [00:16:29]:
We're like, okay, never heard of that, but, you know, we trust you. You crushed our interview, and we think you're amazing. So, yeah, that sounds pretty good. And again, this isn't a knock against Akka. I think this is just that. In our company, we have a ton of infrastructure set up to support the tools that our company has sort of blessed that that are kind of frequently in use. In fact, we have like a name for that, we call it the paved road, which is like, if you're an engineer, you have a lot of autonomy out of what you use, but if you stay on the paved road, it should be an easy path. And Akka, which is a JV, you know, which we use with Java or Scala typically, is not on Ibotta's paved road.
Jesse Spevak [00:17:15]:
So we had to kind of have a contentious meeting, a contentious conversation with the architects at Ibotta and say, No, we really believe in our developer. And we think that she knows what she's doing. And we're ready to sort of make our bed and lay in it. And they were like, Well, I know as long as you know that. So they they let us kinda they gave us just enough rope to I forgot how the rest of that goes. To hang ourselves with.
John Epperson [00:17:43]:
Okay. So so it sounds like you you had an advocate, and that's kinda how you landed on it. Where did you go after you decided that ACCA was the poor choice? Like, what did you end up with?
Jesse Spevak [00:17:53]:
Yeah. So this and this kind of gets into the next problem, which is this the siloed work. So we're starting to work on building this ACCA powered race car of a microservice to pull billing logic out of our rails monolith. And there's sort of two streams of work. There's the development of the Akka microservice, which is we were writing, using Kotlin, which is a JVM. It's kind of a nicer Java. It's what Android apps are written in. It's actually pretty nice.
Jesse Spevak [00:18:24]:
So we have that work going on, and then we have to integrate that microservice, sorry, into, into the Rails model. So I ended up taking on a lot of the integration work and the other developer took on most of the Akka and Kotlin work. So we have these sort of 2 very isolated pieces of work that are siloed. And then, something terrible happened. And I don't blame this person at all because I wouldn't want to be on my team anyway. She decided that she was much more interested in data engineering and she moved to a different team in the company. So, and that's actually, you know, a great thing about working at Ibotta. Like we've got tons of really cool problems, different spaces, and people are almost encouraged to move around to find things that they're, that they like and are good at.
Jesse Spevak [00:19:13]:
So, she made this move. And now, now the, the problem of using a framework that none of us had experience with, and that the general company didn't have a ton of experience with, really became self evident. Because now there was, there was still work left to do on the microservice. And I'm just, I'm just a rails developer. Like, I had to go into there and start writing Kotlin, start reading Yocket documentation to try to wrap my head around what it what this whole, you know, actor system meant. And, it it was, it was tough. And that's why I call it a big mistake. So that's sorry, Dave.
David Kimura [00:19:53]:
You know, from my experience too is that, yeah, Ruby is slow. Now, there's no getting around that when you compare to a compiled language. But holy crap, is it fast too. You know, I have a production application which processes over 500,000 active jobs every single day. And that it does it extremely quick. I don't need it to be faster. I mean, that's plenty fast. On the current setup that it's on, which is 2 cores and 4 gigs of RAM, and we have 2 servers dedicated to the background job.
David Kimura [00:20:29]:
So 2 VMs. It's able to handle that load, and we don't have to worry about it crashing or anything like that. So I mean, that's good enough for us. Now I imagine that it would be able to double that workload before we ever ran into any kind of performance issues where we needed to start scaling up.
Jesse Spevak [00:20:48]:
Absolutely, Dave. And we were originally in the monolith. The way the system worked was it ran on a background scheduled job using Rescue, and the scheduled job basically took roughly 45 minutes to run. So it was running like, kinda like a waterfall, like every 10 minutes, we'd start a new job that would take 45 minutes to run. And so going from that to basically completely real time is a huge improvement. And if, you know, real time for 500,000 events per day versus 5,000,000 events per day are different things, but if you're at real time, it's already this just enormous improvement over what we were working with, which is like this 45 minute loop. So at that point, and I guess I'll say another thing, before I get into, like, how we fixed, how we fixed my mistake. This is actually getting to what Dave said.
Jesse Spevak [00:21:41]:
This is a case of premature optimization. We sort of didn't do the back of the envelope math well enough, or didn't have a clear picture of like, okay, this is really like designed to handle like literally a 1000 times more traffic than our best day. So, you know, we, we didn't that was a, that was definitely a mistake. Like, we should have asked that question and realized, okay, maybe, maybe this is a little too much horsepower, we don't need this, it's too much trouble for what it's for what it's buying us. And on my end also, I was imagining, you know, getting all these events, we call it so the the microservice was producing, like, prediction events. Okay? So predicting every time a piece of content should come down and making an adjustment about when it thinks that that should happen. And so I'm imagining the monolith is listening to these prediction events, subscribe to an SNS topic or SQS queue. And I imagine, you know, thousands and thousands and thousands of events every second, which is way, way too much.
Jesse Spevak [00:22:41]:
And so I was thinking of like all these clever ways to do caching and like to try to figure out when can I drop events so I don't need to hit the database, like when can I, how can I come up with these clever ways to only make round trips to the database when I really need to? And that added time to the development, And it also added a ton of complexity. So that when our 2 systems, when we're like, okay, let's turn them on, let's see what happens, the first error that comes out, because of course there's gonna be an error when you, when you first try it out, That was really hard to debug. It was really hard to understand, like, is this a caching issue? Is this actually an issue with the data coming from microservice? Like, it was really hard to isolate that. And then that obviously moves into the 3rd problem, 3rd mistake was that we made too many changes at at one time. So all these were ways that I tried to shoot myself in my foot in my own foot while working on this very important project.
John Epperson [00:23:38]:
Yep. Makes sense.
Luke Stutters [00:23:40]:
I'm not sure if I understand it, but so you're saying there's kind of 2 separate things going on here. Firstly, you're moving to a completely new technology. So you're taking a large part of the app out. You're putting in a micro service. It's not very micro if you're doing, what, 700,000 things a day. That's not a micro service. That's, your your this is this is this is this is a macro micro or medcro. And the second thing you're doing is you're actually changing the whole interface.
Luke Stutters [00:24:12]:
So you're saying you're going from you you previously had a rescue batch job, and now you're saying, no. No. No. We're not gonna do batching. We're gonna do it all immediately.
Jesse Spevak [00:24:22]:
Yeah. Luke, that's a really astute observation. So we were changing structure and we were changing behavior at the same time, which is something since that point, I've really tried to avoid doing. Even at, like, the micro PR level. If I have a PR that I'm gonna push up, it's either going to add behavior or change behavior, or it's gonna change the structure of the code. And I'm gonna try to not do the 2 things at the same time. So in this case, we did the 2 things at the same time. And at multiple points, we should have known or we should, or in retrospect, we should have known to pause and verify, right? And if we couldn't verify, take that failure as feedback and iterate.
Jesse Spevak [00:25:02]:
And I think that that's really what really strong engineers, experienced engineers know to do, make one change at a time and verify. Like, I haven't it's I guess it's kinda like when I run rspec, right? Or rspec feels like super natural to me. So every time I hit my test suite, I'm like, okay, I expect this to happen. And like, sometimes I'm right, and then sometimes I'm wrong, and that's like really interesting too. But making that initial guess of what's gonna happen is super important. And I think we need to slow down and stop and say, okay, here's our guess as to what's gonna happen. Here's how we're gonna verify it. And if if we can't verify it, here's, like, the corrective action you can take.
John Epperson [00:25:42]:
Yeah. So I think you're you're hitting it, like, one thing that's, like, really important. Right? So one of the things I think that make that that really kind of makes you a strong engineer of sorts. Right? It's not necessarily, like, do you avoid mistakes? Because, shoot, for whatever reason, they keep happening to me. Right? It it's are are you good at cleaning up? Right? Are you good at figuring out that something's not right? Something smells funny. Whatever it is, can you go back and, you know, sweep up your mess? You know, either push it across the finish line because you're close enough or say, actually, this is the wrong path. We should actually back up, go to the last intersection, and go down the other way. So, I I mean, it's definitely yeah.
John Epperson [00:26:26]:
I I I mean, I can speak to experiences too where I was just like, man, I really, like, missed all the flags. And then I kept missing more flags and just kept going. And and why are we surprised that we're in a bad place? Because I, like, ignored everything. So I hear you, man. It and and kudos for for recognizing it at some point and and, and doing something about it.
Jesse Spevak [00:26:48]:
Yeah. So what do we do about it? We unfortunately, I well, not unfortunately. It was a good it was a good learning experience, but, obviously, my happy place is is deep in a in a legacy Rails application, finding all the pathways there, but I had to get really push myself, get out of my comfort zone. I had to pick up Kotlin. Luckily, Kotlin is like a super friendly language, I think, to get into, especially for folks who are coming from Ruby because I think that there's, like, kind of like an emphasis on syntax, on, like, syntactic sugar, on, like, making the code actually, like, readable and look nice, whereas that's not always the case with all JBM languages. So, in that case, in that sense, it was kind of friendly. Also at Ibotta, we were able to like organize kind of a learning group. So, there were lots of folks who were new to Ibotta or new to Kotlin who were kind of trying to solve these similar problems.
Jesse Spevak [00:27:42]:
So, we made a study group, we found cool online coursework, we held each other accountable for making sure that we were, making progress on those things. And I started writing Colin in the microservice to try to, like you said, push across the line. We need to get this across the line, and then we can kinda circle back and deal with some of the underlying problems. And that's important because, like, as engineers, like, at the end of the day, we're trying to deliver value to the businesses that we work for, and not just, like, try out new technology or, like, optimize what techno what's what technological solution we're we're we're implementing.
John Epperson [00:28:19]:
So to just to clarify what what you're saying, you kept you you kept ACCA. You just, you just pushed it across the finish line despite all your your problems.
Jesse Spevak [00:28:31]:
Yeah. So That's
John Epperson [00:28:32]:
how you resolved it. Okay.
Jesse Spevak [00:28:33]:
Exactly. So we have we have problems with it, and we decided, let's get this thing, like, we're close enough, even with the problems, even with our lack of understanding. We're close enough that a total rewrite right now would put us back a long way and and upset the business. So let's push it across the line, and then we can circle back and figure out what to do. So that was that delivering that value, even though it it kinda felt yucky, even though we knew that there were things that we're gonna need to change over the long term, bought us some time and bought us some credibility in the organization.
David Kimura [00:29:10]:
And I wanna circle back to the point you made about moving too quickly or too many changes happening at one time. And I think that's something that a lot of developers might start to experience soon with the whole removal of jQuery from Bootstrap 5.
Luke Stutters [00:29:28]:
So No. Not my jQuery. No. Sorry.
David Kimura [00:29:35]:
So the idea is that Bootstrap 5 no longer has the jQuery dependency. So you might plan to upgrade your rails application, the CSS framework from Bootstrap 4 to Bootstrap 5. And you might say, like, hey. Well, why we're making this change? Why don't we go ahead and rewrite a lot of our JavaScript that was also jQuery dependent? And so I would say, instead of doing that, do one thing. Either first, remove all your jQuery dependency within your application and just have jQuery be a dependency of Bootstrap. And then in another iteration, upgrade your Bootstrap version, removing then jQuery entirely, or do it vice versa, where you do your Bootstrap upgrade first, and then you do your jQuery removal from your application as a dependency. But trying to do both side by side, it's too big of a task, you know, for, you know, 1 person or one team to do right away. I would just handle one thing at a time.
David Kimura [00:30:38]:
And moving slower, you're gonna say, okay. What broke this? Was it the new Bootstrap framework that broke this, or was it our jQuery rewrite? So that way, you're gonna be able to identify a lot more problems quicker before they are reported to you by the customer.
Jesse Spevak [00:30:55]:
Absolutely. I think the the most experienced engineers, you know, one of the best engineers I've worked with, his name's Justin Hart. He was the, like, did the first commit on the on the monolith that I work on. And every time I work with him, that is always my experience. Like, he is, like, we're making this change. Here's what we expect, and we're gonna do it, and then we're gonna verify it, and then we're gonna move on. And it and I'm always like, oh, why don't we do this, this, and this? And he's like, no, no, no. Just this one thing.
Jesse Spevak [00:31:23]:
And it it's illuminating. Right? It it's illuminating. I think it's helped me as a developer in the projects that I'm I'm doing now realizing like, okay, this thing that I that I wanna do, it's actually like 4 different things, and I'm gonna do the first the first thing first, verify it, and then move on from there.
Luke Stutters [00:31:39]:
If you or your loved ones were affected by the removal of jQuery from Bootstrap, why not check out the excellent course from dev devchat.tv? You don't know JavaScript yet 30 day challenge available with the link in the episode shown us.
John Epperson [00:31:55]:
Were we scheduled for, advertisement there? That's just come out of nowhere.
Luke Stutters [00:32:01]:
Just trying to just trying to just trying to keep the ship afloat, man. Come on.
John Epperson [00:32:05]:
Hustle. I was got so this goes along for me with,
Luke Stutters [00:32:10]:
No. No. No. No. No. No. No. No.
Luke Stutters [00:32:11]:
No. No. No. We have to do jQuery now because this is really affecting me. I mean, this is this is terrible news. I I literally can't write JavaScript without putting a dollar sign in front of everything.
John Epperson [00:32:23]:
You can still have a dollar sign. It you just need to assign something to the dollar sign. That's all.
Luke Stutters [00:32:27]:
One of the reasons I started using Maybe not
John Epperson [00:32:30]:
use jQuery.
Luke Stutters [00:32:31]:
Was because it has a little dollar sign in front of a data structure, and that kind of really makes me feel at home. Plus, you know, it's it it feels like money. You know? When I type the dollar sign in, I feel like, yeah, this is gonna make me a millionaire.
John Epperson [00:32:43]:
Oh, man. I'm sorry, Luke.
David Kimura [00:32:44]:
So I guess while we're on JavaScript, I think another premature optimization or rather I like to call them premature deoptimization is creating a new Ruby on Rails application with React, the dash dash webpack equals React. Just thought I'd throw that in there for the obligatory bash on React.
John Epperson [00:33:03]:
I do like stimulus, and I I do feel like I do feel like we can we can let React go now.
Jesse Spevak [00:33:08]:
Yeah. I mean, I don't know. Have you all checked out the Basecamp's email, hey.com?
Luke Stutters [00:33:15]:
Mhmm. Oh, yeah. Yeah. What do you think?
Jesse Spevak [00:33:19]:
Yeah. So I've been I've been using it, and I obviously, I'm a Basecamp fanboy, but I think it's it's pretty amazing what what they're able to do with just HTML over the wire for the most part and not, not relying on some of the frameworks that have come out recently. Also, I think it's super interesting, like the 2020 root on Rails community survey. If you look, the one put out by, Planet Argon, if you look at, like, what JavaScript libraries people are using, so I was expecting that number 1 would be React, and maybe number 2 would be, I don't know, Vue or, like, Ember maybe. But number 1 on there is jQuery. And it's like we're all in this in this jQuery world, and it's not it's not bad. I love jQuery.
John Epperson [00:34:02]:
So so for all of you who along with Luke are mourning jQuery, I'm telling you, stimulus. Go check it out.
Luke Stutters [00:34:10]:
Try it. Understand it.
John Epperson [00:34:12]:
I don't have an what that's fair. I I I thought I found it pretty easy. I I thought it gave me everything that I felt like I was getting from jQuery before, which is a thing happened on the page. And because that thing I mean, Jesse talked about event based architecture here. Come on. This is exactly what JavaScript is. Right? And jQuery, especially. Right? Stimulus is exactly that.
John Epperson [00:34:36]:
Event happens, do something else because this thing happened. Right? Just saying if you love that event based architecture style, stimulus is that. It's just way prettier. And I found it a lot easier to use. And because I don't like dollar signs, I was happy about it.
David Kimura [00:34:54]:
Yeah. And, look, if you want a bunch of different stimulus JS tutorials, check out Drifter Ruby, man. I have a whole bunch on there.
Luke Stutters [00:35:03]:
It's a big plug episode. Everyone quickly plug their thing. Do you know what? Actually, Dave, I have already checked that out, and I still don't understand it. So I need to check it out again. It's it's definitely me. I tell you I tell you what the problem is. Right? I've been leaning on jQuery for so long, and I'm talking about 10 years. Right? And it's not just that.
Luke Stutters [00:35:26]:
I've got my whole arsenal of weird front end stuff that I can pull in. And replacing that big long list of handy widgets, I know will solve this problem, is the is the is what I'm lacking. So yeah. Basic GOM stuff fine, you know, ES 6 is all the way. But if I wanna do something weird, if I wanna do something fancy, the whole point of stimulus is it doesn't have weird fancy stuff. It's clean.
John Epperson [00:35:56]:
So so now that we've decided that Jake Ray is is dying and that hopefully our morning period is a little almost over.
Luke Stutters [00:36:04]:
It is dead. I I know it's dead. I'm just finding it hard to cope. Okay.
John Epperson [00:36:08]:
Fair. So, Dave, you earlier had me a little bit on this train, and I kinda wanted to go back to it or whatever talking about, like, changing multiple things. Because there's so many places where we can find ourselves tricked or or just, like, seduced into it. Right? Where it just seems like the right thing to do. So so my the thing that, like, I like to tell people because I it made total sense to me when I first heard it was, you know, Kent Beck's like, first, you make the change easy, then you go and make the easy change. Right? And the important thing about that, right, is there are steps here. Right? Like, I yeah. Dude, engineering is all about taking those, like, tiny steps and, like you said, testing.
John Epperson [00:36:52]:
Right? And we all know what happens when we change a bunch of things at once, and then we all do it. And and then we yeah. So we go down the the road of mistakes. Engineering is this it requires self discipline. And whenever we don't have self discipline or whenever we, you know, just break it because we're human beings. Bad stuff happens.
Jesse Spevak [00:37:12]:
I almost feel like it's I go into, like, meditative state almost. It's it's it's kinda like the flow state kind of. And there are times where it's like, I pick up a story and say, okay. Yeah. This is easy. I can just kind of, like, half think about it and do it. And when that doesn't work, which it almost never does, I have to, like, get into this, like, very focused, I'm making this one change verifying state. It's almost like like the cartoon Avatar going into your your Avatar state, if you if you guys know that amazing cartoon.
John Epperson [00:37:45]:
Yes. I'm familiar with it.
Jesse Spevak [00:37:47]:
Luke, I highly recommend it.
Luke Stutters [00:37:49]:
What's it called?
Jesse Spevak [00:37:50]:
Avatar and the Last Airbender. I think it came out, like, 10 years ago, 15 years ago.
Luke Stutters [00:37:53]:
Oh, it's the guy with the face the face tattoo.
Jesse Spevak [00:37:56]:
He's got an arrow
John Epperson [00:37:57]:
on it. Yeah. Exactly. You know exactly what
Luke Stutters [00:38:00]:
Right. I will check it out.
John Epperson [00:38:02]:
I've seen that art too. He looks like he's really angry. Exactly. He's he's
Luke Stutters [00:38:07]:
he's got he's got no hair. Right?
John Epperson [00:38:09]:
Yeah. Bald head.
Luke Stutters [00:38:10]:
Yep. That's the guy. Yeah. Alright. Alright. I'm I'm a huge anime watch too. That's a that's a dangerous gap in my knowledge.
John Epperson [00:38:18]:
Yes. So, anyway, it's like you go into the state, just oh, just yet you were you were saying you go into the state when when the the first
Jesse Spevak [00:38:27]:
And it and unfortunately for me, maybe it's because of my age or whatever, but it getting into that really focused state actually, like, costs something, right? It takes, like, a little bit out. You can't maintain that state forever. There's, like, a man pool almost of how long you how long for me I can be in that extreme focused state. And I think that when there are times where I look at a problem, I'm like, do I like, I I almost, like, ask myself, do I need to be in this highly focused state to get this thing accomplished? And more often than not, the answer is yes, but my initial answer is no. And And so I end up wasting time by not doing things as systematically as they need to get done. Am I am I alone on this? Nobody else goes into the guitar state when they when they code?
John Epperson [00:39:13]:
I, I mean go ahead.
Luke Stutters [00:39:15]:
I find that, yeah, people talk about this flow state, and I was I was, I was not a believer for a while until I got something really hard to work on, you know. And it's those problems where you're the kind of limit of what you can do when some you're thinking, can I actually write this? That is when I find you get these kind of periods of intense concentration. And obviously you don't want to be, you know, the edge of your ability all the time. You want to kind of line up the nice easy jobs or, things tend to go disastrously wrong. But one thing I have found is that, you know, when you're doing these really hard problems, and you're like, can can we do it? You know, can is it possible? You always have to come back and redo it. Once you prove that it's possible, then you have to hit it again, and then you come up the 1. So all of the stuff I've done in a kind of state intense concentration tends to get thrown away, but it moves you forward down the old down the road down the road.
John Epperson [00:40:16]:
I always call those my naive implementations. And and it's totally cool to write really terrible code during naive implementation. That's not the point of it. The point is to get from having nothing to having a working thing. The important thing is that after your naive implementation, you decide, you know, decide whether it's worth refactoring, when you're gonna refactor, all that kind of stuff.
Jesse Spevak [00:40:41]:
Yeah. I I love that that idea of a naive implementation, and I think the Akka microservice was our naive implementation. We didn't consider the scale at all. We didn't consider the makeup of the team changing. And as a result, when we went back to fix the naive implementation, right, when we wanted to get rid of the double loops or whatever, we ended up moving to Java from so from Kotlin to Java, and from Akka to Camel. And the reason that those tools made sense is because they were very common in our company. And what we didn't have to reinvent the wheel. We weren't seeing errors for the first time in the entire company.
Jesse Spevak [00:41:24]:
There was sort of a knowledge base to draw from. So the current state is that we have Java, Camel, Spring, Microservice, talking to the rails monolith. It's very stable. It's performing super well now. And, yeah, I mean, getting I guess, a question I have is, like, how can we speed up the process of getting from our naive solution to the the more learned solution?
John Epperson [00:41:51]:
I think, in my opinion, if I had an answer for that, I would be I I would be a super wealthy person. Because in my experience, usually what happens is when you get to the point that you're calling something a naive solution, right, it's usually because you recognize that there's a problem with it. Right? And one of the things that's going on, I feel like, is that you're sort of you're you're kinda burning yourself out on the problem at that point. When you when you sort of have this realization, you're also at the point that you're kinda burning yourself out in this problem. So if you as a as an organization don't have the resources to sort of, like, swap out people, you know, bring somebody in that's fresh, things like that. Right? Or or kind of go back to a huddle and, you know, somehow become reenergized. Like, all the things that we can do are are almost all, like, social interactions, not engineering interactions. Right? And so if if you have a culture where you can kind of deal with that, I feel like people can kind of turn around and refactor and make reasonable choices or whatever.
John Epperson [00:42:59]:
But if you don't, like, I feel like it's really hard. It's really hard for you as somebody who just burned yourself out, pushing something across the finish line that was really hard to then turn back around and be like, I'm throwing you out, and I'm gonna redo this really hard problem again. Right? Like, that's just a really hard thing to do, I think.
Jesse Spevak [00:43:17]:
It's interesting that you say that because at the point that we made that decision, the problem didn't seem hard anymore. At the point that we were like, you know, we can just do this job in Campbell. We had wrapped our heads around the problem enough that we really felt like it wasn't hard anymore. And maybe maybe that's what it takes.
John Epperson [00:43:35]:
You also said a really important word there, which is we. Right? So one of the things that you did as a company to recover from this problem is you went back to this huddle, and you said, hey. I made some mistakes. And and then your team said, that's okay, Jesse. We, as a team, are taking ownership for this. Right? Like and we, as a team, are gonna fix this. Right? Like, that that's what a lot of that communicates right here. And when you do that, like, that's one of those reenergizing moments or, you know, things.
John Epperson [00:44:07]:
And then those decisions become a lot easier in in my experience.
Luke Stutters [00:44:11]:
It breaks my heart every time every every every time I have a have a git commit with a kind of more lines removed and added. Oh, man.
Jesse Spevak [00:44:20]:
Those are the proudest days.
Luke Stutters [00:44:22]:
Right? I know some people like to take the code out, and they're like, oh, I go knocked out, you know, loads of that repeated code. I just think, why couldn't I get it right the first time?
Jesse Spevak [00:44:31]:
That's interesting. The person who taught me how to code, one of the first pieces of advice that he gave me was that you should write a lot of code and throw a lot of code out, and that's how you'll get better at coding.
Luke Stutters [00:44:45]:
That is very, very sensible advice. Very sensible advice. And, by that standard, I've got amazing at coding over the years. Do I, the context is a big mistake. Now just to be clear, this big mistake has had a happy ending. So this isn't this isn't the kind of big mistake I got fired. This is big mistake, we pulled through it, and it all worked out. Am I right?
Jesse Spevak [00:45:06]:
Thankfully. Yeah. So I think so we did a couple of things, I think, that helped us. So the first is, as John said, we were open about these things. We didn't try to hide that things weren't going as well as we had wanted them to. And I think that Ibotta has a pretty strong culture in the sense that we're not trying to throw people under the bus in engineering. If if something crashes, it's not who's gonna get fired? It's like, okay, how do we learn from this? This is a mistake that cost us some money. How do we make sure that we that money is actually teaching us something? So that was part of it.
Jesse Spevak [00:45:42]:
And then I think also we did a good job of communicating, like, to external stakeholders. We communicated to the the finance team, who are kinda one of the main, main, consumers of the data that we were producing. And, you know, really went through in detail, here's where we're at, here's the timeline, like, updating them and checking in with them all the time. And just keeping expectations in line, I think, really helped us out. So even though we we delivered a little bit later than I think we thought we would at the onset of the project, because we were able to communicate that, we, we were not fired. And, yeah, I mean, not only that, we we've hired more people. We're still hiring. And, if you if you're thinking about getting into the mobile coupon space, there's a ton of really cool problems, even if you're not passionate about mobile coupons.
Jesse Spevak [00:46:35]:
And you might get to talk to me in the interview process, which will be fun.
Luke Stutters [00:46:38]:
Speaking of interviews, I see a smooth transition there. Speaking of interviews, I understand that you have a project for which you call MENA SWAN Interviewing. That is I have no idea what Mina Swan sounds, but I see a lot of people saying it in the Ruby community, but I just just not there's one of these weird in jokes I think they have in the Ruby community. It's nice and
David Kimura [00:47:00]:
it's nice and sweet.
Luke Stutters [00:47:02]:
I know. I know. I know.
David Kimura [00:47:05]:
Try humor, Luke. Never pick up on your sarcasm.
John Epperson [00:47:09]:
Good lord. I'm just over here, like like, dying every
Luke Stutters [00:47:14]:
and that's the, what was it, Dave? Sorry. It
David Kimura [00:47:16]:
was, seriously?
Luke Stutters [00:47:18]:
Yeah. Go on.
David Kimura [00:47:19]:
Matt says nice, and so are we.
Luke Stutters [00:47:21]:
Yeah. What is the best thing about reviews community? It's a it's a really, it's a really great, language, and it's a really strong community, really great events, even though, obviously, the events this year have been a bit difficult. So how are you carrying that culture, that tradition over into the interviewing process? What's your gambit?
Jesse Spevak [00:47:43]:
Yeah. So maybe all maybe everybody has, experience or a lot of folks haven't ex have an experience in, let's just say, a not meet us one interview. An interview that maybe is a little more hostile than we'd like. And I think I think a lot of us have experienced kind of broken interviews where it feels more like the person on the other side of the table is trying to prove how much smarter they are, or how much better they are at coding than I am. And that's not nice. Another thing that's not nice is asking someone to do an inordinate amount of work outside of work, that's not paid in the form of like a take home project. So I've done take home projects that have taken me entire weekends, multiple days, and that's uncompensated work. And that can bias your process against people who have outside of work commitments, like families and just, you know, who don't wanna be working all the time.
Jesse Spevak [00:48:44]:
So I thought it would make sense to kind of take the best part of the Ruby community, this idea that Matt is nice, and so we are nice, and apply it to interviewing. Let's like, actually be nice to the people that we potentially could be working with. And, you know, the pandemic has been terrible in so many ways, but it did offer us this opportunity to kind of dramatically rethink what our interview was gonna look like. Because we're not coming into the office, everything has to be remote. And basically, our HR team and our leadership are like, how how can we do this? We've only been accustomed to bringing people in, asking them super tricky things that they have to whiteboard. How can we, how can we translate this to a remote interview? And this is what I proposed, and this is like what we landed on, which is an interview not meant to trick the interviewee. It's an interview meant to simulate what the first couple of days of work is gonna look like. And it's supposed to give, give us as an organization a sense of how much we would enjoy this person as a colleague, how successful they'll be.
Jesse Spevak [00:49:53]:
And the message that we're always trying to send is not, hey, I'm so much smarter than you because I understand recursion or because I understand how to do whatever this type of model, this type of data structure. The message is, you're gonna be successful on our team, and you're gonna like working with us, and we're gonna like working with you. So the way that we do that is basically by giving the person a sample of code from our domain. And it's highly simplified, and it's not and we ask them to just read the code and we say, what is this doing? What do you like about this code? What do you not like about this code? And it's not a bug finding adventure. We're not asking them to find where where an error is gonna be secretly raised or why a test is failing. That's that's kind of beyond. Like, you can't really do that in a in a 20 or 30 minute conversation. We wanna hear how this person would approach an alien code base, which is what their first task is gonna be on the job.
Jesse Spevak [00:50:49]:
Then, we present them with some data, similar, you know, some kind some award events from our system, and we ask them to manipulate that data in, with a pretty simple algorithm. We even tell them what the algorithm is, and we ask them to code it. And we say specifically, we don't care about the answer here. The answer is not interesting. We just told you what the answer is. We actually wanna see what it looks like when you code. Now, what what's your approach? Are you systematic? Right? Are you making guesses about what should happen and checking yourself? Those are the things that we're looking for. We're not looking for some for to see, you know, do you know this random algorithm from your, from your computer science education that you'll never use as a as a web developer at Alana? So those are the those are the big pieces.
Jesse Spevak [00:51:35]:
And I'm stoked because, I got invited to talk at at RubyConf, which is coming up in November, where I'm gonna be kinda outlining this. You all got a a preview of the content there, Exclusive to Ruby Roads.
John Epperson [00:51:47]:
Awesome. I'm definitely curious as you guys implement this, like, when you kind of come back and say, well, this works and this didn't work. Or or this is, like, this is how we sort of had to come up with a way to, you know, because this is a subjective thing. This is this is how we came up with a way to to make this more objective than than it originally was, you know, and and that helped us. Like, definitely interested in seeing this. I I mean, all of your all of the pain points that you're talking about. Right? All this homework that we have, all these, you know, coding challenges that we do. I mean, we we're very I feel like as a community, we we talk about how aware we are that they don't work, but we don't have alternatives yet.
John Epperson [00:52:30]:
And so people continue to use them regardless because they're like, well, I don't know what to do. And people who are lost, you know, just run back to where they came from a lot of times. Right? It's like the squirrel. It's like in the middle of the street and the car is coming. It's like, shoot. I gotta go all the way back. So I I kinda feel like we do a lot of that still. Yeah.
John Epperson [00:52:48]:
There have been there have been various things, like, talked about. I remember I think I've heard various people from thought on various podcasts, like, talk about the fact that, like, they they have you come and spend, like, I don't remember if it was, like, a day or whatever or half day, but, like, a lot of time with them pairing. Right? I know that boot camp does that as well or base camp. Sorry. Wow. Whatever. Base camp does that as well or something similar. They'll they'll actually, I remember heard they had an article about they did some homework or something recently.
John Epperson [00:53:16]:
Whatever. They they did they did
Jesse Spevak [00:53:18]:
a case of the homework, though.
John Epperson [00:53:19]:
Yeah. That's okay. Yeah. I mean, there's there's people that are, like, exploring around these edges, but we don't really have, like, a good way forward, I feel like, yet still. So I'm super interested to see how this turns out.
Jesse Spevak [00:53:31]:
Yeah. I mean, so far, you know, obviously hiring has has been slower than than typical for our for for Ibotta. But we are still hiring. And so far, the feedback from candidates has been really interesting because, you know, we'll get unsolicited emails about how much people like the process and how they hope that this is the direction that the industry goes in. A lot of folks who are interviewing with us are interviewing at other places as well. And I see this as like a competitive, a competitive advantage for us. Right? Does this buy us like $10,000 in salary or $5,000 in salary or something like that? Like, does this make our company more you know, once you get to a certain point, there's like diminishing returns on all these different levers that a company can offer to a developer. And I think that this is maybe an unexplored lever or a lever where there's a ton of room to gain.
Jesse Spevak [00:54:22]:
And when someone goes through our process and comes out feeling like, hey, I'm gonna, I'm gonna kick butt there. The people there are super nice. They didn't make me feel like an idiot. And they contrast that with other interview experiences they've recently had. Hopefully, that that that will play in our favor.
Luke Stutters [00:54:37]:
That is certainly not the current approach to coding interviews. I've been hearing recently more and more people talking about make how to make coding interviews harder. If you go on something like Hacker News, you know, they discuss which framework they need to insert under the fingernails of potential applicants. And I hear if you go to Facebook now and apply for a job, then they, you have to bring along your PhD in computer science. They, then burn it in front of you, mix it into old coffee and make you drink it as part of the interview process. So it's really, I think it's really something that the, tech community needs. We have been doing a few episodes about trying to make community more inclusive, trying to bring in people, trying to keep people in. And that sounds like a really great idea for talk, you know, why don't we be be, nice of people? Try and get the most out of them during interviews instead of subjecting them to torture.
Jesse Spevak [00:55:35]:
I love that image of shoving a framework under someone's, fingernails loop. But the the thing we the thing that I noticed, and I've done a lot of interviews that I bought. I've done, I would say, easily 200 interviews over the past 2 years. We do ton of firing, so it's it's crazy. And the thing that I noticed is that when we were asking people to whiteboard, right, to answer these kinda tougher algorithm algorithm questions, I didn't I didn't always see a one to one correlation between the people who crush that kind of question and the people who I really enjoy working with in either direction, right? There are false positives and false negatives. And I just found there to be way too much noise in that type of diagnostic tool for me to really make a strong decision that I felt confident with that proved out over time. And I'm hoping that this is a remedy for that.
John Epperson [00:56:33]:
Yeah. I mean, you always have to check to make sure that whatever test you're giving. Right? Like, is the you have to decide what you're testing for. And you have to say, is the question that I'm asking actually testing for the thing that I'm testing for? And too often, we jump to this conclusion that, oh, yeah. This sort of this sort of is related. Therefore, it'll let me know. When really you're just you're making a subjective judgment call again and calling it objective. So yep.
John Epperson [00:57:01]:
It is it is a thing that's very painful, and maybe maybe we should we should focus a little bit on this in the future and and have a more in-depth chat about this. I feel like that would be an interesting subject.
Jesse Spevak [00:57:14]:
Yeah. There was a last year. I believe the guy's name is Eric. I'll throw it in the in the chat in a second. He's from Test Double, And he talked about, Test Double is like a rail or a roomie consultancy, based out of Columbus, I believe. And he talked about their hiring process and more in the sense of our goal is to build a process that lets candidates show off their strengths and as opposed to helping us find their weaknesses. And that talk really informed our process at Ibotta. We're trying to find people's strengths and trying to figure out where they're gonna be able to make the most impact in our company.
John Epperson [00:57:52]:
Yeah. I have I have gone for the past, like, I don't know, 4 or 5 Rails comps. I've gone to basically every single talk that was sort of, like, related to the subject. I I do I care about it quite a bit, and I I just kinda hope that we get to a better place. It's yeah. Like I said earlier, my my feelings on this, I think, are are basically what I said earlier that I think that a lot of people are thinking about this, and I just don't think there's a clear answer yet, but I'm super interested at, like, everything that people are trying. Experiment. Hey, Sean.
John Epperson [00:58:23]:
It's gonna that's how I think we're gonna get a little better.
David Kimura [00:58:26]:
Yeah. Well, hey. Looks like we are coming up on the hour, so we need to move things along. Jesse, if people want to get in touch with you, where should they go online and look?
Jesse Spevak [00:58:36]:
I would say, you can find me on Twitter. I I tweet from planet efficacy. So if you can if you can spell that, you can find me. And, you'll find a lot of interesting, movie content, political outreach, and, Vermont progressive rock on that Twitter stream.
David Kimura [00:58:56]:
Awesome. Well, let's go ahead and move over into some picks. Luke, you wanna kick us off?
Luke Stutters [00:59:02]:
I would. My first pick is a version of Windows 10.
John Epperson [00:59:09]:
A bit
Luke Stutters [00:59:09]:
of a strange thing to pick on a, but this is called Windows 10 ameliorated edition. I found it featured on the popular YouTube channel not long ago. And this is a version of Windows 10 porta spyware taken out, which also incredibly boosts the responsiveness and performance, because it's not doing any async network calls back home in the background, if you know what I mean. This is the first operating system I've ever found where to download it, I had to get a bit torrent link from a Telegram channel. So if you want some excitement in your operating systems, check out Windows 10 ameliorated edition. Never seen anything like it.
John Epperson [00:59:59]:
Just just wanted to confirm something here really quick, Luke. Is this a legal version of Windows 10?
Luke Stutters [01:00:05]:
It's got quite a big section on legality on the website. Okay. The website does make it clear that you do need to have a Windows 10 license in order to legally use the Windows 10 Ampereaser edition. But I mean, come on. Hit torrent link through a Telegram channel? Wow. That's, that's quite a anyway, it's it's quite trippy. It's a cool little project. There's lots in the FAQ about their rationale behind it, and it does fly when it runs.
Luke Stutters [01:00:36]:
It's really quick.
David Kimura [01:00:38]:
I heard there were some issues with it with not being able to do certain things because it can't call home to Microsoft. Even some, like, simple rudimentary things that you would think that would be possible, but just kinda breaks it.
Luke Stutters [01:00:53]:
Yeah. I mean, you could call it Windows 10 Lobotomized. I mean, there's barely anything there at all. But it's it's oh, it's a lot of fun.
John Epperson [01:01:01]:
It flies doing nothing.
David Kimura [01:01:03]:
Awesome. Well, John, you wanna do some pics?
John Epperson [01:01:06]:
Yeah. So I have a different pick this week too. You may by the time this this comes out, I have no idea if people are even gonna still be playing this game, but I have been playing and streaming and and absolutely having a blast playing, I guess, that's redundant among us. So if you're not familiar with it if you're familiar with, like, Werewolf or Mafia or kind of games in that nature where you there's another game that I people have, like, compared this to or whatever. Where you, like, sort of have, like, like, townsfolk, and then you have, like, people that are pretending to be townsfolk that are, like, trying to kill everybody off. And your goal is the townsfolk or to try and find the, one in this game, the imposters among you. So this is, like, set in space or whatever. But, yeah, it's an absolute blast.
John Epperson [01:01:56]:
I I really am not sure what to say about it other than it's, like, kind of, like, a great social game. Yeah. It's it's pretty awesome. So I I highly recommend checking it out. I definitely have been doing a lot of streaming of it lately. So yeah.
David Kimura [01:02:10]:
I'll jump in with a few picks. First pick, bamboo flooring. So I love bamboo, and instead of the crummy old chair mat, the little plastic thin plastic that I was using for a long time, I finally went to Home Depot and got some bamboo flooring. And I put glued that to a piece of plywood, and I now use that as my floor mat on my carpeted floor. So my chair chair can just slide around on it. It's really, really cool. And it's been a life changing event here, so it's pretty awesome. And the second thing that I pick is the Elgato key light.
David Kimura [01:02:50]:
So I got those for my streaming setup, and they make a huge difference. So you won't be able to tell on the podcast, but this is with my lights on, and this is with the lights off. Makes a really big difference in quality. So having proper lighting on doing any kind of video work is an absolute must.
Luke Stutters [01:03:07]:
Can you can you do that again, Dave? Just so everyone can see you on the podcast. Yeah. I gotta say that is quite striking. Is that the same company that did the shortcut bar?
David Kimura [01:03:17]:
Yes. The Stream Deck. Mhmm.
Luke Stutters [01:03:19]:
Yeah. That's a pretty cool thing as well.
David Kimura [01:03:21]:
Big Elgato fan. And, Jesse, do you wanna jump in with some picks?
Jesse Spevak [01:03:25]:
Yeah. I've got a couple of picks, different categories. So the first is a talk that I watched recently that maybe you all have seen already, but I recommend revisiting it. So Sandy Metz's RailsConf 2019 keynote, which is called Lucky You. And I think it's especially timely at the, as we approach election season and thinking about equality and issues of inequality in our country. So I highly recommend that talk, it's amazing as all Sandy Met's talks tend to be. Then I have, when the pandemic started, we started working remote. I made a small investment in my work from home setup that I highly, highly, highly recommend.
Jesse Spevak [01:04:08]:
I went out and I got an ErgoDox split keyboard, having some wrist pain on the keyboard I was using, so I know keyboards can be a whole other podcast, but check out ErgoDox. They make a really, really, really good product that's worth the price, and I've been a huge fan of it since getting it. And then my final pick, there's a book that just came out. It's a guilty pleasure. It's called The Trouble with Peace by Joe Abercrombie, a British gentleman, And it is in the grim dark genre of fantasy literature, and I highly recommend it.
David Kimura [01:04:43]:
Awesome. Well, thank you for those picks, and just be sure to post them into our chat section here so we can include them on the show notes. Well, Jesse, thank you for coming on today. It was a lot of fun. I love these kind of talks where we can just humble ourselves and talk about past mistakes and, you know, what we learned from them. So thanks again.
Jesse Spevak [01:05:03]:
This was awesome, guys.
Luke Stutters [01:05:04]:
Really interesting.
Jesse Spevak [01:05:05]:
I listened I long time listener, first time guest, and this was this was awesome. I appreciate it.
John Epperson [01:05:11]:
Yeah. Thank you for coming. Right.
David Kimura [01:05:12]:
Well, that's all for this episode, everyone. Thanks for listening.
John Epperson [01:05:15]:
Take care, everybody.
Hey, everyone. Welcome to another episode of Ruby Robes. I'm David Kamira. And today on our panel, we have Luke Suttters
Luke Stutters [00:00:11]:
Hi.
David Kimura [00:00:12]:
Stuttters. And we have John Epperson.
John Epperson [00:00:14]:
Hello, everybody.
David Kimura [00:00:15]:
And today, we have a special guest, Jesse Spivak.
Jesse Spevak [00:00:18]:
Great to be here.
David Kimura [00:00:19]:
So, Jesse, would you mind explaining just a bit about who you are, some of the things that you're doing, who you work for, and why you're famous, and all that good stuff?
Jesse Spevak [00:00:27]:
Yeah. Absolutely. So my name is Justin Spudak. I'm a senior engineer at a company called Ibotta, which is a cashback for shopping app based in Denver, Colorado. I've been working there for about three and a half years. We are doing some hiring, so check out our careers page. I guess I'm I'm famous as it were because I gave a talk at the first remote Rails comp this past May,
Luke Stutters [00:00:52]:
and
Jesse Spevak [00:00:53]:
I talked about kind of how crummy of a developer I am.
David Kimura [00:00:57]:
Yeah. I think we can all relate to that on a daily basis sometimes. So would you mind giving a bit of a, you know, highlight talk about what you covered at the conference and stuff so we can just kinda pick it up from there? We'll link to the in the show notes, a link to the conference, but just for those who maybe didn't see it.
Jesse Spevak [00:01:17]:
Sure. And and there there's no substitute for actually watching this fantastic talk. But in in more more seriously, the talk is really about my experience as a tech lead at Ibotta, working on a pretty critical project over the course of about 6, 6 months or so. And over that time, I made 4 very big mistakes that put the project in jeopardy. And hopefully they're mistakes that I will learn from and not make again as I continue to lead projects that I brought in, in the future. And my hope is that by sort of articulating these mistakes and what I learned from them, other folks can benefit. And so the 4 mistakes that I made are first, we picked the wrong technology. You can get more into that.
Jesse Spevak [00:02:09]:
We also, as a team, we siloed work. So work was divided up in not the best way. We fell into the premature optimization trap. And then, maybe worst of all, we made way too many changes at one time. So I can I can go into detail on any of those?
David Kimura [00:02:27]:
Yeah. And I think it's fair to say that I, our listeners, other members on the panel have been there before on
Luke Stutters [00:02:35]:
Yeah. I've I've worked with loads of people who've made those mistakes. Obviously, I've never made them myself, but I've oh, my words.
John Epperson [00:02:41]:
Oh, my words. Luke Luke is the perfect developer. Let's be fair. No. I I was actually going to jump in and say that, like, I feel like I I don't feel like I'm alone, but usually when you make a mistake, it cascades into more. For whatever reason, I generally feel like you either you either kind of a rolling along and you feel pretty good about stuff, or you you're just, like, in a bottomless pit. And there's very little in between time. You just suddenly notice that you're you're at the bottom.
John Epperson [00:03:11]:
So yeah. Just saying.
Jesse Spevak [00:03:13]:
And absolutely. And I just wanna kinda call out that it's there's, like, a certain amount of privilege that comes with being able to talk about our mistakes. Right? I'm not worried that my boss is gonna fire me. And I'm also not worried that folks won't take me seriously for giving this talk. If anything, it probably improves my reputation as evidenced by getting to talk to 3 of you gentlemen. So I just wanna call that out because I I I think it's important.
John Epperson [00:03:40]:
I I think that oh, okay. So so speaking of that, I think there's a give and take there too. So in I think this is one of this will get into a thing that I care a lot about this particular subject. Because for most of my shoot. I still feel it today or whatever. It doesn't really matter. I still always feel this pressure to not let people know about the mistakes that I made. Right? Because letting people know about the mistakes that I made makes me more vulnerable to them not wanting me to work for them.
John Epperson [00:04:08]:
Right? Like, fire me. Whatever whatever the bad thing is that it's a boogeyman. It's not real. It's just a pressure. Right? And so I guess what I'm trying to get at is, like, I kinda feel like one of the things that I always care a lot about with mistakes is telling people about your mistakes. I actually have discovered in reality actually makes people respect you more, but it doesn't change the fact that it's, like, really hard, and I still feel that pressure to this day.
Jesse Spevak [00:04:39]:
Yeah. John, yeah. I I agree with what you're saying. I think that for a lot of us, talking about mistakes openly and honestly, and what we learn from them actually builds our credibility. But that's not always the case depending on sort of how you present, what you look like. I think that, you know, you guys recently did an episode, I think, where you you talked about issues issues of hiring and getting equality in a workplace, and I think that that that plays in here for sure.
David Kimura [00:05:08]:
And I would go as far to say that it depends on what the mistake is, what kind of mistake it is. If it is just utter and complete negligence, then those aren't really the kind of mistakes I would really wanna be forthcoming and outright about. You know? But, like, if I went in and always had a VPN tunnel into my production environment, then we got malware in our local work environment, and that transferred over to production and encrypted all of our production data. I don't know if I would really say, like, oh, yeah. You know, that was a silly mistake of mine. No. It's like, okay. Not only have our customers affected, but now, you know, my job's in jeopardy because I decided to always take shortcuts.
David Kimura [00:05:56]:
But something like and what I'm really interested in is your technology, framework or the you used the wrong technology. Can you explain a bit more about the scenario?
Jesse Spevak [00:06:08]:
Sure. Absolutely. So this the high level, we at Ibotta, we have just a wonderful, majestic monolith, Rails application. Actually, originally, the application we've written in Scala. And then after about a month of that, they switched it over and rebuilt the whole thing in Rails, and it's been Rails ever since. So it's going on close almost 10 years at this point. So it's a large application, it serves millions of users. Very cool.
Jesse Spevak [00:06:34]:
And about 3 years ago when I joined, we started thinking we started growing the team and thinking about how we could, like many people have, pull pieces out of the monolith into microservices. So this project in particular was about taking a piece of billing logic from the system, from the monolith, and pulling it out into a microservice. The hope was to make it better encapsulated, easier to iterate on, not depend you know, isolate dependencies, every reason that you think to build a microservice. So we chose actually, before I say the technology, before I get trolled by all the lovers of this technology, I'm gonna preface this by saying, I don't think that this technology is wrong and I don't think it's bad in and of itself. I just think it was not the right technology for the problem and the team.
Luke Stutters [00:07:26]:
Hold on a second. Hold on a second, Jesse, because the slide I'm looking at says big mistakes. It doesn't say small mistakes. It says big mistakes. So they should keep this in mind before we reveal what this technology is, what a big mistake was.
Jesse Spevak [00:07:41]:
Absolutely. This whoever whoever uses this technology here is is definitely making a big mistake. So I know, spoiler, we weren't building like us, a side rails app microservice, which would probably would not have been as big of a mistake. But the issue really was that our team, the the small team of developers, and then the larger team of engineers in the company, really did not have a ton of experience with the framework that we chose. And as a result, we ended up having to do a lot of plumbing, and reinventing the wheel, and just not benefiting from the institutional experience that exists at Ibotta. And unfortunately, right, this, this could work if you're doing kind of like a proof of concept, like, let's show what this technology can do. Let's pick a pretty isolated use case. But the billing logic that we were pulling out about Monolith was basically like do or die, right? If it did not work, it costs 1,000,000 of dollars to fix, or, you know, it ends up costing the company 1,000,000 of dollars.
Jesse Spevak [00:08:46]:
So it really, we were walking on a tight rope and there was no net underneath us. And unfortunately, we decided to, I guess, like walk on our hands instead of, go across normally. So the framework that we used is called ACCA. And I think for a team that knows Akka, this probably would have been really a perfect tool for the job. But our team and our company really did not have a ton of experience with Akka. And so unfortunately, we weren't able to sort of take advantage of it and use it in a way that sort of professional Akka developers likely can.
Luke Stutters [00:09:20]:
So this is kind of like a functional programming thing?
Jesse Spevak [00:09:23]:
Right. It deals with data streams and passing data along in a functional paradigm, and it's meant to accommodate high volume data across highly parallelized system. So, you know, at the time, we went there, well, I'll talk about why we went there in a second, but it, in retrospect, it was, it was something that could handle basically 10000x really what we needed in terms of what it was designed to handle. So just sort of on, on paper, it probably wasn't the best, the best move in that respect. But I could also talk about the team as well and why it wasn't a great a great fit.
John Epperson [00:10:02]:
Yeah. Can you can you talk for just a minute about, like, what was the problem that you were trying to solve? Why did you choose like, what did you need ACCA for?
Jesse Spevak [00:10:11]:
Sure. Yeah. Perfect. So the problem that we're trying to solve, and you have to know a little bit about the Ibotta app. So I assume all of you all of you guys have, downloaded it and are actively using it.
Luke Stutters [00:10:23]:
Absolutely. It's the best app I've ever used.
John Epperson [00:10:25]:
I've definitely looked at it and I was like, nope.
Jesse Spevak [00:10:30]:
That's funny. We did a search. Well, I'll get to that later. But basically, Ibotta is a way for you to get digital coupons. You can get brands put offers in the app, you click on the offer, you show us evidence that you bought the thing that is on offer and then Ibotta will pay you cash back. They'll put, send it to your PayPal account, give you a gift card to Amazon, whatever you want. So the problem that we're trying to solve is how do we make sure that the offers in the app don't exceed the budget that is allocated to them by the brands that put those offers in the app? And that sounds maybe like an easy problem, like there's an easy way to just say, okay, there's $500,000 of budget for Oreo coupons. Just divide $500,000 by how many, how much money we're giving out per coupon, and then you know.
Jesse Spevak [00:11:19]:
There's actually much, like, it's obviously much harder than that. And in order to preserve a good user experience, we need to make sure that we're not yanking content and surprising our users. Like you would be really upset if you went to the store specifically to buy Oreos to get a coupon, and then by the time you checked out, the coupon is no longer in your application. So we have to learn some predictive algorithms to basically guess when we're gonna run out of money and kind of slow the coupon, slow the velocity down as we approach that point.
Luke Stutters [00:11:52]:
Oh, dear. That's that's a that's a that's a dangerous recipe. I must add that I I don't think the bottle is available in the UK at the moment. Is it available in Canada?
Jesse Spevak [00:12:03]:
We are only in the United States right now. Sorry, Luke.
Luke Stutters [00:12:07]:
So That's why it's in my defense. This looks like the perfect storm for me because you've both got the kind of strong financial components. So if you get it wrong, you lose money. And if you get it long, then people will have wasted their time going to the shop, you know. In Britain, it's so small, I can walk to the shop, you know. Sometimes I just open the window and shout at them and tell them what I want. But, you know, I know when I was living in the States and people maybe drive for 2 hours to go to Walmart. So this is this is this is quite a quite a problem you've got there.
Jesse Spevak [00:12:41]:
Yeah. It was it's interesting that you say that because right when I I joined the company, I was kinda put on the team. I was actually giving, you know, my tech buddy, my mentor when I joined the company. This was his project. And as someone new in the company, it was very, it was really overwhelming problem space. Because basically these campaigns and application almost have like a physical momentum to them. So if you imagine trying to stop a moving train, you can't just hit the brakes and expect it to stop on a dime. You have to apply the brakes over some distance to slow the train down.
Jesse Spevak [00:13:17]:
And that's really how the content in the application is modeled. And it's I'm not a physics person, and so it's really confusing.
John Epperson [00:13:26]:
So alright. So this seems a slightly I mean, it's a it's a twist, right, on your classical inventory problem, right, is what it kinda sounds like. So the way that Aka came into this is is what you were taking, like, what all these requests from users and trying to decide whether or not you're gonna, I guess, give them the coupon or not or something?
Jesse Spevak [00:13:50]:
Yeah. That's that's that's sort of right. So we have an event based architecture where our system and for the for folks who aren't familiar with that, that means that basically your system publishes events, which is data that signify that something has happened of interest in the system. So maybe like, you know, shopping cart loaded is an event that you might have in like a typical inventory space or something like that. So we have events that we're interested in, like content awarded, which means that John went to the store, submitted a receipt through the app, and got cash back. So the content has been awarded. So we listen for those events in order to keep track in real time of how much budget is being used, and we basically track that over time to make a rough prediction about how fast things are moving. And so, Akka, so sorry, John, to get back to your original question, Akka is really good at streaming data, at streaming large amounts, high volume, data.
Jesse Spevak [00:14:54]:
And Ibotta will get on the order of several 100,000 of these content awarded events per day, which seems like a lot, but it's actually much lower than I think what Akka can kinda deal with or is designed to deal with out of the box.
John Epperson [00:15:09]:
Yeah. Did you so did you consider alternatives? Did you reject them for various reasons? And and I guess so, obviously, ACCA didn't work out for you, so you must have picked something else that you like better. And how did you arrive at that new thing, and what was that, if that makes some sense?
Jesse Spevak [00:15:28]:
Yeah. Perfect. So yeah. We we basically this this comes down to some some some team issues again and not not an issue with Okta. So the team issue was basically that at the start of this project, we scaled up our team. We're like, this is a lot of work. We need, we need to bring in some artillery. And we brought in a new developer from outside the company who is awesome.
Jesse Spevak [00:15:57]:
She's a rock star and had it was coming from the ad product space. And with dealing with volume, much volume of streaming data at scale much higher than what we needed or we're going to be dealing with in any near near future. And, you know, she was coming from, I believe in Akasha. And so she joins the team. We're excited to have her. We think she's a rock star. I mean, she is a rock star. And she's like, this is a perfect use case for Akka.
Jesse Spevak [00:16:29]:
We're like, okay, never heard of that, but, you know, we trust you. You crushed our interview, and we think you're amazing. So, yeah, that sounds pretty good. And again, this isn't a knock against Akka. I think this is just that. In our company, we have a ton of infrastructure set up to support the tools that our company has sort of blessed that that are kind of frequently in use. In fact, we have like a name for that, we call it the paved road, which is like, if you're an engineer, you have a lot of autonomy out of what you use, but if you stay on the paved road, it should be an easy path. And Akka, which is a JV, you know, which we use with Java or Scala typically, is not on Ibotta's paved road.
Jesse Spevak [00:17:15]:
So we had to kind of have a contentious meeting, a contentious conversation with the architects at Ibotta and say, No, we really believe in our developer. And we think that she knows what she's doing. And we're ready to sort of make our bed and lay in it. And they were like, Well, I know as long as you know that. So they they let us kinda they gave us just enough rope to I forgot how the rest of that goes. To hang ourselves with.
John Epperson [00:17:43]:
Okay. So so it sounds like you you had an advocate, and that's kinda how you landed on it. Where did you go after you decided that ACCA was the poor choice? Like, what did you end up with?
Jesse Spevak [00:17:53]:
Yeah. So this and this kind of gets into the next problem, which is this the siloed work. So we're starting to work on building this ACCA powered race car of a microservice to pull billing logic out of our rails monolith. And there's sort of two streams of work. There's the development of the Akka microservice, which is we were writing, using Kotlin, which is a JVM. It's kind of a nicer Java. It's what Android apps are written in. It's actually pretty nice.
Jesse Spevak [00:18:24]:
So we have that work going on, and then we have to integrate that microservice, sorry, into, into the Rails model. So I ended up taking on a lot of the integration work and the other developer took on most of the Akka and Kotlin work. So we have these sort of 2 very isolated pieces of work that are siloed. And then, something terrible happened. And I don't blame this person at all because I wouldn't want to be on my team anyway. She decided that she was much more interested in data engineering and she moved to a different team in the company. So, and that's actually, you know, a great thing about working at Ibotta. Like we've got tons of really cool problems, different spaces, and people are almost encouraged to move around to find things that they're, that they like and are good at.
Jesse Spevak [00:19:13]:
So, she made this move. And now, now the, the problem of using a framework that none of us had experience with, and that the general company didn't have a ton of experience with, really became self evident. Because now there was, there was still work left to do on the microservice. And I'm just, I'm just a rails developer. Like, I had to go into there and start writing Kotlin, start reading Yocket documentation to try to wrap my head around what it what this whole, you know, actor system meant. And, it it was, it was tough. And that's why I call it a big mistake. So that's sorry, Dave.
David Kimura [00:19:53]:
You know, from my experience too is that, yeah, Ruby is slow. Now, there's no getting around that when you compare to a compiled language. But holy crap, is it fast too. You know, I have a production application which processes over 500,000 active jobs every single day. And that it does it extremely quick. I don't need it to be faster. I mean, that's plenty fast. On the current setup that it's on, which is 2 cores and 4 gigs of RAM, and we have 2 servers dedicated to the background job.
David Kimura [00:20:29]:
So 2 VMs. It's able to handle that load, and we don't have to worry about it crashing or anything like that. So I mean, that's good enough for us. Now I imagine that it would be able to double that workload before we ever ran into any kind of performance issues where we needed to start scaling up.
Jesse Spevak [00:20:48]:
Absolutely, Dave. And we were originally in the monolith. The way the system worked was it ran on a background scheduled job using Rescue, and the scheduled job basically took roughly 45 minutes to run. So it was running like, kinda like a waterfall, like every 10 minutes, we'd start a new job that would take 45 minutes to run. And so going from that to basically completely real time is a huge improvement. And if, you know, real time for 500,000 events per day versus 5,000,000 events per day are different things, but if you're at real time, it's already this just enormous improvement over what we were working with, which is like this 45 minute loop. So at that point, and I guess I'll say another thing, before I get into, like, how we fixed, how we fixed my mistake. This is actually getting to what Dave said.
Jesse Spevak [00:21:41]:
This is a case of premature optimization. We sort of didn't do the back of the envelope math well enough, or didn't have a clear picture of like, okay, this is really like designed to handle like literally a 1000 times more traffic than our best day. So, you know, we, we didn't that was a, that was definitely a mistake. Like, we should have asked that question and realized, okay, maybe, maybe this is a little too much horsepower, we don't need this, it's too much trouble for what it's for what it's buying us. And on my end also, I was imagining, you know, getting all these events, we call it so the the microservice was producing, like, prediction events. Okay? So predicting every time a piece of content should come down and making an adjustment about when it thinks that that should happen. And so I'm imagining the monolith is listening to these prediction events, subscribe to an SNS topic or SQS queue. And I imagine, you know, thousands and thousands and thousands of events every second, which is way, way too much.
Jesse Spevak [00:22:41]:
And so I was thinking of like all these clever ways to do caching and like to try to figure out when can I drop events so I don't need to hit the database, like when can I, how can I come up with these clever ways to only make round trips to the database when I really need to? And that added time to the development, And it also added a ton of complexity. So that when our 2 systems, when we're like, okay, let's turn them on, let's see what happens, the first error that comes out, because of course there's gonna be an error when you, when you first try it out, That was really hard to debug. It was really hard to understand, like, is this a caching issue? Is this actually an issue with the data coming from microservice? Like, it was really hard to isolate that. And then that obviously moves into the 3rd problem, 3rd mistake was that we made too many changes at at one time. So all these were ways that I tried to shoot myself in my foot in my own foot while working on this very important project.
John Epperson [00:23:38]:
Yep. Makes sense.
Luke Stutters [00:23:40]:
I'm not sure if I understand it, but so you're saying there's kind of 2 separate things going on here. Firstly, you're moving to a completely new technology. So you're taking a large part of the app out. You're putting in a micro service. It's not very micro if you're doing, what, 700,000 things a day. That's not a micro service. That's, your your this is this is this is this is a macro micro or medcro. And the second thing you're doing is you're actually changing the whole interface.
Luke Stutters [00:24:12]:
So you're saying you're going from you you previously had a rescue batch job, and now you're saying, no. No. No. We're not gonna do batching. We're gonna do it all immediately.
Jesse Spevak [00:24:22]:
Yeah. Luke, that's a really astute observation. So we were changing structure and we were changing behavior at the same time, which is something since that point, I've really tried to avoid doing. Even at, like, the micro PR level. If I have a PR that I'm gonna push up, it's either going to add behavior or change behavior, or it's gonna change the structure of the code. And I'm gonna try to not do the 2 things at the same time. So in this case, we did the 2 things at the same time. And at multiple points, we should have known or we should, or in retrospect, we should have known to pause and verify, right? And if we couldn't verify, take that failure as feedback and iterate.
Jesse Spevak [00:25:02]:
And I think that that's really what really strong engineers, experienced engineers know to do, make one change at a time and verify. Like, I haven't it's I guess it's kinda like when I run rspec, right? Or rspec feels like super natural to me. So every time I hit my test suite, I'm like, okay, I expect this to happen. And like, sometimes I'm right, and then sometimes I'm wrong, and that's like really interesting too. But making that initial guess of what's gonna happen is super important. And I think we need to slow down and stop and say, okay, here's our guess as to what's gonna happen. Here's how we're gonna verify it. And if if we can't verify it, here's, like, the corrective action you can take.
John Epperson [00:25:42]:
Yeah. So I think you're you're hitting it, like, one thing that's, like, really important. Right? So one of the things I think that make that that really kind of makes you a strong engineer of sorts. Right? It's not necessarily, like, do you avoid mistakes? Because, shoot, for whatever reason, they keep happening to me. Right? It it's are are you good at cleaning up? Right? Are you good at figuring out that something's not right? Something smells funny. Whatever it is, can you go back and, you know, sweep up your mess? You know, either push it across the finish line because you're close enough or say, actually, this is the wrong path. We should actually back up, go to the last intersection, and go down the other way. So, I I mean, it's definitely yeah.
John Epperson [00:26:26]:
I I I mean, I can speak to experiences too where I was just like, man, I really, like, missed all the flags. And then I kept missing more flags and just kept going. And and why are we surprised that we're in a bad place? Because I, like, ignored everything. So I hear you, man. It and and kudos for for recognizing it at some point and and, and doing something about it.
Jesse Spevak [00:26:48]:
Yeah. So what do we do about it? We unfortunately, I well, not unfortunately. It was a good it was a good learning experience, but, obviously, my happy place is is deep in a in a legacy Rails application, finding all the pathways there, but I had to get really push myself, get out of my comfort zone. I had to pick up Kotlin. Luckily, Kotlin is like a super friendly language, I think, to get into, especially for folks who are coming from Ruby because I think that there's, like, kind of like an emphasis on syntax, on, like, syntactic sugar, on, like, making the code actually, like, readable and look nice, whereas that's not always the case with all JBM languages. So, in that case, in that sense, it was kind of friendly. Also at Ibotta, we were able to like organize kind of a learning group. So, there were lots of folks who were new to Ibotta or new to Kotlin who were kind of trying to solve these similar problems.
Jesse Spevak [00:27:42]:
So, we made a study group, we found cool online coursework, we held each other accountable for making sure that we were, making progress on those things. And I started writing Colin in the microservice to try to, like you said, push across the line. We need to get this across the line, and then we can kinda circle back and deal with some of the underlying problems. And that's important because, like, as engineers, like, at the end of the day, we're trying to deliver value to the businesses that we work for, and not just, like, try out new technology or, like, optimize what techno what's what technological solution we're we're we're implementing.
John Epperson [00:28:19]:
So to just to clarify what what you're saying, you kept you you kept ACCA. You just, you just pushed it across the finish line despite all your your problems.
Jesse Spevak [00:28:31]:
Yeah. So That's
John Epperson [00:28:32]:
how you resolved it. Okay.
Jesse Spevak [00:28:33]:
Exactly. So we have we have problems with it, and we decided, let's get this thing, like, we're close enough, even with the problems, even with our lack of understanding. We're close enough that a total rewrite right now would put us back a long way and and upset the business. So let's push it across the line, and then we can circle back and figure out what to do. So that was that delivering that value, even though it it kinda felt yucky, even though we knew that there were things that we're gonna need to change over the long term, bought us some time and bought us some credibility in the organization.
David Kimura [00:29:10]:
And I wanna circle back to the point you made about moving too quickly or too many changes happening at one time. And I think that's something that a lot of developers might start to experience soon with the whole removal of jQuery from Bootstrap 5.
Luke Stutters [00:29:28]:
So No. Not my jQuery. No. Sorry.
David Kimura [00:29:35]:
So the idea is that Bootstrap 5 no longer has the jQuery dependency. So you might plan to upgrade your rails application, the CSS framework from Bootstrap 4 to Bootstrap 5. And you might say, like, hey. Well, why we're making this change? Why don't we go ahead and rewrite a lot of our JavaScript that was also jQuery dependent? And so I would say, instead of doing that, do one thing. Either first, remove all your jQuery dependency within your application and just have jQuery be a dependency of Bootstrap. And then in another iteration, upgrade your Bootstrap version, removing then jQuery entirely, or do it vice versa, where you do your Bootstrap upgrade first, and then you do your jQuery removal from your application as a dependency. But trying to do both side by side, it's too big of a task, you know, for, you know, 1 person or one team to do right away. I would just handle one thing at a time.
David Kimura [00:30:38]:
And moving slower, you're gonna say, okay. What broke this? Was it the new Bootstrap framework that broke this, or was it our jQuery rewrite? So that way, you're gonna be able to identify a lot more problems quicker before they are reported to you by the customer.
Jesse Spevak [00:30:55]:
Absolutely. I think the the most experienced engineers, you know, one of the best engineers I've worked with, his name's Justin Hart. He was the, like, did the first commit on the on the monolith that I work on. And every time I work with him, that is always my experience. Like, he is, like, we're making this change. Here's what we expect, and we're gonna do it, and then we're gonna verify it, and then we're gonna move on. And it and I'm always like, oh, why don't we do this, this, and this? And he's like, no, no, no. Just this one thing.
Jesse Spevak [00:31:23]:
And it it's illuminating. Right? It it's illuminating. I think it's helped me as a developer in the projects that I'm I'm doing now realizing like, okay, this thing that I that I wanna do, it's actually like 4 different things, and I'm gonna do the first the first thing first, verify it, and then move on from there.
Luke Stutters [00:31:39]:
If you or your loved ones were affected by the removal of jQuery from Bootstrap, why not check out the excellent course from dev devchat.tv? You don't know JavaScript yet 30 day challenge available with the link in the episode shown us.
John Epperson [00:31:55]:
Were we scheduled for, advertisement there? That's just come out of nowhere.
Luke Stutters [00:32:01]:
Just trying to just trying to just trying to keep the ship afloat, man. Come on.
John Epperson [00:32:05]:
Hustle. I was got so this goes along for me with,
Luke Stutters [00:32:10]:
No. No. No. No. No. No. No. No.
Luke Stutters [00:32:11]:
No. No. No. We have to do jQuery now because this is really affecting me. I mean, this is this is terrible news. I I literally can't write JavaScript without putting a dollar sign in front of everything.
John Epperson [00:32:23]:
You can still have a dollar sign. It you just need to assign something to the dollar sign. That's all.
Luke Stutters [00:32:27]:
One of the reasons I started using Maybe not
John Epperson [00:32:30]:
use jQuery.
Luke Stutters [00:32:31]:
Was because it has a little dollar sign in front of a data structure, and that kind of really makes me feel at home. Plus, you know, it's it it feels like money. You know? When I type the dollar sign in, I feel like, yeah, this is gonna make me a millionaire.
John Epperson [00:32:43]:
Oh, man. I'm sorry, Luke.
David Kimura [00:32:44]:
So I guess while we're on JavaScript, I think another premature optimization or rather I like to call them premature deoptimization is creating a new Ruby on Rails application with React, the dash dash webpack equals React. Just thought I'd throw that in there for the obligatory bash on React.
John Epperson [00:33:03]:
I do like stimulus, and I I do feel like I do feel like we can we can let React go now.
Jesse Spevak [00:33:08]:
Yeah. I mean, I don't know. Have you all checked out the Basecamp's email, hey.com?
Luke Stutters [00:33:15]:
Mhmm. Oh, yeah. Yeah. What do you think?
Jesse Spevak [00:33:19]:
Yeah. So I've been I've been using it, and I obviously, I'm a Basecamp fanboy, but I think it's it's pretty amazing what what they're able to do with just HTML over the wire for the most part and not, not relying on some of the frameworks that have come out recently. Also, I think it's super interesting, like the 2020 root on Rails community survey. If you look, the one put out by, Planet Argon, if you look at, like, what JavaScript libraries people are using, so I was expecting that number 1 would be React, and maybe number 2 would be, I don't know, Vue or, like, Ember maybe. But number 1 on there is jQuery. And it's like we're all in this in this jQuery world, and it's not it's not bad. I love jQuery.
John Epperson [00:34:02]:
So so for all of you who along with Luke are mourning jQuery, I'm telling you, stimulus. Go check it out.
Luke Stutters [00:34:10]:
Try it. Understand it.
John Epperson [00:34:12]:
I don't have an what that's fair. I I I thought I found it pretty easy. I I thought it gave me everything that I felt like I was getting from jQuery before, which is a thing happened on the page. And because that thing I mean, Jesse talked about event based architecture here. Come on. This is exactly what JavaScript is. Right? And jQuery, especially. Right? Stimulus is exactly that.
John Epperson [00:34:36]:
Event happens, do something else because this thing happened. Right? Just saying if you love that event based architecture style, stimulus is that. It's just way prettier. And I found it a lot easier to use. And because I don't like dollar signs, I was happy about it.
David Kimura [00:34:54]:
Yeah. And, look, if you want a bunch of different stimulus JS tutorials, check out Drifter Ruby, man. I have a whole bunch on there.
Luke Stutters [00:35:03]:
It's a big plug episode. Everyone quickly plug their thing. Do you know what? Actually, Dave, I have already checked that out, and I still don't understand it. So I need to check it out again. It's it's definitely me. I tell you I tell you what the problem is. Right? I've been leaning on jQuery for so long, and I'm talking about 10 years. Right? And it's not just that.
Luke Stutters [00:35:26]:
I've got my whole arsenal of weird front end stuff that I can pull in. And replacing that big long list of handy widgets, I know will solve this problem, is the is the is what I'm lacking. So yeah. Basic GOM stuff fine, you know, ES 6 is all the way. But if I wanna do something weird, if I wanna do something fancy, the whole point of stimulus is it doesn't have weird fancy stuff. It's clean.
John Epperson [00:35:56]:
So so now that we've decided that Jake Ray is is dying and that hopefully our morning period is a little almost over.
Luke Stutters [00:36:04]:
It is dead. I I know it's dead. I'm just finding it hard to cope. Okay.
John Epperson [00:36:08]:
Fair. So, Dave, you earlier had me a little bit on this train, and I kinda wanted to go back to it or whatever talking about, like, changing multiple things. Because there's so many places where we can find ourselves tricked or or just, like, seduced into it. Right? Where it just seems like the right thing to do. So so my the thing that, like, I like to tell people because I it made total sense to me when I first heard it was, you know, Kent Beck's like, first, you make the change easy, then you go and make the easy change. Right? And the important thing about that, right, is there are steps here. Right? Like, I yeah. Dude, engineering is all about taking those, like, tiny steps and, like you said, testing.
John Epperson [00:36:52]:
Right? And we all know what happens when we change a bunch of things at once, and then we all do it. And and then we yeah. So we go down the the road of mistakes. Engineering is this it requires self discipline. And whenever we don't have self discipline or whenever we, you know, just break it because we're human beings. Bad stuff happens.
Jesse Spevak [00:37:12]:
I almost feel like it's I go into, like, meditative state almost. It's it's it's kinda like the flow state kind of. And there are times where it's like, I pick up a story and say, okay. Yeah. This is easy. I can just kind of, like, half think about it and do it. And when that doesn't work, which it almost never does, I have to, like, get into this, like, very focused, I'm making this one change verifying state. It's almost like like the cartoon Avatar going into your your Avatar state, if you if you guys know that amazing cartoon.
John Epperson [00:37:45]:
Yes. I'm familiar with it.
Jesse Spevak [00:37:47]:
Luke, I highly recommend it.
Luke Stutters [00:37:49]:
What's it called?
Jesse Spevak [00:37:50]:
Avatar and the Last Airbender. I think it came out, like, 10 years ago, 15 years ago.
Luke Stutters [00:37:53]:
Oh, it's the guy with the face the face tattoo.
Jesse Spevak [00:37:56]:
He's got an arrow
John Epperson [00:37:57]:
on it. Yeah. Exactly. You know exactly what
Luke Stutters [00:38:00]:
Right. I will check it out.
John Epperson [00:38:02]:
I've seen that art too. He looks like he's really angry. Exactly. He's he's
Luke Stutters [00:38:07]:
he's got he's got no hair. Right?
John Epperson [00:38:09]:
Yeah. Bald head.
Luke Stutters [00:38:10]:
Yep. That's the guy. Yeah. Alright. Alright. I'm I'm a huge anime watch too. That's a that's a dangerous gap in my knowledge.
John Epperson [00:38:18]:
Yes. So, anyway, it's like you go into the state, just oh, just yet you were you were saying you go into the state when when the the first
Jesse Spevak [00:38:27]:
And it and unfortunately for me, maybe it's because of my age or whatever, but it getting into that really focused state actually, like, costs something, right? It takes, like, a little bit out. You can't maintain that state forever. There's, like, a man pool almost of how long you how long for me I can be in that extreme focused state. And I think that when there are times where I look at a problem, I'm like, do I like, I I almost, like, ask myself, do I need to be in this highly focused state to get this thing accomplished? And more often than not, the answer is yes, but my initial answer is no. And And so I end up wasting time by not doing things as systematically as they need to get done. Am I am I alone on this? Nobody else goes into the guitar state when they when they code?
John Epperson [00:39:13]:
I, I mean go ahead.
Luke Stutters [00:39:15]:
I find that, yeah, people talk about this flow state, and I was I was, I was not a believer for a while until I got something really hard to work on, you know. And it's those problems where you're the kind of limit of what you can do when some you're thinking, can I actually write this? That is when I find you get these kind of periods of intense concentration. And obviously you don't want to be, you know, the edge of your ability all the time. You want to kind of line up the nice easy jobs or, things tend to go disastrously wrong. But one thing I have found is that, you know, when you're doing these really hard problems, and you're like, can can we do it? You know, can is it possible? You always have to come back and redo it. Once you prove that it's possible, then you have to hit it again, and then you come up the 1. So all of the stuff I've done in a kind of state intense concentration tends to get thrown away, but it moves you forward down the old down the road down the road.
John Epperson [00:40:16]:
I always call those my naive implementations. And and it's totally cool to write really terrible code during naive implementation. That's not the point of it. The point is to get from having nothing to having a working thing. The important thing is that after your naive implementation, you decide, you know, decide whether it's worth refactoring, when you're gonna refactor, all that kind of stuff.
Jesse Spevak [00:40:41]:
Yeah. I I love that that idea of a naive implementation, and I think the Akka microservice was our naive implementation. We didn't consider the scale at all. We didn't consider the makeup of the team changing. And as a result, when we went back to fix the naive implementation, right, when we wanted to get rid of the double loops or whatever, we ended up moving to Java from so from Kotlin to Java, and from Akka to Camel. And the reason that those tools made sense is because they were very common in our company. And what we didn't have to reinvent the wheel. We weren't seeing errors for the first time in the entire company.
Jesse Spevak [00:41:24]:
There was sort of a knowledge base to draw from. So the current state is that we have Java, Camel, Spring, Microservice, talking to the rails monolith. It's very stable. It's performing super well now. And, yeah, I mean, getting I guess, a question I have is, like, how can we speed up the process of getting from our naive solution to the the more learned solution?
John Epperson [00:41:51]:
I think, in my opinion, if I had an answer for that, I would be I I would be a super wealthy person. Because in my experience, usually what happens is when you get to the point that you're calling something a naive solution, right, it's usually because you recognize that there's a problem with it. Right? And one of the things that's going on, I feel like, is that you're sort of you're you're kinda burning yourself out on the problem at that point. When you when you sort of have this realization, you're also at the point that you're kinda burning yourself out in this problem. So if you as a as an organization don't have the resources to sort of, like, swap out people, you know, bring somebody in that's fresh, things like that. Right? Or or kind of go back to a huddle and, you know, somehow become reenergized. Like, all the things that we can do are are almost all, like, social interactions, not engineering interactions. Right? And so if if you have a culture where you can kind of deal with that, I feel like people can kind of turn around and refactor and make reasonable choices or whatever.
John Epperson [00:42:59]:
But if you don't, like, I feel like it's really hard. It's really hard for you as somebody who just burned yourself out, pushing something across the finish line that was really hard to then turn back around and be like, I'm throwing you out, and I'm gonna redo this really hard problem again. Right? Like, that's just a really hard thing to do, I think.
Jesse Spevak [00:43:17]:
It's interesting that you say that because at the point that we made that decision, the problem didn't seem hard anymore. At the point that we were like, you know, we can just do this job in Campbell. We had wrapped our heads around the problem enough that we really felt like it wasn't hard anymore. And maybe maybe that's what it takes.
John Epperson [00:43:35]:
You also said a really important word there, which is we. Right? So one of the things that you did as a company to recover from this problem is you went back to this huddle, and you said, hey. I made some mistakes. And and then your team said, that's okay, Jesse. We, as a team, are taking ownership for this. Right? Like and we, as a team, are gonna fix this. Right? Like, that that's what a lot of that communicates right here. And when you do that, like, that's one of those reenergizing moments or, you know, things.
John Epperson [00:44:07]:
And then those decisions become a lot easier in in my experience.
Luke Stutters [00:44:11]:
It breaks my heart every time every every every time I have a have a git commit with a kind of more lines removed and added. Oh, man.
Jesse Spevak [00:44:20]:
Those are the proudest days.
Luke Stutters [00:44:22]:
Right? I know some people like to take the code out, and they're like, oh, I go knocked out, you know, loads of that repeated code. I just think, why couldn't I get it right the first time?
Jesse Spevak [00:44:31]:
That's interesting. The person who taught me how to code, one of the first pieces of advice that he gave me was that you should write a lot of code and throw a lot of code out, and that's how you'll get better at coding.
Luke Stutters [00:44:45]:
That is very, very sensible advice. Very sensible advice. And, by that standard, I've got amazing at coding over the years. Do I, the context is a big mistake. Now just to be clear, this big mistake has had a happy ending. So this isn't this isn't the kind of big mistake I got fired. This is big mistake, we pulled through it, and it all worked out. Am I right?
Jesse Spevak [00:45:06]:
Thankfully. Yeah. So I think so we did a couple of things, I think, that helped us. So the first is, as John said, we were open about these things. We didn't try to hide that things weren't going as well as we had wanted them to. And I think that Ibotta has a pretty strong culture in the sense that we're not trying to throw people under the bus in engineering. If if something crashes, it's not who's gonna get fired? It's like, okay, how do we learn from this? This is a mistake that cost us some money. How do we make sure that we that money is actually teaching us something? So that was part of it.
Jesse Spevak [00:45:42]:
And then I think also we did a good job of communicating, like, to external stakeholders. We communicated to the the finance team, who are kinda one of the main, main, consumers of the data that we were producing. And, you know, really went through in detail, here's where we're at, here's the timeline, like, updating them and checking in with them all the time. And just keeping expectations in line, I think, really helped us out. So even though we we delivered a little bit later than I think we thought we would at the onset of the project, because we were able to communicate that, we, we were not fired. And, yeah, I mean, not only that, we we've hired more people. We're still hiring. And, if you if you're thinking about getting into the mobile coupon space, there's a ton of really cool problems, even if you're not passionate about mobile coupons.
Jesse Spevak [00:46:35]:
And you might get to talk to me in the interview process, which will be fun.
Luke Stutters [00:46:38]:
Speaking of interviews, I see a smooth transition there. Speaking of interviews, I understand that you have a project for which you call MENA SWAN Interviewing. That is I have no idea what Mina Swan sounds, but I see a lot of people saying it in the Ruby community, but I just just not there's one of these weird in jokes I think they have in the Ruby community. It's nice and
David Kimura [00:47:00]:
it's nice and sweet.
Luke Stutters [00:47:02]:
I know. I know. I know.
David Kimura [00:47:05]:
Try humor, Luke. Never pick up on your sarcasm.
John Epperson [00:47:09]:
Good lord. I'm just over here, like like, dying every
Luke Stutters [00:47:14]:
and that's the, what was it, Dave? Sorry. It
David Kimura [00:47:16]:
was, seriously?
Luke Stutters [00:47:18]:
Yeah. Go on.
David Kimura [00:47:19]:
Matt says nice, and so are we.
Luke Stutters [00:47:21]:
Yeah. What is the best thing about reviews community? It's a it's a really, it's a really great, language, and it's a really strong community, really great events, even though, obviously, the events this year have been a bit difficult. So how are you carrying that culture, that tradition over into the interviewing process? What's your gambit?
Jesse Spevak [00:47:43]:
Yeah. So maybe all maybe everybody has, experience or a lot of folks haven't ex have an experience in, let's just say, a not meet us one interview. An interview that maybe is a little more hostile than we'd like. And I think I think a lot of us have experienced kind of broken interviews where it feels more like the person on the other side of the table is trying to prove how much smarter they are, or how much better they are at coding than I am. And that's not nice. Another thing that's not nice is asking someone to do an inordinate amount of work outside of work, that's not paid in the form of like a take home project. So I've done take home projects that have taken me entire weekends, multiple days, and that's uncompensated work. And that can bias your process against people who have outside of work commitments, like families and just, you know, who don't wanna be working all the time.
Jesse Spevak [00:48:44]:
So I thought it would make sense to kind of take the best part of the Ruby community, this idea that Matt is nice, and so we are nice, and apply it to interviewing. Let's like, actually be nice to the people that we potentially could be working with. And, you know, the pandemic has been terrible in so many ways, but it did offer us this opportunity to kind of dramatically rethink what our interview was gonna look like. Because we're not coming into the office, everything has to be remote. And basically, our HR team and our leadership are like, how how can we do this? We've only been accustomed to bringing people in, asking them super tricky things that they have to whiteboard. How can we, how can we translate this to a remote interview? And this is what I proposed, and this is like what we landed on, which is an interview not meant to trick the interviewee. It's an interview meant to simulate what the first couple of days of work is gonna look like. And it's supposed to give, give us as an organization a sense of how much we would enjoy this person as a colleague, how successful they'll be.
Jesse Spevak [00:49:53]:
And the message that we're always trying to send is not, hey, I'm so much smarter than you because I understand recursion or because I understand how to do whatever this type of model, this type of data structure. The message is, you're gonna be successful on our team, and you're gonna like working with us, and we're gonna like working with you. So the way that we do that is basically by giving the person a sample of code from our domain. And it's highly simplified, and it's not and we ask them to just read the code and we say, what is this doing? What do you like about this code? What do you not like about this code? And it's not a bug finding adventure. We're not asking them to find where where an error is gonna be secretly raised or why a test is failing. That's that's kind of beyond. Like, you can't really do that in a in a 20 or 30 minute conversation. We wanna hear how this person would approach an alien code base, which is what their first task is gonna be on the job.
Jesse Spevak [00:50:49]:
Then, we present them with some data, similar, you know, some kind some award events from our system, and we ask them to manipulate that data in, with a pretty simple algorithm. We even tell them what the algorithm is, and we ask them to code it. And we say specifically, we don't care about the answer here. The answer is not interesting. We just told you what the answer is. We actually wanna see what it looks like when you code. Now, what what's your approach? Are you systematic? Right? Are you making guesses about what should happen and checking yourself? Those are the things that we're looking for. We're not looking for some for to see, you know, do you know this random algorithm from your, from your computer science education that you'll never use as a as a web developer at Alana? So those are the those are the big pieces.
Jesse Spevak [00:51:35]:
And I'm stoked because, I got invited to talk at at RubyConf, which is coming up in November, where I'm gonna be kinda outlining this. You all got a a preview of the content there, Exclusive to Ruby Roads.
John Epperson [00:51:47]:
Awesome. I'm definitely curious as you guys implement this, like, when you kind of come back and say, well, this works and this didn't work. Or or this is, like, this is how we sort of had to come up with a way to, you know, because this is a subjective thing. This is this is how we came up with a way to to make this more objective than than it originally was, you know, and and that helped us. Like, definitely interested in seeing this. I I mean, all of your all of the pain points that you're talking about. Right? All this homework that we have, all these, you know, coding challenges that we do. I mean, we we're very I feel like as a community, we we talk about how aware we are that they don't work, but we don't have alternatives yet.
John Epperson [00:52:30]:
And so people continue to use them regardless because they're like, well, I don't know what to do. And people who are lost, you know, just run back to where they came from a lot of times. Right? It's like the squirrel. It's like in the middle of the street and the car is coming. It's like, shoot. I gotta go all the way back. So I I kinda feel like we do a lot of that still. Yeah.
John Epperson [00:52:48]:
There have been there have been various things, like, talked about. I remember I think I've heard various people from thought on various podcasts, like, talk about the fact that, like, they they have you come and spend, like, I don't remember if it was, like, a day or whatever or half day, but, like, a lot of time with them pairing. Right? I know that boot camp does that as well or base camp. Sorry. Wow. Whatever. Base camp does that as well or something similar. They'll they'll actually, I remember heard they had an article about they did some homework or something recently.
John Epperson [00:53:16]:
Whatever. They they did they did
Jesse Spevak [00:53:18]:
a case of the homework, though.
John Epperson [00:53:19]:
Yeah. That's okay. Yeah. I mean, there's there's people that are, like, exploring around these edges, but we don't really have, like, a good way forward, I feel like, yet still. So I'm super interested to see how this turns out.
Jesse Spevak [00:53:31]:
Yeah. I mean, so far, you know, obviously hiring has has been slower than than typical for our for for Ibotta. But we are still hiring. And so far, the feedback from candidates has been really interesting because, you know, we'll get unsolicited emails about how much people like the process and how they hope that this is the direction that the industry goes in. A lot of folks who are interviewing with us are interviewing at other places as well. And I see this as like a competitive, a competitive advantage for us. Right? Does this buy us like $10,000 in salary or $5,000 in salary or something like that? Like, does this make our company more you know, once you get to a certain point, there's like diminishing returns on all these different levers that a company can offer to a developer. And I think that this is maybe an unexplored lever or a lever where there's a ton of room to gain.
Jesse Spevak [00:54:22]:
And when someone goes through our process and comes out feeling like, hey, I'm gonna, I'm gonna kick butt there. The people there are super nice. They didn't make me feel like an idiot. And they contrast that with other interview experiences they've recently had. Hopefully, that that that will play in our favor.
Luke Stutters [00:54:37]:
That is certainly not the current approach to coding interviews. I've been hearing recently more and more people talking about make how to make coding interviews harder. If you go on something like Hacker News, you know, they discuss which framework they need to insert under the fingernails of potential applicants. And I hear if you go to Facebook now and apply for a job, then they, you have to bring along your PhD in computer science. They, then burn it in front of you, mix it into old coffee and make you drink it as part of the interview process. So it's really, I think it's really something that the, tech community needs. We have been doing a few episodes about trying to make community more inclusive, trying to bring in people, trying to keep people in. And that sounds like a really great idea for talk, you know, why don't we be be, nice of people? Try and get the most out of them during interviews instead of subjecting them to torture.
Jesse Spevak [00:55:35]:
I love that image of shoving a framework under someone's, fingernails loop. But the the thing we the thing that I noticed, and I've done a lot of interviews that I bought. I've done, I would say, easily 200 interviews over the past 2 years. We do ton of firing, so it's it's crazy. And the thing that I noticed is that when we were asking people to whiteboard, right, to answer these kinda tougher algorithm algorithm questions, I didn't I didn't always see a one to one correlation between the people who crush that kind of question and the people who I really enjoy working with in either direction, right? There are false positives and false negatives. And I just found there to be way too much noise in that type of diagnostic tool for me to really make a strong decision that I felt confident with that proved out over time. And I'm hoping that this is a remedy for that.
John Epperson [00:56:33]:
Yeah. I mean, you always have to check to make sure that whatever test you're giving. Right? Like, is the you have to decide what you're testing for. And you have to say, is the question that I'm asking actually testing for the thing that I'm testing for? And too often, we jump to this conclusion that, oh, yeah. This sort of this sort of is related. Therefore, it'll let me know. When really you're just you're making a subjective judgment call again and calling it objective. So yep.
John Epperson [00:57:01]:
It is it is a thing that's very painful, and maybe maybe we should we should focus a little bit on this in the future and and have a more in-depth chat about this. I feel like that would be an interesting subject.
Jesse Spevak [00:57:14]:
Yeah. There was a last year. I believe the guy's name is Eric. I'll throw it in the in the chat in a second. He's from Test Double, And he talked about, Test Double is like a rail or a roomie consultancy, based out of Columbus, I believe. And he talked about their hiring process and more in the sense of our goal is to build a process that lets candidates show off their strengths and as opposed to helping us find their weaknesses. And that talk really informed our process at Ibotta. We're trying to find people's strengths and trying to figure out where they're gonna be able to make the most impact in our company.
John Epperson [00:57:52]:
Yeah. I have I have gone for the past, like, I don't know, 4 or 5 Rails comps. I've gone to basically every single talk that was sort of, like, related to the subject. I I do I care about it quite a bit, and I I just kinda hope that we get to a better place. It's yeah. Like I said earlier, my my feelings on this, I think, are are basically what I said earlier that I think that a lot of people are thinking about this, and I just don't think there's a clear answer yet, but I'm super interested at, like, everything that people are trying. Experiment. Hey, Sean.
John Epperson [00:58:23]:
It's gonna that's how I think we're gonna get a little better.
David Kimura [00:58:26]:
Yeah. Well, hey. Looks like we are coming up on the hour, so we need to move things along. Jesse, if people want to get in touch with you, where should they go online and look?
Jesse Spevak [00:58:36]:
I would say, you can find me on Twitter. I I tweet from planet efficacy. So if you can if you can spell that, you can find me. And, you'll find a lot of interesting, movie content, political outreach, and, Vermont progressive rock on that Twitter stream.
David Kimura [00:58:56]:
Awesome. Well, let's go ahead and move over into some picks. Luke, you wanna kick us off?
Luke Stutters [00:59:02]:
I would. My first pick is a version of Windows 10.
John Epperson [00:59:09]:
A bit
Luke Stutters [00:59:09]:
of a strange thing to pick on a, but this is called Windows 10 ameliorated edition. I found it featured on the popular YouTube channel not long ago. And this is a version of Windows 10 porta spyware taken out, which also incredibly boosts the responsiveness and performance, because it's not doing any async network calls back home in the background, if you know what I mean. This is the first operating system I've ever found where to download it, I had to get a bit torrent link from a Telegram channel. So if you want some excitement in your operating systems, check out Windows 10 ameliorated edition. Never seen anything like it.
John Epperson [00:59:59]:
Just just wanted to confirm something here really quick, Luke. Is this a legal version of Windows 10?
Luke Stutters [01:00:05]:
It's got quite a big section on legality on the website. Okay. The website does make it clear that you do need to have a Windows 10 license in order to legally use the Windows 10 Ampereaser edition. But I mean, come on. Hit torrent link through a Telegram channel? Wow. That's, that's quite a anyway, it's it's quite trippy. It's a cool little project. There's lots in the FAQ about their rationale behind it, and it does fly when it runs.
Luke Stutters [01:00:36]:
It's really quick.
David Kimura [01:00:38]:
I heard there were some issues with it with not being able to do certain things because it can't call home to Microsoft. Even some, like, simple rudimentary things that you would think that would be possible, but just kinda breaks it.
Luke Stutters [01:00:53]:
Yeah. I mean, you could call it Windows 10 Lobotomized. I mean, there's barely anything there at all. But it's it's oh, it's a lot of fun.
John Epperson [01:01:01]:
It flies doing nothing.
David Kimura [01:01:03]:
Awesome. Well, John, you wanna do some pics?
John Epperson [01:01:06]:
Yeah. So I have a different pick this week too. You may by the time this this comes out, I have no idea if people are even gonna still be playing this game, but I have been playing and streaming and and absolutely having a blast playing, I guess, that's redundant among us. So if you're not familiar with it if you're familiar with, like, Werewolf or Mafia or kind of games in that nature where you there's another game that I people have, like, compared this to or whatever. Where you, like, sort of have, like, like, townsfolk, and then you have, like, people that are pretending to be townsfolk that are, like, trying to kill everybody off. And your goal is the townsfolk or to try and find the, one in this game, the imposters among you. So this is, like, set in space or whatever. But, yeah, it's an absolute blast.
John Epperson [01:01:56]:
I I really am not sure what to say about it other than it's, like, kind of, like, a great social game. Yeah. It's it's pretty awesome. So I I highly recommend checking it out. I definitely have been doing a lot of streaming of it lately. So yeah.
David Kimura [01:02:10]:
I'll jump in with a few picks. First pick, bamboo flooring. So I love bamboo, and instead of the crummy old chair mat, the little plastic thin plastic that I was using for a long time, I finally went to Home Depot and got some bamboo flooring. And I put glued that to a piece of plywood, and I now use that as my floor mat on my carpeted floor. So my chair chair can just slide around on it. It's really, really cool. And it's been a life changing event here, so it's pretty awesome. And the second thing that I pick is the Elgato key light.
David Kimura [01:02:50]:
So I got those for my streaming setup, and they make a huge difference. So you won't be able to tell on the podcast, but this is with my lights on, and this is with the lights off. Makes a really big difference in quality. So having proper lighting on doing any kind of video work is an absolute must.
Luke Stutters [01:03:07]:
Can you can you do that again, Dave? Just so everyone can see you on the podcast. Yeah. I gotta say that is quite striking. Is that the same company that did the shortcut bar?
David Kimura [01:03:17]:
Yes. The Stream Deck. Mhmm.
Luke Stutters [01:03:19]:
Yeah. That's a pretty cool thing as well.
David Kimura [01:03:21]:
Big Elgato fan. And, Jesse, do you wanna jump in with some picks?
Jesse Spevak [01:03:25]:
Yeah. I've got a couple of picks, different categories. So the first is a talk that I watched recently that maybe you all have seen already, but I recommend revisiting it. So Sandy Metz's RailsConf 2019 keynote, which is called Lucky You. And I think it's especially timely at the, as we approach election season and thinking about equality and issues of inequality in our country. So I highly recommend that talk, it's amazing as all Sandy Met's talks tend to be. Then I have, when the pandemic started, we started working remote. I made a small investment in my work from home setup that I highly, highly, highly recommend.
Jesse Spevak [01:04:08]:
I went out and I got an ErgoDox split keyboard, having some wrist pain on the keyboard I was using, so I know keyboards can be a whole other podcast, but check out ErgoDox. They make a really, really, really good product that's worth the price, and I've been a huge fan of it since getting it. And then my final pick, there's a book that just came out. It's a guilty pleasure. It's called The Trouble with Peace by Joe Abercrombie, a British gentleman, And it is in the grim dark genre of fantasy literature, and I highly recommend it.
David Kimura [01:04:43]:
Awesome. Well, thank you for those picks, and just be sure to post them into our chat section here so we can include them on the show notes. Well, Jesse, thank you for coming on today. It was a lot of fun. I love these kind of talks where we can just humble ourselves and talk about past mistakes and, you know, what we learned from them. So thanks again.
Jesse Spevak [01:05:03]:
This was awesome, guys.
Luke Stutters [01:05:04]:
Really interesting.
Jesse Spevak [01:05:05]:
I listened I long time listener, first time guest, and this was this was awesome. I appreciate it.
John Epperson [01:05:11]:
Yeah. Thank you for coming. Right.
David Kimura [01:05:12]:
Well, that's all for this episode, everyone. Thanks for listening.
John Epperson [01:05:15]:
Take care, everybody.
Exploring Tech Choices and Team Dynamics with Jesse Spivak - RUBY 669
0:00
Playback Speed: