Deeksha Goyal is the Senior Machine Learning Engineer at Lyft and Michael Sun is the Staff Software Engineer at Lyft. They delve into the intricacies of machine learning and data-driven technology. In this episode, they explore the challenges and innovations in deploying models into production, particularly focusing on the real-world implications of ETA (Estimated Time of Arrival) modeling at Lyft. They share valuable insights, from the complexities of A/B testing and long-term impact assessment, to the dynamic nature of handling real-time data and addressing unpredictability in route predictions. Join them as they journey through the world of model deployment, bug identification, and career development within the fast-paced environment of Lyft's data-driven infrastructure.

Sponsors

Socials

Michael Berk [00:00:09]:
Welcome back to another episode of adventures in machine learning. I'm one of your hosts, Michael Burke, and I'm joined by my beautiful and lovely co host.

Ben Wilson [00:00:17]:
Ben Wilson.

Michael Sun [00:00:18]:
Cool.

Michael Berk [00:00:20]:
Yeah. His name is just Ben. He doesn't have any background. And today, we are speaking with, machine learning engineers at Lyft, Diksha and Michael. While getting her master's degree from Stanford in computer science, Diksha interned both at Google and Lyft. She also cofounded a startup. And then while she had some free time, she served as a teaching assistant. And currently, she's the ML lead for estimated time of arrival prediction.

Michael Berk [00:00:44]:
Michael studied at UBC, and after graduating, he held software engineering positions at SAP, Amazon, and most recently at Lyft. He works alongside Diksha focusing on ETA and routing. So, Michael, kicking it off with you, before we get into the technical, what is GirlSmarts?

Michael Sun [00:01:02]:
It's a great question. I didn't know you would dig dig that hard back. It's a volunteer program of teaching girls in grade 5 to 7 how to code. And we had, a few programs. We have, like, a software, like, engineering where we just teach them how to code in very basic, coding languages with some if statements and stuff. We also had a UX design program, which was the one that I led and developed the program for. Basically, you draw, the UX on a piece of paper and you take photos on an iPad, and you can link pages together, on an app, so develop a user experience that way. But, yeah, it was basically a fun program to get, girls engaged into the life of computer science.

Michael Sun [00:01:53]:
So, hopefully, that they will take up a career in that, when they grow up.

Michael Berk [00:01:59]:
Awesome. And you've been doing it for 11 full years?

Michael Sun [00:02:02]:
No. No. That was I probably should love doing my LinkedIn. I did that, through my undergrad, and I kinda had to stop, after I graduated because that program was in Vancouver where I studied. And now I'm in Seattle, so, couldn't, do that. Yeah.

Michael Berk [00:02:21]:
Cool. Alright. Well, yeah, I was just curious, when you peruse, you see stuff. But, Ben, it seems that

Ben Wilson [00:02:27]:
you hear the question. Based on your experience of doing something like that, where you're taking a a complete novice and then introducing them to a complex technology stack or the concept of building something. Do you did you learn things and notice things while you were doing that instruction that informs UX design or API design later on in a career to say, I'd I can sort of recognizing patterns that were hard for me to explain and teach to somebody to learn, so I shouldn't build software like that later on?

Michael Sun [00:03:09]:
That's a great question. I think it's less translatable for me just because I mostly work in back end, and UX is more so on the front end. But I've definitely noticed, when somebody starts to do UX, they question everything. Like, why is the close button on the top right? Why couldn't I put it on the bottom left? You know, those are the things that I never think of and I take for granted. But these kids, they they question the thing that makes me question things as well and and how why they aren't designed such way. So that that's one example. And, also, like, it would ask, oh, why do I need to go through different screens? Why can't things just happen on one screen? It's also, like, another question that they'd come up with, through that exercise. So I thought that was pretty fun, to learn.

Michael Berk [00:04:00]:
Yeah. Do you definitely notice when you're entering a new environment, like, as a professional or whatever it might be, you question organizational, things that just are, like, embedded and everybody takes for granted. And, yeah, I would love it if the the exit button on my Chrome window was in the bottom left. So it would just be truly game changing. Absolutely. So what is ETA? Can you explain that as a concept?

Michael Sun [00:04:31]:
To a 3 year old or to someone who's somewhat familiar with technology?

Michael Berk [00:04:36]:
In the context of lift, like, estimated time of arrival. Alright. I'm gonna get there in 5 minutes. But what implications does that have for the business?

Michael Sun [00:04:46]:
Yeah. That's a great question. And Deetra probably has a lot of, things to add. But to at at a high level, ETA goes in to a lot of decision making that we do, at Lyft. So how to set the price, how to set, who to match to who, so matching decisions. And as well as all the ETAs that we display to the users, for how long, potentially a car might be able to approach you at the pickup location without knowing who the driver is gonna be. Because when you request a ride, we have to find the right driver. So how do we predict the ETA for that? And, also, how do we predict the eventual ETA to the destination? As well as, for example, if you need to get to the airport by a certain amount of time, then when do we need to start requesting for a lift for you? So it's a bunch of, interesting, edge I guess, I would call them edge cases for ETAs.

Ben Wilson [00:05:42]:
I've got a question. Oh, sorry. Go ahead.

Deeksha Goyal [00:05:45]:
I was gonna say one thing I'd add to that is, an interesting question we, you know, we've had discussions about on the team is, how do you determine that a new ETA model is good? Like, how do we decide that this is actually good for the company? And as an example, say we have a new ETA model that makes all ETAs for all riders shorter by, like, 10 minutes. Riders would be really happy with that. Right? They would be like, oh, like, I should request this ride. Like, this is a great ETA. Right? But that actually messes up our entire marketplace because that's not an accurate ETA. We're not pricing rides correctly. We're not matching you as efficiently as possible. So that kind of leads us to question, what is the right way to evaluate an ETA model? And it often just comes down to the core accuracy of ETA and and distance itself.

Michael Berk [00:06:44]:
Right.

Ben Wilson [00:06:45]:
Pretty much partially answered before I even asked the question. My question was for you, Diksha, with what is the experience like for being a tech lead that owns one of probably arguably the most important core fundamental model or systems of models that runs that business. It's just like you said, if you get that wrong

Michael Sun [00:07:10]:
Mhmm.

Ben Wilson [00:07:10]:
You could lose customers or you could seriously irritate drivers. You know, there's a lot of weight on the shoulders of the team. What does change management look like there?

Deeksha Goyal [00:07:25]:
That's a great question. So there's a lot that goes into a an ETA model update that has a phase shift, in terms of accuracy. So, like, Michael and I, we recently worked on a project where there was a pretty sizable, improvement in accuracy, And that wasn't something that we could just ship like that because we have other teams that have models that are coupled to ETA. So even if we have higher ET accuracy, we need to make sure that we work with all of our partner teams, like matching, pricing, and make sure that they can accommodate for this change. Even though it is actually a good change for the company, we need to make sure that we're all aligned. So that's in terms of, like, sort of, like, business level metrics, like, what we're tracking, but then there's also a lot of work around, like, basically, when you request an ETA, it's not just when you request a ride, it's not just one ETA. There's multiple ETAs that come out of it. Right? So, like, we have to match you to a driver.

Deeksha Goyal [00:08:31]:
So we look at, like, k nearest drivers, which means we have to get k ETAs. And so, Michael, as, like, the engine lead on ETA, you can probably comment on, you know, how the the scale of of what we're working with here.

Michael Sun [00:08:44]:
Yeah. Sure. So we probably have about 30 callers into our service and, for a variety of different reasons or use cases. Like, I mentioned some of them matching fair, etcetera. So and then as a result of that, we serve probably millions of request, at a at a minute level. So we have to be really mindful of not just the the model accuracy, but also the reliability of the service. I would say also the other side of of what Diksha said is there's also a lot of guardrails on other teams to make sure that ETA changes are not impacting, that much, in the downstream dependencies. So for example, on the matching side, there are times when ETAs could hiccup and the service may face some unreliability issues.

Michael Sun [00:09:40]:
So they have, a way to fall back into something that's also less less, accurate of a prediction, but that would still ensure that drivers could get matched to a ride and vice versa. Our riders could get matched to a driver.

Ben Wilson [00:09:54]:
So that sort of tiered approach, I think Michael and I have talked about it with an a number of people who've been on the show over the last couple years. Like, hey. You you hope for the happy path of the highest accuracy, most sophisticated model, But then you have an a fallback on the data science side. We're like, hey. If we don't return in enough time or something went wrong and we're our monitoring is saying that, hey. The model we're we're putting into staging or into production, you know, worst case, will fall back to our our version on our side of the API. And then downstream, everybody's gonna have their own fallback logic as well. I think that's an important thing to continuously on this show at least for us to mention.

Ben Wilson [00:10:38]:
Like, hey. Everybody does this who's doing this for real. Because if you put all your eggs in one basket, that one model, and you don't have robust deployment going on, you're like, hey. We do shadow deployment for 4 hours and make sure that, you know, metrics coming back are are good or however I don't know how you guys do it, but making sure that you have that cutover and reversion logic in to say things are on fire, move back over to yesterday's build or something. Yeah. I've worked at companies where we like, people weren't doing that. You know, like, we're using ML. It's in production.

Ben Wilson [00:11:15]:
And then something shifts, like, everything is on fire. What do we do? So what's that process like at Lyft when when you're if you guys can talk about it, about deploying, like, a new fundamental new feature? Like, you said that you you just released an accuracy improvement. How do you go through that process of making sure that it's revertable?

Michael Sun [00:11:40]:
I don't know if you wanna take it 1st, Deepgram. You can

Deeksha Goyal [00:11:42]:
go first. Go ahead.

Michael Sun [00:11:43]:
Alright. Okay. I would say this process, takes quite a bit starting from the model sort of proof of concept stage. So Disha can probably talk about that. And once we have somewhat of a confident model that we can prove the accuracy improvements offline, then we get into the phase of how can we roll this out, and how we're how can we bring it into our own service, and then make sure all the downstreams, can react to the changes that's caused by this model. So we basically talk to a bunch of teams. One of the most important teams, that we talk to is the team that lies on ETA distance to, set the pricing, because pricing has correlations with, ETA distance as well. So we talk to them about, hey.

Michael Sun [00:12:33]:
There's this new ETA accuracy improvements. Can you guys, make sure that this this change is addressed in your model as well? So we work with them to train a new version of their model, and then we would, at that point, sort of figure out how we can get into experiment or AB testing for the new version versus the old version for both models at the same time. And then in the experiment phase, Diksha can probably talk about all the metrics that we monitor as well. We figure out whether the metrics are green, basically, good or bad. Sometimes they're a mix. But if we and then we do our experiment review with the national teams, including the pricing team and also the matching team who's in charge of, matching the right driver to the rider using ETA and distance, as well as figuring out what I mentioned earlier, what the right ETA to set when we don't even know who the driver is going to be, when the rider requests for that ETA. So there's these metrics that we look at, how that impacts the accuracy of these downstream metrics with our new versions. And if they look good, then we roll it out.

Michael Sun [00:13:49]:
And then there's, obviously, the process revertable. So we keep the old model, in production for a few weeks just in case something goes bad so we can revert to that. I should also mention before we get into experiment, we do the shadow phase where we have the new model running in shadow, against the production, and we can observe online what the performance of the model is and how that may affect downstream teams.

Ben Wilson [00:14:17]:
That's gonna be crazy complicated for Diksha to handle that concept that you you've got a new idea, a new feature You're putting new data in to say, hey. This is my hypothesis. I tested it. My model works really well. But then you're having to monitor 7, 8, 30 different consumers of your model, and then also having to think about your changes as they impact all of those and ingesting that validation data in that shadow deployment.

Michael Sun [00:14:53]:
How long

Ben Wilson [00:14:53]:
does that take? Like, when you're doing a new major feature, is this like you do 1 or 2 a quarter, or is this much faster development time?

Deeksha Goyal [00:15:04]:
We we're able to ship so, you know, it depends on the type of change. We're able to to ship, like, feature changes to existing models pretty quickly. Like, it could be, like, if it takes, like, say, like, a few weeks add a new feature, we shadow it for a few weeks. By the time we get to experimentation, we are we don't wanna run an experiment on something that that we truly don't know anything about. So when we go to experiment, we feel pretty confident that this is going to look good. It's not gonna degrade business metrics, but it's more of a sanity check because, as you mentioned, ETA affects so many other services at Lyft. So we just need to ensure that other metrics aren't degrading. But, yeah, like, for for simple features, additions to the model, it's not too bad.

Deeksha Goyal [00:15:54]:
But when it comes to, like, fundamentally changing, like, rewriting our entire ETA system, which is something we just recently did, that that that's something that's a lumpier process, and it requires, a lot of buy in from stakeholders, a lot of explanations because we like, in that in that rewrite, we were looking at, like, our experiment review. It was, like, probably, like, 40 metrics that we're looking at. And it's pretty difficult to improve 40 metrics all at once in one experiment. And so there's a lot of trying to understand, like, which of these metrics actually matter. Which of these metrics, if they degrade, it actually doesn't mean that we're doing something wrong here. It means that we need to ask a partner team to fix something. So, yeah, there's just a lot of, like, aligning stakeholders and and getting buy buy in from them.

Michael Berk [00:16:50]:
How do you do stat sig calculations for 40 metrics?

Deeksha Goyal [00:16:56]:
So we before we run the experiment, we we do a power analysis to understand how long we would need to run this experiment for. So just just given, like, an estimated idea of, like, how what the impact would be on that metric, we're able to simulate it beforehand. So our ML platform is actually really, really solid. I know you guys interviewed them just, like, a month or so ago. And in there, we have, really nice tooling to understand, okay, what is the power of this experiment gonna look like? Should it run for 2 weeks? Sometimes some of the metrics we're looking at, we just won't get enough power in 2 weeks, so we have to run it for 4 weeks. And then there's certain metrics where it's like we wanna ensure like like, for example, we wanna see, like, long term impact maybe in terms of, like, long term retention of riders or drivers. For that, we we have to go for for longer experiments.

Michael Berk [00:17:55]:
Right. And then you also guys, you implement something called surrogate metrics. Right?

Deeksha Goyal [00:18:02]:
Surrogate metrics?

Michael Berk [00:18:04]:
Cool. So I I come from a a company that did a lot of a b testing and specifically, with similar 2 sided marketplaces.

Ben Wilson [00:18:13]:
And, one of the

Michael Berk [00:18:14]:
big challenges is determining long term impact of an experiment. And so, I've read in some blog posts that you guys leverage, the concept of a surrogate metric, which is basically a short term metric that is indicative of a long term trend. So what kind of metrics are like, short term metrics do you guys use to sort of estimate the overall business impact over, let's say, a year or even longer?

Deeksha Goyal [00:18:39]:
Okay. Yeah. That that makes sense. Well, for for metrics like that, we're looking at sort sort of, like, there could be trade offs in terms of change in, like, conversion versus change in profit. So it's like a per ride. Is this actually, like, improving the experience of the rider in terms of, like, actually being more accurate? But is there is there a trade off in terms of retention? And so we have some metrics that weigh that trade off that are set by the company. They're not really set by, like, individual teams, and we just have to stay updated with their guidelines.

Michael Sun [00:19:18]:
Got it.

Ben Wilson [00:19:19]:
Cool. We got a potentially super stupid question for you about features and testing. Do you try to model when you're doing a new release of a of something that you're you're not 100% sure? Like, you've done all the the preplanning and done it sounds like you you front load a lot of that design work to understand the problem before you write code, which is really good practice to do and then work on the feature and then monitor it. But if you're doing a shadow deployment, say, in March and checking and saying, like, hey. We're deploying this in North America or in the state of California or something. We we wanna make sure that, you know, our services in big metropolitan areas are are good. Do you try to estimate black swan events that are hard to estimate through some sort of simulation modeling? Like, hey. We wanna estimate when a big sporting event is coming to a downtown metropolitan area.

Ben Wilson [00:20:25]:
How is that gonna how is the model gonna react to that? Is that done in some sort of simulation environment based on prior crazy things that happened?

Deeksha Goyal [00:20:35]:
That's a great question. So digging a little deeper into our ETA models, we actually have one that updates in real time. So it retrains in real time. And that has been really, really useful for these kinds of issues where there are, like, major events or there's, like, a snowstorm, or, you know, even, like, major holidays, like New Year's. Just sort of training on historical data, say it's, like, a month or 2 months of data, might not actually be reflective of what you see in production. So we actually have a final layer that's able to accommodate for that and just learn in real time about these sorts of issues.

Ben Wilson [00:21:20]:
That's fascinating. So stuff like a fire in LA underneath a bridge that completely closed an interstate. You're at the algorithm. We when that happened, do you guys, like, sit around a computer and just watch in real time? Like, how's our algorithm doing? Is it rerouting? Is it is everything changing? Do you do tell me. You guys do that. Right?

Deeksha Goyal [00:21:43]:
Well, when we rolled this out, I remember we were we were, like, oh, like, we should definitely track this new model because Halloween is coming up, and we need so we definitely were sitting around watching, like, okay, is this model doing what we expect it to do on Halloween when we're gonna have a lot of traffic, a lot of closures. So, yeah, we in the beginning, that was definitely, like, an exciting little thing, and leads would be excited to be like, oh, like, did the model accommodate for, like, there was a snowstorm in my hometown or whatever. Like, did it accommodate for that? So, and we found it to be pretty robust so far.

Ben Wilson [00:22:20]:
That's gotta be, like, this awesome feeling where everybody's just sort of looking around at each other. Like, we actually figured this out. This is awesome. Yeah.

Michael Berk [00:22:27]:
Yeah. So, Michael, I was wondering if you could elaborate a bit or Desha on the the underlying infrastructure because you guys are sort of at a an interesting place in ML where you need really fast decision, and you also need to bay basically support, as you mentioned, millions of requests at a minute level. So what does your cloud bill look like?

Michael Sun [00:22:54]:
From sense of how deployment works or how infrastructure is hosted on the cloud?

Michael Berk [00:23:01]:
I guess more on the infrastructure side.

Michael Sun [00:23:04]:
Okay. Okay. So most of our services are hosted on AWS, And, we have Kubernetes clusters that is running on AWS EC 2. And we have auto scaling units that can handle, I guess, load increases, in real time. And getting into a bit more of the service, that goes on to generate the CT at the end, because it's not just a single service or a single model that does this. So we have within, we're part of the mapping org. So within mapping, we have, a team that is in charge of curating on the map data. So you can think of that basically like Google Maps where you you say, here are all the road segments.

Michael Sun [00:23:52]:
Here are all the points of interest on those roads. So that's sort of our base map team who's in charge of doing that. And on top of that layer, we also have a team that is in charge of, observing all the drivers' movements on the map and figuring out their real time speed. And, with those, locations and speeds, we can calculate and the traffic on those real segments, either over a long period of time or in real time, for example, the last 5 minutes. That's how we can determine congestions or traffic, in certain areas. And using this traffic data, then we can come up with a an accurate route for the drivers. So we have, I guess, a service that is in charge of generating multiple routes from going from point a to point b and relying on the real time traffic data. Then we can figure out what is the best route for the driver to get from point a to point b, based on avoiding congestion, for example.

Michael Sun [00:24:57]:
Or we also have real time road closure data, to avoid road closures. And like, Diksha mentioned, if there's an event going on, some road is closed or half traffic, we can use this to route the drivers more accurately. And then the last layer on top of the routes is our ETA production layer where we generate an an accurate ETA based on, the the the, I guess, the starting point and the end point and the potential route that the driver is going to take. So we know we can recommend drivers to route, but we don't know that that's a route that the driver is going to take eventually. So how do we predict the best ETA for our riders knowing that that's the route that we're gonna show the driver? But the driver has a chance to not comply to that route. So that's sort of the problem that the ETA model is trying to to address. And maybe I sort of moved away from the infrastructure question. But I guess, yeah, if you have some specific questions, I can I can answer? But

Michael Berk [00:25:54]:
Well, just one question. So you have sort of this layered model. Is it all in one database? If so, what kind of database? Or do you have different databases to serve different use cases?

Michael Sun [00:26:07]:
You're referring to the ETA model or the different standard model?

Michael Berk [00:26:11]:
The base map and then the driver sort of, real time map and then the routing layer on as well. Is this all just in key value store? How do you store it?

Michael Sun [00:26:22]:
Yeah. Those are in different stores. So Basemath has their own store, of the data, for example, for roads and points of interest. Those are actually in 2 different data stores, points points of interest and the road segments. And in order for to compute the road on top of these, I guess, road sec or the route on top of these road segments, we have to convert the data into, you can imagine, a graph model. So we can perform Dijkstra search on this graph model. And this graph model sort of have to live in memory because Dijkstra needs to run-in memory. So we take what is in sort of, like an s three long term storage, and we convert that into graph model and put that into our memory.

Michael Sun [00:27:09]:
So we can run our route algorithm on top of that. And then finally, we have the ETA model, which is stored in memory as well in some other service. So I would say the data is pretty, distributed and not just sitting in a single place.

Ben Wilson [00:27:25]:
One thing you mentioned about the drivers not selecting a route, I imagine it must be complicated when you're building route optimization if there are rogue drivers in the training data. Somebody who's like, yeah. I I got this this route that I'm supposed to take. I don't trust technology. I'm gonna I know this city. And they always take a suboptimal route. Do you have to detect stuff like that for retraining and say, like or, like, market or label it in some way to say, this person didn't follow our recommendation or, conversely, this person found a far more optimal route that beat the ETA estimation by, like, 40%. Does that use in the analysis phase or in retraining, and and how do you handle that?

Deeksha Goyal [00:28:21]:
So we so we own the full mapping stack. And, I guess, what that means is, like like Michael mentioned, we create our own map, using combination of open source data and some really valuable data we have in house from, like, driver GPS traces. And so we are able to look at historical data and see, like you know, we recommended these routes, but drivers are not going on these routes. And it's really interesting because we have been able to learn a lot about the map and even fix the map, this this sort of open source map, by sending back edits to OpenStreetMap and saying, hey. You know, this is what we call a ghost road. No one we believe this might be a ghost road. Like, no one is going on this road. So there's this community that, like, this this sort of Wikipedia for maps called OpenStreetMap.

Deeksha Goyal [00:29:20]:
It's used by other companies as well, like Amazon, Meta, Snapchat. And all these companies are contributing back with what they find from their fleet of drivers or riders, and then we're able to leverage that in in our algorithms.

Ben Wilson [00:29:37]:
And that'll also update stuff. Like, I mean, roads don't permanently close very often, but sometimes if there's some sort of major construction project, then that'll go in. You'll fetch that latest build on, like, a nightly basis or something to to ETL the open source package in? Or is that or do you do, like, strict version controls of, hey. This hash commit is what's in production right now. We only update after validation.

Deeksha Goyal [00:30:05]:
So, yeah, we we're able to we have for the base map team, they have a lot of automated checks. So, like, they download the latest version, and then they have automated checks by just looking at our driver data. So, like, if there's a if there's, for example, a road that does not exist in our map, but we keep seeing drivers are going somewhere and it's, like, off path, then we're able to run some validation checks like that. And and we have, like Michael mentioned, a whole curation team as well, that's able to curate things and and contribute things back. So it's a it's become a pretty quick feedback loop from down downloading to validation to submitting fixes back to open source. And then but, obviously, that's not gonna fix everything. And so that's why we have these ETA models at the end where they're able to account for uncertainty of accuracies anywhere in the stack.

Michael Berk [00:31:06]:
Cool. So it sounds like the the data are pretty good, and it's a slow and steady progress for improving this data. But, Diksha, you mentioned you guys rewrote a lot of stuff recently. What was the problem or what were the problems that were addressed in the prior version, and how did you address them?

Deeksha Goyal [00:31:22]:
How should we dig into this? So, basically, we launched our own navigation called LyftNav. So if you go on a Lyft ride, you'll see that, majority of drivers chances are you'll see that your your driver is using something called Lyft Navigation. And the idea behind our update was that so drivers and riders, you know, they want the most real time information in terms of, like, closures, traffic, all of that. And so this big, big change that we had was in terms of, like, getting the these high speed signals into, our final ETA models and, basically, basically, unifying itself with this Lyft navigation platform, because with LyftNav, we're actually telling drivers where to go. We're we're deciding the routes now. And so it was a matter of sort of combining the fact that we know where drivers are likely going to go and that we have all this, traffic and closure data and just creating a higher accuracy to model because of that.

Ben Wilson [00:32:43]:
Doesn't sound like a trivial change at all. If you're integrating with the whole new system, that sounds like a monumental data science machine learning engineer and just pure software engineering challenge. How many people are involved in that, and how long did it take to do the product design before a single line of code was written?

Michael Sun [00:33:06]:
Are you referring to the navigation part or integrating our ETA stack into the Integration. The navigation?

Ben Wilson [00:33:12]:
Like, your team integrating with this entire new product feature.

Michael Sun [00:33:16]:
Yeah. I'll say we probably had, including both scientist, engineer, and data engineers, probably 6 people 6 to 8, overall that worked on this project. And over what time frame, I would say 6 months to a year sort of time frame.

Ben Wilson [00:33:41]:
Sounds about right. Yeah. Based on my experience of, like, hey. We have this new feature on the platform. It's time to integrate. We're like, okay. We're gonna take a quarter and just do design and prototypes, and then we're gonna iterate. And then by the time you're ready to release GA to, like, the public, it's, yeah, it's usually about, like, a year for something of that size.

Michael Sun [00:34:00]:
Yeah. I I would definitely echo with that by saying that implementation was not the longest thing. The longest thing was how do we get that model into production? How do we test it out? How do we make sure downstream is also, adjusting these changes correctly. And, also, the long tail was, like, debugging. We had some version of, what we expected offline. Didn't see that online. And how do we debug through these, I guess, imperatives, between what we saw offline and online? That that also took us quite a while as well.

Michael Berk [00:34:32]:
Can you give an example?

Michael Sun [00:34:34]:
Yeah. So one of the things was we saw we have different ETA and we predict both ETA and distance. And we have ETA distance metrics like ME, MAE, RMSE that we monitor offline and also online. And we when we wrote this model out into shadow, we noticed that for a distance, the metric did not reflect what we saw offline at all. So we had to dig into whether that was a data issue or a model issue or an offline parity issue. There's a variety of different variables that we need to, make sure that, is performing as we expect them to, before finding out what the, I guess, where this fix should, be in, before we, I guess, roll this out even into experimentation.

Deeksha Goyal [00:35:28]:
There were some interesting ways in which we debug that. Like, we at one point, we were like, okay. Let's just, like, send from our service. Let's always just send the same value kind of thing. Like, always sending out a distance prediction of x in in shadow, not not in production. And and we're, like, okay. Now let's trace all of our tables and see what's going on. So we had to kind of dive deep into in order to find find the issue.

Michael Berk [00:35:58]:
Got it. And, Ben, in your experience, is this common when you're rolling out from shadow to prod, or does it always work perfectly?

Ben Wilson [00:36:06]:
In my experience, again, with all the models I've ever shipped to prod back when I was doing data science work, the exception was when it went well, and that set off an alarm in my head saying I'm not seeing where it's failing. I need to dig deeper because I've I've never shipped anything into a staging environment where it works the first time, like, as expected. Maybe I'm just bad at what I do, but usually these systems are so complex that it's like, there's gonna be something that's unexpected. And if you're not if you don't get alerted to that right away, that means it's probably an insidious bug that, like, your CI isn't catching or you you're just not realizing what this impact is potentially gonna do. And then moving to the pure software side in recent years for me, I mean, we ship bugs. Everybody ships bugs. And it's an opportunity to to learn and write more robust testing infrastructure and make sure that it never happens again. But it's pretty rare for a a feature of that magnitude of what the 2 of you just explained for that to go well in sort of a that shadow mode of production deployment.

Ben Wilson [00:37:29]:
I've never seen it. Like, not even one time where it just something that takes a year to build. And even in, like, if you're to sum up the number of incremental changes and say, okay. During every incremental test that we did, these private previews, there were no bugs. I've never seen that. That would be Yeah. So abnormal that most people would everybody that I work with would would probably get into a meeting room and say, what's going on? What are we missing? And they would kick off, like, a very intensive bug bash to figure out, like, are we being stupid here? Like, let's figure this out. And I've never seen that.

Ben Wilson [00:38:14]:
Like, people gather like that, but I've seen the, hey. Everything seems to be working fine. And they just tag another person to get fresh eyes on it that doesn't have context, and they find within, like, the first 5 minutes, this API is broken. Or, hey. I'm doing this one thing. Look at this crazy stack trace. I have no idea what's going on. We're not catching this exception correctly.

Ben Wilson [00:38:36]:
So there's always something that you can find with fresh eyes.

Michael Berk [00:38:39]:
Got it. Okay. Cool. That that makes sense.

Michael Sun [00:38:42]:
Alright. I think there's one example I can probably give you about something I found we found that was very interesting. So in the shadow data, we noticed that for rides that are less than 1 minute long, our model is predicting something that's, like, 5 minutes, or something that's just not reasonable if it's, like, just around the corner, for example. And we noticed that we didn't do that. We didn't see this in the offline data because in offline, when we train or when we evaluate the model, we filter out some of the rights. And these were the type of rights that we filtered out. So as a result of that, we kinda had to build logic into our service to say, hey. Just don't predict for routes that are less than a minute long.

Michael Sun [00:39:24]:
Because the prediction is probably not right. And we don't need a model to break for a route that's very short because we kinda know that it's gonna take a minute. So that's something that's, we didn't just didn't see offline, but we saw online. I had to really dig to find out what was going on.

Michael Berk [00:39:42]:
Do you hard code those values, or you still use dynamic modeling? Like, if it's under 10 feet, just call it 0 minutes?

Michael Sun [00:39:50]:
When when we when we looked at the bug, we did some search, dynamic search, to find the what the best value to set.

Michael Berk [00:39:59]:
Got it. Very data driven of you. I like it.

Ben Wilson [00:40:03]:
So that's filtered out from training data, but just due to the sparsity. Like, hey. How many times do we have a 16th of a mile ride? And is that because somebody signed up to go 20 miles, and then they just bailed out because they didn't like the driver in in the first 200 feet and said, pull over. Get let me out.

Michael Sun [00:40:27]:
There are situations where the driver is just around the corner. So the data would be from the driver's location to that pickup. That's a very short ride. There are situations where, the driver's behind the rider, and they have to go around the block to get back. And the model doesn't learn that. Model thinks, oh, they're very close. So I should just say 0 minutes. But act in reality, the route the driver has to take is going around in a circle then come back.

Michael Sun [00:40:56]:
Maybe that takes 2 minutes. So there are situations like that where the model may not perform as well.

Ben Wilson [00:41:03]:
So the local navigation due to architectural and design of inner city traffic is what messes that up. That makes total sense. I remember being stuck on a on a corner, when using your service because of the design of Toronto City streets, in one particular area. And it just had this really weird combination of one directions and then high highway off ramp immediately at where I was trying to pick up, and the driver was sending me messages saying, can you go to the other side of the block? It's gonna take me 10 minutes to get there. And I'm looking at the map. I'm like, it's, like, 300 feet away from me. Like, how is the and then I looked at at the street signs around me, like, oh, yep. Okay.

Ben Wilson [00:41:50]:
And as I walked over, I was like, okay. Now everything says, you know, the correct ETA, and they're right there. But, yeah, that must be incredibly challenging to detect stuff like that in training data.

Michael Berk [00:42:04]:
Yeah. So we've been talking a lot about edge cases, and then a question for both of you. What excites you about ETA? It seems like you spent a lot of time in the mapping space. Do you like working through all these subject specific edge cases, or do you like building the infrastructure? Like, what what about the job and the subject matter is exciting to you?

Michael Sun [00:42:24]:
Maybe I can go first. Like you said, Michael, at the beginning of the show, that ETA just goes into so many things at Lyft. It's very fundamental to the business. It's fundamental to the user experience. And I think it's very rare for me just looking at any product to find a team that's crucial to both the user experience and the business and to have, like, so many other dependent teams, on that one single product. So I think that's that's that makes the space very dynamic. I'm still learning new things, every day, even though I've been on the team for almost 5 years. I was talking to a person from the pricing team, and they were telling me yesterday that they do they do to detect for fraud.

Michael Sun [00:43:11]:
They would send request after a write happens to to see if the ETA prediction after the write is the same as the ETA prediction during the time of the write to detect fraud for for writers that may find loopholes in the system where it always gives them $20 off or for some reason. So that's a use case that we never thought that would be used for, but that's that's what the callers are calling us for. Or during the middle of the ride when, at the by the request for a new destination or, like, a waypoint where they need to go to the grocery to pick something up, they need to predict BTAs again. So that's, like, another use case that we sort of need to handle and make sure that we have accurate predictions for. So, yeah, I guess it's just I'm just learning new things every day, so that makes it exciting and not boring.

Ben Wilson [00:44:00]:
It's gotta be really fascinating as a a design principle how your team had has designed this to say, we're behaving almost like a subcompany within a company and providing a service. It's almost like a SaaS service. You're saying, like, we don't know how people are gonna use these APIs, but we expose them, and you can hit them. I was just saying that that's a fascinating way to handle it, and it's it's a scalable way to handle it too instead of having to say, we have a monolith. Everybody has to embed their logic within the stack for all of these use cases. Good luck maintaining that code base with 30 different divergent paths through a single endpoint. Sorry, Diksha. Go ahead.

Ben Wilson [00:44:44]:
What you were saying?

Deeksha Goyal [00:44:45]:
Oh, I was just gonna say, I think another another nice piece to it just to to add to your point actually. So there's all these you know, we have to think about all these different services, and that leads to sort of this, like, really high paced environment in terms of we need to be thinking on our feet. We need to be quick debuggers. And I I think I just really like having like, practicing that skill set. And there's also just, like, you know, everyone I know, like, uses Lyft in some way or another. Right? So it's just, like, whatever we ship, it's like, okay. This is gonna touch my friends, my family. So there's an element of I don't wanna mess this up because these are people I care about.

Deeksha Goyal [00:45:35]:
So I don't want them to have a bad experience, and it's it's it's cool having having, the ability to sort of to work on a problem that that does that.

Michael Berk [00:45:45]:
Yeah. You sort of take more pride in your work because you know that your grandma is gonna be taking a lift, and you want her to get there on time and safely. So Mhmm. That definitely resonates. Okay. Cool. And, Ben, I know you have to drop, but I I think I have a couple more questions. And one of those questions is, basically, how did you get to where you guys are from a skills perspective? So you hold very prestigious positions at a prestigious organization, And, theoretically, you are good at your job and know what you're doing.

Michael Berk [00:46:17]:
So how did you get there?

Deeksha Goyal [00:46:21]:
How did I get here? I think there's just a lot of, like, constant learning for me. So, and and staying curious. So, like, outside of work, taking classes off Coursera just to sort of stay up to date. But then at work also just, like, asking asking good questions and trying to find out which team is of interest to me. So, yeah, I think I think it's just a matter of, like, staying curious and staying connected with people within your company or with outside, like, having a strong network. That's been really helpful for me to figure out, like find work that actually really excites me.

Michael Berk [00:47:08]:
Got it. So follow your passion. Is that the sort of

Deeksha Goyal [00:47:11]:
the one? I I guess follow your passion, but also, like, I think for me, it's, like, the people I work with matters a lot and, like, feeling like I can learn from them and that we have this sense of, like, we're all really interested in this problem. So it's like, I think I would struggle with working on a problem I'm passionate about, but with people who are not passionate about it themselves. So finding an environment where other folks are also excited about what they're working in.

Michael Berk [00:47:39]:
And it's so if you're so people focused, did you have a hit in terms of productivity and job enjoyment during COVID?

Deeksha Goyal [00:47:47]:
Definitely. It was it was tough. I I mean, when we were in the office before COVID, it was it was so fun. The whole office is pink, and, we would just yeah. We would go in and, like, it just really felt like, a fun environment. So COVID was definitely was definitely tough, but, I think I think folks tried to create an environment that is, like, decent, like, decently social, in some way or another online. And then and now we're now we're back in the office. So so folks go in.

Deeksha Goyal [00:48:25]:
It's not the same in terms of frequency, but people people definitely go in.

Michael Berk [00:48:31]:
Cool. K. What about you, Michael? What excites you at at Lyft? And and, well, I mean, more specifically, like, it sounds like Disha has followed passion to get good at what she does. But, have you followed that same sort of curiosity, or have you grinded through and read textbooks and practiced on on leetcode? Like, what's your strategy?

Michael Sun [00:48:57]:
Definitely practice on leetcode to get here. That's for sure. Textbook in school as well. Although I've never been good on, like, reading textbooks, that's just fine to me. I'd I'd rather doing problems, just get more hands on experience. But before I found Lyft, my requirements for the next job was, like, I wanted to work on a product that people use and a team that works on the product that people can see. And I also work on, ML automation or infrastructure related work. Those were the requirements that I had before I applied the job.

Michael Sun [00:49:32]:
And then when I found ETA, I was, like, super happy that it met everything that I was looking for. And I think I alluded to this earlier as well. It's, that, I've been here 5 years, and every year, there's a different challenge. There's new lessons to be learned. There's new people to work with, like Deepa said. And there's just the space is so dynamic, and we focus on, a variety of different things. And we always need for people to work on these challenges. And like like I said earlier, we we're also finding out new things every day.

Michael Sun [00:50:05]:
So how do we make sure that we address for these new use cases in time? So the these are things that keep me challenged and keep me engaged. And I think as a result of that, I just, do work because I'm curious and I wanna learn. So very similar to Deepgram's answers, I'd say.

Michael Berk [00:50:25]:
Cool. And so both of you guys discussed the excitement of the field. Do you think the future of ETA is going to be algorithm based or improving data quality? Like, what is what are the frontiers of ETA over the next 5 years that will be hopefully surpassed?

Deeksha Goyal [00:50:43]:
That's a good question. I would say it's, like, across everything, like, end modeling, data and everything. In terms of modeling, we have some exciting projects we're working on in terms of, understanding, like, p 95, p 99 cases. Like, how do we improve the sort of, like, outlier experience? And that's something that we've been pretty focused on lately. And, also, looking into sort of industry papers on what other folks are doing for ETA, and making sure that we're also somewhere close to to to to what they're building. So, yeah, those are those are the interests in terms of modeling. Michael, you wanna take the the other ones?

Michael Sun [00:51:36]:
Sure. The other ones, I would say, are also making sure that we have a good story around all of the edge cases. So one thing I mentioned at the beginning was, like, if a user says I wanna get to the airport by 10 o'clock, how many times can we actually guarantee that they get there maybe even 10 minutes before, or, you know, making sure at least that they're not late? And how do we, make our ETA as efficient enough so that drivers can always be assigned to the the passengers? Or passengers can always get a ride, within, for for example, 5 minutes or whatever, promises we make. So I think from the engineering side, it's addressing for these, you know, airports use case, schedule rights use case, and the other internal use cases I mentioned for fraud detection, that kinda stuff. So, having a good story around all those different use cases. Yeah.

Michael Berk [00:52:29]:
Okay. Notice audience that LLMs are not in there anywhere, unfortunately. But yeah. Okay. Cool. That makes sense. Do you have any advice, for people who are potentially in the ML space, and if they wanna get into more exciting roles, how they would go about doing that and, like, how they go about building the technical acumen that would allow them to do that?

Michael Sun [00:52:57]:
Take classes, I would say. I, was taking some classes online as well to just understand the model space. And, technology is constantly changing, so keeping up to date to the new technology and new libraries, LLM is something that maybe we'll look into for ETA prediction. Who knows? And to get into the ML space, I'd say also, find a team that can support that growth is very important. When I came into Lyft, I didn't have any previous ML experience besides the courses I took undergrad, but I had enough trust to work on starting off at the very beginning of some automation for ML, infrastructure and eventually got more more familiar with modeling size. So over time, I just pick up more, like, I guess, expertise in the ML space. But to start off, just finding a manager or a mentor that really supports you on that on that journey is is really important, I would say.

Deeksha Goyal [00:54:01]:
Yeah. I also think I was pretty surprised in industry by how much, folks will, like, trust you As long as you're motivated and really want to solve something, even if you don't have the experience for it, people just trust you because you really wanna do something, and they trust that you'll you'll do a good job. Sometimes more maybe even more than you might trust yourself. But I think finding people like that who will, like, are your allies and that they'll just kind of allow you to explore new spaces or I I think we have we have a lot of that lift. We have managers who are willing to it's actually, like, part of our sort of career development is we have these, like, individual development, like, programs with our managers where it's like, okay. Like, what are my growth areas? It could be like, I wanna learn about reinforcement learning, and then your manager helps and find you a project in that space if possible. So having that space where you can you can actually do that is really nice.

Michael Berk [00:55:05]:
That's super cool. And do you think the organizational rationale is for retention, or is it for seeing the ROI on skill sets and having you grow into a more seasoned developer or both? Neither?

Deeksha Goyal [00:55:16]:
I'd say it's a combination of the 2. I think the company really benefits from having folks stay because there's just, like, so much so much knowledge you gain as you stay at the company. But, also, it's just about, like, as a result of, like, having happy employees, and you're able to actually, like, hire other happy employees too. So it's just like it's one of those things where it's like you don't really see a direct if you create this program, you wouldn't really see a direct ROI, but I think implicitly, it is there.

Michael Berk [00:55:49]:
So one final question for both of you. What is the worst thing about the ETA platform at Lyft, and what is the best thing? And to add a little flavor, it can be about your interaction with it. It can be about the user's interaction with it. It can be anything. Like, what what do you not like about it, and what do you like about it? What are you maybe proud of? What are you maybe embarrassed about?

Deeksha Goyal [00:56:12]:
I'd say the worst thing is that it's it can be difficult to debug things at times. And so we we have a lot of effort around debuggability this year. We've had it last year. I think we have it every year. We have a whole, like, project around debuggability. But it's still never really, like, an ending problem for us because we like I mentioned earlier, for each rider request, there's so many different ETAs. And so another team comes to us and they're like, this looks really bad. What is what why did this why is this bad? It's like, okay.

Deeksha Goyal [00:56:48]:
We have to, like, dive deep into our tables. Be like, is it a map data issue? Is it an issue with the routing engine? Is it an issue with our ETM model? So that's always been a difficult process that we are trying to fix in different ways. Best thing is, I think the scale of the problem. I I really like that and how we have to keep, a pretty, like, good overview of, like, who our callers are and be able to make adjustments to it despite being in this, like, difficult position where it's like, good. We wanna improve accuracy, but we don't wanna degrade business metrics, and we want our partner teams to be happy. So it's like I actually like, that could be a con, but I actually really like that because it it adds more challenge to it. And, yeah. So it just makes it more exciting.

Michael Berk [00:57:43]:
Got it. Couple questions on that. In terms of debugability, so a core issue is that there's many disparate systems and each system is incredibly complex. And then on top of that, there's many different routes which depend on many different systems. So it sounds like one potential solution would be, like, testing each component super thoroughly so that, you know, inputs and outputs are not going to produce something that, is problematic? And then also maybe, like, complex stack traces or something. But, like, what tangibly are you guys doing to make debug ability more possible?

Deeksha Goyal [00:58:20]:
We log everything. Everything that

Michael Berk [00:58:22]:
You guys just, like, read every, like,

Deeksha Goyal [00:58:24]:
time logs. That's true. We have a ton of logs. And then it's like, okay. We have all these logs. Like, how do we organize these logs? How do we we've created, like, visual tools to debug these logs. Like, so yeah. I think though the first step is just logging everything, but then the next step is making sense of all the logs.

Deeksha Goyal [00:58:42]:
And, of course, like you mentioned, it's like we need to be careful with each update we have across the stack, but we can't really do everything. We can't, like, stop all work and mapping while one model is being updated. Like, we have to be able to ship things in parallel, or else we're just really gonna slow down as an organization. So that's why logging is really the the the best way for us.

Michael Berk [00:59:06]:
Got it. And are there, like, logging standards that are ridiculous and super complex, or is it pretty simple?

Michael Sun [00:59:13]:
Like, is

Michael Berk [00:59:13]:
the user interface problematic, do you think?

Deeksha Goyal [00:59:16]:
I think it's kind of there are some guidelines, but for the most part, we have free rein on, like, how we wanna log things, which is really nice. So, like, we'll often completely change it or rewrite our tables once there's, like, new new business needs.

Michael Berk [00:59:33]:
Got it. Okay. Cool. Thank you. Michael, what about you?

Michael Sun [00:59:37]:
Oh, we have run into issues where we log too much, and we had to cut down on the logging. That's one of the complaints from, our infra team. So that's something that we have to juggle with as well.

Michael Berk [00:59:47]:
Like a cloud cost thing or a late like, a query latency thing?

Michael Sun [00:59:52]:
Like, a streaming platform cannot handle the amount of things we

Ben Wilson [00:59:56]:
No. Kind of thing. Okay. Yeah.

Michael Sun [01:00:01]:
Right. The best and worst. I would say, I could probably give one answer for both, and it's the uncertainty. So the uncertainty in the sense that we just never know which route driver's gonna take, and we can always do our best guess. We also would never know which driver is going to be assigned to the rider at the very beginning of our prediction, and we can only do our best estimate. And over time, we gain more and more confidence, though. So, after the driver is assigned to the rider, we also we we sort of get rid of that variable. And once the driver starts on the route, then they more or less adhere to that, to that route.

Michael Sun [01:00:38]:
So we also gain more confidence. So it's a matter of, like, how do we address these uncertainties at different phases while making sure ETAs are sort of consistent. Right? Like, you can imagine if we don't know the driver, we can say, oh, it's it's 10 minutes. But once we know the driver, it could be come down to 5 minutes. How do we make sure that user experience isn't, compensated or is compensated correctly for, like, this gap of uncertainty and this jump between 2 different phases in that. So that's a constant problem that we're trying to solve. But I think it also makes it exciting because it seems like a never ending challenge in this space. And there's always new ways for us to get around that.

Michael Sun [01:01:23]:
One of the things that we have done in the past is with the introduction of LiftMap. We can control our routes. So we can better control our ETAs, with those type of routes. And, our ETA predictions become more accurate with being able to control these these routes suggestions. So yeah. Got it.

Michael Berk [01:01:43]:
So, yeah, if the prediction is different from the route, just change the route.

Michael Sun [01:01:52]:
That's a good point. Yeah. Maybe.

Michael Berk [01:01:56]:
That way, actually like, double clicking into that, that's an interesting thing to account for in terms of assessing model quality. How do you guys think about it?

Michael Sun [01:02:06]:
Assessing model quality. So we have like, Diksha mentioned, we have these metrics that we track whether there's a defective ride in the sense of, is our ETA prediction off by more than 5 minutes, for example, and how good is our average prediction? So, we have metrics like these to help, guide our, I guess, research and, our engineering focus to make sure that we're focusing on the right components of the system, to solve and to improve these metrics over time.

Michael Berk [01:02:44]:
Yeah. But the question was, if the prediction has structured to it and fundamentally could change, based on, like, new routing strategies. Do you just quarantine that as, like, a weird dataset section that needs to be handled differently? Or how do you how do you handle that?

Michael Sun [01:03:03]:
So are you saying that there's routes that, a driver can possibly take that we just couldn't predict? How does ETA address for those routes that the driver could take that we just didn't predict for?

Michael Berk [01:03:15]:
Yeah. I guess, fundamentally, you have a prediction for a route, and then that route will dynamically change. So it's not a a static prediction. Is that an issue?

Michael Sun [01:03:30]:
I guess there's 2 things. The one is we have a route, and the driver just goes along this route.

Michael Berk [01:03:35]:
Right.

Michael Sun [01:03:35]:
And even if they do, that ETA can still change because the traffic pattern changes, etcetera. And so we have different layers to address for this. There's the traffic layer. There's also the ETA model layer and the real time ETA correction layer. The other the other, I guess, issue is the driver could be going on this route, and then all of a sudden make a turn where they, we didn't say to make a turn. And in those cases, we just have to recompute the route, based on where we think they might be going and readjust the ETA based on that on that.

Michael Berk [01:04:07]:
Awesome. So I think we're at about time. I will summarize. Lots of really cool things in this in this discussion. Honestly, a relisten is probably your best bet. But, in terms of model graduation and validation, typically, at Lyft, they run-in the shadow phase or shadow deployment, and then they kick off an experiment where they look at complex, metrics that are sort of indicative of this two sided marketplace. And then the business decides the trade off between potentially, combative or or, at odds metrics, and this is at an organization level. From an infrastructure perspective, Lyft uses AWS with Kubernetes and auto scaling, and then they have this really cool layering of a base map.

Michael Berk [01:04:50]:
And on top of that base map, there's drivers or traffic. And then on top of that, there's possible routes, and this is a in memory graph model. And then finally, the ETA comes in with a real time correction layer. On the career tips side, constantly learn and stay curious. You should focus on the people, whether it's just for brainstorming or camaraderie or just, learning about new things. And it also makes your job more fun. And then when looking for jobs, it's important to be clear on your priorities and expectations, know what are deal breakers. And especially if you're early, at least in my opinion, sometimes just jump in and see what you like.

Michael Berk [01:05:25]:
It's useful to gather those data points. And then, finally, on the big stuff side, one of the issues, for the Lyft infrastructure is handling how complex that infrastructure is and debugging it. So, their strategy is logging everything and creating, sort of tools on top of those logs you can easily parse them. And then another challenge is sort of handling the structured uncertainty in the time series of a ride. And so the way that you guys are handling that is introduce introducing new rules at different phases of the journey and being dynamic. But ETA is a a fast moving space and a challenging space, and I'm sure there's gonna be lots of cool things to come. So if people wanna learn about those cool things to come, Diksha and Michael, where should they go?

Deeksha Goyal [01:06:13]:
There's a lot of blog posts on eng. Onlift.net. We have blog posts there. We also have blog posts on Medium. So those are your best bet.

Michael Berk [01:06:24]:
Cool. And then if they wanna get in contact with you or potentially a hiring page on Lyft, where should they go?

Deeksha Goyal [01:06:32]:
I'll I think it's on lift.com, but they're also welcome to reach out to either Michael or me.

Michael Sun [01:06:42]:
Yeah. Lift.com/couriers. I'll plug that. We are hiring. The entire mapping work is hiring. So definitely feel free to reach out to find out more information or just apply online on LinkedIn. Yeah.

Michael Berk [01:06:54]:
Cool. Well, until next time. It's been Michael Burke, and have a good day, everyone.