Michael Berk and Ben Wilson from Databricks are joined by Brooke Wenig, who has a fascinating background in distributed machine learning. Today’s conversation dives deep into the intersection of AI, environmental science, and career transitions. They explore how individuals like Michael transformed their careers from environmental science to AI, leveraging existing expertise in innovative ways. Ben shares insights on leaping from non-technical roles to data science by embracing automation with Python and machine learning.

We tackle the critical shift in roles, the balance between education and hands-on experience, and the growing disparity between academia and industry. Brooke brings valuable perspectives on project scoping, from aligning success criteria to ensuring real-world value. The discussion revolves around augmenting existing roles with AI, common pitfalls, and transitioning proofs of concept to production.

They also explore the practical applications of language models, the debate over open versus closed source models, and the future of AI in various industries. With a focus on collaboration, the traits of top data scientists, and the implications of integrating AI into non-tech fields, this episode is packed with insights and tips for anyone looking to navigate the exciting world of AI and machine learning.

Join them as they delve into these topics and more, discussing the evolving landscape of AI and how it's shaping careers and industries alike.

Socials

Michael Berk [00:00:05]:
Welcome back to another episode of Adventures in Machine Learning. I'm one of your hosts, Michael Burke, and I do data engineering and machine learning at Databricks. And today, I'm joined by my wonderful and lovely cohost.

Ben Wilson [00:00:15]:
Ben Wilson. I do quarterly planning at Databricks.

Michael Berk [00:00:19]:
And today, we have a very special guest. Her name is Brooke, and she is one of the faces of Databricks. She got her undergraduate in Chinese and then her master's in distributed ML from UCLA. And then right after graduating school, she joined Databricks as an ML solutions consultant and is currently the director of Databricks' ML practice. Now in her free time, she writes books about Spark, cohosts the Databrew podcast, which we just learned doesn't exist anymore, teaches courses, and also speaks Mandarin. And so Brooke, why did you switch from Chinese to distributed ML?

Brooke Wenig [00:00:58]:
So it actually wasn't a switch. I was actually studying computer science in my undergraduate, but I studied abroad in China, not through UCLA. It was a scholarship from the Chinese government, and when I came back, UCLA said we wouldn't take any of your credits. And because of my GPA, I had guaranteed admittance to the master's program. I'd already taken all the computer science and the math classes at that point. I just didn't take any of the physics or electrical engineering courses. And one of the best lessons my dad ever taught me is the answer is always no unless you ask. So I asked, like, hey.

Brooke Wenig [00:01:32]:
I I obviously know my computer science skills, my math skills. Like, would you still do guaranteed admittance for me even if I don't technically have a computer science degree? I can get my Chinese degree, though, in a quarter. And they said, yeah. No problem. And so I could effectively do 2 undergraduates in 5 years or one undergraduate with masters in 5 years. That was kind of a no brainer.

Michael Berk [00:01:52]:
That's super interesting. Alright. 1st, going back to your dad's comment, do you think it's better to ask for permission or just do it and ask for forgiveness? I mean, in this case, you have to ask to enter a master's program probably.

Brooke Wenig [00:02:07]:
Yeah. I I think that guidance is a little bit different. The answer's always no unless you ask of, like, there is a policy and there's no way you can get around it unless you are to ask. So don't always treat no at face value. That's different than asking for forgiveness rather than for permission because that means there is a way around it.

Michael Berk [00:02:24]:
Yeah. Okay. Good clarification.

Ben Wilson [00:02:26]:
Or no policy.

Brooke Wenig [00:02:28]:
Or no policy. Exactly. Yeah. I just want a quick clarification. Dave is on hiatus. It will return. Yes. Timeline TBD.

Michael Berk [00:02:37]:
Right. Excuse me. It it is coming back very, very soon. So, today, we sort of were chatting a little bit before we kicked off the recording about what we wanted to talk about. We were initially gonna talk about managing and the beauty of this podcast is it's very free form. So we can essentially see where the conversation goes and talk about whatever we want. And so what we're gonna kick it off with is the concept of scoping projects. I know this is one of Ben's favorite things to discuss because it's so hard and so important.

Michael Berk [00:03:06]:
And so, Brooke, if you're put into this position of scoping a project, what are the things you look for from the outset? And, yeah, just how do you think about it?

Brooke Wenig [00:03:16]:
So the first thing is to validate whether or not there is a use case. So whenever account teams come to me saying like, hey. We need a scope of engagement, I actually have a form of, like, these are the things you need to tell me before I walk into a customer conversation. What is the use case? Are there any known escalations? Are there any timelines that I need to be aware of? And then I typically like to have a quick 15 minute internal call just to make sure I get as much of the basic information from them so I can focus the time with the customer on, addressing any questions that I have at that point and just regurgitating the information, just making sure we're all on the same page. In terms of scoping, customers are all over the place right now. Everybody wants to say they want to do Gen AI, and a lot of them don't understand them now. And it's like the first step is to take a step back of, like, what is the business problem you're trying to achieve? And let's align on success criteria. Right now, we're getting tons of requests of, like, I want to, build a model that can beat GPT 4.

Brooke Wenig [00:04:13]:
Can you help us with that? And we'll take a step back and say, like, alright. What are the problems you're facing? Is it that you want to have control over the model? Are you having issues with cost? Is it that you want to fine tune the model? Like, why do you want to do this? So take a step back, figure out what is the ultimate driver because, like I said, so many people just wanna say, we're doing our thing. We're building our own, like, company a, b, c, g, p, t. Too many letters there. But, yeah, figuring out what is the actual business problem they're trying to solve, why they want to solve it. And then, oftentimes, there's some politics apply, and they will not outright tell you what those politics are. And so that's where having that information from the account team is super helpful, just so you can understand, like, what is their incentive? What is their motivation? Sometimes the motivation is just like, hey. We have extra budget at the end of the year, and we want to experiment with something.

Brooke Wenig [00:05:03]:
It's not actually mission critical. Just they have budget, and they want to see if it works. Other times, it's our team is going to get dissolved if we don't show value by the end of the year. So it's always helpful to figure out what stage of a customer, you're walking into.

Michael Berk [00:05:19]:
What is your thoughts on whether LLM based ML is is it worth the hype, basically? Because I know that Databricks is investing super heavily. There's a lot of use cases. There's a lot of revenue. But is it sort of an Internet where there's gonna be a bubble that will burst, or is it actually gonna provide value from the outset?

Brooke Wenig [00:05:38]:
I think it's just a question of what stage in the hype cycle is it. I think there is immense value. There's also immense risk. And so, for example, like, the what the use cases that we're seeing the most success in are augmenting customer call specialists. So many customers, rightfully so, are hesitant to have anything customer facing. Like, we've all seen the example like the Chevrolet dealership, like, recommending Ford F 1 fifties. But instead, what we're seeing for, like, cost customer call specialists is instead of starting off with something to completely replace the human, call specialist, it's recommending the next best, response for that human specialist. And so, with one of our customers, we had found an overall 20% reduction in time to resolve a customer query, as well as 15% increased, upsell opportunities and, overall, a much happier customer.

Brooke Wenig [00:06:27]:
So, like, that's a great example of you're able to save costs and you're able to make your customers more happy, and you're actually able to sell more with that increased upsell opportunity.

Ben Wilson [00:06:38]:
Yeah. I mean, from the few customers that I talk to now, it's usually evaluating our API designs. The ones that I see is that seem to be things that people are pushing to production are exactly that, as well. It's like a lot I think a lot of fear that people talk about is like, oh, it's it's gonna replace our job or it's gonna, you know, take over this this thing, and you hear that from, like, junior software engineers sometimes. Like, somebody's gonna, you know, create one of these tools that's gonna write JavaScript for me, and then I'm gonna be out of a job. It's like, nah. Like, we all use these things as like, the successful application of these is an augmentation, and it's information retrieval at its core. Like, how can I get to the correct information as fast as possible without having to, you know, sift through tons of potentially irrelevant information?

Brooke Wenig [00:07:33]:
Yeah. I I'm not a fan of the job replacement argument because it what is it? Like, the 1800 to 98% of humans were farmers. I was like, oh, no. We're automating agriculture. Like, we have these great new tools. Like, people aren't gonna have jobs. And, like, that's not the case. 98% of people don't, are not unemployed right now.

Brooke Wenig [00:07:49]:
And so, like, sure, some jobs will be eliminated, and or greatly reduced, but there will be new jobs that are being created. And so I think that's the side that people are not, realizing is just how many new jobs this is creating as well.

Ben Wilson [00:08:04]:
Yeah. Something that I heard of that blew my mind recently are all the companies that are starting up for the gig market for Gen AI,

Michael Berk [00:08:14]:
where

Ben Wilson [00:08:15]:
companies, and I'm not gonna name names of who they are, but most of the big ones that are building the next gen or, you know, the cutting edge state of the art stuff, you can't do validation testing yourself. Like, you do your first training iteration, Data design team's gonna be doing that, or a QA team internally is gonna be doing that. Then you go to fine tuning, and then you need to find out, are there risks associated with usage of this thing that we just trained from scratch? You're now your internal teams are not gonna have enough time to do that. It's just too much data. So you you create a bunch of training sets or validation sets and then hire a company that that basically does mechanical Turk operations. It's like, hey. Let's get 10,000 people around the world who are all either experts or generalists in these topics and have them tell us if this is bad Mhmm. And then get the results.

Ben Wilson [00:09:13]:
And that it's just a it's a whole new economy that's starting up. I didn't know how many, like, I know a couple of people that are doing that on the side right now. And look at the number of jobs that are coming in each day from these services. And it's in the tens of thousands of projects to be done, and they're burned through within 24 hours.

Michael Berk [00:09:35]:
Yeah. Wait. Okay. So another question for, actually, both of you. What do you think of the closed source versus open source model debate? Like, what what's is there gonna be a winner? Is it a time and a place for each? What are your guys' thoughts?

Brooke Wenig [00:09:54]:
Go ahead first, Ben.

Ben Wilson [00:09:56]:
I wanted to defer to you first because I have a bunch of bias here. So I think it extends to the same principle of just general software. There's benefits and risks to both. So in a closed source system, you might not know what's going on or how it's made or you can't see what's going on. But you're paying somebody to maintain that for you, and you have a a finger to point to and a vice to tighten when things aren't going right. So that company is that's maintaining that, as a service is very motivated to correct that action for you or correct that fault with it or build a feature for you for something. And then an open source is kind of opened up, like, hey. You want something? Build it and give it to the rest of the world for everybody to use.

Ben Wilson [00:11:00]:
It's up to open source maintainers to control whether that should be included, whether that should be released, and whether it's it's safe and useful. Or you can just create your own projects and push it out there, and maybe people will use it. Maybe it'll it'll pick up traction. But we're talking about something that costs so much money to build. The difference between closed source and open source is who's building it, why are they building it, and how much money are they putting behind it. Is that right? Go ahead.

Brooke Wenig [00:11:40]:
So I I think we are talking about closed versus open models because we're not talking about open source. A lot of these models, like, they're not releasing the training code. They're not releasing the training data. And if you follow the LAMA 2 style license, they are restricting who can use these models. Like, a core tenet of open source is anybody can use this. But with the LAMA 2 style license, it's, oh, if you have more than 700,000,000 monthly active users, you you are not allowed to use unless you've, received explicit permission. Yay, yay. And so I think it's really a question of closed versus open models.

Brooke Wenig [00:12:14]:
Ben, to your point you were talking about earlier in terms of having a finger to point, how would that differ if someone is providing an open an open model as a REST endpoint, for example?

Ben Wilson [00:12:25]:
If a company is offering an open model as a REST endpoint.

Brooke Wenig [00:12:30]:
Yeah. Like, Databricks does. Amazon does.

Ben Wilson [00:12:35]:
I think that becomes much more of if we're offering a paid service with that, we're highly motivated to respond to requests or if issues come up because we're making money off of that thing. But in the case of Meta, they launched they they released this model and say, hey. You can use this for whatever you want. You can fine tune it, but we're not gonna show you how, you know, how the spaghetti is made here. They're not they're not motivated to fix anything with that, and they look at their licensing agreement. And there's a lot of caveats in that of like, hey. We're not guaranteeing that this isn't gonna, you know, ins you know, give you instructions on how to build homemade bombs. You know? Mhmm.

Ben Wilson [00:13:22]:
Use it use at your own caution. I think the the motivation for the open model process, for ensuring that it's of a high quality, it's more reputation based than money based.

Brooke Wenig [00:13:42]:
My personal thought on the open versus closed model is it's expensive, to be lever to be using these massive models, if you're actually trying to put use cases into production. And for a lot of customers, they care about low latency, high throughput, low cost. And so we are starting to see a trend of customers wanting to build much, much smaller models. So, like, I was chatting to the customer yesterday. Like, they want to build a series of models that are less than 1,000,000,000 parameters, actually, for a highly specialized use case, because they need they need it in terms of the low latency. And so that's when we're like, you're not going to get a closed source model, or a proprietary I'll just call it proprietary model. You're not going to get a proprietary model off the shelf that's going to be able to perform your specialized task, at the size that you're looking for. And so that's one of the divides I see.

Brooke Wenig [00:14:34]:
I'm also seeing a lot of financial services companies in particular being interested in, pre training models from scratch entirely. They don't even wanna use open source models because they are concerned about, the data that went into it and making sure that, they have high fidelity inform they have high fidelity data, and they know all of the data that went into it as part of regulatory requirements. So for most of our customers, starting off with the REST endpoint, whether or not it's a proprietary or an open model, Like, that that's the part that makes sense to start with. Like, demonstrate proof of concept, proof of value, benchmark it, and then figure out what are your gaps. Is it a gap on quality? Is it a gap on performance metrics? And then figure out which one makes sense for you. I think there's a place for both. I don't think they're mutually exclusive.

Michael Berk [00:15:27]:
Got it. So sort of a function of use case maturity, organization maturity, and then, like, regulatory requirements.

Brooke Wenig [00:15:36]:
Yeah.

Michael Berk [00:15:37]:
Cool. Yeah. So that's super interesting because, Databricks has been investing heavily in the, like, the open source side or the open model side. But I think, Brooke, your prior question about it it seemed like you were hinting at is there really a difference between a closed and an open model if you're not open sourcing the training code and the training data? Because Databricks theoretically are let's not talk about Databricks. The owner of the organization, or owner of the model is theoretically still responsible for its performance. Right? So it's almost as if it's closed source. Is that what you were hinting at?

Brooke Wenig [00:16:14]:
What do you mean by the owner of the model?

Michael Berk [00:16:16]:
Like, the publishing organization that created the model. Given it's not really open to complete retraining or complete modification, you can fine tune it. It seems like it's still on the onus of the organizing party to give even though there aren't official SLAs, it's the onus of the organization to maintain it, ensure quality, theoretically ensure you don't give building BOM instructions. Is that what you were hinting at, or am I off base?

Brooke Wenig [00:16:45]:
I wasn't quite hinting there, but with open with open models, you can grab them at any point in time, and you know they're not going to change. Like, you pulled into your organization, you fine tune it. It's not gonna change. Like, there have been plenty of studies that have shown that for the same endpoint, the GPT 3.5 and 4 models changed answers over time with having all of the other, factor set. Got it. And so, like, that that's one of the challenges if you don't own the model and the infrastructure of the underlying model could be changing, and you don't think it should be changing. So, like, you run all of your tests and, like, okay, great, it passes. But now at the same endpoint that you've been hitting, something gives you certain different answers.

Brooke Wenig [00:17:26]:
Like, how do you control for that? And so there is an element of control as well that you get from the open models that you don't get from any proprietary models because it's up to the provider ultimately.

Ben Wilson [00:17:38]:
Yeah. This is almost like a a Gen AI specific issue when we're talking about model as a service Mhmm. Sort of infrastructure because it's it really is an economy of scale when you talk about, like, what are the solutions? Or if, Brooke, if if you and I were gonna create a company that was offering XGBoost models that we would retrain for these specific use cases, we could version lock those, retrain them every day, and host 10,000 of them on a suite of rest endpoints, and we would probably be paying maybe $50,000 a month in cloud computing costs for hosting that because they're they're super cheap. They're they're lightweight. We don't have special infrastructure we can run those on some optimized CPU or something and probably be fine. When we talk about Gen AIM model, that hardware is stupidly expensive to run these things. When you talk about the infrastructure required to respond to the level like, the volumes that OpenAI is dealing with with how many requests are coming in per second to their, you know, their mainframe. You need a very bespoke implementation where you can only host so many of those versions of these models until you're like, oh, we need we need 5 more data centers.

Ben Wilson [00:19:00]:
Each of them are $40,000,000,000 a piece. So you it starts to become economically infeasible to to, like, version lock and say, well, I want the version from 7 weeks ago, and I'm gonna use that for the next 6 months.

Brooke Wenig [00:19:13]:
Mhmm.

Ben Wilson [00:19:14]:
Opening, I can't do that.

Brooke Wenig [00:19:17]:
Nor would that you

Michael Berk [00:19:18]:
may be

Brooke Wenig [00:19:19]:
And and even more so, like, yes, it's all expensive, but there's just a global scarcity of resources as well. So even if you want to, you you wouldn't be able to acquire the GPUs necessary.

Michael Berk [00:19:33]:
Right. Okay. So, Brooke, shifting back to the scoping conversation, what is the biggest or what are some things that make projects fail, from the scoping phase?

Brooke Wenig [00:19:45]:
Not establishing success criteria. So that's where, like, customers are saying, like, we want GPT 4 quality models. We will instead reframe it as, alright. We are working off of an open model. Like, we'll figure out which one is most suitable for your use case. Our guarantee will be if we fine tune that model, it will outperform the base model that we fine tune for your domain. Because customers, like, they say, yes, we want to fine tune it on our dataset because we want to perform on that domain, but then they want to evaluate on, like, hello, swag or these other, open benchmarks that have nothing to do with their domain. And it's like one of the open problems right now is that there are an industry standard benchmarks for, like, HLS, for FINS, for financial services that everybody agrees on and uses.

Brooke Wenig [00:20:29]:
And so that's what makes the evaluation part so hard, because they'll tell you, like, hey, we've got the data for fine tuning. But they won't necessarily have, like, what are these golden standard, responses that we're looking for for the evaluation. And so that's where it gets really tricky in terms of establishing success. Like, alright. We fine tune the model. Is it a success that that completed, or is it a success that it actually performed well on this evaluation dataset? And so, like, making sure everybody's in alignment and just educating the team as well because, like, this is new for most organizations. They and this is where just really working with the business side and, like, they love actually being educated as part of this. Like, be at the cutting edge, if you will.

Brooke Wenig [00:21:10]:
Yeah. It's it's really establishing success criteria and success metrics from the beginning.

Ben Wilson [00:21:17]:
Do have you found that over the years since the hype started with Genai that the people that you're talking to who are now doing this stuff has shifted at all?

Brooke Wenig [00:21:29]:
We're talking more to, lines of business rather than to, like, data science teams, actually. And so that's what's really interesting. Like, I have this business problem. Like, I am a marketing company, and this is my problem. Or I am, like, the CMO, and this is the problem that I need to solve versus working with the with the data science team.

Ben Wilson [00:21:54]:
So, yeah,

Brooke Wenig [00:21:54]:
it's definitely shifted who we're having the conversations with. We're definitely having conversations more with the with the executive team as well too because they're providing board updates. The board wants to know what their strategy is, as opposed to, like, let me help you build out your MLOps framework for your data science team to help you productionize models more quickly.

Ben Wilson [00:22:11]:
Yeah. We're we're seeing the same thing on the MLflow open source side. The people that we're interacting with now, it's I don't know. We just noticed that over the last couple of months, we're like, we're getting a lot more questions from people that like, this is definitely not a data scientist asking this question. And you look them up on LinkedIn. You're like, yeah. It's a it's like a staff software engineer at this company. They build apps.

Ben Wilson [00:22:38]:
And now they're they're using MOflow, and they're asking, like, why is this designed this way, or why do I have to specify these things? Like, this isn't a model. This is this is effectively a script that I'm defining. I'm using Lang chain. Like, why do I have to do this stuff? And it's just interesting to me how that would shift now that, you know, if we if we talk about people that are writing code in industry, 5 years ago, people writing data science related stuff, building models, training models, you know, putting them out there for for usage. It's probably less than 5% of people touching keyboards. And now with Gen AI craze, we're now opening this up to 90 something percent of people that are titled as, you know, computer science focused in a company. Do you think that that's gonna continue into the future, and where do you think that's gonna land, say, 5 years from now?

Brooke Wenig [00:23:40]:
Alright. I'm not gonna make any prediction 5 years out. But I think that was partly born out of necessity. There's a shortage of data scientists. We've known that for years even before the gen AI craze. But separately, to build a RAG application, you don't need to be a data scientist to do that. For fine tuning and pretraining, you absolutely do, like, understanding, like, what epochs are, learning rates, like, hyperparameters, all of that. But for just putting together a basic application, like, yeah, that's something a software engineer could totally do.

Brooke Wenig [00:24:08]:
Just take some courses and then try building it. You don't need to understand the inner workings of deep learning to be able together a RAG application. So 5 years out, I think everybody's going to be more Gen AI literate. That that's my only prediction for the future, Ben.

Michael Berk [00:24:27]:
Question for both of you. Just from personal preference, what type of stakeholder do you like talking to? Is it the software engineer, the ML engineer, the CTO, the CMO? What's your favorite discussion to have? Alright.

Brooke Wenig [00:24:41]:
I'll go first this time then. CTO. They are the ones that can call the shots. They have the budget. They are technical. So those are generally my, favorite folks because they are the executive stakeholder, for whatever that project may be. So, like, even if the CMO wants to do a project, the CTO is still going to be aware of it. So that that's my preference.

Brooke Wenig [00:25:07]:
Then curious to hear yours, though.

Ben Wilson [00:25:10]:
I wish I had something different as a the only thing that would add on to exactly what you just said is that's the person that knows, like, what's going on and who's gonna resource and staff. My second favorite people to talk to, this might sound stupid, but the actual owner of the project, which most of the time in back when I was doing similar stuff to you in in the field of Databricks, it's usually nobody who's touching the keyboard to write any implementation. It's like the marketing team or sales team, whoever at a company that actually is gonna be using this model. And the reason I like talking to them so much is because they provide faster insight into what actually needs to get built

Brooke Wenig [00:25:59]:
Mhmm.

Ben Wilson [00:26:00]:
Than anybody who's building the thing. A lot of times, the companies the people building it, they didn't even talk to the people that are been gonna be using this. So they have this mental model of, like, we're gonna produce predictions. We're gonna write it to a table, and and this team will just use it. And you talk to the team, like, yeah. We don't even have access to whatever they're talking about. I don't know what this is. It's like, okay.

Ben Wilson [00:26:21]:
Let's think about this a little bit.

Brooke Wenig [00:26:24]:
Yeah. I always give that example with, like, health care in particular because, like, doctors are naturally skeptical of any model models. And it's like, who cares if you built the best model to predict cancer, whatever it may be, if doctors aren't going to use it? It's like, you definitely have to work hand in hand with the person who would be the end consumer.

Michael Berk [00:26:40]:
So one of the reasons that I'm asking that question is, I am put on a lot of projects similar to what Brooke's team does, but typically more in the platform or data engineering space. And the first call that we always have is getting that political context, determining who the key stakeholders are, what's the success criteria, and we try to get it in writing just to, like, cover our bases. I was wondering, Brooke, if you could sort of elaborate on, so you said the lack of success criterias make projects fail. But from a political or, like, organization specific context, what have you seen makes projects fail more than other traits, let's say, that is about customer inefficiency or, like, lack of communication betweens that between teams that's external to your project?

Brooke Wenig [00:27:32]:
A lot of times, it's not thinking about how do we get this to production. I'd say it's not super political for showing proof of value, proof of concept. Like, everybody knows that's early stages. But it's alright. Now we wanna get this into production real estate. Wow. It's actually gonna be really expensive if we wanna be using GPT 4, for all of these thousands of requests that we're going to be anticipating every hour, every day, whatever it may be. And so planning is what it's like the traditional data science problem.

Brooke Wenig [00:27:58]:
The data scientist will build a model. The data scientist will not be the one that productionizes that model. Like, you need somebody from the ML engineering side or the DevOps team to help productionize it, and then who's gonna monitor it. And so what we're seeing is proof of concept, proof of value that's easy to get everybody on board. It's a relatively low time investment, relatively low cost investment to get to that phase. But it's what are the gaps to getting into production? So for a lot of customers, it's security of, alright, we've done this on a subset of data that everybody in the company has access to. If we actually want to roll this out for all of these cases that we have, not just like the HR employee docs everybody has access to, What other datasets can we get? Have we done our due diligence there? Red teaming, testing it out, cost monitoring. There's just so many potential gaps along the way for production that most use cases that we're seeing are still internal use cases.

Brooke Wenig [00:28:52]:
So they might be in production, but they're just not externally facing.

Michael Berk [00:28:57]:
Got it. Ben, so you work internally in Databricks and don't typically interface with customers. Do you see similar issues with productionizing as as a big bottleneck, or is that typically handled well?

Ben Wilson [00:29:11]:
I mean, I interface with customers, usually through engineering support tickets. Like, we broke something or we didn't think the use case through. So the vast majority of things that we get are during production deployment, where they've already done all that planning homework. They know what they're doing. So it's it's the more sophisticated side of of, like, our customer base, or in the open source community. Like, these are people who are finding issues with, like, hey. You guys have this API to build this container, and it doesn't allow me to configure my integrated OAuth. Like, what the hell? Like, why isn't this supported? So it's like, that's going on the road map.

Ben Wilson [00:30:04]:
Sorry. We'll fix that for you. So we don't get a lot of questions or issues with people that are in that formative stage. Like, sometimes you'll get questions about, like, hey. This API is broken for tracking or, you know, for, like, registering my model. This this is messed up. That's that's early phase stuff, and that's more functionality of a of the core product. We don't talk use cases with them, but sometimes we come on and talk to customers when we're getting ready to deploy something new, like a new major feature.

Ben Wilson [00:30:43]:
We'll meet with 20, 30 customers and get to hear what cool stuff they're doing. And the interesting thing to for me is those were the customers. Like, the people that I talk to in these calls now is the ones that I never talked to when I was in the field, because they don't need help. You know? They they they're capable of reading through the source code, figuring it out, and it's usually really, really top notch engineering groups. But they're the ones that are like, yeah. We're gonna test this out. We're gonna break this and tell you how it's broken so that you can improve it. And during those calls, sometimes you hear about, like, what they're trying to do, and you see some really interesting things that people are doing.

Michael Berk [00:31:29]:
Like,

Ben Wilson [00:31:29]:
oh, yeah. There's this model. Yeah. We gotta train. It's great. Data science team did an awesome job. Now we're integrating it into our app platform. And this like, we are having problems with the size of the container that's deployed here.

Ben Wilson [00:31:46]:
Like, why are you including, like, the JVM in this? Like, we'll we'll fix that. Make that a little bit lighter for you. But then they'll talk about what the actual use case is and how they're doing controls and how they're doing, like, failovers. Like, hey. If this thing starts losing its mind, for this use case, we have a fallback that we can do for our app. Or, we also have, like, we're taking everything that's being generated. We're we're submitting, you know, trace events from every request that's made, and that's going into a real time database that we're, you know, continuously monitoring and checking. So you see, like, very sophisticated things in that space.

Michael Berk [00:32:28]:
Okay. I think I think I I'm following. Another question, Brooke. So alright. We've exited the scoping and planning phase. How do you think about delivery, specifically balancing prototyping with just building something?

Brooke Wenig [00:32:48]:
Oh, this this is a good question. This came up this week, for example, like, we have a new feature. We do not know if it will fully address the customer's need, but we know that we can always fall back to the do it yourself RAG implementation as an example. And so, generally, we try to time bound that, as well as add some buffer in. So, for example, like, when we scope engagements, if we realistically think it's gonna take 15 days, we'll put 20 days there. We have no idea what issues we're gonna hit. Just need a little bit of padding time. And so, like, if there's a new experimental feature that we think could drastically improve the efficiency, for the overall customer, we might time down that, like, alright.

Brooke Wenig [00:33:28]:
We've got 3 or 4 days to do this. If, to either address are there any major gaps? If there are major gaps, we abandon right away and we share that feedback with either, the product team, whoever it may be. If there are not major gaps, then we make the product team aware of, like, hey. We are planning to do this. Please let us know if there's any issues they need to be aware of. And, oftentimes, if we do plan to go ahead with that approach, they will invest additional engineering resources, to get an early customer in. And it's a great working relationship as well too because engineering needs that early feedback from customers. So they don't go down a rabbit hole of, like, a single mind track of this is the approach that we want to take, and they start seeing customers actually adopting it and realizing how many different ways they plan to be using it or different things that they didn't think needed to be supported.

Brooke Wenig [00:34:15]:
So it's, like, it's this great relationship because engineering also knows us. We're internal to Databricks. They know we know what we're doing with the product. And so at least there's a level of trust, there's a level of, reciprocation. Like, generally, we will have seen that, the PRD early on. We've then done some early dogfooding, so by the time we start using the customer, start using it for our customer, we know what we are doing. We just don't know the limitations of the product or the limitations, for that given use case. So, yeah, definitely needs to do some time learning.

Brooke Wenig [00:34:46]:
The, worst ones, though, are when customers want to use LLMs for migrations. That's the one that I run away from. And this was all of the craze last year that, of people thinking, like, LLMs are this magic bullet for migrations. And what we had told them, like, you are welcome to try experimenting with it. We don't think it's gonna work for you, and I'll tell you why in a second. But you are going to scope a 100% migration effort. We will add a 15% buffer. So your overall, like, net proposal is larger than if you didn't want to do it because we need that follow back plan of the traditional migration approach, if the LM doesn't work out.

Brooke Wenig [00:35:21]:
And, typically, it won't work for there's 3 types of migrations. There's direct code migration, which is often not what customers want to do. They often want to do a modernization. And it's like LNs are LNs are great at the code translation. They're not great at modernization. Like, try taking some legacy code and converting it to the medallion architecture on Databricks. Like, there's no out of the box element that will do that for you. So that kind of rules out the second one of modernization, which is what most customers want to do.

Brooke Wenig [00:35:49]:
And so where we're seeing elements being most successful for migration related use cases are either asking questions about the legacy code, or generating documentation or generating test cases. And so it's not this magic bullet that customers all thought it would be. And when we explain it to them like that, they're like, actually, yeah, we we we realized that this is not the right use case. Did that answer your question, Michael? I feel like I went on a bit of a tangent there.

Michael Berk [00:36:15]:
Yes. It a 100% answered the question. So, basically, add some buffer and time bound.

Brooke Wenig [00:36:22]:
Exactly.

Michael Berk [00:36:23]:
Cool. Yeah. It's it's funny you say that about migrations, though. They're my team does a lot of migrations. It might be from a different cloud system into Databricks or between Databricks tooling or things like that. And we typically don't support things outside of Databricks. But, yeah, the amount of times that we've just tried to throw an LO on that stuff, and it just breaks immediately. Yeah.

Michael Berk [00:36:46]:
It's unfortunate. But as you're pointing out, being in a information retrieval system and helping you understand the systems, that's where LMs seem to be really, really good. And I use GPT 4 everyday to, like, answer basic questions about how do I use pytest fixtures or how do I do do this or that. Like, it's if it's well documented specifically in the Spark space, like Spark has been around for years, It usually has reasonable solutions to simple problems. So it allows you to be a generalist a lot faster than than other systems.

Ben Wilson [00:37:19]:
There's some tactical stuff with code, though, that I can 100% tell you that every almost every engineer at Databricks uses, LLMs for this particular use case, aside from what you mentioned, Brooke, about docs. Yeah. Definitely. Docstrings, it's exceptional at doing that. And it's that just saved you 15 minutes, for each function or or method that you're you're documenting. But when you're unsure of sufficiently complex implementations, if you're doing something that is reducing the ability for somebody who didn't write it to grok what you're doing. It's really great. So if you have prototype code that you're like, hey.

Ben Wilson [00:38:09]:
I'm just banging out this solution. I wrote this function or this method in this class, and just asking one of these models, like, is this readable? Can this be refactored

Brooke Wenig [00:38:22]:
Mhmm.

Ben Wilson [00:38:22]:
To to maximize readability and understandability or testability? Where they were a year ago was pretty garbage. Like, GPT 3.5, not that great at that. More recent builds of of GPT 4 are exceptional at doing that, where it'll take overly complex implementations and break it down into testable components. We're like, hey. You're doing something here that by rewriting this this internal method that you have in this this other method, externalize that or make it static and make this a little bit simpler, and you look at what it comes up with. And you're like, man, thanks, buddy. And, you know, it's just like the it's like the ultimate, you know, code rubber duck that's just sitting there next to you and and is ready to give you advice. And I find that 9 times out of 10 with that targeted use of those tools for a software engineer, it's really good.

Ben Wilson [00:39:24]:
Mhmm. Because it removes that that mental model that you've already built. You have context on it. You know what this thing's doing that you just built, and it seems simple to you. Mhmm. But asking it and prompting it in that way, like, hey. Can a junior engineer understand what I just wrote here?

Brooke Wenig [00:39:41]:
So I've got a question for you, Ben. So if somebody is studying computer science or learning how to program from scratch, like, they're brand new to it, do you think they should go through kind of the traditional approach of not using any of these augmented tools like Copilot? Or do you think that they should try to do whatever make or do you think that they should learn how to program leveraging these tools from the get go?

Ben Wilson [00:40:09]:
I think learning to code and computer science are not directly related to one another. So learning computer science in an undergrad or grad degree

Brooke Wenig [00:40:23]:
Mhmm.

Ben Wilson [00:40:23]:
That is absolutely never going to go away. Understanding fundamentals of algorithms and logic and how to construct, like, what are the components with that make computer systems work and information architecture and stuff.

Brooke Wenig [00:40:38]:
Mhmm.

Ben Wilson [00:40:39]:
That's so key and critical to understand that stuff. And if you don't have that background and you get into software development, you're gonna have a rough time. You're gonna learn it the hard way, and it's gonna be really painful. But for coding, I see coding, like, actually sitting down and writing code as more of a trade skill, that requires deep knowledge, of the fundamentals of, like, how these things work. But it's it's more of a, like, practice till you get good sort of thing, and you just grow in ex as you get experience in this and have people providing mentorship to you to correct you. That's the whole reason for code reviews and having, like, a a tech lead that's answering your questions. I see that Gen AI shortens that process even for a new developer. And even though people that I've talked to are kinda worried about that, they're like, oh, we shouldn't have junior engineers use these tools because they're never gonna learn the basics.

Ben Wilson [00:41:46]:
Like, I don't I don't actually see that. I think it's it's like them having them having a TL that they can ask questions to no matter how stupid they think they are. And there's no judgment, and they can just get an answer and kinda see What I don't think is ever gonna be successful in software engineering in industry is somebody who's doing copy pasta output responses from a tool Mhmm. Without checking it. I've seen PRs and ML flow. Like, it opensource contributors are like this. Thank you, gbt 4. This, like, this this function that you wrote does not solve the problem, because it can't load the entire code base and see context.

Ben Wilson [00:42:33]:
Yeah. It's like, hey. You're using this API that's actually not designed for this, Or, hey. Let's not use this private API that there's no guarantee that this is gonna stay in this functionality. If we change that, now this implementation is broken. We don't want this this linkage here. So that's dangerous, and that it puts a little bit more sort of workload on maintainers of repositories, whether it's internal or external, like open source or closed source. But that's that's more of a process thing.

Ben Wilson [00:43:07]:
Like, you just follow-up the person and be like, hey. Nah. Let let's not do this, and this is why. Mhmm. Please change this. And, hopefully, that prompts that person to be like, alright. I need to actually do some work here.

Brooke Wenig [00:43:22]:
I like your distinction between computer science and programming. One thing that we continually struggle with is, as part of our interview process, we have a coding assignment. And we know, like, it's not a proctored assignment. We know that people have access to these tools. We ask on our policy that they do not use any form of assistance while doing the coding exam. I guess given that kind of mindset then, since people are gonna be using these assistance as part of their jobs, what are your thoughts on for coding assignment interviews, seeing how productive they are with these tools versus without using any tools?

Ben Wilson [00:43:59]:
I think even with, like, with the honor system, good luck on keeping people honest about that. I don't think that should be an expectation. And I think stuff like take home coding assignments or for job interviews are gonna be a thing of the past because you can't there's no way you can guarantee that that person created that or that they even understand what it is that is being asked. Because you can take the question in the context of that, paste it into a a chat template, and get a response that's probably pretty darn correct. And all they have to then do is just execute it and be like, did I get the right answers that this thing is expecting? Alright. On to the next question. So on our side, we don't do stuff like that. It's all live.

Ben Wilson [00:44:51]:
Mhmm. And not necessarily like, hey. Open up a an IDE and, you know, implement this algorithm. Although there are certain interviews that do that. But it's more being able to discuss an implementation in pseudo code, because you don't care if somebody can write flawless code. Like, nobody does that for their job anyway. You know, we have all of these tools to help us craft a good solution that can be merged into a code base. It's more of understanding the why.

Ben Wilson [00:45:27]:
Mhmm. Why was this built this way, or why did you choose to build it this way? What are the downsides? Like, hey. What happens if, you know, you're using this this list iterator here, and what happens if the data increases by a million fold? What does that do to this algorithm? Mhmm. And if they can't think through that or don't understand that, there's, like, nothing happens. This is resilient. Right? Let and then that's when I open up, like, a screen share and write the actual code for them and say, let's try it. And then you see that it just crashes the, like, the Python REPL. And you're like Mhmm.

Ben Wilson [00:46:04]:
Here we go. Or, hey. We hit the recursion limit in Python with this implementation because we're iterating over nested dictionaries that are, you know, 1500 levels deep. So it I think to answer your question directly, I think it's gonna change, and it has to change.

Brooke Wenig [00:46:23]:
Mhmm.

Ben Wilson [00:46:24]:
But it's gonna it's on the onus of the hiring manager now to find somebody to rewrite that that interview process.

Brooke Wenig [00:46:31]:
Yeah. We intentionally have an open ended, question so that it's not a yes or no. And as part of the following text screen, we do ask people to we do ask pointed questions about the code asking about the approach to try to guard against it. But, of course, there's no way to really ensure that they were even the ones that wrote the code, in the first place as well. So it's a challenge for all, hiring managers, but we we think so far it's working. But, yeah, definitely always a challenge.

Ben Wilson [00:47:00]:
Question, I think it's harder on my side, when you're talking about, like, hey. Design a system that provides a REST API to this sort of service and, get this working. Like, use this database type, and this is the data structure and create a a schema that you can insert data into the, you know, CRUD operations on on a table Mhmm. And then create an interface for that. You could probably use an an open model that's not particularly well trained, and it would still kind of grok how to do that. So, yeah, that's why we do the we're doing this live sort of thing.

Brooke Wenig [00:47:40]:
Mhmm.

Michael Berk [00:47:42]:
So I was just in a wedding in Baja, Mexico. And as we are all, like, awkwardly standing around sipping a margarita, AI comes up because my job is, like, data science or whatever. And, everybody's, like, interested in the field and a lot of people who are data or, like, ML adjacent, so in the data space of some sort are really interested in entering that field. So, Brooke, your team is the best of the best in theory, and they're quite good from what I've seen. Lots of them have PhDs. Lots of them have many, many years of experience building distributed systems and and complex ML implementations. How would you recommend someone start entering the AI space if they have a data skill set?

Brooke Wenig [00:48:30]:
If they have a data skill set. Okay. That that's an interesting question, and it's one where, like, I feel like a bit of a hypocrite with. Because for my team, because Databricks right now is such a hot company, like, we have our pick of the litter of candidates that are applying. And so, like, the bar to get hired today is much higher than the bar to get hired a few years back. Like, all of my tenure team members will choke me, myself included. If I were applying to Databricks today, like, with the same resume I had back then, it would have been picked up, just because of how high the bar is right now. And so, like, if somebody's brand new to, if they're brand new to AI, like, even if they have, like, 20 years of software engineering experience and they're fantastic at that, like, they're so far behind on the AI side that it's hard to get hired into a data science team.

Brooke Wenig [00:49:21]:
And so my recommendation for folks who have data skills but not AI skills is see how they can use AI to make their team or their function more AI driven. And so, like, that could be if you are like, let's say, even if you're not in a tech company or, like, you're let's say you don't even have data skills. Like, you're an environmental scientist, let's say, and you wanna get into AI. Wait. Do you have an environmental science degree?

Michael Berk [00:49:53]:
Yeah. I failed out of computer science, so I studied environmental science.

Brooke Wenig [00:49:57]:
I did not know that. So, yeah, let's say you have an environmental science degree like Michael, and you do wanna get into AI. It's thinking, what are cool project that you could be using AI for? For example, like, recognizing sounds of, like, bird calls, as an example. And so that way, you're able to still leverage your expertise in a domain, but then build up a new skill set so that eventually you can then transition fully into AI if that's the way that you want to go. But I always try to look for for use cases where somebody is able to contribute their skills, while still developing other skills. Because if they're able to contribute their existing skills, they're just like a junior entry level person that needs to be trained up from scratch. And so that's where, like, they would be a good fit for my team, but, like, I eventually try to bring AI into your existing team or function. That way, you're still able to be productive, and you're building up that skill set, throughout.

Michael Berk [00:50:56]:
Crystal clear. Yeah. And if you can sort of leverage your existing situation as a jumping off point and start incorporating new things. Like, the the 0 to 1 shift is often difficult. So if you slowly incorporate, that makes a lot of sense.

Brooke Wenig [00:51:10]:
And it's also easier to do within the company you're at than trying to jump to go to another company to make that switch because you already have the connections. You know how the company operates. You have that trust. And oftentimes, companies are willing to make concessions to keep top talent. And so they will potentially find a way to, like, hey. I want to balance, like, my day job with this, but I want to effectively be like an intern to the data science team or whatever it may be as well. So, yeah, I I definitely recommend trying to do it within your existing role because if you're trying to do 0 to 1, it's it would be hard to land a job at another company, with that sole focus on AI, with limited AI skills.

Ben Wilson [00:51:55]:
You just explained my career path. That's exactly how I got into data science. We were doing factory yield analysis, which is with me and the people that were on that team. This is, jeez, 15, 16 something years ago. I feel older than that. It was, like, 18 years ago, but we were, we were so annoyed with the manual process of going through and doing things in spreadsheets of, like, hey. I need to find outliers in this column of data, and it's got 4,000,000,000 rows in it. Mhmm.

Ben Wilson [00:52:37]:
And just applying, like, basic statistical formulas onto that, iteratively. Like, oh, this this dataset has 15,000 columns, and we have to write a macro that's going through and and basically identifying rows that are, you know, basically SPC rules on this data. And none of us knew programming or how to do this the right way. We're using, like, VBA scripts and stuff. And eventually, we're like, there's gotta be a better way. Let's take some classes and just see if there's a a way to do this. And that's when we all learned Python the first time, and and we're like, oh, there's these, like, stats, you know, modules out there that we can use. And then we we started learning more and saying, isn't this what, like, the model stuff is? Like, this machine learning stuff, like, we can just use, like, a logistic regression model and see what happens.

Ben Wilson [00:53:36]:
We trained it, and we're like, hang on. This just saved us 34 person hours per week by just this one model. What else can we do? And 3 years later, we had a suite of over a 100 models that we're running in production every day that we're automating all of this stuff that we hated doing. And we're saying, like, oh, yield just went up 37% on this line because of all this stuff automating what we couldn't get to before. And, yeah, management we didn't even ask. Like, management just said, you're all now, like, the data science team. Like, just do this. We're like, okay.

Ben Wilson [00:54:20]:
And they're like, hire 15 more people to make more of this.

Brooke Wenig [00:54:24]:
Mhmm. But then exactly those 15 more people that you hire, they all have to have the skills that you all had already picked up. So it's one where it feels like the bar keeps getting higher and higher.

Michael Berk [00:54:34]:
Yeah. Brooke, do you prefer people with that sort of Ben background where it's homegrown and they've learned it over time? Or do you prefer the PhD and distributed ML?

Brooke Wenig [00:54:46]:
So one thing I pride myself in is that I think we only have 3 people on a team of 14 with the same degree, and, like, of course, that degree is computer science. Everybody else has a distinct degree. It's like we do have an environmental science degree on the team. Actually, we've got philosophy degrees on the team, stats, math. And so I do think there is something to be said for some level of continued studies, but that's not a substitute for hands on practical experience as well. So, like, there's a balance between the 2. I will say the average level of education on the team is between a master's and a PhD. Most of the team has master's.

Brooke Wenig [00:55:26]:
There's one postdoc on the team or or one team or one team member who did actually 2 postdocs. So generally, we look for folks who have some form of grad studies because ML is not commonly taught at the undergrad level. Undergrad level, like, they might take 1 ML class if that. There are some schools like Stanford, Berkeley, like, those are schools that actually have proper ML courses for undergrads. Most of them don't. Most of the computer science programs are teaching very, I don't wanna say antiquated concepts, but just they're not teaching the things that industry needs. They're not teaching cloud skills. They're not teaching Git.

Brooke Wenig [00:56:03]:
They're not teaching ML. And so that's where I will generally air for somebody who has a master's or a PhD so they have those ML skills, as well as a few years of industry experience. Because that way, they'll have picked up the things that they needed to pick up for industry, but still have that strong foundational understanding of ML. We actually only have one team member who only has an undergrad degree. Interesting.

Michael Berk [00:56:26]:
Yeah. I remember, in undergrad, I spent 3 weeks on Perceptron in one class, and I was like, come

Brooke Wenig [00:56:32]:
on, guys. I remember my LDP class in, undergrad was using LISP and all using overhead slides. Like, they did not change in 30 years.

Michael Berk [00:56:45]:
Yeah. It's and just out of curiosity, do you know why that is? Because it seems like like academia lags behind industry in terms of tooling, which makes sense. You don't use, I don't know, get Docker clouds as much. But why don't they just, like, ask?

Brooke Wenig [00:57:02]:
I I there's this brain drain from academia to industry right now. Industry is so lucrative, that top academics are just being pulled into industry. And I think it's also the kind of divide of it's hard to so, like, schools like Stanford and Berkeley, like, Databricks cofounders are also professors at Stanford and Berkeley. Like, they have that flexibility to be able to keep professors on. Like, Matej still teaches courses. I think Ali has been on a bit of a hiatus for quite some time now. But, like, Jan, one of the other cofounders, like, he's a professor at Berkeley. And so schools that are more connected to tech, I think, have a better understanding, like, that balance of professors, spending time in industry and they can feed that back into the classroom.

Brooke Wenig [00:57:46]:
But, typically, that's only at the grad level, to be able to get that cutting edge, more seminar style, courses, rather than the undergrad. Like, it takes a lot of work, to update a course syllabus. I remember at UCLA, it took 2 years to approve a new course. And so, like, by the time you approve a new course in the space, like, it's already gonna be out of date. Like, I remember when I went through my computer science degree, they said the half life on a computer science degree is 7 years. And it's much shorter than all of the other fields. And I think that half lifetime is continuing to decrease. I don't know if I call it 5 years now.

Brooke Wenig [00:58:23]:
But, yeah, it it's hard to keep up with industry, particularly when professors don't have that opportunity. I remember I believe UCLA had a limit of, like, only on Fridays could professors do advising for industry. And so with that limited feedback loop, it's hard to kind of get that information flowing both ways. And so, like, Databricks does have a university alliance program where we do share some content with, universities. We give them free access to Databricks. But that's not an entire semester long course. Like, that is a lot of work to build and to maintain. So I think there's some fundamental issues going on there, but, hopefully, other schools start to pick things up.

Brooke Wenig [00:59:01]:
UCLA now has, I think, 2 undergrad ML courses, but, yeah, most of it's at the grad level.

Michael Berk [00:59:08]:
Got it. Okay. One final question for you. What are the traits of a top data scientist on your team?

Brooke Wenig [00:59:18]:
Grit. Grit, curiosity, and collaboration. And those are all things you can't teach in school. So, like, you go to school to learn the foundations. All of those other skills are ones that you can't learn from just reading through textbooks. And so for me, collaboration is one of the most important ones of, like, how well do you work with other team members? How well do you work cross functionally? Because I do not care how brilliant an individual is if they're not able to lift up the rest of the team and the people around them. So that's collaboration, curiosity. It's doesn't matter what your prior experience has been if you don't have the potential, the drive to continue to learn new things.

Brooke Wenig [00:59:58]:
It's like we saw that with the shift to Gen AI. I only had one person on my team with serious NLP chops prior to the Gen AI craze, and now the entire team is delivering on Gen AI engagements. It's like if they didn't have that curiosity drive, we only hire people for past experience without that curiosity or that potential. Like, they wouldn't have made that transformation. And grit is being willing to not only work smart, but also work hard, and, like, wanting to work on tough and challenging projects. And I would say that that was something that Databricks culture very much hired for early on is assessing out grit. And, like, that that, of course, changes over time. Like, I'd say those are probably my 3 key hiring principles.

Michael Berk [01:00:42]:
Super well said. On collaboration, can you elaborate, though, what, like, what does that mean? Like, is it communication? Is it teaching and mentoring? What it like, what are is the actual activity of collaboration?

Brooke Wenig [01:00:55]:
Collaboration could be, like, hey. I'm struggling with something. Does anybody have a second pair of eyes? Like, team members jumping on, like, yeah, I'll help you out. Other ones like volunteering information back like, hey, I struggled with this. Like, here's my hacky workaround for all these lane chain issues. It's also identifying, like, these are areas that aren't working well in Databricks. Like, hey, our field team isn't as conversant in Gen AI as we need them to be. Like, what are ways that we can upscale them and contribute our knowledge back? And so, like, one way is, like, we could do a rotation.

Brooke Wenig [01:01:24]:
Like, they work on a professional services engagement that we oversee. We get some temporary additional bandwidth. We're also helping to transfer our knowledge back to them. So it's really trying to find ways that are win win for everyone and just doing the right thing for the company as opposed to solely focusing, like, I want to be the owner of this thing. I don't want anybody else to touch it. So, yeah, it's really focusing, like, what's the right thing for the company, what's the right thing for the customer, and not trying to overly focus on the I. It's more about the we.

Michael Berk [01:01:56]:
So it's more like context aware knowledge transfer?

Brooke Wenig [01:02:00]:
It's more than just knowledge transfer, though. Like, it is actively seeking out to do things with other team members as well. So, for example, like, co presenting at a conference. Like, that's when we're always tell the team, like, it doesn't detract from you if you have a co presenter. What it means is you have somebody else that you can bounce ideas off of. Somebody else that could take over the presentation should you fall ill as well. Like, it's overall less work on you, and you get the same amount of credit whether you present solo or you present as a pair. And so it's looking for folks who want to work with other team members and they see the value from it.

Brooke Wenig [01:02:32]:
And, like, that's something that I evaluate in the interview process. If they just say I I I, they're trying to take credit for others' work. If they say we too much, then I'll ask, like, alright. What exactly was your particular role? So, like, it might be like I was the tech lead and we accomplished blank. Like, that that's what I love to see is, like, giving the team the credit for that. And so, like, with the team, I always encourage other team members, and I love seeing, like they give each other shout outs of, like, thank you for helping me figure this out. Like, you saved me 10 hours of sorting the bugging issues for a prompt. I would have never figured out the solution had you not helped out.

Brooke Wenig [01:03:07]:
So it's really trying to help out other team members, and doing the right thing, without necessarily regard to your own reward or credit.

Ben Wilson [01:03:18]:
That also really shows with the interactions that we get with your team, on our side. Not only are so you can always tell when it's somebody who works for you, when when there's something from the field coming into our our team channel, because there's always a reproducible code snippet that will that actually has import statements at the top and everything. It's like, this is a fully functional thing to reproduce this issue. And then there'll be a series of not like, hey. This is broken, but more like, I expect this API to do x. It does y. Can it do x plus y? And here's the justification for it. And usually, when we read that, when we read a report like that, you can always see the Slack history, because it's really short.

Ben Wilson [01:04:12]:
It's, like, 4 or 5 back and forths. And at the end, it's usually, here's a link to a PR that I just filed to fix this for you instead of the other, you know, group of reports that we get where it's like, hey. This doesn't work. You know, like, what what doesn't work? You know, I tried doing this thing. Like, can you give me a or, hey, this threw an exception. Like, can you give me the stack trace? It's like, it's clear that your team thinks about these things from how to solve a problem perspective as well as you can also tell that they've probably discussed this internally amongst themselves. They're like, hey. I tried that as well.

Ben Wilson [01:04:51]:
Yeah. It's totally broken. Like, let's tell engineering. It's a it's a special culture that you've helped build on that team.

Brooke Wenig [01:04:59]:
Thank you. Yeah. And and it's something that I do intentionally foster. So, like, for my team, I typically will cap the amount of time that they spend on professional services engagements about 32 hours a week. And so with the remaining 8 hours, like, that's all hands 1 on 1. But the rest of the time is spent, working with r and d. And so, like, with the product champions program that we have internally, working closely with a dedicated end of team. And if not, that's testing out the new features, because we are gonna be using them on projects, and it is way easier if we catch these issues early on, we're able to provide constructive feedback than once it's already gone GA, if we need to make, like, a breaking change to an API.

Brooke Wenig [01:05:37]:
So, yeah, definitely something that we intentionally do. And that's when we're like, it the sum is greater than what's the phrase? The whole is greater than the sum of its parts of working together with engineering, to try to make the product better rather than trying to say, like, I built this awesome thing that's a workaround for all of this. Yeah. Like, isn't this great? It's like, no. The the the right solution is first principles. Let's see how we can make the product better, and work with, R and D on that.

Ben Wilson [01:06:05]:
And we greatly appreciate it.

Michael Berk [01:06:08]:
Yeah. Yeah. That resonates like crazy. I will I'll see what we can do. Awesome. So we are out of time. I will quickly summarize. Today, we started talking or started the conversation talking about scoping.

Michael Berk [01:06:23]:
Brooke's tips is start off with a use case checklist and ensure that there's an actual real world value that can be provided by the solution. And then once that's affirmed, you can enter a conversation, typically a call, where you can discuss success criteria. Couple common pitfalls of planning is not getting that success criteria ironed out and then also not thinking about moving the POC into production. On the proprietary versus open model side, it's typically a function of organization maturity, use case maturity, and then also regulatory risk. So, choosing between whether it's gonna be an open model or a closed model, you can sort of think along those three axes. And then if you have data skills and you wanna move into AI, try to your existing use cases to actually leverage some AI in some form. And, typically, in your current organization, you'll have more leeway than trying to enter a position without experience. And finally, top data scientists have grit, curiosity, and collaboration.

Michael Berk [01:07:20]:
So, Brooke, if people wanna learn more about you, your team, maybe potential roles on the team, where should they go?

Brooke Wenig [01:07:26]:
Databricks.comforward/careers. Cool. We're always hiring.

Michael Berk [01:07:31]:
Alright. Well, until next time, it's been Michael Burke and my cohost, Ben Wilson. And have a good day everyone. We'll catch you next time.