In today's episode, Ben and Michael dive deep into the intricacies of software development, innovation, and team dynamics. This episode explores the critical balance between building in-house tools versus leveraging open-source solutions, with real-world examples from Databricks.

They discuss the creation and eventual abandonment of a benchmarking tool for warehouses and discuss the importance of evaluating user demand, effort, and impact before committing to development. They emphasize the role of empathy, constructive feedback, and team collaboration in driving successful projects. They share strategies to influence behavior within organizations, the significance of a blame-free culture, and the art of leading difficult conversations with stakeholders.

From detailed discussions on customer feedback loops to practical advice on automating mundane tasks, this episode is packed with insights that will help you navigate the complex landscape of software development. So sit back, relax, and join us for a thoughtful and engaging conversation on how to turn challenges into opportunities for growth and innovation.

Michael Berk [00:00:05]:
Welcome back to another episode of adventures in machine learning. I'm one of your hosts, Michael Burke, and I do data engineering and machine learning at Databricks. Today, I'm joined by my cohost.

Ben Wilson [00:00:15]:
Ben Wilson. I teach myself TypeScript so I can write scripts at Databricks.

Michael Berk [00:00:21]:
So impressive. Just blows my mind every time I hear it. So we are on hour, 1 so we we're 1 hour and 41 minutes deep into our recording and just press play. Prior to this, we have been talking about some very, interesting things about the human condition and how to convince people of your idea, and even more specifically, how to convince someone that your idea is their idea. If it's their idea, they will believe it more more strongly and they will champion it throughout the organization and hopefully drive change. So, we're gonna start this episode with a bit of a scenario, something that both Ben and I have seen. It refers to the buy versus build and it depends upon, a variety of factors, but let's just set it up really quick. So I am a engineer on my team and I think that there's some people maybe in my organization who are not quite as technically competent as my amazing self, so I'm going to go build a framework that makes joins in Spark a lot easier.

Michael Berk [00:01:27]:
And this con this framework will leverage YAML as a config. We will write out files and, read them back in, and that config will determine how a join is performed. So these these analysts that only know SQL and that can only read English, they're gonna be joining things left and right. Ben, what are your initial thoughts on this framework?

Ben Wilson [00:01:54]:
I've seen this framework, many times, and I've always had the same reaction to it, which is confusion. When I first see it, I wonder, was it easier to build that than it would have been to teach the people to just do the basic applied engineering tasks? And I've never gotten a straight answer from anybody who's built one of those. Usually, it's, well, we're not resourced to train people, or we we don't have training material for that. Like, well, how many and then I'll ask, like, how many hours did it take you to build this? And usually, it's a very large number, and the maintenance burden of it is usually pretty pretty substantial. But I never get a I never get a, like, a confident answer to any of the questions that I ask about why does this exist. I usually get, like, a panic look from people when they start talking about it. They're proud at first. They're like, hey.

Ben Wilson [00:03:00]:
Look at this cool thing I built. And then I start asking, like, why did you build this? And I get this, like, defeated you you kinda see that the stages of grief unfold in front of you as somebody realizes that they might have made a mistake in creating this thing. It's usually when you're talking to somebody, like, the job I used to do, the job you currently do, it's because they're having troubles with what they built or what they're maintaining or, k, what we have this interface, but you guys keep on changing Spark or you keep on changing APIs. And can you support these new features and stuff? Or how do I how do I make this work with this new thing? I can't migrate to an a newer version. So it it's never at the point when they build that that it's brand new and there are no issues. It's usually, like, a year later, they're like, help me.

Michael Berk [00:04:02]:
Right. Yeah. There's definitely selection bias in the people that we talk to in the field. People typically, they they don't have enough, staff to do what they wanna do, or they don't have the technical expertise to do what they wanna do. And for this scenario specifically, I've seen it a bunch, like, a little bit too much. And depending upon the audience, it's actually a pretty simple checklist to get them to to change their perspective. But I'm curious, Ben. I I would love your feedback on my checklist.

Michael Berk [00:04:30]:
So let let's say we go into a room and and they've abstracted some monstrosity on top of a a solid open source framework that actually works. What I typically like to say is, first, the amount of maintenance that is required on this open sore on the wrap around the open source framework is just not worth it. 2nd, I say that if you guys want job security, that's great. But if you wanna go to another organization with skills that are transferable, don't build an in house solution that can't transfer. Instead, learn the base framework. That makes your job skills a lot more relevant. And specifically for engineers, like mid level and below, they'll love hearing that. They're like, oh my god.

Michael Berk [00:05:15]:
If I become an expert at companies x y z's library, that doesn't transfer. But if I become an expert at Spark, that does transfer. Those are the first two that I typically say. What are your thoughts?

Ben Wilson [00:05:30]:
No. Those those are good. I usually do a bunch of prework before I get into the to preface this, my main motivation in getting somebody to listen to advice that I've that I'm pretty confident that they don't wanna hear in the first couple of minutes of meeting me, is to read the room. And what I mean read the room is I've never been in an engagement with a a customer like that where they built something like this, where there's not one person in the room that everybody defers to. So if you ask questions about, like, hey, what are you working on? And what's your most challenging thing that you've built? Or what are you most proud of? Those simple soft questions, I don't care what they're answering at all. I don't care what they build. It's irrelevant. There's tons of people building cool stuff all the time everywhere.

Ben Wilson [00:06:38]:
Software is an ephemeral thing, you know, it's it's always evolving, and there's a lot of people who know how to do really cool things with it. So I don't I never really got that excited because I've been at places before doing that job where I saw people do stuff like that. Like, yeah, it's pretty cool. After you see your your 10 thousandth project of somebody doing something cool or, you know, participating in a hackathon where you see a bunch of cool things that are built, nothing becomes cool anymore. It's just like, is this useful? Yes or no? What problem does it solve? So I'm asking those questions for one purpose only. Who's the person that talks the most? Because that's the person who came up with the ideas. And then you start asking progressively more challenging questions about they're still soft level. They're not they're not antagonistic, but it's more just to see if that person who is answering all of those, provided they're not the manager, who speaks up first when you question what they've done in a very innocuous way? That's the person whose idea was guaranteed.

Ben Wilson [00:07:55]:
Who are like, hey. We're doing this, and this is why it's so important everybody get on board. And you can then further test what the dynamics of the room are by asking up an intentionally stupid question and preface it with, like, hey, this might be a stupid question or something. You know, feign ignorance about what it is that we're working on. Usually, if you've seen stuff like this built, you know, like, how this got built and why it exists. But you wanna be able to see, like, was this a a resume builder that somebody thought was a good idea, or is somebody a true believer who thought that this is the right way to do something? So who defends it the fastest? And that'll be the person def like, definitively, that will be the person who came up with the idea. And if it's all that one person, you know you have group type a in my book, which is one person who has a lot of confidence and seems to dominate that that group organization, they probably believe very strongly that they're the smartest person in the room. And that could be true.

Ben Wilson [00:09:09]:
It could be, like, the most technically competent person. It could be the most senior person. But you really wanna feel out that dynamic and look at other people as you're asking the questions just to see their face, like their facial expressions. Are they smiling when you're asking that one question that's like, hey, this might be dumb, but, like, why did you build this? And if you get some people who kinda, like, grin back or roll their eyes, you're, like, okay. There's other people in this room that know this is stupid. And it like, the reason we're talking about this and the dumpster fire that it is is because these other people can't actually talk to that that lead person who seems to have all the great ideas, the genius in the room. So it then becomes you know, navigating around that, there's ways to do that that I found are successful. But then you also have, like, the type b environment where everybody's answering collectively as a group and giving the same sorts of answers.

Ben Wilson [00:10:11]:
So it's a very very cohesive team that came through group think together to think that this is a good idea. And there's no one person that's kind of dominating as as the, the voice of why these decisions were made. That takes a different tact of eventually convincing them. Not that, like, my idea is the right way, because I I never approach anything that way. I I wouldn't even be able to count how many times I've been wrong. But the idea of what I'm what I'm doing over time with working with that team is to figure out what don't they like, why did they go down this path in in the first place, and then get their ideas for how to get out of this situation.

Michael Berk [00:11:09]:
Got it. How do you do this over Zoom, especially with people who have cameras off?

Ben Wilson [00:11:16]:
I always used to ask that people turn them on.

Michael Berk [00:11:22]:
Because that's your line? It's like, hey. We're all gonna be friends here. Let's turn our cameras on, or is it turn your cameras on now. What is what's your what's your line?

Ben Wilson [00:11:30]:
I'm not a threatening guy. That would show I'm kidding. A lot of people have told me that I'm threatening. But, definitely not do, like, an antagonistic, like, hey. I've got my camera on. Why isn't yours on? Yeah. That doesn't set a good tone. But I don't think I have, like, one particular pattern of it.

Ben Wilson [00:11:53]:
Sometimes you don't even have to say anything. You just join the meeting, and people start doing intros, and you just explicitly turn your camera on. A lot of people see that, and they're like, oh, he wants us to see it, you know, his face, so I I should do that as well. A lot of times that just works.

Michael Berk [00:12:11]:
Yeah.

Ben Wilson [00:12:11]:
We have policies internal in in engineering and Databricks that most people will turn their cameras on. It's, like, the expectation. Because being able to see somebody's face when you're talking to them is very effective for communication.

Michael Berk [00:12:25]:
Yeah. Way more efficient. Okay. So step 1, get your bearings, read the room, figure out who is who. Let's say we throughout that process, we've determined that this new feature is horseshit, and it should not have been invested in. What are your thoughts on proceeding with next steps in terms of changing behavior?

Ben Wilson [00:12:49]:
So instead of well, I'll tell you what doesn't work. And I know it doesn't work because I've tried it, early in my time with talking to people. Walking in being the answer person and just adopting a dictatorial tone of, like, this isn't how you should do this. This is, you know, here's the proper way to do this. You're gonna get maybe 20% of the time, people will be in such a state of dejection that they actually will listen and just want you to tell them how to do it. The other 80% plus time, you're just gonna get resistance. You're gonna get anger. You're gonna get people that are actually just flat out offended.

Ben Wilson [00:13:38]:
Not a lot of people wanna hear straight out to the gate of talking to somebody who's who they feel is an expert coming in and being like, you guys suck. Like, everything you do sucks. That just feels terrible, and it's not productive. It and it's not true. You know? They're professionals. They've built something maybe without guidance that they thought was solving a problem that was the right way to solve it. But it's they obviously aren't in a place where they feel like what they've done is good. So you get a lot further by just listening and being understanding with them, particularly in tech.

Ben Wilson [00:14:22]:
A lot of people don't think that that's a factor, but if you're somebody that comes in, even if you are an expert and you're just being kind of a jerk and calling it like you see it and being blunt like that, there are tons of people that behave that way. And what you're gonna inspire is fear. And you're gonna inspire people to be like, okay. This person has told me something that I kind of already knew, but they're saying it in such a way that makes me feel bad or makes me feel like I'm incompetent. You've now set the stage for expectations to be not a an interaction or relationship with you that is team based or collaborative. It's now I'm coming in to tell you how to fix the problem that you created. You get a different dynamic when you do that. Will you get the same out will you get an output that solves the problem? Sure.

Ben Wilson [00:15:20]:
They'll get something else, some other way of doing something. They're not gonna like you. And I know a lot of people think that that doesn't matter, but that matters more than anything else. So if you are interacting with the team and they are really completely inspired by how you handled that situation. Like, hey, this person is, like, they're really nice, and they're really helpful and collaborative, and they wanted to work with our team. You've now built that relationship with them where you can talk about other things other than this immediate problem that they need to solve, but they all feel like the solution that is gonna be built, they were a part of. Because you're listening to their ideas and you're incorporating their ideas. They're not idiots.

Ben Wilson [00:16:08]:
You know? They they all have minds, and they can come up they just don't have as as broad of experience maybe as as a consultant would. But they they can think, so leverage that. And a lot of times, you're gonna get ideas from them that you never thought of. So best idea wins. Like, hey, let's all collaborate on this. And the the sooner you establish that rapport with a team that has kind of done something that they've gotten off the rails with, the sooner you now have a more cohesive team because we as a species love to solve problems. The only thing we love more than solving problems is to solve problems together. And that goes across all personality types.

Ben Wilson [00:16:54]:
Everybody loves that.

Michael Berk [00:16:57]:
Yeah. That does make sense. Alright. I had a few reactions. One of them is I completely agree about getting people to like you, and I think it's up to your own style. I typically am the small talk guy at the beginning of the call that says some, like, stupid funny joke, and then we all laugh, and then we start working. But there are many different ways to do it. That's just a a a vibe that has worked for me.

Michael Berk [00:17:28]:
Another point is you said that assuming that someone is an idiot or that they built something wrong is not correct. I think that's so vital. Empathy is about truth seeking. It's about being completely honest. And as your as an employee, seeing the truth is sort of your job. I don't think there's any or there are probably I I'm going out on a limb, but I think there's very few roles where seeing the truth is not important. And so it's not about being nice in my opinion, although, like, hopefully, you're a nice person. It's about going to the root cause, the root truth.

Michael Berk [00:18:06]:
And no one would build something because they're dumb. That's not a motivation. People would build something because they want a resume booster because their boss told them to and they were too scared to push back. Because 7 of like, there's many reasons, but being an idiot is just not the truth. So don't go off that as a first principle for action.

Ben Wilson [00:18:24]:
Mhmm.

Michael Berk [00:18:25]:
Those were a couple things. But, Ben, I had a a question to turn it back around to you. So alright. We've read the room, and we've sort of set more of the stage for, understanding motivations. Do you tell people when they're wrong? And if so, how?

Ben Wilson [00:18:49]:
It's much more effective in my opinion, and high functioning tech organizations do this by nature of how they do business. If you walk up to somebody and say, hey. What you built sucks. You could interact with somebody who has the fortitude to take that in stride and agree with you and be like, yeah. Let's fix it. There's not a lot of people that I've met that can do that and are, like, genuinely interested in just making something better and owning up to the fact that, yeah, I made all these, like, these mistakes. I created all these problems. Yep.

Ben Wilson [00:19:34]:
That was dumb. I'm gonna I'm ready to fix it. You're generally gonna get people that you're either gonna fake that response, but then their their feelings get hurt, you know? And that's that's important thing to to kinda be aware of. So the better way that I have found that works in a 100% of situations is to have them get to a point where they admit themselves to you openly without you prompting them to do it. You don't lead them down a path of, like, tell me that this sucks. Come on. I wanna hear that that you know that this is terrible. You get them to just admit it after going through the process of talking about it.

Ben Wilson [00:20:20]:
So usually, if somebody built something, there's a reason they built it. So you get down to those, as you said, like, first principles of projects. So sometimes you can say, hey. Can we can we take a look at the original design of this? Like, where's the documentation about the project plan for this thing? The vast majority of times, you're gonna have nothing come back. You'll or you'll get like, oh, it's more organic. We built it over time. Like, even if you can see that, that's fine. But you got to lead them down this path of realizing that there's a better way to do stuff.

Ben Wilson [00:20:58]:
So if you ask questions about the things that you know is a proper way to build projects and they don't have it, they'll start thinking like, wait a minute. We don't do that. Is that how things are done or how things should be done? And sometimes they'll ask like, well, if we had the design, would this event ever been built? They'll think that in their head. Like, we probably never would have built this if we had put it on paper first.

Michael Berk [00:21:26]:
Wait. Let are you down to do a little role play?

Ben Wilson [00:21:29]:
Sure.

Michael Berk [00:21:30]:
Cool. Alright. I am I as you were talking, I was thinking through the worst abominations that I've built over the past 2 years at Databricks. Of course, I've never done anything bad for a customer, so we'll just talk about an internal tool. This internal tool, was born out of a really great need and was mediocrely built. It was the best I could do at the time, and they're also sort of hard problems. And since then, we've iterated upon it and have a a very robust solution to solve this issue. So it's now been solved by a team of several primarily field people, but maybe some engineers as well.

Michael Berk [00:22:06]:
So this tool was a TPC DS benchmarking one clicks solution. And what all of that means basically is when you're looking to benchmark whether a a warehouse, so thanks, Snowflake, BigQuery, Databricks SQL, you name it, whether a warehouse performs, there's a bunch of representative datasets and representative queries of OLAP workloads. So online analytical processing, that's typically what an analyst would would run against, to to do their day to day job. So this dataset and these sets of queries are meant to be representative of that type of workload. And my tool, again, was born out of necessity. I had worked with a couple customers on benchmarking against warehouses, Databricks warehouses specifically, and, setting it up is hard. Doing concurrency is hard. And so what I look to do is create a one click solution that basically, if you just install this library and press play, it will be a configurate driven, solution that would actually go and load test against a warehouse.

Michael Berk [00:23:13]:
Mhmm. Since doing that, I've realized that it was a complete and total waste of time. It got no adoption whatsoever, at least waste of time from that aspect. I learned a ton, generally had a good time doing it, but, and met a lot of cool people along the way, but it got literally zero adoption. And we canned it and have done it properly. So let's roll back a year. Say, I have just completed this. I just did my tech summit talk.

Michael Berk [00:23:39]:
I'm really proud of it, and I have my heels dug in that this is the best solution ever. What would you do to change my mind?

Ben Wilson [00:23:49]:
I would start with that same question, like, why did you build this? And not in a rude way. I wanna hear your perspective of what problem were you solving.

Michael Berk [00:24:00]:
So the scenario is multiple customers had asked, me to do benchmarking against Databricks SQL warehouse. I ended up building tooling and then reusing this tooling across customers. And after the, like, 3rd customer or whatever, I went to the Databricks SQL SME channel and said, hey. I've done this 700 times. Would this be a good thing to repurpose and maybe put in an an internal repo that's public? It's currently public as well. And, hopefully, other people can use this for their own benefit.

Ben Wilson [00:24:33]:
Sounds legit to me.

Michael Berk [00:24:35]:
I agree.

Ben Wilson [00:24:36]:
Yeah. That's the purpose of code. Right? Like

Michael Berk [00:24:38]:
That it Sure.

Ben Wilson [00:24:39]:
Make it so that you don't have to do a bunch of annoying stuff over and over again. So far, so good. How many people have used it so far?

Michael Berk [00:24:48]:
Just about, like, plus or minus 0.

Ben Wilson [00:24:52]:
And how many people will have to do this workload at the company? Like, 4 customers.

Michael Berk [00:24:58]:
How many people or how many projects? Either way, it's a lot.

Ben Wilson [00:25:03]:
People and projects. Are we talking 10, 100, a 1000 in the last year?

Michael Berk [00:25:09]:
That is a great question. Every single Databricks customer account of sufficient size would have a warehouse play, and we want to basically compare Databricks performance with some other warehouse. So Databricks has 10,000 customers. Let's say let's say 2,000 of them are mature enough, for this tool, but theoretically many more. And let's say so let's call it 2,000 projects over, I don't know, 300 people, 200 people.

Ben Wilson [00:25:40]:
Okay. And how many times does it run per customer?

Michael Berk [00:25:47]:
It it my tool has run 0 times because no one uses it. But how many times is this scenario run? It's typically once at the initial compete when you're deciding what tech stack to land on.

Ben Wilson [00:26:00]:
Cool. So it's a onetime use at certain customers and but a whole bunch of people would would have to potentially use it.

Michael Berk [00:26:09]:
And it's and it's very expensive to do right. You requires a lot of subject matter knowledge, and, also, it's not trivial to set up if you're gonna do it right. So it's it's pretty time intensive per customer.

Ben Wilson [00:26:21]:
And this tool is gonna acquire the datasets, load them in a specific way that's transparent to the user, give you a bunch of reports about how everything is set up, and then it has an interface that allows you to just run the tests with some preconfigured infrastructure.

Michael Berk [00:26:38]:
Yeah. Like, you have 6 parameters like concurrency, max load. You can also do load shapes. And, yes, everything is open source and transparent.

Ben Wilson [00:26:49]:
And how many customers demand this be run? Is this something that consultants do to prove a point, or is this something customers demand is run before they accept?

Michael Berk [00:27:02]:
That's a great call out. I think probably, like, 30% of customers demand, 70% of consultants use it to sell.

Ben Wilson [00:27:11]:
And how much time savings are we talking about? Is this, like, without using the tool, this is gonna take 2 weeks of effort of 8 hour days? Or is this, it's annoying to do it manually, but I can get it done in a day?

Michael Berk [00:27:27]:
Well, it's a twofold issue. The first is having the subject matter expertise to do it right. That like, learning all of that is months of work, on top of a full time job. And then input

Ben Wilson [00:27:40]:
So when you say do it right, does that mean that the report result will be completely erroneous and non useful? Or if you do it wrong, you still get a report that just doesn't happen to be accurate.

Michael Berk [00:27:59]:
There will be a report. The report will be erroneous, and that may lead to incorrect conclusions.

Ben Wilson [00:28:06]:
Okay. So that's the the first problem that I see with providing tooling for something. So if somebody can do the job themselves, but they still get an output that may be within some margin of error that still makes the product look good. There's no incentive to use something that is

Michael Berk [00:28:28]:
Let me clarify. It their result would make the product look worse or just unbelievably better. It's it's not within a margin of error, typically.

Ben Wilson [00:28:38]:
Are we talking worse as in, hey. If you set this all up wrong, it looks so bad that no customer would ever use it. Like, hey. This is a 10,000 times less performant than your doc your, like, your your marketing documentation says.

Michael Berk [00:28:54]:
10000 times is a lot, but I've seen that happen multiple times where it's a couple orders of magnitude worse than it actually is

Ben Wilson [00:29:02]:
due to, like, that's not a problem. Does a customer immediately say, I'm not gonna use your product?

Michael Berk [00:29:10]:
Sure.

Ben Wilson [00:29:12]:
So if they do that, does the person who's running the test then phone phone a friend and say, hey. Why is this why is this so broken? And then some expert comes in and fixes it real quick, gets the test to run properly, and then it No.

Michael Berk [00:29:26]:
It's usually they're like because it's an easy answer, And they are if they're not already sold on the platform, it's an easy, like, this platform sucks. Let's go use a different one. The experts even came in and built the tool.

Ben Wilson [00:29:40]:
Interesting. So I'd say that's high stakes, and that's more of a people problem than a tooling problem if the people running the test can't detect that there's an issue with it. So that that would be more educational and standards based. So it seems to me like there should be a tool to to automate that if that's the case.

Michael Berk [00:30:01]:
I agree the need is there.

Ben Wilson [00:30:03]:
The other side is a little bit scarier to me, though, because that's gonna be an uphill battle fighting against. If you're convinced that your implementation is the correct way to do it and you've validated that and got signed off by a bunch of different people who are, like, the people who built the system, like, the engineering teams that built catalyst and and photon and stuff, like performance optimization engineers. They're like, yeah. This is the right way to do it. And then people can go and manually run the test, and it performs better than the results of your tool, everybody's gonna think your tool is broken. So if they get better performance by running the testing correctly, those numbers incentivize the customer to sign up even if they're incorrect. Mhmm. Because it makes it look like it's faster, better, cheaper even if the real product on their real data isn't gonna run that that well.

Ben Wilson [00:30:59]:
So that's where you're gonna get that's where this no longer becomes a software problem. This becomes an institutional, like, behavioral problem.

Michael Berk [00:31:09]:
Okay. Because that's

Ben Wilson [00:31:12]:
the that's, like, sabotaging, you know, for personal gain. Because the people running the test wanna get the cell. Right? They want money. They want that account. So if they see a faster number with the way that their expert air air quote expert is running it, they're gonna go with that number even if it takes them 3 days to run it manually in however they were doing it. So you'll probably never get anybody to use it if they think that they have a way to make like, cheat that number to be better.

Michael Berk [00:31:42]:
Sure. But, yeah, let let's say this is replacing the expert. They don't even know that there's a incorrect way to do it that would make the product look better.

Ben Wilson [00:31:53]:
Yeah. Then you've got a marketing problem, Assuming that your implementation is is solid, and it's like when you're talking about building a tool for people to use, the key to adoption on that 100% is you have to overcome a threshold of acceptance, which is related to how much easier is this than anything else. So if it's something that needs to get done, that's really important, there's a way that people know how to do it manually. You're creating a tool that's gonna try to do the same thing in an automated fashion. If your tool isn't just dumb simple or is faster than doing it manually and is not designed in such a way that it's so intuitive to use, you're probably not gonna get adoption, or you're gonna have to fight against that impetus. So I was like, yeah. I don't really need a tool. I know how to do this.

Ben Wilson [00:32:50]:
But if the tool is that good where it's like, hey. This is like a one liner to just instantiate this, and it does. It automates hundreds of tasks for me, and I don't have to think about anything. I don't need additional configs to be written. I just run this thing, and it it gives me the answer. Then it's like, yeah, it's pretty good. You're gonna see adoption. Right.

Ben Wilson [00:33:13]:
But now you're asking a question about, why isn't somebody using my tool? We're now moving kind of like, assuming that the tool is very easy and very high level and and automates all the the annoying stuff.

Michael Berk [00:33:26]:
Mhmm. We're

Ben Wilson [00:33:27]:
now getting into the realm of product design, which is, hey. We built this cool thing. Why doesn't anybody use it? We don't do that on our side when we're building new stuff. I mean, you saw it recently with, like, tracing. Right?

Michael Berk [00:33:45]:
Yeah.

Ben Wilson [00:33:47]:
The process of of building that, if we had just built it and then wrote docs and just did a release, maybe wrote a blog or something, we're not gonna get a lot of people using it or talking about using it. So what we do is you identify customers who have this need. Those are the ones that we were building it for in the 1st place, and you talk with them every week and have them test early builds and point out what's working, what's not working, what additional features are gonna be more relevant. And every week, you're continually talking to them until you get to that point where, like, hey. We think we have the first, like, official version of this that we're about to release. You have them basically give the thumbs up, thumbs down on this. And then any sort of bugs that are in it, you, you know, scramble and fix real quick. And then you do your release, and then you talk about it.

Ben Wilson [00:34:42]:
You do, like, a pseudo marketing campaign around this. You get the word out there. Like, hey. This is now a thing. It doesn't stop there, though. There's still work that's done after that release where you're doing promotion of this. And then can you're going back to the design board again. Like, how do we further lower like, everybody that we've had used this with the way that we are doing it have all these amazing things to say.

Ben Wilson [00:35:12]:
And they think this is really important and really great. Well, now you start looking at the product and saying, can we make this easier for people to use? Can we make it so that this just turns on automatically for them when they start using it? So when they need it, they already have it. Yeah. And that's a whole another product design that you're going through and then getting feedback on that. And maybe you have to make adjustments to the the thing that you built in order to make it work in that way.

Michael Berk [00:35:42]:
Alright. Let's take a bit of a step back, though. So, I 100% agree with everything you're saying, but how would you go about convincing me, the angry stakeholder that loves this product through and through, how would you convince me of your ideas for what the next steps should be? You just told me the next steps, but is that what you would do? You just outright say, alright. These are the 7 things that you should go try.

Ben Wilson [00:36:08]:
I'd start asking leading questions to see if you understand that that's maybe an effective way to see if you can get usage of it up.

Michael Berk [00:36:19]:
Alright. Let's hear it.

Ben Wilson [00:36:20]:
Let's say, how many people have you shown this to?

Michael Berk [00:36:24]:
Like, 2. Like, 5. I don't know. Small number.

Ben Wilson [00:36:29]:
So we'll we'll use the upper end. We'll say 5. How many people could use this tool?

Michael Berk [00:36:36]:
300 over 2,000 projects.

Ben Wilson [00:36:39]:
Okay. So what's your plan on expanding from 5 to 50, and how are you gonna do that?

Michael Berk [00:36:46]:
Oh, light bulb. I don't have a plan. I don't know. I've never marketed a tool internally. Ben, what do you suggest? Is that the the inflection point that you were looking for?

Ben Wilson [00:37:01]:
I mean, maybe. My homework to you at that point would be go find 5 more people and have them use it. But don't don't be there when they use it. Don't be in the room. Don't be hovering over their shoulder pointing out how to use it. Give them the tool and the docs, and just ask for candid feedback. Say, hey. Can you tell me if this sucks or not? And have them run it.

Ben Wilson [00:37:31]:
You're gonna get a lot of feedback. So if you self select the those first five people or those 2 people, that those are usually safety selections. That's just how our brains are wired. We wanna get somebody that we that kinda knows us and, you know, they're probably gonna be somewhat promotional in whatever we worked on. They're not gonna be blunt about things if it sucks. They're gonna sugarcoat things, or they're gonna have more positive than negative things to say. So it's good to get that in the first phases, but get people that you know are willing to give you feedback that most people wouldn't wanna hear and but tell them that as well. Like, hey.

Ben Wilson [00:38:21]:
If you see things that you don't like, like, please tell me because I wanna fix those. Or if you have ideas of ways to make this better, let me know. I wanna I wanna do that. But then after that first phase, think about it as, like, a a business to customer relationship. You might have a good relationship with them, but the reason that you're building things for them is because they're giving you money in order to build those things. So there's incentive there. Like, they stop giving you money if you make garbage. So you make less garbage, they give you more money.

Ben Wilson [00:38:57]:
It's it's kinda like how that whole thing works. And you can also do that with peers. You know, if there's somebody who who has to do this job, but they don't personally know you, if you engage with them, they're no they're no longer influenced by that personal relationship of, like, being nice effectively. They'll just be honest. And you might not get it from every single one of them. Some people just aren't naturally nice. They don't wanna say negative things about something that somebody did. But if you canvas enough people, you're gonna get some people who are just brutally honest, or just wanna be providing constructive feedback.

Ben Wilson [00:39:41]:
And they might say some things like, hey. I I don't know why you built this thing in Python. Like, why isn't this a bash script? They can just do all this, and then it would be simple, you know, and just run. Or it could be somebody giving the exact opposite advice of, like, why isn't this a scholar library that has all of these, you know, traits defined for this configurate? You can you get all all sorts of random responses, but if you collect enough data on this and you give them a series of questions to answer as they're evaluating it, and those questions might be like, will you use this? Yes or no? You know? Do you think this is a good idea to use at customer sites? Yes or no? What are the the five things you hate about this most? What are the five things you love about this most? You collect enough information from all of these random people. You're gonna get, you know, a of issues. You're gonna get a of good ideas that were in this thing, and you're gonna get, you know, feedback of, was this even worthwhile? Yes or no? But I would also ask, if you were presenting this tool as something that you've already built, and you're, like, you're ready to ship it, like, it's done. I would say, hey, Michael. Like, what were your what was all the feedback that you got when you asked, like, 20 people before you wrote your first line of code? Like, can I see all of those responses? And we could go through that together.

Michael Berk [00:41:13]:
Yeah. So in my scenario, I was simply solving an issue that I had and that I know because I'm a subject matter expert and a genius that everybody else has this issue. And so I just built tooling that solved it and then package it up in a nice, very easy to use, framework, and anybody who thinks that it's not perfect is wrong.

Ben Wilson [00:41:36]:
So did they say that when you like, before you built it? Like, in

Michael Berk [00:41:42]:
Well, no. They're just idiots. I didn't I didn't get feedback because people are dumb.

Ben Wilson [00:41:48]:
Okay. Let me tell you how we do it, and I'll just give you Nice. An explanation of of how how it's done when we build stuff. So that's where this conversation would go.

Michael Berk [00:42:01]:
Yeah.

Ben Wilson [00:42:01]:
I would I would get that because that admission that you would make, even if you gave that flippant response, which I know you wouldn't give that flippant response. But even if you had, internally in your subconscious, you're immediately gonna be thinking like, oh, no. He just asked me a question that I I totally should have done that before I wrote this thing. Like, gotten some sort of understanding of, like, is this a problem that needs solving? And if everybody was like, no, this isn't important, you're never gonna convince them that this is worthwhile. If everybody just dunks on the idea, move build something else. Like, go find another problem to solve. If it's something that makes your life easier and it works really great for every time you have to do this, use it, man. Build it.

Ben Wilson [00:42:54]:
But build it as lean and as as, like, low level as you possibly can. Bash script. Like, hey. I need to automate this thing because I hate spending 3 days setting all this crap up.

Michael Berk [00:43:06]:
Just get it done.

Ben Wilson [00:43:07]:
Write a script. Yeah. GSD code, man. All of us have loads of it in engineering. Just like, hey. I've got these random JavaScript scripts or Python scripts or bash scripts that automate something that I hate doing. I've got stupid stuff like I'm looking at it on screen right now. I have, like, many Mac users.

Ben Wilson [00:43:33]:
I have ZSH as my terminal manager. I've got all my ZSH for all my formatting and stuff, and I've got a very unique scheme that my brain has, like, enjoyed looking at to be able to find information very quickly in that. But I'm so freaking lazy, that every time that I, and I also hate using standard terminal on Mac because it's annoying to me about how the window layout is. And I like to put a bunch of plugins into it. So I use something called I terms. So I can have, like, 12 terminal prompts open all pointing to different repositories in different places on my operating system to automate stuff. But whenever that program updates or my computer restarts, it it uses a like, the standard template when you restart that thing. That's just nothing is basically enabled, and you just get a simple command prompt of 1 window.

Ben Wilson [00:44:30]:
So then I have to go and, like, create all these different windows split, you know, create split horizontally, split vertically, like, do all this nonsense to get my 12 windows back. And there's templating that you can do in Iterm that'll say, like, start 12 windows. However, you can't apply configurations through, like, basically saved scripts within that program. It just doesn't support it. So I just wrote a bash script that says, hey. When I start up or when the terminal starts up, execute this bash script, which just automates all of that, sets up my conda environments, my pie end. If I'm doing Scala dev work, it ensures that I'm using the like, it maps to one of the 4, like, JVM versions that I have installed. It's if I'm doing, like, JavaScript work, I, you know, con configure through a con like, a config file to, like, use node 20 for this.

Ben Wilson [00:45:33]:
But if I'm doing it in another environment, I can change that to be, like, use node 18. So that that's a GSD script. It sets up a dev environment. I'm not gonna build a tool, or I'm gonna, like, modify source code of Iterm in order to get it to, like, do this special thing. Because it just takes me, like, an hour to write that script that'll just settle this up for me.

Michael Berk [00:45:57]:
Okay.

Ben Wilson [00:45:58]:
So we all have tons of stuff like that. Never gonna share it with anybody, unless somebody asks, like, hey. I'm using Iterm, and I hate having to manually do this. They're like, hey, I I got a script. And then somebody's like, this is awesome. Great. I'm gonna change it a little bit because I don't like that your color scheme sucks. I'm like, sure.

Ben Wilson [00:46:17]:
Go ahead, man. It's fast script. So we do we share stuff like that all the time and have little things like that. So if it's just for you, build it how you're gonna build it. But you'll know whether you're gonna do it that way, or I need a Python library that it has, like, a high level API that has some configuration that I can pass in for these different edge cases for testing, get validation first.

Michael Berk [00:46:47]:
Got it. Yeah. Ben and I were recently at the data and AI summit for Databricks, and it's the annual conference, and we are part of, several dinners. And one of those dinners, it was really interesting to see him give advice, because I feel like a lot of junior people want to prove that they know things and then give answers. But there were a variety of scenarios posed by customers to Ben, and Ben simply said, well, when I was working with 16,000 customers back in my day, this is how we did it for all 16,000. You can do it that way or you cannot, but if you do do it that way, at least for those customers, it works a lot better. And that sort of it was a lot less involved than I expected your answers to be. It was, just like, this is how we've done it in the past.

Michael Berk [00:47:42]:
It's worked, like, 77% of the time, and you can do it this way and have 77% odds, or you can do it your own way and have unknown odds. You pick. Mhmm. So what is that? Just like to hammer home the the question about how do you get people to have the idea that you want them to have, it seems like you just set up a scenario that made that idea the logical conclusion, but you stayed very far removed from the actual decision. Is that sort of the the structure that you provide, or do you have more thoughts on that?

Ben Wilson [00:48:20]:
I mean, I I try to give people the benefit of the doubt when they're asking me a question they're not looking for I mean, they might be looking for the the just tell me what to do answer. And I'll flat out tell them, like, I don't know what's good for you. I I have not walked that mile in your shoes. I don't know what you deal with on a day to day basis at your company. It could be that if I give the answer that I think is right and you go and try that, you're gonna be like that dude's an idiot. Like, of course, this doesn't work. But if I give them options of, like, hey, I've seen things similar to what you're talking about, and here's what worked, here's what didn't work, or this is what a lot of people choose to do, ball's in your court. Now that process of what I've done is I've given them data that they can now act on.

Ben Wilson [00:49:18]:
I'm not giving them the answer that the data would inform for them to make a decision. And you're gonna find that people are more willing to listen and retain what it is that you're talking about if they've had to go through that cognitive load of making a decision, even if it's something that they don't realize they're making a decision on. But that's a way to do effective technical communication where you're proposing just giving information or giving more context to somebody that they can then make an informed decision on. And now they own that decision. It's not, well, Ben told me this because this is the way things are done, but more like, well, Ben told me about all these these stories of, like, things that he'd seen. And I kind of was going, like, leaning on that on that direction. And to begin with, I now that I have this additional context, I now I'm more convinced. So I'm I'm doing this.

Ben Wilson [00:50:19]:
That's what's going on in their head, and you can help them make a decision by giving them factual data that you've seen. And you can also do it with the the decision of, like, your project that we were just talking about. And by using those leading questions about, hey. If you want if you're wondering why people aren't using this, did you go through the like, a process that would have given you that data in order to determine make make that decision? And by doing that post hoc, like, you've already built the thing. And, like, if I was being blunt with you, I would have been like, who told you to build this? Like, why did you build this? And that's just rude. Right? That's not that's not productive. You're gonna feel attacked. You're gonna feel like, I'm not gonna listen to this guy.

Ben Wilson [00:51:10]:
He's just being a jerk. But if I ask you that question, like, how did you collect the information about why this was so important to build and and promote? And you'll have that inner realization, like, oh, so that's how this is done. But that'll now be a lesson because you're now aware of the right way to do that. You'll never ever forget that. You'll be like, hey. Before I write anything that I'm gonna try to get people to use, people need to tell me that they need this thing. Right. And then I need to involve them in the process of, hey.

Ben Wilson [00:51:47]:
I have version 0.0.1 that only does this, like, very small subset. Can you test this and tell me if it works the way that you expect it? Or how does this feel when you use it? Is this good? Yes or no? Would I use this? Yes or no? And you get that feedback before you've already invested a ton of time and energy into something, where if you wait till after you release it, and you get all that negative feedback, or people are just like, don't care, not interested, don't need it, you get that later on. You can't go back easily and fix that stuff, because you've already built the whole thing out. Right.

Michael Berk [00:52:25]:
Cool. So your way to reframe my very stubborn self who has dug his heels in and will not think that anybody's feedback is valid, You would just say, hey. This is how we do it at Databricks and maybe consider opening your mind and then hoping that I I make my own conclusion that you're potentially right.

Ben Wilson [00:52:49]:
I mean, I would continue to ask questions until I hear that from you. I wouldn't tell you like, hey. You should have done this differently. I'm waiting to hear that sentence of like, man, I really should have, like, asked for help or asked for feedback very early on and throughout the process if I wanted people to promote this.

Michael Berk [00:53:09]:
Cool.

Ben Wilson [00:53:10]:
And if I don't hear that, that's when I'll go into the I mean, by telling it in a personalized way of, like, this is how we do it at Databricks Engineering, that's partially true. This is how software is developed everywhere. Like, every big tech company that's involved in generating software for other people to use, that's how thing this is agile software development. This is it in a nutshell. Like, do product design, get feedback, release first iteration that's buggy and broken and is kind of a mess, get feedback, find bugs, do bug bashes, iterate on next, you know, version of it, get feedback, fix things, build new features, and you do that progressively as long as there's still excitement and requests for new features to be added. If it's gonna make you money or it's gonna make customers happier, it's gonna solve more problems, keep on building onto it. But if it solves everything that it set out to do, and you're not getting any more requests from it, but, you know, you can do instrumentation track, how many people are using it, or just ask people, like, hey, are you still using this? Whatever. Having that that feedback loop tells you when it's been long enough when you it gives you a stopping point of when does this become feature complete, and it's now in maintenance mode.

Ben Wilson [00:54:35]:
And then also, when do I turn off maintenance mode? When is this no longer relevant, and I don't have to support it anymore, and I can just delete this thing? Because nobody needs it anymore. And that's the whole it's not something Databricks specific. This is, like, just software like, product software development

Michael Berk [00:54:56]:
Yeah.

Ben Wilson [00:54:56]:
At its at its core. Right. But a lot of people a lot of teams don't do that because they they don't have customers of their software or of the things they're doing. They have a problem to solve. So when you're talking with the team that is maintaining some applied usage of a framework, Like, hey. We're trying to predict fraud, you know, at our company, and we have this model and all this this data that's coming in. Can you solve that problem in a very simple way that's easy to maintain, is flexible, and sure. But to your end customer, which is the fraud department and, you know, your CFO who's worried about how much money is going out the door from fraudulent activity, they don't care how you solve that or what you built to, like, solve that.

Ben Wilson [00:55:56]:
So those teams that are working on building software to solve a problem like that, they don't have any incentive to go and ask somebody, like the CFO. They're not gonna be like, hey. Do you think we should build a framework for this, or should we just, like, build the simplest thing possible, like a notebook with its version controlled that like, we have a notebook that trains the models. We have a notebook that handles tracking and logging of the models and versioning. We have a notebook for, like, deployment. It's very easy for us to read and write comments in and, like, get it so that we can collaborate on this. Or should we go and build, like, a fraud detection framework that is a, you know, compiled code that now has to go through this very complex CICD process in order to, you know, be run-in production, and it's a nightmare to maintain. Nobody knows, like, when it goes awry, nobody knows how to debug it because it's so complex.

Ben Wilson [00:56:59]:
That end customer of theirs doesn't care. They care if it's wrong, but that is tangential, completely tangential to how that thing is built.

Michael Berk [00:57:11]:
Right.

Ben Wilson [00:57:12]:
But their fellow customers in that respect, the the person who's working on that project, the customer is their peers who are on call. Can they fix this thing when it blows up? Right. Like, really quickly? Can they close that ticket that comes in? Like, yep. It, it threw an exception on line 347, but I I can totally fix it in half an hour and deploy the fix and everything's good. But on the framework route, it's like, oh, we're gonna we're gonna have to take, like, 3 or 4 days to to fix this because this is pretty complicated.

Michael Berk [00:57:51]:
Right.

Ben Wilson [00:57:51]:
And yeah.

Michael Berk [00:57:53]:
Heard. Okay. I have one more question about the framework. So the steps that are materializing in my mind, first is read the room. 2nd, root cause the issue. 3rd, collaboratively brainstorm on solutions to the issue, and then hopefully go do them. But it feels like there should be a 4th step of gathering whether the the the input that you provided actually led to a solution. And if you work in an organization and you're, for instance, doing these steps within your team, it's pretty easy to see, alright, did the feature get shipped, did the process change, whatever it might be.

Michael Berk [00:58:30]:
But do you have a a good way to go back and gather feedback on how successful your input was? Because that would help you iterate as a person as well.

Ben Wilson [00:58:42]:
I don't know if there's a a good way to do that or a point of doing that that's based in, like, collecting of data. I don't I've never worked in a team that actually does something like that, so I don't I don't really know. But I know that, like, teams if the team that I'm on right now, like, nobody cares about stuff like that, we care about what the team has done. And, you know, there's individual recognition for, like, hey, this person built a super cool thing. That's for their own personal career growth and getting exposed to new things to make to have them grow as an engineer and wherever they want their career to go, you know, you work on projects that are related to that or roles within a project that are adjacent to that. But on a, like, evaluating, hey, who built this one feature, and how is it performing over time, and what was their impact to to the creation of this over time? It never is clean like that.

Michael Berk [00:59:51]:
Right.

Ben Wilson [00:59:52]:
It would be impossible to disambiguate all of the other influences that happened because once you ship it, it's now in maintenance or it's in feature improvement. So somebody else is gonna touch it, Or somebody else might go and refactor the whole thing. So now you're like, well, it's not one person. It's like 4 people. There's one person that originally built it, but a bunch of people have changed it and improved it. So it's really the team that is building these things.

Michael Berk [01:00:22]:
Right.

Ben Wilson [01:00:22]:
Okay. That's a good point. It's always a team event. It's a team event once you go into production. It can be a team event for big things that are like, okay. There's too much work here for one person to do. So that person might have been a project lead for something, and they delegated respond like, different implementation things to other people. That happens all the time.

Ben Wilson [01:00:47]:
But once it ships, like, it's just the team that supports it or sometimes the entire organization if it's super critical.

Michael Berk [01:00:56]:
Right.

Ben Wilson [01:00:56]:
That would be like trying to attribute, like, well, what one engineer at Databricks can we attribute, like, the performance of the data frame API? Right? Like Yeah. I know who came up with the idea. I know who built the prototype, but it's probably a 100 people have touched that code base that it falls with that, at least.

Michael Berk [01:01:24]:
Yeah.

Ben Wilson [01:01:24]:
And now it's probably upwards of 400. So, yeah, it's nobody looks at stuff like that. We look at the project as a whole and the entire organization to say, like, are we keeping this in a state of constant improvement over time, and are people successful with this, and how much usage do we have with this?

Michael Berk [01:01:48]:
Cool. Alright. I am absolutely crystal clear on how to influence behavior within an organization. Do you have any closing thoughts before I wrap?

Ben Wilson [01:01:59]:
I mean, I think the biggest part of it when having count like, these sort of difficult conversations is to really just double down on empathy. Like, put yourself in the position of that person and think about how you would wanna collaborate with somebody to fix something that you screwed up. Do you want somebody who is interacting with you that feels like they have your back and nobody cares whose fault it was. Like, it shouldn't be a blame filled thing. It's irrelevant. If you want somebody to grow and you want somebody to to learn these new techniques, work with them. And that is like, the ability to work with somebody and get them to see alternative ways to think about something is just giving them more wisdom, and they'll learn more and they'll get better. So that 5, 6 years from from that point that you're having that conversation, they're gonna do that for 5 other people.

Ben Wilson [01:03:04]:
And all of a sudden, you're you're building more of this sort of awareness and knowledge within that organization of people being like, hey. Before we go and do something like this, let's plan this out. Let's talk to people. Let's get feedback. And then maybe we'll go and build it this this way. Never underestimate the ability or the power of peer review.

Michael Berk [01:03:29]:
Heard. Okay. Cool. Well, today, we talked about how to influence behavior within an organization. It seems like it's a 3 step process or at least to get started. The first is read the room. If you're part of the team, that should be easy. But if you're a con external consultant specifically, it's important to have cameras on and try to figure out whose decision actually matters.

Michael Berk [01:03:53]:
2nd is root cause the issue. So figure out the true reason why there's an issue. And then 3rd, collaboratively brainstorm on solutions and then do them. Some do's and don'ts on the do's side, listen and be kind. This is also called having empathy. And then when looking to change behavior, try to get the person to have the idea. The reasoning for that is they'll champion the idea a lot more and build on it a lot more easily if they think it's theirs. One little tip when, reading the room is figure out who talks the most.

Michael Berk [01:04:26]:
Typically, whoever speaks about your questions, they are the owner of that project and incepted that project. And then finally, if all else fails and you're getting a lot of pushback, you can take a step back and say how you have solved the solution in the past. Then one very tangible don't, don't come in and be the answer person without context or empathy. No one likes a smart ass, myself included. And instead of being the answer answer person, give anecdotal pieces of data that lead people in the right direction. Anything else?

Ben Wilson [01:05:00]:
Excellent recap.

Michael Berk [01:05:02]:
Cool. Until next time, it's been Michael Burke and my cohost.

Ben Wilson [01:05:05]:
Ben Wilson.

Michael Berk [01:05:06]:
And have a good day, everyone.

Ben Wilson [01:05:08]:
We'll catch you next time.