Michael_Berk:
Hello everyone, welcome back to another episode of Adventures in Machine Learning. I'm one of your hosts, Michael Burke, and I'm joined by my lovely co-host.
Ben_Wilson:
Ben Wilson.
Michael_Berk:
And today we're gonna be talking about a thing that I know legitimately nothing about. So I am going to be learning along with you guys as we ask Ben scintillating and amazing questions about ML frameworks. So to start off, Ben, can you define the difference between applied ML? and an ML framework.
Ben_Wilson:
Sure.
Michael_Berk:
Thank you.
Ben_Wilson:
And just to forewarn everyone, I don't claim to be an expert on them any more than anybody else who builds them and supports them. But there's a general principle of design around these things and about an approach to development that I've noticed personally. There's a big difference in how you approach these two things that you just asked about from a development perspective. So getting that out of the way. A framework is generally a generic structure of code that allows you to perform certain tasks. in a way that you don't have to worry about or concern yourself with orchestration. And what that means is there's some repeatable tasks that are common to a lot of applications of software. Let's say you wanted to you wanted to get data from a database. And you wanted to be able to take that data and use it to fit a model and then have some sort of scoring on a holdout validation set that you could know how good the model is. Now, without a framework, you're going to be writing the direct interface to the APIs of the library of that database. you're without a framework for the ML perspective, you're going to be implementing the algorithm. So let's say we're doing random forest. Well, you're going to go and look at what the components are from how the random forest algorithm works. And you're going to write the implementation about like, hey, here's how I do differential, you know, differential entropy calculations. And this is how I'm going to determine what the variance is. And you're going to write those equations in code. And Then for scoring, you're going to, that one's probably the easiest one. You're just going to run a holdout validation set through your model implementation of the, the scored weights, which by the way, you would, you would be writing all of the code for that too, to record that, store it in an object that you could handcrafted. And then you would just perform the transformation through the equations based on the learned weights and, or the learned decision tree. And then you would write the very simple scoring calculation of a row-wise iterator, and you would get a result. Now, 15, 20, 30 years ago, when people were doing what is now referred to as machine learning, that's how implementations were. And there are still shops out there that do that because they need to. They have some bespoke algorithm. They you know, data storage paradigm that they just, there's no framework that's out there. There's nothing in the open source and maybe the company's not big enough to have a software development team that builds a framework to interface with these different systems. So they have to roll all their own. And it's a lot of work. Now, a framework of the three things that we just talked about, in the example of Python. If we want to interface with the database, well, there's the SQL Alchemy framework, which allows for a high level API that allows you to interface with different database backends. And it will auto-generate the type of requests in that engine's, you know, DML language, data manipulation language. It'll issue the query, it'll handle creating a cursor that you can, you know. basically open that connection to that database, issue the query, and it opens a pipe for data to be returned to wherever you're executing this code. And it handles a lot of other things. Like if you need to update data, it handles CRUD operations for you. It does a lot of the really nitty gritty details of database management from a non-user perspective, but more from like an engineering perspective that an end user shouldn't have to worry about. So we would use that framework. And then for data manipulation and feature engineering work, we might use something like a native Python in a data manipulation framework. The most famous one that most people are familiar with is Pandas. Pandas is a high level abstraction around messing around with data in a tabular format. And then... When we want to talk about the model, if we're building a random forest, we have scikit-learn, which is a framework around these implementations of low-level algorithms that we don't have to worry about reinventing the wheel there. And then for scoring, we could use sklearn as well. Or you could use many of the scoring algorithms that are in packages like stats models. So all of those things that I named, they're all frameworks. and they provide that high level abstraction over complex code so that you don't have to keep on
Michael_Berk:
Thanks
Ben_Wilson:
rebuilding
Michael_Berk:
for watching!
Ben_Wilson:
that for every project. Applied ML is that project. Regardless of whether you build everything yourself or you use frameworks, there's going to be a custom combination of the utilization of either frameworks or the raw implementation in order to solve a business problem. And tying the... that explanation into the discussion we're having today, there's a big difference in how you approach building those two things. So if you're building everything for applied ML or just applied software in general, you're thinking about high specialization to the task at hand. So if we're building this particular model and we're trying to... classify whether our users are committing fraud or not. Well, there's gonna be specific data manipulations that we're going to do, and maybe post-processing from the prediction operations that we're gonna do on those results that are highly specific to the task of detecting fraud. And our code is gonna reflect that because it's needed for that particular use case. Whereas in a framework, you don't build stuff like that into a framework. You might build a tool or utility that allows somebody to craft that logic in an easy way, but you don't bake into a framework, all of the nuances and esoteric needs for every project that's out there.
Michael_Berk:
Got it. So it sounds like a framework is like a Python package and a not framework uses those Python packages to implement stuff. Is that more or less correct?
Ben_Wilson:
I mean, you could package up your project code and create
Michael_Berk:
Valid.
Ben_Wilson:
a wheel. So it's more the higher level concept of, is this package or artifacts that you're creating this computer application, is it intended to serve one purpose or many purposes? And frameworks are meant
Michael_Berk:
Thanks for
Ben_Wilson:
to
Michael_Berk:
watching!
Ben_Wilson:
support many purposes in an abstract way.
Michael_Berk:
Got it. Okay, so that makes a lot of sense. And an applied ML solution is just for that specific use case. It's usually
Ben_Wilson:
Yes.
Michael_Berk:
not meant to generalize.
Ben_Wilson:
I mean, I've seen people do that. I've been guilty of that early on in my career. I'm like, well, if I, if I built some, you know, I'm going to create this class in this, this project and it'll allow me to do all of these, these different things and I'm going to make it robust to support, you know, let's say for modeling purposes, like, oh, I need to I'm going to build this interface so that I can do K means and K median and PCA and random forest and linear regression. But the actual project just uses a linear regression model. So I wasted two weeks of time building this module in an abstract way that lives within the project code that 90% of that code is never going to get touched. It's dead code. It was a learning experience. It was kind of valuable there, but it was a waste of time. And it's code that was getting tested for no particular reason.
Michael_Berk:
Got it. So for on the applied ML side, it seems like we're looking for concise, yet thorough and effective solutions. So with that, as a lazy data scientist that I am, I would love to cut some corners. So maybe I don't want to write a ton of tests. Maybe I don't want to have a bunch of object oriented structure. What are some corners that you think you can cut on the applied side that you can't cut on the framework side?
Ben_Wilson:
On the framework side, keep in mind that nobody's perfect. No team is perfect, no tool is perfect. Bugs happen. Test coverage is never perfect. It never will be. Things break because we're human. Well, and as frameworks grow in size, they grow in complexity generally. And the more moving pieces you have, the more probability that... some use case that's out there, you're not going to think of testing it or it's going to break in an unexpected way, which you then have to fix. But the whole concept of cutting corners with frameworks, the way to cut corners in framework development is don't try to solve all the problems. You're never going to do it. And the pursuit of doing that is going to... increased complexity for your project at the sake of stability and the ability to just keep it going. Any project's going to hit a critical mass at some point where no matter how many humans you throw at the problem, how many hours you throw at it, you're never going to be able to maintain it anymore. And there are notable packages throughout history that have hit that. point and even companies start throwing tons of resources at them to keep them going before eventually abandoning it for something simpler and more sophisticated but less complex. So from the framework side, keeping it simple and keeping the code in a testable, readable, like human readable way is paramount. widely accepted design principles and a sort of a style guide that There's a contract sort of either unspoken or it could be written Amongst people that are maintaining a framework of this is how we're gonna write code. This is how we're gonna test our code This is our standards and those standards are enforced by pull requests and code reviews So the group collectively works towards maintaining a high level of excellence through having everybody check each other's work and nobody's above that. You don't merge code to master blindly if you're working on a framework. Now from an applied ML perspective, ways to cut corners is don't test framework code. You don't need to. The people maintaining the framework, that's what they already do. Now, There are times where there's bugs that get released in any framework. There's gonna be some unexpected thing. And you having a unit test in your applied framework that's testing functionality of that framework code, it doesn't buy you anything because you're not gonna be the one fixing it. I mean, you could write, you know, draft a pull request and submit it to the... open source framework package and say, hey, I have a fix for this bug that we detected last night. Can you take a look at it and merge it? That'll gain you favor for you, for the maintainers. They could be like, wow, this person's awesome. They saved us from having to fix this. Great. So if that's your goal, awesome. There's better ways to test that than putting it in your application framework code. So Avoiding testing functionality that's part of that core module is probably worthwhile. So if you're testing something in pandas and you're doing some manipulation, you're like, hey, I have this if conditional, I want to update index positions where this value is above this value and clean up outlier data. What you can do is take your actual that has outliers in it and run it through your logic for making sure that you're catching those and resetting them to the appropriate value. But limit the test to just that. So it's a pseudo integration test, sort of. It's like, hey, our raw data coming in, we know it's bad. Let's make sure that we don't, like after we run it through this process, that our custom logic for that column conditional. is doing what we want. So you're testing your if statement basically. But what you don't want to do is do stuff like, hey, after I change this column, make sure that I still have columns in the data set from before I did this manipulation. Because you're now testing pandas. Does pandas randomly drop columns, yes or no? We should hope not. Python practitioners would know that within an hour of a new version of pandas getting released. That would never happen because they do extensive testing before releasing anything. So you don't need to test stuff like that. You don't need to test, hey, I added a column, do I still have the right number of rows? Of course you do. Pandas isn't randomly dropping rows of data when you're doing a column operation. So it's just wasted. Wasted cycles of effort to come up with tests like that, and it's wasted computation, wasted time, and it's just more lines of code. And if there's one thing that I could tell a lot of people who are doing applied ML, your project's value does not, is not related in any way to the number of lines of code of your implementation. If anything, you should be striving for the inverse of that.
Michael_Berk:
Thanks for watching!
Ben_Wilson:
Strive for the fewest lines of code. to solve the problem that you possibly can. But I've seen a lot
Michael_Berk:
Yeah.
Ben_Wilson:
of people try the other thing. They're like, look, I've got 65,000 lines of Python code in this project. It's like, why? Like you could probably do that entire project in less than a thousand lines of code. Like you don't need
Michael_Berk:
Thanks for
Ben_Wilson:
an
Michael_Berk:
watching!
Ben_Wilson:
object oriented to implementation of churn risk modeling. You know, these 37 classes that you've created serve no purpose whatsoever. Cause there's no abstraction here. Just convert this to functional programming. Write 20 functions that do all of this. Test those functions to make sure they're working right. And create a main method, and you're good to go.
Michael_Berk:
I personally strive for the biggest set of files because I like maximizing upload time. So whenever I'm committing to git, I want it to take as long as possible. So that's the other angle. If you like working very inefficiently, just write as much code as possible. I promise you it's super fun.
Ben_Wilson:
And everybody looking at your code is going to hate you
Michael_Berk:
Oh,
Ben_Wilson:
because
Michael_Berk:
that's so true,
Ben_Wilson:
when
Michael_Berk:
yeah.
Ben_Wilson:
you do a PR and you look and you're like, hang on, you changed 6,500 lines of code in this PR. Three other people, four other people, five other people, they have to read through all that stuff and make comments and try to figure out what you tried to do. It's risky and it's a great way to create bugs.
Michael_Berk:
and create enemies within your team. Okay, cool. So let me sort of see if I can break this down into layman's terms or an analogy. So frameworks are sort of let's say we're going to build a house. That's our project. Frameworks are let's say the brick or the raw materials, the wood, the tile, etc. that you're going to be using. And you should have faith that those things won't crumble and break. They're the building blocks of whatever you're making. And you should trust them and you don't need to test them. But there's a brick maker, there's a tile maker, there's a lumberjack, and all of them, it is their responsibility to ensure the quality of the component parts. So when we're building up applied models, we leverage component parts, but there are people that do have to go in and actually build those frameworks such as Ben. So is that basically what we're talking about here?
Ben_Wilson:
Yeah, and if you choose not to use a framework to do this, this thing, we're talking about building houses in this analogy. What if you choose to not, you're like, Oh, I can, I could do the wood part by myself. Well, you now have to do all the stuff that that lumberjack does. You now have to become savvy in forestry management. You have to know how to plant trees and grow them and know when to cut them. Know which ones to cut, which ones not to cut. All right, once you fell it, how do you get the log off the side of the mountain? Well, you have to now know how to use all this heavy machinery to move that stuff around. Now you got to learn how to drive an 18 wheeler truck and where to take it to a processing mill. You have to know how long to dry that wood out. You know, there's all this things that are involved in building those frameworks that are highly specific to a certain skill set and the vast majority of people doing applied data science work. or applied software engineering in general, don't need to know how to do that. Just like a home builder doesn't need to know how to cut a tree down. They need to be good at using wood to frame a house. And a lumberjack is not gonna come in and be like, oh, I'm gonna build a house. That's why you never see like framework maintainers go like, hey, we came up with this great ML use case. It doesn't happen. Cause it, you know, A, there's not enough time in the day to do all that if you're supporting a framework. And B, it's just not, you know, that group speciality.
Michael_Berk:
That makes a ton of sense. Okay, cool. So let's sort of switch gears a bit and start thinking about how we scope projects. And Ben and I earlier this week were chatting about this thing called Halmeier's Catechism. It's a mess, just Google your best estimate of how it's spelled and something will pop up. But the background of this is we were looking at this DARPA website. And DARPA is the Defense Advanced Research Projects Agency. projects like flying missiles and talking dogs and all these other things. And on the website, at least this is what I was referencing to learn about Halmeier's Catechism, they state that DARPA operates on the principle that generating big rewards requires taking big risks. And then the question is, how does the agency determine what risks are worth taking? So George H. Halmeier, who was a researcher in the 70s, he crafted a set of questions anyone scoping a project go through and evaluate whether this program is worth it. So these sets of questions are very prominent in software and Ben, could you name a couple open source repos or even projects that you've worked on that leverage these principles?
Ben_Wilson:
Uh, so these principles are at the forefront of all product decisions made at pretty much any big tech company and most small ones too. So there are the guiding principles of everything that. Databricks and engineering decides to build. So you have to answer these questions before. If you have this great idea, you're like, Whoa, what if we had this tool that could do this thing? Well. If you approach a PM about that, they're going to hand you these questions and they're going to say, answer these, please, write down your answers, and then we'll talk about it. And that could be, you know, somebody in any position within the company can propose an idea like that. And if they want to champion it and run with it, that's kind of how it's going to go. Whether you're an engineer, an executive, or, you know, anybody at the company really would have to go through this. And other big tech companies do the same thing. These principles inform, it's not just software that does this either. Uh, other industries that I've worked in before working in software. I've seen these questions before. And then, you know, I knew the story of, of this man's history about, you know, his LCD display invention and, and how all of that revolutionized a lot of high tech manufacturing. But the project work and focus that he was able to bring to an entire industry is, you know, one of the reasons why he's so famous and why everybody continues to use these principles. They seem like common sense, but they're really not.
Michael_Berk:
Yeah, a lot of project scoping questions do seem like common sense, but they're really really valuable when everybody becomes aligned on the first principles and the basics. Because often people make jumps and assumptions that aren't always warranted. So Ben, you down to walk through them and chat about each of the points? Cool.
Ben_Wilson:
Yeah, of course.
Michael_Berk:
So first one, according to the DARPA website, is what are you trying to do? Articulate your objectives with absolutely no jargon. So how do you approach this type of question?
Ben_Wilson:
So this question is written in such a way that is beneficial to everybody who's going to review the other answers to the questions. And it gives a democratic approach to your project. Most people are coming up with ideas for products to build in industrial manufacturing or software to build in our industry. They're highly steeped in jargon and technical terms that are esoteric to what they're working on. And the intent with this first question is the layman's term test. Like, can you take what you're about to write and read it to your grandparents or your parents or somebody who is completely not in your industry? Can you explain what this thing could potentially do and have the person who's making your morning coffee at Starbucks understand what you're working on. And that's critically important because not everybody that's gonna be reviewing this proposal is going to have the same technical background that the person proposing it is gonna have. So in order to get the most amount of minds and perspectives on this idea, you have to eliminate all of the specific technical jargon. You don't talk about implementation details. because it's irrelevant. It's just, hey, what is this thing that you're trying to propose? Use plain language.
Michael_Berk:
Exactly, and that's one thing that ML frameworks and applied ML both have in common is it should be understandable. It should have a clear value proposition. It should have a clear purpose. And that's really important to articulate at the beginning. Sometimes I have started projects without articulating this and then like two weeks in I'm like, holy shit, this is completely pointless. I'm doing nothing valuable. that everybody is sort of on the same page of what this is supposed to do is incredibly valuable. One additional point is that I've actually... I haven't done a single project in the past two years that doesn't start out for the problem statement. And I often write by thinking, so I often write in a Google Doc, what is the project, how is it used, et cetera, but I have not done a single project without an outlying initial problem statement. Cool, so,
Ben_Wilson:
Nor have I.
Michael_Berk:
yeah, nor has been. Cool, so moving on, the second one is, how is it done today and what are the current limits? So Ben, how does that fit into this whole project scoping setting?
Ben_Wilson:
So there's two points in that that are important to get an answer on from the person proposing the idea and it's to help eliminate personal bias of the person who comes up with the idea. We are inherently egotistical creatures, humans. We think that our ideas are best ideas, right? So we have this cognitive bias of, Hey, I've noticed this problem. I think I have a unique solution for it. And I think this is important. You propose it. If you don't, if you don't answer this second question about how do people do this today, that first part of it, if there is no answer to that, then you're assuming extreme risk because you're coming up with something novel that nobody's doing this today. So this is a new thing that you're, you're now gambling basically. that whatever you're coming up with, people are gonna think is valuable. And that's extremely risky. That doesn't mean don't do it. That means articulate to the people who are collectively looking at this project and its potential value that, and you have to be honest with these answers too. Like you can't make stuff up here because people are gonna independently verify this by interviewing people. But if you articulate like, The process of doing X is 37 steps of complex stuff. And we have this, and it's extremely needed. Like everybody has to do this thing. We wanna provide a solution that can reduce the complexity of that from 37 steps down to three. That's a pretty big value proposition. Like, hey, users are gonna see, hey, there's all this. And we're talking about like framework code here, but. That's how a lot of new APIs are created and projects for frameworks are generated. It's because people have looking at how somebody has to do this right now. And then a company just says, well, it might be internal users or it could be, you know, if you're a SaaS company, it could be your users who are your clients who are doing this thing, you evaluate how, how they're doing it right now. And you say, well, let's provide an API that simplifies this. and makes it so that we can save, you know, 20 person weeks per company per year. If we implement this and any company that sees that like, Hey, this, this software company just released this to the open source, this saves me from doing all this annoying stuff and all my projects and also has all of these features that we really need that we currently can't do because of time and resources. People are going to use your stuff. So that's really what this question's aiming towards, is like, hey, what's the risk of adoption here?
Michael_Berk:
What are your thoughts on building versus buying?
Ben_Wilson:
Um, jeez man, it depends. Uh, it really, really depends. What are you trying to do? Like what problem are you trying to solve? How novel of a problem is it and how well does. you know, offerings that are out there solve that problem or potentially solve that problem. If you have, if there's a solution that somebody's selling or there's an open source package out there that gets you 90% of the way there or even 60 or 70% of the way there, that 60 or 70% can save you months of work and can reduce your test burden and complexity of maintaining this thing. But you come to the build discussion sometimes when you've exhaustively tested everything that you can. Like, hey, I've tried to solve this problem with all these open source tools. And not just me, but seven of us on the team have tried to do this. We can't solve this with existing things. So at that point, you fill out this questionnaire for the catechism and say, hey, do we have to build this thing? like build it ourselves. How long is that going to take? How many people do we have? What's our current workload at the company? What's, what's the cost of us doing this and maintaining it as well? You know, walking into the assumption of building something, you should be thinking, how much is this going to cost us to maintain for 10 years? Cause that could be the life cycle of, of that solution. And you're going to have ongoing maintenance for that for a decade. How many people do we need? What's their salaries? How much time per week is this gonna, you know, estimate all that stuff? And you take that entire total sum of money and also weight against the fact that people would be tasked with maintaining something that you built versus building new things based on existing technology and stuff that you can do from, for you open source stuff or interface with a SaaS company and have them do all the heavy lifting of complexity. Uh, somebody will make that determination usually in management and it's up to the team and the team lead and that, that team manager to draft up that comparison to executives and say, Hey, this is going to cost us $1.5 million over the next three years if we build this, or we can buy a solution from this proprietary software vendor that they offer this service, we just need to give them our data and it costs $150,000 a year. So an executive is going to be like, I just want the problem solved. I don't care how it gets done. And I want to the difference between 150 grand versus a million. They're going to go with 150 grand. So that's really the economics of how I look at stuff like that.
Michael_Berk:
Got it, so it's just a complex ROI calculation. And yeah, that actually leads very nicely into the third point, which is what is new about your approach and why do you think it will be successful? How
Ben_Wilson:
Yeah,
Michael_Berk:
do you evaluate that?
Ben_Wilson:
this one's very important for people who haven't up until getting to this question done their homework of really evaluating what's already out there. So in order to answer this question, you have to evaluate other options that are available and you should be listing down all of the things that you found like, well, we need this thing to do X, Y, and Z. you should be listing the fact that Y and Z could be handled with this open source package. X isn't part of that open source package, but we could couple another package with this other one and just write some glue code, effectively gluing these two frameworks together to solve this problem. What you don't wanna do is do this evaluation and say, oh, we've looked and we can't find anything that solves X, Y and Z. And then during review time, some senior engineer is looking through and you're like, wait a minute, there's this package that is part of core language in Python that does, you know, TUV, WX, Y, and Z. Why don't you just use that? And that review and the list of things that are purportedly new in what you're doing can help illuminate the fact that this already exists. which can save you from releasing something, potentially the open source, or just spending a ton of time reinventing the wheel. If somebody else has gotten there first, just use their stuff. Use your creativity in other ways.
Michael_Berk:
100%. Yeah, and moving on to the fourth of eight total points. This is one that I have been struggling with at Databricks, which is who cares? And if you're successful, what difference will it make? Because I personally would love to build a bunch of cool stuff, but it turns out that you actually have to make money. So you need customer buy-in, you need people to be invested, you need there to be a need in the industry. So Ben, how do you go about scoping that need?
Ben_Wilson:
I don't. In fact, you kind of alluded to it with what you were saying about working at Databricks. We don't decide any of that stuff. We ask customers. We ask them to be, you know, we sign non-disclosure agreements with them and say, hey, we're not going to repeat anything you tell us during this meeting. But just tell us if this is something that you would use. Tell us if you think this is stupid. completely honest with us, or if it's an idea coming from them, we just politely listen to everything that they're saying. We take notes and anything that's sensitive. Of course we don't take notes about, but the general concept of what they're trying to communicate to us, like, Hey, we really need this functionality. We'll interview that one customer to get that, that first idea. And then we'll float that idea without saying that, you know, company A mentioned it. We'll just say, what do you think about this idea to company BCD? And you'll talk to other, other companies that are within their industry in a similarly, you know, safe manner. And if everybody's saying, yeah, this would be awesome if we had this or yeah, we really need this. We, we were going to ask you about it, you know, next quarter. Uh, so if you get this buy-in from a lot of people. And they're saying, yeah, we're going to use this thing if it comes out and here's the features that we want to see. That's that process. And that's how you answer this question. But there's times where we've had ideas that either are generated from internally within engineering or they come from Databricks field or they come from customers and everybody from a technical perspective, the people that are going to be building it. They get kind of excited. They're like, this will be cool to build. This seems like it's really important. And we go and talk to a bunch of customers and everybody kind of looks at us like, we've got two heads. They're like, no, that's stupid. Like, why would we use that? We don't get disappointed, we don't get frustrated, we just say, all right, let's do something else. Because there's always work, you know? There's no limit. If you were to sequentially tag the backlog of ideas that are out there, there's decades worth of work to be tackled. So there's always gonna be something that could replace that priority.
Michael_Berk:
Got it. What percent of the time are customers wrong in knowing what they want?
Ben_Wilson:
Um, man, what a loaded question. Uh, I couldn't give you a percentage at all, but. I'd say that in the overall, let's say if a customer answered question one properly, and they said, and that's what we do when we interview customers for use cases, we don't wanna hear about their specific needs of this one project, of applying this thing, it's irrelevant. What we wanna know is, in layman's terms, what is it that you're trying to do? Like how is this solving a problem for you? And if that aligns to what we're already building or aligns to something that everybody has a common need for that, that use case, then that's what gets built. And we talk to them and share with them what the design is going to be. Say, Hey, here's how we're thinking of solving this problem for you. And this is what it would kind of look like. So a lot of times that does happen where. they have this ask that is either so specific to their industry or their company or their particular project that nobody else would be able to use if you just list out those requirements and build to that. Nobody else would ever use it because it's irrelevant to everybody else. But if you abstract out that idea of like, hey, we're trying to solve this thing in this way and it's built in an abstracted way. so that they can adapt their use case to that implementation. That's how you get successful framework additions built. And that happens all the time. In fact, it happens the other way as well. So when you're building an implementation and you're showing a prototype to a number of different companies, that's why you do private previews. And that's why you do prototype demos and request for feedback. That's that whole process. We want to know, does this meet your needs, or does this totally suck, or does it need this additional functionality that we're going to implement later? Most feedback is pretty brutally honest, which is how we like it, and it helps build better software.
Michael_Berk:
Got it. Yeah, if I'm being totally honest, I would say that beyond general technical competence and just competence in general, I would say that value scoping and identifying opportunities is the single most important thing an employee can be good at. In a prior role, I was working in decision science and so many different people would come to you with different needs, different questions, and just be like, hey, solve my issue. And it's your job to sort of read between the lines, abstract out, and then develop more general solutions that ideally can be modularized into other areas. But people rarely know the solution that they want. They just know the pain point. And if you can gather lots of different data points, it's extremely valuable.
Ben_Wilson:
Yeah, I couldn't agree more. I mean, in fact, That's more important in my opinion, too, than being able to effectively tune a model as a data scientist. Because you can be the best in the world at algorithm development and applied use of algorithms and how to tune them and how to optimize your feature set to eliminate possible overfitting. All those technical skills that usually come with experience. testing stuff out. But you can be the best of the world in that. But if you can't talk to people and understand what they're trying to solve or get you to help them solve, you're just useless in a company. Keep your resume up to date if if that's the way you want to be because eventually that company is going to realize that no productive thing is coming out of the team or out of that person and they'll be asked to leave.
Michael_Berk:
Yeah, exactly. So we have I think four-ish more, but we're just going to skip to the last one. If you guys want to review them on your own, feel free. But the three that we're going to be skipping are what are the risks, how much will it cost, and how long will it take. All of them are about identifying potential issues in bottlenecks and are essentially a reason to not do the project or be wary as you go through the project. But one thing that I want to ask Ben about is this specific question. What are the midterm and final quote unquote exams to check for success? So as you're developing this project along the way, how should you think about measuring whether you're on track? And at the end of it all, how should you determine whether you have succeeded in implementing the solution?
Ben_Wilson:
I mean, for software, this was one of the, this, you know, sort of several defining principles that informed agile software development, which is instead of. So the industry that this guy came from, you know, originally, before he started working for DARPA, working for RCA Labs in Princeton, New Jersey, and is working on, you know, electrical engineering tasks, specifically for displays. Invented a bunch of super cool stuff. But when you're talking about inventing a new technology and building prototypes, that's d***. just like how software used to be built because software, people that were building software 30, 40 years ago, uh, came from physical engineering backgrounds. You know, they were electrical engineers who were like, oh, I can now use a computer to do this thing. And, uh, you know, it's kind of how that industry got started. And when you approach a project in software like that, it's called waterfall development. And you gather a list of all of your requirements up ahead of time. You spend a lot of time building all of this stuff to meet all of those requirements and at the end of. Construction of your prototype, you send it out for review. People evaluate it, test it. And that's kind of that, that midterm check. And once everybody agrees that. all of the subsequent repair work that's done to it or adjustments to that software are done, then you send it out for a final review, which is that final exam, and you get an acceptance or rejection of it. Waterfall is super dangerous in software. People have known this for many, many decades now, so nobody really does that anymore. Although, I mean, I have seen companies do it with applied ML and applied data engineering tasks. It's just It's not what like recommended. So instead in agile development, you have this principle in your mind of constant exams. So instead of, you know, saying, hey, we're gonna build this thing that does these 127 things. And we're gonna make sure that all 127 features are built out before we show it to anybody. And it's complete. That's really risky because what if You know, half of those are totally useless. 10 of them are broken. And then you've failed to build 15 things that people actually care about. So nobody's going to use your stuff that you built. So the constant, the agile process of constantly doing checks of, Hey, I have a new feature. We're going to release a candidate for this. We want people to test it out. We do our own internal testing. You know, this week while we're chatting right now, last week, this week, and next week, and you were a part of it earlier this week, where we did dog fooding, which is software development term for eating your own dog food, like testing out your own code. We're working on Animal Flow 2.0, and we're having people test our release candidate build and telling us, hey, tell us how this is broken and we'll fix it. And we don't do that just for the 2.0 release. That's done. the 2.0 release, it's going to be done for the 2.1 release and the 2.2 release and it's never ending. And when new features are proposed and implemented, we're testing that out, we're making sure that it behaves right. It might need adjustments later on, we might have breaking changes later as as the project evolves over time. So it's really important for that final question when following this for product design. particularly for ML as well, is understanding that this is a mutable thing. Your project is not just done when you release the first set of predictions. You could be rewriting that entire project from scratch five times over in the first two years. Like every line of code could change. It's not the code that's important. It's the problem that you're trying to solve. Nobody cares how you solve it. Nobody cares how pretty your code is, or, you know. how sophisticated the algorithm is that you've, you're using or that you've implemented. It's the project that matters. And knowing that every time you're gonna have a new idea to solve or a new change to make is thinking about, are we successful at what we set out to do? How are we validating that?
Michael_Berk:
Yeah, and if you're curious whether you will have tendencies to stick to projects too long, ask yourself if you're okay with stopping books mid-chapter or even mid-sentence. Lots of people such as myself really want the satisfaction of going to the last page, closing the book and being like, I'm smarter because I read a book. It's actually a lot better if you spend your time doing the right books instead of completing the wrong books just because that's what you picked up. So Ben, do you complete books or do you stop midway?
Ben_Wilson:
Um, I don't think I've ever, I mean, there've been a couple of garbage books that I have purchased in the past and they end up on a used bookshelf somewhere. But the vast majority of the time I finish a book if it's halfway decent, but I never read to the end of a chapter. And sometimes I never read to the end of a paragraph while I'm, I'm reading. put a book down, pick it back up two, three days later. I can even read multiple books in the same day. So I never used to be like that before I was working in data science, ML engineering and software engineering now because it used to drive me nuts. Like, oh, I need to get to the end of the chapter. I need to finish this. Not so much anymore. I think your brain changes when you are working on things. with such a level of asynchronously where you have to do context switching so often throughout the day. And that doesn't mean switching from working on something technical to jumping on a meeting where you're talking about business stuff or that sort of context switching is jarring for anybody. But from a technical context switching where you could be working on seven, like seven different problems in one day. For efficiency, it doesn't really work to wait until you finish each one before going on to the next one. The same thing is for like project work. When we start a new sprint, sometimes I start two or three ideas all at the same time and I'll have different branches on my local machine. None of them are committed yet because I'm still thinking through them and working and doing. you know, prototype testing and stuff, but I'll switch between them because I might be able to kick off a test suite that takes three hours to run. And instead of sitting around waiting three hours to see if that idea worked out, I'll pop smoke over to the next one, start that one. So time management, uh, that comes with, with just experience and stuff. But yeah, psychologically, I think a lot of people that get into this industry, highly technical people like completeness. They're like order and disorder bothers them. And it used to bother me as well.
Michael_Berk:
and it currently bothers me. But we're working on it.
Ben_Wilson:
Mm-hmm.
Michael_Berk:
Cool, so I'll quickly summarize. So applied ML is sort of for a single use case. And you typically leverage frameworks as much as possible and rarely build very low level operations or applications. ML framework code instead are design patterns that are used to make writing code easier for multiple use cases. So they're very modular and it's paramount to make them readable, well designed, and quote unquote specific. They should solve a problem and solve it very well instead of kind of solving many problems. And then a couple of tips when leveraging these frameworks and just scoping an ML project. Make sure that you are actually scoping the value of the project and ensuring that there are customers or consumers on the other end of it. And Hal Myers Catechism can help a lot. And then another point is it's important to have humility in your knowledge of what the user needs, but don't take their ideas as fact. So listen, but try to read between the lines and figure out what their actual pain points are instead of forcing your ideas upon them. Anything else Ben?
Ben_Wilson:
That was a perfect summary.
Michael_Berk:
Beautiful. Alright, well until next time, it's been Michael Burke
Ben_Wilson:
and Ben Wilson.
Michael_Berk:
and have a good day everyone.
Ben_Wilson:
Take it easy.