In today's episode, our hosts Michael, Ben, and special guest Keith Goode delve deep into the transformative role of AI and machine learning in modern HR practices. They tackle a range of topics, starting with the innovative use of AI to streamline surveying and sentiment analysis in employee evaluations. They explore the exciting potential of AI models in technical data collection, particularly for interviews, and discuss how these models can assess candidates' sentiment and confidence levels, providing valuable insights into their fit for specific roles.

They also hear about the emerging trends discussed at the recent Databricks Data and AI Summit, where generative AI for resume screening took center stage. They debate the challenges and opportunities of leveraging AI to reduce information overload in analytics, particularly within the complex hiring process. They emphasize the importance of explainable AI models, consulting scalability, and the perennial issue of data cleansing in HR.

Additionally, the episode touches on the critical aspects of diversity and inclusion in the workplace, the influence of new legislation on workforce diversity modeling, and how companies can configure HR systems to suit their unique needs. They share insights into using advanced tools like XGBoost for predictive modeling, highlight the significance of face-to-face interactions in interview processes, and caution against over-reliance on automated resume screening.

Join them as they navigate these thought-provoking discussions and more, shedding light on the intersection of AI, machine learning, and human resources. 

Michael Berk [00:00:05]:
Welcome back to another episode of Adventures in Machine Learning. I'm one of your hosts, Michael Burke, and I do data engineering and machine learning at Databricks. Today, I am joined by my cohost,

Ben Wilson [00:00:15]:
Ben Wilson. I talk about MLflow at Summit.

Michael Berk [00:00:20]:
And today, we are joined by Keith. He started his career as a systems engineer at EDS. And since then, he's taken on a variety of roles ranging from product management, solution architecting, business development, pretty much you name it. Currently, he is VP of services at Zeroed in a data focus or sorry, a HR focused data solution. So Keith, we were chatting before the episodes as we usually do, and I'm curious your take on a semi controversial topic. What role should data play in the hiring process? Should we completely automate it? Should it be augmenting human decisions or something else?

 Keith Goode [00:00:58]:
Great question. I I think, I think in the future, we're gonna see, obviously, increase in the amount of, amount of value that data can add to the hiring process. Will it ever become fully automated? I doubt it. I think there's still gonna be a level of human interaction, to ensure a person's real, that there's someone behind there, things that maybe the data couldn't catch. But it is growing, immensely. And it's not just the the current data, the data from the resume, but it's also utilizing historical data to help make that decision even better. So I think we're gonna see it grow. We're just starting to see the amount of data that's going into the the resume.

 Keith Goode [00:01:43]:
I think it's it's gonna grow even more. And and as I've mentioned, looking at more historical data and coming up with more, better models, machine learning models that can help that decision making process. But I don't see it turning over to a fully a 100% automated. Perhaps in some, some roles, that are well defined, and the criteria is extremely well defined. But, for the masses, I I still believe there's going to be a a a human oversight to it.

Michael Berk [00:02:14]:
Cool. So I have done a variety of hirings. Ben has done a variety of hirings, Keith. I know you've done a variety of hirings as well. And I've been surprised at what candidates outperform expectations. Typically, if you're hiring a rock star, I like, in my experience, I've been able to see pretty early, and I've been right. Obviously, that's a there's a lot of selection bias within this. But there's been some people that have really surprised me when I thought, alright.

Michael Berk [00:02:44]:
They're gonna be okay. But then they come in with a great work ethic, a great attitude, and they go and learn really fast and become an expert quite quickly. So I understand that, resume screening and when you look at it on paper, that's quite easy to screen out. But, Ben and Keith, I'm curious your take. How do you think about finding those, diamonds in the rough?

 Keith Goode [00:03:07]:
Ben, do you wanna start? Or

Ben Wilson [00:03:10]:
I mean, I can talk about my process. And it does start with with reading a resume. I'm more of a red flag person, which I when I think about applying machine learning, even simple algorithms, I think that part of what I have done in the past could be 100% automated. So although it's it's probably not the simplest model to build, I look for hyperbolic statements. Things where somebody's claiming something that I know just by reading a few sentences that they're probably taking credit for a whole bunch of other people's work, or what they're explaining in the amount of time that they gave to that project is infeasible. There's no way that they could have done what they claimed to have done in the scope of what they're they're stating. And that's that doesn't disqualify it. That just means this is what we're talking about on the face to face interview because I I wanna hear this story.

Ben Wilson [00:04:16]:
But I also look for really simple things in resume writing, where it's the same thing that I would expect from somebody that It's my peer or, subordinate to me, which is when we when there's a success, it's a we. When there's a failure, it's a me. And if you can explain things in those terms and have that tone, and also it's really great when I see in a resume that somebody claims that they messed something up and then learn from it and fix it. That's sort of another flag. It's a green flag for me. But going through that that process of identifying that, I'm I'm looking at tone and, like, how is this person going to work in a team, and are they being honest?

 Keith Goode [00:05:04]:
Yeah. Those are great points. And, you know, I and to to go on to that, I I think I'm a a believer and and have seen a lot of out of aptitude testing to help lead clients as well. So depending on the the role, that, you know, they're coming in for, but, aptitude testing can be be a good sign. And, you know, going on with that with that resume aspect, I think we are seeing where some of the the models out there, machine learning models, can really start delving into to some of those criteria that one might see and that you then are picking up on the resume. I think that's a great opportunity for some model generation. But to answer, Michael, some of your question, I believe it's also fostering the environment where certain people can be successful. So, you know, it's kinda like, is it, genetics or is it nurturing? I I do believe the aspect of of nurturing candidates.

 Keith Goode [00:06:04]:
I've seen good candidates come into a difficult place and leave quickly. And one might say, well, that was a bad hire decision. I don't think so. So there's two ends to that that I think one could could argue.

Michael Berk [00:06:21]:
Interesting.

Ben Wilson [00:06:22]:
Yeah. I've definitely lived that. It's I think there's nothing more jarring than having a really great interview at a company with 1 or 2 people, and you're like, man, I got a really good feeling about this place. And then right after you're done your onboarding and you get put into a cubicle and you start working, and you just see, like, a very toxic hostile environment, people being belittled next to you or, you know, managers that are completely checked out. Nobody really cares. You're like, what am I doing here? And then you start, you know, updating the resume and sending it out. So, like, I I can't waste my time here. Yeah.

Ben Wilson [00:07:06]:
It's tragic when it happens when you have, like, a really good person who gets into a role and then they just leave. Yeah.

 Keith Goode [00:07:14]:
Yeah. And you you think about that. I mean, some some would argue, well, that's just, you know, either the candidate's fault or poor decision making, but, you know, the underlying result is it was neither.

Michael Berk [00:07:27]:
Yeah. Okay. Cool. Another potentially saucy question. So the data knows who the person is on paper as we were alluding to before. How do you handle diversity and inclusion initiatives when, someone may not look the best on paper, but you want to hire them via an affirmative action type of model?

 Keith Goode [00:07:55]:
I'll start with a a thought. I I believe that at the organizations I've seen most of the time that that gets down to more of a a human decision making process where, you know, the machine language or the the, data may point a certain direction, but there's always that human interaction, that can override, and make those decisions. So, you know, we see that happening. That's how to answer that question.

Michael Berk [00:08:23]:
Yeah. It's it's, just an interesting concept to think about, having data try to enter into human, like, soft judgment types of, decision making processes. Ben, how do you guys handle diversity and inclusion?

Ben Wilson [00:08:41]:
So we're very much a it doesn't matter, sort of organization. So there's no, like, the requirements to come and do the job that we do means that there's no human bias that's placed on any sort of, like, national origin, you know, original language that you speak growing up or your gender or anything. Nobody cares. And I mean that in a positive way is that everybody gets the exact same fair shot, and there's it's not so much heavily loaded on things that would be biased in society coming into an interview, which is is everybody knows in the tech industry, you look at the distribution in computer science or math or physics programs is predominantly white males in America that are in those programs that get fed into software engineering jobs historically, and that's changing over time. So we now have the most internationally and gender diverse software engineering group that I've ever seen. Lot of people from all around the world, from all sorts of different backgrounds because they're just good, and they exhibit the the cultural values that we have, which is you don't interact differently with anybody anywhere, in the teams. Everybody, it's not like we're all we're one big happy family. We're all one big happy, you know, respect all of your peers sort of organization, and everybody's voice matters.

Ben Wilson [00:10:29]:
And that's distilled from on top by the founders. And it's that inclusion is just part of the the cultural value. But I've been places where I saw the the opposite end of that, where every the only thing that people care about is what school did you go to, which that that's bias as well. Like, do you come from an upbringing that that silver spoon that's in your mouth helped you get to that place, and people that were struggling in other places, couldn't get to that that college or couldn't, you know, work at that that Fortune 100 company. It's not on your resume. So people just don't even consider that candidate, And it's toxic, and you just perpetuate the cycle of, you know, furthering exclusionary tactics that that doesn't even look at at somebody. They won't even get a face to face interview because they're like, oh, you didn't go to Dartmouth. You know, it's it's or Princeton.

Ben Wilson [00:11:29]:
It's ridiculous, and the company suffers for it.

Michael Berk [00:11:32]:
Ben, before we kick it off, you were saying, right, okay, diversity and inclusion background.

 Keith Goode [00:11:43]:
Yep. Ben was there was a thought that that came to mind when we listened to that, and and some of our clients experienced, it's fairly new. In In New York, they passed legislation in the state that any predictive modeling that they do regarding their workforce and people has to have biased audits performed against them from an EI perspective. I thought that was pretty interesting, you know, where things are going. Your your organization, Ben, you know, seems like it's a very inclusive environment in general just because that's the way it is. But you also have that legislative perspective that's trying to to also force that type of environment out there as well. So 2 two ways of getting there, I guess.

Michael Berk [00:12:38]:
Yeah. So at 0ed in, what exactly are you guys doing different than all the other data driven HR solutions?

 Keith Goode [00:12:51]:
So as we talked about before, HR is from a HR systems perspective is very siloed. Organizations in the HR typically do best of breed, whether it's their payroll system or learning management system, performance management system. The there there are great systems out there that perform those tasks very well. And, yes, there are some organizations like Workday that have a lot of those, processes already built into one application. But still, there are many organizations that haven't taken that leap and still have their processes in solid different, applications. So where I would say we do a great job is we work with our clients and we don't force them to put their data in our format in order for them to say in order for us to say, here are your 1,000 metrics that you wanna, you know, monitor. Some of our competitors do it that way where they're like, okay. Give us all your data in this format, and we'll give you, you know, these 500 metrics.

 Keith Goode [00:13:49]:
Our approach is is really more working with client, understanding we we we have templates in those different areas that we start with. But if an organization already has data feeds that they're moving throughout their organization to other vendors, we like to utilize what they already have in place, try not to reinvent that wheel. And we certainly, we wanna make it easy for them. We wanna make it easier for them to to to start getting, you know, metrics and getting value out of their data. And, of course, every organization feels that they're unique in the in the way that they they manage their data. So we like to try to, keep that uniqueness and and that feel for them. And so our application is highly configurable when it when it comes to that. We, you know, we we are able to work with multiple, you know, ETL routines to get get into what we call a staging environment.

 Keith Goode [00:14:36]:
And then, we have templates that we'll start building our metrics from, but we work with our clients to understand, okay. Well, you know, how how does your organization count their employees? Is it at the beginning of the month, end of the month, end of the period? Or, you know, often, we we utilize what's called a daily average headcount that gives more fidelity in terms of how people are moving throughout the organization. So just, you know, unique things like that. And there's work with a client. There's hundreds of of things that then you start opening up the filling back the layers of the onion and find out that, you know, this company does something a a very unique way, and and they wanna preserve that. So I I would think that's, one way we differentiate ourselves in the market. We also will build out, predictive models. For example, take flight risk.

 Keith Goode [00:15:23]:
Most systems or a lot of systems that handle people will have a a flight risk model. Maybe they're using client data and a benchmark of all historical data or all data from other clients. And that's great. It might produce a a fairly accurate, prediction that they can then, you know, assign to to individuals. But what we're finding is our clients, they want they wanna know a couple of things. They wanna know what's going into the model. So a lot of times, those systems are black box. They're just producing a a a metric where, you know, we'll work with our clients.

 Keith Goode [00:15:56]:
We'll identify the items that go into the building the model. We'll discuss the model with them. We'll change out features on, you know, what's gonna drive a higher, a higher, rate of accuracy. And so, you know, we kinda come up with a very unique, prediction, and a model for each client. Then, the other aspect I think is that we're we've generated what we call, explainable AI. So after we generate a prediction, anytime we generate a prediction, we also have a method to to roll back through the prediction or any any prediction case, for example, flight risk. If it identified Sally as a high flight risk, the client often says, well, why? And so we can take that that prediction, run it back backwards through the model, and identify what features the AI model, utilized mostly to come up with that prediction. And then that gives the the client a little bit better method to to, you know, determine what they should do.

 Keith Goode [00:16:58]:
Should they, you know, increase pay? Is it perhaps, you know, a problem with their supervisor? Whatever the case might be. But it gives more insight in terms of that prediction. You know, when the it's a black box prediction, you don't always get that.

Michael Berk [00:17:13]:
Nice. And can you speak to any of the methods? Just a out of curiosity, like, how they work on the back end.

 Keith Goode [00:17:20]:
Are you referring to any, model tools that we're using or data data

Michael Berk [00:17:25]:
that I'm asking? About a lot of, very important aspects of machine learning. So just straight up prediction models, model explainability, a variety of other things. And so I was just wondering if you could speak to any of the stack you're using, any of the the packages or libraries, any of the models specifically.

 Keith Goode [00:17:42]:
Sure. We like to utilize, XGBoost. It's a nice, algorithm for prediction, and it also has a nice, utilize several explainers for that. That's where we've mostly spent our time with it. Are you familiar with with that model?

Ben Wilson [00:18:01]:
Yes. Oh, yeah.

 Keith Goode [00:18:02]:
Yep. So, yeah, that's one of the primary ones we you we'd utilize for for prediction.

Michael Berk [00:18:09]:
Nice. And it would make sense that decision trees would lend well to explainability. So when you build out a decision tree, it will actually you can visualize the splits. And sometimes, like, salary at 30,000.26 $1,000 or whatever. Like, the the splits are a bit, specific, but oftentimes, if you extrapolate out what the splits are actually saying, it's there's some really insights. I remember countless projects where I would build out trees, visualize them, and then I mean, again, they can be overoptimized, to for prediction and less so for explainability. But if you build a stable model, there's often insights in those splits.

 Keith Goode [00:18:49]:
Yeah. And that's why we we really try to work with our clients to identify and and, you know, recreate or or generate the model and and kinda keep tweaking it until we come up with the right features that go into it. The other thing that that you know, as as well as the explainable AI that comes out, we can also utilize a a what if analysis tool. So, you know, think of a a really nice waterfall chart for each prediction, and then I get it shows, you know, what each feature did for the prediction, and then you can maybe tweak that and say, okay. Well, if we, you know, if we we promoted Sally, how would that would that keep her to stay? Or, and, of course, you know, sometimes based off features you use, you can't change those things, but it's nice to see what would happen anyway.

Ben Wilson [00:19:32]:
Yeah. Ben, I'm curious. Go ahead. Simulation modeling tools become very interesting when applied to human resources data in my experience. I've worked on a couple of projects internal to companies where people ask stuff like that. Like, hey. We're not gonna give you any of the employee names or any anything about what department they work in, but here's the data. And if some you know, working on some of these projects, it was kind of shocking what they were collecting on, like, how employees were doing things.

Ben Wilson [00:20:06]:
It's like, man, how did you get, like, the open rate of of their emails, in this format? They're like, well, you know, that comes as this tool that we use. Oh my god. It's cool. You know, it's useful. What about the sentiment analysis that you get from their internal communications and how that changes over time. They're like, well, we don't know how that works, but we we work with the vendor that does that. But we don't know we can't read what the the conversations are. I was like, that's good that nobody can see that.

Ben Wilson [00:20:36]:
But you start looking at all that data, and then you throw it into, you know, a tool like what if, and you start saying, like, woah. How do I like, if I bump this thing up, is this indicative of this other behavioral thing? Like, why why did this employee all of a sudden change in this quarter? And then you go back and look through the data, and you're like, oh, they didn't get a promotion, even though they're this is 1 year after they're up for it. And you start telling, you know, the HR team, like, what's going on here? Like, here's the here's the data you need for this employee to talk to them and see how they're feeling. It's it's pretty interesting when you start doing stuff like that. Well, if they were promoted, what does the model now predicts their their flight risk is? And it's like, oh, it drops way down. Like, maybe that's a driver for them.

Michael Berk [00:21:26]:
Yeah. Ben, I'm curious how you think about, something that we alluded to earlier, which is sort of the selection bias of only having data typically about the people that you hire. How do you think about, incorporating training information about people that you potentially passed on that could have been really

Ben Wilson [00:21:42]:
good or really bad? For a hiring decision, I think you need some sort of you know, you you can't just look at who did you hire and what what were the behaviors of that. You also have to feed in all of the reject data. And if you really wanna do a scientific analysis of something like that, you could only really do that, I think, in a very large company that has Scrooge McDuck money to, you know hey. We can we can handle hiring 10% of we don't know if these people are gonna be successful, but they're sort of borderline marginal. And out of that 10% of additional hire that maybe they weren't, you know, 5 stars across the board on on all the interview panels. They had a couple 3 stars, maybe a 2 star from 1 person. You'd probably find that there's a number of people that do exceptionally well at that company. Maybe they're just bad at interviewing.

Ben Wilson [00:22:41]:
Who knows? Or maybe they they like the wrong sports team, and they got into some discussion, you know, after the interview and just annoyed the the interviewer. Or the interviewer is having a bad day. Who knows? I think there's so many infinite realm of possibilities of things that can go into, you know, evaluation ratings from a a face to face talk. But I know there are companies out there that do that. I've worked at them before. And it's I think it surprises everybody when they do at those companies that I worked at that were doing that process intentionally, they would have reviews. So, you know, the typical, oh, we are we're gonna have 30, 60, 90 of check ins of a new hire, and we're gonna evaluate how everything's going. I don't think that works that well, actually.

Ben Wilson [00:23:30]:
I think it works better if it's 30, 60, 90, 365, and then you're doing, like, a 3 year review as well afterwards. And you're saying and then correlating that back to the initial interview data and saying, are we broken as an organization? Are we even making the right decisions? Do we need to revise, you know, our interview process? And sometimes, like, smart companies will continuously do that. Like, Keith, as you said, that that test the the the aptitude test that you can you can do, if you're not looking at that data 1 year out correlated to employee performance and happiness or competency, and looking back and saying, okay. For everybody that we've hired that did like, got a perfect score in the aptitude test, are they technically super confident at their job a year on out? Mhmm. And if not, then our screening's broken. And then the behavioral stuff, like, interview interviewers who are qualified to interview for positions should be going through some sort of continuous training. But part of that continuous training is, how did you rate that person, and how are they doing at the company now? And same thing for early stage mentors. Like, did you do a good job getting this person prepared for success at this job? I think data can answer all of that.

Ben Wilson [00:24:49]:
I'm curious, Keith. Like, is that data that some companies work with your your company on?

 Keith Goode [00:24:56]:
It it is. It and that's exactly what we do, but I I think you'd be totally surprised by how many companies don't. And the reason why they don't is because their talent acquisition data was in one system, and the performance in HR system was in another, and we're talking about from an HR perspective, who typically doesn't get IT resources, who, you know, comes from more of that, you know, touchy feely world. And IT resources are off doing finance and productive, production problems and things like that. So most of the time, it doesn't happen just because they don't know how to correlate the numbers from 2 different systems. Something that easy. But, and and you think about it, you know, from a data perspective, A candidate information is not employed, so they have a totally different ID system. And because they're in a, different system, when they do become higher, often, they're just generated a new ID, and that link is broken.

 Keith Goode [00:25:59]:
So what we work with our clients in bringing data together is is making that connection back so so that we can't reform. And it's really valuable. I mean, it's highly valuable. So, you know, you can look at that information during that that candidate process and then look at it 2, 3 years later, and how well that that person doing. That can also be used for modeling future. So it's it's really important. And if you don't mind me just touching on something else that you mentioned, and that was, you know, the the qualitative assessments and and evaluations. You know, we we primarily work with our clients, quantitative data, you know, data from different systems.

 Keith Goode [00:26:42]:
But we do also have, a surveying and assessment component and kind of built into our application. And if things, for example, engagement surveys, new hire surveys, you know, exit surveys, and all those can be be because we're working with the client's data already, we can easily set up those recurring evaluations based off of logical aspects, whether it's, you know, 30, 60, 90, 365. We can easily set that up and and have that, go out and collect those evaluations, and many of our clients utilize us to doing that. We're actually take trying to take that a step further. I don't know how much evaluation data you guys work with, but we're we're thinking about a way of using AI so that instead of saying, I wanna generate a, a survey that evaluates people's engagement or people's sentiment around something, you know, where instead of instead of being a psychometrician where you have to come up with your own survey questions and your answers and so forth, We're thinking that if you go to an LLM and say, okay. I wanna find out, you know, what people think about x. And at that point, it generates the surveys. It works with our applications to, you know, identify the people that it's gonna be sending it out to.

 Keith Goode [00:28:08]:
And then when it sends a survey out, no one's ever coded a question. And it may look, you know, it's not just a one time here of the questions. It's like, okay. When a person starts answering it, the the, the model the AI model would would be able to say, okay. Well, I'm gonna start asking this type of question. So it's a total, like, virtual assessment that, you know, based off of how a person's answering, the AI is going to know how to phrase the next question in order to get the results back on what the original person was trying to find. What do they think about x? So, you know, I'd love to get your ideas if you ever seen anything like that. You think it's possible? Would it, know, I'd love to open that up for further discussion.

Michael Berk [00:28:57]:
I'll jump in super quick. This is a really awesome use case that I hadn't even thought of until you brought it up. Maybe we could I mean, we're still a bit away, but, from, like, this tech being robust, but maybe we can switch out. I I don't even wanna say the initial phone screen, but at least, like, technical data collection, where you can just chat with a model and have the model keep poking and prodding until it gets a satisfactory answer. I I'm wary of offloading that to a computer with the current state of tech, but phone screens, I know, are really just challenging, especially if, like, your technical recruiter is very lenient on who gets through, then the hiring managers is bogged down with half hour or 15 minute conversations. And so augmenting this to some degree, at least for the data collection, definitely not the decision making, but for data collection, is a really interesting use case. What's your take, Ben?

Ben Wilson [00:29:59]:
I don't know about the I mean, you could fine tune an LOM, and it would be a really complex process to do this, but what you described, I think, is technically feasible. You would have to generate a whole bunch of training data for exactly this and, you know, wide, broad, diverse set of questions and how to do, you know, effectively the conversational lineage. But I think I think it's possible to do that. But what I when you started talking about that, first thing that popped in my head is multimodal models where you could have, like, a transcription and a recording of the actual interview. So phone screen is a sort of an antiquated term these days. Usually, it's over, you know, Zoom or Hangouts or something. You have that video feed and that audio feed of that that interaction with the the interviewer. There's nothing really stopping you from slapping on a post interview analysis of that with something like like, hey, you want speech to text trans transcription? Everybody's using Whisper for that these days.

Ben Wilson [00:31:18]:
You can slap on a different head to that model and have it analyzed provided that you've, you know, fine tuned it to just evaluate sentiment and what are the levels of confidence that this person has in their interaction? And you could say, are they eloquent? Are they well spoken? Do they speak in a manner where they're assertive about what they're claiming that they know? And those could be criteria that you could now have a effective qualitative adjudication of that candidate. Like, hey. For some roles at your company, that might not be important that somebody is assertive. You might not want that that you know? Or you don't need that that sort of I know what I'm talking about, and this is how confident I am in my presentation. You want that in a salesperson, definitely. It's something that, you need. You need that in an executive. You need somebody who's commanding, who has the ability to to communicate their thoughts in a way that is compelling and people listen to.

Ben Wilson [00:32:25]:
But do you need that in an an independent contributor who's working on modeling? Maybe, maybe not. Maybe you don't need that. Maybe you don't want that. So effectively mapping some of these personality traits by the way that they present themselves in front of others could be a good way to say, is this person a good fit for this particular role that we're thinking for them? Because the human level of bias in that, particularly in an in an initial face to face meeting a stranger, evaluating whether they're good for your your role, I think humans are inherently flawed and biased in those adjudications, whereas an impartial observer could augment that that interviewer's perception of things and say, here's the data of the this model that's been trained on millions of interactions. This is what the model detected. Does this align with what the human detected? And that can be something that on an an interview panel review board, you say, hey. The model really liked this person or this is this detecting that it's a good fit for this personality mapping to this role? The person that did the interview, they're pretty far off. And you can see, like, this did the does everybody else's model like, interactions that they have with this person, do they all agree? Could be interesting.

Michael Berk [00:33:44]:
Yeah. That that is that is, interesting. It made me think of this advice my mom retooled for me when I first started, doing interviews. She, like, read a book or a New York Times article, and this super fancy executive would go. And depending upon the different candidate's role, he would ask them, how fast did you drive here to the interview? This is back when you had to drive to interviews. And, if it was a salesperson, they had to be at least 20 over the speed limit. If it was a marketing person, they had to be right on the speed limit. And then if it was, like, a legal something, someone in the legal space, they had to be 20 below the speed limit.

Michael Berk [00:34:24]:
And based on that, like, he would find these, like, heuristics on, personality and how that would fit into the specific role. So, yeah, the Ben, I think you're you're hitting the nail on the head with that, which is we need sort of heuristics on how someone will do in the role, and personality can be attained, at least in theory, from these fancy Gen AI models. At a minimum, you can have eloquence, pace, cadence, that type of thing in speech. But do you think we could go so far as to say, are you a good personality match for the role?

Ben Wilson [00:34:57]:
I mean, provided that it has the ability I mean, I'm not saying that this would be a trivial model to build or a trivial system of models to build, but I don't think it's too far off. If you had the the money and the time to do it, you could build that system. And it would just be a a suite of models that that would be fine tuned for particular classification task.

 Keith Goode [00:35:28]:
Also kind of fall back to, you know, there's a lot of models out there that personality models, Myers Briggs, things like that. Yeah. So do you wanna reinvent that wheel or not? I don't know.

Michael Berk [00:35:42]:
True. That is a good call. So, Keith, I have another question for you. This week was Databricks Data and AI Summit. We publish episodes about a month after, so at time of publishing or time of listening, it will be after it. But, I was working the generative AI booth for way too long, and at least a couple hundred people came up and asked questions. And about, like, 10 of those questions were, how do I implement a resume screener for or using LLMs? Because everybody's executive is saying use AI, and people are saying, well, maybe I can augment the hiring process by using generative AI to at least extract information from resumes and maybe even make decisions on whether they should pass. Do you guys think about using Gen AI for that use case?

 Keith Goode [00:36:37]:
Yeah. I think it's a a strong case, and I would probably argue it's being somewhat saturated already. Most most, recruitment management talent acquisition systems out there are looking into doing that. The the challenge I would have with just doing it from from where we are is that I think they're tip it's kind of the problem I said before. They're they're gonna be looking at it from the data that's collected either in the resume, this some of the the the screening process, and it's gonna be subject around that silo data. I I think, you know, I would look more intuitive if there was still a gap in its ability based off of just that silo to say, okay. Well, at least really bringing historical data and others elements, then then perhaps, you know, we would be more interested in it. But I think that's something that, you know, those those talent acquisition systems are spending a lot of money to to implement, and they could probably, you know, do a pretty good job with just that amount of data as it is.

 Keith Goode [00:37:40]:
It's not something that we really go after initially. Where where we see utilizing, say, generative AI and LLM is twofold. Some that you see right now is where a a user could say, okay. I wanna find more information about x headcount or how headcount interacts with turnover in these departments. So some of those, our competitors and us will will use l LM to help generate, say, a query or point them in the right direction within the application to pull that information, whether it's a dashboard, whether it's a report, whether it's just querying data and presenting that. And that's all well and fine. It's basically utilizing an application's metadata to help navigate the person to the right area. It's it's a great use case in, a time saver, click saver, if you will.

 Keith Goode [00:38:35]:
But we're gonna take that a little bit further where we wanna try to work with with data, not just the the metadata about what we're capturing, but also the data itself. So imagine summarizing, an entire dashboard. Summarizing and then having a conversation to go deeper. Okay? Here's here's a dashboard, HR dashboard containing headcount, turnover, mobility. It's just some some things. For the last 4 quarters, broken down by organizational structure. So imagine throwing that at an LLM and then being able to have come back and summarize what the user is seeing because, you know, a lot of our dashboards, you know, we're presenting to an executive, to an HR person. Sometimes, even though they know the metrics, they know that having, you know, looking at the dashboards.

 Keith Goode [00:39:32]:
Oh, okay. That looks great. And they walk off, and they do whatever they were doing before. But perhaps having a summary, you know, or having the LOM say, you know what? This is what you should be looking at. This is, you know, we're we're detecting something happening over here. And I think those are are good use cases, that a generative AI model could use, not just from the metadata, but also from from the data itself. So does that make

Michael Berk [00:39:58]:
sense? Yeah. I think so. I mean,

Ben Wilson [00:40:02]:
it does to me quite a bit. It's I think that's the holy grail in analytics these days is exactly what you described. You know, you take a a company that starts an initiative 10 years ago where everybody is, like, let the data decide. We're a data driven organization. What happens 5 years on from that? You now have all the ETL setup, all of the BI setup. You've acquired, you know, enterprise licenses for Tableau or Power BI. You've hired 15 people in the analytics department to answer questions about the business. 5 years on from that, what do you see? Information overload.

Ben Wilson [00:40:46]:
I've been at places where I've gone into it's a tableau to to kinda see, like, I need insights on the business. I need to, like, figure out answer to my question here because I'm curious, like, should we focus on building a model for this part of the business or this other one? You start looking through dashboard after dashboard after dashboard. Like, hang on. We have for our product lineup, we have 847 dashboards. I don't even know which one to use. They're all running every day. They're all up to date. They're all cutting the data in different ways.

Ben Wilson [00:41:23]:
And then you end end up having to ask a person and say, hey, I just need to see, like, are we gonna be like, should I build something that's gonna optimize, you know, revenue generation for product x or product z? They're like, well, you need to go to these 7 different dashboards, and then there's a meta dashboard that kinda combines those. Can you just tell me which one? Like, just give me a link. So I from an executive point of view, that's why they have assistants that are interfacing with the BI team. And sometimes where it's just, I don't wanna look through all this stuff, just tell me give me a dashboard, build it for me that answers this one question. And it becomes daunting to run your business when you're I mean, it's I think it's impossible in in today's modern age to run your business without analytics. It's just a bad idea. But in the information overload age, which I think we're in now, for business analytics, how do you find the needle in the haystack? And I I think that makes a ton of sense. I think there's gonna be much more I I know that Databricks is working on it.

Ben Wilson [00:42:34]:
The ability to do exactly what you're doing is models that have been fine tuned on extracting, relational data, you know, structured table format table data and making, through the use of of agents. They can say, I'm gonna run a query on this raw data, and I'm gonna figure out what comes out of it based on this algorithm that I I have access to. And then I'm gonna write a human readable format that says, here's the top five things that are problematic that I'm seeing in your data, and here's some suggestions of where to go next to dig in deeper. And tools like that, I think, for what we're talking about on the HR process to say, are we hiring correctly? Or do do we need to make a cultural change rather than trying to say, you know, our kitchen's clean, and it's just our diners that are messed up, or we need to find better diners. We're like, do do you need to change the menu? And maybe, like, these tools analyzing the data that a company is collecting are the ways to sort of expose that to say, yeah. Let's let's change our direction of of the food we're making here so people will come like, the right people will come in.

Michael Berk [00:43:51]:
That makes sense. Yeah. Yeah. It's it's so interesting. The hiring process is a binary decision maybe with a when and a how much, but distilling hundreds of pieces of information into that decision is is really complex and challenging. So, yeah, any tools that can augment would definitely be helpful. And, Keith, I'm curious. How do you guys approach information overload? Do you just throw everything at the person and say good luck, or do you try to steer?

 Keith Goode [00:44:19]:
That's a it's a great great issue, and I think we're we're entering into that information overload age. Right? So I think, as I mentioned, some of the things with LOM is gonna help, you know, get summarize the data, so being able to take large quantities of dashboards and dashboard data to be able to first summarize and then conversation through that to to pinpoint, you know, what a person should be looking at. So that's definitely, I think, the direction it's gonna be going in. You know, currently, right now, we we also are are highly consultative with our clients. So, you know, we we don't have a one size fit all. We we work with our our clients to understand their uniqueness and and, again, what their their business problems are and put those in in in front of the client in terms of what reporting, what metrics, what graphs, what dashboards are gonna make the most amount of sense. So right now, it's still probably a a a manual effort, but we see that changing very quickly.

Michael Berk [00:45:16]:
Got it. And have you seen the consulting model scale as in from

 Keith Goode [00:45:21]:
a business perspective?

Michael Berk [00:45:22]:
No. So no. Like, sorry. When you can when you do this consulting model within your business, is that scaling, or because if you have a software product that's automated, anyone can use it. Right? So does that sort of limit the organization's growth?

 Keith Goode [00:45:37]:
I I don't believe so because it's, you know, we we've tried to take out the the, we've tried to make things repeatable as much as possible and configurable, so we're not building from scratch each time. So we look at a lot of, features to to copy than just tweak. So, you know, we're trying to, reduce that that level of effort to get something unique for for a client. But, you know, there is there is a level of of resource needed, and, of course, that's going to impact scale. But, you know, we feel that it's it's important and and clients are willing to pay for that.

Ben Wilson [00:46:17]:
Yeah. I don't see there's an alternative, though. Like, from what you're from what you're saying in in my own personal history with dealing with teams like that within an organization, you're a 100% right. Like, if they do have an IT person, it's to help resort and set HR passwords or access the systems. You're never gonna get I I've only seen in a handful of companies that I've either worked with or talked to. They're like, yeah. We have we have, like, dedicated software engineers for HR. I'm like, what company is this? And you realize, like, oh, you're a FAANG company.

Ben Wilson [00:46:50]:
Like, of course, you have, like, a a team of 15 engineers that just do HR stuff, and no other software engineers talk to them. But that like, those teams don't exist at 99% of companies out there. So, yeah, if you're getting into this business, it's not like you can tell the head of HR, like, yeah. Give me access to Greenhouse, you know, just just set up the API for me and, make sure that we have a token. They'll be like, what are you talking about? So, yeah, you need consultants to go in there and be like, here's what we're going to do, and here's the data we need to get. And if you give us access, we'll we'll get that working for you. So that makes sense.

 Keith Goode [00:47:33]:
From an HR perspective too, you know, there's a lot of over the last several years, a lot of shifting from one siloed application to, oh, you know, we use this learning management system. Now we use this one. We went from Lawson to Workday. We went from you know? So in order to capture that historical data, you know, they may have it. It's probably in, you know, very simple format with a ton of it, and they'll need a way to to, you know, bring that all together. And so there's still a lot of that old school ETL data cleansing that needs to go on.

Ben Wilson [00:48:12]:
Cool.

Michael Berk [00:48:13]:
So I have one final question for you, Keith. It seems like you've held a variety of leadership positions, and you've had to do the the manual human version of this hiring process. So do you have any tips on how to hire?

 Keith Goode [00:48:29]:
Well, like I said before, I I you know, depending on on the types of roles, I I do like aptitude testing, from a from a high level perspective. I think that that certainly helps. You know, and to to kind of go off what Ben was saying before too, I I think there is a level of experience in looking at at resumes, and and there's a lot you can weed out. I also try to weed out things that, are buzzwords. I I don't like to see a lot of buzzwords in a resume, and that kind of is a is a red flag for me. You know, this is the the main things I I would say in it. You know, for, interviews are still important. One day, I think we'll get to more help in the decision making process, but that's those are the things that that that I'm looking for now.

 Keith Goode [00:49:23]:
And, you know, I'm also a firm believer that we, you know, we do make a a higher decision to create the the environment, the nurturing environment for them to succeed. So it's it's bidirectional. It's not just the candidate, but it's it's the environment and and making environment for someone to be successful.

Michael Berk [00:49:44]:
Well put. And then, Ben, I have one more question for you. Do you think from a code screening perspective, specifically in open source, we could remove the coding interview and just look at someone's historical contributions?

Ben Wilson [00:50:01]:
Historical contributions. That'll work if provided that there's a bunch of stars that align. So if you are somebody who has contributed to open source, yeah, we're looking at that. We're not looking at the code, though, per se. I mean, we do. When we are evaluating candidates, we're looking at their commit history. So, specifically, what was the the first commit that they pushed? How many comments did it get or change requests? And that's not a detractor. It'd be like, oh, they they shipped something garbage or whatever to for the first review.

Ben Wilson [00:50:42]:
It's how do they interact with with helpful comments and suggestions. Do they blindly accept, you know, some, like, in an instance where the the advice might have been bad? Did they just go and do that? It's like, okay. Maybe they don't really know what's going on here. Or is it did they refuse to make the change or argue nonstop back and forth with a maintainer of a project, those would be red flags. But if they're if they've made the effort to, like, implement something of sufficient complexity and then work with the maintainers, and they're very personable and pleasant, and that's that's just, like, huge plus ones every time you see that. You're like, yeah, this person's great. Like, they seem like they would really fit in on on our development team. But that is not, like, a requirement for any position at atrigs, in engineering, because you can come from a company that has a 0 open source policy.

Ben Wilson [00:51:47]:
You're building code in private repos, but you are either not allowed to contribute to open source based on company policy or you're too busy keeping the lights on and building features. You we're not looking for somebody who lives, breathes, and and eats software, and I think that's kinda dangerous. Like, those people do exist. They're very rare. Like, hey. I worked my my 9 to 7, and now on my weekend, I'm contributing to open source. Like, cool. Great.

Ben Wilson [00:52:18]:
That's your passion. Not gonna not gonna knock it. But the rest of of people, they're off hiking in the mountains. They're spending time with their family. That's super important. They don't spend your free time just doing work all the time. That that's bad. You're gonna you're gonna burn out really quick if that's how you approach it.

Ben Wilson [00:52:37]:
So we we take all that into consideration. We don't have access to their contributions internally because, you know, proprietary code. So that's what the testing assessment is for. But I don't think testing assessment should be testing the ability of you being able to write, you know, elite code. I don't think that's very useful. It's more like, give an actual real problem that's unique, see how they think through it. Doesn't matter if they get it completely correct. Let them mess up.

Ben Wilson [00:53:12]:
Let them self correct and talk to them about it. And, you know, the the face to face interactive code test is always more effective in my I mean, in my experience, because you're able to really see how somebody thinks through a problem and then gauge what their reaction is to them messing something up. Like, do they freak out? Do they get super hostile or or defensive? Probably not somebody you wanna work with. But if they're they're humble and and good at what they're doing and wanna discuss and get excited about discussing this problem, you're kinda like, yeah. This person's great. So I I a 100% believe in assessment tests. Just I don't like the fire and forget ones. And particularly on today's age with code generation, you can get GPT 4 to generate some code for you that will pass a lot of code interviews for the non, like, elite software companies.

Ben Wilson [00:54:11]:
It's not that hard. So I don't think they're very effective anymore.

Michael Berk [00:54:18]:
Heard. Alright. Well, let me summarize really quick before we wrap. A lot of interesting things we discussed today. A few tips specifically in the HR space. For HR, model explainability is super important. People wanna know why a model is generating the prediction that it is. Also, it's important to monitor the outcomes of the decisions that were based on your model's output.

Michael Berk [00:54:43]:
So, typically, this will take time. Sometimes it's a year. Sometimes it's 2 years. If you can find early indicators of success, that's great, but it's important to understand that the model is a vehicle for decisions. Model accuracy is not what we're optimizing here. LLM based resume screening is a very saturated market. So if you're doing a start up, maybe do something else. And then when hiring, consider using aptitude testing, and buzzwords are typically red flags.

Michael Berk [00:55:10]:
So, Keith, if people wanna learn more about you or your organization, where should they go?

 Keith Goode [00:55:15]:
Yep. Our our website, 0edin.com. And, also, feel free to connect with me in LinkedIn. Handles Keith a Good. I'd always love to carry on this conversation. I find it fascinating. And, you know, you guys have some great questions, and and I love your your insight into what you're doing. It's been fantastic.

 Keith Goode [00:55:35]:
I wish we had more time to keep going, but, now it's good. And I always love to carry on the conversation.

Michael Berk [00:55:42]:
Yeah. No. Likewise. We we always get into, like, a a nice little rhythm right around this time. So yeah. And HR is such a fascinating, field. So, but, unfortunately, we have lives and jobs. So with that, I will wrap.

Michael Berk [00:55:58]:
Until next time, it's been Michael Burke and my co host Ben Wilson. And have a good day, everyone.

Ben Wilson [00:56:03]:
We'll catch you next time.