Tristan Sloughter has been an Erlang Developer for over 19 years. In this episode, Tristan joins the show as he talks about their project called “OpenTelemetry release of Erlang/Elixir.” With the use of this protocol, in your application, developers can collect, process, and export data. He also shares his journey toward shifting his focus from OpenCensus to OpenTelemetry and gives a background about it.

Hey everybody and welcome to another episode of Elixir Mix. This week on the panel we have... me. Such a wolf. I'm alone today. But I have a special guest. And that is... Tristan Slaughter. So Tristan, why don't you tell the audience why you're here and what we're going to talk about?

 

I'm here because we recently, we, the OpenTelemetry project, released an Erlang and Elixir 1.0 of their API and SDK. OpenTelemetry is a merger of two projects, OpenTracing and OpenCensus. OpenCensus I was involved with a number of years before the merger, and now it's been about two or three years that I've been involved with OpenTelemetry. And it's the focus of my work at Splunk. And I've been involved with Erlang for, oh geez, 19 years or something like that now.

 

Oh boy! 9th? Good gosh. I wasn't even in the industry back then. I was still going to school!

 

I'm getting old, yeah. Wow. So yeah, the main focus. Why I was asked to be on was the open telemetry release of Erlang and Elixir.

 

Yes, indeed. That is the focus of the day. But I mean, they just said, okay, you've been running for 19 years now with OpenCensus. For like, how long more long? I mean, how long that? Because I actually still remember when OpenCensus and OpenTracing were two different things. Like, I don't even sure what we exactly ended up using. But back on the project a few years ago, I was looking at the different possible solutions for the Beam, right? And then... I definitely stumbled across those two different things. So why

 

Yes.

 

don't they give us a bit of a gist, like a history there, like how you ended up doing what you're doing.

 

Yeah, so I don't know how long it's been with OpenCensus over five years probably, but I got started with that because I was at a small startup and we ended up moving to Google Cloud and I was looking at observability options and Google didn't... There was some stuff that I was trying to talk to them about supporting to get decent Erlang support and to even work with open tracing, because I was going to do some open tracing implementation.

 

Mm-hmm.

 

And I ended up they ended up giving me on a call with people who were working on open census, and this was before it was public. And so.

 

Okay, exciting.

 

Yeah, so I started out talking with them about what their goals were, and it fit pretty well because one of the issues with open tracing was it's pretty lax on actual spec details. And so all the vendors are pretty different, and so things don't work across vendors very well. And that's a big

 

Mm-hmm.

 

problem when you're not supported by vendors, like Arlang and Elixir usually aren't. So I got on that project right then before it launched. And after having been working on that forever, when the tracing and census projects decided to merge, they talked to me about coming on to continue doing what I was doing with the Erlang and Elixir.

 

Nice, nice, very exciting. I mean, I'm not sure that everybody who listens to this podcast is familiar with what tracing actually is, so why don't you give us a short introduction to that?

 

Yeah, and first I should say that one of the differences with Census and OpenTracing was the OpenCensus encapsulated metrics and logging as well. And OpenTracing

 

Mm-hmm.

 

is totally focused on tracing. And tracing is a lot like your local tracing if you're looking at the function calls that are happening in the Beam. but it's usually added in specific places where you want what they call spans, which are the lowest level of a trace. They're meant to encapsulate some operation. The key point for open tracing, open senses, open telemetry is that this is distributed tracing. the context, what the operation, the current span is when you're saying, making an HTTP request to a server is carried along. And so another span, because there's a parent-child relationship there of these operations, is carried along with it and you create a new child even though you're on now a different node. And so being able to build that, that graph of relationships and be able to pinpoint where things are going wrong, where your slowness is happening by looking at a full distributed system instead of each, each, uh, each one individually.

 

Yeah, yeah, yeah, I definitely can relate to that. I would maybe add that the nice thing about, especially when you like combine all of that, right, also like metrics and logging, then you could also have locks attached to particular spans, but how this like maybe looks at at the end of the day is that you really have this graph where you can see, okay, like basically you have like horizontal lines, those are the spans and they are, they can be if you render them visually and then you can see, okay, spans below that, their child spans, their parent spans, and they can really see what are the different things doing at every point and how long do they take. Just to give you our listeners and you an example where I saw that in action, it was super useful. I used to work on an event source system, so we're very event driven. And then we also added span information and span IDs as metadata to each event, and then we could also see which actions were triggered from events down the pipeline even in a completely different part of the system. and still could then see the relationships and the causes and the effects. And that was insanely useful and something misbehaved. So, yeah, I just, if anybody who is not familiar with that thing yet, you really ought to check it out because I feel it, it looks and like massage blocks and searchable blocks are great, but there's combining all of those different observability tools together. That is just a whole new level of understanding a system in depth.

 

Yeah, and open telemetry, because it combines all those into a single spec, is really able to cross those boundaries even better than, because services vendors like Datadog and Splunk and others, certainly provide that functionality on top of what they call open tracing, but it's ad hoc on their end usually. But open telemetry actually defines stuff like exemplars, which are Traces are logs that are parts of metrics. So the metrics themselves, when you're looking at metrics, you can click and see, oh, this goes along with this trace that was happening when this metric was recorded,

 

Nice.

 

things like that.

 

Nice. So what is then the usual setup you would run this with? Because, I mean, of course, you have libraries and different implementations in different languages which follow that spec, but that needs to flow somewhere at some point to be usable. Right? So how would usually, in your experience, look, might the setup look like, for example, if I have like a beam application, be it the elixir or earling, it doesn't really matter, right? But I have instrumented, I have a whole open telemetry things integrated. I'm using spans, I'm using, I'm adding locks to those, having metrics might measure. And then they, you take them and do what.

 

As in where would you ship them?

 

Yeah, for example, for example, just to give people like an idea about how this might work in real

 

Yeah.

 

life, you know?

 

Well, the grit, the, if you're getting started and you just want to see, do I think this is going to add value to

 

Mm-hmm.

 

me? Great place to start is Zipkin and Jaeger. And those can, you can just quickly run them locally, throw up Docker compose and have those visualizing your spans for you. But that doesn't give you, you know, a complete picture because the There's no metrics, there's no logs. Oh, well, I first should have mentioned that with the OpenTelemetry collector, which is a separate process that ingests the OpenTelemetry protocol, we have a protocol for traces, metrics and logs, yet another protocol, yet another spec, everything. And it... And

 

It's protocols all the way down, specs all the way down.

 

the collector has a little... dozens and dozens of supported exporters. So you can, if you set up the collector to accept your traces, metrics, and logs, you can then ship them to many different vendors to try them all out. So you've got Honeycomb, Lightstep, Splunk, DataDog, and use their free trials and see how it all meshes together. Locally, I think, I haven't tried this myself, but I know that these days, Grafana and Prometheus, they have their tempo and Loki and I think that Grafana, tempo, Loki for metrics, traces and logs. I don't know how well tied together all those are when ingesting open telemetry, but it's a possibility that those are a great option for running in Docker Compose. uh, seeing that, seeing what you gain from, uh, combining all those into a single, uh, protocol and spec. There, there was somebody the other day asking, uh, who has this very setup or is attempting this very setup in Docker Compose and the Elixir slack open telemetry channel. So, uh,

 

Mm-hmm.

 

if anybody is interested in that, they can hop in there and scroll up and hopefully the... history hasn't been lost yet.

 

Interesting. Okay. When I actually did some research on OpenTelemetry a while back, initially I found this distinction between the collector and the spec for actually collecting and gathering those things. I found that confusing at first, but at some point it made click. It was a bit hard to wrap my head around. Okay, wait. Oh, wait. That's a completely different thing. I also need to... Which is responsible of collecting those things. But yeah. I think one question which definitely was in my head when I first heard about the project, and I would assume is also in the heads of some of our listeners, is there is no relation between telemetry and open telemetry, right?

 

That is correct. Uh, telemetry pick the name before open telemetry pick the name. Uh, and of course I didn't have veto power, so

 

Hahaha

 

I couldn't say, Hey, this new library and elixir and Erlang is using telemetry name that's going to be confusing over here to stop it. I also thought it was a pretty good name, but yeah, it's very, that is certainly confusing. They do, uh, telemetry. There we have in open telemetry handlers for taking telemetry events and creating spans and metrics, or not metrics yet, but spans, so metrics to come. So they'll work together, but yet they're completely separate projects and with different goals.

 

Yeah, that was, like I said, that was a very confusing moment for me. I mean, naming

 

Ready?

 

is important, right?

 

Yeah, it's been very confusing for people.

 

So hopefully this clears it up for some people.

 

Yes.

 

Okay, so maybe because we invited you based on the release of OpenTelemetry for Erlang, Elixir, JavaScript and Ruby in version 1.0, can you give me the gist of what that means maybe for you personally also, but also for the project as a whole, right?

 

I mean, the main thing that 1.0 meant to signify is stability in the

 

Mm-hmm.

 

API.

 

Mm-hmm.

 

So for me, it means kind of handcuffs, but.

 

I get it.

 

can't change it or if I change stuff, I have to leave backwards compatibility. And we're given, I don't remember the exact numbers, so I don't really wanna say a number, but in the years, in year to years of what we're supporting. So if you adopt open telemetry, early in elixir and you wanna keep using the same version, we ordered the same code instrumented, we're not gonna break it is. we have for API usage for some underlying SDK stuff. If you're writing a custom exporter, something to send spans to a specific custom backend, you

 

Mm-hmm.

 

have,

 

Mm-hmm.

 

there has been some changes there recently, but in the API, if you implement the instrument in your application, we're guaranteeing to the best of our ability that it's not gonna break on you.

 

Yeah, that makes sense.

 

And the project as a whole has that as a guiding principle to have this stability. Especially, I mean, it's since it's big name companies, you know, involved in using this, they do not want. And if you ever wanted to get involved with the project, you can check out the OpenTelemetry website and get, find the special interest group SIGs and join some of them. And it's always a... Long discussion when API changes or something are proposed in the spec and how do we make this backwards compatible? It's taken very seriously.

 

Yeah, I think it makes sense with products of that size. And especially if you already have a significant adoption, then...

 

Yeah, and what we're asking people to do with it, put it into their, not only their projects, but their libraries and all this. And I know when Elixir telemetry is really taken off, so it's probably not, we're probably not going to see open telemetry instrumented libraries at least anytime soon. And it's still going to rely on middlewares and all that. But I'm hoping that if we show the stability that people fear it won't have, I mean, a couple A few years ago, I was telling people to adopt open census and now I'm telling them to adopt open telemetry so I can understand

 

Hahaha

 

the fear. But open census, of course, is still working. But I mean, it's not adding new features or anything like that, but it's gonna provide the stability.

 

I'm actually curious. I mean, we've not talked about open telemetry a whole bit, but you also said earlier, right, you were involved with OpenCensus and you went through the whole merger, right? So how was this? Because I mean, those were two different projects. Were they already collaborating before that? Or how do I imagine how do I have to imagine like two big projects like this coming together and creating something new that sounds challenging?

 

Yeah, I was surprised. I was not involved in the initial like discussions, which, uh, I've never asked. I have my guesses of who was, you know, directly involved. And I was surprised when I did hop on when stuff started becoming more solidified. So they knew that it was actually moving forward, just how much they had agreed that this was going to be the direction. And because, yeah, all the people who are involved with OpenTracing were. Because to me, we were adopting more of open census than we were open tracing, but they felt it was the right direction and they were all

 

Mm-hmm.

 

on board. See, I don't know what happened before I got truly involved, but it seemed to have been decently smooth.

 

And then like in the time after that, after that merger was like official, let's say that, like was there a lot of busy work to come to create something new or like, how do I have to imagine this?

 

Yeah, I mean, as I said, I've been working on this for what, three years and that's,

 

Eh, eh.

 

and the 1.0 of Erlang and Elixir was released a couple months ago or something.

 

You're fair enough, I guess.

 

And that's, and there's still a mental amount of work to go. And I've been involved a little here, there at Paternity Leave at one point in like the middle of all this. I got disconnected for a while, but the for a few months, but doing the tracing spec and they were trying to get a metric spec at the same time and it was just too much work because just nailing out the differences in tracing was quite an ordeal. I could tell

 

Mm-hmm.

 

from the people with different opinions and different backgrounds and different experiences in the industry. And, but I think it came out, uh, really great. Uh, I've been surprised at times and, uh, at times I've seen things with the implementation where I'll say, I don't know if this makes sense when I'm doing it in early in this way, and then I'll come back, you know, rethink it for a bit and I realized, ah, they did, they thought this out pretty well and it, uh, it protects the API from potential future, you know, things like that. being able to change things in the future or not change things. So it took a long time to develop the spec, but I think the time was worth it. And like the metrics are still not completely done. Though there are, I think Python and.NET have what they, stable releases of metrics. Most like most high level stuff is done on the metric side, but there's still some questions around like exponential histograms are still being worked out, things like that.

 

cool. I gotta say, it's definitely one of these projects I've observed from the far, but I never had the pleasure of looking into the inner doings of it. So, how has it been for you personally? As somebody who has been involved with a project that long, and especially from this one technology perspective right Erlang? Is this something you expect to be also now when you hit 1.0 right where you said earlier you kind of feel like you now have handcuffs? Is this something you expect to be doing in the long run like helping it evolve or is that something where also you as a person might say okay I got it here now it's time for something new? I mean you've been with Erlang for what what did you say 19 years right?

 

Yeah, rough. I'm 37. I'm pretty sure I started when I was 18 or 19. So yeah, somewhere in there. But yeah, in it for the long haul, I find the topic interesting. And Splunk is gracious enough to pay me to do this stuff.

 

So how does that work? I mean like, this is a startup, right?

 

What's that?

 

You said Splunk is a startup, right?

 

No, no, the startup was when I started working on open census.

 

Oh, okay, sorry, sorry, yeah, you're right, I remember, yeah.

 

Splunk is like 30,000 people or something like that company. They got their name in the industry with logs many, many years ago. When I was at Heroku, we were using Splunk and it was really

 

Okay.

 

great. Really amazing for log search.

 

Okay, for anybody who's interested, I'm adding a link. I will be adding a link to the show notes. So if you want to check out Splunk, you can go there.

 

Yeah, man, a lot of people don't realize that we have an APM offering for Spanish straight metrics and all that. I think it's just a logs company, but not anymore. And they're major backers, number one contributor to open telemetry. I mean, that's how I got my job at Splunk was because of my work in open telemetry while I was at, at that time I was at Postmates. And I wanted. They got bought by Uber. So I was looking for a job and yeah, I spoke to people working on the project at Splunk and yeah, hired me on. And yeah, so there, I spend basically 50% of my time on my

 

Mm-hmm.

 

maintainer duties because I'm a maintainer in open telemetry. So I get to work on learning and elixir aside. And then... 50% on what actually profits Splunk because they probably had like a, they certainly have Erlang and Elixir users, but they don't really know it because they're not big enough or they're big enough, but they're Erlang and Elixir uses compartmentalized in the company. So it's not their main technology. So it's not as known. So I'm. would be great to change that to have more. There have been, there's probably potential customers out there, I'm sure. Like when we were working at Heroku, we are all of our stuff from the Erlang nodes was going to Splunk, but it's not like they knew that it was Erlang, it's just another ingester they considered Heroku, I'm sure a Ruby company.

 

Yeah, that makes sense. That makes sense. Okay, so now... So you said you're in for the long haul, right? And now that this thing has at 1.0, but yeah, handcuffs. But what would you like to see next? I mean, I'm

 

So.

 

aware that of course you're not in the position to call the shots, right? But still, like, I mean, if... If you had a magic wand, right? Like, where would you like to see OpenTelevator evolve into?

 

I should mention, so for early Elixir and those other languages in the blog posts like Ruby, 1.0 is only for tracing. So I'm still working

 

Okay.

 

on metrics, still working on logs. And so yeah, there's still a lot to do. And aside from that, within tracing, there's still a lot to do around user, the... User experience, automatically collecting things like an open telemetry, we call them resources, which are information about the system that's running, what it's running on. So like if it's on Kubernetes, it would have a pod name and the IP and maybe the service it's in. That kind of stuff is automatically attached to your span. Well, yeah, your spans information. And

 

Mm-hmm.

 

We don't have any of that yet in early in elixir. It's not automatically detected. And you got to do this for AWS, Google, all of them. So there's still a lot of work there. Also, if anybody's listening and is interested in getting involved with some simple tasks, those kind of things, they're simple enough. They're not just copy. paste attributes because you do have to read the APIs of Kubernetes, AWS, GCP, but simple enough tasks if anybody wants to get involved. Those are good ones.

 

Where would they check it out? So where would they go if somebody wanted to be involved?

 

Yeah, GitHub open dash telemetry is the organization. And then we have open telemetry dash Erlang is the main repo and there's issues for stuff like resource detectors. And then there's also open telemetry dash Erlang dash contrib and that's where integrations with like Phoenix and Ecto are and there's plenty of work to be done there too. I mean, those are. those work. So if you decide to give OpenTelemetry a try, you'll get a lot of mileage out of those integrations, but there's certainly work to be done. I know in LiveView, there's still discussions in there. I'm not much of an elixir person, so I don't pay that much attention to exactly where they're at with stuff like LiveView integration, but I know there's still work to go.

 

Yeah, fair enough. Can't expect everybody to see the light that is Alexia. No, I'm just pulling on it. I know, I'm not gonna address an old or like, yeah, for that.

 

fear change, the, what was it again, the, oh, and yeah, that reminds me of the, within Erling itself, there's a lot of interesting work that can be done around integrating with open telemetry. Cause like I mentioned, one of the key things with distributed tracing is you pass this context from one node to another.

 

Mm-hmm. Mm-hmm.

 

We're talking between nodes all the time. but it's not HTTP, it's distributed early. So how do we do that? And how do we do it in a way that right now it's the user's responsibility. They have to create copy out of the process dictionary into a variable and then pass it as part of the message. And it would be great to streamline all that. And so that's something we've been looking at for years now talking with the OTP team as well. of how to do that. Cause one cool thing listeners might not have heard of, some of you would is seek trace, S-E-Q underscore trace in the beam. And that actually is distributed tracing built into the beam. It's been there forever.

 

Wow.

 

Yeah. Using Lamport clocks to get the order of things. And it's actually, It will pass context in messages for you without you having to do anything besides set the context in the token. And so you can already in Beam secretly pass context between nodes, between processes, and I mean, it's almost perfect, but because of the way it's built, you can only have one. So if we say,

 

Mm.

 

took that over, you wouldn't be able to do live Sheik Trace investigations. And we don't want to break that. The same with General Earl Tracer. There's some issues there around, you can only have one. I'd like to see that have the ability that you can add trace points that create spans and open symmetry on a running system, eventually. That, because I think if we do it in such a way that it's, not a core feature, but something you can turn on when you're debugging, you can add the spans. That wouldn't interfere with the core use case of Erlang's tracer. You'd still be able to use it for all your other tracing needs. But with SeqTrace, originally you could only have an integer in its context. It was just a single word of memory that you could have, and that got a couple of DPs ago, because we were working with the OTP team to say you could have any terms. So technically we can put the context in there and automatically ship it to the other process, other node. But yeah, there's still those questions around it. And so yeah, I'm excited about the future there, figuring out how do we really adapt this stuff to what being already provides you and make it even better than what all the other languages have so we can show our advantages even within this space.

 

Yeah, that makes a whole lot of sense. I haven't even considered that. I still remember like when we initially added like tracing and all those capabilities into an application I mentioned at the beginning, we had the same situation, like, okay, how do we pass this information through the system without polluting, because I mean, there was a discussion, do you want to add this additional argument to each function that felt really dirty? because it's this cross-cutting concern, right? Like it's orthogonal to your application logic. It doesn't really belong there. So we ended up using the process dictionary, which was still felt hacky, but better than

 

That's

 

nothing.

 

the perfect place for the process dictionary.

 

Yeah, I mean, it worked out nice enough at the end, but it was always something where I was sitting and it would be so nice if there was something more here, something which is like deliberately designed for that kind of information. So yeah, very exciting to see what the OTP team and what this collaboration potentially can bring us in the future. I feel, I mean, like... We already talked about telemetry, right? Not being open telemetry, but I feel there is a lot of motion in the beam world around, okay, how can we make things observable? How can we figure out how the system behaves at runtime? And all of, there's also a lot of wisdom in what the, the Erlang community, to be honest, has been doing for literally decades now.

 

Yeah, I mean, that's one, like the main thing that eventually sold me on Erlang

 

Hmm.

 

started out as a student and there was a lot of intriguing stuff about it. But once I got into industry, what really sold me was the observability was that I could,

 

Yep.

 

like when I, if I ever have to touch in production a system written in anything else, like go, I don't know what to do. There's no, you can't hop on a shell and safely. take a look at what's actually going on.

 

Yeah, definitely, definitely. That's definitely like a secret superpower. And I always tend to say that the beam is just a piece of technology, which is deeply pragmatic, deeply pragmatic in all the good sense.

 

Yep.

 

And that's part of it.

 

Yeah. I mean, yeah, it was written for industry and it shows it was written for getting things done.

 

Yeah, yeah, yeah, definitely. Okay, maybe to come back to a question like, where do you expect to see OpenTelemetry go in the future? I mean, like, maybe you already know some things that might be happening if you're willing to share that if not fair enough, right. But what is like the next thing the community and the beam world can expect to see already set like logs and metrics are still being worked on. So probably movement in that space, right? But

 

Yeah, pretty much all focuses on logs, metrics and semantic conventions, which semantic inventions are what will allow, uh, like the same spans and logs and metrics across vendors to be utilized the same way. So it adds

 

Mm-hmm.

 

specific conventions around like how you set an HTTP status code on a span, how you set the response size, what term you actually use there. So it's parsed. the same way by every vendor. And so aside from all the work going into metrics and logs, that kind of stuff, and once it's adopted by the vendors, we'll really see the improvements, I hope, in just the experience for Erlang and Elixir developers if they start trying these vendors, because things will just work. Where before, they would have to kind of hack around, depending on which vendor

 

Mm-mm.

 

they were using. change certain things to actually work for them. And hopefully it'll just work.

 

It's both like this abstraction layer of having open telemetry and open telemetry collector, right? Where you already said earlier that you might have exporters. I can definitely see the value there from being on the table. I mean, as I said in the beginning to try out things, right? And if it then really just works. No, that's like, I don't know. Like, I mean, I have a soft spot for all topics of observability. And that's like, Oh boy, I'm blushing. That sounds amazing.

 

The Collector is really nice because the vendors are writing their their exporters from it and so it's a centralized place where because there is still some at this time changes that need to be made in order for their backends to understand the open telemetry stuff. So that all gets done there. You don't have to re-implement it in every language or anything you send with the open... limit your protocol to the collector and the Datadog processor does what it needs to do to export it to Datadog and

 

Mm-hmm.

 

it works.

 

That makes a whole lot of sense for me.

 

They

 

I mean

 

are

 

that it-

 

like Splunk accepts open telemetry directly. I think, yeah, and Honeycomb, Lightstab, I'm fairly sure I haven't used, I don't think, yeah, Lightstab, Honeycomb, Splunk, we all accept open telemetry directly instead. You don't even have to go through the collector though. I still recommend setting up the collector because it's useful for debugging too.

 

Makes sense. Okay, Tristan, is there anything you would like to add, anything you would like to let the audience know, because otherwise I feel if we've painted in complete enough picture, right, so we could transition to pics. But anything else you would like to let a lot of people know out there?

 

Uh, I think we fairly well covered everything. I just want to get across the, please take a look and help out if you're interested

 

Yeah.

 

in this. Yeah. Cause we know we need a lot of help, especially, uh, even just, uh, reviewers and stuff to, uh, keep this stuff pushing along. Cause there's a lot of changes coming in from the community and contrib for improving, uh, the integrations with stuff like Phoenix. But there's only so much time to review it, so

 

Hmm.

 

we need all the help we can get.

 

That's always the thing with like projects like this, especially open source, right? Like there is never enough people to get all the things done you would like to see get done. So if people want to reach out and get in touch with you personally, Tristan, how would they do that?

 

I guess what I'm on Elixir Slack so you can just look for Tristan and on github you'll see me my name T-Slaughter under many many of the open telemetry commits on the Erlang repo so they can find me there and message me wherever you want.

 

Okay, then I'd say we can transition to the fun part of a podcast. Pics. So do you have any picks for us, Tristan, or should I start? What do you prefer?

 

You can start.

 

Okay, I already kind of blew bloom what I had earlier because we did a double recording earlier. So I'm just going to do some fun picks this time around and I'm going to pick a TV series I've recently started watching. It's on Amazon Prime and it's called Invincible and it's a comic TV series. It's based on a comic and it's basically superheroes, like a traditional superhero story. And I don't really want to spoil anything, but I want to say this, like the very first episode is like, okay, there's this guy, his dad is like basically Superman, he doesn't have his powers yet, and during the first episode he gets his powers. And then something happens at the end of the first episode where you are left like, what the fuck just happened? Which the series plays with viewers' expectations. Let's say that. And... So if you are into superheroes at all and don't have any problems with animation, then check it out, watch the first episode, watch it till the end, and then you can make judgement whether or not this is for you. So Invincible is my pick for this week.

 

Nice. I'll have to check that out. My picks since, well, I only saw that you were doing the picks today, and the top thing on my mind, because I've been using it the past few days on the Open Telemetry project, is Gradualizer.

 

Hmm

 

And there's an ELECTR version of it that uses it, which I almost want to say is named GRANIT, but I'm probably wrong. I should... probably pull that up. But Gradualizer has been amazing. There's rough edges, so if you try it out and run into things, definitely let them know they're very very responsive, very helpful, and it just improves it for everybody. But because I dialyzer all my projects and that helps with a lot of stuff, but Gradualizer caught so much. So many things that were wrong. Uh, it's, and it's so fast when you're used to dialyzer, which takes forever. And the gradualizer just fits everything out. And I thought something was wrong. It was so fast.

 

Hahaha!

 

Did it actually do anything? Uh, yeah. So that's, that's my big pick. Anybody who's been on the fence about checking it out, do so.

 

To be honest, I was under the impression that the project stalled, but apparently I'm wrong.

 

It might have at one point, but it seems to be back on. And I think it might even be, I don't know why I think this or what I saw, but it feels like Erlang solutions might be involved with the Elixir version. So, uh, it might have, that might've picked it back up. But yeah, the,

 

Interest

 

uh, yeah, the maintainers have definitely been responsive. I already got a patch in, uh, to make some stuff easier with the rebar three plug in. Uh, And I'll probably do some more on the Rebar3 plugin because I wrote Rebar3, so the co-creator of it, that helps. Try

 

That's

 

to help.

 

super interesting. I was actually at the Beamstock Home Conference where it was originally announced as also the more academic approach. He laid out his reasoning about what dialyzer is doing is success typing and what we are doing is really adding this gradual type system and we need to redefine a little bit what the primitives do mean. He had this whole talk about that. So it's... I was super excited for it back then because it made a whole lot of sense for me. Um, then a few, I think like a year back, I wasn't the impression that the development halted, but so, so great to see it's, it's still alive and kicking because I definitely think there's a lot of, a lot of value to be, to be gained from an approach like that.

 

Yeah, I've already gained a lot of value from it. Uh, and I know Fred Haber, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he, he,

 

Yeah.

 

he, he, he, he

 

You can also go back a few episodes and then we interview,

 

to review

 

interview

 

that. Nice.

 

Louis Bill Fultz. So like he has been on the show twice now. So the last time he was here, we talked about his type set implementation of ODP actors, which was also pretty cool.

 

Oh, very cool, I gotta listen to that.

 

Okay, then it was a pleasure having you Tristan. Thank you for coming on the show.

 

Thanks for having me.

 

And you, I hope you all enjoyed listening to us rambling about open tracing and tracing and all that confusion. And tune in next time when we have another episode of AlexiMix. Bye bye.