Marcos Ramos is a Senior Software Engineer. He joins the show with Allen to talk about, Debugging and Tracing in Erlang | AppSignal Blog. He explains the process of debugging and the tools that he is using. He shares his methods and tips for tracing and debugging.

Hello and welcome to an episode of Elixir Mix. I'm Elixir Mix. I'm your host, Alan Weimann. I'm hosting alone today, so sad. But not so sad because we have Marcos Ramos here,

 

Hello everyone.

 

who wrote a very interesting article about debugging Elixir. I know we talked about debugging before, but it's always good to hear from other people what's been their experience other than just us. So welcome, Marcos. How are you?

 

Thank you, Allen. Yeah, I'm good. Great. Thank you for the invite. We're glad to be here. And yeah, when I wrote this article a couple of months ago, it was basically writing about my daily experience with debugging. So this is something that I'm really comfortable talking about.

 

Yeah, we were talking before the show, like my experience with debugging is just I own specs everywhere. I totally forgot about the, the deep, uh, GPD

 

DBG.

 

is it deep? GPD, DGG, PPPT, I don't, I forget the characters now. It's

 

Yeah, it's a short hand for Debug.

 

Ah, that's what it is. Okay. Cause I think there's also GDB, which is like a debugger too, if I remember correctly.

 

Yeah,

 

So,

 

but GDP is a C, C++ debugger for native applications on Linux.

 

Okay.

 

It's a different piece.

 

And another different basis, PDB, which

 

Yeah.

 

is the Python one.

 

Yeah.

 

Okay, it's good to hear you know about that one. Yeah, I've used that one quite a lot in when I was working with Python. So I remember it's like import PDB, PDB.setTrace or something, and then it stops right there, basically like pry. It's been a lifesaver. But so it's, we said DBG. And yeah, I mean, maybe we can start from the top of your list because actually you don't even cover Do you actually cover it? I know you do cover it, but with Tracer.

 

I think I just talk about it, like very,

 

Yeah.

 

very briefly.

 

Okay. Yeah. So you mentioned you got Ion spec, the I ex pry in dbg. And I think we said dbg is like a brand new module put in right

 

Yeah, so, so DPG is a, I, at least how I see it is a replacement for I own inspect. When you use I inspect, you do have like some sort of introspection about whatever you're looking at. But with a DPG, you have more powers on what you can see from that variable from that function call. For example, it gets you the formated output. So you can just base on an AI session, for example. and the itu work. If you're running on a XSession, it will stop and you can pride a code. You can look at the variables, foot can calls, all this stuff. And it works in the same way as Ion Inspect in the sense that you just put it there and it will work and it return the same term that it was reading. So you can pretty much replace Ion Inspect with dbg at any point. There are some caveats like for example if you're running a Phoenix application on IEX, if it encounters a DPG call, it will stop the request because it was queued to Pride and you have to confirm on the console that you actually want to Pride that code. So apart from that, I think it's a good replacement and I've been using it a lot. A lot.

 

Well, it's a little bit confusing, right? So you're talking about dbg the function call, right? Because there's also

 

Yes,

 

dbg the module, right?

 

there is the DPG without the two dots in the front, which is the new function from Elixir 1.40, and

 

Yeah.

 

this one is a replacement for IO.inspect. Yeah,

 

Yeah.

 

and there's also the 2.dbg, which is a whole model from Erlang. So even though they look like they're separate things.

 

Yeah, definitely separate things. So don't don't get them confused because

 

Yeah.

 

like that, I went to Alexa and typed in dbg and I'm looking at your your article. You don't even discuss the dbg function that I can see. And maybe I need to check again.

 

I just

 

I don't

 

talk

 

think I've

 

about

 

seen. Yeah.

 

it and I say, hey, it exists. You can use it.

 

Yeah.

 

But

 

But you don't give an example. You don't go deep

 

yeah.

 

in on it.

 

I think AppSignal has an article

 

Thank you.

 

about DBG.

 

It's possible. Yeah.

 

Yeah.

 

It's brand new, like you said, one dot 14, which is

 

Yeah.

 

obviously the latest version. And I know one of the one of the big one of the big maintainers wrote it. I forgot who it was. I think it's I forgot the name now. But I know it's it's kind of a big deal. Like I'm kind of curious about the background of that. Why they thought that was a good idea to add in. I mean, I'm not saying it's bad, right? But I mean, I own Spectworks wonderful for what it is. But do you know anything about the background of why this needs to be there? why they decided to edit it, I think.

 

As far as I recall, it was because sometimes the IOS inspect would cut some of the outputs while DBG can print everything and DBG can also print in a way that you can reuse it and just based on the NIX session or in a script file. So

 

Okay.

 

I think it's more of a need that arose like over time from developers. they eventually added it to the language.

 

That makes sense. I think I ran to an issue recently with one of the developers over here. He was trying to output JSON and he kept the slashes in there

 

Yeah,

 

for

 

the

 

when

 

skip

 

you have

 

characters.

 

like, yeah, and then he's like, why

 

That's

 

is this

 

a

 

not

 

kind

 

working?

 

of annoying.

 

I'm like, yeah, I'm like, that's because you're, you need to remove those back slashes. That's why. I'm like, yeah, I'm like, that's because you're, I mean, it gets output as a string rather than printing it as a string.

 

Yeah,

 

Right.

 

and with TPG you don't need to require ix.pry or anything.

 

Mm-hmm.

 

And it's really more a developer quality of life too.

 

Yeah. Yeah, it gives some pretty good information, like, like what line something is on. Let me actually go back to that. I was just looking at that. And you don't even need to give it anything to it.

 

Yeah,

 

I'll make you just take

 

and

 

this

 

the cool

 

round and

 

thing

 

code.

 

is that if you put it in a call of chain calls with the

 

Mm-hmm.

 

pi-vaporator, it

 

Yeah.

 

even says where it stopped, so you can see which step your stopped.

 

Hmm. Yeah, it gives the line of the file to right.

 

Yeah.

 

Yeah, the line of the file, the name of the module, the name of the function, which is quite interesting and gives you

 

So

 

like

 

for example, things that probably everyone does is, but a lot of calls to IOT dot inspect over the application. And you don't know where it's coming from. Usually people use labels to filter that. Hey, I'm here step one, step two, whatever.

 

That's what I use.

 

Yeah, and with DPG, you know, the module, the line, the function that's been called, which is really neat.

 

Okay, yeah, it does. I think I started to use it a bit. And it looks like you can, yeah. There's also, there's more options besides a label that you can have. Like this is, there's like a width you can do for IO Inspect.

 

Yeah, you can

 

Label.

 

do

 

And then,

 

a whole sort of things.

 

Yeah.

 

But I think when you're writing code, you really want quick information. That's why people rely a lot on IO.inspect because it's easy, it's well known, it's been there like forever. And that's why I'm here. I'm here to help you. I'm here to

 

Yeah.

 

help you. I'm here to help you. I'm here to help you. I'm here to help you.

 

Mm hmm. Yeah, definitely. It's, uh, it works great. And I, I totally forgot about it because it's just, yeah, it's not, it's not been, it's a brand new thing. Um, I mean, prior, I think I stopped, I don't even use prior. I thought prior was pretty cool to have, but do you actually use prior when you're debug?

 

Sometimes, yeah, for example, sometimes you want to see what's happening during the execution of a function, not after. So you can just require pry and then call ix.pry. For Elixir, for Phoenix applications, that's really good because you usually run mixed.phs.server on the terminal anyways. So it's really easy to grab the information that's going on there. Especially when you're doing maps or reduces when you have loops and then you can see the evolution of the accumulators or something like that. That's when bright really shines in because you can stop, you can see, you can check execution. Otherwise, if you put like an a.inspect call inside a noon.map, if it has like a thousand of items to process, you just get like a very noise console. Sometimes we don't want that.

 

Yeah.

 

And then it really depends on what we're doing, right? So,

 

Yeah. Yeah. That, that is the annoying thing is like, I remember debugging this thing with, uh, with one of my co workers and he had I own specs everywhere. And I was like, can you just remove the ones that we just don't care about right

 

いしー

 

now because it's very noisy. And then like your term look on his handle, like so much characters also just lines

 

Yeah,

 

of code coming out.

 

and the thing about what I really like about the debugging tools that we have on their long-end elixir is that you have a specific tool to solve a specific problem. So for example, I'm doing development very quickly. An IOPN dot-and-spec call will do the job, but sometimes you need a little bit more and then you can use dbg or you can use deprive from rex. Sometimes you don't have access to the code, like in a runtime environment, like production, staging, whatever. And Elixir and Erlang also offer tools to debug this kind of environment. And that's, I think that's the main point of these two articles, when we start debugging by changing code and adding what's called probes, I'll call probes inside the code And you have tools for that. But if you don't have access to the code and you want to probe it anyways, you can also do that using dbg. And I think that's very, very interesting to see, especially because with the dbg tool, the debugger, not the function, by having the power to know, to see the execution of things in production, you can, see things that you never see in a local environment. And there are stuff that you can only see in production because users are using it. There is a lot of real-world data, you know. And I think that's the big value and the big win for having such a tool inside your language, your framework.

 

Interesting. So, okay. Excuse me. With the debugger, you don't even need to have the source code or anything. You could just take a running instance and then hit a debugger on it.

 

Yeah, it's easy when we talk, but the functions are kind of confusing. But essentially with the debugger from your launch, you kind of write a rejects for function calls. You give it to it and you give this rejects, but you also tell to the debugger which process you want to list them. So for example, if I have, for example, a gene server that has a handle call function, for example, I can tell the debugger to only listen for that process and only for the GIN calls. Handle call, sorry, for handle call calls. And from that point on, I can see what's happening in that process and I can get notifications whenever that process gets a new call. So, and this is one of the examples that I wrote for the article where we have two separate nodes and you activated the bugger on these two nodes, on the process that sends the message, and on the process that receives the message in another node. When you send a message from the IXGL in one computer, you see the whole execution, like the user typed in, I received the message to send, I sent the message to another process, and I printed this message on the other node. You can do this from one computer, as long as it's connected to the cluster. And that's pretty much it. You have lots of small functions that you can use to make these rejects, let's say, of functions more specific. But that's the whole idea. You'll tell where to look, and it will look for you, and you get notifications about that. And that's it. I hope you enjoyed this video.

 

Hmm. No, that, yeah, that, that's part of the runtime tools, right?

 

Yeah.

 

Yeah. So I don't think a lot of people actually know about the runtime tools and actually what's in it. I mean, the tracing for sure is in that the, so the buggers also included in that runtime tools.

 

Yeah, so it's available for all Elixir and other developers and on whatever environment you are in. It's a good thing, for example, I've used this tool to debug a problem where I have Phoenix WebSockets and Phoenix PubSub sending messages in a cluster. And sometimes we would drop some messages. And we didn't know why we could not reproduce. And then by attaching the debugger to two nodes, I was able to see the messages being exchanged by the PubSub. And then I didn't find the problem. which was a case statement that was not handling a message properly. And because the way the process was modeled, we, it was exploding in a way that it was not really, we were not able to detect what was really happening. But with the debugging, we were able to see what was happening and see the message has been yet changed.

 

You're saying it's exploded, so you're saying the process was crashing or what

 

The process

 

exactly

 

was

 

was happening?

 

crashing, but it was hiding the error for us. So even

 

Thank you.

 

though

 

Thank you.

 

we knew that the problem was there, we knew that something was going on, we really didn't know what was happening until we connected the debugger and we saw the messages and we saw, okay, we are not handling this payload correctly.

 

And what you're trapping messages is that why you'd never saw the error coming out or what? Because I'm kind of curious about how you would be, now you not have the message.

 

Yeah, let me just remember, we had a user sending messages and this message is goes over a WebSocket and this WebSocket published on the PubSub. And then I don't remember if we changed the payload and one of the case statements was not ready for that, but it was an async process, you know, so and this process was not attached to the main was with task async no link.

 

Okay.

 

We didn't know why and it was not crashing the whole tree. But when you saw the message, you understood what was happening.

 

I see. Okay, interesting.

 

Yeah, and that's the thing. With the actor model, even though 99% of the time, we use the stuff that Phoenix provide for us, that actor provide for us, and we have all this process all right there. When you start working with processes and adding your own stuff on top of the existing frameworks, sometimes you get this kind of problem where a process go wild or it breaks and it doesn't affect your supervision tree. And that's why these tools exist, because this problem happens. You know?

 

OK, and then you said this, you did this all through tracing and that helped her, that was debugger, right?

 

Yeah.

 

Interesting. So you could actually run in, like, you're running two instances of the debugger at the same time and then walk in through two different nodes.

 

You can run just one instance, connect the nodes, and then

 

Okay.

 

the instance where you run the debugger can see all the nodes that you have connected.

 

I see.

 

So for example, if you have a VPN with your production environment, you can connect the debugger into one machine, and then you can see the whole cluster, and then you can debug the whole cluster.

 

Yeah, I'm just trying to think about, because I mean, when people think about debugger, like usually they're thinking about like, okay, you know, single line execution, I'm stopping my program here. But like Elixir, it's like you're actually just pausing a process there and

 

Yeah.

 

everything else is still running. And so, I'm just trying to think about how to do this, and how to do this. And I'm just trying to think about, because I mean, when people think about debugger, like usually they're thinking about like, okay, you know, single line execution, I'm stopping my program

 

You

 

here.

 

can do that. You can do that. The way I use it is that I tell the debugger to send a message with the variable to my process. Right? So you can extract the parameters of a GIN

 

Oh,

 

server call, for

 

interesting.

 

example, and get it

 

Okay.

 

on your machine where you're looking at. So you can extract the parameters of a GIN server call, for example, and get it on your machine, where you're looking at.

 

Oh, sorry, it's a little bit late over here. Okay. That's interesting.

 

Thank you. Bye.

 

Yeah. I had a late night last night too. I had another podcast recording, so I apologize if I'm yawning. Um,

 

No problem.

 

okay. So, oh, okay. Cause the only thing about debugger is like you, you usually, you pause the execution, but this one, you know, for debugging, you don't have to pause it. Cause what you're describing sounds more like tracing rather than pausing,

 

Yeah,

 

rather

 

yeah.

 

than debugging.

 

Yeah, the name is confusing. You can use the

 

Okay.

 

dpg2 to debug and to tracing, right? But you can, yeah. And that's it. Thanks for watching. I hope you enjoyed this video. I hope you enjoyed it. I'll

 

So you can pause if you want and inspect what's going on.

 

see you in the next video. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye.

 

Or you can say, when you get to the sign execution,

 

Bye.

 

send me a message and let me know what's the current state of certain variables. So I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test. I'm going to go ahead and do a quick test.

 

Yeah, and that's why it's really dangerous to use this in production, because you can't stop the whole process from working, because you told the debugger to stop the process, to pry, to see what's going on there. That's why I usually try to do as much on local hosts as possible, if not, and if I can connect to the staging or production environment. When I do that, I really want to do, possible and not change stuff too much and try to not be right. The bugging code that's hard for the environmental process. Because when you enable this thing, everything gets really slow, really slow. So you have to be careful when using that.

 

Yeah, but this stuff is really confusing. Right. I'm, I'm walking through your article and to me, I'm thinking, okay, pry, you know, all this stuff for prize in there. Um, you're like for debugging, right? You have to run debugger at least. So DBG is also like debugger. So debugger full, full word will give you a graphical debugger.

 

Yeah.

 

DBG is a text-based debugger.

 

Exactly. They're the same

 

Okay.

 

tool, just

 

Yeah.

 

different interfaces.

 

So that's a little tiny bit confusing, but okay, it makes sense. You know, obviously sometimes you don't have a UI or you don't need or want a UI. Makes sense. Now, here's the tricky part is you have another module called int. No relation to integers is my understanding.

 

It's actually for interpreter.

 

interpreter. Yeah,

 

Yeah.

 

exactly. But it's like, Hey, so I'm running the bugger. Okay, DBG debugger. Okay, got it. Once the different ways of accident, the same thing. Not a big deal. But now the end thing kind of throws me off because it's like,

 

Yeah.

 

okay, but you're saying that you need to use this in order to actually work with the debugger.

 

Yeah. So the pressure model.

 

So you guys cannot see at home. He's smiling

 

Yeah.

 

because he knows it sounds very confusing as I'm explaining it. Yeah.

 

It

 

Yeah.

 

was confusing when I was learning this stuff. For example, if you look

 

Okay.

 

at some functions from the DPG module, you see stuff like TPL. What the hell is TPL? And then when you read it, you understand that it's trace pattern local. Or you have to trace all patterns to the local

 

Yeah.

 

node. But when you look at just TPL, it doesn't really make sense. But yeah. Anyways. So, why do you have to use the interpreter module to work with debugger? Basically, the debugger knows how to stop processes and get processes and inspect processes. But with the interpreter module, you tell where it's going to do that. So, and I even write this in the article. Yeah, found it. The idea is that you start a debugger process and then this process will hold state about your debugging stuff. You tell the debugger what process you want to debug. And with the interpreter module, you can start doing stuff like, okay, stop at line 60 or stop at the function called X because the interpreter module has the context about the code itself. and then you use the two together to interpret your code at runtime. That's more or less the idea. I'm not sure if I was clear enough.

 

Yeah, no, it makes sense. I guess it's going back to solid principles, single responsibility kind of.

 

Not good

 

Okay.

 

at naming though.

 

Yeah, the name is, I've never seen the word int used for something besides an integer. That's why it took me a moment to like, what am I doing? An interpreter,

 

Yeah.

 

like I, of course, yes, we're using interpreter, but I nearly forgot because it is a compile language, right? Usually people think of interpreters, they just run the code without the language in between, right? So

 

Yeah,

 

you

 

and

 

have

 

if you look

 

int.

 

at it,

 

Mm-hmm.

 

the.ni is like

 

Yes.

 

interpreter load this module. And

 

Eh?

 

then because you have the compiled module, like the.bin, and you have like the source code, and you have to bind the two together at runtime because it doesn't exist at runtime, the code. So you kind of have to bind them together so you can work with. And then same thing for interpreter when you want to add a break. This is actually the budding stuff when you want to

 

Okay.

 

stop execution. And then you say, hey, I'm looking at this code, and I want to look at this module, and I want you to stop at this line. You have to tell the interpreter to do that, you know?

 

I'm trying to read what the word and I means. I don't know if they have a explanation of that. Let's see.

 

It's a new interpretation,

 

I'm

 

new

 

okay.

 

interpret.

 

Yeah, because they put I and N I together. So I interprets the module only at the current node and I interprets the module at all known nodes. So does

 

Yeah.

 

that mean that you could have the same module with different implementations? Again, I guess you can because you can have two at the same time.

 

Yeah, you can have two or lung nodes not sharing same applications.

 

Yeah. So how would you know which one you're looking at then?

 

That's a good question, never thought about

 

And

 

it.

 

nope, you need to think about that.

 

I never run a cluster with different applications. There was never a concern for me. I

 

Yeah,

 

never run a cluster with different

 

I mean,

 

applications.

 

yeah, but I mean, that's one thing. The other thing too is, um, if you're in the middle of say, like a hard code upgrade, and you run this thing, some nodes are running an old version, some nodes are running a new version,

 

Yeah.

 

which is a bit confusing. But I'm still a bit confused about how does it get the actual source code then. So that means you have to be running this within the context of the nodes running source code. So you would say, okay, I X dash X dash S max and then connect.

 

So,

 

and I

 

exactly. So the

 

Okay.

 

compile code, the compile code dot being file has the information about lines, even though the source code is not there, right? And that's how we can get documentation

 

catch you.

 

and other stuff. So you can tell the interpreter to stop at a specific line and then you can see that. When you run this thing locally, you can see the source code and you can see the line stopping with the GraphQL interface Sort of the bug in a way that we are used to like in a Graphical ID something like

 

Mm-hmm.

 

that But when you don't have access to code you have to use other resources like the Function of function to Sorry forgot the name Fun to MS which translates functions to match specs. Sorry. Yeah match specs Well, yeah, I think I got confused here because to use the int module, you have to have the source code while the match specs you use when you don't have access to the source code. You only have access to runtime information.

 

Okay. Yeah, that makes sense. And I just looking around like some more, um, when you do have the debugger, you have to be a little bit more careful, right? Like where it read, Alexa gives you a nice friendly interface. And then, um, when you look at it in the debugger, right, you're going to get a different looking variables with the underscore at the beginning and then the number at the end for when you rebind everything.

 

Yeah.

 

Yeah, I know I had this problem trying to explain to people that like, okay, these are actually immutable. But then you're like, no, you're changing it here. I'm like, no, no, no, I'm not. I'm not changing it. I'm rebinding it. It's totally different,

 

Yeah.

 

man. Totally different.

 

Have you ever seen how Elixir solved the problem?

 

Well, I mean, I heard about it. I know that they're doing some stuff right at the hood, but it looks like you can kind of see that where you have the

 

Yeah,

 

underscore name

 

when

 

at.

 

you debug within the Erlang debugger, you see the actual variables.

 

Yeah.

 

So you have, let's say you have an X variable. And then when you rebind that, you actually have on the Erlang version of that code, you have X underline 1 and then X underline 2. For Elixir, it's all X. But behind the scenes, they are different variables, actually.

 

Yeah. How come when I'm counting from zero, I thought we're programmers?

 

Maybe they come from zero, I don't remember, it was being a while.

 

Okay. I was just thinking about that. I got this, you know, I ran into a, I was debugging some code with somebody the other day. He's like, and he's like, this is the zero th one, but it said like one on it and like, you know, normal people thing. And I'm like, doesn't say one. He's like, no, but we're programmers. So we got to talk about this in terms of zero.

 

Thank you. Bye.

 

All right. All right. Fine. As long as we're on the same page.

 

I also get confused with the shorthand for anonymous functions, where

 

Yeah.

 

the first variable is actually the number one. Instead

 

Oh

 

of

 

yeah,

 

being the...

 

okay you're talking about the ampersand.

 

Yeah,

 

I forgot the name

 

Empress,

 

of that style.

 

it's

 

Yeah. Yeah.

 

the anonymous... It's a shorthand

 

year.

 

for anonymous functions.

 

Yeah, is that what's called shorthand? I thought there's another name for it. Ampersand function, the capture operator, right? The capture

 

Yeah,

 

style,

 

exactly.

 

that's the one. Yeah, it took me some time. It's also a little bit weird. Maybe there's something that they have to use the first and second. And that took me some time to figure that part out because it was just so weird. It's just not elixir like in my immediate thought process. Yeah. Okay. Um, yeah, well, I'm good. You I'm good. You talk over that. Um, actually, I want to actually zoom in. I want to actually zoom into this picture. Kind of my machine, but the picture you have an article is a little bit small, even when you click on it to make it bigger. But yeah, because like, I mean, if you're If you have an eight no cluster and you say, okay, at this line. do some debugging activity, whether it's stop or send me a message. That means for any of these running nodes and any of these processes could be triggering that. So if it's a very busy process, you may have a ridiculous amount of messages coming in, right?

 

Yeah, that's where the match specs come in handy. Match specs are something that are very hard, very cryptic, and I never work with them directly. I always use some sort of to translate it to a match spec. But with match specs, it's kind of a reject for functions, for token calls. And you can specify what kinds of attributes you're looking at. So for example, you can do a match that have a specific value for the first variable. So even though you may have like eight nodes connected, several process sending messages all the time, you can still use matchbacks to future very specific information. So let's say that I'm tracing, I'm trying to trace a specific user or a specific message from that user. And I know that in one of the distributions, Then I can write a match spec that only match functions that have that user ID. So

 

Okay.

 

the debugger will take care of the rest for us.

 

That's like similar to like the TPL you're talking about, right? Yeah, over here, DBG

 

Yeah,

 

TPL, similar to

 

yeah,

 

that one, no?

 

yeah. So

 

Okay.

 

in the article, what I used was like a very, very naive version of a match spec that was matching all function calls of those

 

Mm.

 

functions, but you can be very specific of what you want to see from your cluster.

 

Yeah, you basically use the... Isn't that a similar style of like the recall that the line? So that's a DBG module. And then particularly, is it the TPO? Yeah, TPO.

 

Yeah.

 

Because I think the last one is about the arguments, right?

 

Yeah. And you

 

Thank you.

 

see

 

Thank you.

 

the first, second,

 

match

 

third,

 

spec.

 

fourth.

 

Okay.

 

Yeah, the fourth argument is the match spec.

 

Uh-huh.

 

And I use the name to match spec to match like every function call. So with the TPL, I'm saying, OK, I want to look for the messaging.message in server module, the handle call with three arguments with this match spec. And this match spec that I used captures all calls to that function, but you can filter out what things you want to see or things you don't want to see.

 

Yeah, the trace pattern, it uses the same trace pattern, it says so hard for me to understand this pattern and how how it works, like I understand like a bunch of underscores and stuff, but this one, you're just passing an empty list and that basically means everything.

 

Yeah.

 

But how do I mean, I'm going to learn this pattern, right? But how, what's

 

Ah,

 

the good place to learn it?

 

yeah, so

 

Even your, it's not easy. That's for sure.

 

it's not easy. Even Erlang provides a helper function so you can write a function that is translated to a match spec. And if you look at to fun to EMS, you'll see what it means because it's a whole thing about position and match-packing. And I don't really know how that works. to be honest.

 

Okay, yeah, you can pass in a function and then that's going to interesting.

 

But you have to be careful with this function, dbg.front2ms, because it generates specs, but you have to tweak around to work with Alexa. It's not

 

Ah,

 

like

 

I see.

 

super, super different stuff. It's just you have to be careful, because this spec that's generated, sometimes it can get really complex to read. And you have to know what's going on there to do the appropriate fixes.

 

Okay. Yeah, I'm happy to hear that somebody else is not really understanding the match spec because I do remember that one. That match spec is used in so many places, right? So we talked about

 

It is.

 

the the tracer

 

It is.

 

DBG ETS was the other was last one that might

 

Yeah.

 

bring up ETS is a big one.

 

Yeah. To be honest, I've been using chatGPT to write this stuff for me because it's just...

 

So that's the useful part. Okay.

 

Yeah.

 

It could get it can actually guess them properly and put into our elixir.

 

Kind of, kind of,

 

Okay.

 

so what you can do is like create your function, use dpg and fun2ms, get what the, what chat dpg is for you, and then you can compare it, test around, it's not really, I would say that it's not like an exact science because you have a good feeling there.

 

Okay. Well, it's, it's so you're saying even chat, GPT cannot figure this thing out

 

Yeah.

 

entirely. All right, then I don't feel so bad anymore. Usually chat, you

 

Ha ha ha 

 

can figure everything out these days. I don't feel so bad anymore. OK. Yeah, and you can also do a custom handler for the tracing, too, right, and for debugging.

 

Yeah. And that's what I use in the art course. Well, because

 

Yeah.

 

the debugger from Erlang will spit Erlang stuff. And

 

Yeah.

 

it's very Erlang-y, if I can say that. And then...

 

Oh yeah. Like strings or have the binary

 

Exactly.

 

stuff at the end. Yeah, that's really annoying.

 

So I use that handler to actually translate it to something more of the elixir world that I can edit.

 

That's nice.

 

And this is where you actually want to connect to the outside world. If you want to connect this to

 

Thank you. Thank you.

 

OpenTelemetry or Telemetry and send data. Of course, this is not something that you want to do in runtime, because you're making the application slow so you can debug it. But it's also a possibility that you have. You can connect your tracer information directly to your tracer vendor. Your telemetry vendor. And the idea for this article was more like, hey, there is this tool and it's possible to match back. It's possible to connect stuff to the outside world. You can run it in clusters. You can run it in your local machine. That's how you use it. And that's the starting point because it really depends on what you're going to do, what you're going to inspect, what you're going to debug trace. And if you look at the dbg module or the int module, you see that's a lot and lots and lots of functions. And they each do something very unique and very different. And it really depends on how you do that. So the idea of this article was more like to present the main ideas like

 

Hmm.

 

function match packs and debugging and interpreting a module, connecting nodes, you know. And also the example that I gave that I wrote for this. where two different nodes exchange messages was intended for showing showcasing these tools. So you can see like a starting point, like, hey, I need to do this, I need to debug my application, where do I start? And

 

Yeah.

 

I think this article does a good job at starting and showing the way to start debugging and stuff. get the ideas like, okay, you have process, function calls, nodes, and how this all plays together. With this information, you can write your own tracing stuff, your own debugging scripts.

 

Yeah, I like that you show like an example script like, okay, you know, let's let's put the last few pieces together into an excess script, Lexar script, or I don't

 

Thank you.

 

know

 

Thank you.

 

what that's actually called. I guess it's called Lexar script. So

 

Yeah.

 

without ex file, uh, you know, you create your handler, your custom handler, you start the debugger, you start the tracer, uh, you attach this to the nodes and you know, you just started checking everything out and how it's all going. And that's, that's really nice. That's, uh, that's like a nice little like that you can add into your production system to see what's going on. Yeah,

 

Yeah,

 

I like that

 

and

 

one.

 

you can use the script as a starting point and customize it for whatever you want to look at in your application.

 

It's not copyrighted. There should be a license on this one, right? MIT or something

 

Oh, it's,

 

so we don't

 

yeah,

 

know.

 

it's actually open source. It's in the, oh my GitHub account. It's, I think it's MIT. I'm sorry, I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry.

 

How

 

I'm

 

is

 

sorry.

 

it?

 

I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry. I'm sorry.

 

Oh, great.

 

Yeah.

 

Now I can use this and don't need to give you any, I think you don't

 

Yeah.

 

need to give any feedback or any, you don't need to

 

Yeah,

 

point

 

I

 

back

 

think

 

to anybody.

 

MIT is very permissive. Do whatever you want. I don't care.

 

Yeah.

 

Right?

 

Yeah, I quite like the MIT license just because I don't need to worry about anything. It's like, here's something cool I made. But guess what? No warranty. Do whatever you want. Don't care.

 

Yeah.

 

So MIT license is pretty nice. Yeah, this is pretty cool. Yeah. I mean, the stuff you can do with Elixir and Erlang is crazy. Like I actually, I learned something when I was watching this video where like this insurance company, they had a Super Bowl ad, their, their site got swamped. They found an error. They, they traced it out. And then the really cool thing is that they managed to fix the bug locally. And then without even deploying that you can actually just code push your module into

 

Yeah.

 

a running system. And then it just updates. I was like, whoa, that's, it's not a hot code. even it's technically not a hot code upgrade in terms of what

 

Yeah.

 

we think about, you know, do I need to play me? You just like connect and just push and it's like, Oh, that's awesome.

 

Yeah, that's not something that I really wanted to do like in when I was when I'm on call Like you have

 

Yeah.

 

to you have the nerves to do that, right?

 

Well,

 

But

 

I mean,

 

they really

 

if you ever need to restart your service, it's gonna be the old code, right? That's

 

Yeah.

 

the problem. Ha ha ha.

 

Yeah. But one thing that I really like about Elixir is really the process and the actor model because you can do so much crazy stuff with that. And as I said, I'm working with AI lately over the past two months. And

 

Yeah.

 

even though the tooling is not there yet with Bubblebee and NX, it's evolving fast. But for example, we have this use case we receive some messages from a customer and you have to process some models to extract data from this messages. And on Elixir, I can just run TaskAsync, run this model on this input, and I can just wait. So I can run like dozens of models at the same time in parallel. And if one of them crashes, it's OK because there are different processes. I don't care. And I was writing this, I was thinking, if I had to do this like in Java or Python or whatever other application that doesn't have these actual models, it would be like a pain because I would have to start a thread and then I would have like semaphors and locks and all sorts of stuff that I can just like, I don't even have to think about that because it's there, you know.

 

Hmm.

 

I think this is the secret source of Elixir that really makes me happy as a developer.

 

Yeah, I just wish other people would give it a chance, you know, everybody just

 

Thank you.

 

like,

 

Bye.

 

oh no, Alex, there's too new. It's like, not exactly. It's kind of like another flavor with some good stuff on top of Erlang, which is battle tested and supreme.

 

Yeah,

 

It's

 

when you see the debugging stuff and the applications and all the tooling that we have on Erlang and Elixir, these are all stuff that we see new projects showcasing as new stuff. And

 

Hmm.

 

when you look at Erlang, it was already there all this time, in stability, supervision trees, you know.

 

Well, the observer itself is actually pretty crazy. Like,

 

Yeah.

 

I don't know of, I guess JVM has something similar, but

 

G him

 

I

 

this,

 

can't think of

 

this, this habit.

 

they have a launch pad or something launch something, I think I forgot the name.

 

You have a different set of different tools because

 

Yeah.

 

Java is really old at this point. So you

 

Yeah.

 

have a different UI, so you can look at that. But

 

probably

 

yeah,

 

Java

 

you have

 

swing.

 

this, yeah. You have all this stuff there. And that's it. I hope you enjoyed this video. I hope you enjoyed it. I'll see you in the next video.

 

Yeah, but I mean, like anything else that has something like this, I mean, I guess you can maybe you can count the chrome tools

 

Kind

 

for

 

of,

 

JavaScript

 

yeah.

 

people in the front end kind of. But like, what else has something like this? I can't think of anything.

 

like building in the language where you can just open a terminal and observe and start. I don't

 

Yeah.

 

think we have that.

 

Yeah, that's like

 

I

 

the crazy

 

might be

 

part.

 

wrong

 

It's

 

but I never heard of it.

 

exactly that's like the crazy part. It's like, come on, why? Why does this thing? It's built in the 80s have like a built in GUI observer where you can start doing whatever you want. And just kind of running nodes and push fixes, you know,

 

Yeah.

 

everybody wants to build a hot, hot code push updates. Well, now you can, but I don't recommend it. It's because if it gets

 

Thank

 

restarted,

 

you. Thank you.

 

that's the problem.

 

But even if you do something like that, you can still use tools like tracing and debugger to

 

Yeah.

 

monitor what's happening. And that's why I said that the dough exists because the problem also exists. So probably they run the same problems that we run in the 80s, and that's why this stuff has built at that time. They are not facing all the stuff that all the problems that we see today. So, yeah, that's it. I hope you enjoyed this video.

 

Oh yeah. Yeah, definitely. I heard they used to like test this stuff by just yanking out wires out of the system and see what happens. How does this thing recover?

 

Yeah. And Erlang was designed to run on computers that would break, but on clusters that should not break, right?

 

Mm-hmm.

 

And I think all the good stuff that we see now comes from the pain of these developers 30 years ago that had to create all this stuff so their system would work flawlessly. And now we can benefit of all this tooling that was built over these last few years. 30 years.

 

Yeah. Yeah. And I need to crack open. I mean, I think maybe kind of let's, let's kind of ask you about like, how did you manage to learn all the stuff? You didn't just crack open the early documentation and start reading from cover to cover, I'm guessing, right?

 

No, first because it's a very technical documentation. Even for developers, it's hard to read because it assumes that you know a lot of this stuff. And I'm not an alone developer. I know a bit of alone to do some stuff here and there. But I'm a lexadeveloper, right? And the first thing that I tried to have was an IDE that could debug Elixir code for me. for VS Code, we do have some extensions for IntelliG, but none of them really work flawlessly. Like I was the Java developer before and the debugger on these ideas is like super, it works really well. Even JavaScript, if you debug on a Chrome browser, it's like perfect debugging tools. And we don't have that on the IDs that we have for Elixir. And that's where I started options to debug Elixir code. And then I stumbled across... some forums and some people trying to do the same thing. And they would all say, hey, Erlang has this tools, right? You can use that. And then it was really like a trial and error until you understood how everything fit together. So it was more like a process and not really something that I intended to do because it was for me, I would just press a button on my ID and get the break points and everything. But this thing was not there. That's why I went for DDBugger and stuff.

 

Yeah. I still miss having the debugger within my IDE, but this still works. I mean, the nice part is that you can use this one within a running system. That's probably the nicest part.

 

Yeah. And also, because everything is immutable in Elixir, it's easier to debug. So even though I miss having an option or tool to debug my code on my IDE, it's not like a dear breaker for me because it's just more easy to work with Elixir because of immutability, processors, and other stuff. So 99% of the time that I was debugging something, it was trying to discover what the hell is the value of a variable. Why, what changed that? And I don't have to care about that anymore.

 

Yeah, actually, that definitely is something that was there before. Yeah. But now we don't have that. Now it's okay. Uh, what changed this new binding? Like, or, or why did it? Why is my thing like that now? You know, as it's going through the pipeline,

 

If you

 

that's

 

avoid

 

a huge one.

 

rebinding variables, this is also a problem that you don't have.

 

True to.

 

Yeah.

 

Yeah, I mean, I'm gonna start using the pipeline a lot more. I think it's a lot more easier now. It's just, I don't know, simple logic errors here and there. Well, for you, it's I mean, actually, something that happens a lot, which I think I forgot the name of the language. Now, there's another language on the BM or on the beam that tries to do a type safe version on the beam. Do you remember I'm talking about? We

 

It's

 

got the name

 

L?

 

now.

 

No.

 

No, not Elm, but similar. Dang, what is the name?

 

safe. Ling.

 

Yeah, Gleam, I mean, they actually somehow try to get like the messages to be typed, which is interesting. They have some way to get either exact or pretty close to it. So like you're talking about that issue where one knows expecting a form of the data structure, then the other one was sending

 

Mm-hmm.

 

so that

 

Yeah

 

possibly could have been caught by by gleam. If I remember correctly.

 

Maybe, maybe.

 

Yeah.

 

I don't have a strong opinion for type versus type languages, I think they each have their own use case, you know? But I do miss sometimes, it doesn't need to be like enforced types, more like annotations. If at least the compiler could give me some hints that something might be off, that'd be way, way, way more helpful. having a strong type system. And I've never seen some language do that, like annotating variables so you can track changes on that.

 

Okay. Yeah, I do like your article though. I think I might keep it up and just kind of walk through it because yeah, for one of my projects, we have a lot of kind of new people to Elixir and they've been trying to debug it and they've been doing, we have a mostly API based backend. We're not using LiveView sadly yet. I have to try to sneak that in somehow. But they basically hard like test everything, manually test everything, which is just painful process doing the whole setup again. and trying it doesn't work and then they make a code change and it's kind of a painful process. So it should probably make things a lot easier.

 

There is also inside Erlang, I forgot the name of the module, where it can spin an entire VM and connect to that. So for example, if you want to test distributed systems, you can actually, with one computer, one node, you can write one of these scripts like I did for the debugging and start nodes with your code, connect to them, and test if you're talking to each other. I think it's master and slave, it's the name of the modules and with that you can... Yeah, I think that's it. Gee.

 

Okay, slave notes. It's

 

Yeah.

 

interesting choice of words these days.

 

Yeah.

 

But, okay.

 

I think this is going to be replaced on future

 

probably.

 

versions of RL.

 

OK, interesting. Okay, let me see this one. Stay. Yeah, let me read this after after maybe probably tomorrow. Just laid over here. This is interesting. I didn't, you know, this is the, this is the problem with Erlang and Elixir. It's too damn big. You can't keep

 

Yeah.

 

up.

 

Not only that, but... There are a lot of cool stuff going on there that we don't know.

 

Yeah. Okay, because I, yeah. So there's a whole module actually called slave. Interesting.

 

Yeah.

 

I didn't actually get to the whole article. Okay. goodness. There's just too much stuff. This is this is giving me I think I feel a bit anxious right now. There's

 

Yeah.

 

too much stuff to learn. Okay. All right. Well, no, not, you know, I'm a little bit upset today. I joined this call today because now you give me homework again. I always get homework every time I have a good guest on.

 

Ha ha ha ha!

 

Yeah, this is

 

Well,

 

pretty cool.

 

if I learn this, anyone can learn. I mean,

 

Come

 

it's

 

on now,

 

there.

 

don't put yourself down, you know, he's supposed to put just let me pull you up, you know.

 

But I mean, I'm not like English is my second language, of course, even though

 

Yeah,

 

I've been

 

me too.

 

used for years. And so you understand me, right? When I say

 

Yeah.

 

you can understand this thing.

 

Yeah.

 

The documentation is not that hard. to understand, especially deer lung documentation. But this is something that

 

How do you actually learn a programming language in another language? I feel so, how do you, it sounds so difficult for me to like, how do you do that? I don't know if I can do that.

 

Yeah. I think I never...

 

a couple of translations happening at the same time, right? You're translating the English into Portuguese. And then you had to translate these, these contexts into another thing. And you're like, okay, how do I relate this to my natural way of thinking?

 

In the beginning, when I started learning, Fregrami was at school. And we had this what we call Portugol, which is like a Portuguese version of algorithmism and stuff.

 

Okay.

 

So I learned programming, or at least the logics of programming, using Portuguese. And then we translate that to a programming language. At the time was C. And because C is a very simple language, related to languages, English, Portuguese and C. But with time, the concepts get really just natural as you work with this every day. You know. So I'm going to go back to the first part of the video. I'm going to go back to the first part of the video. I'm going to go back to the first part of the video. I'm going to go back to the first part of the video. I'm going to go back to the first part of the video.

 

Hmm, okay, interesting. Yeah, I don't know if I can do that. I could barely comprehend English sometimes these days.

 

Hahaha.

 

It's it is my first language, but I sometimes I feel, oh, I don't even know what's what's going on. But yeah, especially when I listen to some of the English over here, it's not about their English is good or bad. It's just like the choice of words is sometimes really difficult.

 

Yeah.

 

Like, when I get into the subway, they say, please, a light from this side. I'm like, what the heck is a light? I literally

 

Thank you.

 

like,

 

Bye.

 

I'm not kidding you. I lived in China for like three and a half years. And I moved to Hong Kong within a week of being in Hong Kong. The

 

Mm-hmm

 

English is so different than us that I actually had to download a dictionary. An English dictionary to start to like, because there's so many different words that, you know, we speak differently, speak more simple, I think, in US compared to UK. So it takes some

 

Do

 

time.

 

they watch, do they have like US media in there?

 

Yeah, well, so that's the thing though, like, of course, there's a huge influence, obviously, of UK stuff, like everybody over here calls it air con. I call it AC.

 

Mm-hmm.

 

Of course, there's a Chinese name for it. And there's lots of Europeans over here too. So they also call it air con, I think, because it's just proximity to UK, right? And yeah, so like, there's just different words in different ways to say things like I had a when I was in China, actually, my my boss was Dutch. And I'm of course American. You know, the internet thing that. router traffic. So I called that a router. He calls it a router. And both him and I were talking to this Chinese girl about the router. And so he was saying router blah, blah, blah, blah, blah, and then I said router blah, blah, blah, and we're trying to give her directions to do something with the router. And she was getting confused about what the heck we're talking about, because we use different pronunciation for the same word. So I feel sorry for other people for this different pronunciation stuff.

 

Yeah. I've worked with people from Scotland and

 

Oh wow.

 

also

 

Yeah.

 

from Southern United States with that very very southern accent.

 

Oh yeah.

 

So but because I'm like English is not my first language, it's easier for me to get the context of what are they saying because I can my brain will automatically try to connect the dots, you know.

 

Yeah. Yeah, we, I was just in UK and we were, we were in a lounge with my wife and I and there was a couple from Ireland next to us, very thick accents and my wife is not so used to it. For me, I just know I on TV. So I have a guest. I can kind of follow what's what they're talking about. And they were talking to us and I was looking at her and like, does she really understand everything that's going on? Because even I'm

 

Hahaha.

 

a little bit, some of the words I couldn't quite get. So, and I get the same thing. I have another French guy. I work with his accents very thick, but I luckily he doesn't, he doesn't get upset with me when I, and I'm like, did you say this? No, no, no, I said this. Oh, okay, okay. Gotcha. So it's good. Awesome. Yeah. Thank you for writing an article and for, for explaining some stuff. And like, again, giving me from homework now to, to work on,

 

Thank you.

 

I

 

Bye.

 

have to look into the slave notes. That could be interesting. And I think I might try to teach that. Maybe do like an internal about tracing and debugging stuff in production. That's pretty cool stuff.

 

Yeah, and it could become handy. Hopefully,

 

Yeah.

 

you'll never need this in production, but if you ever need it, it's good to know.

 

So sorry, actually, I think I asked but didn't quite get at least what I think is a sufficient answer about how you actually got into the stuff. Like I said, you didn't just crack open early documentation. Did you did you read about this like on another article and I got you into looking at these things? Or how did they kind of company radar?

 

I was looking into, because I didn't have debugging tools on my IDE, I started to look in for options on forums, on Elixir forum, on several places, and they all pointed me to using Erlang built-in stuff. And with that, it was kind of a necessity. It was like, it's something that I wanted to do. For me, I would just press a button. debugging on my IDE and call it a day. But over time I just started to learn like piece by piece, you know, first with IO inspect, then the observer, then the debugging. So it was like more of an accumulated knowledge base that I got.

 

Gotcha. Yeah, because there's a couple of Erlang books out there that talk about running Erlang in by scale or in scale written.

 

One thing that I really like is Erlang in Anger. Have you

 

Yeah,

 

ever seen that book? It's amazing.

 

I started reading but I haven't finished it. Yeah, I heard it's

 

It's

 

really great.

 

amazing. It's amazing. I think every Elixir developer should read that. It really makes you understand the runtime and how it can go crazy. It's

 

Yeah.

 

an awesome reading. Thank you.

 

Yeah, I think I got to the part where they started talking about, um, you know, how do you handle like lots of load? Like, do you start to shed it and stuff? And I was like, this is interesting. How would you know how to shed stuff and how would you even do that? And I think that they go a lot more in depth in there. I think it's still like the first third or something of the book.

 

Mm-hmm.

 

It's, it's on my desktop again. Another thing I have to read and just no time and it's not pressing. So that's the other thing too.

 

Thank you. Bye.

 

It'll get read eventually, hopefully. Maybe by the time I retire. But cool.

 

Hehehe.

 

Yeah, is there anything else that you wanted to talk about in terms of the debugging journey you went on or something else you think we should know about before we move on to pics? Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah.

 

Um...

 

Yeah.

 

Personally, I think that's it for people that want to start using this stuff. I would say to just connect to nodes, start doing this stuff and see for yourself. I think that's the best way you can learn is just try Berserk.

 

Yeah, well put. OK, cool. Then let's move on to PIX. Yeah, so I actually use a lot of Kubernetes in production. I just find it easier just to run the container. So I've been using this tool called OpenLens. I don't know if anybody's ever heard of that one before, but FreeBase. It's not easy to find how to get it for free, but how to use it for free. But it's kslens.dev. and just kind of works out of the box. If you're working with Kubernetes, it's pretty easy to use and I guess it free, which is probably the biggest one. I'm pretty happy with using that one. Do you have a pick for us? I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know

 

Anything goes?

 

Yeah, anything. No

 

Right,

 

limit.

 

so

 

Sky is the limit.

 

recently I started reading stuff with that. Recently I started reading stuff that my my wife is recommending me. I started reading all this stuff from. What's the name of the guy? Neo Gaiman. I started with Sandman and now I'm reading American Gods. Pretty good books. But else, I'm also reading one piece. I think it's the first anime thing that I've been ever reading in my life. Like it very much. And for the text stuff, I would recommend reading Erlangenanger, very cool book. And... else. What's the last thing I was reading? Yeah, just this stuff. Mio Gaiman and Erlen Nienanger. That's my pick for today.

 

Awesome. And with that, thank you for coming out. Hopefully have a good on the future.

 

Alright, thank Alan and yeah, thank you for inviting me.