In this episode, Caleb and Shawn discuss Playwright, an end-to-end testing framework for web apps. There are several established end-to-end testing frameworks including Selenium and Cypress but Playwright was new to us. Shawn found it while doing research for an app that he is currently working on so we decided to talk about what he has learned using Playwright so far. The framework supports multiple languages including TypeScript, JavaScript, Python and .NET. There is a lot of intelligence under the hood including auto-wait and web-first assertions. Listen in and see if Playwright could be the end-to-end testing framework that you use on your next project. Have you used Playwright before? Let us know your experience on Twitter at @dotnet_Podcast.

Hello and welcome to another episode of Adventures in.NET. I'm Sean Clayboy, your host, and with me today, co-host Caleb Wells. Hey, Caleb. Hey.

 

Hey, how you doing, Sean?

 

Good. Good.

 

Good.

 

Uh, yeah. Been a busy week for me

 

Hmm.

 

and, uh, I nearly, there's lots of yardwork I need to do, but it's just too hard to get outside. Hot to get outside. So triple digit temperature. So I'm just going to try to stay cool inside and just let my grass grow.

 

Yeah, hey, you know, it's your grass is happy outside and you're happy inside. So, you know, everybody's happy. There you go.

 

Happy, happy Joy Joy.

 

Yep.

 

All right, so it's just the two of us today. I thought maybe we could talk about some things that I've been working on lately. And the thing that I've been tasked with lately is doing some end-to-end front-end testing. So I went out and evaluated some of the tools that are out there, compared some of the ones that I've worked with a little bit in the past and some of the new ones that have came out and things like that. So kind of the things that are looking at were Selenium. Selenium's been around. for a long, long, long time. A lot of people know it for using front end testing and end testing and things like that. So it's been out there for quite a long time. It's got a lot of support, a lot of community out there to help you with things on it. But I don't know, it seemed like it kind of been surpassed by some of the newer tools. So I looked at those, I looked at Cypress. Cypress I've used a little bit before in the past and it was nice. It's got a nice environment. It's very friendly, but it does have some limitations and things like that out there. So, uh, some of the things that I found, it doesn't do well with multiple domains. Um, doing authentication for all the different tests can be a little bit tricky, um, from what I've learned. And, and so I kind of looked at some other ones and I stumbled upon playwright. And I'd never heard about it before, but you know, it was when I was trying to just research. the ones out there, I kept on finding people saying, check this out, check this out. So I looked at Playwright and it's actually a Microsoft product, which surprised me. I said, if it was Microsoft product, you'd think I would have heard about it with all the involvement that I have with people that do Microsoft things and some of the groups that I belong in. So, yeah, so it was surprising to me, but it definitely is a great product. And I liked it because of all the different features that it had. And it integrated well for me into VS code. So there's a plugin you can use right in VS code to do playwright stuff. And it makes it really, really easy. And one of the nice things about playwright is you can write your tests in like five different languages. So whatever language you're comfortable in, you can write your test in. So you can write it in TypeScript, which is what I ended up using, but you could also just write it in plain JavaScript. You can use Python. So they have an API where you can just use it in Python and run your tests there. Of course it runs in.NET. So if you want to write C-sharp tests in Visual Studio or VS Code, you can do that with Playwright. And it also has a... Java capabilities. So pretty much most of the major environments that you might be needing to run end-to-end tests, front-end tests, things like that, you can do that in Playwright. So that was really cool. So where I started on it is, yeah, is that in VS Code, it actually has a recorder. So you can launch this recording session in VS Code and it will bring up a browser. And then you can just click around the browser and type in whatever you would normally do if you wanted to do it as an automated test. So and it automatically records all the different selectors for the elements. So it's gonna say, okay, I'm gonna look for this CSS ID or this ID or this class, or I mean, whatever it needs to do to isolate what you're clicking on, it'll do that and record it in the test. And then it also records your events. So as you click on something, it'll record that as an event there. And so you can do that through the whole process that you would run a write a test. stop it and then go back and modify the test. Normally you're gonna have to modify it to make it more generic and support how you would really wanna have it done automated way. And one of the really nice things about it is it has the ability to save your authentication. So

 

Oh,

 

if

 

nice.

 

you're running the test and you're doing a whole bunch of different tests, it will actually. save the browser's cookies and local storage into a file for it to reuse every time it runs a test. So it's going to run this test and it's going to load in that file. So it's going to restore all the cookies and record your local storage. So it doesn't have to re-authenticate. There was a downside in that the system that I was using was actually storing authentication in the session, not in cookies are in local storage. So it doesn't grab the session when you tell it to grab authentication. There is a way to do it in Playwright. And I tried to work with it and it just ended up being a bit difficult. So if you're gonna do authentication where you're putting things in the session, it's gonna take a little bit of work to get that set up. But it's totally doable. So that was all good. That worked for me. And then you write your tests and it runs it in multiple browsers, which is a really nice thing about Playwright as well. So it supports more browsers than Cypress did. So Playwright will launch a Chromium browser. It will launch a Firefox browser. and it will even launch a WebKit browser all at the same time. So when you're running your tests, it will open up three different windows. Or if you have more tests than that, it'll open up, you know, a bunch of different windows and running them all in parallel and you can run it in a headed mode. So if you want to actually watch the test, how it works and, and, and kind of catch some ideas of what you want to change, or maybe you need to debug something that's not working right. You can actually watch it as it does it. Then of course, when you're actually going to run it, maybe in your, your CI pipeline, something like that, or your CD pipeline, it will run it headless. So it'll be more performant that way.

 

Do you, is this app going to be used in the mobile space? And if so, how does Playwright, do you know how it handles mobile web emulation?

 

Yeah, it actually has a mobile emulation in it. So I don't know

 

Nice.

 

the full list of all the different ones that it can emulate, but it can at least emulate Google Chrome for Android and mobile Safari. So it'll do those. And I think they have some more that I haven't experimented with. So that was really nice there. So my current project isn't really geared toward mobile, but it's nice to know that if I need to, on some project in the future, I can do that with mobile. So that's great.

 

Yeah,

 

And one thing

 

no,

 

that

 

that's

 

also

 

very

 

made

 

cool.

 

it easier on the tests for Playwright versus some of the other tools is I didn't have to tell it to wait for things to show up. So as you click on something and then you have, maybe it has to change the page or something has to, some sort of delay, you don't have to tell it yourself to do that. It will automatically do it. Now it has a set amount of time it will wait before it'll assume, okay, this isn't working. But that's adjustable. So if you need to configure something and you know this step is gonna take longer, you can put in saying this step, wait this long. Or if you just wanna kind of slow things down and make sure that everything has longer to wait before it auto fails, you can do that in the global configuration as well. So that's all great.

 

Yeah. Does it work with a Shadow DOM? Right, because I know a lot of the spa frameworks rely on that.

 

Um, I didn't really look at it, but, uh, but I think it will. I think it can see through into the shadow Dom and let you get into the different frames of that, um, you know, fairly easy. So, yeah. So pretty much all the different things you want to do it. It can do it. And, uh, it has a, like, especially in, in VS code when I was using it, I wanted to debug it so I could launch the test in. VS code in debug mode and set my break points and then see, okay, this didn't work the way it was supposed to. What's it seeing as, you know, was it able to find that selector after I changed it, or is it one of those timeout issues or things like that? It can do all of that. So it can launch it and you can do it there. Or one of the other things you can do is it will actually record what it sees as it run through the tests. So maybe you need to debug something that's in your CD pipeline. You can tell it to record all the different screenshots and it will do a snapshot every time it goes through that and so it will write a report to a location that you want to. And then you can go back and look at that and say, OK, this is where it failed. And at that point in time, this is what it saw. And this is what it saw for the the dom and this is what the screenshots look like. All those types of things. So very, very powerful and doing what you need to do and figuring out what's not working right when it's not working right. So really good. Yeah, all the all the browsers, you know, because it's going to launch different processes, so they're all isolated.

 

Mm.

 

One thing that I had to do in my test is we were using external authentication. So we were going out to Azure to do the authentication and it was able to follow that. So that was something that was really nice that, where we have a local domain for the main website but once it goes out to authentication, it goes out to Azure, so the domain's gonna change. Some tools don't really jive with that.

 

I first didn't

 

They don't

 

like

 

want

 

it.

 

you to switch

 

Yeah.

 

domains. Yeah, so. Yeah, this worked and it did it. So

 

Nice.

 

it didn't take too much work. You know, I worked and I did authentication. I did some basic tests within about three or four days of just learning and playing and evaluating and doing things like that. Yeah, it was awesome. I really can't think of something that I wouldn't be able to do that I would need to do in this tool.

 

surprised that neither one of us had heard about it until you mentioned it a couple of podcasts ago right because it seems awesome I actually just pulled up the website it's playwright.dev and and I was curious if you dealt with this but it says that you can actually have it generate code by recording your actions and then it will save it into the languages it supports for you

 

Yeah,

 

That's

 

yeah, that's that's yeah.

 

some next

 

When I was

 

level

 

talking

 

stuff.

 

about you, you launch VS code and then hits this little record button. Yeah, that's that's what it's doing there

 

Okay,

 

is it's

 

okay.

 

it will launch the browser and as you click

 

Yeah.

 

on things, you can actually see the code in VS code being generated. So step by step,

 

Okay.

 

you can actually watch you're clicking on something. A line of code

 

Yeah.

 

appears in your VS code as this is what this is what you would have to write in order to have done

 

That's

 

what

 

wild.

 

you just did. Yeah,

 

Yeah.

 

yeah, that was really cool. Cypress

 

Very cool.

 

had some of that capabilities, those capabilities, doing a recorder

 

Yeah.

 

and things like that, so that was good for Cypress too, but this was just so easy. And like you said, it can generate it in multiple languages. So if somebody is more of a backend C Sharp developer, they could have it generate C Sharp code for them, and it would

 

Very cool.

 

Another thing I kind of experimented with is I could do direct API calls, which was really cool. So I want to skip touching, testing the front end in the browser. I could test our endpoints, each one of those to make sure that those are working as well. So I was able to successfully do an OAuth request call to get a bearer token. Save that. And then. issue that against our API on the back end to say, am I getting a successful call? And then the data that I'm getting returned, is that what I'm expecting? And all your expectations, you could use all sorts of different expectations you can do, just like you do with any unit test type thing. So you can say, I expect this to return, you know, a JSON document with these properties and the property can, you could say it has to be this value, or you could write a regular expression and say, the value for this property must be in this kind of a regular expression valid format. So I could

 

Gotcha.

 

say that it's got to be start with these letters and then have any four digits or something like that. You could do that with the regular expression against the test there. So very, very nice.

 

Yeah, absolutely. You've got me thinking about how we could use this with the new Blazor apps we're developing. How

 

Oh yeah.

 

would that one work, right?

 

Yeah,

 

Yeah.

 

so, yeah, just, you know, if. Download the extension that they have VS code.

 

Okay.

 

It's just going to go into the extensions for VS code in, you know. Look what's available and get playwright. And, you know, it'll install and then you just do like an MPM or MPX playwright space test, and that's how you would launch your test. But the extension, it'll give you another icon on the left for extensions in your test areas. And then at the top of that, there's a toolbar where you can go in and say, start recorder. And then I did this all in a TypeScript project. So that was really simple to do there and give it the configuration file. And the configuration file has a lot of different options that you can get in there and set up. you know, your base address, if you want to have everything going to a certain URL, it can have that global config for authentication. So every test it will run this, you know, login thing first. And then you can make a lot of your tests conditional. You can say, you know, if, if this fails or that fails or, or if you, I, if I see this selector, then do this thing. Or if I see a different selector, you can do that. So you can do conditional type of tests and selection and actions in your tests. So yeah, it worked great for me.

 

Very cool. Yeah, I'm definitely

 

So

 

going to

 

how

 

have

 

do you think

 

to

 

you would use this?

 

check it out.

 

Yeah. How do you think

 

How

 

you

 

do

 

would

 

you think?

 

use? Yeah.

 

Well, I mean, really, it sounds like it runs the gamut, right, of testing UI functionality and the flow there, but then authentication, API endpoints. My only question is, if we could actually, you can install npm packages in Visual Studio, right? As part of a project, and then you can have MS build targets to run these. I'm wondering if it would work there. Because right, all of our developers are Visual Studio with the whole, you know,.NET, C Sharp, the whole shebang deal.

 

Yeah, so I would

 

It sounds

 

do it there.

 

awesome.

 

You know, yeah,

 

Yeah.

 

I would do it there. I think there's probably an extension to put it in VS Code and then have it

 

Hmm.

 

run the recorder and do the generating of the code within for C Sharp instead of

 

Hmm.

 

TypeScript. It's just that I was doing mostly

 

Catch

 

front

 

you.

 

end type stuff and I already had, it was an Angular project that I was testing, so I already had TypeScript

 

Yeah.

 

in there. So I was able to just, I made a separate project that was just a playwright test. So rather

 

Okay.

 

than actually integrating it into my Angular project, I kept this separate

 

Gotcha.

 

because we want to launch this, we want to launch it after it gets deployed.

 

Mm.

 

And so because of

 

Smart.

 

that, we don't want to, you know, we don't want to build the whole application into this little test container to run the tests and run there. So we made it a separate project so that at the end, once it's been deployed in our... our test environment or then our staging environment and even out into production, that it then launches a new container that runs these tests and generates the reports. So you can set up your environment file so we can read. I was able to successfully integrate this into an Azure pipeline. It can read the...

 

Yeah.

 

environment variables, bulls that you assign to the pipeline. So in your

 

Yeah.

 

in your gamma file, your or your pipeline within Azure, you set up your environment variables, and then playwright can read those so it knows the various different things for this pipeline, I need to use these values or this endpoint, so on and so forth. So because I'm running in test or I'm running in in stage or things like that, you so you can set those all up. And the script will read those right out of out of your pipeline.

 

is awesome.

 

So,

 

Yeah.

 

yep, it deals with frames. So if you can, you can deal with that. It doesn't have a problem with doing frames. It can do mocking and things like that. So lots of different things

 

Accessibility

 

there.

 

testing.

 

Like I said, it can do

 

Ah.

 

screenshots. So

 

Yeah.

 

if you want it to have the hell at this point in time, take a screenshot, it can do that. And then I say, generate a report out there for you to review and say, yep, everything succeeded or this test failed, that test failed. So when it's over, you go for there.

 

Yeah, I'm looking at their docs right now, and I just came across testing for accessibility. And they extend off of Axe, which I'm sure most people who have to deal with accessibility have used at one point or another. So that's really cool, right? Because there are so many things that go into that, the minutia of accessibility. Pretty broad, the deeper you get.

 

Yeah, so here's a

 

So

 

here's a

 

yeah.

 

here's a in their docs here. It says talking about it can emulate an iPhone 11. So you can say,

 

Ah, yeah.

 

you know, MPX playwright, Kojan and device equals iPhone 11. And then you can say go out to like Wikipedia or something like that. What what it has in the doc. So it can do all of

 

Yeah.

 

that. Yeah,

 

Nice.

 

I just I wish that they would give an easier way to save the session. out into the file.

 

Mm.

 

So maybe they'll add that

 

Gotcha.

 

at

 

Right.

 

some

 

All

 

point in time, but

 

right.

 

yeah. And then it, the docs are really good. It talks, it has all the different API endpoints that you can call into to use for this. It talks about electron. So you'd be writing an electron thing. It can go at an electron. That's under the experimental section. So that's something that probably they're, they're working on a bit, but lots of different things and I was trying to see if I could find a real quick a way to you know run the recorder in Visual Studio to generate C sharp. I don't see the extension for that listed here right off the bat

 

Well,

 

but I'm sure

 

if

 

it's there.

 

anyone who's listening has used Playwright and done it in Visual

 

Yeah.

 

Studio, let us know. Yeah.

 

Yep.

 

Very

 

There's

 

cool.

 

community links.

 

I appreciate you

 

Yeah. Oh, there. Yeah.

 

digging into it. Because this, right, you mentioned it. And I looked at it briefly. But then, you know, work comes back around. So it's nice to sit down and look at it a little closer. It's obviously very well done. So.

 

It can even deal with tabs. You know, I don't know how

 

Hmm.

 

it's doing that, but since

 

Okay.

 

it's launching its own little browser that it controls, it can somehow tell that to switch tabs. That's what it seems like in this, in what I'm reading about it right now. So,

 

Yeah.

 

yeah.

 

That would be interesting, right? You're in an app that you own, and you click on a button or a link that opens in a new tab pointing to something else you own, and it goes from the end-to-end testing of one tab to the other. That would be wild. Yeah. Hm.

 

Yeah, I mean, that's some of the hardest things to do and to test testing for is, you know, opening new tabs or pop

 

Right.

 

ups or dialogues and things like that. So it could do that. You know, I had to go in there and kind of futz a little bit with some of the selectors because sometimes in each one of the browsers, they might not use the same selector. So I did have to write some conditional branches saying if I'm running the chromium test. look for this selector, but if I'm running the WebKit test, the selector is different. So I have to say, you know, do this. So that was one thing that's disappointing, but that's not really the tool's fault. That's just because the browser vendors don't all, you know, have the same selectors that they're going to register for.

 

Good stuff.

 

And it was fast.

 

So

 

It was fast.

 

is there anything else you can think of that you want to mention?

 

I think that covers pretty much what I learned. There's a lot more of the details you can get in by going through the documentation on the website. But really, that's the gist of what you need an end-to-end tool for, is to

 

Great.

 

be able to test the different browsers, handle all the different types of environments that it might need to be ran in, and handle authentication, and all those things. So. Yeah, if

 

It

 

you're

 

sounds

 

looking

 

like

 

for that.

 

it really it checks so many boxes, right?

 

It does.

 

So,

 

Yep.

 

yeah.

 

And then it's it's. Out there in GitHub, so. In the code for it is out there, so if we want to contribute to it, they can contribute to it. There's an issues tab, so there's definitely some things that people are having some issues with, but. There's a bunch of outstanding PRs from people. So people are trying to fix things and, uh, yeah, contribute and just make it a better product. So, yeah, even though it's for Microsoft, it's, it worked for me. Hopefully it worked for you and work for everybody else. Yeah.

 

Yeah.

 

So somebody is a specializing in, in, and, and tests and, and they've used, you know, Cypress or Selenium or some of the other tools that are out there. Yeah, let us know how this one compares.

 

Yeah.

 

Send us a message. You know, I'm on Twitter. I am at.NET SuperHero.

 

Yeah,

 

And

 

please

 

reach

 

let us

 

out,

 

know. Absolutely.

 

reach out, reach out. Yep. All right. So that kind of covers this for today. Um, you know, kind of a short little discussion, but hopefully something people can benefit from. Um,

 

Yes.

 

we can move on to picks. I will

 

Sounds

 

make

 

good.

 

a pick today and it's something that's a little bit different. Um, I don't, this kind of popped up on one of my. Articles to read feeds somewhere about this guy that was a beatboxer and he went on The Voice but in France and so he's from France and he went on The Voice from France but he's a beatboxer but not a normal beatboxer. He actually beatbox makes sounds of instruments so not just drums and beats and things like that but he can actually simulate like an orchestra or various different instruments in the orchestra and things like that. And he uses a little one of the little looping devices and so he'll start with one instrument and then have that loop and

 

Mmm.

 

then he'll add in another instrument for the different voice, so on and so forth. And then he sings as well. So I'm going to post a YouTube video of his all the different stages that he went through on the voice. But he also has other videos and things like that out there. I mean, he did. He did like He did Starway to Heaven by Led Zeppelin.

 

Hmm.

 

That was really cool. His version of that he did in the air tonight by Phil Collins. He did a Queen, a couple Queen songs. So he's doing some good stuff out there. So if you want to check out a very interesting beatboxer, check out this video that's in the show notes. Oh, let me just say his name in case somebody doesn't get the link. He goes. by the name MB14. His name is Mohammed Belk here, something like that. So I'm sure you can just search for MB14 beatboxer and find what he does.

 

Yeah, I'll have to take a

 

What's

 

look.

 

your pick?

 

My pick today, so it was Prime Day recently, last week or whatever. And I got a notification about a 16 terabyte NAS internal hard drive that was less than $300. And I was like, ooh, all right. So I checked it out and looked at the reviews. picked it up and installed it in our home media server. And I just looked it back up on Amazon. And it's still discounted. It's not under 300. It's 325. But that's a pretty good price for 16 terabytes. And it's their. Red Pro NAS version, so it's supposed to be long shelf life. So if anybody is looking for more storage, which, of course, you can never have enough storage, then

 

Yeah, that's

 

check

 

a pretty

 

it out.

 

good deal because I about a month, month and a half ago, I bought a 12 terabyte external drive and I think it was about 250 bucks, 250, 260 bucks, things like that. So if you got 16 for 40 or 50 bucks more than me, that's definitely a good deal.

 

Right?

 

All right. So I already mentioned my Twitter handle, you know, if they want to give us feedback, want to mention stuff or even let us know that they have somebody that would like to be a guest on the show, they can reach to me. I am at dotnet superhero. Yep.

 

We get two

 

Yeah.

 

in one show. And then I'm at Caleb Wells Codes.

 

All right, guys.

 

Well, Sean. Yeah,

 

Thanks,

 

been good.

 

all of our listeners, for listening to the show. We love to hear from you, so get in touch with us. And we'll catch you all on the next episode of Adventures in.NET.

 

Bye y'all.