Speaker 1: 0:07

Hello, welcome to Microsoft Community Insights Podcast, where we share insights from community experts to stay up to date in Microsoft. I am Nicholas and I'll be your host today. In this podcast, we will dive into the world of container security, but before we get started, I want to remind you to follow us on social media so you never miss an episode, and it will help us reach more amazing people like yourself. Today we have a special guest called Josh Duffy. Could you please introduce yourself, please?

Speaker 2: 0:33

Absolutely. First, the intro. That's the longest five seconds I've ever experienced. Very exciting, I'm just kidding. Yeah, so my name is Josh Duffy. I'm a senior cloud advocate at Microsoft.

Speaker 2: 0:45

My background we were chatting a little bit about this before we went live. Let's see here I'll give this a succinct version, if I can. We can dive into more. So I wanted to be a video game designer because I played a lot of video games in high school, decided I didn't want to have $100,000 of student loan debt right starting life. So I decided to go to a community college and the only thing computer related they had was this computer networking degree, an associate's degree, and it turned out to be kind of an introduction to everything. I had a networking class, a full semester on printers, which was very painful I still dislike printers to this day. But anyway I landed a help desk job from there, got really into like system administration, worked my way up to like a one man IT shop running a small bank and racking servers, and then I kind of got into PowerShell and so a lot of people have seen my name before. It's probably because of the stuff that I used to do with PowerShell. But anyway, that led me kind of on the DevOps path.

Speaker 2: 1:35

Previous to Microsoft I was an SRE at Stack Overflow and then I got recruited in by Mike Robbins, who's familiar in the PowerShell community too, to be a docs writer or a content developer, I think is the technical term for that, and I focused on Ansible and Terraform. I had written a book on Ansible and successfully implemented it in a couple enterprises and stuff previously. I did the docs writing for about 18 months and then I got the opportunity to join the cloud native team here at Microsoft in the DevRel organization for advocacy work, which was a good split for me between some of the content creation and speaking which I enjoy doing, and I'd done that a lot in my past. I ran a PowerShell user group and other user groups before, spoken at many things, different conferences, so it seemed like a good fit. And then I landed somehow in container security for probably like the past two years or so. So that's like I don't know how long have I been working Like 12 years, I don't know. 12-year snapshot right there.

Speaker 1: 2:29

So before you land your job at Microsoft and you are in Stack Overflow. Did you have experience in Cloud Native or just be using PowerShell to create like meetups?

Speaker 2: 2:43

I wouldn't have put it under like cloud native, but I definitely had a lot of cloud experience. What's the introduction to Azure? So before Stack Overflow, the company I worked for had AWS and Azure, had to work in both clouds and we had on-prem so we had to do all that. So I'd been working in the cloud for a number of years, probably before the term cloud native came about and just I don't know, it seemed like the things that I was naturally doing in the cloud and then having them change, I suppose, from on-prem and stuff, and then it just became a term. So it's one of those situations where you've just been doing a thing for a while and then all of a sudden, uh, the broader ecosystem comes up with the term for what it is that you've been doing for a while.

Speaker 1: 3:23

Yeah, because I remember watching some of your episode on Bash. You know Bash as well as PowerShell. It's quite handy.

Speaker 2: 3:32

Yeah, I got forced into that once. I Very interesting paradigm shift. So outside Microsoft was Windows System Administrator, powershell guru doing DSC like even wrote my own. Actually, dscv3 just came out and they have the ability to do JSON representation. Back in 2016, I did a talk where I wrote a module that abstracted DSC into JSON, so that's pretty cool to see that. I don't think they used any of my work or anything like that, but just that idea transferred. Yeah.

Speaker 1: 4:00

Okay, so today's session is the world of container security, so let's talk about the main, like the introduction. So what's container security and why is it important?

Speaker 2: 4:12

The area that I've been working in container security is pretty broad, so I'll narrow it down to the areas that I've been working in. I've been working with a couple open source projects Notation, which is a digital signer you can think of it that way yeah, and then they're ratify, which is an emission controller for Kubernetes, and then copacetic, so that would be for the patching, and all those are related to the software supply chain and containers. So container security most people would think immediately think chain guard that's really focused on the images and make sure that those images in the best possible way. That's a big part of it. A lot of the work that I've been doing is on somewhat on the other side.

Speaker 2: 4:48

Right, okay, we have this good image that we trust or want to trust. How do we signal that trust? And that's where the signing comes into play. And then how do we know what's in here and that would be something like an S-bomb and all of these are. It's been an interesting thing to watch in the industry because it's more work and definitely not something that the average developer wants is more work. But there is a lot of pressure from especially like government agencies and working contracts with government agencies where they're now mandating these types of things to improve the supply chain, to avoid supply chain attacks from happening. It's an evolving ecosystem.

Speaker 1: 5:24

One of the important points is keeping your images up to date and don't have updated images, because it would lead to having vulnerabilities in your code.

Speaker 2: 5:32

So yeah, I never writing. So I wrote a series on Copasetic, which is a patching tool, and in writing that it just brought me back to my first kind of epiphany as I was learning Kubernetes and getting into containers, when Docker was really starting to come on the scene. And one of my first thoughts I remember because I had battled Windows updates like my entire career VM updates and I was like, oh, finally I'm not going to have to worry about this problem again. Here we are, 10 years later. I'm having to worry about, you know, vulnerabilities of my images that I have to patch or you can patch by updating the base image. We can get into more detail with that later, but at the end of the day there's still vulnerabilities in these container images and systems. It didn't solve the patching problems per se Exactly. You know like we still have to have systems around updating things to make them not vulnerable.

Speaker 1: 6:22

Okay, so the free tool, Copasetica, and the other one take it there. Cncf approved yeah sandbox. Can you use it via Helm or is it like automated Ratify? You can.

Speaker 2: 6:39

So that is the only of those three projects that would sit on your cluster. The other two, notation and Copasetic, are both CLI tools that have GitHub Actions, so we have to automate it.

Speaker 1: 6:50

So what are some of the use cases of like impact, use cases that you think both of the tools do have? Do you just have to use all of them side by side?

Speaker 2: 7:02

You can use all of them. So here let me. Actually let me just show you a repository. Would that work? Yeah, here let me, let me clean up my very messy desktop.

Speaker 1: 7:13

Oh yeah, before I mention, I remember the tool that we use A net scaler, no, a new netter.

Speaker 2: 7:19

Oh, I've not heard of that one.

Speaker 1: 7:20

It's like a container scaler thing, it's like a container-skeletal thing.

Speaker 2: 7:23

It's like an images thing. Does it have a scanner?

Speaker 1: 7:29

Yeah, but you have to pay for a license. It's not open source, I think. Ah, okay, it may have a container-skeletal license.

Speaker 2: 7:35

Share screen. Let's see. Oh no, Is it going to make me have to drop? I've used this before. Let's see here Window.

Speaker 1: 7:43

I want this one. I think it's sent Yep.

Speaker 2: 7:46

All right, fantastic. So this is a little outdated. I will get to. If my build talk gets submitted, I will definitely take another pass at this, but here this is kind of. This will give you the full picture here. So if we look at this, this little graph here will give you an idea. So there's another tool I haven't actively contributed to, but I use all the time, which is Trivy. So Trivy is a vulnerability scanner. It's probably you, might it might be using your tool, might be using it.

Speaker 1: 8:12

I think you can scan like infrastructure tool, like Terraform code as well using Trivy oh yeah, yeah, yeah, misconfiguration and secrets and stuff.

Speaker 2: 8:20

Sure, absolutely yeah. So Tribu is just a vulnerability scanner and misconfiguration scanner, for lack of a better word. I'm sure their bio will give a more accurate description. But essentially, what that allows you to do is to generate a report of, like what's vulnerable in this image. So what you can do is you can hook these all up with GitHub Actions, which I'll show you in a second, and then the next step would be wouldn't it be super cool if we could remediate those vulnerabilities based on the report?

Speaker 2: 8:44

And that's where Copasetic comes in. So Copasetic will take care of OS level vulnerability patching for container images and it can ingest trivia results to do targeted patching. It also has the ability to just patch everything to the latest on your system If you want to. You're feeling risky and you want to, you want to go that route, you can, but the trivia one will just I found a Cve for libc, I'm going to update libc or whatever that might be, and the report takes care of that, uh, and then? So now we have this image that we feel pretty good about. How do we signal trust?

Speaker 2: 9:14

And that's where the notary project or the notation clr comes in, and you can more or less stamp. You put a digital signature on that container image, a notation, so another another option would be Cosign from Sigstore, so you could use that as well and that creates a digital signature on the image and that becomes important later on. But there's a distinction between. Well, I think they can do both. If you use the beta features for Cosign, they both can use the referrers API or it'll create, like another artifact, a signature artifact on your registry, and I think I even might have some packages that we could see what that looks like on here. We'll see if I do no, but maybe yosh does so.

Speaker 1: 9:50

Have you used this on production environments for, or is it just like poc?

Speaker 2: 9:55

well, I know it's being used in production inside microsoft, but the nature of my role is I they don't let me in production very often, right? So they actually just, uh, you kick me out of, like production tenants and stuff, and that's just the nature of my role in the advocacy. You know I'm not responsible for any internal production systems. I don't want the Wasm cloud, it's all it's my cloud.

Speaker 1: 10:15

Yeah, so to update the image, you can just have different base files for each of the image user.

Speaker 2: 10:27

And then you can update it right via helm or cli. Yeah, uh, so there's different patching strategies. If you want to one second and we'll go in a little segue about like why would you even want to patch an image and not just update the base image? Because that's a valid, that's a valid question. I wanted to show signature real quick. Is this the right one? Now I want packages and not there, so so this is a signature here. This SHA-SIG, I guess. Yep, and so that's what Cosign does by default is it'll put the signature on the repository Notation by default, will make it through the refer, so it won't show up here. It'll actually be in, like the manifest metadata, if you were to like pull up latest and in here we would have some information to the digest for the signature and it can do that with. It works with, I think, most registries, any OCI compliant registry it works with.

Speaker 1: 11:10

So you have to modify the image. If you want any custom image, make it own.

Speaker 2: 11:15

It'll do it for you. Yeah, yep, so the copacetic. You could overwrite the tag if you wanted, or you could append, like attach, like a hyphen one to do a counter, but we can go into that in a second. So we'll pause right there, though, and just you know, like why would you even consider patching container images when, like, this is the build pipeline? I could just bump up, uh, the base image version. When you can do that, then you know, I would say patching, don't do patching.

Speaker 2: 11:39

I think what was it? Chain guard just had like something on their x profile of a big billboard that they put up, that they had like patching something, and they crossed it out and they said secure future. I don't disagree. Like, if you have the ability to use a better base image that doesn't have CVEs, do that. The problem comes when those vulnerabilities are upstream and you have a dependency on some other image and their cycle is longer than yours and they can't resolve that cve. You're basically locked right, like I can't upgrade a version because it's going to break something or because there just is another latest version yeah, not a lot of the like couple seconds.

Speaker 1: 12:15

Like the upstream repo probably won't tell you the vulnerability, like if you used to look at a changelog, it probably won't say just show the features. So you have to scan it to see if that is very good.

Speaker 2: 12:26

Within the hash right image itself exactly, exactly, yeah, and so it's most useful when you've got an upstream dependency that you just can't update the base image for. How do you get rid of those vulnerabilities, and copasetic is a good way to automate that. Technically, it's super cool using build kit to to break apart your container and patch it and create a new page. It's pretty cool.

Speaker 1: 12:46

Have you used this for Azure DevOps before, or is it just CLI?

Speaker 2: 12:50

Oh yeah, it works in Azure DevOps, I believe. What's the GitHub action equivalent for Azure DevOps? Is it just a task?

Speaker 1: 12:57

Yeah, it's a task.

Speaker 2: 12:58

Yeah, so there's a copacetic task for it.

Speaker 1: 13:01

Okay, and is it for the nodejs? Now, the other one is trivia. So I know that is a trivia and then. Okay, so a lot of them have to test. So is it possible, just like, like, if you create one, one base, one file, to update image instead of updating lots of image for each of the?

Speaker 2: 13:19

the tool to have like a golden image. Is that what you're referring to? Yeah, uh, yeah. So let's see here. Let's go to copasetics documentation, one of like. You have a dot net image of.

Speaker 1: 13:25

Is that what you're referring to? Yeah, yeah. So let's see here. Let's go to Copasetic's documentation, one of like you have a NET image of NET and you can update that, then it will pass the latest without and then you can pass it with whatever image you want, whether it's a patch image or an actual NET, that is image I think what you.

Speaker 2: 13:39

So you could do this, uh, dynamic tagging where you basically say, like here's, here's the base tag that I'm patching, but then I'm always going to override this version tag. So the patching you would want to do per version. So there would be another version of this. Like there would be another tag for a point 1.25, you would have another one that is always patched. That's, that's one tagging strategy that you could you could do. That'd be the dynamic one. The other one is the, the incremental one. So you can just see there in the docs, you know, like the first time it goes through and patches it's going to be hyphen one. The second time will be hyphen two. The problem with that is you, you have to go up and up, you have to go back and update your manifests and I'll have this ready. This would be cool to add in. But I'm working on setting up a sample repo that uses Dependabot to be able to come back and say, hey, I detected a new patch version of your image, just like it would any other dependency update right, and do a PR into the repo to say, hey, let's update the image to a new patched tag.

Speaker 2: 14:34

The issue with the other supply chain tools down the line is if you're not using the tag and you're using the digest. So, for context and people that don't know, each tag has its own digest. If you look back at this, the SHA right here is the digest, so you can actually reference container images in your manifest by digest. And that's the reason that's good is because it's not mutable, meaning if the image has changed in any way, it gets a new digest and it won't pick it up, which is inconvenient if you want that. But it also improves the security, because now you know for certain which exact tag and version of the image you are going to have deployed. But if copacetic patches changes the tag and then that digest is actually used by the signing tools. So the other tool that I didn't get Okay.

Speaker 1: 15:19

so whenever you use the signing tool, do you just get a new digest you do yeah, okay, yeah, I'm curious to know. Sure, is there any like open source tool that's equivalent to this, that's like a web competitor, or is this, is the one that's well known in microsoft or in the cnc? Uh?

Speaker 2: 15:39

which tool.

Speaker 1: 15:40

Which tool? Whether it's going to be copacetic or notary.

Speaker 2: 15:46

Notary. So the notation CLI. Its counterpart would be cosine. So the cosine CLI from Stixor. They just, they, just. They just create the signatures in a different way. The biggest.

Speaker 2: 15:56

Here's what I would say If you have a mature paykey, public key infrastructure inside your company like you're a large enterprise you have your own keys. Notation is what you want to use. But if you don't and you just want to have a signature, cosign actually has something called keyless signing where it will basically get rid of the private key so it can't be used again, but then the public keys are made available. I believe it was a cold record. So that way you can verify the signature but you don't have to maintain your own PKI infrastructure. So that would be the big delineation, at least in my mind. If you've got a mature PKI system and you need to use your own certificates, that's where notation shines. And then cosign is on the other side, where maybe you're doing more local development or you just don't have the infrastructure, don't care to have the infrastructure and you want signing, and I'm sure they have offerings to kind of offload the PKI infrastructure and they have, I'm sure they have offerings to kind of offload the PayCat infrastructure.

Speaker 1: 16:45

I'm just not super familiar with their product suite, so I can see this improving by having a little dashboard, seeing any outdated images, like when people like internal what developer or DevOps can monitor, and then we can just like get, we can just update the image accordingly using automation, whether it's GitHub or Azure DevOps Yep, there's something that can be built in like monitoring on it if we just update the image accordingly using automation. Whether it's GitHub or Azure DevOps Yep, there's something that can be built in like monitoring on it.

Speaker 2: 17:06

Kubescape might be doing something like that. This is a tool that popped up on my radar recently. I haven't dove into it yet, but it does use. It integrates with Copasetic.

Speaker 1: 17:14

Okay. So this might actually be kind of what you're envisioning you probably can interface into like a GUI, right? Yeah, when people like a front-end GUI. Okay that's quite nice. It's easier for like developers or anyone to have login and concede vulnerability and patch it themselves by running the automation tool.

Speaker 2: 17:33

Yep, that would be. It's hard for me sometimes to put on my S3 platform engineering hat and want to build those types of things, but those are lots of time. This is more putting together open source tools, something. Yeah, a product could definitely sit on top and use all of these things together and make it a lot more seamless, and I think that's honestly what needs to happen to make it more palatable for companies and organizations. This is a lot of additional work, but this is what it would be doing under the hood. There are a number of features inside Azure. One would be the continuous patching for ACR, so it uses ACR tasks to do the copacetic stuff for you and then it takes it out of your workflow and makes it more of a platform checkbox type task.

Speaker 1: 18:09

Yeah, because normally we're trying to make everything self-service so people can just run a pipeline and then it will update the image. So we're just putting the YAML and submit to PR.

Speaker 2: 18:19

So whether it's looking at the dashboard to see vulnerability or not, yeah, this would all take care of it inside of a pipeline, but this would require all the dev teams to kind of adopt a similar workflow or a templated workflow. Right, yeah, so the other end of it.

Speaker 1: 18:32

Oh, go ahead. What's the result you get at the end? So do you get, like a result of something that's outdated, images with SAP vulnerability and then someone can actually check whether it's an actual vulnerability or not?

Speaker 2: 18:47

I don't know of anything that there's. I think there's some Azure features that would be able to do continuous scanning on your registry and then present you with diagnostic information there. I haven't looked into it too much. That's what I'd imagine it would be going Right now. It's point and click in the workflow. Trivia runs inside that report. You would see your vulnerability list. It's not displayed on any kind of dashboard. It's displayed here. There are flags with trivia that you can use to stop your workflow and fail it if a certain number of CVEs exist, even after you patch or during the scan.

Speaker 1: 19:18

But yeah, there's no dashboard, at least in my current map markup there we go and you say you can, this can be integrated with github dependent bot or something that does the pr scanning for vulnerabilities and stuff, so you don't really have to scan it yourself. You can get pr. That's been raised and just so.

Speaker 2: 19:36

It's just reviewing really yeah, that's the kind of the end to end for if you're just wanting to focus on GitHub workflows, and Dependabot obviously works for Azure DevOps too.

Speaker 1: 19:48

Because I think I use something similar called Renovate. It does image for versions, but I think it probably can integrate with this. It's similar to Dependabot in a way Okay cool, great with this. It's similar to the pentabot in a way okay, cool. So have you got any? Do you know any examples where this is used in companies, or it's just microsoft at the moment? You know where?

Speaker 2: 20:12

alibaba has been pretty good, uh, with adoption on ratify. They've got some solutions. So, yeah, let me go to the deploy phase. Um, a little bit and just about that, and then we can hang on to more customer adoption. So, if we look at the bottom part, so it assumed like we've got a signed image on our registry and now we want to deploy to our cluster. How do we put some teeth to all this work that we did during the build step right To produce this artifact that we trust?

Speaker 2: 20:38

So ratifies and gatekeeper combined, so you deploy them together with a helm chart, and what that allows you to do is do admission control for digital signatures or other verification policies, so you can define some kind of rego policy. And in this example it's looking for does a digital signature exist for this public certificate, right? And so if I can't validate the signature with my public key, that signature either doesn't exist or is invalid, and I'm not going to let the pod run on this cluster. And so there's a lot of good use cases there, trying to think what are the other customers? By name or if I should say names, but yeah, there are definitely customers using it. A lot of these are actually being baked into features right Like features inside ACR features inside AKS or whatever it might be.

Speaker 2: 21:20

For example, like with Ratify and Azure, you can define an Azure policy that converts to a Ratify configuration, so that way you don't have to apply another kubectl thing. It can just be any AKS cluster that gets deployed to this resource group has to have signed images, and then that translates down to Ratify.

Speaker 1: 21:37

Yeah, I guess the good thing about it is that it's a security aspect as well, because it constantly gets updated as well with the images, and it's being maintained by the CNCF, so that's a good side about it.

Speaker 2: 21:52

Yeah, all those projects are. I don't know about Trivi, but Copasetic, notation and Ratifier are all CNCF projects.

Speaker 1: 21:57

Okay, brilliant. Do you have a picture of what it will look like as a result when you run it?

Speaker 2: 22:04

Yeah, I think I have some successful runs, or maybe I don't. I deleted them, I don't know. We'll find out.

Speaker 1: 22:09

No worries, so it'll just give you a trivia. I know trivia just gives you SVE, whether it's infrastructure code or any code in general of the list, but for the rest it just gives you the hash. So it just trivies that it's bring up right Most of the results.

Speaker 2: 22:26

Well, here let's just look at the workflow, we'll see. So this is a little update. I'll show you a more recent one for people that are more on the platform engineering side, because a lot of this is being done on the developer side, which might be hard, and maybe you just care about patching the images in your registry and I'll talk about a continuous patching workflow here in a second for at least GHCR. So yeah, if we just let me bump this up, the general workflow is we're going to log into Azure, log into ACR, we're going to build our app, right, and then we scan it. So we scan with the trivia action that's going to output this patchjson and then we're going to patch it. So this can be replaced. There's a GitHub action for this. Now this can be replaced. I'll show you that in the next one.

Speaker 2: 23:08

We're going to patch the image. So I'm just outputting what the digest is going to be and then push the patch image. So I'm going to push the patched image after Copasetic patches it. So Copasetic is going to patch it. It's going to create a new image locally with the taking prefix like hyphen one or whatever taking strategy you're going to use, and then I'm capturing that digest at the most optimal time, which is right after it's built. I'm going to capture it. So there's no chance that I'm I'm grabbing the wrong digest and someone put another layer on this image with something malicious in there and I didn't know.

Speaker 2: 23:41

So I'm going to grab that digest right there and that's where notation comes in. Notation has a, an action as well, so I'm just setting it up and then I'm signing it and then that puts a signature on the refer API that can be validated with ratified later. And then so if you looked at, I think I have terraform code in here. This sets up the aks cluster and then I'm using it in the demo I actually set up ratify. So it's not, it's not part of my infrastructure as code, because it's part of the demo script, but all this you can walk through as a demo. This demo sh goes through every single step in the workshop, so that's from like a pipeline perspective. But let's say that you wanted to like what would it look like to continuously patch a registry? And so I started this repo and I walked through this in a blog post series for people that are interested.

Speaker 1: 24:22

Okay, it looks like. Oh, go ahead. This would go through like ACR, like for your, and it's patched all the images to see if they're using the correct images or not, if they have any vulnerability.

Speaker 2: 24:34

Yeah, to see if they're using the correct image or not, if they have any vulnerability, yeah. So there's a continuous patching. Private preview for the AZ-CLI that basically does just this. I just did it for GHCR and I wrote a little. It gave me an excuse to write some code which I can show you in a second.

Speaker 1: 24:48

Does this look at images that's, for example, older than three years or three months, or something?

Speaker 2: 24:53

It'll look at everything on your GHCR repo, everything. So essentially what it'll do. If we look at this little flow chart, it sets up, it gets your images from GHCR, it outputs a matrix and then it loops through the matrix and patches. So let's just look at the workflow real quick. So here's the setup right here. Make it a little bigger.

Speaker 2: 25:14

I wrote a little go CLI. I turned it into a CLI instead of just a go file, which we can see in another branch. There's actually quite a bit of logic in there to detect the tags. So the advanced logic that I put in there would basically calculating the tagging strategy. So I'm only going to grab the latest. If, like, net monitor has a hyphen one, I'm only going to patch hyphen one and not zero, because zero came before hyphen one, and then if I patch both, they'd be at equal patch level. You should, you know. Then why not just flatten it down to one tag? So there's a little bit of strategy there. And then I'm calculating the incremental on that and that looks like let me open a new tab.

Speaker 2: 25:47

The output for that CLI improved looks like this. I called it contiguous. I thought it was clever. Maybe it does. I don't have a screenshot maybe, but essentially you would do this list and then it would output a JSON of here's your current tag and here's your next tag. And then GitHub Actions is pretty cool because you can just use from JSON to generate your matrix, yeah, and then basically go through.

Speaker 2: 26:09

So for each image in the matrix that I identify that needs a patch, I'm going to go through and generate a trivia report. I'm going to check the vulnerability count. So if the vulnerability counts zero, I'm not going to bother patching, because Copasetic is just going to fail and say I couldn't patch anything. This patch tag is taken care of and then the new version is CLI. But this is just some bash Again. This is why I've had to learn more. Bash is because of GitHub Actions, to be honest. This calculates the next patch tag based on a tag, then it goes through and runs Copasetic logs into GHCR and then pushes the patch tag. So the next part of this workflow would be like okay, I've got all these new patched images on my workflow. How do I automate that final step, which would be the PRs, into the code repos that are using those images?

Speaker 1: 26:54

And that's where Dependable will come in. So that's next on my list to tackle. Okay, yeah, it's quite good that you can automate this and probably can make it a self-service for anyone or developers to run.

Speaker 2: 27:03

Just add the images, yep, and so here's an example, here's the setup, and I only had one image, but two tags that needed to be patched, right, so let me shrink it a little bit so you can see, but then, yeah, it just goes through. This one didn't even need to be patched, so it skipped it.

Speaker 1: 27:23

So this is probably good. You can probably build in this with some alerts as well. When someone's image has been outdated for like one month, you can alert you so someone can patch it.

Speaker 2: 27:32

Yeah, you could this continuous patching thing. I have it on an on-demand trigger, but you could a cron job, like whatever you wanted to do, just kind of show you like what's possible with it. Like I said, cubescapes doing something with it, integrating with copacetic to make this more full-fledged. This was just to give an idea of, like what you could build with it and a lot of people are doing with that. I I think, yeah, in ACR they're working on the continuous patching thing, solving a lot of these problems as you dig into it.

Speaker 2: 27:57

There's issues, like I mentioned, with the signing, like how do we make sure that the signatures are still valid? How do we update? It's funny just last year I contributed the logic for Ratify to support inversions of keys. So basically it'll look at your, your keyball instance and if you have more versions that you've created for a key, it'll update, auto update the cache, which solves the next problem that the continuous patching does like. Okay, what if I signed it with a new key? Do I have to go then update my verification engine? The answer is no, it'll. It'll dynamically pull um. So that was kind of cool to see that that work get used, okay so, yeah, it looks like it's quite a good tool to quite batches.

Speaker 1: 28:36

Is there any? So, to get this set up and running, is there any challenges that you have to do? I take it one of the challenges you have to know more batches right or just use your task well.

Speaker 2: 28:47

So the way that I've set it up, if you I'm tomorrow's on my to-do list, tomorrow is to take the. So if you want to use ghcr, that's what I scope. Mine to you tomorrow is to take the. So if you want to use GHCR that's what I scoped mine to you could actually just take my actions and plug them in and it would work. The only thing that you'd have to figure out if you're not using GHCR is figuring out what tags you want in your matrix, and you could manually define that. So if we go back and look at this one, I had an example.

Speaker 2: 29:10

I had an example of using just a static matrix. So if you knew the tag names that you wanted, you could just define your matrix like this, right? But I wanted to be a little bit more pragmatic, and so then that required me to go in write my own logic against GHCR. So that would be. The biggest gap right now is you need some mechanism to feed into the copacetic workflow what images you care about. So if you want, you can use my Go code. That's fine.

Speaker 1: 29:35

Yeah, it's open source anyway, so it's good. So do you reckon that the tool that you show us today is the future of container security, like containerized?

Speaker 2: 29:44

I would imagine it's kind of the foundation. So I think they're like what you were talking about earlier. These will be wrapped into other tools that probably cost a little bit of money. That just make it easier, right. That's kind of the trade-off with open source. But if you want to build your own type of solution, all the tools are there to improve the supply chain and again in the CNCF, so like they're stable and you know that they've got good investment in them, they're being used internally for Microsoft. They're being used internally at other big competitors too. One interesting thing AWS and Microsoft are both working on Notation, so they're collaborating. We're collaborating on that one. But yeah, so the tools are there to put it in there if you want to go the open source route. But I'd imagine there's going to be some cool tools that kind of wrap all this and make cool button Right.

Speaker 1: 30:29

Make everything a button. The only good side about it is that this is free. It's open source. Instead of other tools that you have to pay it, you know to use this stuff enterprise tool. That's more of a better suited. So that's about okay. So, as this episode is coming to end, we always like to get to know our guests. What do you normally do in? Your spare time, josh, like your job as an advocate on your job and then learning about security in containers.

Speaker 2: 31:02

Yeah, well, the last few years. So I just my wife and I welcomed our third child a year ago. He just turned one last month. So the kids keep me pretty busy, so my spare time isn't as fruitful as it used to be, but I like to read, so I do a fair amount of reading. Lately it's been I've actually been revisiting some old books PowerShell books, no, actually more like productivity-type books Deep Work by Cal Newport, I think it was released in 2016, 2018.

Speaker 1: 31:30

And then UltraLearn. I just got it right here on my desk I can show you. Okay, like you read 50 a book, but not kindle.

Speaker 2: 31:33

I have a kindle um, I use the kindle to evaluate whether a book's useful or not. So like I'll download the sample, I'll burn through the sample on my kindle, uh. But I do prefer a physical book, um, even technical books, which are more annoying to do because you have to. But anyway, yeah, I like the physical book. So I I don't know, I'm, I'm all kind of over the place. I read a little bit of Rust, worked with a colleague on some Rust recently.

Speaker 1: 31:59

That's not the language right Rust.

Speaker 2: 32:01

No, yeah, the language Rust, yeah.

Speaker 1: 32:03

Oh, okay, yeah, it's not very popular. So I don't think if you use Copilot to tell me about Rust, I don't think Copilot would know what's Rust it but Rust. I don't think a lot of people would know what's Rust. It's never helpful. Yeah, yeah, Rust is quite hard. It's like Ruby. It's quite an outdated language as well.

Speaker 2: 32:20

So it's probably more people know Ruby than Rust. To be honest, I think I would have had an easier time picking up Rust, moving from PowerShell as my primary language than Go was. Just because of all, it is still static, but it has a lot of niceties to language Once you understand. They're very cryptic. At first they just look like hieroglyphics, but they're actually very useful shorthand macros that Go doesn't have. That makes it made the transition a little hard for me, but I do really enjoy Go still. It's made me learn things at a much deeper level, that's for sure.

Speaker 1: 32:49

So I've remembered speaking to you before this goes live and you said you're going to the bill in May.

Speaker 2: 32:58

Potentially. I have a CFP submitted, so I'll have word back. I don't know, maybe in the next month or so, so I might be there.

Speaker 1: 33:06

Yeah, so it's going to be the same session as this talk.

Speaker 2: 33:08

Yeah, yeah, it's this one right here, yep.

Speaker 1: 33:10

Okay, nice, so yeah, so if anyone wants to interesting to meet josh, he might be in bill so you can catch him there. Absolutely okay, uh, thanks for joining josh. So yeah, as normal. Stay tuned and see which is going to be on youtube and spotify.

Speaker 2: 33:27

Thank you, bye thanks for having me.