Ep 746: Review of Anthropic Claude’s Viral New Computer use: Fun Party Trick or Real Agentic Workforce?

This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life. In their new computer use tool, Anthropic just released one of the most viral AI launches ever and what could be a step change not just for the AI industry, but for the future of knowledge work. And if I'm being honest, one of the biggest pieces of long term job security for us human knowledge workers right now is the fact that we're the ones opening the programs on the computer, clicking around, saving and uploading, you know, the stuff that I call human duct tape on this show. As for the high value important tasks, many of us are already relying on AI to do the heavy lifting and that kind of relegates us to just the human duct tape that opens, closes, copies, paste, uploads and downloads on a computer. But now Anthropic's new computer use tool that they just rolled out can kind of do those very things like an actual human behind a computer. Computer use in theory can open programs, browse the web, access your local files and use a mouse in keyboard to navigate around just like we do. So as we put AI to work on Wednesdays in our weekly segment here on everyday AI, we're going to tackle the new computer use feature from Anthropic and answer the question if it's a just a fun party trick, or if it's a real, a genetic workforce. All right. The big picture. Claude's new computer use tool went absolutely mega viral on Twitter. I'm talking about 75 million views, which is a lot. Yeah, if you're not on the Twitter machine, what people call acts I call Twitter. Yeah, this is one of the most viral AI product launches ever, at least by attention and eyeballs on the product. And Anthropic showcase the ability to literally control your Mac desktop via your phone. Yeah, you can do this all with your phone right now with a combination of the dis bass dispatch feature, which they released earlier in March, and the brand new computer use functionality. So computer use can be used on your mobile phone via the dispatch app, or the dispatch feature in the Claude app, or it can be used on the desktop Mac app via co work or Claude for paid pro and max users. But it does obviously require full use of your computer. So we'll talk about that later. Some of the downside. So this might not be kind of an AI agent that sits down with you and works alongside with you like you might want it to be. So stick with me for the next 20 ish minutes and on today's show. You're going to learn why the most viral AI feature right now is maybe much less useful than you might want it to be. You're going to learn what Claude's computer can actually do versus what it looks like it can do. And you're going to know whether this is a genuine agentic workforce tool or just a research preview gimmick. All right, let's get into it. If you're new here, welcome to every day AI. My name is Jordan Wilson and well, this is an unscripted unedited live stream podcast and free daily newsletter helping everyday business leaders like you and me, not just keep up with what's happening in the world of AI, but how we can make sense of it to grow our companies and our careers. So welcome on the journey. If that's what you're trying to do, it starts here, but the cheat code is at our website, your everyday AI dot com. Go sign up there. We're going to recap the most important things from today's episode as well as give you all the other AI news you need to know to be the smartest person in AI at your company. So let's get in and talk a little bit now about the new computer use feature. So here's well, what it can actually do on your computer, it can open apps, it can move your mouse, it types, it can fill out forms, it can navigate your entire desktop. If you give it access to right, we're going to tackle some of the security considerations here in a little bit. But right now this works both inside Claude code and Claude co-work on Mac OS. So this is a desktop app that you download and in doing so after you, you know, go through all the permissions, it can literally control your entire computer, not just the Mac app, not just the Claude app, right? It is available right now for paid plans only. So you do have to be on a paid plan. And if let me just cut to the chase, this is only truly useful. I think for apps that don't have a dedicated CLI connection, right, command line interface, if they don't have a useful API, if it doesn't have like an MCP, or if it doesn't have a direct Claude integration, right, which that sounds like a lot of ifs, but it's actually not because I would venture to say that the overwhelming majority of kind of apps or where we work, because the reality is as knowledge workers, all most of us really do is we're inside apps, right? You're inside Salesforce or Slack or Outlook or, you know, looking on different websites, right? You're in Excel, you're in Microsoft Word, all for the most part we do, we spend all of our day in apps, right? And many of those apps do have a direct integration with Claude. And if they do, using this computer use is overkill, right? At that point, it is just a party trick, because it is slow, right? It's slow, but effective. But if you're just thinking, oh, well, you know, I can just kind of put together everything that I use and just do it all via the computer use, that would be an extremely token inefficient way to tackle this. Because yes, it is number one, slow. And number two, it does eat up tokens like I eat pancakes at the original pancake house, like it goes through them so, so quickly. So if you do want to set this up, here's how you do it. All right. And podcast audience, you know, sometimes our new Wednesday series that we do, putting AI to work on Wednesdays, sometimes they're a little more visual, right? But if you are following along just on the podcast, I want to do my best to describe what's going on on the screen. I am going to do some live demos here in a little bit. Right now, if you want to know, well, how this works, how do you get it to set up? So number one, you have to download the Claude app. If you don't have it already, it's only available right now for Mac. So not Windows yet. And you do have to be on a paid subscription. All right. So if you check those boxes, then what you're going to do inside your settings, but in it's the settings for the desktop app, which is different than the settings. If you go to Claude.ai. So you're going to want to open your desktop app, look in the settings on the left hand side, it's kind of hidden down there. So you're going to want to look, there's specific desktop app settings. You're going to click on general. And then there's going to be a new computer use toggle that you want, that you're going to want to click on. There's other things you can do from there. You can go to denied apps, right? So if there's certain apps that you absolutely don't want Claude to access, you can add those apps there. And as you go along and we'll see, I'm going to do some of these things live. We'll see how that goes. Because yeah, it's slow and I've done a couple, you know, practice demos on this, but it could go off the rails. That's the other thing, right? You do need to be careful. And so this is not something I recommend that you kind of quietly or sneakily install on a company computer. I wouldn't do that, right? If you are a decision maker at a company, you do have to say, is this something that we want to do? I would say it's at least worth sandboxing. But you do have to keep all of these security implications in mind, because when you do give a large language model, complete control over your entire computer, obviously, a lot of things can go very, very wrong. Right. I do these things as a small business owner myself. I understand the risks. Right. I kind of keep tight guardrails around private or sensitive information when I do these types of things. But keep that in mind. Add denied apps. And then once you do that, you do also have to go into your Mac settings and grant access for accessibility settings in screen recording. So that essentially allows Claude to perform different actions on your computer and to see things, right? That happened on your computer. Then dispatch, I think, is where this actually becomes useful. And we'll get into that a little bit more as we do the demos. But if you don't know dispatch, dispatch was released a couple of weeks ago. So Entropic has been on a complete tear and they've actually released so many things. It's kind of confusing to keep up. All right. So dispatch was released about two weeks ago. And this is just a way to communicate with Claude co-work from the mobile app, because there's not a dedicated Claude co-work app on the phone, but there is a dispatch kind of feature that allows you to talk to Claude co-work and talk to Claude code. So you're not saying, okay, I'm confused. Why would I want to do that? Well, as an example, right now on your mobile phone, there's no way to access something on your desktop. So you go through with dispatch, you go through a one time pairing process. But then essentially, right, I think of some instances where this would have been very helpful. Right. And it's funny, the, the kind of little demo or use case that Entropic led with was, you know, Hey, you know, using dispatch, I'm running late. There's this file on my computer. Can you send it over to me? Right. That's happened so many times. Luckily, my wife bailed me out. I was doing a big keynote in training in Chicago last year and I'm like, Oh my gosh, I, you know, I don't have this file on my laptop. It's saved on my desktop at home. It's not in the cloud anywhere. Right. So my wife, who is awesome, bailed me out via. It was the pre dispatch dispatch. But now, right, as long as you have the app running on your home, quote unquote, home computer, your desktop computer, you can access any file just via dispatch, but that's not computer use. Right. So the way that it does this is when you set it up originally, well, it can just run things through the command line. So it's kind of taking a backdoor approach. Again, within the guardrails that you set up originally. So a little confusing, but dispatch is just a way that your Claude app on your iPhone and your Claude app on your Mac can kind of talk to each other and access files that you've already set up via folder access. But now the combination of dispatch and computer use is actually really cool. So there are these now new settings or features inside dispatch. So same thing on the desktop app, you're going to want to click on dispatch. And then there's a couple new options on the right hand side. And one of them is enabling computer use. There's also the option to keep your computer awake and allow all browser actions. And I don't know why the allow all browser actions is actually a toggle because no matter how many times I tried to get this to work, you still have to always click allow like a trillion times, which takes away from its utility. And I hope that anthropic fixes that portion because it's extremely buggy in that regards. So that's a short, not very short primer on the setup. So and then when you do that, anthropic does kind of warn you, right? So it verifies and I'm just going to read this prompt that anthropic gives you. It says turn on computer use. Claude will take screenshots of your screen and control your mouse and keyboard. You'll approve each app, but not to confirm each step Claude performs. Keep in mind, some actions can't be undone. Apps you approve could open other apps that you haven't approved. Websites and docs could contain malicious instructions that misdirect Claude. And it says close anything sensitive Claude can see your screen. And then as a reminder, the bottom it says this is a research preview. Start with tasks where mistakes are easy to fix. And I think this is a good kind of call out by anthropic just saying straight up, even you can approve certain apps, but those apps might open other apps. And maybe show an example of this. We'll see. I don't want this to turn into, you know, another 35 minute show when I say stick around with me for 20 minutes. But even in instances where I tell computer use to use a certain app, sometimes it's like, no, I don't want to. I'm going to use a different app. All right, so we'll get into that a little bit more. But here's how Claude's computer use actually works. Well, it takes a screen chat to your screen decides what to do next, acts and repeats the loop. So yeah, it's slow. All right. And you can choose which model to use. And, you know, maybe there's a trade off there. I've been using Opus a little bit more because when it comes to agentic and computer use benchmarks, it's the best, but it's also very, very slow. Right. I'm starting to test some different workflows in sonnet and in Haiku as well. But it's pretty slow because it's essentially just using computer vision and it's screen-shotting everything on your computer. And then it's, you know, kind of directionally slowly moving them out. So you do need to watch for the orange outline on your screen. So this is a newer feature. And this shows you that Claude is using computer use because sometimes you can even ask it to use computer use. And well, it might not. Right. So even as an example, Claude says, if, if you have different connectors already connected, like Gmail or Slack, and you say, go to Gmail.com and log in and do A, B and C via computer use. It might do that sometimes, or it might actually just default to using your connected integrations inside Claude. So if you do kind of look for that orange outline on the screen, that confirms to you that, yes, Claude in this case is using computer use. And it's not just running in the terminal or using, right? One thing I found when I tell it to use, you know, an agentic browser, right? Like I'm trying to use, you know, computer use to control perplexities, Comet or to control Chantix Atlas. When I do those things, it will open them, but it'll actually just do the task in Chrome because it has a Chrome extension, the Claude Chrome extension in there. So even when I tell it, hey, go open, you know, Chatcha P.T. Atlas and do this task. Well, it's like it'll open Chatcha P.T. Atlas, but then it's going to decide to do the task elsewhere and not use computer use really at all. So you do have to kind of monitor it. So yeah, those connectors run first, Chrome sometimes run second, even when you don't want it to in full desktop control really only kicks in as a last resort sometimes. So I do hope in the future that there's a quite literal, a toggle that you can put on that forces, you know, the Claude app to use computer use by default, versus it going through connectors or Chrome first. And then like I said, already dispatch is what makes this worth considering because one of the downsides, which I'll talk about that more in a minute, is you can't really do too much on your computer when you're using this. So I think right now, one of the best use cases for this is when you step away from your computer, right? So being able to use the dispatch, the dispatch app to essentially do any work that you would normally want to do in front of a computer. I'm one of those for me. I don't know if it's because my fat fingers are because I'm used to having all these big, right? I usually have, you know, these two 25 inch or 24 inch screens that I'm always working on in multiple desktops on those screens. I stink. I absolutely stink at doing anything on the phone. So I do like using the dispatch feature because then essentially it's like, okay, well, I'm using my entire computer now wherever I am, right? And being able to dictate that with my voice is really nice. But here's the limitations that no one's really talking about. Yeah, this thing got, you know, 75 million views online, but no one's talking about the things that don't work very well. The biggest thing is it just screen hijacks, right? So a lot of times when I'm working with different agentic browsers, they kind of leave you alone. Right. Let's just use as an example for PlexiD's comment or chat. GPT Atlas, right? If they are doing agentic browsing, well, they're going to take up that portion. This is different because it uses your mouse. You can't really do anything. So I can have even just a lot of different agents or agentic browsers kind of going on their own desktop on my computer or their own screen or their own monitor. You can't really have that with cloud computer use because it's screen hijacks. It mouse hijacks. So if you think that, oh, this thing's going to work alongside me on one side. And I'm going to do all my stuff. You can't really do anything while computer use is running because it needs your, it needs your mouse. It needs your screen. So you can't really do anything else. The other thing, it is slow. It is finicky, especially when you're using multiple monitors. Right. We'll see when I do my live demo, if that works. And the instruction following just breaks down under real conditions. I already talked about that, you know, telling it as an example to use chat. GPT Atlas and computer use, it'll open it and then use Chrome on its own. And also the permission prompts are required for every new app. In every, from what I've seen in, in my own testing, every single new run. Right. So even setting something up on a schedule, you might think, oh, my gosh, computer use on a schedule, I can run this every single night and, you know, do these little tasks. Well, yeah. But at least for me, at the beginning of each run, I have to authorize it each time. So yeah, you can do that via the mobile app on dispatch, but it kind of takes away this concept of being able to proactively automate and schedule computer use. Right. Anthropic in the app says you just have to do it once and then it'll work. And at least for me, it doesn't. It's permission prompting each and every time. I can't wait for the equivalent of the dangerously skip permissions. Right. If you use cloud code, you probably know that I can't wait for the equivalent of this for computer control or computer use. They didn't even name this thing, right? Because technically computer use, no one talked about this either. This is another limitation, I guess. If you're going to look up computer use, you know, like, oh, let me find some information on this. Good luck. It's not on their website. You got to go find their tweet. And actually they came out with a quote unquote computer use product. I believe in October 2024. Right. So it was essentially a even slower version of this. You had to run it via Docker, right. And it didn't work at all. So yeah, they didn't even really give this thing an official name. Most people are kind of calling it computer use, but technically, Anthropa came out with a computer use product in October 2024. And they don't even have this on their website. So. All right. What could go wrong? Let's go live. All right. And this is for our live stream audience. This might seem awkward. I'm going to go ahead and cut off my camera. All right. Because I'm going to use my phone because that's what I record on. And I'm going to run some of these examples on dispatch. So I'm going to pull up and share my iPhone screen as we kick this off. So give me a second here. You're probably going to go blank here for a brief second. All right. Hopefully, hopefully we do this right. Live stream audience, let me know you should see my computer screen. All right. And let's go ahead and well, let me first describe what we have going on. So I am mirroring. This is a live, although it's being a little finicky here. There we go. You'll see this is the live view of my iPhone. I have the iPhone. Claude app open and I have the Claude Mac app open and I'm sharing my entire screen. So you have my Mac app here on the left. You have my iPhone app here on the right in my desktop running in the background. So we'll see. Hopefully this works. I might have to drag some things into view, but we'll see. So I just sent a prompt that I already had kind of saved in because you didn't want to. I already said I had fat fingers and I hate my phone. So I'm not going to try to type this out. But I am doing something fairly simple here. And hopefully this works. Okay. So I said for this task, use the computer use function, open codex on my computer. Right. So I'm having Claude use codex. This should be fun. All right. And then I'm saying go. All right. Here we go again. All right. It's asking me to allow for this session, even though I've already allowed this multiple times. I'm going to go ahead and click to allow. All right. Here we go. Okay. So a lot of things just jumped on my screen. Don't worry. I'm going to tell you what I said. So I said for this task, use the computer use function, open codex on my computer. Go to the build Mac OS Pomodoro timer entry on the left hand side and click into it. At the bottom of this page, find the bash command to run the program in the terminal, run the command, open the app, then click around the user interface for a few screens. Then send a message to codex with one specific piece of feedback or improvement based on your findings. And then I also said, I was trying to run this task a couple of times and then it kind of refused. It's like, I just did this, right? And I'm like, I know I just did it. Please do it again. All right. So that's all we have doing. So now I'm going to try not to really move my mouse, but hopefully we'll be able to see what's going on. And I won't have to jump in because like I said, it does sometimes run on other screens because I do have a kind of two screen set up right here. Okay, cool. So it did this and it ran this app. So it did the first part. Well, so side note secret. It's a little app that I've been working on a productivity app. I wish I had more time, right? But I tell you a lot of times I always have like codecs and Claude code open and there's like three or four apps that I've been working on apps that I've used myself. But I think ultimately, probably a lot of people would want to use, you know, maybe, maybe I'll share more on that in our community. So if you're in our inner circle community, maybe go bug me about that and be like, Hey, what's up with that? So anyways, Claude code is doing what it's supposed to it. Launched this app. So it opened first codecs. It found this command to open this app that's still in development. It's clicking around the actual screens, right, which is really cool. So this was not an option to do right like a week ago before this came out. And you will kind of see for our live stream audience, you can kind of see a faint orange glowing highlight around it. So that tells the user that it is using computer use. And well, also, it sees everything that's on the computer. So if I had anything else on that screen, it is going to see that. So again, when you talk about privacy and security implications, that's why. So if you have your text messages up or your bank information in another window on the same screen, and you're sharing an entire screen, it will see all of that. All right, so now one thing I'm not the biggest fan of. I'm going to actually make that a little bigger there. OK. AI moves too fast to follow, but you're expected to keep up. Otherwise, your career or company might lag behind while AI native competitors leap ahead. But you don't have 10 hours a day to understand it all. That's what I do for you. But after 700 plus episodes of everyday AI, the most common questions I get is where do I start? That's why we created the Start Here series, an ongoing podcast series of more than a dozen episodes you can listen to in order. It covers the basics for beginners and sharpens the skills of AI champions pushing their companies forward. In the ongoing series, we explain complex trends in simple language that you can turn into action. There's three ways to jump in. Number one, go scroll back to the first one in episode 691. Number two, tap the link in your show notes at any time for the Start Here series, or you can just go to starthereseries.com, which also gives you free access to our inner circle community, where you can connect with other business leaders doing the same. The Start Here series will slow down the pace of AI so you can get ahead. Cool. So it did it correctly. Claude code went in, opened codecs, launched this app, clicked around, find some things that maybe didn't work correctly. And now it's typing a message to codecs. Unfortunately, I think I kind of, okay, there we go. I was going to say I had to move the mouse to try to resize the Claude code window so we could see what's actually going on. But it did it. So it said, let me kind of click over here. So it gave a message to codecs and it said in, it said UI feedback. In the next tab, the accents, red for overdue, orange for due today, neutral for upcoming so users can instantly triage their query at a glance without reading each label. So there you see right there. All right. This actually worked, right? Like this I knew was going to be consistent and was at least going to be a decent enough demo that you could see and understand. You know, some of the utility from this new feature from Anthropic. There's a lot of other things and I don't know why. Claude computer use is just really bad at browsing the internet. So I tried it in Claude code with the, sorry, I tried it in Google Chrome with the Chrome extension, not very good. Yes, it does just take screenshots of everything. But the browser use, you know, on all benchmarks, the Anthropic models usually do fairly well. So for whatever reason, it really struggles. I don't know if this is again a bug. Again, keep in mind, this is a research preview. So if you wanted to do a bunch of agentic browsing, probably not the best use case right now. Right. And this might sound crazy, something like chat, GPT's agent mode, which hasn't been touched or updated. It seems like in a year and isn't that good. It's actually pretty good for doing things like that, probably better. Or using like what I use codex has a great playwright and interactive playwright integration, which is essentially just a very good dedicated Chrome browser codex is amazing at that. So for whatever reason, Claude code not great at instruction following in the new computer use. It's not great at using the browser, but for other things, opening other apps, like what I just did here was actually pretty good. Right. It worked. And it came back. It said, you know, it reported back to me in the kind of dispatch terminal here. It said, done. Here's what I'm going to do. I opened codex. I found the project you told me I ran the batch command. You know, I clicked through the four different screens and I sent this feedback to codex. So did a pretty good job. And you can kind of see in the background here codex is is running and doing all the updates that Claude code talked about. All right. So let's wrap this thing up and I'll answer the question. Is the new computer use that went mega viral? Is it a fun party trick or is it a real, agentic workforce multiplier right now? And well, the answer is today it's a party trick. But I think the architecture points to something much, much bigger. Right. We talked about this, you know, earlier this week on the show. And I think I mentioned this briefly last week as well. I think this is the next layer of knowledge work. I think this actually will be pretty big. And I do see this rolling out pretty quickly to Microsoft, right, which might be shocking. But little do most people know it seems like Microsoft is now going to have a dedicated senior I think is a vice president working on an open claw integration. Right. So Microsoft has actually been shipping a lot of things very quickly. They shipped a version of co-work, right? Like very similar to Anthropics, co-work Microsoft has co-work Microsoft has tasks. So I wouldn't be surprised for something like this a fully integrated or something like this, a fully functioning computer use product like this for Microsoft. And when that happens, I do assume that open AI and Google wouldn't be too far behind, right? Google already has this built into their their browser with the Mariner Project. You know, open AI obviously has this on a virtual browser, virtual sandbox via the chat, the agent mode. So I do assume that when open AI goes to the super app, this will be a future functionality. I assume that Microsoft and Google will follow suit as well. So I do see this even though it's buggy. Yes, it's a research preview right now. It's party trick, but I also firmly believe that this is the future of work. I don't think this is the end medium, right? But I think I mentioned this before. I think there is a brief period of maybe a half year to a couple of years. Where this is going to be the next interface, right? Until more and more websites, more and more apps do a little bit better at supporting agentic protocols, you know, A2A, you know, MCP, there's all these, you know, ACPs, these agentic checkout protocols, all these different things until, you know, essentially the entire Internet, all SaaS products support those. This is going to be the new human duct tape. It is going to be the AI duct tape. And that is an agentic computer using layer. So even though it's not great right now, I still think it's promising. And I do know that Anthraatic will continue to ship updates because they've been shipping like crazy. So I do think that this will improve. So I hate being that guy that always says this is the worst it'll ever be. But this is the worst it'll ever be. So you should probably go in now and start using it and get familiar. And a couple pieces of advice for me. I've been using this a lot since it first came out, plan to run at any task at least two to three times. And you have to very closely audit the results even for very simple workflows. Right. So Anthraatic, you know, recommended to start with, you know, simple tasks. And I agree, but you really do have to run them multiple times, look at the results and even ask Claude like, Hey, this didn't go right. How should we reframe this so it actually works consistently? And I think this is going to be the first one of the first major pieces for an actual super agent that works everywhere. Because like I said, there's obviously great AI agents out there that can automate all of this without you having to look at anything. But I think the majority of enterprises right now, well, they're using different software or for whatever reason, you know, having things locally and not in the cloud where, you know, they're not able to integrate things directly. So I do think a lot of what us humans are doing, right? The copying and pasting between different AI systems, you know, exporting data, uploading it into a large language model for evaluation. I think this is the next layer. It is the agentic computer using error, not error, although there's actually been a lot of errors, but layer. And I do think it's important for you as you put AI to work this Wednesday to get familiar with it, even if anthropic Claude isn't your go to might be worth signing up for a month to give it a go just to see how it works. All right, that's it. I hope this was helpful. The review of anthropic Claude's viral new computer use feature. Hey, if this was helpful, let me know. All right, if you want to see more shows like this, we do these more hands on demos on Mondays on or sorry, on Wednesdays. On Mondays, we bring you the AI news that matters. That's a recap of the biggest and most impactful trends and stories and happenings of the week on Fridays. We do feature Fridays going over some of the sometimes looked over or glossed over features and updates from large language models. And then on Tuesdays and Thursdays, we rotate shows, right? So if this is helpful, please consider take 30 seconds, subscribe to the show, leave us a rating. I would really appreciate that. If you're listening here live on the live stream, please repost this tag. One of your friends on LinkedIn that needs to hear this. And then when you're done, go to your everyday AI dot com. Sign up for the free daily newsletter. Thanks for tuning in. We hope to see you back tomorrow and every day for more everyday AI. Thanks y'all. And that's a wrap for today's edition of everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more magic visit your everyday. Dot com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.