Using only 20% of your business data is like dating someone who only texts emojis. First of all, that's annoying. And second, you're missing a lot of context. But that's how most businesses operate today, using only 20% of their data. Unless you have HubSpot, where all the emails, call logs, and chat messages turn into insights to grow your business. Because all that data makes all the difference. I would know because I use HubSpot at my company. Learn more at HubSpot.com. The new way to do AEO and SEO is via API. You know what's interesting? We had a call recently with a company that is literally they do documentation for dev companies. Let's say they're doing, just call it $10 million a year. They raise a lot of money. Let's just say $24 million to do this documentation. The interesting thing is they've gotten to that level by having a powered by button at the very bottom of the documentation they do for everyone. Anyway, my point of saying this is that I think they use an API. So ClickFlow, we have an API as well that people use to generate content with. But I think the way you're going to do it is like people are just going to log into Cloud Code command line interface and just say, hey, can you just do this for me? Can you build this for me? They can plug it into their agents. The agents do the work. The agents could go out there and buy stuff. There's a ramp CLI now. The agents market it, agents type of thing. So I think most SEO companies now, or AEO companies, whatever you want to call them, the way to do it is via API. And we have the documentation for ClickFlow. And we use, we use, actually use that documentation company for it. I think it is using API because you need it to plug into your agents to get the work done faster. And if you aren't doing it that way, I feel like, you know, it's not required right now, but I do think it's, it gets work done quicker for you. Did you see that LinkedIn thread on how they know and how they mark that their content and the stuff that AI is creating, they know that it's AI generated? You mean like Google? Yeah. I've seen that. And I'm a big believer that they genuinely do know, you know, I was a believer even before I saw that. I was a believer that they do know that one content is AI generated, because if you look at all the stats in the data, even though both Eric and I have our own AI content writing tools, it does not rank well. And you could get short term traffic, but over time we see AI content just not do as well. And it doesn't mean you shouldn't use AI to write content. It means you should use it in different parts of the content writing process. It's great for ideation. It's great for structure. It's great for research. A lot of the writing, I highly recommend doing it manual. And I believe it's going to keep going that way because if AI doesn't care if the content is written by AI and then they crawl it and then use it to create more content and keep learning, it becomes a self-fulfilling prophecy and then it starts hallucinating and they don't want that. So what we do for a click flow, so what I've heard the customer say, so what I've heard from our people working with the customers is that the content that they generate sometimes are just YOLO and let it publish. But similar to how like Finn does it with customer service, right? For like, sometimes I have a hard time believing this and I've seen the content quality myself. I'm like, this is really good, right? But they say this is better than any other content generation tool that they've used and that we have a human in a loop, right? So that's what helps. So when they say like the content helps them get more citations and it ranks better, it does. But there's a lot of caveats. There's one, we have a human in a loop. Second, we spend a lot of time on iterating on it. And so that goes back to what you're just saying about Harvey. We can't just let it YOLO on its own. But I do think if we can get to a point where we have a lot of data around it and we can just release our own LLM down the road, like a vertical one, that's like down the road. But that's at least what we're seeing on the click flow side. We've been seeing something interesting with content. So you know this, a lot of companies are publishing listicles on their own website. If you look at like chat GPT and when they're doing the searches, I was with Sam from Right Sonic. A lot of the prompts now have site colon commands. So like chat GPT will run a site colon command on your own website to see the articles you have to see if it's relevant to what people are asking. And a lot of companies are creating articles like the 10 best web design shops, the 10 best really well like a year and a half project management shops and whatever CRMs and they're creating their own pages for the services or products they offer. And what we're seeing is even though it can help you with GEO and helping you do better on chat GPT within six or so months of creating a lot of those listicles, assuming you do it in quantity, we see it hurt your organic search traffic. So it's just like you're, we think that's a wrong approach because you're trying to do better in one place at the cost of another channel. And we don't think in the long run, at least at my company, NPD, that you should be publishing this content on your own website or even publishing the content because this kind of content is something where it'll help you with the algorithms now, but we don't think it'll help you with the algorithms in the long run because it's just easy to manipulate. Yeah. If acquiring customers has been a struggle for you and you are trying to figure out how 80O works, answer engine optimization or the new version of SEO, how paid works, how all this AI stuff is going to play into your customer acquisition strategy and check out my ad agency, singlegrain. It's singlegrain.com and it looks like it's a good fit will help you with a free marketing plan. So again, go to singlegrain.com and we'll see you on the other side. So businesses can sell directly via Meta and chat GPT ads. I think we're going to see a lot more of this. We're not saying by the way, get rid of your website. I think the website is still important because the more information you can give agents to crawl, the better. So documentation around your software, very important in my mind. Whatever other information you have, get it all out there. Do more podcast interviews. Get your YouTube channel out there. Keep doing what you're doing because that stuff, they need stuff to ground on. It's not going away. This is a random note. This is a marketing related, but you want to know what my wife does to try to make us all healthy every single week? What? Make you wipe your butt with clean toilet paper and organic? No, but I don't know. I just bought a bunch of organic toilet paper. I don't know what my toilet paper is. Let me tell you what I bought. Go ahead. So what she does is she chops up different vegetables in different colors because radish is a different color than cabbage and all this kind of stuff. She tries to get all the major vegetables in her body every single week, not every day because it's too hard to get all of them in. Then that way we have all the different colors and vegetables because she saw some report on how different vegetables have different things for you and the colors are all natural and she just chops up so tiny in food like lasagna that you can't see and you just eat it. That's good. That's smart. That's a lot of work. What does she use to chop it all? Her hand? So let me tell you this. So by the way, we'll move on. Don't worry guys. We'll bring this back to marketing in a second. So I bought this, okay? Real 3Plysoft and strong bamboo toilet paper. And I just started using yesterday like 4.6 stars, 4,000 reviews. I saw this on Instagram and I bought it because someone was writing something about the toilet paper users, PFAS and all that because I was using like the Amazon one, the Charmin one. It's like, no, you got to get this. If you don't want PFAS in your butt, what's PFAS? Forever Chemicals. I don't know that. Yeah, so now you know. See? Better for your butt. Well, typically when we're home, we don't have toilet paper. What, you use your hand? No. Our toilets are bidets. And then we have wet wipes. Okay, yes. But I don't know what the wet wipes are in. It's toilet paper. But the wet wipes are made from like bamboo too or something like that. So it's like, mine's plant-based. The dude wipes, they're expensive. We don't use dude wipes, but we use another organic brand. I think it's organic for the wet wipes. I'm pretty sure everything in our house is organic, even though it doesn't mean much for food. But we have bidets that clean you and then you have the wet wipes on top of that. Sometimes though, the bidets, we'll just leave it at that. I'm just going to send you this so you can buy. Okay, guys, let's bring this back to why AB testing is dike. So back to marketing, guys. So we're getting close to the end. So yeah, we are. We still have 14 minutes left. So AB testing was the gold standard for product decisions for 15 years. The best AI companies abandoned it. So this is from Akash Gupta. For a generation of PMs, experimentation meant designing a controlled test, allocating traffic, waiting two weeks for statistical significance, and hoping you had enough sample size to learn something. That loop trained an entire discipline to think in cycles of weeks. E-Vals compressed the loop to minutes. Three components, a set of inputs your product needs to handle. Number two, a task that generates outputs. Three and a scoring function that produces a number between zero and one. You run on your laptop, no production traffic, no two week wait, no data engineering pipeline. The math on what this changes is staggering. Teams running evals are doing 12.8 experiments per day. That's roughly 384 per month. A traditional AB testing team runs maybe three over a quarter. One team makes more 1150 variations. The other has explored nine that learning gap compounds every single week. So the point of saying this, so he mentioned on Corgoyal, build the eval platform behind Vercel, Replet, Ramp, and Notion, $800 million valuation. He ran an eval from scratch on this episode, went from a score of zero to 0.75 in under 20 minutes. That's a PM shipping a measurable quality bar before writing a single line of product code. So let me unpack this for people. So Andre Carpathi, super AI researcher, did, he released auto research. I think these two kind of go hand in hand because auto research allows you to run compute experiments very quickly, right? AI experimentations. We're talking like every five minutes it's running one and he's just leaving his computer on and doing. He's found things that he couldn't. He's like, dude, over like 20 years of doing this, like they found a bunch of things that he didn't even know were possible. Okay. I think when it comes to evals, evals are just, it's just a simple way of saying, hey, what is defined as a successful test versus a not successful test? That's all it is. Okay. And then you're just running it over and over, but we're going to see a lot more of this. And depending on what you're doing, if you're sending cold emails or if you're running paid traffic or whatever, you do need a good enough sample size when it comes to marketing. And you can't just say you're going to do 384. So I think it depends on your data sample size. But now when you set these agentic platforms up or your agents up, you're just going to say, hey, I want you to run these tests and I want you to, every week I want you to run evals and just so it can recursively improve over time. That's where this is all going. And that's where a lot of this stuff has already gone with AI, but not. It's kind of starting to encroach on marketing. So this way he's talking about, we've seen this happen for years. This is, it's new to a lot of people, but it's not new in the Bay Area or the startup world. There's one issue with this model. It only works if you have high volume. If you don't have high volume, you can't do it. I think for most companies who are heavily venture funded startups or large enterprises, they can't leverage this because they don't have volume. And a prime example of this is if you're a new entrepreneur, you set up a D2CE com site. You're doing drop shipping. You're not even creating your own product. You're running Facebook ads and you're getting five sales a day. And that's it. So 150 a month. You don't have enough conversion volume to run this playbook. I think what's going to happen. So exactly what we're saying. But I think we're going to be using a lot more synthetic data in the future. Well, if you think about, you know, MMM, MMM has been out for ages. And it's what people have MMM, media mix modeling. OK, yeah, yeah. Right. So I'm talking about M&Ms. No, when you think about and some of the solutions that Google offers and a few of the others, right? Media Indian and the list goes on and on. When you think about it, you're taking data from what happened in the past to figure out what's most likely going to happen in the future. And it's saving a lot of time and money. And companies are also starting to look at not just, hey, were we spending our money and where should we? They're also looking at increment, incrementality in which was this person actually buying because of the ad or were they already on the checkout? I'm going to buy anyways in the ad just took credit. And that's another big problem in marketing as well, because we spend a lot of money on advertising. But yet was the sale actually caused by marketing or was it not? And, dude, we see this all the time. And the way you see this is not what I'm talking about. Incrementality or causality. It's when you have the executive team just being like, what would happen if we turn off all our ads? Let's try that for a week. Right. That's their way of saying, did the ad actually cause or does marketing actually cause more sales for us? Or is it just taking credit for sales that we would have already gone? And it isn't that simple because if you just turn off your advertising and your marketing for a week, a lot of what you're closing in that current week came from people who saw your brand and from advertising and marketing that happened, you know, months ago or a year ago. So it of course isn't that simple. But, you know, when you think about incrementality testing and you think about MMM because of AI, it's more easier for companies of all sizes. To use this stuff while it wasn't possible for smaller organizations in the past. Yeah. I think what people can learn from this. So what Neil is saying is true. This is not necessarily new on the marketing side, but I think on the PRD side, so a product requirements document, this guy is basically saying that evals are the new PRD. And so when you have a very specific spec driven document and it shows like it defines what a win and what a loss is, I think this is going to be built in a lot more to whatever agents I'm building, for example, whatever workflows that I'm building. So it's just on top like, Hey, I want you to have an eval here, right? That's just going to be built into whatever scaffolding that you have in the future. And that's going to allow us to do these things. Sometimes you might need synthetic data. Sometimes you can't do it, right? But I think we are, I think me, for example, I've been required to, we were doing PRDs maybe like a year, year and a half ago and like me, PRD, right? Doesn't make sense. But now evals, when you just build it in, it's it reduces a lot of noise. That's what I think is going to a lot more people are going to start doing it that way. Yeah. Yeah. But yeah, with all these changes that's happening with data, you know, not just synthetic data, because I know a lot of corporations are using synthetic data too. And when I'm talking about synthetic data, let's say you're having, you're doing running Facebook ads and they want you to pass in data on your customers. Well, a lot of corporations these days are passing in synthetic data. So then that way Facebook doesn't have their quote unquote customer data. Facebook hates it because when, when you don't pass in your real data, it makes their ecosystem not as strong and it doesn't help them for future, you know, companies who are running ads there. But because of technology, you're going to see two things. One, you're going to see easier for startups to get up and running and have the power that a lot of these enterprises had because things like MMM are much more affordable and easier to do. The second thing that you're going to see is it's also easier for companies to leverage things like synthetic data and protect their own first party data and not pass as much through. So then you're going to have these platforms like Meta and Google not be as effective from a targeting perspective because they're not getting passed in all the data from corporations that they would like. All right. So this, this can be a nice final topic for us. So this is also from Akash. So every CEO will be running an open claw within 12 months. So Mark Zuckerberg, Meta CEO is building his own in-house. So a couple of days ago, right? Jensen Huang stood in front of 30,000 people at GTC and said, every company is an open claw strategy. So now you have Zuckerberg building a personal CEO agent and it's a Fortune 5 company, right? And so, you know, a CEO at Meta scale sees maybe 1% of information that determines whether a decision is right or wrong. The other 99% gets filtered through VP's, Chiefs of Staff, dashboards and whatever made it into the pre-read. The agent replaces the filter. Think about what a CEO agent actually does. It ingests every product metric, every internal thread, every customer escalation. Every competitive intelligence report across every team simultaneously. Then it surfaces the three things that actually mattered this morning before every one to one one on one. It pulls that person's team metrics, open headcount, recent launches and the two things they said they delivered last quarter. When the CEO asks what happens to our glasses timeline, if we move 200 engineers to AI infra, the agent gives a first pass answer in minutes instead of a two week strategy team exercise. And it never forgets the person who remembered why the company killed that project in 2019 left two years ago. The agent didn't. So Meta employees are already running their own versions tools called myclaw and second brain engineering output is up to up 30%. Power users up 80% year over year. Zuckerberg is doing what his employees are doing, applying it to the highest leverage seat in the company. So this is exactly what I'm doing internally right now. I have everything connected and it's proactive with telling me it does forget. Don't guys don't don't think this is perfect. Memory is an issue with AI right now. But I do think that if you're applying it at the highest level, like this, for example, or even like the way I'm doing it right now, it cuts to it cuts the noise. Very quickly. And I can execute a lot faster. And I think we all like to move faster. I agree with you. This I still have. I don't understand the concept by AI can't figure out the memory issue. It doesn't seem like a big deal. You tell it something once and for some reason it doesn't always follow. I do you remember everything? You have really good memory. No, but this is a computer. Like it should be able to like hard code and not forget unless you tell it otherwise. The computer is made out of right. Yeah. Sad. Like we conjured it and then we put it in the truth. I think I think it's going to get better over time. And I think the memory problem should be solved. I don't know. Maybe in the next year or two. But I do think just the decisions I like, man, it's so Christmas. I can move a lot faster. For example, last week I made that CFO agent and it's like, oh, you have seven hundred thousand dollars in cuts over here. I was like, wait, what? Yeah. Found money, man. Right. It's great. It's great. I love it. So so when I when I make a copy for you, you can use it for yourself. You can just duplicate it and use it for yourself. You can use your own scaffolding on it. Oh, a gift to you. I was at an event a month ago and I met someone who's working anthropic and they're in charge of cutting up a lot of this stuff. And I asked him the memory question and he gave me an explanation on why AI isn't perfect on recalling everything. And it was too complex. I didn't understand it. And I was like, when will it get solved? And because that's what I was really looking for. Right. And like, I don't really care how you saw. Just solve it for me. Yeah. And he's just like, he's like, it's not going to be easy. We don't see it happening in the short run, but it'll get better over the next few years. That was his words. Not necessarily being solved in a few years. It'll get better and make the errors less often. That's what I'm really saying. Yeah. We expected as a compound over time. And as we talk about this, dude, like this is the worst I'll ever be. So I don't know. My eyes continue to burn to the ground every night. And I don't know about you. My eyes don't burn to the ground every night, but they do if you use them too much, right? No, they don't. No, I know. You have a laser, right? Yeah, I never had that issue. You're due. Yours too. OK, great. No, it does. All right. So Neil's preserving his eyes. No, no, they are being stupid. But yeah. Anyway, guys, that is it for today. Please don't forget to rate, re-subscribe and we'll see you tomorrow.
