Nate B Jones
Gemini 4 Argon Is #1 On A Leaderboard. Here’s Why You Still Can’t Use It.
Ask the tutor about this video
Sign in with a verified email and ask the tutor anything about this video.
TL;DR
Research summary
▶ Benchmarks give way to customer fit
The speaker says people can stop following AI leaderboards because benchmarks do not show whether a model is useful, especially when few people can access it. They argue that leading AI products are built by people who use them intensely and fix what fails, making customer obsession the more relevant benchmark. They describe Anthropic as focused on enterprise, OpenAI as having a growing enterprise arm and a large consumer business, and Meta as focused on consumer AI. They question how Google’s Argon model serves customers: Google earns from compute, serves many people through Search, and offers Gemini, but the speaker does not yet see how Argon connects to those uses. They want Google to succeed and say more AI players could make things safer and better for consumers. The speaker frames this as a broader challenge for companies and leaders building products.
▶ Building products around evolving AI
The speaker asks how companies can make money with AI when the underlying intelligence may keep changing. They contrast older software roadmaps, with releases every month or two, with the challenge of building products atop evolving models and helping users understand how AI may change their lives in a positive way. They call this a difficult product challenge receiving substantial investment, and say companies will make little progress without understanding their customers and what those customers do. In their view, the important issue beneath benchmark news is whether builders use AI products, notice failures, and make those failures matter; they describe this as both a business and an engineering discipline.
▶ Using Dots and Muse to manage subscriptions
The speaker describes extending an earlier personal-subscription review with Muse to business subscriptions, whose information was spread across accounts and services. During a flight, they used ChatGPT’s Dots tool and built-in connectors to gather financial sources and email, then had Dots create a combined personal and business subscription model. They fed the spreadsheet to Muse, which they say was faster and could navigate the web to take action, including cancellations; they valued delegating the work when short on time. The speaker describes Dots as slower but strong at connectors and combining large amounts of information, and says they use it for complex work that can run over time. At the time of speaking, Dots was processing more than 20,000 X likes and bookmarks to find posts about books; it had processed 741 books toward a reading list, a count the speaker expected would update. The example illustrates how hands-on use reveals where each product works well.
▶ Useful agents and products for specific users
The speaker says businesses should use the products they sell so they know which use cases work and which do not. They see room for many AI winners and argue that working agent utility can help some of the planet’s 8 billion people across varied use cases. They call for more companies, including Google and others, to put AI into customers’ hands in relatable ways, and express optimism about builders and companies with existing customer relationships. Because people face uncertainty about AI and are expected to become knowledgeable about it, the speaker encourages building products in specific domains that evolve with AI and serve particular customers. As an example, they mention a product using AI image recognition to help children identify trees and plants on forest walks. They do not know whether it will succeed, but like its apparent consideration of parents’ interests in discovery, outdoor engagement, limited screen time, and access to AI benefits.
▶ Moving beyond chatbot design and asking about Argon
The speaker says many AI products may fail, but believes there is more opportunity than is commonly recognized. They describe the field as trapped in the chatbot format, saying other products have borrowed from ChatGPT’s design and added features such as toolbars and places for artifacts. They argue that AI product design is still at an early stage and needs broader design thinking and greater understanding of customers. Although Argon reportedly had strong benchmarks, the speaker wants Google to explain where it will appear, which customers it serves, whether it will be in Search or Gemini, where it succeeds in Gemini, and what makes it more useful than the current Gemini model. They call for more customer obsession and optimism, saying doomer discourse can limit hope about what widely available intelligence could do.
▶ AI as perspective and empathetic support
The speaker recounts someone using Muse after a child had an accident involving a toy and an eye; the child and eye were fine. The person wanted perspective in the moment while understanding that Muse is not a doctor. The speaker says they also use AI to think out loud and think better, describing it as extending their ability to remember, think, and understand events. In a screenshot shared by the friend, Muse considered how the son might be feeling and suggested that the child may need time rather than a lecture about toys. The speaker presents this as an emotionally supportive response that avoided giving medical advice while showing empathy, and as an example of customer-focused product behavior.
▶ Daily use cases and product experience as the bottleneck
The speaker says they would rather discuss practical AI use than benchmarks, frontier research, or mathematical breakthroughs. They compiled 26 agent use cases from the previous 3 months, including tax preparation, travel planning, children’s school schedules, and understanding medical history, to offload complexity and focus on what matters. While they acknowledge that AI is not perfect and that model limitations contributed to problems in 2024 and 2025, they now see product experience and customer obsession, rather than models or benchmarks, as the bottleneck. They want more customer-focused AI experiences and invite people to share examples. They mention Instinct, an AI startup that launched group chat with AI to help friends coordinate plans such as getting together, seeing a movie, or choosing where to have dinner. They are unsure whether it will catch on but see it as a product pitch for a particular use case.
▶ Evaluating models by customer value
The speaker asks viewers to share examples of AI products that work for customers. They note rumors that Fable 5.5 is about to launch and may be an impressive model, while cautioning that online claims are not always true and that it may or may not lead leaderboards. They expect it to serve enterprises because, in their view, Anthropic is focused on solving enterprise problems; they describe the model as another tool in that effort. The speaker plans a full Fable 5.5 review based on the value it brings customers rather than its leaderboard score. They conclude that the next year in AI will be a year of product, not benchmarks.
Full transcript
[0:32] the different labs and the different models out there, we see a really clear map of lab to customer except for one instance. So, I'm going to go through. Anthropic, they focus on enterprise. They focus on how you can professionally use AI to succeed. Open AI, they have a growing enterprise arm, and they also have a very, very big consumer play. If you look over at Meta, Meta is deep on consumer, and Muses the latest AI release there, and their whole motion,
[1:02] their whole push from Zuck is all about AI for consumers. What about Google? Well, Google makes a ton of money from compute. Google obviously serves a lot of people with search, and Google has Gemini. What I don't understand is how Argon supports those use cases. Why do we need that kind of cutting-edge model? And that got me thinking about the importance of having tight customer-product relationships in the age of AI. And that customer obsession
[1:32] is something I would love to see more of from Google because I'm someone who thinks that the world is better off with more AI, not less. And especially in a world where we're talking about powerful AI, you want that power in the hands of more people, not less. So, if you're one of those people who's worried about AI, and you're like, "I don't know that I trust Anthropic. I don't know that I trust Open AI. I want other players." Well, you want another player in the game? Great. Let's have Google in the game, too. I would love there to be more players because I think that more players, generally speaking, make things
[2:03] safer and better for us consumers. So, I want Google to succeed. And I say this with love, but I don't yet see how Argon is connecting with customers. And I think that that leads me to some larger questions for all of us. If you work at a company, this is for you. If you work in enterprise and leadership, this is for you. You need to understand, in a world where intelligence is evolving every single week, how do you make money with AI? And
[2:34] especially, how do you make money with AI when the underlying model you're using is probably going to keep evolving? You know, when I was in product, for a long time, we would just release software. And the software that we released, we could update it, you know, once a month, once every 2 months, and we would have a road map, and that's great. I don't know that road maps work anymore. I think a lot of us in product would say they don't. And the larger problem is that you have to build a product with an evolving intelligence
[3:04] underneath that also needs to help users understand what is actually going to happen to their worlds as AI changes in a good way, not in a doomer way. That's a really tall order. It's a tough product challenge. It's a challenge that literally a trillion dollars is getting thrown at right now. But if you don't start with understanding who your customer is and what they're doing, you're not going to get very far. If you think about this Benchmark news, the Benchmark news
[3:35] doesn't matter. Let's peel that back. Let's look at what's underneath. I think the best AI products are being built by people who obsessively use them and notice what goes wrong and who make those failures matter. And this is a business discipline as much as it's an engineering discipline. And I want to share an example because I ran into a very practical use case when I was using AI myself. So, I shared a few weeks ago that I used Muse to review my personal subscriptions, and I talked about that in earlier video. But, I
[4:05] wasn't satisfied with stopping there because I also have business subscriptions, and the same problem existed across more than one part of my life. And the information was scattered across lots of different accounts and lots of different services. So, on a plane flight that I took recently, I went further. I used ChatGPT's Dots tool to connect to all of the different sources I needed. I could connect to a Mercury, I could connect to other financial uh sources, and I could have OpenAI's built connectors help me bring all of my subscription material together
[4:35] along with email to double-check it, and I could use Dots to process all of that information and to build me a financial model of subscriptions across both personal and business all in one place, which I never seen before. Afterward, I could take that spreadsheet and feed it to Muse and have Muse be like my hunter-killer drone. Like, Muse went through and helped me action against all of those subscriptions and make good decisions against it. Partly because Muse is faster than Dots right now. Partly because Muse is able to really
[5:07] use that speed to navigate the web and get things done directly. So, I can just delegate stuff like a cancellation. I actually had someone reach out to me when I shared that I was doing this and say, "Why would you use Muse to do that? You can literally click through and do it yourself." And I said, "Yeah, I I can, but it's awfully nice when I am strapped for time to be able to hand a job to an AI and have it just get done." And so, when we're talking about customers and we're talking about what matters, I use that story because I think it illustrates the importance of
[5:38] actually using the AI. And when I use Dots and I use Muse, I see where the AI sings and where it doesn't by trying it out in products like that. So, Dots is a little bit slower than Muse. Dots is very, very good with connectors. Dots can hold a lot of information and put it together coherently. And I find that I use both. I find that I lean into Dots where I need complex work done over time and I can just hand it off. Right now, Dots is working through literally more than 20,000 likes and bookmarks on my X
[6:11] account and chewing through them and finding all of the individual ones where I liked or bookmarked a post about a book because I want to have one place for a book list cuz I love to read where I can go back and pick a book off. And I've never been able to do that before because the job was too big and required too much analysis. And Dots is taking a matter of days to process through that quietly and is up to 741 books that I'm sure it will update in a few minutes, but it's up to 741 books that it has processed through. So,
[6:42] what I am sharing with you is the customer experience that comes when you use an AI and push it and discover how it works. And what businesses need to do more of is they need to use the products that they are selling so that they are confident in the specific use cases that work or don't work. I think there's room for a lot of winners in AI and I don't think we need to be complaining about good agent utility that works. If it
[7:13] works and the agent does the job, then for some number of us, it will be really helpful. There are 8 billion people on this planet. We have a lot of different use cases and there's a lot of room for agents to solve that. And when we look at the world that way, one, I think it leads me back to where I started, which is this idea that we need more players in AI, like Google, but beyond Google. Players who can bring AI into the customer's hands in a way that's relatable. I really am optimistic
[7:45] for builders right now and for people who are in companies with existing customer relationships. We are in a world where building for customers has never been more important because we are all facing the uncertainty around AI together. There's no place to run from it. Everyone has to be an AI expert now and everyone isn't prepared for that, but we're all expected to be that in various ways. And so this is our chance when we are looking at our individual spaces, our
[8:15] individual domains to actually sit down and say, "How can I build something that I would want to use in this space that evolves with AI as AI evolves and that is really, really useful to particular customers." And your customers could be anybody. Like I saw an example recently of a new product that's out that really focuses on using AI image recognition. And it's a very simplified product to allow kids to go on forest walks and
[8:45] they can actually recognize the trees and plants around them. I don't know how that will do, but I loved it because as a parent, I want kids to be able to find out about what's around them, keep them plugged in to the world that that they're living in, not have them have too much screen time, and still have the benefits of AI. And so that's an example of a product that got thought through for parents, that got thought through for kids. We need more products like that. We need more people who are thinking about where AI can go and how AI can deliver useful
[9:15] solutions. And I don't know that all of them will work, but I do know that a lot more room is out there than I think we realize. I think we are trapped in the tyranny of the chatbot. We've seen chat GPT get to what, 1.2 billion users right now? Every other product since then has in some sense borrowed from that original design philosophy. It has taken the idea that is chat GPT and said, "Well, now we're going to add a toolbar. Now we're going to add a place for artifacts."
[9:45] We are just at the beginning of figuring out what AI means, and we need wider design sense. We need more sympathy and understanding for customers. And so when I think about Argon, and I know that it had great benchmarks, the first thing I think about is where in Google's surface does it appear and for what customers does it appear? And that's why I'm asking Google, please release it. Please share it. I want it to succeed. Does it go in search? Does it go in Gemini? Why does
[10:16] it go in Gemini? Where does it succeed in Gemini? Why is it more useful than the current Gemini model? What is it about Argon that makes it sing? And that's just one example. I think we need more customer obsession, and we need more customer optimism in AI. I think so often right now the doomer discourse that's coming is keeping us from being hopeful for our customers. It's keeping us from being hopeful for what intelligence can do if intelligence is everywhere. There was someone that I was talking to
[10:46] over the weekend who used Muse to help figure out what is the right approach with a kid who had an accident with a toy in their eye. The kid is fine, the eye is fine, don't worry about it. But they wanted to talk to someone who is there who could give them perspective, and they know that Muse is not a doctor, but they needed some perspective in the moment. I found myself talking to AI when I need perspective in the moment, too. Not because I'm forming some kind of weird parasocial relationship with the AI, but
[11:18] because having it there helps me to think out loud and think better. In other words, I can think of so many specific use cases where AI effectively extends my ability to remember, extends my ability to think, extends my ability to understand what is happening. And I see that happening in individual moments like the doctor moment with my friends. And it happens more when we obsess over the customer. One of the things that my friend was sending me is she sent a
[11:48] little screenshot of the Muse text, which she noticed that Muse thought about how her son was feeling. Hey, your son probably had a hard time. They probably don't need a lecture from you about toys right now. They're going to be okay, but you just need to give them a minute. It's a model that actually had a moment of delight, a moment in a difficult emotional moment where the parent is worried, the kid is worried, and this AI finds a way to walk the line, to not give medical advice, but also to have some empathy.
[12:18] That's customer obsession. That's something that we need more of. When I think about the future of AI, and I think about why I am hopeful for builders in this moment, those are the kinds of stories I think about. We have enough stories about benchmarks. We have enough stories about researchers that are breaking the edge of the frontier. We have enough stories about the Navier-Stokes equation and all of the math. Do you notice I don't cover a ton of the math breakthroughs on this channel? That's on purpose. I don't think for a lot of us it makes sense.
[12:49] I think it makes more sense to talk about how we are actually able to use AI for daily stuff. I actually compiled a list today of all of the different things I've used agents for. It came to 26 separate use cases in just the last 3 months. Stuff around tax preparation, stuff around travel planning, stuff around figuring out kids' school schedules, stuff around understanding medical history, all of this different stuff that I had just found because I
[13:20] was digging in and trying to sort of offload more and more and more of the complexity so I could focus on what matters. I know that it's not perfect yet. And partly in 2024 and 2025, the imperfection came from the models. More and more and more as I think about it, the imperfection in our AI experiences, the pain points, it's not coming from the models. It's coming from the product experience. This video is being made because I think the models aren't the bottleneck anymore. I think the benchmarks are not the bottleneck anymore. The product experience, the
[13:52] customer obsession is the bottleneck right now. And I don't see as many examples as I would like to of truly customer obsessed AI experiences, and I would like to see more. By the way, if you know of some that I have not mentioned in this video, and I know there's a lot of startups out there. There are a lot of people who are looking for customer obsessed experiences. There's people who are innovating. Instinct, another AI startup, just launched group chat with AI, so you can throw an Instinct into your group chat with your friends, and Instinct can help coordinate the next,
[14:24] you know, time you get together, the next time you see a movie, the next time you have dinner, where you have dinner, all of that stuff. We'll see if that takes on, but at least it's a pitch for a particular kind of customer used case. We need more of those. I would like to hear from you where you see those examples working because I think it's really, really important. And as I talk about benchmarks, there are rumors that Fable 5.5 is about to drop. Rumors that say it's an incredible model. I've certainly seen amazing things on Twitter. We all see amazing
[14:56] things on Twitter. Not all of them are true. Fable 5.5 may or may not lead the leaderboards, but what it will do is it will work for enterprises. How do I know that? How do I have a lot of confidence in that? Because that is what Anthropic is obsessed with. They are obsessed with solving problems for enterprises. And Fable 5.5 will be another weapon in their arsenal to deliver extraordinary value to enterprises because that's what the entire company is obsessed with right now. And yes, they're obsessed with delivering a better model, but they have that business model context and that customer obsession. I will, of
[15:27] course, do a full Fable 5.5 review, but I won't do it because it topped a leaderboard. I'll do it because of the value that it's bringing to customers. That's really what I care about. And so, if you see this and you're like, "Where's Fable 5.5?" that's where it is. That it's coming. I'm going to have the review, and that's why I'm going to cover it, not because of what it what it scores on the leaderboards. The next year in AI is going to be a year of product, not a year of benchmarks.