Nate B Jones
Opus 5.5 frente al resto: ¿es el nuevo estándar del sector?
Pregúntale al tutor sobre este vídeo
Entra con un email verificado y pregúntale al tutor lo que quieras sobre este vídeo.
TL;DR
Resumen de investigación
▶ El modelo LEGO y el coste por tarea
El narrador presenta Opus 5.5 como un modelo cuya escritura le resulta más fácil de dirigir y cuyas capacidades visuales tridimensionales considera extraordinarias. Explica que quiere medir la eficiencia por la cantidad de trabajo necesaria para completar una tarea, además del precio de los tokens: cuentan también las lecturas de archivos, los intentos y las correcciones que hay que pedir. Como demostración, muestra su logo de gorro y gafas convertido en un modelo LEGO de 514 piezas, con base y soporte transparente, un archivo del modelo, lista de piezas, lista de compra de BrickLink, un video y un folleto de 63 páginas con 58 pasos de montaje. Dice que las piezas y los archivos coinciden en el total de 514.
▶ Tokens, tarifas y comparación de costes
El narrador afirma que el trabajo del modelo LEGO requirió 89 millones de tokens y consumió el 1 % de su uso semanal de Claude con Opus 5.5. Estima que esos tokens habrían costado al menos 44 dólares a tarifas de API; compara esa suma con una suscripción mensual de 200 dólares, que calcula como aproximadamente el 25 % de su precio, aunque dice que no pagó esa tarifa por la tarea. También menciona archivos para comprobar conexiones geométricas y compatibilidad entre ladrillos, y destaca el uso de código para producir trabajo visual. Según las tarifas que cita, Opus 5.5 cuesta 4 dólares por millón de tokens de entrada y 20 por millón de salida, un 20 % menos que Opus 5 y un 60 % menos que Fable 5.1, al que atribuye tarifas de 10 y 50 dólares, respectivamente. Advierte que la afirmación de Anthropic de que las cargas de trabajo típicas cuestan alrededor de un 40 % menos no significa necesariamente un 40 % menos de tokens: propone considerar entrada, salida, contexto, tarifas y consumo de suscripción. Menciona reportes de GitHub y Lovable sobre menos pasos y de Spotify sobre tareas más baratas y rápidas, además de informes de usuarios; interpreta esa eficiencia como una mejora potencial del valor por tarea.
▶ Medir la tarea completa y recordar las dificultades de escritura
Retomando el modelo LEGO, el narrador dice que acertar antes el diseño y responder mejor a los cambios puede dejar más margen para iterar. Aclara que el consumo depende de la complejidad: una ciudad completa, como la que dice haber visto de Lower Manhattan, usaría más tokens que su modelo. Insiste en medir el trabajo entero y no detener el recuento ante una imagen o captura atractiva; en su ejemplo, considera el diseño, los archivos y el coste de tokens como parte del resultado completo. Describe herramientas de código para crear gráficos tridimensionales manipulando objetos, materiales, luces y cámara, y dice que los modelos de Anthropic se han vuelto buenos usándolas bajo instrucciones sencillas. Luego contrasta su experiencia con Opus 4.6, que recuerda como excelente para escribir y programar, con dificultades posteriores en 4.7, 4.8 y Fable. Presenta como ejemplos de quejas individuales comentarios públicos sobre comportamiento discutidor y sobre instrucciones de escritura que se reconocían pero luego se ignoraban. Su valoración es que la experiencia propia importa: poder obtener lo pedido a la primera, dirigir el modelo y evitar que se resista.
▶ Escritura, dirección e iteración visual
El narrador dice que Anthropic reconoció problemas de comunicación sobre Opus 5 y que su anuncio de 5.5 señala escritura más clara y mejor seguimiento de instrucciones como correcciones importantes. En su opinión, Opus 5.5 se acerca a lo que apreciaba de 4.6, con más inteligencia detrás, y le permite avanzar rápidamente tanto con escritura como con tareas complejas. Define la buena escritura asistida por IA como aquella que preserva la intención: simplificar no debería eliminar la complejidad central de un argumento, dar calidez no debería debilitar una decisión firme y expresar confianza no debería borrar incertidumbre necesaria. Considera que los cambios pulidos pueden empeorar la claridad, y valora la calidad del resultado por encima del método usado para producirlo. Afirma que 5.5 es fácil de dirigir; por ejemplo, puede pedir que se conserve la incertidumbre o que una parte sea más fácil de entender. Para el modelo LEGO propone cambios concretos, como alargar el gorro sin tocar las gafas, ralentizar el montaje, añadir pasos o modificar piezas para lograr una apariencia más suave, manteniendo coherentes las piezas y las instrucciones. Añade que una escena hecha con código permitiría cambiar selectivamente objetos y movimientos, y seguir desarrollándola mediante conversación sobre la cámara o la forma del gorro.
▶ Tareas prolongadas y límites claros
El narrador cita el ejemplo del anuncio de Anthropic en el que Shawn Hanse, de Cleo, dejó a Opus 5.5 trabajando sin supervisión durante 18 horas en una tarea de ingeniería que abarcaba seis repositorios. Señala que ese tipo de autonomía implica muchas decisiones y que hay que volver y comprobar que el código funcione. Cuenta que probó dos o tres tareas nocturnas y encontró mejores resultados cuando definía condiciones claras para detenerse y una evaluación o definición precisa de terminado. Advierte que la persistencia del modelo puede prolongar el trabajo si no se acota el encargo, con consumo innecesario de tokens. Como ejemplo, dice que especificar de antemano el número de piezas y la complejidad del modelo LEGO evitó que el sistema tuviera que decidir por sí mismo el alcance y seguir ampliándolo. Para él, hay que definir el espacio de trabajo del modelo y concluir cuando la escena funciona, los cambios están aplicados y las comprobaciones necesarias se han hecho.
▶ IA que ayuda a desarrollar IA
El narrador amplía la conversación a las herramientas internas de los laboratorios, incluidas otras IA que los consumidores no ven. Atribuye a Anthropic la afirmación de que, desde mayo, Claude escribió más del 80 % del código integrado en su base, y dice que OpenAI describe a investigadores usando agentes para escribir código, ejecutar experimentos y resolver problemas. Interpreta estos ejemplos como IA que ayuda a diseñar IA, mediante investigación asistida o mejora recursiva. Propone que herramientas más capaces podrían ayudar a investigar y probar cambios con mayor rapidez y acortar la espera para mejoras útiles, aunque reconoce que no conoce el historial interno completo de cómo Anthropic produjo Opus 5.5. Como posibilidad, plantea que los usuarios señalen problemas mientras equipos con herramientas de IA investigan cambios. Dice que ve avances en escritura y que su criterio personal es usar los modelos que funcionan, fijándose en el coste por tarea.
▶ Más margen de trabajo y valor para los usuarios
El narrador plantea que, si el modelo LEGO y sus archivos se producen con menos esfuerzo y tokens, puede parar y gastar menos o usar el margen para hacer una animación, ampliar el modelo o empezar otra tarea. En su opinión, la eficiencia deja más espacio para decidir, iterar y trabajar en otros proyectos, y aligera las tareas complejas. Relaciona esto con la pregunta de cómo las empresas entregan valor de frontera eficiente y responden a sus usuarios; atribuye a Opus 5.5 la combinación de respuesta a comentarios y eficiencia por tarea, pero dice que OpenAI y Anthropic tienen trayectorias y modelos internos distintos. Afirma que el intervalo aproximado entre lanzamientos importantes se ha reducido a unas dos semanas y menciona que una cifra reciente que vio era de unos 18 días; vincula parte de esa rapidez con herramientas de IA usadas por los laboratorios. Expresa optimismo sobre futuros modelos. Como referencia de progreso, dice que en 2026 Opus 5.5 completó el modelo LEGO de 500 piezas usando el 1 % de su uso semanal, una tarea que, según su estimación, habría superado su benchmark LEGO dos o tres meses antes. Concluye que las elecciones de los usuarios sobre qué modelos usar y pagar dan a los laboratorios motivos para atender a la experiencia de trabajo.
▶ Cómo comparar modelos en tareas propias
El narrador recomienda evaluar la ergonomía de los modelos con los materiales y tareas habituales de cada persona, como hojas de cálculo, documentos o un repositorio de código. Propone repetir una tarea anterior con el mismo punto de partida, prompt y configuración disponible para el modelo previo, y registrar el consumo total de tokens, una estimación de la factura a tarifas de API y la parte del uso semanal consumida. Sugiere comparar con modelos anteriores y contabilizar el alcance completo: intentos fallidos, correcciones, intervención necesaria y calidad del resultado, que debería igualar como mínimo el estándar previo. Según su experiencia, Opus 5.5 superó a modelos Opus anteriores y a Fable en algunos de esos trabajos porque fue más barato y rápido y requirió menos explicaciones, aunque presenta esto como valoración personal. También aconseja conservar los casos en que un modelo no funcionó, no solo los éxitos, para identificar dónde mejoran las tareas y dónde no. Indica que el valor dependerá de la frecuencia con que los resultados sean utilizables y de cuánto cambie el trabajo diario; menciona como posibles áreas con mejoras distintas el trabajo visual, la escritura y la investigación.
▶ Un ejemplo personal y la recomendación final
El narrador cuenta que, tras observar el criterio visual de Opus 5.5, trasladó a ese modelo el trabajo visual y de diseño de un proyecto suyo. Dice que le confió, con un solo prompt, una tarea de diseño de alrededor de 90 elementos, que el modelo la completó y que no consumió demasiado de su presupuesto semanal; a partir de sus pruebas, cambió el flujo de trabajo del proyecto. Recomienda crear un conjunto de pruebas con encargos reales y ofrece enlazar un prompt para analizar patrones de uso previos y preparar uno personalizado, distinto de su benchmark LEGO y sus propios archivos. Cierra sosteniendo que medir tareas de manera consistente entre versiones permite comprobar si los usuarios obtienen más valor y reconocer una mejora. Invita a elegir una tarea cuyo resultado importe, medirla y repetir la medición con cada lanzamiento, y pide comentarios sobre Opus 5.5.
Transcripción completa
[0:30] 50 bucks a week. A task uses about 1% of my weekly allowance, a task I will show you in this video. It's a complicated one. That's 50 cents of the subscription I'm actually paying for. So, in this video, I'm going to show you the calculation, the work behind it, a sample LEGO build cuz we're going to have fun, and most importantly, I'm going to show you how to figure out whether you're getting that improvement on your own tasks. So, let's get into it and how it works. Think about what you're paying for when you use a model. There is a price for the tokens, the
[1:00] chunks of information it processes and generates. But how many times does it need to read the files to do that? How many attempts does an AI make to make a change that you care about? How many times do you have to keep explaining again and again? How many times does it churn out? All of that is part of getting the job done. So, when a model gets the job done efficiently, the price of what you're getting done can go way, way, way down. Now, I get it. This gets really squishy really fast. Your tasks are not my
[1:32] tasks. You could spend all day with exactly the same model and get a very different result because you're asking it to do different work, and models are spiky. That's what I want to get into with 5.5 in this video, how you can figure out what the efficiency gain is worth to you. So, look at this. It's my logo in LEGO. Say that five times fast. It has the beanie, it has the glasses, it has a clear column that holds up the glasses and the hat over a nice base. Now, I'm a LEGO guy and I like beanies, so of course this is what I wanted to
[2:02] make. You can watch the pieces come together in the video and then see the finished model from the front and it lines up with the logo. It's beautifully put together. And look what came with it. There's a 63-page instruction booklet with 58 different build steps. It shows which parts to add and even tells you when a piece has to attach from underneath. It gave me a model file, a parts list, and a bricklink wanted list. I can order it all. The model and the list both agree this is a 514-piece build.
[2:33] That's a lot of work. Do you want to guess how many tokens that was? It was 89 million tokens. Do you want to guess how much of my Claude usage that took up with Opus 5.5? This is all Opus 5.5. 1% of my weekly usage for that. And those 89 million tokens, they would have cost me at least $44 if I was at API pricing. I didn't have to pay that $44. And by the way, if you do the math on a $200 a month subscription, we're talking about 25% roughly of the subscription price,
[3:03] but I didn't have to pay that. And it only took 1% of my usage for a week. And for all of that value, I get multiple files. I showed you a video file. I talked about the parts. I talked about the instructions. I also got a whole check file that checks the geometric connections between bricks. And I got an LDraw file that actually makes sure that the LEGOs can connect and and and get put together. And this gets at a larger pathway that I think is under discussed
[3:33] that Anthropic is actively using when it comes to AI development. And that is using code to do visual work. So, what does all of this cost? Opus 5.5 charges $4 per million input tokens and $20 per million output tokens. Those rates are 20% below Opus 5. They're also 60% below Fable 5.1's $10 per million input tokens and $50 per million output tokens. So, we are seeing significant decreases
[4:04] for really, really solid intelligence that in many ways mirrors what Fable can do on ordinary tasks, but is much more efficient and much cheaper ultimately. And so, these are useful baseline numbers, but it's important to know how much the model chews on those tokens to actually do the job. This is where that task efficiency comes in. Now, Anthropic says, "Typical workloads cost about 40% less on Opus 5 from a combination of lower prices and of course, the model using fewer tokens to get the work
[4:35] done." So, keep that in your head, right? 40% less cost doesn't tell you that a task used 40% fewer tokens. Instead, think of it as input, output, cash, context, those all have different token rates, and ultimately, to understand the efficiency improvement, you are going to need to look at the token usage across all of those buckets, as well as the API bill, and then look, if you're a subscriber, at how that chews up your usage. But, the 40% seems plausible. Other users are pointing to this same kind of savings, the same kind
[5:07] of efficiency that I'm showing you here. In the reports Anthropic features in its announcement, GitHub and Lovable both describe Opus using fewer steps with 5.5, and Spotify describes doing the same tasks more cheaply and quickly. Uh and I see individual users reporting this all over X. And that's what makes this much more interesting than a typical model release to me. It's the idea that we can listen to our users, we can respond, we can use AI tools inside a company to build a better model that is more task efficient, and that
[5:37] delivers more value. So, think about that Lego build one more time. Claude has to work through that model, right? Claude had to work through the animation, had to work through the information in those build files, recheck everything. Every time something needs another attempt, Claude had more work to do. If it gets the design right sooner and responds more efficiently and usefully when I ask for a change, we have a lot more room to make this LEGO build better together. And that's the kind of efficiency that we're going to care about in real work. It's whether we
[6:08] can build it and then iterate it and feel like we have lots of room in the window to continue to improve together. Does that automatically mean that every single scene will use fewer tokens because we steer more cleanly? Well, it's going to depend on you, right? How complex is your scene? I saw scenes that looked very complex with the 5.5 release. I saw someone building an entire city. I think I saw Lower Manhattan made with Opus 5.5. Those look cool. They're going to use a lot more tokens than my LEGO build. But, one of
[6:39] the things I want to call out is we need to stop counting our tokens too early. If you've got a great-looking frame, if you've got a great-looking screenshot, that's wonderful. You still have work to do. And so when we talk about tasks, not just LEGOs, real work tasks, the real work needs to be the whole job and we need to measure that whole job. And that's why I did the whole job in terms of planning out the LEGO build and then I gave you the token bill for it. And that's why I'm impressed with the
[7:09] task efficiency, not because I got, you know, a particular screenshot back or because the video looked nice. So, code like 3.js lets you build three-dimensional graphics in a browser. It lets you manipulate the objects and materials and lights and have a camera angle. Uh and there are lots of different coding tools we can have for visual jobs, but I mentioned 3.js because the model, especially Anthropic models, have gotten extremely good at using coding tools like that under
[7:41] direction, under a simple prompt from you to produce beautifully tasteful results. And this is also what I love about the writing improvements in this model. Think back to Opus 4.6. It seems ages ago, right? Anthropic felt like it was at the top of their game, right? The writing was excellent, the coding was excellent. I wanted to keep using Claude all the time. And then, as we went through 4.7 and 4.8 and Fable, getting the work I wanted became more complicated. And that's not just me, by the way. I am actually reflecting back a lot of sentiment that the community fed
[8:13] back to Anthropic over the course of the last few months, particularly around writing. You can find that frustration out in public, right? Bram Cohen wrote about Claude's argumentative behavior very publicly in a memorable article entitled Why Is Claude Turning Into an And you can even find this in Anthropic's GitHub issue tracker, where a user described writing instructions being acknowledged and then ignored. And look, these are individual accounts that I'm sharing, but you can start to see a pattern when you see enough of these.
[8:44] And I've seen them all over the web. And what really matters is your experience, right? It's your and my experience. You can recognize when a model gives you what you ask for the first time, when it gives you exactly what you need, when it's steerable, when it doesn't fight back. And of course, Anthropic has a lot riding on getting that experience right. The company's growing rapidly. They're reportedly pursuing an IPO. People still have to choose Claude for their work. If the model makes you spend your afternoon correcting it, that experience is going to affect what you choose to open
[9:14] tomorrow. And Anthropic has acknowledged the issues, right? Anthropic says it listens. Specifically, the 5.5 announcement names communication feedback about Opus 5 as a big issue that they were working to fix, and describes clearer writing and better adherence to writing instructions as one of the top fixes in 5.5. So, for me, 5.5 feels so so much closer to what I loved about 4.6, but of course with so much more intelligence behind it. I can give it direction and get somewhere with the
[9:45] writing really quickly. I can give it direction like I did for Lego and get somewhere really quickly, really efficiently on complex tasks that of course I could never attempt with 4.6. And when we talk about writing, I think we often assume that all AI writing is AI slop. That's not true. Good AI writing preserves your intent. A good revision has to preserve your intent. Suppose you ask for a paragraph to be simpler. The model can make it shorter by removing the complication that made your argument worth reading and then you have to fight with it to put it back. Or
[10:16] it can make the tone warmer by softening a decision you wanted to state firmly. It can make a sentence more confident than you meant by erasing uncertainty that you know you needed to communicate. Those edits can sound polished while making the writing less clear and useful. And this is exactly the line between AI slop and useful writing. And this is why I've often advocated for the difference around writing to be defined by the quality of what you make,
[10:47] not by how you made it. Because in the end, it's either communicated clearly or it's not. And this tool, Opus 5.5, definitely makes it easier to communicate clearly. And this is also why I care so much about steerability and it's one of the things I loved about 5.5. If I want to say, "Hey, keep this, keep the uncertainty, make this part easier to understand." I can just talk into the model and when the correction comes back right away, I don't have to redo it, right? And I found that Opus 5.5 is very steerable in that regard.
[11:17] Think about a revision to the Lego logo that I showed earlier in this video. Suppose I wanted the beanie to be like a tall hat, but I want to keep the glasses exactly as they are. Or I want the assembly to slow down so I can follow it. I want more steps. Those are specific changes. Uh I can change the bricks. I can change the parts list and the instructions to agree with how I want it to look. Maybe I want it to look smoother. I want to make sure that when I give those decisions, every single revision
[11:47] across a complicated 500-piece build reflects my intent. And this is where Claude's visual work gets super interesting. With a scene built from code, it's possible for Claude to go in and specifically change the relevant objects and movements surgically. So, the coding ability of Claude becomes a way to visualize and make the image and animate it and revise it cleanly, and I think that's really special. You can have a conversation about the camera or the shape of the hat and keep developing the same scene together as you figure out what you
[12:18] want. Now, what about overnight jobs? How does this model do? How does Opus 5.5 do on jobs that run overnight? In Anthropic's announcement, Shawn Hanse at Cleo describes leaving Opus 5.5 with an engineering task across six repos unattended for 18 hours. These are things we've heard before, right? And that's a lot of room to get work done. It's also a lot of decisions that the model needs to make autonomously while you're off sleeping, right? Doing something else. And in that world, it's
[12:48] really important for us to come back and make sure that we have good working code. And so far what I've found, and I've done overnight tasks with Opus 5.5 as well. Uh so, this is not just me relying on others. I did two or three overnight tasks as I was testing it early on. What I found was Opus 5.5 does better with those long-running builds when you give it clear stop conditions and a very clear and determined eval or definition of done at the end. The more
[13:19] you're clear about that with a long-running assignment, the better off you are. And that's not a particularly new or even particularly special to 5.5 insight, but I did find it was important to ensure that the the passion that Opus 5.5 seems to bring to a problem doesn't just lead to a really long-running job without a determined stop condition. And so, the lesson I've taken from 5.5 is that it's important to define the universe you want the model to work in. This thing just loves to go and go and go, right? And I love that persistence,
[13:49] and I wanted to attach to the job I gave it so I don't chew tokens unnecessarily. It comes back to efficiency. And so, with a package like this, with the logo LEGO that I built, what I wanted to do was give it a definition of done. And that's why I specified at the top how many pieces, the complexity that this would have to be so that it did not have to do that itself. And that saved me a bunch of tokens. It it enabled the model to not have to model the entire world of my logo hat and arbitrarily decide and
[14:20] then decide it wasn't doing good enough and then push itself harder. These models push themselves. And so, it's up to us to give them a world that they can operate within and be efficient. And then when we're finished, we're finished. The scene works, the changes are there, we've checked the things that need to be checked, the LEGO logo works, right? And I want to connect that to the bigger story because the labs are getting better tools for doing this work as well. And some of those tools are their own AI models, ones you and I haven't seen. We tend to talk about a
[14:50] release as though it came out of nowhere. Here's the model, here's the score, where does it rank? Meanwhile, Anthropic says that as of May, Claude has authored more than 80% of the code being merged into its code base. That matters, right? 80%. So, that's AI helping to design AI, right? It's And we talk about this as recursive self-improvement. We talk about this as AI-assisted research. There's different words, but it tells you how much the work inside the company is starting to
[15:20] change. OpenAI describes researchers using agents to write code, run experiments, and troubleshoot problems. That is a lot that the labs are using AI on to design AI. And this is going to matter, and it's going to help us understand why Opus 5.5 is the way it is. And this matters because the way that we are using AI is actually shaping AI tools like Opus 5.5 that the consumer sees in ways that make it easier for us to use those models. We often think of
[15:50] AI as being less responsive to consumers, but in this case, it's kind of the opposite, and I want to tell that story. We don't have the full internal history of how Anthropic produced 5.5. But what I think is encouraging is the possibility that connecting these dots opens up. Think about it this way. Users like you and me can point out something that makes a model difficult to work with. And the people fixing it have increasingly capable AI tools helping them investigate and test changes
[16:20] rapidly. And that can shorten the wait for really useful improvements. And with 5.5, I can see strides in things like hear about every day, like writing. And I want to look at what those strides let us do. People ask me, "Nate, don't you have a favorite model?" And I say, "I use the models that work, right? I don't pick favorites." Uh and so my job is just to report on what's working, report on cost per task, and be glad that we are living in a world where we have so much intelligence. Here's the part that gets me excited ultimately. Suppose I can get this LEGO model and the build
[16:50] files working with much less effort and fewer tokens as I've been describing. I could stop there, and I could spend less. Or I could go farther and build something else. Maybe I work on the animation, maybe I want a bigger LEGO build. Uh maybe I want to go beyond LEGOs and don't do real work as I as I've been describing. I actually do do real work, not just LEGOs. But ultimately, because it's task efficient, Opus 5.5 leaves me more room to make the decisions that matter, to iterate and then to pick up another project. And
[17:21] when I do that work, it becomes something that makes me feel like the load is lighter on complicated tasks. And that gets at why we use frontier intelligence in the first place. And that is going to be the larger story as we look into the IPOs for both Open AI and Anthropic is ultimately how are they delivering extraordinary frontier value that is task efficient and how are they listening to their paying users and responding in real time. The larger story of Opus 5.5 is that all of that
[17:53] came together in this release and you can see the Anthropic team responding to user feedback. You can see the model coming out and being task efficient and you can see how that pays off in terms of what users can use. Now, every lab has their own journey to go in here. Open AI and Anthropic are in different places. I'm not saying their internal models are nearly the same. They're obviously different. They have different model lineages. It's part of what makes it great as a consumer. Where we are right now is we are living in a world where AI is
[18:24] helping AI development make a real difference for consumers. We have shrunk the approximate number of days between major model releases to about 2 weeks. I think the number I saw recently was about 18 days. Part of how we have gotten that fast and part of how we have gotten a chance to have that much responsiveness to what we need as frontier intelligence users is because the labs have AI tools to help them get better. And so when I look at the
[18:54] future, I'm optimistic. I am hopeful because I'm looking at this and I'm saying, "Wow, what is Anthropic got cooking next? What is Open AI got cooking next?" I bet it's pretty cool. And so here's where we are. We have a model that is good enough in 2026 that I can do that 500 piece Lego build that I showed you, which would have maxed out and broken my Lego benchmark just 3 months ago or 2 months ago. And it only uses 1% of my weekly
[19:25] usage. That's where we are. I can't wait to see where we get in 2 or 3 months. It's going to be so cool. And this is why I think our experience as consumers matters to the larger story. This is why I started with what I actually built with Opus. I wanted to share it. I want you to share it with me. We all have work we need to do. We know which decisions we want preserved. We can explain where the result fell short. We also choose which models get our money, which models get the next assignment. When a model helps us finish efficiently and responds really well to
[19:55] direction, we have a good reason to keep using it. And those choices give the labs a reason to care about the details of working with their systems. Ergonomics of frontier models matter a lot. They matter a ton. And so you might be wondering, "How do I get a useful answer, Nate? How do I find out what my cost per task is? And is Opus helpful to me? How do I know? The with the work I actually do, not with the Legos, with the materials I've got." So, for this one, here's what I would do. I would
[20:25] take the files that you are usually working on. Maybe it's spreadsheets for you. Maybe it's docs for you. Maybe it's a code repo. But whatever it is, run it again. Run a task you ran before when a new model arrives. Try and use the same starting point, use the same prompt, and give the new model the exact same setup the old one already built. And then let it run and see what it does. And make sure you track it just as I tracked the total tokens. And I did a approximate
[20:55] bill count against API costs. And I also measured the weekly usage slice. Do that, right? Test it and see. And then benchmark it against previous models. And and the more you are able to understand the full scope of the task, the attempts that failed, the corrections, the work that happened along the way, how much you had to get involved. One of the things I loved about the LEGO builds is I barely had to get involved at all. Like it was beautiful. It just got done. I got other things done in the meantime. The
[21:25] ultimate result has to meet at least the same standard as your previous model. And the reason I'm talking about Opus 5.5 is it did a lot better than previous Opus models at this. It did better, in my view, than Fable cuz it was so much cheaper and faster. And so I didn't have to keep explaining the same things over and over again. I didn't have to pay as much in terms of weekly usage. It was cheaper to do. It was less guidance. It just it made a lot of sense. And so when you were doing this cost per task work, don't just keep the successes. Keep the moments where you
[21:56] were like, oh, this model didn't do it for me, right? It didn't work for me. This release didn't work for me. The more we're able to document that, it tells all of us what we need to know to actually make use of AI, right? Cuz what we ultimately care about is how often something is usable. How often something is spent across attempts. And then at that point, you can start seeing where the improvement lies. Maybe visual work gets dramatically better and writing improves, uh, and maybe the research side isn't there as much. How much of
[22:28] your day changes on that? And that's going to determine what this release is worth to you. Let me give you an example from my experience. When I used Opus 5.5 and I saw the visual taste that the model had, I actually switched where I was going for visual taste and design work on a project I'm working on and I gave it to Opus 5.5 because I liked how Opus 5.5 handled the visuals and I trusted that a complicated, it was like 90-some element
[22:58] visual design job was something that I could hand with one prompt to Opus 5.5 and I and I knew it would get it done and it did and it did not chew up my weekly budget. And so, I basically reasoned from what I could learn and test and changed where my workflow went as a result. I'm doing that all the time. And so, for you, my challenge to you is make sure you build a real assignment testing kit for models. If you haven't done that, yes, of course, I
[23:28] can help you with that. I have a whole prompt to work with your model to look at your previous usage patterns and to come up with a realistic just for you prompt that is not a Lego bench, right? That is not whatever Nate's design files were. It is for you. And so, with that, if that's something you're interested in, I'll put the link, you can go get it. That's fine. But ultimately, I want you to see the larger story here. We are in a world where we can give feedback to the labs, they can respond, we can see better models coming out, and we can get
[24:00] more done. And the only way we know if that works for us is if we are actually tracking our tasks and making sure that those tasks actually get more value back. So, pick something where you're going to care if the result is good, measure it, and come back and measure it again. Make sure that you test it consistently across model releases. That's how you get excited. That's how you know when a breakthrough has happened. That is my take on Opus 5.5. I'd be curious for your take on Opus 5.5. Let me know in the comments.