Nate B Jones
¿Cuándo me preparará la IA unos huevos revueltos? Fui a NVIDIA para averiguarlo.
🇬🇧 EN🇪🇸 ES
Pregúntale al tutor sobre este vídeo
Entra con un email verificado y pregúntale al tutor lo que quieras sobre este vídeo.
TL;DR
Resumen de investigación
▶ Presentación de Cosmos y trayectoria de Mu
El entrevistador plantea que, según la conversación, podría llegar antes un robot doméstico capaz de lavar y doblar ropa correctamente que uno capaz de preparar huevos revueltos perfectos; Mu responde que cree que sí, y destaca que no está claro que el modelo conozca la física. Presenta la entrevista desde NVIDIA y explica que un modelo del mundo puede producir acciones para un robot, en vez de limitarse a redactar instrucciones. Mu se identifica como vicepresidente de Cosmos Lab y cuenta que, hace unos diez años, defendió que el contenido visual acabaría generándose con aprendizaje profundo, una idea controvertida cuando los resultados tenían baja resolución y figuras humanas deficientes. Describe NVIDIA como una empresa que ofrece a desarrolladores GPU, CUDA, bibliotecas y modelos, además de CPU y redes, para ayudarles a crear productos y aprender qué arquitectura de computación hará falta. La conversación introduce el tema de cómo los modelos del mundo difieren de los modelos de lenguaje.
▶ Qué es un modelo del mundo y para qué sirve Cosmos
Mu sostiene que las personas forman modelos del mundo antes de adquirir lenguaje. Describe los modelos del mundo como un conjunto de modelos para tareas concretas, por ejemplo explicar por qué ocurre algo, predecir lo siguiente o navegar en el espacio. Cosmos busca ayudar a desarrolladores de IA física a crear mejores vehículos autónomos, robots y sistemas para coordinar máquinas en fábricas. Entre las capacidades necesarias incluye comprender escenas, como comprobar si un trabajador sigue un protocolo de seguridad o detectar defectos en una línea de producción. También plantea usar simulación para probar políticas de conducción y robótica antes del despliegue, dado que las pruebas con flotas reales consumen tiempo y no cubren fácilmente todos los escenarios. Según Mu, durante los aproximadamente 1,5 años posteriores al lanzamiento de Cosmos fueron desarrollando modelos para distintas necesidades y, en una presentación de junio en Taipei, anunciaron una suite que reúne comprensión, simulación y capacidad de acción en un solo modelo, aprovechando representaciones compartidas del mundo.
▶ Modelos en tiempo real y distintas formas de escalar
Ante la necesidad de que un robot reaccione en tiempo real, Mu dice que Cosmos 3 tiene tres tamaños —los nombres de dos aparecen transcritos como «super» y «nano», y el menor como «h»— y que buscan que puedan ejecutarse en dispositivos como Thor o DGX Spark. Los modelos pequeños son menos capaces; el modelo «super» está destinado al rendimiento de frontera y puede funcionar en centros de datos u ordenadores más potentes. Sugiere combinar modelos integrados para navegación inmediata con otros en centros de datos para razonamiento de orden superior que pueda esperar. Sobre el escalado, enumera más datos y mayor tamaño del modelo, y añade el escalado en tiempo de prueba: una descripción más detallada puede mejorar el contenido visual generado. Como ejemplo, se podría especificar el movimiento de un brazo robótico segundo a segundo. Explica que un razonador puede ampliar gradualmente la descripción de una simulación y que varios modelos dentro de un sistema pueden corregir el resultado de forma iterativa. Resume estas dimensiones como escalado de datos, del modelo, del tiempo de prueba y de agentes.
▶ Datos visuales, física y arquitectura de Cosmos 3
El entrevistador observa que el mundo físico parece ofrecer muchos datos, y Mu matiza que el lenguaje es conciso y semánticamente significativo, mientras que en lo visual puede haber oclusiones, cambios cuyas causas no se ven y señales de mayor dimensión. Aprender fenómenos como la dispersión causada por el viento es difícil. Mu afirma que un aumento de datos puede mejorar la simulación de aspectos físicos, pero que también recurren a simuladores de física —menciona herramientas cuyos nombres se transcriben como «Universe» e «Isaac»— para generar datos sintéticos. Añade que la red neuronal sigue siendo una caja negra: no puede afirmar que conozca la física, aunque intente aproximar mediante datos las reglas que gobiernan lo observado. El entrevistador usa la caída de una botella y el valor de 9,8 m/s² como ejemplo; Mu acepta que los motores físicos pueden aportar interacciones newtonianas difíciles de encontrar en datos reales. Describe Cosmos 3 como una arquitectura de tipo transformador con un componente de razonamiento lingüístico, un generador de video de simulación y acciones. Mu señala que las acciones importan porque causan cambios observables y pueden ser la salida necesaria para controlar un robot. Considera que un razonador podría llegar a revelar qué regularidades físicas aprendió, pero dice que aún no han llegado a ese punto y que no cree que el modelo haya descubierto explícitamente las leyes de la física.
▶ Aplicaciones: verificar políticas y controlar dispositivos
Mu describe dos aplicaciones: usar Cosmos para desplegar simulaciones y puntuar la tasa de éxito de políticas de conducción, ayudando a seleccionar qué puntos de control probar con una flota limitada; y usarlo como modelo de política, con resultados de recoger y colocar objetos que NVIDIA presentó al lanzar Cosmos. Explica que un modelo de política busca controlar un dispositivo físico a partir de instrucciones y observaciones visuales. En el ejemplo de un coche que puede girar a la izquierda, la política genera una trayectoria de puntos y, si aparece otro vehículo, toma nueva información visual y predice otra trayectoria. El entrevistador destaca que la salida es una acción espacial y temporal, no solo texto. Para un sistema físico, Mu describe un harness como una orquestación de modelos y herramientas: incluiría robots reales, quizá herramientas de calibración de cámaras, un modelo de política que produce acciones y un inspector visual que comprueba si se completó la tarea. Ambos señalan que hay más similitudes entre modelos del mundo y modelos de lenguaje de las que el entrevistador esperaba.
▶ IA con agentes y especialización para el mundo físico
Mu compara modelos autoregresivos, que generan secuencias paso a paso, con modelos de difusión, que eliminan ruido gradualmente, y destaca que los sistemas con agentes pueden dividir problemas difíciles en subproblemas y usar modelos y herramientas en varios pasos. Considera que esa estrategia también será importante para la IA física. Espera una transición hacia agentes físicos capaces de controlar robots y usar herramientas digitales relacionadas, aunque cree que la separación entre agentes digitales y físicos se irá difuminando. A la vez, sostiene que persistirá la especialización: la eficiencia, el coste y las limitaciones de cómputo importan, porque un robot no necesita todas las capacidades y no es viable poner un ordenador enorme en un dispositivo pequeño. Por eso prevé arquitecturas distintas para diferentes aplicaciones; defiende GPU programables que puedan servir tanto para entrenamiento como para inferencia. Concluye que todo tenderá a ser agéntico, pero que la especialización determinará las necesidades de arquitectura y cómputo. El entrevistador propone que las habilidades especializadas de los agentes de software podrían tener un equivalente físico para que un robot aprenda tareas como cocinar o doblar ropa.
▶ Habilidades físicas, simulación y recompensas verificables
Mu acepta la idea de habilidades físicas y sugiere que la experiencia de un robot, incluidos los registros de intentos y resultados, podría procesarse para ajustar cómo realiza una tarea. Dice que los archivos de habilidades equivalentes a los del software aún están por llegar, aunque hay trabajos iniciales. La diferencia clave, según él, es que las pruebas digitales se pueden ejecutar rápidamente en un ordenador, mientras que verificar IA física es más difícil. Por eso quieren construir modelos del mundo: en vez de crear físicamente mil cocinas distintas, se podrían simular muchas y desarrollar a escala mediante la interacción del agente con el modelo. Sobre la generalización en hogares diversos, Mu afirma que los datos y las recompensas verificables son retos importantes. Algunas tareas son difíciles de evaluar —por ejemplo, si la comida está buena, puede requerir que la pruebe una persona—, mientras que doblar ropa puede comprobarse visualmente frente a una instrucción. También compara insertar un cable en un equipo de centro de datos, que podría verificarse con pruebas de software. Sostiene que las tareas cuya recompensa pueda verificarse en simulación avanzarán más rápido, y coincide en que probablemente se consiga antes un robot que doble bien la ropa que uno que prepare huevos revueltos perfectos.
▶ Familias de modelos y apuesta por la apertura
Al preguntarle por las familias de modelos y los planes de NVIDIA, Mu afirma que distintas aplicaciones requieren especialización. Enumera modelos para lenguaje y programación, Cosmos para modelos del mundo, otros para robótica humanoide y conducción autónoma, y un modelo para atención sanitaria; algunos nombres están deformados en la transcripción. Defiende la diversidad de modelos y la apertura. Dice que una API puede ayudar a resolver un problema, pero deja oculto lo que hay detrás; recuerda que aprendió de código abierto como estudiante y atribuye a la apertura oportunidades de innovación y de comprender el funcionamiento de los sistemas. Presenta el apoyo a desarrolladores y la disponibilidad de herramientas y modelos abiertos como parte de la orientación de NVIDIA.
▶ Predicción sobre Astra y cierre
Ante la pregunta sobre qué podría superar las expectativas en los próximos doce meses, Mu dice que, si hubieran hablado dos semanas antes, quizá habría mencionado lo logrado por Astra. Describe un ciclo en el que se usa Blender para crear un espacio 3D, se proyecta y se comprueba si coincide con la descripción; según él, esa recompensa verificable permite desarrollar capacidades con rapidez. Considera que las escenas creadas por Astra son sorprendentes y que una próxima sorpresa podría surgir al conectar estas ideas con la IA física y el uso de herramientas físicas. Plantea que alguien podría encontrar una recompensa verificable que ya sea posible pero que no se haya aprovechado todavía, y activar así nuevas capacidades. El entrevistador comenta que investigaciones publicadas pueden difundirse y permitir que otros aprendan. Mu expresa que difundir el conocimiento y habilitar a cada persona es importante para mejorar el mundo; cierran agradeciéndose la conversación.
Transcripción completa
[0:00] So, so we're saying that it's it's going to be faster to have a robot in my house that will do the laundry and fold it correctly than one that will do perfect scrambled eggs. >> I believe so. It's hard to to say that it knows the physics. Um, you know, we actually don't know if you know the phys. But instead of creating a thousand different kitchen, you can quickly create a thousand different kitchen in the war model. As a uh enter the physical world, you'll see more and more policy models. Today's conversation is about world models. I'm here at NVIDIA.
[0:31] People know NVIDIA because of chips, but they actually do a ton of model development as well. I'll be chatting with Ming Yu, who is the head of the Cosmos world model family. For those of you that don't know, a world model is a model that produces actions the way LLM produce text. And so for example, if I have a robot arm, I could tell the robot arm to take an action like picking up the water bottle and that is a direct
[1:01] action that's produced. It's not like it's writing text that says pick up the water bottle. It actually is issuing guidance to a robot to do that. World models are already around us. I've got a little robot lawn mower named Mallister and it ms my lawn. It has a tiny world model inside it and it just outputs where that little robot lawnmower goes next on my yard. That is just a little bit of what I dove into today. So, I'm really excited to chat. Maybe you start by introducing yourself. Tell us about who you are, what you're doing at NVIDIA, all of that. >> Yeah. So, my name is Mu uh and the vice
[1:34] president of uh Cosmos Lab. Uh I lead uh Cosmos Lab uh for Cosmo platform development. uh and you know I start my career uh as a geni researcher. Uh I was the first group of people uh who believe that uh uh the future visual content uh will be generated by deep learning. >> But 10 years ago when I did a job
[2:06] interview here I told our chief scientist that I know today we are using computer graphics to generate visual content. Mhm. >> But in the future it going to be using AI. >> Uh and it was quite controversial at the time because >> you know the image could be generated by AI model at the time was so limited resolution is small and often the generated human look like zombie. uh we just I just feel like uh you know this
[2:37] is such a a nice uh paradigm uh put all the problem into a learning problem and you don't need to have all kind of handd designigned rules to create a 3D world just datadriven and create uh as long as this uh uh paradigm scale uh then visual content will be created right uh through AI and you know in the last 10 year we have see uh we have seen tremendous progress in the field like gradually one
[3:08] by one >> and now I I think most of the I would say you know that's growing maybe exponentially the amounts of content uh in the uh vure uh is generated by AI now day yeah >> super exponentially it's growing super >> yeah yeah exactly >> so you were early so 10 years ago you saw this vision that AI was going to be a big part of video creation now. You're in charge of the sort of the Cosmos family and and where that's going.
[3:39] >> I think everyone assumes that Nvidia is about GPUs because that's all we hear, right? >> Um, but I I learned that we have more software engineers at NVIDIA than you have hardware engineers. And so it's a software company, too. There was the Hugging Face acquisition recently. Tell me a little bit about sort of within the NVIDIA family. What's it like to be focused specifically on I guess we could call it a world model. Is that a fair way to describe it? >> Yes. World model. Yeah. So, uh media is
[4:09] a developer company. >> So, we build our computers to help developers build uh amazing product to serve their customers to you know and developers often need more than just the GPUs, right? uh you also need a CUDA so that you can better program our GPU to extract them all the muscle powers from our GPU right and there are also uh libraries we build on top uh and also like a AI model we build on top so we
[4:41] are kind of building layers of software on top our computers now our computer including GPU CPU and networking devices connect them together uh so that we can better serve our developers >> uh you know we providing tools so they can kind of save some energies and move faster. Imagine they may figure out way to build better tools than what we provided right but also through building these tools we better understand what kind of computer architecture we need to have for the future challenge. So we are
[5:13] building uh we are kind of building vertical to better uh knowing how to build our kind of computers uh to serve the whole ecosystem. No, that makes a lot of sense and I think that focus on the customer is something I've certainly seen talking with uh NVIDIA engineers in other contexts as well. So it seems to go through the whole company. Certainly heard it from Jensen. >> You better go first. >> Yeah. Yeah. Exactly. Uh so when we talk about world models and Cosmos, I think a
[5:43] lot of us come to the table around AI at least after chat GPT and we think in terms of tokens, we think in terms of LLMs. >> Uh world models are a different game. So can you introduce for us like what are world models? How do we get to this point with world models? Uh and what is cosmos like? >> Yeah. So um people have the some model of the world before they have language. >> Mhm. >> Okay. >> I have a one-year-old so I I know this. [laughter] >> So uh and uh so language um is very
[6:18] nice. Uh it's symbolic. Uh it's semantic. uh it's a uh document how people think about everything right so uh when GBD uh I think it's the first major AI breakthrough uh it makes sense it happened in the language domain so uh enormous amount of information there encapsulate uh and learn from how human deal with scen and then can start have chat with you now can help you with the
[6:48] code generation review right so what model is from a different perspective, right? So, uh people build war model uh try to accomplish different task >> try to understand for example why this thing happen >> or to predict what going to happen next >> or or have a map so that they can navigate in the space. So I I think one model is actually a set of models not
[7:19] just one model. So you build a model for some purpose. >> Yeah. >> And you live in the war and then allow scene you want to achieve you need some model of the war. Mhm. >> So then we have different kind of war model and the war model uh in this Cosmos effort uh this Cosmo platform that we want to build is to help physical AI developers to build better self-driving car, better robots, better way to orchestrate different machines uh
[7:51] in the factory. >> Yes, >> because we have our goals set for physical AI in these important verticals. So we then start to think about what kind of war model best position uh for the to to address the need of developers in these domains. And from there we realized okay we need a war understanding uh be able to explain what happened like whether this worker is following the safety protocol uh to operate this machine uh based on the
[8:21] handbook or you know is is this in this production line whether there's a defat uh from the visual inspection >> course. Yeah. Oh, self-driving car, right? They need to um so to build self-driving car, once you have a policy model, how you drive the car, >> you need to have a way to verify this policy model work, you know, uh safely, you know, accurately. Right. That's right. >> And one way is to, you know, have a fleet of cars and driver and go out to
[8:53] do the testing. And you can see it's pretty time consuming and some of the scenarios and you could not easily hit. So you your test may not be complete but if you can create the war model you know something that the self-driving car would perceive right and and now you you just have your policy in touch with the war model and you can know it's uh its performance it's accuracy it's safety you know before you deploy the the car in the face. There are also kind of
[9:23] simulation kind of war model that uh help uh needed for physical air and I gave example on uh self-driving car but robotics humanoid robot it's the same you need to be able to test you need to have some kind of proxy of the war so you can do the testing so then there's a war understanding part there's a war simulation part these are uh since we identify critical for physical AI and over the past 1.5 years um you know
[9:55] since the uh launch of Cosmos we've been gradually building uh model for each blend and in June uh this uh uh comput uh you know Jensen's uh keynote uh in Taipei uh we released this cosmos suite where we fuse world understanding and world simulation and also war action capability into a single model uh so that uh you know we can put that into a single model because even though they
[10:25] are for different purpose but they model the same world right so then there's a underlying um representation they could be shared among them so we put them together what we are thinking about war model you know is for physical AI and we identify the key scene they needed and we try to bring them uh together so that we have a a a simple way for developer to leverage uh to solve their problem. >> And I think you know when I think about
[10:55] world models, one of the things that's really intriguing to me and I think you got it at the beginning of of your explanation, you talked about this idea that we as humans develop effectively a world model even before we can speak because we're modeling the rooms we walk around in as babies, etc. Um, and that resonates for me when we think about robotics and where we are with a robotics moment. And you talked about the simulation piece a bit later. Like if I am trying to navigate a factory floor,
[11:26] >> it becomes very useful to be able to model that factory floor and do that in simulation with, you know, the forklift coming across or the person walking across and how do you handle all of the the the pallet falls over? How do you handle all these edge cases? One of the things that stands out though as I think through that scenario is that Cosmos would need to be able to be deployed so that it is very very real time and responsive in order to enable the robot to navigate safely and to work in real
[11:56] space safely. How has that understanding of the need to operate in real time shaped the way you've architected the model? >> We build model of different sizes. >> Uh so uh in Cosmos 3 we actually have three different sizes. um super nano and h uh h is the smallest one. Okay. Uh and we build it in a way that it can run real time on each device such as uh media just thor or dig spark.
[12:28] We want one model lip where robot lips. >> So there's a smaller model of course of course smaller model is uh less capable compared to larger model. We also have uh the largest one in Cosmos 3 is the super model. That one is for frontier performance and that one may live in data center or you know uh more powerful uh computer right but some of the navigation test yes um you have to be real time but there's some higher order
[12:59] reasoning task you you can wait you can be patient true yeah >> so I I think a mix of models of different sizes and some live in embedded device some live in data center might be a good combination to address the need in physical AI. >> That makes a lot of sense. Like I could I could see different use cases popping up. You know, one of the things when we talk about the scale of LLM is we talk a lot about scaling laws, right? This idea that test time inference is something that that enables us to scale the longer
[13:29] you think the more you scale uh you know scale of parameters etc. Do you have equivalents on the world model side where there's sort of scaling laws for world models? >> Yes. So uh scaling uh happen in multiple aspects right. So data scaling more data yes uh model size scaling yes bigger model uh better quality >> it also interesting part is that uh well for the uh visual content generation simulation part of the war model the
[14:01] scaling can the model uh there's also test scaling that's uh let me explain uh more concretely so we know u If the description test description of the visual content uh that you want uh to generate is more detailed then the visual content tend to be better right. So you can think that u if you can you can do loom kind of test
[14:33] scaling you keep reason thinking describing what is the scene you want to generate. >> Yeah. And then you have more like a detail. I mean it trinkets. I'm not saying we do that but you can say hey this pixel should be uh a 25 100 you know RGB color. >> I just >> as an example >> as simple. So but basically if you get very detailed description uh you can get very high quality video output right you
[15:03] can say hey uh from the first second to second uh the robot is doing this and the ham is moving 30° and next second another 10° right so this detailed description give uh very good generation result and this is one kind of 10 time scaling by by sinking longer uh describing more and you get a better result, right? Yeah. So, >> so I just uh I'm a coffee nerd, so I
[15:33] think of it as like the robot arm is pouring the coffee into the ChemX. Right. In that world, I think what you're saying is you can invest more in the fidelity of the labeling of the motion in real space. And if you invest in very high fidelity labeling across that space so that you know where it's moving exactly in the world as it pours and all of that then you have a much richer uh data set to reason across and you can replicate that motion. You can also scale up and
[16:03] infer from that into other novel motions etc. >> Yeah. Yeah. >> Got it. So, so that's uh a more detailed motion description happen in the test time >> as the so the the way we do uh simulation is uh uh we have a uh reasoner yeah >> uh to uh based on the the instruction and gradually expand what uh need to be >> uh contained in the simulation output
[16:34] >> and that is the sinking process that you keep generates the description like what you say earlier the the motion >> that's right yeah so I and I didn't want to communicate it's a bunch of humans doing that like the model itself is doing that yeah exactly >> and in this agentic war right so maybe you don't achieve this in one shot you can multiple uh model code inside a harness that you fix the issue and then they say um so I think the robot need to be three fingers left
[17:04] you know like this kind of correction so um So the scaling can happen in many different dimensions, right? Data scaling, model side scaling and test side scaling and also the agent scaling. That's what I would say >> that and that's super interesting to me. It gets at something that I think and maybe I want to suggest it to you and see how you feel about it. But it feels like one of the underlying differences when we talk about LLM versus world models is that if you get into the world model space, the data availability is very very rich. Um whereas with LLMs we
[17:36] talk about the idea that we've sort of sucked up the whole internet and now we have to like scale with sort of artificial data etc. But I look at the world around me there's rich data everywhere from a world model perspective and so it feels like it's a richer data environment. So uh in some sense yes but also um language is very concise uh semantically meaningful right and you see everything in the text >> in the visual world there are a lot of thing you don't see right the precise
[18:07] action caused to this change and there's occlusion that that's a >> allow observability and and you are also dealing with a more like a raw signal not like in language you be more symbolic. >> True. >> The things you need to model are more than what I I shouldn't say more because language also it's a challenge but I want to say uh even though you have a tons of data but the dimension of the output is much larger and the scene you want to capture is more.
[18:37] >> Yeah. >> Yeah. Yeah. Like like >> you need to learn the physics right. uh how uh when you have wind blower to blow something and how it's spread out those things are not easy. >> No, they're very difficult. Do you find I think this is one of the really interesting implications when you start to scale up these models. Do you find uh to a certain degree an emergent understanding of physics as the models start to understand things better? So uh we see that when the data uh size
[19:11] increase uh the the capture of the the the uh the physes uh it can simulate improves right so and but often we found that we need to leverage uh like a phys simulator like universe or isac that's model uh that's a you know built-in engine uh use those phys engine to generate synthetic data and use them to improve
[19:42] the models physics uh performance and I want to say that um um after so many uh years of um um let me put this way so many years of people putting tons of effort into the deep learning neuron network >> um 2010 remain a black box to us so it's hard to uh to say that it knows the physics. uh you know we actually don't know you know the phys but uh to some
[20:12] extent try to approximate uh the rule governing the data gen uh you observe uh and and the we know the rule behind uh what we see you know is governed by physics right so try to approximate that part but indirectly through data yeah >> so I'm going to give an example and make sure that we understand for the audience here so if I take this water bottle here and I were to drop it would accelerate at 9.8 m/s squared. >> We know that from equations. And so I
[20:43] think what you're saying is you can do that and have the model look at a lot of video of that and look at a lot of images of that and it starts to approximate the appearance of that falling pattern, that acceleration pattern, but you can also accelerate that understanding by giving it a physics engine to tap into during training. >> Yeah. >> Like omniverse where it starts to sort of dig into that. Does that >> Yeah, that's what I'm saying. Yeah. So um like a a bow hitting another bow. >> Yeah. >> You know this kind of data may not be easily available you know and so uh use
[21:15] f engine to simulate this uh Newtonian motion uh interaction between object and help. Yeah. And so uh as we are talking I've been thinking about one way you know let's let's think about this right. So let's say we have a maybe I let me briefly introduce the Cosmos 3 architecture and we come to my point earlier. >> Uh Cosmos 3 is a mixture of transformer architecture. >> Okay. >> So we have a reason to uh sync in
[21:46] language based on YC sync in language. >> Oh people don't know that. So you have a language component as well in there. Okay. >> Syncing in and we also have a generator. >> Okay. uh based on the uh thinking the representation learned in the listener and you generate uh you know simulation video [clears throat] >> you can do symbolic reasoning over here in the language >> symbolic and this uh video and action we also have action in right so I want to mention action is important because uh what you observe in the world is the
[22:18] deformation caused by some action taken >> yes >> and in robotics uh sometimes your desired output is action So that's what action to take to >> and you don't want to send tokens to do that. You just want to say here's the action that you should take and move the servos. >> Yeah. Yeah. So it's a everything in the neural network is represented a token a sole experiment right >> whether these uh um war model can learn physes right so uh if we keep um if we keep uh dropping both yeah uh okay uh
[22:50] and we didn't uh assuming the uh the reasoner doesn't know about the law of physics right and but it keep observing that can we use some kind of uh uh sudis learning can be uh sometime somehow figure out that the reasoner need to come out a precise 9.8 a the coefficient uh and to model this and then precisely describe how housing going to drop right if that happen uh I think I can more
[23:20] confidently claim that's uh uh this war model and the phys but I think in the reality um yeah I think we haven't reached that that uh state uh we are uh the model yes >> I don't think it's fig out yet u but this reasoning we may have a a way to look into what the model truly learned. >> So I I think if I understand correctly, we obviously have the mimicking now from the cosmos model family. Uh and you see
[23:51] a path in the future with the reasoning tower there to start to understand how the model is processing and reasoning across this information. So we can eventually start to see to what degree does the model really understand physics. That that makes sense. more like uh uh recent is more like the um you know also the same architecture that people use for coding. >> Yes. >> Like like you have a tons of observation from the reward video. >> Yeah. >> Can you see this? Can you learn the
[24:22] program? >> Yeah. >> Yeah. And to govern TypeScript, can you learn JavaScript? Yeah. >> Uh more like the phys description, right? how this particle is moving and so they pass away I believe but I don't think we are there. >> Yeah, that makes sense. >> But what's interesting to me is even though we're not there, the mimicking is good enough that we're already seeing applications for these world models. Um can you share some of your favorite applications for the Cosmos model family? >> Yeah, so um so there are a couple of uh
[24:52] applications. One is uh for policy verification. >> Okay. >> Right. So you have a a driving policy uh like how do you know it's good enough you can use Cosmo model to uh to roll out and give a score the success rates for each uh policy and from there you can >> um you know use it u you know for your selection and you can limit your limited flee to those more promising checkpoints the policy
[25:22] >> and Cosmo can also be used as a policy model. uh world aure model. Uh so when we release Cosmo, we show that it achieved very good uh pick and place results. >> Yeah. So so because this model is an only model is on model on this modality text video and action. >> So when you have action out it's a policy model and because it's learn with the world dynamics modeling language modeling. So you can follow the instruction and see the dynamics and
[25:53] predict the right action to come the task. Yeah. And and I think people are finding more application of uh of our model. Yeah. >> Just so we kind of dig in a bit. I think policy is something that people aren't used to hearing when they hear about AI and LM is probably a new term. >> When we talk about policy, how would you describe policy >> to someone who may not be deep in AI? >> Okay. So passing model is more like a uh an AI uh with the intention uh to
[26:24] control a physical device. Got it. >> What kind of uh command you would given based on the test given and the visual observation um presented. >> So like one example would be if I'm in the self-driving car and the green arrow shows I'm allowed to turn left and the policy says that I can turn left. The policy will give you the the way points the trajectory. >> Yeah. And it's like you turn left by following these dots along the road. Yeah. >> And it's a uh is streaming. So it keep
[26:54] generating because as you make a turn there might be another car try to cut and there's a new >> and now you have to redirect the policy. >> The time policy will take the visual input and predict a new new trajectory. >> Got it. Got it. And this is what we mean by action. You're literally there's a vector in space and time. >> Yeah. that you are predicting and directing against. >> Yes. >> Which is super cool. >> And as much as that sounds like that would be compute inensive and impossible to do in real life, you're driving around them right now. Like they're already here.
[27:24] >> As uh AI enter the physical world, you'll see more and more policy models. >> We will. And I think part of why I wanted to have this conversation is it feels like it's important for us to start to think about it more broadly as as a community. Um >> you mentioned harness earlier. >> Yeah. We're we're a little bit familiar with LLMs and harnesses now. Um what does a harness look like for a world model? How is it different? >> So uh allow similarity. Um so uh I think LN harness uh you know it has you have a
[27:56] also have a policy in the harness right and the policy there is about >> uh the security uh permission you know what kind of resources you can do you're allowed to hack into other people's [laughter] computer those different thing right but harness is you know let's uh take the software piece out uh harness is orchestration of model and also tool use right u and so I think the harness for physical will be very similar but the tool will be very
[28:28] different >> uh it will be the real robots u uh and maybe you need some camera calibration tool >> uh and the model uh might just might generate might be a policy model uh that generate action to control the robot >> and there might be also a visual inspector the model can be also a visual impacttor to measure whether the robot completed the task. >> One of my big takeaways and surprises is I learned about more similarities between world models and LLMs than I
[28:58] realized before. >> So that you know the thing I I like about agentic AI is that um you know uh as a AI researcher uh I've been in this field for a long time. We studied many different AI models. >> Yes. And now two mainstream uh two two kind of models uh are mainstream uh GBD auto request style l next to competition and division model you remove noise
[29:28] >> gradually one at a time. So these two model has the nature have the characteristic of divide a complicated problem into small problems and you solve one at a time right one token at a time instead of gening the whole paragraph >> that's like a diffusion model generates the whole experience at once whereas this is just one token at a time >> so diffusion gen the whole experience through gradually remove the noise. >> Oh so so okay there you go. Yeah, because that's how it works, right? You
[29:58] start with raw noise and you have a seed and then you start to like >> Yeah. So you do multiple step. What I like about HD AI is it's trying to solve a challenging problem through multiple model codes and tool codes. It had a characteristic of divide a challenging problem into small problems and execute one by one. And what's more exciting is that uh in ln or division model uh all the stop problem is homogeneous. It's the same nest token prediction problem
[30:30] is the same like a noise removal problem. But in LM sometimes you do the model code and sometimes you give different system problem to the model so that it achieve different task and this way of divide a changing problem into small uh easier problem. I think it's a very scalable approach based on what we have observe. Yeah. >> Yeah. No, that makes a lot of sense >> and and so so so that why I believe that um agent AI going to be also critical
[31:00] for physical AI. You have challenging problem in physical AI and it makes sense to solve it u in some problems and through some kind of uh more task specific graph execution. >> So and this is really where I wanted to go. you're anticipating me. So you're saying like we have two basic ways to solve problems. We can either do token prediction and token architecture or we can sort of go with the diffusion space where you're taking away pieces methodically as you go and then you have the diffusion space.
[31:30] >> Um and that that ability to decompose the problem is scalable both in terms of agents that we would operate in software but also in the robotic and physical space. And I got really excited about that because when you think about it, the the agents that we use in software space, like if you think about the agents you have on your machine, you have skuomorphic sort of designs that evoke robots. Like we think of these agents as little robots in our machine. So you'll have
[32:00] the little like robot face and little antenna, all of that. And we chuckle about it and we laugh and it's software design. >> But I think it gets at something deeper. I think it gets at this idea that we have a mental model of the robotics world that is tied closely to this idea of agents that operate and are able to manipulate, >> you know, bits in the case of software, but atoms in the case of hardware. >> Um, and so maybe, you know, talk to me a little bit about how you see agents and robotics in this world with Cosmos and
[32:30] kind of what what the next year or so might look like for use cases. So I think uh so in LM we have see um now gradual and you know more transition to agent world. >> Mhm. >> I think the same thing going to happen uh in physical AI. >> Oh that's super interesting. It going to be Yeah. So, uh you're going to have uh uh the physical agent make took control the robot and maybe some other digital
[33:00] tool uh suitable for this physical attack. I think going to happen and um and I think the the line between these uh will I mean will gradually be blurry. you know they they are a big let me put this way uh we still believe we require specialization right so uh in the end uh um with the uh development of AI when AI model all become very kind of powerful
[33:32] efficiency will become a major uh selection criteria you know >> it already is on the software side >> it's yes and you have this capacity Um you know they say the model size capacity uh there are so much you can pack. >> Yes. >> Right. Uh if you're a robot never need to solve Olympia mass problem. >> Mhm. >> Um it might be good in the earlier you provide some training. Right. But later you probably don't want to waste your
[34:02] capital this deal. And also the infant shape will be different right there. Think about um when s go to the data center >> um you can batch uh the API requests from different user >> uh and solve it one shot. Uh so we know LM is more like a memory bound right and but when you pack in you can leverage the uh the compute layer >> but in the field when your robot and trying to execute
[34:34] >> um you cannot wait right to so so it means that the the compute uh need to happen on different application going to be different. >> Yeah. And because of the computer difference I think there will be a v variety of different model architecture design right they all will share some similarity but they all have some specialization. So then get back to your question. I think the whole idea of dividing a changing problem into easier
[35:06] sub problem >> uh is universal right scales. >> Yeah, it it it scale but uh in the end I think you can if you don't care about cost uh you probably can have a universal one to maybe solve all of them >> but in the end you you're going to cost going to matter. You cannot put a superco computer inside a tiny >> no one will do it. Yeah. >> So then because of this uh there will be specialization in architecture and because there are so many different applications so they're going to have
[35:36] many different uh stuff >> uh and that's why media believe in GPU right so you need a programmability you need a GPU um you know be training uh the GPU have to be funible right when you change the architecture you can still train and when you change the architecture can do inference I think those are >> you know pretty exciting So go back to your question. >> Um I think that uh everything going to be agentic >> uh specialization matters
[36:07] >> and specialization will trigger different architecture and compute need. One of the things that we've started to see uh certainly in the last few months has been this idea that you can have um a fairly general purpose intelligence on your computer uh something from the Claude family, something from the OpenAI family, whatever. Now the open source family with hugging face >> um and you can start to give it as part of your harness specialized skills. >> Um and the skills are sort of like recipe cards. They enable the agent to
[36:38] learn very quickly to do something reliably. One of the things I've been thinking about as we talk about agents in robotics is this idea that you might have a capacity envelope, a general form factor for a robot, but you would have perhaps something analogous to what we call skills in software that would enable that robot to go from cooking an egg to folding your laundry. >> Yeah. >> Is that something is is that a a good way to think about it? >> Yes. It's a physicalized skill, right? The
[37:08] the difference is that uh uh instead of keep trying this uh uh digital tool uh and pass the log uh you keep scramble the act and you may have a >> uh you know also a lock need to be you know process hey hey when I scramble I find something and later maybe I need to do something differently. I think so. Yeah I could totally agree with you. Do does like at this point in sort of where we are with physical AI do we have the equivalent of the skill.mmarkdown file
[37:40] that we have with software or is that still to come? I >> think it's still to come. I think uh we have see early work uh trying to bring this in uh and but I I I the major difference is in digital AI verification testing can happen very fast. You just need a computer. In physical AI, it's totally a different story, right? >> It really is. It's hard. >> Yeah, it's hard. And that's also why we want to build uh war model because if we
[38:10] can uh have a good policy of the wall and the robot interact with the wall >> uh then you can trade the compute uh for for development philos velocity, right? So instead of creating a thousand different kitchen and you now just have your uh physical agent interact at the wall model uh then you can quickly create a thousand different kitchen in the war model and you know you gain scale from from that interaction. Yeah
[38:40] >> maybe we can touch on that a little bit. So we we were talking actually I think just before the camera started rolling about this idea of verifiable domains and less verifiable domains. Um, when I think about robotics and kind of the real world that we navigate, a lot of those domains outside of manufacturing, especially in the home, are not super verifiable. Everyone's kitchen layout is really different. Everyone's bedroom is different where they fold the laundry and put it away. And you think that's chaotic. Wait till you're in a, you know, a bedroom folding laundry or
[39:11] you're in the kitchen trying to cook eggs. um how do you think about the way you get to generalizable intelligence in those physical spaces? >> Yeah. So uh we we know that uh if we can have verifiable uh reward uh and development going to be super fast >> coding mass exactly okay and the physical AI the challenging is data is a challenging getting verifiable reward is
[39:41] another big challenge probably even bigger right so I'm cooking something right uh maybe I can simulate everything but in the end whether the food is good or bad. You need to be consumed by a human. >> That's right. >> It's very difficult. [snorts] Uh but I think there are things that easier uh to verify. Right. For example, I I want to fall laundry >> uh and uh then uh I can look at the visual output and check whether this is following my uh >> you know uh the my command uh my
[40:13] instruction. If I want to insert uh cable into the chest in the data center, I can run the um you know the uh the software test and the verify this work or not. So there will be some verifiable reward be needed for physical super challenging to get >> whether the food is good but there will be some that is feasible with some robot but there will be a lot of them you can
[40:43] uh do it in simulation a lot can do in simulation so what I think is those uh that can be uh verified through simulation will be so faster >> and that makes a ton of sense It's sort of like the equivalent of coding in agent space where you can verify the code runs. >> Yes. So, so the more we can ship uh sims to simulation, the faster we can do verifiable reward and the faster the physical going to uh evolve.
[41:15] >> So, so we're saying that it's going to be faster to have a robot in my house that will do the laundry and fold it correctly than one that will do perfect scrambled eggs. >> I believe so. >> I think you're right. I think you're right. Um, I wanted to just touch base. I There's been so much news about models and Nvidia recently. Uh, there's the Neotron family, there's the Cosmos family, and of course, there's the new hugging face acquisition. Can you help me understand how is Nvidia thinking about the model family as a whole right now? And what is the larger goal or
[41:46] purpose that you have in mind as you start to to move into the next year or so? So uh so a media believe that uh the AI will be keep evolving >> uh and uh there will be uh different application require different specialization so that's why we have this model family like neimotang for language for coding cosmo for war model uh groups for human know and alo for self driving and also this bionimo for
[42:16] healthcare. So we believe uh they uh they need to be diversity and we deeply believe in openness right. So uh openness so API are great it help you solve the problem but you know you only have asset to API you don't know the seam behind >> it's all >> it's a box it's a bread box and so for you know as a graduate when I was a graduate student I I I I learn from
[42:49] other people's open source code >> yes >> and innovation happen because of the openness right and and I don't think uh we want to I believe the world will be better place if uh people can still understand what happened behind and still can >> ingest new idea and and and create uh better and better model and I think this is you know a media is for developer right so we do want developer have more tools enabled empowered to do creation
[43:22] that's why a media so uh so much into the open source open model sense. >> Yeah. >> Well, my last question for you is a fun one. Um I always like to ask where do you think that if if you see models get to this point and we'll go beyond Cosmos. It can be anything you want to predict. I if it gets to this point in a year, if this use case is unlocked within the next 12 months, it's going to uh surprise you to the upside, right? It's going to be something where you're like, I didn't
[43:52] expect it. the scaling laws held more than I expected and now this use case that we can't do today is unlocked. Um, and you could do that obviously for Cosmos. I think that would be interesting, but for anything you want. I I would be curious like you're deep in this space. If you look out 12 months and you were like, "Wow, this went great. I'm surprised at how well this went. Now we can do this. What is this?" >> Yeah. So, I have to say that uh uh if you ask me uh if we have this talk like two weeks ago, >> Yeah. Uh I might describe what Astra has
[44:23] achieved [laughter] >> and here we are >> and here we are very few skill people can create 3D world. Yeah, >> I thought this is a challenging problem, but I I think HR show us that uh well through this uh um this uh uh verifiable reward. Uh you use Blender uh you call in Blender create a 3D space 3D wall and you project and you verify whether that match your description. This loop you
[44:53] know >> uh give you a [snorts] verifiable reward. So you can quickly create amazing uh capability and we have seen amazing scenes people share about what's Astra created. Yeah. So uh I I think that that is uh pretty mindblowing and I I think my next surprise will be how the connection of this idea to the physical AI the physical tool use. Yeah. So there may be an amazing way to find a verifiable reward is what we can achieve
[45:24] today but not everybody realize and somebody just put the trigger pull the trigger and then execute. I'll be very surprised. Yeah. And that's one of the things that I find really interesting about the AI space is I I've seen over and over and over again that there's these innovations that are possible in this space that one or two or three researchers can put together a paper >> and they can put together a thesis for how to move AI forward in a particular domain maybe around verifiability in physical space >> and then all of a sudden it's out there right and all of a sudden we all can learn from it and we all can grow. So
[45:55] it's it's very yeah I can't wait a lot to look forward to. >> Yeah. I think um I think the knowledge diffusion uh enablements of every individual is critical to make our world a better place. Yeah. And I think many people uh from the research background deeply believe in the idea. >> They do and that's it's one of the things that makes me hopeful. Okay. >> Yeah. Well, thank you so much for taking time. I really appreciate it's been a super fun conversation. Likewise. >> Yeah. Yeah. Very good.
[0:31] People know NVIDIA because of chips, but they actually do a ton of model development as well. I'll be chatting with Ming Yu, who is the head of the Cosmos world model family. For those of you that don't know, a world model is a model that produces actions the way LLM produce text. And so for example, if I have a robot arm, I could tell the robot arm to take an action like picking up the water bottle and that is a direct
[1:01] action that's produced. It's not like it's writing text that says pick up the water bottle. It actually is issuing guidance to a robot to do that. World models are already around us. I've got a little robot lawn mower named Mallister and it ms my lawn. It has a tiny world model inside it and it just outputs where that little robot lawnmower goes next on my yard. That is just a little bit of what I dove into today. So, I'm really excited to chat. Maybe you start by introducing yourself. Tell us about who you are, what you're doing at NVIDIA, all of that. >> Yeah. So, my name is Mu uh and the vice
[1:34] president of uh Cosmos Lab. Uh I lead uh Cosmos Lab uh for Cosmo platform development. uh and you know I start my career uh as a geni researcher. Uh I was the first group of people uh who believe that uh uh the future visual content uh will be generated by deep learning. >> But 10 years ago when I did a job
[2:06] interview here I told our chief scientist that I know today we are using computer graphics to generate visual content. Mhm. >> But in the future it going to be using AI. >> Uh and it was quite controversial at the time because >> you know the image could be generated by AI model at the time was so limited resolution is small and often the generated human look like zombie. uh we just I just feel like uh you know this
[2:37] is such a a nice uh paradigm uh put all the problem into a learning problem and you don't need to have all kind of handd designigned rules to create a 3D world just datadriven and create uh as long as this uh uh paradigm scale uh then visual content will be created right uh through AI and you know in the last 10 year we have see uh we have seen tremendous progress in the field like gradually one
[3:08] by one >> and now I I think most of the I would say you know that's growing maybe exponentially the amounts of content uh in the uh vure uh is generated by AI now day yeah >> super exponentially it's growing super >> yeah yeah exactly >> so you were early so 10 years ago you saw this vision that AI was going to be a big part of video creation now. You're in charge of the sort of the Cosmos family and and where that's going.
[3:39] >> I think everyone assumes that Nvidia is about GPUs because that's all we hear, right? >> Um, but I I learned that we have more software engineers at NVIDIA than you have hardware engineers. And so it's a software company, too. There was the Hugging Face acquisition recently. Tell me a little bit about sort of within the NVIDIA family. What's it like to be focused specifically on I guess we could call it a world model. Is that a fair way to describe it? >> Yes. World model. Yeah. So, uh media is
[4:09] a developer company. >> So, we build our computers to help developers build uh amazing product to serve their customers to you know and developers often need more than just the GPUs, right? uh you also need a CUDA so that you can better program our GPU to extract them all the muscle powers from our GPU right and there are also uh libraries we build on top uh and also like a AI model we build on top so we
[4:41] are kind of building layers of software on top our computers now our computer including GPU CPU and networking devices connect them together uh so that we can better serve our developers >> uh you know we providing tools so they can kind of save some energies and move faster. Imagine they may figure out way to build better tools than what we provided right but also through building these tools we better understand what kind of computer architecture we need to have for the future challenge. So we are
[5:13] building uh we are kind of building vertical to better uh knowing how to build our kind of computers uh to serve the whole ecosystem. No, that makes a lot of sense and I think that focus on the customer is something I've certainly seen talking with uh NVIDIA engineers in other contexts as well. So it seems to go through the whole company. Certainly heard it from Jensen. >> You better go first. >> Yeah. Yeah. Exactly. Uh so when we talk about world models and Cosmos, I think a
[5:43] lot of us come to the table around AI at least after chat GPT and we think in terms of tokens, we think in terms of LLMs. >> Uh world models are a different game. So can you introduce for us like what are world models? How do we get to this point with world models? Uh and what is cosmos like? >> Yeah. So um people have the some model of the world before they have language. >> Mhm. >> Okay. >> I have a one-year-old so I I know this. [laughter] >> So uh and uh so language um is very
[6:18] nice. Uh it's symbolic. Uh it's semantic. uh it's a uh document how people think about everything right so uh when GBD uh I think it's the first major AI breakthrough uh it makes sense it happened in the language domain so uh enormous amount of information there encapsulate uh and learn from how human deal with scen and then can start have chat with you now can help you with the
[6:48] code generation review right so what model is from a different perspective, right? So, uh people build war model uh try to accomplish different task >> try to understand for example why this thing happen >> or to predict what going to happen next >> or or have a map so that they can navigate in the space. So I I think one model is actually a set of models not
[7:19] just one model. So you build a model for some purpose. >> Yeah. >> And you live in the war and then allow scene you want to achieve you need some model of the war. Mhm. >> So then we have different kind of war model and the war model uh in this Cosmos effort uh this Cosmo platform that we want to build is to help physical AI developers to build better self-driving car, better robots, better way to orchestrate different machines uh
[7:51] in the factory. >> Yes, >> because we have our goals set for physical AI in these important verticals. So we then start to think about what kind of war model best position uh for the to to address the need of developers in these domains. And from there we realized okay we need a war understanding uh be able to explain what happened like whether this worker is following the safety protocol uh to operate this machine uh based on the
[8:21] handbook or you know is is this in this production line whether there's a defat uh from the visual inspection >> course. Yeah. Oh, self-driving car, right? They need to um so to build self-driving car, once you have a policy model, how you drive the car, >> you need to have a way to verify this policy model work, you know, uh safely, you know, accurately. Right. That's right. >> And one way is to, you know, have a fleet of cars and driver and go out to
[8:53] do the testing. And you can see it's pretty time consuming and some of the scenarios and you could not easily hit. So you your test may not be complete but if you can create the war model you know something that the self-driving car would perceive right and and now you you just have your policy in touch with the war model and you can know it's uh its performance it's accuracy it's safety you know before you deploy the the car in the face. There are also kind of
[9:23] simulation kind of war model that uh help uh needed for physical air and I gave example on uh self-driving car but robotics humanoid robot it's the same you need to be able to test you need to have some kind of proxy of the war so you can do the testing so then there's a war understanding part there's a war simulation part these are uh since we identify critical for physical AI and over the past 1.5 years um you know
[9:55] since the uh launch of Cosmos we've been gradually building uh model for each blend and in June uh this uh uh comput uh you know Jensen's uh keynote uh in Taipei uh we released this cosmos suite where we fuse world understanding and world simulation and also war action capability into a single model uh so that uh you know we can put that into a single model because even though they
[10:25] are for different purpose but they model the same world right so then there's a underlying um representation they could be shared among them so we put them together what we are thinking about war model you know is for physical AI and we identify the key scene they needed and we try to bring them uh together so that we have a a a simple way for developer to leverage uh to solve their problem. >> And I think you know when I think about
[10:55] world models, one of the things that's really intriguing to me and I think you got it at the beginning of of your explanation, you talked about this idea that we as humans develop effectively a world model even before we can speak because we're modeling the rooms we walk around in as babies, etc. Um, and that resonates for me when we think about robotics and where we are with a robotics moment. And you talked about the simulation piece a bit later. Like if I am trying to navigate a factory floor,
[11:26] >> it becomes very useful to be able to model that factory floor and do that in simulation with, you know, the forklift coming across or the person walking across and how do you handle all of the the the pallet falls over? How do you handle all these edge cases? One of the things that stands out though as I think through that scenario is that Cosmos would need to be able to be deployed so that it is very very real time and responsive in order to enable the robot to navigate safely and to work in real
[11:56] space safely. How has that understanding of the need to operate in real time shaped the way you've architected the model? >> We build model of different sizes. >> Uh so uh in Cosmos 3 we actually have three different sizes. um super nano and h uh h is the smallest one. Okay. Uh and we build it in a way that it can run real time on each device such as uh media just thor or dig spark.
[12:28] We want one model lip where robot lips. >> So there's a smaller model of course of course smaller model is uh less capable compared to larger model. We also have uh the largest one in Cosmos 3 is the super model. That one is for frontier performance and that one may live in data center or you know uh more powerful uh computer right but some of the navigation test yes um you have to be real time but there's some higher order
[12:59] reasoning task you you can wait you can be patient true yeah >> so I I think a mix of models of different sizes and some live in embedded device some live in data center might be a good combination to address the need in physical AI. >> That makes a lot of sense. Like I could I could see different use cases popping up. You know, one of the things when we talk about the scale of LLM is we talk a lot about scaling laws, right? This idea that test time inference is something that that enables us to scale the longer
[13:29] you think the more you scale uh you know scale of parameters etc. Do you have equivalents on the world model side where there's sort of scaling laws for world models? >> Yes. So uh scaling uh happen in multiple aspects right. So data scaling more data yes uh model size scaling yes bigger model uh better quality >> it also interesting part is that uh well for the uh visual content generation simulation part of the war model the
[14:01] scaling can the model uh there's also test scaling that's uh let me explain uh more concretely so we know u If the description test description of the visual content uh that you want uh to generate is more detailed then the visual content tend to be better right. So you can think that u if you can you can do loom kind of test
[14:33] scaling you keep reason thinking describing what is the scene you want to generate. >> Yeah. And then you have more like a detail. I mean it trinkets. I'm not saying we do that but you can say hey this pixel should be uh a 25 100 you know RGB color. >> I just >> as an example >> as simple. So but basically if you get very detailed description uh you can get very high quality video output right you
[15:03] can say hey uh from the first second to second uh the robot is doing this and the ham is moving 30° and next second another 10° right so this detailed description give uh very good generation result and this is one kind of 10 time scaling by by sinking longer uh describing more and you get a better result, right? Yeah. So, >> so I just uh I'm a coffee nerd, so I
[15:33] think of it as like the robot arm is pouring the coffee into the ChemX. Right. In that world, I think what you're saying is you can invest more in the fidelity of the labeling of the motion in real space. And if you invest in very high fidelity labeling across that space so that you know where it's moving exactly in the world as it pours and all of that then you have a much richer uh data set to reason across and you can replicate that motion. You can also scale up and
[16:03] infer from that into other novel motions etc. >> Yeah. Yeah. >> Got it. So, so that's uh a more detailed motion description happen in the test time >> as the so the the way we do uh simulation is uh uh we have a uh reasoner yeah >> uh to uh based on the the instruction and gradually expand what uh need to be >> uh contained in the simulation output
[16:34] >> and that is the sinking process that you keep generates the description like what you say earlier the the motion >> that's right yeah so I and I didn't want to communicate it's a bunch of humans doing that like the model itself is doing that yeah exactly >> and in this agentic war right so maybe you don't achieve this in one shot you can multiple uh model code inside a harness that you fix the issue and then they say um so I think the robot need to be three fingers left
[17:04] you know like this kind of correction so um So the scaling can happen in many different dimensions, right? Data scaling, model side scaling and test side scaling and also the agent scaling. That's what I would say >> that and that's super interesting to me. It gets at something that I think and maybe I want to suggest it to you and see how you feel about it. But it feels like one of the underlying differences when we talk about LLM versus world models is that if you get into the world model space, the data availability is very very rich. Um whereas with LLMs we
[17:36] talk about the idea that we've sort of sucked up the whole internet and now we have to like scale with sort of artificial data etc. But I look at the world around me there's rich data everywhere from a world model perspective and so it feels like it's a richer data environment. So uh in some sense yes but also um language is very concise uh semantically meaningful right and you see everything in the text >> in the visual world there are a lot of thing you don't see right the precise
[18:07] action caused to this change and there's occlusion that that's a >> allow observability and and you are also dealing with a more like a raw signal not like in language you be more symbolic. >> True. >> The things you need to model are more than what I I shouldn't say more because language also it's a challenge but I want to say uh even though you have a tons of data but the dimension of the output is much larger and the scene you want to capture is more.
[18:37] >> Yeah. >> Yeah. Yeah. Like like >> you need to learn the physics right. uh how uh when you have wind blower to blow something and how it's spread out those things are not easy. >> No, they're very difficult. Do you find I think this is one of the really interesting implications when you start to scale up these models. Do you find uh to a certain degree an emergent understanding of physics as the models start to understand things better? So uh we see that when the data uh size
[19:11] increase uh the the capture of the the the uh the physes uh it can simulate improves right so and but often we found that we need to leverage uh like a phys simulator like universe or isac that's model uh that's a you know built-in engine uh use those phys engine to generate synthetic data and use them to improve
[19:42] the models physics uh performance and I want to say that um um after so many uh years of um um let me put this way so many years of people putting tons of effort into the deep learning neuron network >> um 2010 remain a black box to us so it's hard to uh to say that it knows the physics. uh you know we actually don't know you know the phys but uh to some
[20:12] extent try to approximate uh the rule governing the data gen uh you observe uh and and the we know the rule behind uh what we see you know is governed by physics right so try to approximate that part but indirectly through data yeah >> so I'm going to give an example and make sure that we understand for the audience here so if I take this water bottle here and I were to drop it would accelerate at 9.8 m/s squared. >> We know that from equations. And so I
[20:43] think what you're saying is you can do that and have the model look at a lot of video of that and look at a lot of images of that and it starts to approximate the appearance of that falling pattern, that acceleration pattern, but you can also accelerate that understanding by giving it a physics engine to tap into during training. >> Yeah. >> Like omniverse where it starts to sort of dig into that. Does that >> Yeah, that's what I'm saying. Yeah. So um like a a bow hitting another bow. >> Yeah. >> You know this kind of data may not be easily available you know and so uh use
[21:15] f engine to simulate this uh Newtonian motion uh interaction between object and help. Yeah. And so uh as we are talking I've been thinking about one way you know let's let's think about this right. So let's say we have a maybe I let me briefly introduce the Cosmos 3 architecture and we come to my point earlier. >> Uh Cosmos 3 is a mixture of transformer architecture. >> Okay. >> So we have a reason to uh sync in
[21:46] language based on YC sync in language. >> Oh people don't know that. So you have a language component as well in there. Okay. >> Syncing in and we also have a generator. >> Okay. uh based on the uh thinking the representation learned in the listener and you generate uh you know simulation video [clears throat] >> you can do symbolic reasoning over here in the language >> symbolic and this uh video and action we also have action in right so I want to mention action is important because uh what you observe in the world is the
[22:18] deformation caused by some action taken >> yes >> and in robotics uh sometimes your desired output is action So that's what action to take to >> and you don't want to send tokens to do that. You just want to say here's the action that you should take and move the servos. >> Yeah. Yeah. So it's a everything in the neural network is represented a token a sole experiment right >> whether these uh um war model can learn physes right so uh if we keep um if we keep uh dropping both yeah uh okay uh
[22:50] and we didn't uh assuming the uh the reasoner doesn't know about the law of physics right and but it keep observing that can we use some kind of uh uh sudis learning can be uh sometime somehow figure out that the reasoner need to come out a precise 9.8 a the coefficient uh and to model this and then precisely describe how housing going to drop right if that happen uh I think I can more
[23:20] confidently claim that's uh uh this war model and the phys but I think in the reality um yeah I think we haven't reached that that uh state uh we are uh the model yes >> I don't think it's fig out yet u but this reasoning we may have a a way to look into what the model truly learned. >> So I I think if I understand correctly, we obviously have the mimicking now from the cosmos model family. Uh and you see
[23:51] a path in the future with the reasoning tower there to start to understand how the model is processing and reasoning across this information. So we can eventually start to see to what degree does the model really understand physics. That that makes sense. more like uh uh recent is more like the um you know also the same architecture that people use for coding. >> Yes. >> Like like you have a tons of observation from the reward video. >> Yeah. >> Can you see this? Can you learn the
[24:22] program? >> Yeah. >> Yeah. And to govern TypeScript, can you learn JavaScript? Yeah. >> Uh more like the phys description, right? how this particle is moving and so they pass away I believe but I don't think we are there. >> Yeah, that makes sense. >> But what's interesting to me is even though we're not there, the mimicking is good enough that we're already seeing applications for these world models. Um can you share some of your favorite applications for the Cosmos model family? >> Yeah, so um so there are a couple of uh
[24:52] applications. One is uh for policy verification. >> Okay. >> Right. So you have a a driving policy uh like how do you know it's good enough you can use Cosmo model to uh to roll out and give a score the success rates for each uh policy and from there you can >> um you know use it u you know for your selection and you can limit your limited flee to those more promising checkpoints the policy
[25:22] >> and Cosmo can also be used as a policy model. uh world aure model. Uh so when we release Cosmo, we show that it achieved very good uh pick and place results. >> Yeah. So so because this model is an only model is on model on this modality text video and action. >> So when you have action out it's a policy model and because it's learn with the world dynamics modeling language modeling. So you can follow the instruction and see the dynamics and
[25:53] predict the right action to come the task. Yeah. And and I think people are finding more application of uh of our model. Yeah. >> Just so we kind of dig in a bit. I think policy is something that people aren't used to hearing when they hear about AI and LM is probably a new term. >> When we talk about policy, how would you describe policy >> to someone who may not be deep in AI? >> Okay. So passing model is more like a uh an AI uh with the intention uh to
[26:24] control a physical device. Got it. >> What kind of uh command you would given based on the test given and the visual observation um presented. >> So like one example would be if I'm in the self-driving car and the green arrow shows I'm allowed to turn left and the policy says that I can turn left. The policy will give you the the way points the trajectory. >> Yeah. And it's like you turn left by following these dots along the road. Yeah. >> And it's a uh is streaming. So it keep
[26:54] generating because as you make a turn there might be another car try to cut and there's a new >> and now you have to redirect the policy. >> The time policy will take the visual input and predict a new new trajectory. >> Got it. Got it. And this is what we mean by action. You're literally there's a vector in space and time. >> Yeah. that you are predicting and directing against. >> Yes. >> Which is super cool. >> And as much as that sounds like that would be compute inensive and impossible to do in real life, you're driving around them right now. Like they're already here.
[27:24] >> As uh AI enter the physical world, you'll see more and more policy models. >> We will. And I think part of why I wanted to have this conversation is it feels like it's important for us to start to think about it more broadly as as a community. Um >> you mentioned harness earlier. >> Yeah. We're we're a little bit familiar with LLMs and harnesses now. Um what does a harness look like for a world model? How is it different? >> So uh allow similarity. Um so uh I think LN harness uh you know it has you have a
[27:56] also have a policy in the harness right and the policy there is about >> uh the security uh permission you know what kind of resources you can do you're allowed to hack into other people's [laughter] computer those different thing right but harness is you know let's uh take the software piece out uh harness is orchestration of model and also tool use right u and so I think the harness for physical will be very similar but the tool will be very
[28:28] different >> uh it will be the real robots u uh and maybe you need some camera calibration tool >> uh and the model uh might just might generate might be a policy model uh that generate action to control the robot >> and there might be also a visual inspector the model can be also a visual impacttor to measure whether the robot completed the task. >> One of my big takeaways and surprises is I learned about more similarities between world models and LLMs than I
[28:58] realized before. >> So that you know the thing I I like about agentic AI is that um you know uh as a AI researcher uh I've been in this field for a long time. We studied many different AI models. >> Yes. And now two mainstream uh two two kind of models uh are mainstream uh GBD auto request style l next to competition and division model you remove noise
[29:28] >> gradually one at a time. So these two model has the nature have the characteristic of divide a complicated problem into small problems and you solve one at a time right one token at a time instead of gening the whole paragraph >> that's like a diffusion model generates the whole experience at once whereas this is just one token at a time >> so diffusion gen the whole experience through gradually remove the noise. >> Oh so so okay there you go. Yeah, because that's how it works, right? You
[29:58] start with raw noise and you have a seed and then you start to like >> Yeah. So you do multiple step. What I like about HD AI is it's trying to solve a challenging problem through multiple model codes and tool codes. It had a characteristic of divide a challenging problem into small problems and execute one by one. And what's more exciting is that uh in ln or division model uh all the stop problem is homogeneous. It's the same nest token prediction problem
[30:30] is the same like a noise removal problem. But in LM sometimes you do the model code and sometimes you give different system problem to the model so that it achieve different task and this way of divide a changing problem into small uh easier problem. I think it's a very scalable approach based on what we have observe. Yeah. >> Yeah. No, that makes a lot of sense >> and and so so so that why I believe that um agent AI going to be also critical
[31:00] for physical AI. You have challenging problem in physical AI and it makes sense to solve it u in some problems and through some kind of uh more task specific graph execution. >> So and this is really where I wanted to go. you're anticipating me. So you're saying like we have two basic ways to solve problems. We can either do token prediction and token architecture or we can sort of go with the diffusion space where you're taking away pieces methodically as you go and then you have the diffusion space.
[31:30] >> Um and that that ability to decompose the problem is scalable both in terms of agents that we would operate in software but also in the robotic and physical space. And I got really excited about that because when you think about it, the the agents that we use in software space, like if you think about the agents you have on your machine, you have skuomorphic sort of designs that evoke robots. Like we think of these agents as little robots in our machine. So you'll have
[32:00] the little like robot face and little antenna, all of that. And we chuckle about it and we laugh and it's software design. >> But I think it gets at something deeper. I think it gets at this idea that we have a mental model of the robotics world that is tied closely to this idea of agents that operate and are able to manipulate, >> you know, bits in the case of software, but atoms in the case of hardware. >> Um, and so maybe, you know, talk to me a little bit about how you see agents and robotics in this world with Cosmos and
[32:30] kind of what what the next year or so might look like for use cases. So I think uh so in LM we have see um now gradual and you know more transition to agent world. >> Mhm. >> I think the same thing going to happen uh in physical AI. >> Oh that's super interesting. It going to be Yeah. So, uh you're going to have uh uh the physical agent make took control the robot and maybe some other digital
[33:00] tool uh suitable for this physical attack. I think going to happen and um and I think the the line between these uh will I mean will gradually be blurry. you know they they are a big let me put this way uh we still believe we require specialization right so uh in the end uh um with the uh development of AI when AI model all become very kind of powerful
[33:32] efficiency will become a major uh selection criteria you know >> it already is on the software side >> it's yes and you have this capacity Um you know they say the model size capacity uh there are so much you can pack. >> Yes. >> Right. Uh if you're a robot never need to solve Olympia mass problem. >> Mhm. >> Um it might be good in the earlier you provide some training. Right. But later you probably don't want to waste your
[34:02] capital this deal. And also the infant shape will be different right there. Think about um when s go to the data center >> um you can batch uh the API requests from different user >> uh and solve it one shot. Uh so we know LM is more like a memory bound right and but when you pack in you can leverage the uh the compute layer >> but in the field when your robot and trying to execute
[34:34] >> um you cannot wait right to so so it means that the the compute uh need to happen on different application going to be different. >> Yeah. And because of the computer difference I think there will be a v variety of different model architecture design right they all will share some similarity but they all have some specialization. So then get back to your question. I think the whole idea of dividing a changing problem into easier
[35:06] sub problem >> uh is universal right scales. >> Yeah, it it it scale but uh in the end I think you can if you don't care about cost uh you probably can have a universal one to maybe solve all of them >> but in the end you you're going to cost going to matter. You cannot put a superco computer inside a tiny >> no one will do it. Yeah. >> So then because of this uh there will be specialization in architecture and because there are so many different applications so they're going to have
[35:36] many different uh stuff >> uh and that's why media believe in GPU right so you need a programmability you need a GPU um you know be training uh the GPU have to be funible right when you change the architecture you can still train and when you change the architecture can do inference I think those are >> you know pretty exciting So go back to your question. >> Um I think that uh everything going to be agentic >> uh specialization matters
[36:07] >> and specialization will trigger different architecture and compute need. One of the things that we've started to see uh certainly in the last few months has been this idea that you can have um a fairly general purpose intelligence on your computer uh something from the Claude family, something from the OpenAI family, whatever. Now the open source family with hugging face >> um and you can start to give it as part of your harness specialized skills. >> Um and the skills are sort of like recipe cards. They enable the agent to
[36:38] learn very quickly to do something reliably. One of the things I've been thinking about as we talk about agents in robotics is this idea that you might have a capacity envelope, a general form factor for a robot, but you would have perhaps something analogous to what we call skills in software that would enable that robot to go from cooking an egg to folding your laundry. >> Yeah. >> Is that something is is that a a good way to think about it? >> Yes. It's a physicalized skill, right? The
[37:08] the difference is that uh uh instead of keep trying this uh uh digital tool uh and pass the log uh you keep scramble the act and you may have a >> uh you know also a lock need to be you know process hey hey when I scramble I find something and later maybe I need to do something differently. I think so. Yeah I could totally agree with you. Do does like at this point in sort of where we are with physical AI do we have the equivalent of the skill.mmarkdown file
[37:40] that we have with software or is that still to come? I >> think it's still to come. I think uh we have see early work uh trying to bring this in uh and but I I I the major difference is in digital AI verification testing can happen very fast. You just need a computer. In physical AI, it's totally a different story, right? >> It really is. It's hard. >> Yeah, it's hard. And that's also why we want to build uh war model because if we
[38:10] can uh have a good policy of the wall and the robot interact with the wall >> uh then you can trade the compute uh for for development philos velocity, right? So instead of creating a thousand different kitchen and you now just have your uh physical agent interact at the wall model uh then you can quickly create a thousand different kitchen in the war model and you know you gain scale from from that interaction. Yeah
[38:40] >> maybe we can touch on that a little bit. So we we were talking actually I think just before the camera started rolling about this idea of verifiable domains and less verifiable domains. Um, when I think about robotics and kind of the real world that we navigate, a lot of those domains outside of manufacturing, especially in the home, are not super verifiable. Everyone's kitchen layout is really different. Everyone's bedroom is different where they fold the laundry and put it away. And you think that's chaotic. Wait till you're in a, you know, a bedroom folding laundry or
[39:11] you're in the kitchen trying to cook eggs. um how do you think about the way you get to generalizable intelligence in those physical spaces? >> Yeah. So uh we we know that uh if we can have verifiable uh reward uh and development going to be super fast >> coding mass exactly okay and the physical AI the challenging is data is a challenging getting verifiable reward is
[39:41] another big challenge probably even bigger right so I'm cooking something right uh maybe I can simulate everything but in the end whether the food is good or bad. You need to be consumed by a human. >> That's right. >> It's very difficult. [snorts] Uh but I think there are things that easier uh to verify. Right. For example, I I want to fall laundry >> uh and uh then uh I can look at the visual output and check whether this is following my uh >> you know uh the my command uh my
[40:13] instruction. If I want to insert uh cable into the chest in the data center, I can run the um you know the uh the software test and the verify this work or not. So there will be some verifiable reward be needed for physical super challenging to get >> whether the food is good but there will be some that is feasible with some robot but there will be a lot of them you can
[40:43] uh do it in simulation a lot can do in simulation so what I think is those uh that can be uh verified through simulation will be so faster >> and that makes a ton of sense It's sort of like the equivalent of coding in agent space where you can verify the code runs. >> Yes. So, so the more we can ship uh sims to simulation, the faster we can do verifiable reward and the faster the physical going to uh evolve.
[41:15] >> So, so we're saying that it's going to be faster to have a robot in my house that will do the laundry and fold it correctly than one that will do perfect scrambled eggs. >> I believe so. >> I think you're right. I think you're right. Um, I wanted to just touch base. I There's been so much news about models and Nvidia recently. Uh, there's the Neotron family, there's the Cosmos family, and of course, there's the new hugging face acquisition. Can you help me understand how is Nvidia thinking about the model family as a whole right now? And what is the larger goal or
[41:46] purpose that you have in mind as you start to to move into the next year or so? So uh so a media believe that uh the AI will be keep evolving >> uh and uh there will be uh different application require different specialization so that's why we have this model family like neimotang for language for coding cosmo for war model uh groups for human know and alo for self driving and also this bionimo for
[42:16] healthcare. So we believe uh they uh they need to be diversity and we deeply believe in openness right. So uh openness so API are great it help you solve the problem but you know you only have asset to API you don't know the seam behind >> it's all >> it's a box it's a bread box and so for you know as a graduate when I was a graduate student I I I I learn from
[42:49] other people's open source code >> yes >> and innovation happen because of the openness right and and I don't think uh we want to I believe the world will be better place if uh people can still understand what happened behind and still can >> ingest new idea and and and create uh better and better model and I think this is you know a media is for developer right so we do want developer have more tools enabled empowered to do creation
[43:22] that's why a media so uh so much into the open source open model sense. >> Yeah. >> Well, my last question for you is a fun one. Um I always like to ask where do you think that if if you see models get to this point and we'll go beyond Cosmos. It can be anything you want to predict. I if it gets to this point in a year, if this use case is unlocked within the next 12 months, it's going to uh surprise you to the upside, right? It's going to be something where you're like, I didn't
[43:52] expect it. the scaling laws held more than I expected and now this use case that we can't do today is unlocked. Um, and you could do that obviously for Cosmos. I think that would be interesting, but for anything you want. I I would be curious like you're deep in this space. If you look out 12 months and you were like, "Wow, this went great. I'm surprised at how well this went. Now we can do this. What is this?" >> Yeah. So, I have to say that uh uh if you ask me uh if we have this talk like two weeks ago, >> Yeah. Uh I might describe what Astra has
[44:23] achieved [laughter] >> and here we are >> and here we are very few skill people can create 3D world. Yeah, >> I thought this is a challenging problem, but I I think HR show us that uh well through this uh um this uh uh verifiable reward. Uh you use Blender uh you call in Blender create a 3D space 3D wall and you project and you verify whether that match your description. This loop you
[44:53] know >> uh give you a [snorts] verifiable reward. So you can quickly create amazing uh capability and we have seen amazing scenes people share about what's Astra created. Yeah. So uh I I think that that is uh pretty mindblowing and I I think my next surprise will be how the connection of this idea to the physical AI the physical tool use. Yeah. So there may be an amazing way to find a verifiable reward is what we can achieve
[45:24] today but not everybody realize and somebody just put the trigger pull the trigger and then execute. I'll be very surprised. Yeah. And that's one of the things that I find really interesting about the AI space is I I've seen over and over and over again that there's these innovations that are possible in this space that one or two or three researchers can put together a paper >> and they can put together a thesis for how to move AI forward in a particular domain maybe around verifiability in physical space >> and then all of a sudden it's out there right and all of a sudden we all can learn from it and we all can grow. So
[45:55] it's it's very yeah I can't wait a lot to look forward to. >> Yeah. I think um I think the knowledge diffusion uh enablements of every individual is critical to make our world a better place. Yeah. And I think many people uh from the research background deeply believe in the idea. >> They do and that's it's one of the things that makes me hopeful. Okay. >> Yeah. Well, thank you so much for taking time. I really appreciate it's been a super fun conversation. Likewise. >> Yeah. Yeah. Very good.