Nate B Jones
I Stopped Knowing What My Computer Was Doing. Then I Asked OpenAI Why.
🇬🇧 EN🇪🇸 ES
Ask the tutor about this video
Sign in with a verified email and ask the tutor anything about this video.
TL;DR
Research summary
▶ Opening: working alongside agents
The host describes talking to an agent with Whisper Flow while a cursor moves through windows and performs UX acceptance testing, leaving them unsure what is happening across their computer. The conversation’s premise is how AI can move beyond chat to meaningful work: the host plans to discuss work practices, future trends, careers, skills and AI fluency with OpenAI guests, while acknowledging they do not have all the answers. The opening also includes remarks that voice is an efficient way to provide context, while reading is faster than listening, and a report of an agent finding a tax error that saved thousands of dollars.
▶ Guests and surprising Astra uses
Andrew says he has worked on the Codex app and other projects at OpenAI for a little over a year, after about eight years as a startup founder. Ash says he joined about three years earlier, first leading the enterprise team, later working on consumer products, and now leading a productivity and engineering group focused on useful agents for knowledge workers and developers. Asked about surprising Astra uses, they mention people making 3D environments and games in Blender. Ash recounts a family member turning a home-renovation blueprint PDF into a walk-through 3D model and making some renovation decisions based on it.
▶ Making computer use accessible
Ash says that in knowledge work, showing less of what an agent is doing while still providing enough information to build trust has made the experience more accessible. Andrew explains that developer tools could expose scripts, selectors and clicks, whereas a broader audience needs simpler cues, such as a picture-in-picture view and a friendly pointer; he says the underlying model capability is similar. The host says they had underestimated the potential of computer use: once it became fast and fluent, it could handle tasks such as government paperwork where an MCP or API may not be available. The guests agree that MCP can be faster and more token-efficient in some cases, but say universal MCP coverage is unrealistic and computer use can serve as a general connector.
▶ Computer use, everyday complexity and integrations
The host asks whether the computer is becoming the agent’s workspace, describing agents moving through windows while the host speaks to them. Andrew frames computer use as a way to reduce the complexity people face in websites, PDFs, forms and even fax-based processes, with examples such as locating vaccine information or sending a form to a sports camp. Ash says connectors and plugins have become more useful as models improved, and that each additional connector can multiply product capability. The team focuses on providing the tools third-party services expose, and on making connectors reliable, fast and token-efficient; Ash says they will continue this work and build better connectors for specific products where useful.
▶ Building fluency through daily use
Ash recommends trying an agent before doing a familiar task manually, such as updating a presentation or sending an email, and says moving from doing everything oneself to starting with automation is a significant step. Andrew says OpenAI’s environment uses Codex and ChatGPT alongside ordinary third-party tools, which they have connected. He describes people asking how to do things and being told to ask by voice; he says the tools can pull together information from Slack and Google documents. The guests describe adoption spreading function by function, with different tipping points depending on model capability and available context. Coding was an early fit because local CLI tools were available; legal work also adopted early, in part because models could read local Microsoft documents. They say the timing depended more on role-relevant context becoming available with adequate performance than on whether a role was considered technical. Ash notes finance and business colleagues creating and sharing apps with ChatGPT, while lacking tools such as Git and version control.
▶ Sites as new work products
Ash describes sites as a way to create an AI-native artifact when a document, slide deck or spreadsheet does not suit the task, especially when something needs to be interactive or persistent. A person can describe a problem without knowing how to express requirements in engineering terms, and the model can create an application; Ash says this may let knowledge workers express more than they could with conventional documents. The guests say work on these sites can resemble iterative product development: employees create and maintain tools for internal audiences, who act as customers. Andrew says the cycles can be faster because internal tools do not need to preserve compatibility for a billion ChatGPT users, and such work can teach software developers to iterate.
▶ Skills, judgment and the role of human work
Ash expects models to improve the quality of work products, their ability to understand intent and their ability to bring in context and cite sources. He says ideas, validation and iteration with agents will become more important, and that shorter cycles may let people handle more tasks in parallel. He also identifies convergence and taste as bottlenecks when it becomes easy to make many things: teams still need clarity about what to build and how to shape a coherent product. The host reflects that busy work can be delegated, leaving people to decide how to use their time, provide ideas and avoid producing low-quality work. The host also describes feeling liberated to create things beyond their engineering skills and says meaning lies in the end product.
▶ Token efficiency and cost per outcome
The host describes experimenting with a workflow in which Luna, set to extra high, orchestrates Astra executors and subagents, while not all agents receive the full context window. The host says this has been very token-efficient and asks for other approaches. Andrew says larger models increase pressure to be efficient per unit of delivery, cites cases where Astra is an order of magnitude more efficient than previous models, and says code mode improves computer use. He says the team is trying to build some multi-agent task specialization into default model behavior. Ash adds that they work on token-efficient skills and computer-use harnesses. He says cost should be judged by how much was spent to complete the desired work, rather than token count alone; a model can cost more per token while being more efficient overall, so users should assess ROI.
▶ Outcome-focused and proactive assistance
The host describes using AI to analyze product data, identify groups with retention or onboarding problems and share reports with the team, tasks that previously involved finding data tables, writing SQL and building dashboards over days. Ash says they use such experiments to learn what can be productized and mentions a recently released data plugin intended to help with data questions and streamline reports. He says a future product should proactively surface useful questions, such as a 9:00 a.m. report identifying retention blind spots. The host cautions that proactive recommendations require confidence in the context, insight, its direction and the system’s confidence. Ash agrees that the area requires care: overnight inference could consume a user’s usage budget for a few low-quality insights. They say token efficiency and model choice matter, and the quality bar is higher when a user has not explicitly asked.
▶ Voice, visual output and ongoing direction
Ash says voice is available but underused: people need to discover the capability and become comfortable speaking instead of typing or tapping. Andrew calls voice an efficient way to provide context, while reading is faster than listening, and says combining voice input with generated visuals is promising; he mentions visualized skills and generative UI but says they have not done anything big yet. The host says they may want to speak context while receiving visual information and questions rather than spoken replies. The speakers discuss agents asking clarifying questions, including questions sent to another device to unblock long-running work while someone is away. They also discuss interrupting or redirecting an agent mid-run, including when the user changes their mind; Ash says the interaction should feel fluid and bidirectional, and that the interface could make this clearer. The host describes using agent suggestions as a bouncing-ball process to clarify intent, alongside goal-directed execution. They discuss allowing some experimentation or randomness within a plan, with the host saying they want instructions followed but some room to explore.
▶ From individual adoption to team productivity
Asked how AI adoption translates into team productivity, Ash says a power user can identify a repeated team problem and build an AI-supported process or product. Examples include turning a recurring financial-model update into a shared ChatGPT site and publishing bugs for non-retentive cohorts into Slack so a team can use them. Ash says these shared outputs help others learn by seeing practical examples and receive value directly, rather than relying only on abstract training.
▶ Difficult tasks and recent breakthroughs
Andrew describes a long-term challenge of keeping a cross-platform app fully native across platforms while features move quickly; he says coordinating changes in lockstep has been difficult, though Astra produced a good rewrite in Rust and GPUi across several platforms in one attempt. Ash recounts an automation that had searched for ways to save money for about two months without finding anything useful, then found a significant tax error in his favor the previous week, saving thousands of dollars. He says finding it required coordinating information, including recalling an event from multiple years earlier, and that with enough context models can exceed what he could do alone. The host describes Astra producing Lego designs at a thousand-piece scale, compared with earlier attempts that topped out around 50 or 10 pieces, and recalls an earlier, cumbersome process for creating a children’s book that can now be requested more directly and printed. The conversation ends with thanks.
Full transcript
[0:00] Is the computer my computer or is the computer really Codex's computer and I just have a corner and talk to it? Because I find a lot of the time I'm using Whisper Flow and I'm talking to the agent and then there's this like cursor running around on these windows and it's doing UX user acceptance testing on something I'm building and I don't even know what's happening in all of my windows anymore. when the context relevant to that role became generally available to the mind. >> Voice is by far the most efficient way to get context in and to communicate
[0:30] information. >> But reading is much faster than listening. >> And then actually like last week it found a pretty significant error in my taxes >> in your favor. >> In my favor. >> Ooh. that ended up saving, you know, thousands of dollars and I was like, this is something that I would have never have figured out >> and I underestimated the scaling law potential of computer use. I get to be here at OpenAI today. I am so excited. I know so many of you have asked more questions than I can count about what I
[1:01] can ask these guys at OpenAI. The the conversation I'm about to have is one I've been so excited to share with you because it gets at the heart of what makes AI go from just the chatbot in your pocket to something that does meaningful work. So I'm going to be sitting down with Ashe and Andrew and they're both deep into that transition building the future of how we all work together. And really we're going to go beyond the interface. What we're going to talk about is what does it feel like
[1:31] here at Open AI to do work differently in the age of AI. No, they don't have all the answers figured out, but we're going to talk together about what that looks like in practice. We're going to talk together about lessons we can all take for how we work differently. And we're ultimately going to get to, I think, a point in the conversation where we can all understand some of the long-term trends that you can see from inside this building that are going to shape where we should go with our careers, where we should go with our
[2:01] skill sets, where we should go with our fluency and AI and what we focus on learning over the next 6 months to a year. So, I've been super excited for this. I can't wait for you to dive in. Andrew Ash, I'm so excited we get to chat. Uh maybe first could you tell us about who you are and how you came to be at OpenAI? >> Sure. Want to start? >> Yeah. Um I'm Andrew. I started uh work on the Codex app which is now our whole desktop app and work on a variety of things there. Um I've been here about a year now, a little over a year.
[2:32] >> Cool. >> And where were you before that? >> Um I was always in a startup world. So I was a founder for >> I don't know something like eight years. um doing little things and then I you know got a chance to connect with some people here and uh things have just really taken off. >> Yeah. Yeah, that makes sense. Yeah. What about you? >> Cool. I'm Ash. I've been here for threeish years. I joined OpenAI to lead the tragedy enterprise team actually and then I bounced around a bunch doing a bunch of consumer stuff and then recently I've been focused on leading
[3:02] our productivity and engineering pillar um which is our goal is to bring useful agents to everyone starting by focusing on knowledge work and developers. And I guess maybe that's a great place to jump in. I am curious with the launch of Astra, Astra's now not just inside the walls, outside the walls, everyone has it. What do you guys see as the most surprising use case that you've seen for Astra out in the wild since it's come out? >> The most surp I mean the fruitfly has
[3:32] been uh a little wild. >> That has been a little crazy, hasn't it? >> Yes. But I I think the the one I didn't realize would take off, and part of it's just because it's fun, is is the Blender. Um people just simulating these 3D environments. It's not the most useful thing that we use it for internally, but it's been pretty wild to watch people create these games and environments that they walk around in that, you know, photorealistic. Every model that comes out has these spikes of things that can just do really well. And uh one of Astra's spikes has been 3D and
[4:04] it's been >> And did you guys know that 3D was a spike internally when you released it? >> I think we had people who were playing with those capabilities. I think you know kind of alluding to what Andrew is saying like I think we're surprised by how much it's resonated. Like I had a family member who's like doing a home renovation like just paste a PDF of their blueprint into Astra and made an entire like 3D model of the renovation that you can like walk through and like fully accurate and they actually made some like change decisions based on that
[4:35] which is like pretty mind-blowing like how that's like cross the chasm from just like a power user tool to something that anyone can use to to leverage their life. What are some of the design and interface differences the way you guys have had to think differently when you're starting to build for we'll call it non-verifiable domains in the knowledge work space? I think with knowledge work we've really seen that like less is more like the the more you can sort of abstract away exactly what the agent is doing but still provide enough that people can trust that it's
[5:05] providing the right outputs. Um that's made it much more accessible to people and >> yeah I mean if you take the computer use capability which is really state-of-the-art as an example we've always been able to do some amount of computer use. >> Mhm. when it was just a developer tool, we could get away with showing the scripts that were running to [clears throat] do this computer use, right? If it's going to click around this app, we can show the scripts, you know, it's running that the selectors it's clicking on the things like that. >> Um,
[5:36] >> as we've shifted this to to hit a larger audience, we have to build out like >> you can't show that script. You got to know that that's now using notes or using the browser and put a little, you know, picturein picture display of it clicking around and add the pointer and right. So they're all coding agents under the hood. So we added, you know, friendly pointer that clicks around and and things like that. But fundamentally, it's it's very similar in what the model is good at. >> You know, computer use has been one of those vocally self-critical moments for
[6:07] me. In 2425, looking at where computer use was, nobody had it right. like it was janky. Yeah. >> Any implementation I looked at. >> And my thesis at the time was that if you use a computer as a human, you're going to be good at using it as a human. And if you use it as a machine, you're going to want a machine readable interface. You're going to want an MCP. You're going to want an API. You're going to want something that you can interact with. And I underestimated the scaling law potential of computer use. And when it got good enough, there's just this massive tipping point into
[6:38] latent compute that doesn't require MCP and doesn't require API. And now I find myself using all of these use cases that are suddenly fast and they're suddenly fluent. Like I can use it to look through a bunch of annoying like paperwork that I never would have had access to via MCP or API because like if it's government paperwork, they're not building that. Like that's not happening. >> And now it's just done. Like I can just take care of it. And I think that's been >> an aha moment for me, but also for a lot
[7:08] of folks who are watching is how good that's become as quickly as it has. >> Yeah. It's really the the the universal connector. >> Yeah. >> Right. There are many cases where MCP is faster, more token efficient. >> Yeah. >> But the world in which you get MCP for everything that you do is just that's not >> it's a non-existent world. Yeah. >> So you're right. I mean, computer use for a while for us was it was an impressive party trick >> where it's it's very visual. It it seemed impressive, but to actually work with it
[7:38] >> is kind of slow, right? And now it's so fast that >> we miss sometimes if an agent in the background is using it where it's like, oh, we don't we didn't realize that it actually switched from MCP to computer use for that one thing that isn't in the MCP. It just it just happened, right? And you look back and you're like, oh, interesting. I was clicking around. I didn't even notice. Right. Um, it's crazy. >> We are running into this situation where we are asking ourselves, is the computer my computer or is the computer really
[8:08] Codex's computer and I just have a corner and talk to it? Because I find a lot of the time I'm using Whisper Flow and I'm talking to the agent and then there's this like cursor running around on these windows and it's doing UX user acceptance testing on something I'm building and I don't even know what's happening in all of my windows anymore. And so I'm curious when you think about knowledge work and where it's going, how do you think about compute availability? How do you think about how we will be using computers as humans versus how we will be directing agents in the computer experience? >> I mean in many ways we we we spent
[8:39] decades building up this computer UI apps websites like thing, right? That is far more complex than I think anyone really wanted. >> Yes. >> To do everything, right? And you feel that in your life, you know, I I have two kids and filling out the forms on I have three. I feel you, >> right? You know, [laughter] life has become so worked work workshaped with this stuff and it's it's actually it's worse because these tools are often like
[9:11] PDFs, right? Or you're faxing something. Pediatricians still use fax. It's >> all it's all like that, right? It's far more complex than we ever wanted it to be. And I there are some ways that I see the codeex moment, the chachi team moment with computer use as some some sort of like let's get back to what technology was supposed to help us with, right? Describe your problem. >> Um, describe what you're what you need help with and it's going to click around
[9:41] that. It's going to find the website. It's going to scan the PDF. It's going to, you know, send the facts. It's going to file this away so that on my phone later I can be like, "Hey, when was the tetanus vaccine that we got?" like >> can you pull up this >> form and send it to the sports camp, right? Like all of this stuff is is I think >> helping navigate the complexity that's just been built over decades and kind of never really dealt with, right? >> Never dealt with. In fact, it's on average gotten worse every year. I find >> I don't even think twice about asking codeex to do that anymore. And it feels
[10:12] like there's that's not just an intelligence problem, that's also an integrations and data availability problem. And I'm curious how you guys thought about walking over that bridge, how you thought about sort of opening up the computer beyond computer use with like, you know, the email integrations you guys have done, the Slack integrations, etc. Um, and as you look ahead, to what extent are you looking at the integrations piece as it's mostly done because now computer use is so good versus no, we still want to invest very heavily here because there's more capability when we invest in these deep
[10:43] integrations. I think for a long time, you know, chatbt has had connectors, plugins, whatever you want to call them for a while. Um, the model was not always the best at using them. I think in the past 6 months or so, suddenly, much like with computer use, there was like a flipping point. Suddenly, it was like, oh my god, we need to give make sure it has access to anything that someone would want to give it access to because that just each incremental connector or plugin that we add like multiplies the power of this product. is
[11:13] we focus a ton on latency, reliability, etc. to make sure that like when the model uses this connector, a it has access to as many possible tools as third party service provides. Like we're reflecting all of those, b like they work and they work reliably and c they're fast and token efficient. Um, and I think we'll continue doing that, but in the cases where we can make a better connector and for for those specific products, like why not, right? Like we're just giving an additional power to the model. And I'm curious when you look at dog fooding internally like you've been describing, how have you
[11:43] seen uh the shape of work start to change, what is the thing that you sort of know internally as a fluency habit or as a learning habit or something that has been really helpful inside this company that you would want people to know? But I think one thing that we at least for me this is the my personal trick is try it like you know you you come to work or or you're home wherever you're like looking at your computer and you have something to do you're like oh I need to update a presentation or I need to send someone an email or I need
[12:13] to do whatever and you're like natural instinct is going to be to go start doing that >> and I think something that was very >> important in my journey was like to just be like no I'm not going to do that I'm going to try [laughter] it >> in codeex or chatbt Yeah, I think that first step of going from like doing everything on your own to starting with automation I think is is a big gap. >> We've set up our environment to be very codeex and chatb native, right? So the whole thing kind of works and we use normal third party tools for plenty of
[12:43] our work, right? It's not like we've made everything. We just we've made sure that they're connected well. >> And so, you know, people will be asking questions like, hey, what why why do we do a certain thing or like how do I connect to this thing? And the answer is usually like just talk talk to it like press the voice button ask. And it's it's um it's really interesting how you can kind of get through like cut through the noise, Slack, Google documents, like all of the stuff that people have worked on and kind of get the full picture. I think it's interesting to watch or you
[13:15] know everybody here uses codeex or chat work now for most of their job and for a while that wasn't yet true, right? Um there was a period between I don't know the first half of this year that like function by function was making that kind of grand transition to how they worked and for each one it was like a slightly different tipping point. Um, coding was obviously the first one, right? Um, and then who was who was
[13:45] second? >> I heard legal was >> early because there's so much material, right? >> Um, and I think it's like a combination of what types of work the models have been good at at what time, but also then what kind of connectors and file access, like what does your context look like? The coding one was the easiest because everything was just like a local CLI, >> right? >> Yeah. >> Like you don't need a Git connector.
[14:15] You'd need the Git CLI. You need the GH CLI, right? >> And so, so we didn't have a great connector story in I don't know, January, December, right? But you would code with it. And then it's like legal legal's got local Microsoft documents all over the place which the model could read. And then it was it was kind of during this first half of the year that like connectors got really good and then computer use got really good. And so depending on the context that was required for any given role, it kind of like tipped tipped that over the edge
[14:45] when you know this critical mass right >> when the context relevant to that role became generally available to the >> generally available in performance and yeah so it was it was actually more that than like I guess the normal stuff that you would expect about how technical a role is or something like that. >> That's a really good point. In fact, one one of the things I've been taking away is it feels like the question of whether a role is technical is becoming less and less of a question I find useful.
[15:16] >> Mhm. >> Yeah. It's been one of the things that has been super cool to see in the past few weeks. like I've been talking to or spending some time with people in the finance team and business teams and um they've been moving away from Excel in some places to like create these using chatb sites like effectively apps but then they're sharing across their teams and like iterating on over time and like making new versions of and updating for like new use cases and things like that and so for all intents and purposes they are you know developers in that in that
[15:47] realm and like they don't have a lot of the tools it was like kind kind of anxietyinducing for me because it's like okay they they don't have any of the tools that we have in order to make our jobs easier like git and version control and stuff they're kind of just yoling it with you know the what we've provided them and so there's so much more to build in order to like make their lives easier but like the fact that they're able to do it now itself is I think super cool >> I think touching on sites is interesting I know there's been a ton of investment in that space for you guys in the past call it month or two do you see a like
[16:19] if you look at knowledge work and kind of where knowledge [clears throat] work happens. Does sites feel like it's a big new general purpose lever around knowledge work? >> I think there's this idea of like artifacts that are created. I think historically, you know, docs, slides, sheets, >> um, PDFs, etc. With AI, like maybe there's like a new like AI native artifact in the future. Um, I think the thing that sites does a really good job of is like sometimes the thing that you're trying to express doesn't fit cleanly in any of these buckets. like
[16:49] maybe it needs to be interactive, maybe it needs to be persistent in some way, etc. And maybe you because you're quote unquote nontechnical might not even know that like you know the requirements but you don't know how to articulate them in a way that like an engineer might but you can just describe your problem to Andrew's point to the model and it can create this application for you. Um and it fills that gap of like okay you could never do this in a sheet or a doc or slide deck um but now you're able to do this with a slide. So, so I do think like for knowledge work it enables people to be much more expressive than they otherwise would be. So I think
[17:19] that's going to be a huge unlock. >> It feels like in most situations most job roles there's some degree to which it's iterative. I would be curious to what extent you have seen differences in user behavior when people are say in work versus people in codecs and how they are starting to use perhaps a more rhythmic or iterative workflow versus maybe a goal oriented development workflow where you just have to get the code to production and it needs to be shipped. >> Definitely more similar than they have
[17:51] been. I think you know over the past number of years software development has become more iter iterative incorporated that's a fair yeah >> um so I think like the shape of if you were to call it the product development life cycle of like an internal tool that is being built as a chat site looks a lot like your best practice on how to give something to customers right so everybody's kind of now surfacing something to some sort of customer right you know their HR might make a site to
[18:22] help us identify talent and like that we are the customer of that product that they are maintaining. So everyone's sort of starting to maintain a product of you know that captures their type of expertise for another audience. Um I think in many cases their cycle is faster because it's a you know smaller internal audience right they don't worry about >> backwards compatibility of a billion chat GBT users and so um sometimes they teach software developers here a lesson on [laughter]
[18:52] >> you can just iterate and >> of course you don't have a billion users but >> yeah that's an interesting point as well. What is it that we should be betting on in terms of skills? what is it that we should be betting on in terms of uh trend lines that we can keep an eye on and continue to expect to improve that are predictable enough that we can say in 6 months X or Y is going to be a lot better. I think that creation of these work products for lack
[19:24] of a better word like sites or the tools or documents or whatever is going to continue to improve the model's capability to do it and that's on a mult multiple dimensions. So like one is just like the quality of the outputs right two is like the ability for it to understand your true intent like what you mean the ability for it to pull context in to site sources accurately whatever all those things are going to get a lot better. Um, I think what that means is then what you're bottlenecked by is like ideas. >> The ability to like think about what to
[19:55] do next to have the agent come up with something to to Andrew's point, validate whether or not that's useful with either yourself or like the people you work with and then go back to the agent and ask for like iterate. So I think the skill of like working with the agent is the thing that is going to stay constant and like can you continue to get better at that and then you can assume that each iteration the quality will keep getting better and you know maybe the iteration cycle gets shorter and then you can do 10 things in parallel where previously you were only able to do five or so on and so forth. I think the the
[20:26] other side of that is that it has become so easy to make things that taste on convergence is another bottleneck that we sometimes have which is like there's there's a lot being made at any given time. >> Yeah. >> Right. We want to ship as focused as cohesive and coherent a product as possible to help people >> solve their stuff. So, how do we take that thing over there that looks cool and those three things and that thing
[20:57] and that thing and put that together, mold that into something that makes sense, is coherent to people, is tasteful, right? Cuz now it's like you can ship anything. We could ship every single product that's ever been >> You can you should Well, that's another question. [laughter] Um, I think that's that's one is really having clarity of thought of like >> what what do we really need to do here? >> And that like both of you called out mental skills like you talked about the
[21:28] fluency of working with agents. Um, and you talked about decision making and kind of how you make good decisions with good taste. Um, I think to me one of the things that I've been reflecting on is when I was earlier in my career, there was this obviously there was work that was intellectually rigorous that I had to do, but there was also lots of busy work that would just fill the dates as well. And now all of that is gone. >> Like largely the busy work is something I can give to an agent and it just gets done.
[21:58] But that means that the question then bounces back to me, what am I doing with my time? How am I maximizing my agency? Do I have the ideas to actually give AI what it wants? Am I making decisions that are tasteful or am I pushing slop? And it feels like that's a question that we've all been posed. There's a level of slight discomfort on having to, >> you know, reinvent yourself over and over again. Um, at the same time, you know, I I grew up like a software engineer, but
[22:30] also a designer, and there were always things that I saw in my head, wanted to make that were beyond where I was on the engineering side. >> Mhm. >> And I feel somewhat liberated to be able to deliver things without the busy work. Right. >> There's no inherent meaning in that. It's the meaning is in this the end that the thing that you created. One of the things I'm curious about is how you guys think about uh token efficiency. It feels like we like, you know, maybe
[23:00] April or so token maxing was the thing and everyone was talking about token maxing and token leaderboards and now the pendulum is swung back. I've been having a ton of fun switching the usual assumptions. So, a lot of the usual assumptions are you pick the smart model as the orchestrator and the dumb model is the executor. I've been playing with switching it and having Luna be the orchestrator on extra high and having Astra be the executor with sub agents and then I just poke Luna on agency and not uh under scoping what I'm building
[23:31] with another Astra model every now and then but it doesn't get the full context window so it's not chewing up all those tokens and then the super long running orchestration thread lives in Luna and I found that that is tremendously token efficient and it still gets me a lot a lot of the way to where I want to go. And I'm I'm curious like, have you heard of that? Is that a surprise to you? Do you have other tips or tricks for how you're token efficient? Um, yeah, >> I have heard of this. I Yeah, I've heard I've heard more and more of it. Um, >> as this pressure comes in,
[24:02] >> not just because people didn't realize they should be token efficient, but because as the models get larger, it kind of pushes you to be more token efficient, right? Um, per unit of delivery. Um I mean I think we've seen some real leaps in efficiency on Astra, right? There are cases where Astra is just an order of magnitude more efficient than our previous models and that's phenomenal. Um and the code mode uh stuff for computer use makes it even
[24:32] better. Um, I think we're also we're also trying to take some of the things you described in, you know, I've got this multi- aent setup where I'm picking this model for this and this model for this and this model for this and baking a little bit more of that into how models work by default. >> No, I got it. >> And [laughter] and you know, we want to be able to bring that to everybody, >> right? >> Yeah. Yeah. And then like Andrew mentioned like we've been doing a lot of work on kind of the the details like making sure that you know the skills that we're publishing are the most token
[25:02] efficient for the model. Making sure that the harness and how it's using computer use with with things like code mode is is is working in a way where like we're using as few tokens as possible. So it's something that we want to continue pushing over time. Like I think the what you were mentioning about Astra doing the execution isn't surprising to me because one thing that we've seen is like when we're when you're purely measuring like tokens per per you know unit of time it's not the exact measurement that you should be thinking of even as a costconscious consumer because really what you want to measure is like how much did you spend
[25:32] to do the unit of work that you wanted to accomplish. Exactly. And so like you know while while Astra may you know be more expensive on a per token measurement it's like much more efficient. And so like what we want to do is like figure out how people can validate that like the ROI not necessarily just measuring like how many tokens you're spending and multiplying by price. >> And I get people who say well my employees use it for email uh they use it as an answer engine uh and they use it for coding. And then there's just
[26:03] sort of this long pause and then you're you're sort of supposed to kind of fill in and say and then the next thing is how do you think about building an outcome focused product in a world where knowledge workers may not have the ability to cleanly articulate outcomes at the start of the process but may get there over time. like I I often use it to go through data, find groups that are not quite retaining on the product or you know getting onboarded fast enough and just like
[26:34] giving me clarity around that >> you know jobs that would take days beforehand to like you know find the data tables to write the SQL queries >> to build the dashboards and say oh no that was the wrong query it doesn't tell me anything so now I got to go back and do it again and nowadays I can you know get a report that's kind of like hey like this is an area of opportunity for us and I share that with my team and >> I think that you know we play a lot with how can we push it in different areas and it's not always you know ready here and we're doing it a lot for our
[27:05] research of what can we productize and so I I realized that you know people don't have the time to play with it the way that we do and we've got to bake some of this in so the other day we released a a data plugin for example that helps with a lot of these these um data questions and just really streamlining like how reports are made and things of that nature. Um but I think this is something that we want to to play with, right? Is you shouldn't have like if we if if the model knows how to answer any question you type into
[27:36] it, then you shouldn't have to ask every one of those questions. Like we should be able to provide you a 9:00 a.m. report on, hey, here's the things that you're not asking, right? Yeah, >> your job is retention and you're not asking about these groups, but they are actually a huge blind spot for you, right? Like >> they're turning, right? Exactly. Like that's the type of thing I think we need to get right next. >> Yeah. Yeah. And and proactivity is sort of taking the context problem and
[28:06] pumping it up on steroids because then you have to not only be confident of the context, but also confident of the insight and confident of the directionality of the insight and confident of your degree of confidence so that you know when you're coming out and saying something, you're able to say, "I know this is true because X, Y, and Z, and therefore I recommend C action." >> Yeah, >> it's super interesting. It's an area where you want to be careful, >> especially with efficiency, >> right? You can't run this thing and just
[28:36] do inference all night, super expensive in inference for 12 hours while you're sleeping and eat up somebody's entire, you know, usage bill >> for, you know, three lousy insights that are kind of spammy, right? >> So, this is something that we that we are thinking about a lot and getting right. >> Yeah. The token efficiency piece comes in there. The model choice I'm sure comes in there. >> Um >> the bar is generally higher for how good we have to be when the user hasn't explicitly asked question. >> I think of voice as an interface of
[29:06] course. When you guys think about model capability, computer use where these things are going. >> What are the trends in interface design that you guys are paying attention to as you think about where say codecs or work could be expressed in a year or so that like we wouldn't even think about now? I think I mean you touched on voice so I think the one to to call out like >> I think the this is one where like you see early adopters so far ahead of the curve of everyone else like when I walk around the office
[29:37] I see some people with like six different mics on their desk and that's what they use to communicate with you know codeex or tragedy media work all day but that's clearly not made it um out to everyone and I think part of that is you know the product needs to do a much better job people discovering that this capability even exists. Part of it is like we need to bring people along on like changing from you know typing or tapping to like speaking and like how much more fluid and and frictionless that is as an interface. Um, so I think we have a lot of work to do there in
[30:08] terms of like making it feel like this is the next thing like codeex or chatbt work is or chbt is with you everywhere um because you have access to this voice interface for people who get it they truly get it but we haven't done enough to like show people that that's possible. >> So even beyond like building a net new interface I think voice is one where like we have it already but like it's about how do we bring it to everyone. >> Yeah. >> I think the the interesting other side of voice is that voice is by far the most efficient way to get context in and to communicate
[30:39] information, >> but reading is much faster than listening. >> That's right. >> And so I I think like playing with this combination of voice in and like generative visuals out is very interesting to me. And we've done a little bit with our visualized skill and our generative UI, but I I don't think we've >> done anything big here yet. >> Yeah. Um, there are many times where I want to be speaking into this this product, into this model, providing context at a speed I can't type,
[31:10] >> but not having it speak back to me, having it show me things and, you know, ask me questions. I'm a very visual person, right? Um, so we're we're looking at a bunch of that, too. >> Yeah. >> Yeah. I I I think that there's more to do here, too. This the the size of interaction needed is not always the same. Mhm. >> And so as we think about things like token efficiency, even >> sometimes a single question can unblock like a ton of token usage and a much better outcome. Just some clarifying
[31:41] questions. You'll see Astra asking more questions than previous models. But as tasks get long, the ability to direct questions to different devices you might have, >> you know, like I don't, you know, people talk about coding at the playground while their kids are at the playground. I'm like, well, I don't I don't love that idea. Like, I want to spend time with them. >> You want to be spending time with the kids. Yeah. >> But it's also very frustrating to come back to my desk and see something done in a way that >> isn't wrong, but like one question could
[32:12] have made this better. >> Yeah. >> Like just send me the yes or no out while I'm out. I'll hit, you know, I'll give answer and like unblock you. Right. >> The same way that sometimes somebody, you know, sends you a Slack message to just get unblocked, right? >> Yeah. >> Um, so we'll do more there. Yeah, I think and that's another example of like smart proactivity it seems like where it's like learning to ask the right question. Another workflow pattern I've noticed in myself is I am increasingly reading the intermediate messages that are printed and I am aggressively
[32:44] shaping the run of the agent as I see those me messages shape out. Sometimes it's like I'm directly responding and saying no no no no that's wrong do this over here differently. And sometimes it's actually taking a print out and in the middle of the run handing it to another agent saying, "This doesn't make a lot of sense to me. Can you give me fresh eyes on this?" And then >> the runs are long enough I can cycle that back around and say, "Okay, this is how I think we should proceed." >> Do you guys see sort of mid agent work
[33:14] activity as something that's an emerging category across a wider user base? Is that just me being weird as like a a power user or what's been your experience? >> I do think that as the scope of what the agent can take on gets larger, so like both like the value of it and then also the time it takes to do it, >> like we really need to nail the experience of like what does it feel like to interrupt or interject in the middle of a roll out. Not only for the things that you're describing where like maybe the agent is doing the wrong thing, but maybe the user is wrong. Like
[33:44] I've had many situations where like I ask it for something and then 20 minutes later I'm like wait a second that's not >> that was wrong. >> Yeah that's [laughter] not at all the thing that I want. I need to go back to it and make that and that needs to feel like really fluid. So I think that's somewhere we need to invest in more but definitely something that we're seeing where like people are like starting to learn that this is birectional the entire time not just like >> question response question response. Um I think there's probably much more we can do in the interface to make that clear. >> Yeah. No, that makes a lot of sense. Another pattern that I've observed in
[34:16] myself, we talked about iterative work earlier. One of the ways that I have found and that others I've talked to have found that they actually work with AI is they use AI almost as like that uh bouncing ball idea generation machine and you're kind of bouncing. You're like, "No, that's not it, but I wouldn't have thought of that. No, that's not it, but I Oh, that's it." And like you have to go through a series of nos and the nos help clarify your intent. Um, I don't I don't know there's an interface for that yet, but it feels like that's a
[34:47] very different mode for me than let's get work done because I have clear intent and I need both in certain situations. Like in one case it's do I understand what I want and in the other case it is okay now let's get it done. There was a very similar conversation I was having I think it was yesterday about the idea of like randomness or noise. >> Mhm. that like this is a parameter that you can if you're you know using every knob on like the API you can set
[35:19] parameters like this but if if you zoom out and you're like hey I've got this plan that I want to be achieved whether whether that's with a model or even with a company of people right there's a certain amount of like noise and jitter and experimentation that you accept that is somewhere between zero and like chaos >> because you know that It actually helps maximize. >> Yeah. >> Like global maximize, right? Like you're not going to get stuck in like a local maximum situation because you've got a
[35:50] little bit of play where it's like, oh yeah, these people over here are trying something new that maybe might be a waste. And I I feel I feel very similar about these models where I'm like >> I want you to follow the instructions, but I want you to have like a little bit of a little bit of playroom so that when we're talking about this stuff, like yes, it might take a little bit longer, might spend a little bit more, but you're kind of outside of the narrow band. I don't know know exactly what that looks like, but I'm with you on this. >> Yeah, >> that resonates for me. >> I would be remiss if I did not ask cuz I
[36:21] get asked it a ton by leadership. if they're rolling out AI in an organization almost always the initial few months is it's very much like okay we've got some people who are adopting this and they're super passionate we've got group B and they're adopting us kind of figuring it out and group C is like I don't know what is this and when you talk about group A which is the passionate adopters almost always the story is one of individual productivity and the next question I get asked is what about teams and so my question to you is when you think about how AI has
[36:53] kind of tipped over these departments inside OpenAI over the last 6 months. How has that translated into team level productivity or team level uh ability to get more done? I think in all cases it's identifying a process or a problem that your team faces maybe on some repeated basis and then that individual the power user the early adopter whatever them going after and figuring out how can we make this better with AI. I think like the biggest hero cases we've seen is like someone's like, "Okay, we have to
[37:23] update this financial model every two weeks. I'm going to make that into a chatbd site. Now other people can do that too. Now it's more expressive, etc." Or I'm going to create an automation that put publishes all of our bugs for our non-retentive cohorts into Slack and now everyone on my team can start pulling from those things. And so it's like getting beyond like you creating an output for yourself and creating something that your team can see and get value from because I think for like you know leaders in organizations who are like excited about deploying AI. I think when the early
[37:53] adopters are encouraged to do that then people also learn cuz they learn by seeing like oh that's something that I could have done and they also get value immediately versus like trying to run you know a training or something where it's like much more abstract. >> Yeah. Yeah, that makes a ton of sense. Okay, last question. I'm really curious. When you guys are using AI daytoday, what is I most people I know have sort of an NP hard problem, a problem where
[38:23] they just keep like running AI at it and eventually it cracks. Eventually the scaling laws come through and there's something that works. Um, for me, I had one of those moments when a I could finally read my six calendars and actually tell me what was available in my day and not just read two of them and give me false windows of failability. Uh, what is that for you guys? What is that moment that was really difficult? And has AI crossed it yet? Have you had that moment where it's like, oh yeah, okay, this was really hard. I never thought I would get there. I thought it
[38:53] would be a while and now it hit it and I'm surprised. There's like a pet one that I do every time there's a new model and it's it's hard because it's it's not just technically challenging. It's a it's a coordination problem which is >> you know we we have this crossplatform app >> and I have this dream of someday like someday with any given powerful model you should be able to maintain the like absolute platform native on all platform right you know you're doing like old style objective C UI kit on or [laughter] app kit on Mac right you're
[39:24] like yep like we're win UI3 on Windows 11 right and this is a coordination problem because we're moving so fast that it just it has been impossible to move all the features in lock step. >> Like if if we were moving a tenth of the speed, this wouldn't be as much of an issue. But we use crossplatform because of the speed, right? And someday like we'll get back to it. We'll get back to it. We'll be able to fully rewrite all of these apps at exactly the perfect thing for them and keep them all in lock step and make a Rust version. I don't know. Um [laughter] we're not there yet,
[39:55] but um we're getting a we're getting a lot closer. I can tell you that. I mean, I I have the Codex app rewritten like every time we have a new model, I'm like, >> "Hey, try it try it in GPUi with Rust, you know, like >> and just see what happens." >> And just see what happens. And uh >> Astra got Astra got a pretty good version out on like several at a time. So, >> okay, >> we'll see. >> Yeah, fingers crossed. Okay, >> yeah, I mean, I guess taking the other side of this is something that just got crossed like for a long time. So, I was a very early adopter of like connecting AI to like everything in my life either
[40:26] via the browser or connectors. And I have an automation that's like um you know, how can I save money in my life? I'd say like probably 2 months and it never found anything useful. Um and it's actually quite complicated problem cuz it goes through my email. It is logged into a bunch of stuff via the browser. It has some connectors, etc. >> Um you know, it's like maybe it asks me to cancel subscription or whatever, but that's it. And then actually like last week it found a pretty significant error in my taxes >> in your favor. >> In my favor. >> Oo.
[40:56] >> That ended up saving you know thousands of dollars and I was like this is something that I would have never have figured out. Like this just would never have happened because the amount of coordination required to figure this out. >> It was huge. Like I had to remember something that happened multiple years ago and pulled that forward and all of that. And so for me that was like pretty eye opening again of just like >> the more context you give it the models have gotten to a point where like >> they very much exceed what I'd be able to do on my own and so we I need to continue pushing in my personal life in that direction. >> Yeah. Yeah. One of the ones that Astra
[41:28] crossed for me that made me chuckle. Uh so a few months ago I was in Seattle. Tibo had flown up. I had a brief chat with him and at the time I said, "Tibo, make me a model that can make Legos cuz I love Legos. I want a model and and like it's saturating out at like 50 pieces, 10 pieces at the time." >> And now, yeah, I can hit a thousand piece Lego set. It's not a problem. >> It's It's wild to cross the physical. Yeah. >> Right. I remember a few models ago when I did like the first um book for my
[41:59] kids. >> Yeah. >> And I was stringing them together. I was like, "All right, you're going to write a script and then you're going to take the script and you're going to write an image genen prompt for every page and you're going to pass that into the long ago. >> It was not." And now you can kind of just say like, "Hey, make my kids a book about themselves, like a kids book. Go get it printed and it will." And it's just like once things enter real life, you're holding them. >> It feels different. >> It feels different. It does. It does. Well, thank you for the time today, guys. I really appreciate it. It's been a lot of fun. >> Yeah. Thanks. A lot of fun. Uh, as a
[0:30] information. >> But reading is much faster than listening. >> And then actually like last week it found a pretty significant error in my taxes >> in your favor. >> In my favor. >> Ooh. that ended up saving, you know, thousands of dollars and I was like, this is something that I would have never have figured out >> and I underestimated the scaling law potential of computer use. I get to be here at OpenAI today. I am so excited. I know so many of you have asked more questions than I can count about what I
[1:01] can ask these guys at OpenAI. The the conversation I'm about to have is one I've been so excited to share with you because it gets at the heart of what makes AI go from just the chatbot in your pocket to something that does meaningful work. So I'm going to be sitting down with Ashe and Andrew and they're both deep into that transition building the future of how we all work together. And really we're going to go beyond the interface. What we're going to talk about is what does it feel like
[1:31] here at Open AI to do work differently in the age of AI. No, they don't have all the answers figured out, but we're going to talk together about what that looks like in practice. We're going to talk together about lessons we can all take for how we work differently. And we're ultimately going to get to, I think, a point in the conversation where we can all understand some of the long-term trends that you can see from inside this building that are going to shape where we should go with our careers, where we should go with our
[2:01] skill sets, where we should go with our fluency and AI and what we focus on learning over the next 6 months to a year. So, I've been super excited for this. I can't wait for you to dive in. Andrew Ash, I'm so excited we get to chat. Uh maybe first could you tell us about who you are and how you came to be at OpenAI? >> Sure. Want to start? >> Yeah. Um I'm Andrew. I started uh work on the Codex app which is now our whole desktop app and work on a variety of things there. Um I've been here about a year now, a little over a year.
[2:32] >> Cool. >> And where were you before that? >> Um I was always in a startup world. So I was a founder for >> I don't know something like eight years. um doing little things and then I you know got a chance to connect with some people here and uh things have just really taken off. >> Yeah. Yeah, that makes sense. Yeah. What about you? >> Cool. I'm Ash. I've been here for threeish years. I joined OpenAI to lead the tragedy enterprise team actually and then I bounced around a bunch doing a bunch of consumer stuff and then recently I've been focused on leading
[3:02] our productivity and engineering pillar um which is our goal is to bring useful agents to everyone starting by focusing on knowledge work and developers. And I guess maybe that's a great place to jump in. I am curious with the launch of Astra, Astra's now not just inside the walls, outside the walls, everyone has it. What do you guys see as the most surprising use case that you've seen for Astra out in the wild since it's come out? >> The most surp I mean the fruitfly has
[3:32] been uh a little wild. >> That has been a little crazy, hasn't it? >> Yes. But I I think the the one I didn't realize would take off, and part of it's just because it's fun, is is the Blender. Um people just simulating these 3D environments. It's not the most useful thing that we use it for internally, but it's been pretty wild to watch people create these games and environments that they walk around in that, you know, photorealistic. Every model that comes out has these spikes of things that can just do really well. And uh one of Astra's spikes has been 3D and
[4:04] it's been >> And did you guys know that 3D was a spike internally when you released it? >> I think we had people who were playing with those capabilities. I think you know kind of alluding to what Andrew is saying like I think we're surprised by how much it's resonated. Like I had a family member who's like doing a home renovation like just paste a PDF of their blueprint into Astra and made an entire like 3D model of the renovation that you can like walk through and like fully accurate and they actually made some like change decisions based on that
[4:35] which is like pretty mind-blowing like how that's like cross the chasm from just like a power user tool to something that anyone can use to to leverage their life. What are some of the design and interface differences the way you guys have had to think differently when you're starting to build for we'll call it non-verifiable domains in the knowledge work space? I think with knowledge work we've really seen that like less is more like the the more you can sort of abstract away exactly what the agent is doing but still provide enough that people can trust that it's
[5:05] providing the right outputs. Um that's made it much more accessible to people and >> yeah I mean if you take the computer use capability which is really state-of-the-art as an example we've always been able to do some amount of computer use. >> Mhm. when it was just a developer tool, we could get away with showing the scripts that were running to [clears throat] do this computer use, right? If it's going to click around this app, we can show the scripts, you know, it's running that the selectors it's clicking on the things like that. >> Um,
[5:36] >> as we've shifted this to to hit a larger audience, we have to build out like >> you can't show that script. You got to know that that's now using notes or using the browser and put a little, you know, picturein picture display of it clicking around and add the pointer and right. So they're all coding agents under the hood. So we added, you know, friendly pointer that clicks around and and things like that. But fundamentally, it's it's very similar in what the model is good at. >> You know, computer use has been one of those vocally self-critical moments for
[6:07] me. In 2425, looking at where computer use was, nobody had it right. like it was janky. Yeah. >> Any implementation I looked at. >> And my thesis at the time was that if you use a computer as a human, you're going to be good at using it as a human. And if you use it as a machine, you're going to want a machine readable interface. You're going to want an MCP. You're going to want an API. You're going to want something that you can interact with. And I underestimated the scaling law potential of computer use. And when it got good enough, there's just this massive tipping point into
[6:38] latent compute that doesn't require MCP and doesn't require API. And now I find myself using all of these use cases that are suddenly fast and they're suddenly fluent. Like I can use it to look through a bunch of annoying like paperwork that I never would have had access to via MCP or API because like if it's government paperwork, they're not building that. Like that's not happening. >> And now it's just done. Like I can just take care of it. And I think that's been >> an aha moment for me, but also for a lot
[7:08] of folks who are watching is how good that's become as quickly as it has. >> Yeah. It's really the the the universal connector. >> Yeah. >> Right. There are many cases where MCP is faster, more token efficient. >> Yeah. >> But the world in which you get MCP for everything that you do is just that's not >> it's a non-existent world. Yeah. >> So you're right. I mean, computer use for a while for us was it was an impressive party trick >> where it's it's very visual. It it seemed impressive, but to actually work with it
[7:38] >> is kind of slow, right? And now it's so fast that >> we miss sometimes if an agent in the background is using it where it's like, oh, we don't we didn't realize that it actually switched from MCP to computer use for that one thing that isn't in the MCP. It just it just happened, right? And you look back and you're like, oh, interesting. I was clicking around. I didn't even notice. Right. Um, it's crazy. >> We are running into this situation where we are asking ourselves, is the computer my computer or is the computer really
[8:08] Codex's computer and I just have a corner and talk to it? Because I find a lot of the time I'm using Whisper Flow and I'm talking to the agent and then there's this like cursor running around on these windows and it's doing UX user acceptance testing on something I'm building and I don't even know what's happening in all of my windows anymore. And so I'm curious when you think about knowledge work and where it's going, how do you think about compute availability? How do you think about how we will be using computers as humans versus how we will be directing agents in the computer experience? >> I mean in many ways we we we spent
[8:39] decades building up this computer UI apps websites like thing, right? That is far more complex than I think anyone really wanted. >> Yes. >> To do everything, right? And you feel that in your life, you know, I I have two kids and filling out the forms on I have three. I feel you, >> right? You know, [laughter] life has become so worked work workshaped with this stuff and it's it's actually it's worse because these tools are often like
[9:11] PDFs, right? Or you're faxing something. Pediatricians still use fax. It's >> all it's all like that, right? It's far more complex than we ever wanted it to be. And I there are some ways that I see the codeex moment, the chachi team moment with computer use as some some sort of like let's get back to what technology was supposed to help us with, right? Describe your problem. >> Um, describe what you're what you need help with and it's going to click around
[9:41] that. It's going to find the website. It's going to scan the PDF. It's going to, you know, send the facts. It's going to file this away so that on my phone later I can be like, "Hey, when was the tetanus vaccine that we got?" like >> can you pull up this >> form and send it to the sports camp, right? Like all of this stuff is is I think >> helping navigate the complexity that's just been built over decades and kind of never really dealt with, right? >> Never dealt with. In fact, it's on average gotten worse every year. I find >> I don't even think twice about asking codeex to do that anymore. And it feels
[10:12] like there's that's not just an intelligence problem, that's also an integrations and data availability problem. And I'm curious how you guys thought about walking over that bridge, how you thought about sort of opening up the computer beyond computer use with like, you know, the email integrations you guys have done, the Slack integrations, etc. Um, and as you look ahead, to what extent are you looking at the integrations piece as it's mostly done because now computer use is so good versus no, we still want to invest very heavily here because there's more capability when we invest in these deep
[10:43] integrations. I think for a long time, you know, chatbt has had connectors, plugins, whatever you want to call them for a while. Um, the model was not always the best at using them. I think in the past 6 months or so, suddenly, much like with computer use, there was like a flipping point. Suddenly, it was like, oh my god, we need to give make sure it has access to anything that someone would want to give it access to because that just each incremental connector or plugin that we add like multiplies the power of this product. is
[11:13] we focus a ton on latency, reliability, etc. to make sure that like when the model uses this connector, a it has access to as many possible tools as third party service provides. Like we're reflecting all of those, b like they work and they work reliably and c they're fast and token efficient. Um, and I think we'll continue doing that, but in the cases where we can make a better connector and for for those specific products, like why not, right? Like we're just giving an additional power to the model. And I'm curious when you look at dog fooding internally like you've been describing, how have you
[11:43] seen uh the shape of work start to change, what is the thing that you sort of know internally as a fluency habit or as a learning habit or something that has been really helpful inside this company that you would want people to know? But I think one thing that we at least for me this is the my personal trick is try it like you know you you come to work or or you're home wherever you're like looking at your computer and you have something to do you're like oh I need to update a presentation or I need to send someone an email or I need
[12:13] to do whatever and you're like natural instinct is going to be to go start doing that >> and I think something that was very >> important in my journey was like to just be like no I'm not going to do that I'm going to try [laughter] it >> in codeex or chatbt Yeah, I think that first step of going from like doing everything on your own to starting with automation I think is is a big gap. >> We've set up our environment to be very codeex and chatb native, right? So the whole thing kind of works and we use normal third party tools for plenty of
[12:43] our work, right? It's not like we've made everything. We just we've made sure that they're connected well. >> And so, you know, people will be asking questions like, hey, what why why do we do a certain thing or like how do I connect to this thing? And the answer is usually like just talk talk to it like press the voice button ask. And it's it's um it's really interesting how you can kind of get through like cut through the noise, Slack, Google documents, like all of the stuff that people have worked on and kind of get the full picture. I think it's interesting to watch or you
[13:15] know everybody here uses codeex or chat work now for most of their job and for a while that wasn't yet true, right? Um there was a period between I don't know the first half of this year that like function by function was making that kind of grand transition to how they worked and for each one it was like a slightly different tipping point. Um, coding was obviously the first one, right? Um, and then who was who was
[13:45] second? >> I heard legal was >> early because there's so much material, right? >> Um, and I think it's like a combination of what types of work the models have been good at at what time, but also then what kind of connectors and file access, like what does your context look like? The coding one was the easiest because everything was just like a local CLI, >> right? >> Yeah. >> Like you don't need a Git connector.
[14:15] You'd need the Git CLI. You need the GH CLI, right? >> And so, so we didn't have a great connector story in I don't know, January, December, right? But you would code with it. And then it's like legal legal's got local Microsoft documents all over the place which the model could read. And then it was it was kind of during this first half of the year that like connectors got really good and then computer use got really good. And so depending on the context that was required for any given role, it kind of like tipped tipped that over the edge
[14:45] when you know this critical mass right >> when the context relevant to that role became generally available to the >> generally available in performance and yeah so it was it was actually more that than like I guess the normal stuff that you would expect about how technical a role is or something like that. >> That's a really good point. In fact, one one of the things I've been taking away is it feels like the question of whether a role is technical is becoming less and less of a question I find useful.
[15:16] >> Mhm. >> Yeah. It's been one of the things that has been super cool to see in the past few weeks. like I've been talking to or spending some time with people in the finance team and business teams and um they've been moving away from Excel in some places to like create these using chatb sites like effectively apps but then they're sharing across their teams and like iterating on over time and like making new versions of and updating for like new use cases and things like that and so for all intents and purposes they are you know developers in that in that
[15:47] realm and like they don't have a lot of the tools it was like kind kind of anxietyinducing for me because it's like okay they they don't have any of the tools that we have in order to make our jobs easier like git and version control and stuff they're kind of just yoling it with you know the what we've provided them and so there's so much more to build in order to like make their lives easier but like the fact that they're able to do it now itself is I think super cool >> I think touching on sites is interesting I know there's been a ton of investment in that space for you guys in the past call it month or two do you see a like
[16:19] if you look at knowledge work and kind of where knowledge [clears throat] work happens. Does sites feel like it's a big new general purpose lever around knowledge work? >> I think there's this idea of like artifacts that are created. I think historically, you know, docs, slides, sheets, >> um, PDFs, etc. With AI, like maybe there's like a new like AI native artifact in the future. Um, I think the thing that sites does a really good job of is like sometimes the thing that you're trying to express doesn't fit cleanly in any of these buckets. like
[16:49] maybe it needs to be interactive, maybe it needs to be persistent in some way, etc. And maybe you because you're quote unquote nontechnical might not even know that like you know the requirements but you don't know how to articulate them in a way that like an engineer might but you can just describe your problem to Andrew's point to the model and it can create this application for you. Um and it fills that gap of like okay you could never do this in a sheet or a doc or slide deck um but now you're able to do this with a slide. So, so I do think like for knowledge work it enables people to be much more expressive than they otherwise would be. So I think
[17:19] that's going to be a huge unlock. >> It feels like in most situations most job roles there's some degree to which it's iterative. I would be curious to what extent you have seen differences in user behavior when people are say in work versus people in codecs and how they are starting to use perhaps a more rhythmic or iterative workflow versus maybe a goal oriented development workflow where you just have to get the code to production and it needs to be shipped. >> Definitely more similar than they have
[17:51] been. I think you know over the past number of years software development has become more iter iterative incorporated that's a fair yeah >> um so I think like the shape of if you were to call it the product development life cycle of like an internal tool that is being built as a chat site looks a lot like your best practice on how to give something to customers right so everybody's kind of now surfacing something to some sort of customer right you know their HR might make a site to
[18:22] help us identify talent and like that we are the customer of that product that they are maintaining. So everyone's sort of starting to maintain a product of you know that captures their type of expertise for another audience. Um I think in many cases their cycle is faster because it's a you know smaller internal audience right they don't worry about >> backwards compatibility of a billion chat GBT users and so um sometimes they teach software developers here a lesson on [laughter]
[18:52] >> you can just iterate and >> of course you don't have a billion users but >> yeah that's an interesting point as well. What is it that we should be betting on in terms of skills? what is it that we should be betting on in terms of uh trend lines that we can keep an eye on and continue to expect to improve that are predictable enough that we can say in 6 months X or Y is going to be a lot better. I think that creation of these work products for lack
[19:24] of a better word like sites or the tools or documents or whatever is going to continue to improve the model's capability to do it and that's on a mult multiple dimensions. So like one is just like the quality of the outputs right two is like the ability for it to understand your true intent like what you mean the ability for it to pull context in to site sources accurately whatever all those things are going to get a lot better. Um, I think what that means is then what you're bottlenecked by is like ideas. >> The ability to like think about what to
[19:55] do next to have the agent come up with something to to Andrew's point, validate whether or not that's useful with either yourself or like the people you work with and then go back to the agent and ask for like iterate. So I think the skill of like working with the agent is the thing that is going to stay constant and like can you continue to get better at that and then you can assume that each iteration the quality will keep getting better and you know maybe the iteration cycle gets shorter and then you can do 10 things in parallel where previously you were only able to do five or so on and so forth. I think the the
[20:26] other side of that is that it has become so easy to make things that taste on convergence is another bottleneck that we sometimes have which is like there's there's a lot being made at any given time. >> Yeah. >> Right. We want to ship as focused as cohesive and coherent a product as possible to help people >> solve their stuff. So, how do we take that thing over there that looks cool and those three things and that thing
[20:57] and that thing and put that together, mold that into something that makes sense, is coherent to people, is tasteful, right? Cuz now it's like you can ship anything. We could ship every single product that's ever been >> You can you should Well, that's another question. [laughter] Um, I think that's that's one is really having clarity of thought of like >> what what do we really need to do here? >> And that like both of you called out mental skills like you talked about the
[21:28] fluency of working with agents. Um, and you talked about decision making and kind of how you make good decisions with good taste. Um, I think to me one of the things that I've been reflecting on is when I was earlier in my career, there was this obviously there was work that was intellectually rigorous that I had to do, but there was also lots of busy work that would just fill the dates as well. And now all of that is gone. >> Like largely the busy work is something I can give to an agent and it just gets done.
[21:58] But that means that the question then bounces back to me, what am I doing with my time? How am I maximizing my agency? Do I have the ideas to actually give AI what it wants? Am I making decisions that are tasteful or am I pushing slop? And it feels like that's a question that we've all been posed. There's a level of slight discomfort on having to, >> you know, reinvent yourself over and over again. Um, at the same time, you know, I I grew up like a software engineer, but
[22:30] also a designer, and there were always things that I saw in my head, wanted to make that were beyond where I was on the engineering side. >> Mhm. >> And I feel somewhat liberated to be able to deliver things without the busy work. Right. >> There's no inherent meaning in that. It's the meaning is in this the end that the thing that you created. One of the things I'm curious about is how you guys think about uh token efficiency. It feels like we like, you know, maybe
[23:00] April or so token maxing was the thing and everyone was talking about token maxing and token leaderboards and now the pendulum is swung back. I've been having a ton of fun switching the usual assumptions. So, a lot of the usual assumptions are you pick the smart model as the orchestrator and the dumb model is the executor. I've been playing with switching it and having Luna be the orchestrator on extra high and having Astra be the executor with sub agents and then I just poke Luna on agency and not uh under scoping what I'm building
[23:31] with another Astra model every now and then but it doesn't get the full context window so it's not chewing up all those tokens and then the super long running orchestration thread lives in Luna and I found that that is tremendously token efficient and it still gets me a lot a lot of the way to where I want to go. And I'm I'm curious like, have you heard of that? Is that a surprise to you? Do you have other tips or tricks for how you're token efficient? Um, yeah, >> I have heard of this. I Yeah, I've heard I've heard more and more of it. Um, >> as this pressure comes in,
[24:02] >> not just because people didn't realize they should be token efficient, but because as the models get larger, it kind of pushes you to be more token efficient, right? Um, per unit of delivery. Um I mean I think we've seen some real leaps in efficiency on Astra, right? There are cases where Astra is just an order of magnitude more efficient than our previous models and that's phenomenal. Um and the code mode uh stuff for computer use makes it even
[24:32] better. Um, I think we're also we're also trying to take some of the things you described in, you know, I've got this multi- aent setup where I'm picking this model for this and this model for this and this model for this and baking a little bit more of that into how models work by default. >> No, I got it. >> And [laughter] and you know, we want to be able to bring that to everybody, >> right? >> Yeah. Yeah. And then like Andrew mentioned like we've been doing a lot of work on kind of the the details like making sure that you know the skills that we're publishing are the most token
[25:02] efficient for the model. Making sure that the harness and how it's using computer use with with things like code mode is is is working in a way where like we're using as few tokens as possible. So it's something that we want to continue pushing over time. Like I think the what you were mentioning about Astra doing the execution isn't surprising to me because one thing that we've seen is like when we're when you're purely measuring like tokens per per you know unit of time it's not the exact measurement that you should be thinking of even as a costconscious consumer because really what you want to measure is like how much did you spend
[25:32] to do the unit of work that you wanted to accomplish. Exactly. And so like you know while while Astra may you know be more expensive on a per token measurement it's like much more efficient. And so like what we want to do is like figure out how people can validate that like the ROI not necessarily just measuring like how many tokens you're spending and multiplying by price. >> And I get people who say well my employees use it for email uh they use it as an answer engine uh and they use it for coding. And then there's just
[26:03] sort of this long pause and then you're you're sort of supposed to kind of fill in and say and then the next thing is how do you think about building an outcome focused product in a world where knowledge workers may not have the ability to cleanly articulate outcomes at the start of the process but may get there over time. like I I often use it to go through data, find groups that are not quite retaining on the product or you know getting onboarded fast enough and just like
[26:34] giving me clarity around that >> you know jobs that would take days beforehand to like you know find the data tables to write the SQL queries >> to build the dashboards and say oh no that was the wrong query it doesn't tell me anything so now I got to go back and do it again and nowadays I can you know get a report that's kind of like hey like this is an area of opportunity for us and I share that with my team and >> I think that you know we play a lot with how can we push it in different areas and it's not always you know ready here and we're doing it a lot for our
[27:05] research of what can we productize and so I I realized that you know people don't have the time to play with it the way that we do and we've got to bake some of this in so the other day we released a a data plugin for example that helps with a lot of these these um data questions and just really streamlining like how reports are made and things of that nature. Um but I think this is something that we want to to play with, right? Is you shouldn't have like if we if if the model knows how to answer any question you type into
[27:36] it, then you shouldn't have to ask every one of those questions. Like we should be able to provide you a 9:00 a.m. report on, hey, here's the things that you're not asking, right? Yeah, >> your job is retention and you're not asking about these groups, but they are actually a huge blind spot for you, right? Like >> they're turning, right? Exactly. Like that's the type of thing I think we need to get right next. >> Yeah. Yeah. And and proactivity is sort of taking the context problem and
[28:06] pumping it up on steroids because then you have to not only be confident of the context, but also confident of the insight and confident of the directionality of the insight and confident of your degree of confidence so that you know when you're coming out and saying something, you're able to say, "I know this is true because X, Y, and Z, and therefore I recommend C action." >> Yeah, >> it's super interesting. It's an area where you want to be careful, >> especially with efficiency, >> right? You can't run this thing and just
[28:36] do inference all night, super expensive in inference for 12 hours while you're sleeping and eat up somebody's entire, you know, usage bill >> for, you know, three lousy insights that are kind of spammy, right? >> So, this is something that we that we are thinking about a lot and getting right. >> Yeah. The token efficiency piece comes in there. The model choice I'm sure comes in there. >> Um >> the bar is generally higher for how good we have to be when the user hasn't explicitly asked question. >> I think of voice as an interface of
[29:06] course. When you guys think about model capability, computer use where these things are going. >> What are the trends in interface design that you guys are paying attention to as you think about where say codecs or work could be expressed in a year or so that like we wouldn't even think about now? I think I mean you touched on voice so I think the one to to call out like >> I think the this is one where like you see early adopters so far ahead of the curve of everyone else like when I walk around the office
[29:37] I see some people with like six different mics on their desk and that's what they use to communicate with you know codeex or tragedy media work all day but that's clearly not made it um out to everyone and I think part of that is you know the product needs to do a much better job people discovering that this capability even exists. Part of it is like we need to bring people along on like changing from you know typing or tapping to like speaking and like how much more fluid and and frictionless that is as an interface. Um, so I think we have a lot of work to do there in
[30:08] terms of like making it feel like this is the next thing like codeex or chatbt work is or chbt is with you everywhere um because you have access to this voice interface for people who get it they truly get it but we haven't done enough to like show people that that's possible. >> So even beyond like building a net new interface I think voice is one where like we have it already but like it's about how do we bring it to everyone. >> Yeah. >> I think the the interesting other side of voice is that voice is by far the most efficient way to get context in and to communicate
[30:39] information, >> but reading is much faster than listening. >> That's right. >> And so I I think like playing with this combination of voice in and like generative visuals out is very interesting to me. And we've done a little bit with our visualized skill and our generative UI, but I I don't think we've >> done anything big here yet. >> Yeah. Um, there are many times where I want to be speaking into this this product, into this model, providing context at a speed I can't type,
[31:10] >> but not having it speak back to me, having it show me things and, you know, ask me questions. I'm a very visual person, right? Um, so we're we're looking at a bunch of that, too. >> Yeah. >> Yeah. I I I think that there's more to do here, too. This the the size of interaction needed is not always the same. Mhm. >> And so as we think about things like token efficiency, even >> sometimes a single question can unblock like a ton of token usage and a much better outcome. Just some clarifying
[31:41] questions. You'll see Astra asking more questions than previous models. But as tasks get long, the ability to direct questions to different devices you might have, >> you know, like I don't, you know, people talk about coding at the playground while their kids are at the playground. I'm like, well, I don't I don't love that idea. Like, I want to spend time with them. >> You want to be spending time with the kids. Yeah. >> But it's also very frustrating to come back to my desk and see something done in a way that >> isn't wrong, but like one question could
[32:12] have made this better. >> Yeah. >> Like just send me the yes or no out while I'm out. I'll hit, you know, I'll give answer and like unblock you. Right. >> The same way that sometimes somebody, you know, sends you a Slack message to just get unblocked, right? >> Yeah. >> Um, so we'll do more there. Yeah, I think and that's another example of like smart proactivity it seems like where it's like learning to ask the right question. Another workflow pattern I've noticed in myself is I am increasingly reading the intermediate messages that are printed and I am aggressively
[32:44] shaping the run of the agent as I see those me messages shape out. Sometimes it's like I'm directly responding and saying no no no no that's wrong do this over here differently. And sometimes it's actually taking a print out and in the middle of the run handing it to another agent saying, "This doesn't make a lot of sense to me. Can you give me fresh eyes on this?" And then >> the runs are long enough I can cycle that back around and say, "Okay, this is how I think we should proceed." >> Do you guys see sort of mid agent work
[33:14] activity as something that's an emerging category across a wider user base? Is that just me being weird as like a a power user or what's been your experience? >> I do think that as the scope of what the agent can take on gets larger, so like both like the value of it and then also the time it takes to do it, >> like we really need to nail the experience of like what does it feel like to interrupt or interject in the middle of a roll out. Not only for the things that you're describing where like maybe the agent is doing the wrong thing, but maybe the user is wrong. Like
[33:44] I've had many situations where like I ask it for something and then 20 minutes later I'm like wait a second that's not >> that was wrong. >> Yeah that's [laughter] not at all the thing that I want. I need to go back to it and make that and that needs to feel like really fluid. So I think that's somewhere we need to invest in more but definitely something that we're seeing where like people are like starting to learn that this is birectional the entire time not just like >> question response question response. Um I think there's probably much more we can do in the interface to make that clear. >> Yeah. No, that makes a lot of sense. Another pattern that I've observed in
[34:16] myself, we talked about iterative work earlier. One of the ways that I have found and that others I've talked to have found that they actually work with AI is they use AI almost as like that uh bouncing ball idea generation machine and you're kind of bouncing. You're like, "No, that's not it, but I wouldn't have thought of that. No, that's not it, but I Oh, that's it." And like you have to go through a series of nos and the nos help clarify your intent. Um, I don't I don't know there's an interface for that yet, but it feels like that's a
[34:47] very different mode for me than let's get work done because I have clear intent and I need both in certain situations. Like in one case it's do I understand what I want and in the other case it is okay now let's get it done. There was a very similar conversation I was having I think it was yesterday about the idea of like randomness or noise. >> Mhm. that like this is a parameter that you can if you're you know using every knob on like the API you can set
[35:19] parameters like this but if if you zoom out and you're like hey I've got this plan that I want to be achieved whether whether that's with a model or even with a company of people right there's a certain amount of like noise and jitter and experimentation that you accept that is somewhere between zero and like chaos >> because you know that It actually helps maximize. >> Yeah. >> Like global maximize, right? Like you're not going to get stuck in like a local maximum situation because you've got a
[35:50] little bit of play where it's like, oh yeah, these people over here are trying something new that maybe might be a waste. And I I feel I feel very similar about these models where I'm like >> I want you to follow the instructions, but I want you to have like a little bit of a little bit of playroom so that when we're talking about this stuff, like yes, it might take a little bit longer, might spend a little bit more, but you're kind of outside of the narrow band. I don't know know exactly what that looks like, but I'm with you on this. >> Yeah, >> that resonates for me. >> I would be remiss if I did not ask cuz I
[36:21] get asked it a ton by leadership. if they're rolling out AI in an organization almost always the initial few months is it's very much like okay we've got some people who are adopting this and they're super passionate we've got group B and they're adopting us kind of figuring it out and group C is like I don't know what is this and when you talk about group A which is the passionate adopters almost always the story is one of individual productivity and the next question I get asked is what about teams and so my question to you is when you think about how AI has
[36:53] kind of tipped over these departments inside OpenAI over the last 6 months. How has that translated into team level productivity or team level uh ability to get more done? I think in all cases it's identifying a process or a problem that your team faces maybe on some repeated basis and then that individual the power user the early adopter whatever them going after and figuring out how can we make this better with AI. I think like the biggest hero cases we've seen is like someone's like, "Okay, we have to
[37:23] update this financial model every two weeks. I'm going to make that into a chatbd site. Now other people can do that too. Now it's more expressive, etc." Or I'm going to create an automation that put publishes all of our bugs for our non-retentive cohorts into Slack and now everyone on my team can start pulling from those things. And so it's like getting beyond like you creating an output for yourself and creating something that your team can see and get value from because I think for like you know leaders in organizations who are like excited about deploying AI. I think when the early
[37:53] adopters are encouraged to do that then people also learn cuz they learn by seeing like oh that's something that I could have done and they also get value immediately versus like trying to run you know a training or something where it's like much more abstract. >> Yeah. Yeah, that makes a ton of sense. Okay, last question. I'm really curious. When you guys are using AI daytoday, what is I most people I know have sort of an NP hard problem, a problem where
[38:23] they just keep like running AI at it and eventually it cracks. Eventually the scaling laws come through and there's something that works. Um, for me, I had one of those moments when a I could finally read my six calendars and actually tell me what was available in my day and not just read two of them and give me false windows of failability. Uh, what is that for you guys? What is that moment that was really difficult? And has AI crossed it yet? Have you had that moment where it's like, oh yeah, okay, this was really hard. I never thought I would get there. I thought it
[38:53] would be a while and now it hit it and I'm surprised. There's like a pet one that I do every time there's a new model and it's it's hard because it's it's not just technically challenging. It's a it's a coordination problem which is >> you know we we have this crossplatform app >> and I have this dream of someday like someday with any given powerful model you should be able to maintain the like absolute platform native on all platform right you know you're doing like old style objective C UI kit on or [laughter] app kit on Mac right you're
[39:24] like yep like we're win UI3 on Windows 11 right and this is a coordination problem because we're moving so fast that it just it has been impossible to move all the features in lock step. >> Like if if we were moving a tenth of the speed, this wouldn't be as much of an issue. But we use crossplatform because of the speed, right? And someday like we'll get back to it. We'll get back to it. We'll be able to fully rewrite all of these apps at exactly the perfect thing for them and keep them all in lock step and make a Rust version. I don't know. Um [laughter] we're not there yet,
[39:55] but um we're getting a we're getting a lot closer. I can tell you that. I mean, I I have the Codex app rewritten like every time we have a new model, I'm like, >> "Hey, try it try it in GPUi with Rust, you know, like >> and just see what happens." >> And just see what happens. And uh >> Astra got Astra got a pretty good version out on like several at a time. So, >> okay, >> we'll see. >> Yeah, fingers crossed. Okay, >> yeah, I mean, I guess taking the other side of this is something that just got crossed like for a long time. So, I was a very early adopter of like connecting AI to like everything in my life either
[40:26] via the browser or connectors. And I have an automation that's like um you know, how can I save money in my life? I'd say like probably 2 months and it never found anything useful. Um and it's actually quite complicated problem cuz it goes through my email. It is logged into a bunch of stuff via the browser. It has some connectors, etc. >> Um you know, it's like maybe it asks me to cancel subscription or whatever, but that's it. And then actually like last week it found a pretty significant error in my taxes >> in your favor. >> In my favor. >> Oo.
[40:56] >> That ended up saving you know thousands of dollars and I was like this is something that I would have never have figured out. Like this just would never have happened because the amount of coordination required to figure this out. >> It was huge. Like I had to remember something that happened multiple years ago and pulled that forward and all of that. And so for me that was like pretty eye opening again of just like >> the more context you give it the models have gotten to a point where like >> they very much exceed what I'd be able to do on my own and so we I need to continue pushing in my personal life in that direction. >> Yeah. Yeah. One of the ones that Astra
[41:28] crossed for me that made me chuckle. Uh so a few months ago I was in Seattle. Tibo had flown up. I had a brief chat with him and at the time I said, "Tibo, make me a model that can make Legos cuz I love Legos. I want a model and and like it's saturating out at like 50 pieces, 10 pieces at the time." >> And now, yeah, I can hit a thousand piece Lego set. It's not a problem. >> It's It's wild to cross the physical. Yeah. >> Right. I remember a few models ago when I did like the first um book for my
[41:59] kids. >> Yeah. >> And I was stringing them together. I was like, "All right, you're going to write a script and then you're going to take the script and you're going to write an image genen prompt for every page and you're going to pass that into the long ago. >> It was not." And now you can kind of just say like, "Hey, make my kids a book about themselves, like a kids book. Go get it printed and it will." And it's just like once things enter real life, you're holding them. >> It feels different. >> It feels different. It does. It does. Well, thank you for the time today, guys. I really appreciate it. It's been a lot of fun. >> Yeah. Thanks. A lot of fun. Uh, as a