The Infrastructure Nightmare Nobody Is Talking About

[00:00] I would love to hear from you like, you know, tell me about Emma. Tell me about how you're here at OpenAI, what you're doing, etc. >> So, my name is Emma. I joined OpenAI back in 2023 to lead um the data platform infrastructure engineering group here. Um so, what we do is anything that involves data systems. That got some bowels of creating any product or any research system or etc. There's a lot of data systems that underlie it. So, my group actually take cares of all of that. So, whether it's big data, so if you want to do analytics, if you want to crunch some
[00:30] data, um or if it's like streaming, like, you know, streaming processing or like event buses, or if it's ML infra, like you want to do some ranking algorithm, etc., like feature stores. Um or if it's and then we get to even like higher-level abstractions which builds on top of this, which is like piping data from one system to another, making sure that's done securely in a scalable way, um as well as something unique to OpenAI, which is like preparing training data um and eval data, and making sure the system's consistent in kind of that kind
[01:00] of load. So, that's all on data platform. Um so, we're we're very low-level as a team, like product teams that sits on top of us, research teams that sits on top of us. We really serve every team in the company, um whether it's like go-to-market, you know, trying to sell customers, understanding what customers are doing, or finance trying to close the books, or like doing M&A deals, understanding what's going on. Um or even HR, um you know, but also like, you know, for the business itself, right? Building the product like personalization is a feature that
[01:30] is very much reliant on data to crunch the um the information used for memory. Um really every product touches us in some way, whether you're logging into OpenAI or API um or integrity. It's all plugging into data systems underneath. >> So, I I I guess maybe the first question I would have is how has that world, how is your job, how has the team's role shifted maybe in the last year, maybe in the last 6 months, last 3 months, things keep moving really quickly and I'm
[02:00] curious how you feel like things have evolved for you and the team. >> Yeah. >> Just as you see the models get better and sort of how that's impacted the data layer. >> It's all very new for us, too. I would say a year ago, it wasn't that different from, you know, artisanal software engineering of the of the year. Um but really in the last 6 months or so, things really started to accelerate. I think Codex got a lot better. Um the models are also like getting much better much like very very fast. Um so we're seeing like really rapid changes in the
[02:30] last 6 months and I don't think this is the end of it. It's it's very much the beginning and I think we're going to see even bigger changes coming up very soon um that I don't know if anybody has a clear idea what's going to look like in 6 months. Um but in terms of on the platform teams like ours, um we we we kind of see what's happening in terms of accelerating our own work, which is the exciting part. We're also seeing how how much it's accelerating everybody else around us, which um if if like different parts of the
[03:01] organism are growing at different rates, then you can imagine that there may be like actual problems that are caused by um using a lot of agents and autonomous like coding tools. So >> Yeah. That makes a ton of sense. >> Yeah. I I can dig into this as well. But uh in terms of just on the on the data infrastructure front in terms of how we're accelerating things, um we're using Codex as well as like agentic tooling for things that are just like
[03:31] uh we didn't really imagine that we can use like agents there before. For example, our release process um So okay, so I'll back up a little bit. So we have a lot of um basically proprietary LSS software that we use and we basically patch those um like underlying software pieces up um every so often and then we release that and then we use that in systems, right? So, we have our own release process for these underlying components. Um and it was it's a very manual process to test, validate, um make sure like to
[04:02] run all the like, you know, cases through, and then to promote it to like the next day. Let's say from staging to um canaries. And then from canaries to like prod, right? And then we have a lot of different packages like this, you know, dozens. So, every time we need to do a release process, it's very manual. People have to like watch the jobs, like it's couple hours, sometimes days, and then you have to remember to like check all the results and promote it. So, now that's all controlled by the agent. The release process is the agent itself. It does everything um that I mentioned autonomously. It will ping, you know, us in Slack to let
[04:33] us know what the status of things are. If something is is broken, it'll also like try to do its own triage and give suggestions on what's going on. So, we're completely hands-off. That's like hours of time that people are spending like every day. Um and it's done a fantastic job, probably better than humans can. Uh so, that's one area. And the other area is like um we've been using a lot more um we've been just capturing like the the knowledge the specialized knowledge that we have as infra teams into more skills um and more kind of
[05:03] like capture in the agent itself um so that users can take advantage of that, right? So, um So, before there was just like we need to build a lot of guardrails to like you know you you know users can do this wrong and then it'll end up being like a really long-running job, it'll break. But now we can code that in skills. So, then when users are using that, it actually is like extremely smart about what to do and how to um debug. A recent um kind of example is um we launched a skill for users to help
[05:33] export data for training purposes. So, that used to be a very manual process. It's going to take hours. There's a lot of sharp edges in terms of what could go wrong. And um with the new kind of like Codex-assisted like um you know, skills kind of way of doing things. Uh the user launched this job and then just went to sleep and then the agent was running in the background. It actually found some issues. It got blocked. Um and then normally if, you know, if a person's blocked, you have to ping us and then find out what's going on. The agent was actually able to
[06:04] go into the internal systems itself because it's all one big code base and figure out what might be wrong there. So it went into like four or five different internal systems and check the code there and be like, "Hey, why is this Why is this breaking in this way?" Um and then it also pinged us in our support channel, but this was midnight, so we didn't see it. Um and then and then it went and actually like three layers deep, it found like this tiny bug that existed three layers deep. It went and somehow patched it and fixed it. Um and basically worked around it in a
[06:34] way that the job could continue. So by the time the user woke up in the morning, it was completely done. Like no conversations need to be had. Um so that's kind of where we're at right now. Um I can give you more examples on how we're using like kind of agents, but there's like more kind of I think more like kind of vanilla cases, like I think everybody's using this way and that we have some like um kind of uh data tooling um that we launched for all users. So kind of think about um autonomous dashboarding, autonomous notebooking, you know,
[07:05] uh and in order to make that really good, um we take a lot of user feedback and they will just like put in Slack and say, "I want this feature. I want this feature." So we basically have the agent try this. We have Codex go pick things up um and really try to do it on its own. And we have created tooling so that Codex can plug into like the full cycle. Like it can it can it supports browser use, for example, right? It can load the whole DOM. It can figure out like it can do the clicks and figure out what's going on and it can validate its own behavior and its own um
[07:35] uh kind of like fixes. And so it can do the full loop. So essentially uh for many tickets, not all, I would say. It's not like quite completely there yet. For many tickets, it takes a like a user like a comment and some description. And then Codex will just execute on it. Full stack, full loop. And then it'll come back with a PR with all the the proof of how it fixed it with a video and everything. >> Wow. >> And then, yeah, so it's it's really accelerated a lot of parts of engineering for us. Um, and then I'll talk about the other piece that I mentioned earlier, which is
[08:05] like, okay, what if the acceleration is uneven? And I I think I'm seeing this a little bit right now. I think it'll even out longer term, but right now this is what we're seeing is that if you are on a team where um, the surface area or or like basically the spread of um, the damage if something breaks is like a little bit more limited. Let's say you have a like you're you're still building like um, like a project that um, hasn't been launched in production yet. You're
[08:35] iterating on it or it's like um, you know, only launched alpha customers. You're iterating on them very, very fast. So, if you think about like a lot of teams, they can go very, very fast. They can vibe code completely right. Like Codex can just like turn out like features by features. But if you're on a very like like a like a like um, like a root level team, um, like an infrastructure team where if I change one thing, it'll like basically affect like thousands of different teams, um, it's harder to go uh, just complete vibe coding. We still have to have a lot of guardrails in place. We still have to do
[09:06] a lot of manual checking to make sure that like it doesn't, you know, affect things um, in a way that will like, you know, harm a lot of users. And um, uh, and so so like I don't think the model is like quite I think it's going to get there, but it's not quite there yet in terms of creating like 100% one shot at like perfect code. So, we have to iterate on it a lot more. So, you can imagine it's like a lot of users who are sitting on top of our platform are vibe coding like all these new applications and things. Um,
[09:36] and then they land on our platform which we're in charge of running, right? If you imagine like a Spark job or like a Flink workload. Um, if they break, often times we we we will check with the user, and the user probably like has less of a clue of what's going on because it's Codex generated. We've had instances in the past where like users are like, "I don't even know what Flink is. I don't really know how how to use it." Um, but you know, but it's it was working, I thought, you know, so I can't really help you. You guys should figure out like what's broken and fix it. So, there's almost like a transfer of
[10:07] responsibility and load onto a lot of these platform teams who are running the code. There's more code, right? There's a lot more code. It's being created really fast, and the teams are responsible for ultimately running it successfully and making sure that it's not breaking. Um, they're inheriting a lot of the the burden. So, so what are we doing for that? So, um, at OpenAI, we're also thinking about how you know, we don't have a good term for it. So, so you know, sorry for I'm using my own terminology, but it's like almost like a defense in depth kind of um,
[10:37] kind of strategy where it's like, okay, on the code review side, um, this is kind of the interface that we have right now. We will evolve from here. But, how do we make autonomous agentic code reviews happen for each team so it captures our specialized knowledge to make sure bad things don't happen? Right now, we still have to spend a lot of manual effort looking at code and inspecting it. Yes, we're using Codex to inspect it, but it's it in order for things to to really be tied down, we we really still need to do that. How do we encode all of that knowledge and know-how and all of our
[11:07] run books and all the past incidents and all of that and have this agent kind of do that work for us? Um, I think some people, again, it's an unsolved problem. These are all unsolved problems. Some people are like, okay, why can't we encode that in like agent MD files and like the the the agent that's writing the code itself, right? Why doesn't it just know to do the right thing? Um, it's isn't open question, but in my perspective, I do think the incentives will always be somewhat misaligned. That's why we have, you know, people who write code and people who review code, and they're separate people, you know? I
[11:37] I think it'll be it'll be hard for the model to like juggle those two things uh consistently. I think there should be a multi-agent architecture for for this kind of thing. Um and where a separate agent will will look at the code and review the code and um kind of like, you know, it's kind of like a code owner's like plus plus situation where it's like if this code touches like different teams, then the different teams, different agents will come in and review the code. And make sure that it's like, you know, tip-top according to that team's like specifications. Um but then also every
[12:07] other layer, on the infra operations layer as well. How can we make that autonomous, right? If we if we find like a bad workload, how can we sequester that a lot faster in an autonomous way where it doesn't have to like page us? Um we've also seen incidents where like a user, again, um you know, would could could vibe code something that um they really didn't intend to. They they like flipped um feature flag from like uh you know, like uh that that they didn't mean to turn on, and it just took took the entire Kafka cluster down, you know? Those kinds of things like how can we build systems
[12:37] that can very quickly capture these kinds of erroneous usages and and and do something about it. Because I don't think the world is force users to write manual code or like review their code completely manually, understand every single thing. I don't think that's where we're going. We're going to like full autonomous mode, where it's like code producers and code reviewers and code deployers and code um and also like just people running It's all autonomous. It's all agents um every layer. How do we get there um safely? And so I think
[13:08] the models are getting really good. So I think we'll get there pretty fast. So it's a very exciting world where we're in right now. Um yeah, I'll stop there. >> No, there's there's a lot to dig into there. Um I could pick at probably any number of half a dozen threads. One of the ones that I think popped out for me is you talk a lot about sort of framing your world in terms of the kinds of autonomous problems that are solvable at the platform team layer versus the kinds of autonomous problems that, you know, teams that sit on top
[13:38] app teams might might engage with. And maybe we can dig into that a little bit because I think that you have a really great point that I've seen chatting with other companies as well where they feel like there's this huge separation and difference between what the front-end app teams can go after with perhaps less less need to be completely correct on code versus where platform teams have to be near 100% correct and need to insist on holding that standard even as they're supposed to speed up. And so maybe if you can dig
[14:08] into a little bit more like what are what are the pieces of this where you feel like we have a handle as a platform team on where autonomous agents are at now uh and then maybe we have line of sight to some problems that we think are linearly solvable uh for platform teams and then from there maybe we can blue sky a little bit and you and I can hear a little bit about how you're thinking about sort of the longer run. And I And by the way, I fully realize agents are going so fast the longer run is probably like 6 months or something. Like this is the world we live in, but I I'd love to hear your perspective.
[14:39] >> Yeah, I mean it's a really tough question because it's so open-ended right now and nobody knows exactly what to do. Um I can tell you what our thoughts are um and uh all of these things are are more conjectures right now than things that are like you know, close to completion. Um but I I really do think we need a better at least at you know, as as one of the first things that we should do is um a a better like code review harness almost where it can also plug into
[15:10] different tool calls and also um plug into different knowledge bases so that it can perform autonomously what a human right now needs to do. Uh we we have like a a code review bot, but it's not um, kind of specialized. And I think it needs to be specialized because, you know, there's just so much vast human knowledge that goes into a code review that's not captured by like a generic agent that just has access to like mono repo um, or even like the tools. Uh,
[15:40] yeah, so I think iterating on kind of a framework for that will be really important. And that's something we're looking into as well. Um, but I think uh, yeah, the like to to your question is like, what is the difference in this world between like, you know, these teams are like, you know, maybe building front ends and, you know, apps that running on platforms and the platforms itself. Uh, yeah, I I think until we solve this problem, the disparity will get very great. That's the thing with AI is
[16:11] it's very much power law dynamics. And we're seeing this, you know, in the industry as well. Like people who are accelerating with HI AI can go much much faster than um, teams without. And so, even within companies that dynamic needs to be noticed and dealt with or else it gets um, it can really harm a company's like, you know, uh, ability to like, you know, just run its business. Um, so, I guess I'm sorry. Like I don't have like really great answers to be like, oh, this is what we should do to like solve it. Um,
[16:42] but I do think it's something that we should pay attention to um, and invest in. So, on on our end, we're investing in these harnesses, we're investing in like autonomous like operations underneath. But I think there may be even more of a fundamental shift in how things get operated. I can imagine because of power law dynamics um, that like a lot of like um, uh, like, you know, higher layers, you know, higher layer abstractions, teams on top will just be like um, extremely agentic.
[17:13] Whereas the the platform layers um, increasingly need to like uh like support more and more load and therefore it needs to like temporarily grow as well in order to build a system to catch up and once it catches up then we can grow like together. Um but right now it's almost the scaling laws of the upper layers are like AI scaling laws and the lower layers are human scaling laws and that's not sustainable. So how do we get the lower layers to be into in get into the AI scaling laws as
[17:43] well? We need to build those platforms and I think we I think we're super focused on like code creation and feature creation and not about operations and running right now and I think that's the next level lever that we need to pull. And I think we're we're working on that right now. So hopefully in another 3 months we'll have a better answer for you. >> No, well I mean we can always come back and we we can chat some more. Um You know another thread maybe that I a lot uh you mentioned sort of the whole concept of someone sort of typing in Slack something they'd like to see and
[18:13] the agent picking it up and running with it. It got me thinking a little bit about different pathways that uh agents unlock for communication between teams and orgs. Um so like if we're talking about communicating between teams and orgs I'm thinking yes Slack is one where the agents and the humans are together but if you think a little bit more creatively arguably like distributing a plugin for Codex would also be a form of communication and you talked about that as well. We're trying to give app teams essentially your brain, your
[18:43] ability to understand how their proposed data poll is actually going to work with your platform and review that code before they decide to set it up and stand it up on their own. Do it it it's sort of playing with that abstraction a little bit we can be very creative about how how far we want to take that thread. How have you seen um agent communication patterns shifting how teams work across that platform app team boundary? Mm. Yeah, this is a very good question as
[19:14] well. There's so many layers to all of these. Um I'll talk about like maybe the surface layer, which is like people are increasingly using agents to like communicate with others like via Slack. There's a lot of generated messages at this point, right? Um they they look like they come from humans, but if you read it, it's very obviously agent generated. They tend to be very verbose. Um so so for folks reading it, it takes a long time and sometimes it's a little bit even harder to like get to the point where it's like, you know, it's too diplomatic or too wrapped up. Um
[19:44] and then so what people are doing is they're digesting this with our Codex as well. So it's like Codex, right? And then Codex, rewrite this all back to human language for me. What is the point here? Um we're seeing a little bit of that and I think that's actually like not a bad thing. Um uh you know, every time we chat with Codex, every time you give Codex anything, it it it, you know, like it basically builds a hive brain even even even better. So like the fact that we're mediating through Codex is not a is not a bad thing. It'll it'll help us in the future
[20:14] as it accumulates more of this knowledge. Um but you can imagine as a human in the middle, you know, it sometimes is jarring to see Slack become like a little bit like, you know, filled with like agent to communication. Um so I think there's there's a little bit of like, you know, sense of like, oh, things are really changing, but also like excitement that, you know, we can do this. Um another layer is uh you know, like increasingly as we kind of launch more kind of capabilities um
[20:45] as a platform team or as any team like surfacing with users, uh we're we're trying to get to a point where our agents are good enough that it can handle a lot of the user communications, right? So we actually launched this um this kind of support Slack support bot. Uh the initial iterations were not that good, but now the models have gone very very good and we can leverage Codex. Um the support is actually getting a lot better. So, um it's actually able to answer a lot of really great like hard questions very intelligently. Um and
[21:17] previously, you know, what I mentioned before is kind of like what users feels about. Previously, when users see an automated response, they instinctively is like, "I don't even want to read this. It's super long, you know? Like I just want a human to answer my question." But because of quality of answers have gone so much better, um and because it has tool use and all of this as well, it's not just like searching through like a dead Slack corpus, um we see that interaction pattern changing as well. So, people are more open-minded about like, "Oh, this is a generated Slack response." It but it could be very helpful. So, I see people
[21:48] reading a little bit more. So, it's shifting the other direction as well. Um as models get really really good, I can imagine like these are all solvable problems by by, you know, with with model capabilities, right? Like whether it's super long, it looks super like it's formal, and like too much detail. Like that's also solvable. Um including like, you know, how do you like answer questions better? That's also solvable. So, I can see the future where >> Mhm. >> It right now, in my perception, I think the uh you know, the agent can do like sometimes as good a job, but on the most part like maybe less than average than a
[22:18] human does in terms of communication. But I can see very quickly that can change. It could be superhuman. It gets to the point super fast. It's also like very good at like understanding how humans perceive and understand information to make it like super condensed and and and and context-aware. So, if I'm in this channel, I'll talk about these kind of people, I will use this type of language. If they're in for people, I should talk about this, you know? Like you know, really like game the psychology even. Um and, you know, other teams will have their agents that that could do the same
[22:48] as well. I can see that happening very quickly. But we're not quite there yet. >> Yep. Yep, that makes sense. Um maybe we could play with like what one of the things that's been underneath the conversation so far for me has been thinking about how on a platform team on the infrastructure side of things, you have a different at least a potentially different set of primitives that you're working with, that you're giving to agents to address and to deal with. And I feel like when you talk about the need to have agents scale into the infra layer, scale
[23:18] into the platform layer appropriately, it feels like part of the conversation is basically ensuring they can work with the right primitives that scale for you versus primitives that scale at the app layer. It's It's perhaps It's an open-ended question, right? You can come back and say, "No, no, no, I think the primitives are the same." But like I wonder if you would say that. I wonder if you would say, "No, we have some different primitives that we want agents to work with." >> Um I'll just like give some examples. Like and maybe I'm like being simplistic about like, you know, a lot of other
[23:48] areas as well, but let's say you're you're building a front-end app, right? Like you need the code base. Um you need like a browser. You need to be able to like control the browser and whatever to test things. And if you have a back end, you need like, you know, a live service and and being able to spin that up and connect to that, right? Um but for the most part you can probably just like um use like, you know, like fake data or whatever to to stub data to to kind of test it. But if you are like supporting, let's say um
[24:19] a Spark cluster, for example, you know, a few thousand nodes and we have like, you know, dozens of clusters across different regions. And let's say the cluster is having issues, right? The agent needs to connect to so many different tools in order to understand what's going on. And it still connect to like the logging, the observability, needs to connect to like Kubernetes, needs to understand the pod, and needs to control like all these things. Um there's, you know, with with a Spark cluster, it's not just like one service. It's connected to like dozens of other services. There's like the shuffle service. There's also like, you know,
[24:50] our our our cluster routing service. And there's the quota management service. So you have to connect to all of those and understand those two in a live setting. It's not code, right? It's not It's code, yes, but it's not It's It's live code running on a platform. I need to figure out very, very different systems, plug into all of them, understand what's going on, put the picture together. So, just in terms of number of connectors, the capability that you need to have to delve into all of these different areas, and also to do live operations in a way that's secure. You can't just like try
[25:20] different things. Oh, let's try this, you know, see if that like solves it. It just probably >> Doesn't go well. >> Doesn't go well, exactly. So, this is where I feel like the capabilities and the primitives will be different from, >> you know, um you know, a lot of things. But, you know, it's getting there. Again, the models are getting very, very good, very, very smart. We just need to apply them in these settings more and more um to test them out. So, it's like a little bit of chicken and egg problem, which we're we're solving currently right now, which is like, you know, we don't trust it enough to do a lot of
[25:50] live operations, and therefore like it's hard to get a lot better at it. So, therefore we're like setting up more kind of isolated for it to do it right now. We're not quite there where we trust it to completely take over, but we've done a lot of different things with agents so far. For example, we already have agents pull um for like statuses for different services and give us updates on how things are going during a deploy or during incidents, etc. Very good at getting like information. We haven't gone to the point where we fully trust it to like actually come up with like fixes and
[26:22] apply them quite yet, but it can. We let it ask it to suggest things, and then we'll we'll we'll we'll monitor and look at it ourselves. Um but again, all things solvable by model capability, and we're getting there. >> Yeah, that's the other piece that like is popping out for me is like I think in a sense the the most of the conversations I'm in, what I hear is conversation about models and code. That is a lot of what you've talked about, you know, generating code, this and that. Um and it's framed as a mostly solved problem these days. It's
[26:53] framed as something that the models have become very good at. And I think that your this conversation opens up a really interesting uh sort of more nuanced picture where you can say it's one thing to say that the model has grasped app-level codebases and can kind of get into that and mostly solve that. It's very different to talk about how you support platform and infra teams and how you make sure the model's able to operate autonomously in that space. And when even when you were saying, you know, the the tool calls are there's
[27:25] like many many more tool calls just to understand what's going on and to understand what's going on correctly, it sort of got me sort of chewing on this idea that effectively platform teams are on the cutting edge of where scaling laws are going. And like you have maybe an order of magnitude, maybe two orders of magnitude more complexity that you're dealing with. And therefore, as the scaling law comes up, you're sort of seeing that it hit app team first, but you're feeling I'm sort of feeling that cutting edge of like it's not quite there, but we can see it getting there as you deal with a much more complex environment. >> Mhm. Yep. Yep. Yep. And it adds to the
[27:58] complexity that everything around you is growing very very fast. Like all the app layer like, you know, logic is growing very very fast. So, not only do we have to deal with the complexity of like trying to upgrade our systems to be identical, we also are dealing with a deluge of scaling problems. Just pure scaling. >> down the from the app layer, right? The app layer is like, "Okay, we're done with the call. Now we're going to just scale like crazy." And like they expect platform to keep up. >> Yes. Yes, exactly. So, so we're we're kind of in a double whammy right now. Um, but but then again, I'm I'm very
[28:28] optimistic. We already have like really awesome like ideas and solutions in place to solve all of these and we're very very hopeful. I think we're like like you like we're talking about, I think because of the complexity of kind of platform teams, the model needs to get a little bit stronger for it to be able to do this safely. Um, and we're very hopeful cuz we see the progress there. >> Yep. Yep, that makes sense. If you were to give some perspective, you know, I have lots of folks who are deep on data, deep on platform, deep on infra, but of course they don't work at OpenAI and and they're in my audience. I'm
[28:58] I'm curious what you would sort of tell them as like their Swiss Army knife or their pocket guide to sort of how to handle some of the complexity they're dealing with, because I hear from them some of the same challenges, but of course they're not working inside a hyperscaler and so they don't necessarily have access to some of the cutting-edge stuff that you're going to be able to pull from. And so in that world, you know, what would you say to an infra engineer? What would you say to a someone leading a data team um who who isn't working at a hyperscaler? >> Mm, that's a really good question. Um
[29:30] I think a lot of us on platform teams and infra teams are feeling a little bit underwater. So I would say to those teams first try to get yourself a little bit more time so that you can innovate out of this. Um and and for us, for example, we you know, rolled out support bots. Again, this this buys us more time, right? With the deluge of user requests, like at least some of these less urgent ad hoc requests can be dealt with by an agent. Um we encoded a lot of our best practices in agent MD files.
[30:01] Um so that, you know, the agents usually respect those. Um we encoded a lot of our best practices and skills. Again, hopefully like agents will hopefully respect those. And um um but agents are you know, just just to to say it, they're very very innovative. They've they've done things where like, "Okay, that was not that's an internal API, it should never be exposed. I don't know how you found out about that." Um so yeah. So so even if you have it all in skills and agent MD, it still can do very squirrely stuff. Um
[30:32] and uh and just like build in if you have the resources to just build in a little bit more reinforcement and shoring up of your of your system to ban bad actors. I think Yeah, I I it's it's not intention. It's not intentional, right? Um but but a a lot of these a lot of these PRs generated um they can they're they're almost quite adversarial like the case I mentioned before where they're like doing things they're not supposed to do, changing internal APIs to do the thing that they want but you know, will break everybody else. Um you
[31:02] you just got to put in more systems and like maybe obfuscate those APIs from them, make it less accessible to to like you know, a potential indicator to kind of build that defense as well. And once you buy yourself enough time, innovate on your systems, right? Build that like harness tool so that you can have an agentic response to PRs, right? And build those systems um like for example, isolated like you know, testing systems so you you feel confident in like a very minimal agentic kind of operations um
[31:32] response to like you know, some some critical systems and therefore hopefully roll that out in production as well. You need to invest in like wholesale up- grades of your system and so buy yourself time and like start thinking about what needs to be changed underlying to to make this possible for you to scale. >> Yeah, I that's just going to be a fascinating tidbit for a lot of folks right there. I think this that like one of the things I'm hearing really pop out is I hear a ton from people who are earlier in the AI transformation journey
[32:03] obviously than than folks at hyperscalers than you are. Um and sort of the the telegraph from the future that you're sending is basically when you get this AI transformation to a certain point it starts to take off and your app team start to widely adopt coding agents the pinch point that starts to emerge is that you now have uh n highly goal-directed agents who are coding in your system and they are going to hit your data and platform layers really hard and they may hit it in a way that feels adversarial even if that is
[32:33] not the intent of the human because the agents are very good and very goal-directed and really go after what they want to get. >> Yes. Yes, exactly. Exactly. So, so exactly like you said, it's unintentionally, but sometimes it can get like a little adversarial in terms of just a deluge and also in terms of the methods that the agents use to get out your systems. >> Yep. Yep. Yep. >> Yeah. And ultimately, we want to make it so that like we're like our systems are innovative enough and responsive enough
[33:04] and fast enough in an autonomous way where these are not problems. Like again, no bad intentions, right? What is the user trying to do? Why does it need to do this and get around things? Like how can we make that better for the users and go faster together, right? That's ultimately the goal. Um but we just need to make sure that we are, you know, investing those systems. >> How How do you think about the sort of hierarchy of utility around You've mentioned uh plugins, you've mentioned skills, you've mentioned markdown files. When you're thinking about architecting
[33:34] something that is intended to provide a guardrail or intended to provide some kind of input to an agentic system, how do you go about sort of segregating that out in a way that you you feel confident you've got the right form factor for that for that tool? >> Mhm. It really is an iterative process. I don't think we figured it out. Like all of these are attempts to make it better and all most of them have made it much much better, but we still see like gaps. Um uh >> Got it. >> It Yeah.
[34:04] >> it sounds like buy a spaceship and then you figure it out from there. >> Yeah. That's why I keep on advocating for like agentic response to like >> Yeah. >> uh changes, right? Uh right now like skills are These are These are model mediated responses, yes, but it's not like fully like multi-agent world where I can encode my values um separate from um you know, like Code Grader's values. And then we can have like a balanced dynamic. We're not quite there yet. I think that's the missing ingredient. All of these things you've described are like we still rely on that one single model to do the right thing.
[34:35] One single agent to do the right thing. I Again, this is my personal opinion. I don't think that's enough. >> Yeah, you were talking about the idea of like a a need to separate incentives between a coding agent and a code reviewer agent, for example. >> Yeah, yeah, yeah, yeah. >> Yeah. >> That makes sense. And so if you were to think about sort of buying your your team time back, the it sounds like just triaging inbound and starting to take action on inbound requests is one of the first places that you look. >> Yeah, so for a lot of platform teams like ours, like we have like literally
[35:06] the whole company's our user base. Like we have like 6,000 users essentially, right? Um if you look at like our support channels, like each one of them has like hundreds of questions every day. It's like, oh, you know, my service uh is missing this like access to this service. Like can you like make that, you know, happen for me? Um this job is broken in this way. I need someone to help take a look at this and why this is. Like there's just like so many different very like high cardinality type of like, you know, questions being asked. >> Yeah. >> And um
[35:36] you know, we we dedicated resources to just answer those questions and we still do. Um but we're we're really heavily invested in training our model to do this better, especially for things that are like more question-oriented, like debugging. Um so like why is this happening? Giving solutions. I think there's you know, we're we're we're at In this front, the model is beyond the frontier. It can do this, you know, it can debug like a job um fairly well. It can connect to the sources and and do these and the users can like try those out because it's low consequence. It's a single job, you know, if it fails, it can try it again.
[36:07] So the models are like very, very good at that. So we just need to roll that out more broadly um to all of our systems. The other thing I mentioned, which is another team, where a lot of the requests are more feature-shaped. It's like, I want this dashboard to have like be able to hide this line when I click this thing, right? That's a different type of response and we have a different agent dealing with that, right? And that agent literally just runs a whole loop and tries to like take as much off the plate of of our our engineering team as possible. So, for every team, the response will be very different in terms of what agent will solve your problem the most and and and
[36:38] fill in that gap. And then, um because the the model capabilities are changing so fast, like what it can do can change. So, every month we have to go and re-evaluate. Can it do this now? Can it do this other thing now? Um uh but yeah, so we're very optimistic on where it's going, but right now it's able to help in specific ways and we're kind of like seeing what it can do now and try to, you know, help ourselves a little bit in what it can do. But like seeing that very quickly it's going to be able to do more and take advantage of that. >> Yeah, and that's another piece maybe we
[37:09] can tug at that thread before we close here. I it feels to me like you have an intuitive sense of where you would want to push, experiment, or play with the model where you're like, "I'm not sure if the model can do this yet. It feels like it's on the edge of the capability set. Let me go and kind of nudge it and play with it and see what we get." I think my my impression as we talk is that that feels very organic for you and I'd love it if you unpack that a little bit for folks who maybe struggling with and trying to understand like, "You
[37:40] know, I'm in a complex code base. I'm in a complex environment. I understand exactly what you say when you say the model isn't there yet on something. How do I know when to push it? How do I know when to play with it? How do I know when to experiment with it?" Um yeah, so we have um And the same for folks are using our models as well. Internally, we have previews of models are coming up. So, we do get to play with a a few more versions of them. So, we do always try to test the most frontier model um with existing kind of capabilities. The other thing we're trying to do across
[38:10] the different teams is we actually have our eval library. Um so, for different teams they have evals for the core capabilities that they want the agent to do. And we're still flushing that out because again, with greater model capabilities we can add even more like types of things we can do. But for example, this eval for you know, full stack like figuring out like you know, feature sets and and and deploying that. We have evals for that and you know, like the models are pretty good at some things, not good at other things. When a new model comes up, we run our evals and be like, oh, it's really getting good at
[38:41] this. Maybe we should try to get it to do this, you know? So we do have those and then we do do a lot of live testing. We're always like pushing the frontier on things. You know, this may be a more of a manager like you know, culture thing is like at Open AI we we're always pushing people to like try more things with it. Have you tried this other thing with it? If someone comes back like, oh, this took me 6 hours to do is like, have you pushed Codex hard enough to try to do this? >> Why 6 hours? Come on, quickly. >> Yeah, and the answer maybe it's like it literally just has to take 6 hours and
[39:11] so but we are just we're pushing people all the time, right? To like try new things. And there's an internal culture of experimentation as well. It's like we kind of send kudos to people who have innovative use cases. We all share with each other how we're using Codex and how it's accelerating and new workflows that's coming up and and we have learning sessions where we swap these things but also just like culturally we just like there's admiration for people who are able to do this. Find innovative working use cases with agents. And it's considered cool here, it's considered like you know, you're you're innovative
[39:42] here. So that itself creates this kind of culture to like people really want to try what else we can do with it. What as soon as a new model comes up like people are experimenting. Things that they tried and didn't really work out before, they'll try again, right? Again, this is where the eval library come up, you know. We already have a set of things that we really would like the agent to do and we can just keep on trying and see if it improves there. >> And I will say like most of the larger teams that I talk with do not have the discipline yet of maintaining a private
[40:14] eval suite that is designed to test emerging model capabilities. So a model drops, they're going back and they're either looking at production and they're saying, "Do we want to try and swap it in and see what happens?" which is really scary. Um or they're saying, "We don't have a defined process for this. We need to assign someone at some point to look at this." And both of those are fairly inefficient ways to explore the space. And I think that one of the things that is a great takeaway is invest in that eval suite. Um and even if you're not in a hyper scalar, you're still going to get new models coming in.
[40:44] You will still get the chance to use that and it will be very efficient for you to understand what's going on. >> Yes, yes. And you it doesn't have to be heavy investment. You can do a very janky eval suite. It's just like you have a notion doc with like all of the evals and expected out outputs and just like run it through the model every time it comes out. It can be very janky. It doesn't have to be fancy. Um and that's kind of like, you know, a scrappy way to do it. >> Yeah, it's funny that you say that because one of the things that tipped for me I and I love how this ties into this idea that you want to be exploring
[41:14] and finding new ways to use models creatively. I had a tipping point with Codex uh just this week uh where uh it's obviously much smaller scale from a data perspective, but what I was doing was I was spending time essentially assembling a local folder um with a bunch of meeting transcripts, with a bunch of uh research that the agent had done. Um and my goal was actually turn that into a variety of documents. It wasn't just one. Uh I think it was eight. And for the first time I was able to tell
[41:45] Codex, "Okay, we've had a conversation about this. You've got a local folder with a bunch of stuff, right? It's got text of various types from different uh input settings. Like some of it is formal reporting, some of it is very informal conversation. Your task is to now simultaneously produce eight different documents." And it did it. >> Wow. >> Um and those And those it was really cool. >> Yeah, eight different documents. That that is pretty cool. I have never tried that. I need to go try that. It's like swapping kids right here. It's like
[42:15] you're you're you're now you're the cool dog, you know. >> No, it was really fun. And and I like I could feel it like with for me it feels like the the wind is like blowing my hair back. Like it's like wow, the future is here. Like I'm doing eight docs at once. Uh and then inside the same context window, I was able to give it edits and it was able to agentically version out the eight docs and rewrite them until I got them to where I wanted them to be. It was it was really fun. >> Wow. Wow. Wow. That must have been like
[42:46] crazy. Also, like think about the time saving. Eight docs in the past, man. We're we're in a brave new world now, for sure. >> That's right. That's right. Uh and maybe that that leads us to the last question I have for you, which is uh we talk a lot about autonomous AI, autonomous infra, kind of where the models are going in this conversation. Uh human attention continues to be very scarce. And so, where do you allocate yours? How do you think about how your human attention scales, your team's human attention scales? >> Uh such a good question. Um I think as an
[43:18] infra engineer or a software engineers in general, um we spend a lot of time thinking about scalability. >> Mhm. >> And I I think we're we're right in the midst of that. It's like the perfect thing for an infra-minded person to think about because it's all about scalability right now. And so, I I think a lot about what systems need to be upgraded in order to scale, what people systems as well. Like if I want my my folks to like really maximize their time here, they need to like leverage our um agentic coding a lot more or like agentic tooling a lot more. So, I encourage that. Also, on the
[43:49] systems levels, like what are the systems we need to fix right now and change and uplevel so that we can respond to the onslaught. Um This this is this ties into all these other things I talk about as well. And then, another thing is just, you know, in this world, there's a lot of information. Um and nobody really knows how this is going to play out. And you know, in my role, a big part of my job is to kind of let other people, including um, other leaders uh, at the company know what's going on on the ground, right? So, uh, you know, this
[44:21] this kind of like difference in acceleration, it it could be a problem if it continues like this, right? It could be a problem in terms of like, you know, reliability or scalability may really be compromised if we don't invest in um, our tooling. Uh, so a big part of my job is also just like talking to various folks and making sure that people understand what's going on the ground. >> Makes a ton of sense. Um, well, if you had one thing to say, and maybe we'll frame it as like, if you had one thing to say as an AI-powered leader to advise other leaders on like, this is
[44:51] how you should think about your job and how it's shifting, what would it be? >> Ah. I think business as usual is not going to fly anymore. If you're a leader, you need to be a visionary. You need to like wear that visionary hat and be like, things are changing rapidly. How do I navigate in a world where things are changing rapidly? How do I get all the information I need to understand where we're going? And then give clear direction to your your own folks. Um, and encourage them to not be scared
[45:21] of the brave new world. I think human a lot of times human like the first instinct is like, oh my gosh, it's like this is scary. Is my job going away? You know, all these things. You need to provide that positivity and like, um, kind of just like inspiration and kind of like a vision for them to realize that this is actually a really exciting time to be, you know, and we can all be at the forefront of this. Um, so yeah, just provide the support for your team. >> Love that. Well, thank you for taking the time. I really appreciate it, Emma.
[45:52] >> Thank you so much, Nate. >> Yeah, it's been really fun. I'm glad we got a chance to chat and I I'm look looking forward to hearing how like the models make your job easier in the future. >> I know, right? It's going to get serious in the next few months. I know like we have some really awesome stuff coming up. So, it's going to be great. Codex is like amazing. Everybody should use it. >> Yeah. Yeah, it's been a lot of fun to use for me, too. I've I've been watching the like I I actually do token burn counts every day and like I have Codex let me know how many tokens I burn. I'm like, "Wow, you can see the power law
[46:22] here." because I'm suddenly using it more and more and more. So. >> Yeah. Yeah. Yeah. I love it. I love it. Let's go. >> Yeah. There you go. Amazing. Thank you so much. >> Okay. Thank you so much. >> All right. Bye. >> All right. See you.