The interview transcript below has been edited for length and clarity.
Emilie Schario: I'm thinking about cost per PR. It's really easy to look at that $600 bill that he ha- had for that day of work and say like, "Wow, that's so much." But then when you divide it over how many PRs he worked on that day, it's like, "Oh, actually, it was only about $30 per PR." And then it, it changes how we think about the ROI because the spend is not the problem.
The spend with no return on that spend is the problem.
Sam Witteveen: Hey everyone, welcome back to Beyond the Pilot. Today's episode is a little bit different. We're pulling this one from a live panel that I moderated at VentureBeat Transform. It was all about agentic coding in the enterprise, and my guests were Emily Sherio from Kilo Code, which a fun fact, got acquired the morning of the panel.
They got acquired by Anaconda. We also had, uh, Jared Go from Symbiotic, who's doing a lot of work with physical AI, and Amjad Jain talking about what they're actually doing at Replit. We get into what people are actually doing in agentic coding in production right now. Code review, not just code writing, has become the, the real bottleneck, and how teams are measuring success of their actual token spend.
On top of that, we also look at where things are going in the next six months, and this sort of really compounds with, uh, VentureBeat's own research that we're seeing just how fast things are changing in agentic coding and how teams are actually building software. Make sure to listen to Emily's, uh, own unique strategy of how Kilo is measuring success for pull requests.
That one I found really fascinating. Enjoy the pod. We'll talk to you soon
Matt Marshall: This is VentureBeat's Beyond the Pilot podcast. It's about enterprise AI in action. I'm Matt Marshall. Today's episode is presented by OutShift by Cisco, Cisco's emerging tech incubation engine and driver of agentic AI, quantum, next gen infra, and beyond.
Sam Witteveen: All right, everyone. Uh, it's great to have you. So we've got a really interesting, uh, panel now. I think the original title from this was g- was sort of like Vibe Coding, and I kind of was like, "Eh, that's like kind of last year or at least last month." So, I, I guess, okay, to kick it off, let me, let me, uh, ask a little bit about...
You know, I- Yesterday we had the CTO of Instacart here, and the first question I asked him was, does his team still read code, right? And that turned into a very interesting conversation of where it turned out that he was saying basically 97% of them don't, right? And they actually sort of measured this. So let, let's start with, with this.
Emily?
Emilie Schario: Uh, their agents do. Right. I would say most people are, uh, reading each other's prompts, and they have reviewing agents who are writing, reviewing the code. But, uh, unless something's really broken or debugging, 99% of the time engineers are not reading or writing code anymore.
Sam Witteveen: Jared?
Jared Go: Yeah. I think for me it's really about directing the focus of AI.
That's one of the things I try to teach the folks at Symbolic, where it's like if you ask the AI, you know, "These are my criteria. Let's look at it from the lens of, you know, security, elegance, you know, being clean, concise code, water tightness." If you direct the AI in that way, I think that does most of the heavy lifting for you, and you can like really not directly review lines of code.
Amol Jain: Okay. Yeah. We've at Replit gone very agentic, but, but this is one area we've kept it a little bit more conservative, where essentially we run an agent that would look at a PR and, uh, assess a risk score to it. And if the risk score is low, author just self merges. But for others, we do have human reviewers sort of coming in and, and, and, and, and reading the code and, and giving feedback.
Sam Witteveen: So reading PRs is different than reading code. Uh- Yeah?
Amol Jain: That's fair. Yeah. You would say? And reading code- Right ... pretty much no at this point. At this point, it's like the PR is ready to go, and, uh, the agent does all the work, and you take on when it's a ready artifact.
Sam Witteveen: J- J- Jared, y- so you're doing physical AI.
So, um, is that different? Like again, like w- for you, are you reading the PRs at least or, you know, and something like that? So
Jared Go: different teams do. I think, you know, one, one thing that's really important about robots is safety is really important. So I think that's a big focus to our teams are, you know, you need to do intensive simulation.
You need to have reviewers for code. But there are a lot of things that are peripheral, like your logging system or something else that might not need that scrutiny. So I think it's really like you gotta make sure you have your quality gates, and you make a conscious decision about what's necessary in each case, but it's not one necessarily uniform rule or process across every single thing.
Sam Witteveen: Okay. So, okay, I, I'm gonna ask you guys your stack, but your stack is gonna be pretty obvious, right? So e- everyone, uh, Emily's from K- uh, Kilo Code. They just got acquired this morning, which- Ooh ... you know, deserves um-
Emilie Schario: Thank
Sam Witteveen: you. Thank you Uh, so do you wanna k- let's just start with that. I'll come back to you guys about the stack in a second.
Yeah. Um, tell us a little bit about Kilo Code. Uh, and then also, so, uh, about you, you got acquired by- Anaconda ... Anaconda, right? Uh, very famous sort of Python ecosystem company.
Emilie Schario: Yes. Okay. So Kilo Code is a, a agentic engineering platform. We wanna meet developers everywhere they are, uh, in VS Code, in- on the CLI, in JetBrains, on the cloud, on their phones, wherever they might be.
Uh, we pair that with 500 plus models through the Kilo Gateway, or you can bring your own. We are-- We believe in a multi-model future, where your software that you're using to do agentic engineering should be decoupled from the model that you're using to do it. Um, and with Anaconda, we can now bring the power of Kilo Code to the enterprise in secure environments and secure Python packages that Anaconda is known and trusted for.
So really excited. We use Kilo to build Kilo every day. Uh, so all of our developers are using Kilo, their, uh, as their coding agent. We use Kilo Code Reviewer on our PRs, often with a different model. Uh, and then we use, um, Kilo Security Agent to help keep our platform secure, manage our CVEs and vulnerabilities as they're repor- reported.
Um, as well as, uh, one of my favorite ways to use Kilo every day is, um, what I s- call killing my to-do list. So I don't, uh, add something to my to-do list. I kick off a cloud agent to do it, and now with the Kilo iOS app, I can do that right from my phone. So really trying to meet people everywhere they are.
Sam Witteveen: So ju- just quickly, uh, the reasons for Anaconda ge- uh, acquiring you seem to me that one... So one of the things that I, I found fascinating was that you're doing 10 trillion tokens a month.
Emilie Schario: Mm-hmm.
Sam Witteveen: Right? Through the Kilo Gateway. That's some, some serious, uh, traction there. I... And not only, that's, uh, a large percentage of that I'm guessing is open models, right?
Mm-hmm. Uh, how many models do you support again?
Emilie Schario: Over 500.
Sam Witteveen: Okay. So that sort of gives us a, a sense, and, and Kilo itself basically can r- route between those and-
Emilie Schario: Correct. We have, uh, multiple auto models in Kilo that we can route for you. Um, auto efficient is one example where, uh, we, we handle the routing for you based on the kind of task.
So you select auto efficient, we categorize the task, and we'll route it to the correct model, giving you the best bang for your buck. And often what we see is that companies or individuals will use, like, a really expensive frontier model, think a Opus, Fable, Sol category model, uh, to do their plan, kinda architect out whatever they're working on, and then they'll switch to auto efficient or another open weight model to, to get more bang for their buck, and that's how they're really gonna, uh, be able to leverage AI as much as they want, but stretch their dollar along the way.
Sam Witteveen: So I, I would just say quickly, this has been one of the themes in preparing for this conference of, uh, talking to a lot of the people that we've been talking to. Uh, they just don't want to talk about it publicly, this whole idea of having a router which does all of this. And, uh, many of them will admit that yes, they have something like that, but when it comes down to details, they are very reluctant to sort of, you know, actually say anything about wha- which open models they're using, uh, that kind of thing.
Can you tell us a little bit about what are the, you know, what are the most popular o- open models? 'Cause you've got this amazing data.
Emilie Schario: Yes, and, uh, we make it all public. Great. So kilo.ai/leaderboard, you can see what models people are using in Kilo today. If you're using Kilo Auto Efficient, as you're using it, it's gonna show you what model is being used.
Um, the other thing- Okay ... thing too is we also respect the, uh, model provider limitations that you provide. So if a customer is in the EU, they only want providers in Europe. They can set that configuration in the Kilo back end, in the application, and then when their models are getting routed, they're only gonna be routed to providers in Europe.
So I can't say, "Oh, when you use auto efficient for that task, you're always gonna get this particular model," because that's not how it works. It's factoring in what's important to you, what limitations you've set, what data retention policies you've established, um, for your organization, what bring your own k- what keys you've brought in, what commits you might have, and it's factoring all of that into that routing decision, again, so we can give you the best bang for your buck.
Sam Witteveen: Congratulations. Thank you. It's a very big deal. Okay, guys, back to your stack. So I'm hoping that you're not just gonna tell me KiloCode- ... 'cause otherwise it's gonna look like a, an advertisement. Jared, what do you use?
Jared Go: So, um, at Symbiotic we use a combination of Cursor, and we also have our, our homegrown, um, agentic harness we call SymCloud.
Um, you know, w- we are sort of like there's pros and cons of each. I think one nice thing about moving a lot of our workflows to our own clou- cloud native systems, we can just build everything in there and hooks to everything we want. We can control the tool surfaces and connectors and everything like that.
And for a company like us doing physical AI, I mean, there's a lot of data at warehouses, there's data, you know, bot logs, there's everything. So trying to pull that all into one place really nicely is like, you know, doing in the right way, making it efficient, making it, um, you know, reusable knowledge from across the organization is, is important to us.
Sam Witteveen: Do you, um, so Cursor's an interesting one 'cause a lot of people have left Cursor, right? Or, you know, at least, you know, at, at, at some point. Yeah. Um, i- is it sort of like dictated that everyone in the organization has to use the same thing, or is it sort of people pick their own thing?
Jared Go: The way we do it is we do have guidelines 'cause I think there's too much fragmentation if we let everyone pick their own tool.
So we kind of centralize around tools, but we keep our ears open to like if, if the tool's not meeting the needs or something like that. And so in our case, we mostly fold the functionality that's missing into our tool. But, um, you know, we're trying to train about 900 engineers. Um, you know, I wrote a book, uh, called Beyond the Vibe, and where I tried to tr- you know, teach people the steps you need to do to do these kind of things.
And part of this was like, you know, there's a lot of understanding about models that people don't have and how you train them to like, you know, learn the right skills and like theory of mind, understand what is the model thinking and perceiving so I can work with it better. So there's a whole bunch of stuff that I think we're passionate about, like transforming our organization.
Sam Witteveen: Kamaal, Replit. What are you guys doing at Replit?
Amol Jain: Yeah. We've, we've, um, you know, internal tool that we've built that's lets... It's essentially sort of self-driving for software engineering at this point, where engineers are, are, are sort of giving it the, the task and, and how to verify whether the outcome's successful.
And it sort of executes that end to end of planning, implementing and, and, and, you know, testing, uh, getting code reviews, m- merging, navigating CI, et cetera. And, uh- Sorry, go for
Sam Witteveen: it. No, no. So I was gonna say, so that, that sounds definitely more like a full w- full function suite tool rather than just writing the code.
It's more sort of the processes around it as well.
Amol Jain: Absolutely, yeah. It's, it's, it's almost, you know, the idea was human on the loop, not human in the loop. We wanted to sort of solve that bottleneck. And, uh, the fact that, you know, it's, it, it, it's a fleet of agents that run in their own cloud VMs and, uh, with access controls and behind token proxies and such, they're secure.
So we connect them to all the systems. You know, everything from GCPs to Datadogs to, to Slack, to everything else. And that means it has the right context. It had tools like browsers to self-test, that it can really go that end-to-end mile.
Sam Witteveen: Do you guys, um, make your own skills or MCPs or something to go with that?
Or...
Amol Jain: Absolutely, yeah. Uh, exposing certain things via MCPs. Uh, we encourage folks to create skills and then share them. We have a repository of them. Um, yeah, it's been really fascinating to see how people have been using it. Um, I think my favorite sort of anecdote recently has been a very gnarly bug deep in our systems that one engineer, uh, really couldn't repro or solve, so it, it goes to what we call a manager agent and told it to, "Hey, figure it out."
The manager agent told it to go to sleep. It spun up a bunch of agents that reproed. One of them found the repro, then spun up a bunch more agents to try a few fixes, found the fix. Six hours later it had, it had the PR ready for it for this gnarly vir- bug that a lot of people were very puzzled by. And that's, that's been pretty amazing to see.
Sam Witteveen: So it sounds like you're very pro the, the... If we're looking at sort of build versus buy for suite tools and stuff like that, you're very pro the sort of build your own of what you guys actually need. Does that fit in with perhaps, uh, uh-
Emilie Schario: Yeah.
Sam Witteveen: Obviously you're sort of building one of the tools, I'm guessing
Emilie Schario: You know, I still think a lot of the calculus comes down to where do I want my team to spend their energy.
Right. So I'm not going to build a CRM. Um, I might build something that's kinda tech adjacem- adjacent. A lot of our support workflows we've built in-house. Our team, every member of the team has access to Kilo and is using Kilo, whether they've got the word engineer in their job title or not. Uh, so we're definitely building more than ever, but I wouldn't say that we're not also buying.
We're buying where it's not core to what we're, we're building for the business.
Sam Witteveen: Right. Chirag.
Jared Go: Yeah. You know, this is the most important decision I think organiztions, uh, organizations are looking at making now. And I think just 'cause you can build something doesn't mean you wanna sit with the maintenance cost, you know, being on call for that, you know, critical product or something like that.
So you have to think about it really carefully, and I think it, it's on leadership to really send the right message that it's not like, "Hey, these all coder can just go wild and, like, make everything." Because in reality you don't, you know, teams don't wanna necessarily do that. Mm-hmm. So I think, yeah, just respecting that and thinking a lot about that.
Sam Witteveen: So, so would you say the cost has moved from, uh, you know, the, the investment in building to really just the investment in maintenance? If you, uh, if you... Sure, you can build something pretty quick nowadays- Yeah ... but maintaining it, uh, long term is gonna be the hassle.
Jared Go: Yeah, like, you know, greenfield is so easy for agents.
Brownfield we all know is where the actual challenge lies. Mm-hmm. So it's like, you know, you don't wanna accumulate, "Oh, I built it in a week. Now, oh, I'm gonna spend the next month helping work through all these bugs or whatever," right? And also product decisions are key, right? As agent I don't think... They don't, they don't make strong product decisions in a lot of cases.
So those are things that you just take time and effort, so that takes away from your other focuses for your team.
Emilie Schario: And it's, it's the last 10% that's hard- Yeah ... right? It's like y- getting the pretty UI or the pieces, the, you know... Going for that is the easy portion. It's the, like, specific edge case. Those are the things that really drain the level of effort spend.
At,
Amol Jain: at, at Replit we have a little bit of a different stance, obviously. You know, selling, letting people build internal tools or SaaS replacements is our bread and butter, so we, we actually encourage people to, to build things themselves. And, uh, it's almost like folks waking up and figuring out what generic expensive SaaS we could kill today.
Um, and, uh, our job then as, as, as, as the product builders of Replit becomes how do we make that last 10%, how do we make that maintenance easier, where any non-technical person could do that. You don't need engineering for it. So, you know, we have a- agents that once you build an app on Replit and publish it, that are monitoring it to make sure if it fails, they'll auto-fix it or make sure s- things are secure, or improve your app for you, make it really easy to do that.
And I think for that, that's shifted the calculus for us quite a bit internally. Not quite CRM yet, but, uh, but, you know, we just churned a, a seven-figure go-to-market software recently. We just built it in-house.
Sam Witteveen: This series is brought to you by OutShift, Cisco's incubation engine. By creating an open, interoperable infrastructure, OutShift is enabling agents and humans to share intent, context, and reasoning.
The
Matt Marshall: cognitive evolution for agents is here. Explore the Internet of Cognition at outshift.com.
Sam Witteveen: And so what are, what are the lessons then for-- that people should take away for that? If they do wanna do-- go that route, uh, and they are, you know, prepared to put in the effort and stuff like that, what are the tips that you would give them, uh, to be able to get that last ten percent, to be able to make the tool that can maintain it?
Amol Jain: Yeah. I, I think, I think it's important to be judicious like we've, we've been discussing here. It's important to know how far to take this. Uh, you have to go in knowing the maintenance and, and, and, and such costs. Um, and I think once that this is a, a solution, I, I mean, uh, you know, you should buy Replit to do it.
Um, to make the job easy. It's, it's, it's quite hard, but you really have to think about security and, and maintenance and such.
Sam Witteveen: Interesting. All right. The next question I wanted to talk about was token maxing. So, you know, we, we've seen, uh, I guess... And guys, tell me if I'm wrong, but like sort of last December, we saw an inflection in the models, right?
Where we saw some of the Anthropic models suddenly get a lot better. The game then became, for the first three months of this year, who could token max the most to try and do things the most. Probably the last two months at least, everyone's been going, "Hang on a minute, this stuff can get expensive." Mm-hmm.
Uh, so I'm curious to get your sort of take on this. Perhaps, uh, Emily, let's start with you.
Emilie Schario: Yeah. I mean-
Sam Witteveen: And you, you actually had a really good anecdote, so please tell us about the- Yeah ... that one as well. The-
Emilie Schario: Uh, so there's two sides to this: what am I hearing from my customers, and what am I seeing as an engineering leader?
And so much, uh, uh, from the customers we're hearing all the time, "Hey, I accidentally spent my whole AI budget for the year, and it's, uh, July 15th, so what do I do now?" Um, and we're really pointing them to the workflow that I mentioned earlier. You want the expensive models for planning, and then you want to use open weight models, more affordable models, uh, for everything else.
And the reality is, especially when it comes to agentic engineering, to coding itself, all the models know how to code. Um, and the, the quality differences come down to the prompting. So realizing where you can really uplevel your team to help them get the most of-- out of the models they're using is gonna make a big difference.
Sharing skills, strong guidance, MCP servers with the right information, these are all things that are really gonna empower, um, other thing-- models other than the state-of-the-art models. Um, uh, so that's one side of it. Internally, uh, and this is what I was mentioning earlier, I've got one engineer who, if he were driving, I would say he has a heavy foot.
Uh, he is, you know, consistently at the top of our usage board. Um, and I regularly have to nudge like, "Hey, what are you doing there?" And so one metric I've been paying attention to and, and being honest here, I'm still TBD on whether this is the best metric, but I'm thinking about cost per PR. Um, so you know, he spent-- It's really easy to look at that six hundred dollar bill that he ha-had for that day of work and say like, "Wow, that's so much."
But then when you divide it over how many PRs he worked on that day, it's like, "Oh, actually, it was only about thirty dollars per PR." And then it, it changes how we think about the ROI because the spend is not the problem. The spend with no return on that spend is the problem. And so if we can be really intentional about like what we need isn't a, a metric of how much we're spending.
What we need is a metric of how we capture the ROI for that spend. And I think as engineers, you know, you might ship something that makes it out today, but you might ship something that doesn't make it out till next week. The closest proxy we've probably got is a pull request merged. And so cost per, per pull request is the metric that I'm paying attention to right now.
It feels like the, uh, closest proximity for how I can measure value. But I think, you know, three months from now, we might not be talking about token maxing at all, so we'll see.
Sam Witteveen: Mm-hmm. Jared, what about yourself with the physical AI stuff?
Jared Go: How do you- Yeah. So when we-- Yeah, when we rolled out these tools, I think the, the thing you need to think about most in your organization is the controls and giving like the distributed control 'cause it's like, you know, the worst thing that happens is everyone's like, "Oh, I need more tokens.
Go talk to this team." And then this team is like, "Well, let me talk to your manager." You don't want this triangular stuff, so you need a tool. Like what we built early on is a tool surface where managers can kinda see what were the recent PRs, what was the amount of usage, how does it trend over the course of the month, and then they're empowered to go right there and click, "Okay, I'll give you another, you know, two into the next tier."
And one important thing we did too is we have tiers, so we don't try to make it too fine 'cause otherwise you're like, "Oh, should this person get, you know, three fifty, three seventy-five?" We just have s- you know, set tiers, and you can move people between them.
Sam Witteveen: And that's a, that's a financial tier or,
Jared Go: or different tokens?
Yeah, it's a financial like, you know, let's say, you know, a hundred and fifty bucks at the start, you know, four hundred and fifty, you know, above, and then you can make exceptions for
Sam Witteveen: more. This is per day, per month- Per
Jared Go: month ... per year? Yeah, per month. Okay. Per month. And I think, you know- Y- connecting that loop is really important because that ties managers into the value chain.
So exactly what Emily was saying about, like, you know, they're the people who can assess, "Okay, you're doing good work. Yes, you know, I recognize I put in a justification." But you need to give your finance org predictability. And so they're having a cap and just saying, you know, "We know how many people went up in cap this month," makes a big difference when you're trying to, like, corral these costs and c- and, you know, make things efficient.
Sam Witteveen: I'm curious, do you find, uh, when you, when you peg it to s- a financial amount like that, that then it encourages more use of open models or cheaper models? Uh, so that, that, okay, then the person st- suddenly now has to think about how they budget the- themselves rather than just, oh, you've got, you know, uh, s- sixteen million tokens per day or whatever.
Jared Go: So I'll give you a very quick story that's kind of funny. So yeah, I think we... I don't know if, uh, there are other Cursor users, but there was a change that Cursor made where we were grandfathered in under, like, the four cent per request plan even for frontier models. And then they were like, "No, now everyone's gonna pay full," right?
And so that was a shock to everyone. And having the limits in place, everyone stopped to think about, okay, am I using this efficiently? Am I picking the right models? So I actually thought that was a really great outcome where everyone was much more aware of how their usage ties to cost and could make good decisions.
And we had all sorts of good discussions. People were saying, "Hey, you should try this model. You should try this. This works better for this C# code, this whatever." So I think that, yeah, they... people need to be aware, I think. Mm-hmm.
Sam Witteveen: Amal, what are you guys doing at Replit?
Amol Jain: Yeah, um, th- th- that, when I was describing our stack, I was talking about that internal tool.
Um, it started out with engineering and soon spread to the rest of the company. Um, one of the folks looked at the bill eventually, and they realized someone on the support side had blown through an insane amount of money. When we looked under the hood, it was because they were running an automation using GPT 5.5 Pro Max.
Mm. And, um, I, I think that was sort of like the first time we, a, a couple of months ago, where we sort of started to look s- uh, you know, take a look at, like, the return and the investment. Mm. And so- because so far, uh, at least till that point, the ROI was rather clear. Mm. We could see engineering productivity three X, so no one had questioned it yet.
And so we finally started questioning it, finally started sort of, you know, very basic things like, A, visibility to everyone. Not in a way that i- it's anti-productive. It's still just be aware of what kind of spend you're doing. And then very basic things around model routing, of course. Like, that's, that's a, that's a very simple thing to do.
And even sensible defaults. Uh, most tasks do not need the frontier. Um, so we're investing a lot more there 'cause it also, you know, is something we do for our core product as well.
Sam Witteveen: So I think let's open it up. Do people have questions that they would like to ask the panel?
Audience Member: Mm-hmm.
Sam Witteveen: Uh, we've got one here. Uh, we've got someone with a mic.
Audience Member: Good afternoon. Thanks for this discussion. Uh, this is a question for Amal. So as you think about providing the service of Replit to vibe coders, um, have you thought about actually providing this routing layer to the-- basically to your ISVs so that they can take advantage when they build products to distribute to their customers?
Amol Jain: Yeah, great question. So, uh, so far our stance has been very much we as the harness builders have a great sense of that cost versus capability, uh, sort of, sort of spectrum, where we are essentially making the, the, the decisions on users' behalf of what model to use when in what capacity to minimize cost and maximize, max-maximize capability.
And I, I think that's worked really, really well so far, especially for the types of users we serve that are, that are, that, you know, most non-technical users wouldn't be able to-- a fable to them is still a poem, not a, not a model. So that's worked out pretty well so far. I think-- But that could continue to change.
That's some-- Those are discussions we are trying to have. Um, there's, there's clear trade-offs of if you let users select the models, then it's the-- it's a lot more responsibility on them to sort of navigate. Mm-hmm.
Jared Go: Uh, I have just one short thing- Yeah, go for it ... which is like, you know, so for example, in our harness we have, like, costs are upfront, and that was actually a big debate.
Some people are like: Well, if you put the cost, you might discourage people from using it. And I'm like: Well, they should think about it. Like, if I did this task and it says, like, eighty-five cents, well, that's pretty good depending on what it said, you know? So I think just for us, like transparency and making sure costs are visible is I think a key, a key thing.
Sam Witteveen: Do we have another question here in the middle?
Um, I think that was a really good metric on, like, measuring the usage on, like, PR creation. But do you think, like-- You're also thinking about PR merges? Because what I see is, like, the burden on the reviewers have gone, like, crazy up, right? Where it's like you-- A developer would create five PRs in a month, now an AI agent can create five hundred in a day, right?
So it's like maybe the measurement might be, like, PR merges and then, um, um, um, yeah. I mean, uh, code deletion is also very underappreciated in the industry. And I feel like the more that, you know, people can delete some of the code that maybe is-- can be optimized or maybe is unnecessary, I think that's a really good...
Is that something that you're looking in your, um, company as well?
Emilie Schario: Yeah. Thank you for the question. When it comes to PRs, like, if we think about cost per PR as, as spend over PRs, it has to be over PRs merged. Work done that isn't merged might as well have not happened. So we have to take all the work done, um, sometimes in the process of figuring out the final solution.
Like, it's not a bad thing that all work doesn't get merged. Uh, so-- But you have to factor in PRs merged as the denominator for sure. Um, and then the, the other side of your question, um, agents do not like to delete lines of code, for sure. I think that's where, uh, people judgment still comes in. Code review is still the bottle-- is currently the bottleneck in the process, whereas development was before.
But I think one of the big reasons code review is the bottleneck is because we've changed agentic engineering. We're all talking about developing software in a new way. But when you get a room this big talking about what code reviews looks like in this era, you're gonna get fifty different answers. And so we haven't converged on what code review means quite yet.
We've got our opinions at Kilo. I see what my best engineers are doing, and I have, like, a blog post in drafts about this somewhere. But, uh, I think every team needs to be rolling out a ki-- a code reviewer agent or bot. We've got Kilo code reviewer ourselves that we're using. Uh, you-- I think you mentioned, Amal, that small PRs are getting, like, a, a bot review and merge.
Different teams are taking different approaches here. Uh, you don't want something that's toc-touching your billing infrastructure or off auto merge. But, uh, we don't need review to look like the way review did in, say, twenty twenty-two. And so it's figuring out what review looks like next. I think that's gonna be the big theme of the next six to twelve months, for sure.
Sam Witteveen: I would point out, um, there's a, there's a conspiracy theory going around, uh, that I talked to the panel about this. I... So I, I do a lot of work in research, and one of the, the conspiracies is that a lot of the research labs are actually getting agents to file pull requests on open source things, and then see what gets merged to know whether that was a good training signal for training your models to get better, uh, a- at that.
So th- there are certain labs that have been accused of, of it more than others. Yeah,
Audience Member: I
Sam Witteveen: am- Um, all right, we don't have a lot of time left. Let's, uh, just quickly to wrap up, I- I'll introduce- If, if we'd spoken six months ago or, you know, perhaps eight months ago- Do you want me to go? ... it would have looked totally different, this panel.
Do you want me to go? Right? So pull out your crystal ball. Where are we gonna be in six months from now? Amal, you- you s- kick it off.
Amol Jain: Wow, that's... Predictions are always hard. Um-
Sam Witteveen: We do research. Well, just in the general direction, what do you think's gonna change? Oh,
my bad.
What are you starting to see changes in?
Amol Jain: Hold on. I, I, I think the, the model selection part, I, I think we're gonna see a big switch, big sort of like emergence of open source and open weight models that are used for, for majority of tasks and, and frontier models sort of reserved for these, these key decisions, key architecture decisions and such.
Um, I think, I think that's a shift we're gonna, we're gonna see pretty aggressively.
Jared Go: Jared. I think for me it's really like very long unattended work loops. And I think, you know, I don't know if I believe the stats, but you know, the Codex team was saying a lot of their tasks are like eight to 12 hours. I don't feel like, you know, you get the quality out of that little interaction today even with frontier models, but I think it's certainly going up.
So we need to build the harnesses that have enough information and tooling and structure to support that kind of work, which we're, most companies are not built for, you know? So you don't want to have to like wait for someone, oh, they have to upload this document to you for the agent to continue its work.
So really seeing a 24 hour, 48 hour of productive work, you know.
Sam Witteveen: Interesting. Emily, what are you gonna be doing at Anaconda six months from now? What is-
Emilie Schario: Yeah, I think we're gonna see a lot more, um, model sovereignty as a push, whether that's, uh, geographic sovereignty of models or, uh, companies wanting their models on their infrastructure running in closed environments or isolated environments.
Um, I think we're gonna see companies double down. We've been talking about moving a lot of workflows to AI, but the, the level of model we have now, if they haven't already, they're in the process of moving it, but part of that is gonna be isolating from the external environment. So sovereignty of models is going to be really big.