Instacart Cut AI Coding Cost per Merge 75% with CTO Anirban Kundu

Beyond The Pilot Podcast


Instacart's CTO stopped his engineers from reading code. 97% of their codebase is now AI-generated — and they've cut the cost of serving Clementine, their AI shopping agent, to one-eighth of what it was eight months ago.

Anirban Kundu (AK), CTO of Instacart, breaks down the internal architecture behind that number. The core insight: the harness matters more than the model. Instacart built a proprietary system called Maverick that routes planning tasks to frontier models and code generation to faster, cheaper open-source models like Luna and DeepSeek — cutting development AI costs by 75% while hitting their highest-ever weekly code merge count. Engineers are now migrating a 13-year-old Ruby codebase to Go and Rust without knowing either language.

On the product side, AK unpacks Clementine's technical stack: a multi-layer query understanding system that replaced a frontier model with a fine-tuned SLM, semantic IDs built from full catalog embeddings to surface substitute products in real time, and an intent-based architecture that — by the third session — shifts users from ingredient-level searches to natural-language cart building. The result: measurable basket size growth in a grocery e-commerce category he calls "massively under-penetrated."

AK also covers Instacart's MCP strategy for navigating the personal agent ecosystem (Meta Muse, Claude Duet, OpenAI Dots), why they've actively steered partners away from browser-based integrations, and what comes next: synthetic users, per-user experimentation at scale, and fully segregated enterprise code instances generated on demand.

🎙️ GUEST: Anirban Kundu (AK) | CTO, Instacart

🎙️ HOSTS: Sam Witteveen | VentureBeat

---

00:00 Intro — Why the harness beats the model

00:40 Episode intro & guest background

01:20 97% of Instacart's code is no longer read by humans

02:20 The journey to 100%: legacy pruning and engineer buy-in

03:00 Migrating Ruby to Go and Rust — without knowing the language

03:40 From code generation to full codebase regeneration

04:00 Prompt-and-loop architecture: the "why" behind the code

05:00 Multi-model planning: frontier for strategy, open source for execution

06:20 Building the eval and verification layer

07:00 Instacart's internal harness: "Maverick" explained

08:00 Why the harness is more valuable IP than the model itself

09:20 59% of companies struggle to swap models — Instacart's approach

10:00 75% cost reduction and record weekly merge count

10:40 Luna and DeepSeek vs. frontier: speed and cost trade-offs

11:40 AI acceleration on the ML side, not just code generation

12:00 AutoML with Bayesian optimization: shipping models overnight

12:40 Combining GBM/neural nets with SLMs for long-tail model coverage

13:40 Semantic IDs: embedding-based product catalog relationships

14:20 Clementine: intent-based cart building explained

15:40 How Clementine uses purchase history and real-time fulfillment data

16:40 Multi-layer semantic search: inverted index + LLM-cached results

17:20 How the search architecture was rebuilt over the past year

19:20 Business results: search conversion growth and basket size lift

20:20 How users shift behavior by their third Clementine session

21:40 Navigating the personal agent ecosystem: Muse, Claude Duet, OpenAI Dots

22:20 MCP over browser-based integrations: why and how

24:40 Clementine's frenemy strategy: connectors as acquisition funnels

25:20 Personalization moat: purchase history as a competitive advantage

26:00 Blending organic and ad results inside Clementine

27:20 Instacart's enterprise offering: Storefront Pro and Cart Assistant

28:40 Per-brand customization: persona control and product highlighting

30:20 Code regeneration enabling segregated enterprise instances

31:40 Open vs. proprietary model routing inside Clementine

32:40 Fine-tuning SLMs for search long-tail and Clementine query understanding

33:20 How Instacart reduced reliance on frontier model providers

34:00 Automatic model escalation inside the developer harness

35:20 What's next: synthetic users, hyper-personalization, and beyond AB testing

37:00 The verification problem: managing non-deterministic output at scale

39:00 Closing thoughts and what AK will return to discuss

---

Subscribe to VentureBeat: youtube.com/@VentureBeat

Apple Podcasts

Spotify

LinkedIn

Newsletters: venturebeat.com/newsletters

#EnterpriseAI #AIAgents #LLMInfrastructure #AIDeployment #MLOps


The interview transcript below has been edited for length and clarity.

AK: So the harness, I would say, is the place where we spend the most amount of time, both for cost reasons, but more like capability reasons. The models are so capable and similarly capable at this point in time. I won't go into the model that we're using on the frontier side, but it's not the most advanced one.

It's actually something that is a little bit older that open source models can actually handle that pretty well, and now it's our job to figure out how to route it based on domain understanding. I would say it's probably even 70 or 80% of the effort more than the model choice itself.

Matt Marshall: This is VentureBeat's Beyond the Pilot podcast. It's about enterprise AI in action. I'm Matt Marshall. Today's episode is presented by Outshift, Cisco's emerging tech incubation engine and driver of agentic AI, quantum, next-gen infra, and beyond.

Sam Witteveen: Today, we're welcoming Anirban Kundu, also known as AK, the CTO of Instacart.

Before Instacart, AK led engineering at Uber for delivery. He ran product and engineering at Postmates and even before that was the CTO of Evernote. AK and I caught up at Transform this year. We had a fascinating discussion around sort of engineering and where things were going from that point of view.

Today, we wanna recap some of that, but we also want to basically look at where things are going with product and some of the other interesting things that Instacart is actually doing. AK, welcome to the pod.

AK: Thank you. Thank you. It's good to be talking to you, Sam.

Sam Witteveen: At Transform this year, you had some really amazing things to say, and I think the one that caught everyone's attention was that you told us about 97% of Instacart's code is no longer read by humans.

AK: I would say 97% of the cases, yeah, we stopped reading code.

Sam Witteveen: You stopped reading code?

AK: Yeah. I actually make a joke with my team now by the summer of next year, we're not gonna be able to write "Hello, World" in Ruby. We're just... we don't know how to do anything without that.

You may ask "Why is it 97? Why not 100?" There are parts of our code that are driven by legacy, driven by a boatload of code that is dead or not active or half active or things of that sort, we have to slowly prune those parts out, so that's one part of it that we're going through.

Sam Witteveen: How's that going?

AK: It is going well. Are we there at 100% of not reading code? No. I was just in a... We had a leadership conference a couple of days ago, and it was definitely a journey of getting all the engineers to be bought into the fact that, hey, you don't have to review this code or...

But think about what fed into the code more than thinking about the code itself. But our goal is that by the middle of next year, we will be not looking at code anymore. Along the journey of that, one of the things that we have also seen is, for optimization reasons, both from a cost, but primarily from a latency perspective, we've actually started converting some of our code from Ruby to Go.

And some of that is by engineers who have never written Go code or have never learned Go code, and it's like we're making this transition to Go and Rust in a couple cases just by engineers who don't understand that language but understand the idea of how computer science works and how processes work and how interactions work, and then defining that and driving that without caring about what that code is.

So we've already started making that journey in a few places. Hopefully, by the middle of next year, we will stop looking at code altogether.

Sam Witteveen: One of the things you brought up at Transform that I thought was really interesting was you also talked about how everything's moving away from just writing code to much more being about writing the tests, writing even the prompts to allow you to regenerate the code.

So that was one of the things that I found fascinating, was this whole idea of being able to just regenerate the entire code base, maybe not every week, but when you want it to kind of thing.

AK: So I actually think there is the part about regeneration, and we'll talk about that in a second, but we've actually gone even further beyond the regeneration part of that equation.

We're not doing that fully there, but for new code, we are actually getting closer and closer to that point. But one of the things that we've come to learn is we call it the prompt. It's not just fundamentally the prompt, but we've moved into prompt and looping. The looping is where the system automatically is building on itself with certain exit conditions that are based out of it.

But what we've basically come to learn is that can we optimize that prompt even over a period of time? So it's not just that the prompt is locked in time. Can we optimize the prompt? What feeds into that? And we'll talk about that in a second. But then subsequent to that, yeah, it basically starts regenerating that code on an ongoing basis.

In fact, in my own personal projects right now, every three weeks it regenerates the entire code base completely. Two weeks ago, I was running into one part of it that was just extremely slow, and I just told it, "Go implement this in Rust and make this one little optimization." Two hours later, it was completely redone in Rust with the evals, and now it's actually running in Rust in a multithreaded Rust way rather than a Python multiprocess mode.

But we're beginning to do that in a couple of places. But the prompt is interesting, so now we're also beginning to move into, in a couple teams, where design, product, and engineering are beginning to build together to create that output. So when that prompt is getting created, two other pieces of information are coming along the way, which is the journey of, A, why are we doing this thing?

Because that why is important in making the prompt better over time, and it also allows us to prevent mistakes because there might be an assumption of, hey, why don't you just go do this instead of not. So we have this thing where we're talking about long history that we're maintaining and then feeding that in the prompt.

So the prompt is also partially getting that built that way, and then there's a human part of that equation to fill the prompt. And then the last part of it is, yeah, you can regenerate the code in multiple languages, which is happening in a couple of places where we've moved from Ruby to Go, but then in a couple places it's regenerating it continuously.

Our goal is that by the middle of next year, we'll be able to move all of our most computationally intensive portions that we've built over the last 13 years at Instacart into this model of rewriting it all into Go, and in some cases rewrite it even further into Rust to get those optimizations without having to keep touching the code.

Sam Witteveen: So when you talk about that sort of loop, right? So I find that fascinating. So you're talking about the why being a key part of that, not just the prompt. I'm guessing some kind of verification layer whether that's sort of tests or something like that. How are you thinking of constructing that loop?

'Cause it sounds to me like you're almost building your own harness kind of thing, to use sort of some trendy words.

AK: You caught on right away, Sam, in terms of the eval portion of it. We know we have to build that verification layer. We're still working on that side of it. Depends on the coverage and depends on the semantic coverage, not just on the, hey, you've got lines of code being tested kind of thing.

So there is some work that we are doing on that perspective. So that eval portion of that verification of the prompt is not fully there. That's something that we are putting as an entity on that thing. However, there are two parts of that harness, and yes, it is a harness that we are building. The two parts of the harness are, A, the feed of this information, this context about why you're building this, how you're building it, things of that sort.

And then the other part of this harness is which models do we end up using and what do they generate as an output? So for instance, when we're in the planning phase, so when the product manager, the designer, and the engineering lead sits together and builds this out, we're actually using frontier models for that in a couple of different variations of it.

And in fact, we're actually doing multiple models that are evaluating each other to make sure that the coverage is good. But then the subsequent part of it is that the harness actually drops down into cheaper, faster models to be able to build out the code. And so there's even this two-layer mechanism that generates that code that we are beginning to push out.

So we have a system internally called Maverick that we are building out that portion. Along with that, there is a building happening of the verification part of the equation, which is something we're working on. So you're absolutely right that the verification of that prompt and making sure that nothing's changed or broken and doing it in a quorum fashion is something that we're working on at the moment, yeah.

Sam Witteveen: What you're telling us is really that you've built a lot of IP around this, right? So when we hear certain companies talking about everything's moving to the model, it's all just the model kind of thing. It seems to me you're saying the opposite, right?

If anything, you guys are actually building lots of IP around that.

AK: What we've come to realize is that the harness is probably even more important, things that have to be tuned than the model itself. Yes, there's optimizations that happen on the model, like the model has some pieces of information that makes it better.

Like for example, if you wanna get third-party information into the model, frontier models are better for that rather than the code-focused models or the open source models. So those are there, and that's why during the planning phase, that's what's being used. But what's even more important is how do you engage with the model?

What are the things that you feed into the model? How do you drop into these different layers? How do you pull other pieces of information, especially in the intranet, into it? Those are the things that matter a lot more. And what matters even more than that is how do you achieve this consistency of execution across your entire development base.

So it's not just your top engineers who are thinking that way or the ones who are more advanced in their thoughts about how to use AI are using it and getting the benefits of it. It's across the board. You're democratizing that capability. So yeah, the harness is even more important from a cost, consistency, and a coverage perspective.

Sam Witteveen: VentureBeat's been doing a lot of research surveys lately about a whole bunch of different topics. One of the key things is the Agentic Orchestration Report, and one of the things that came out of that was that 59% of companies actually saw a drop in performance when they were changing models, and they were actually finding it quite hard to change models.

So it seems like you are not in that situation. Tell us your secrets.

AK: So, Sam, I won't say that it is easy to change models. It is work. It is complex and there are things that you have to optimize for it. However, saying all of that, the necessity and the benefits that you get out of having these different tiers of models is immense.

Example, if you were to use a frontier model for everything, obviously there's the cost considerations that come in big time. Compared to the summer when we were talking to now, we have dropped our cost to about a quarter on a per merge basis. So last week was the highest number of merges that we've ever done at Instacart on a single week basis.

So it, like no fluctuations. In a normal week, it was by far the highest. But not only that, the cost has dropped by about 75%. So there are benefits of that sort that happens, but that has involved some amount of training and teaching folks about how to think about things. However, saying that, this part about the separation of getting the benefits of planning coming through from a frontier model has been so powerful, but then going into whether that is Luna or DeepSeek, so we're using both of those right now, primarily on the code generation parts, it's so much faster.

So when developers engage with it, it's so pleasant compared to frontier models that are just significantly slower, especially if you use something like Counsel on top of it. They're just so slow compared to Luna and DeepSeek in these particular cases, that you get the benefits of the speed, you get the benefits of the code generation, and yet you've had the thinking happen in a different thing.

So is it hard to use multiple models? Yeah, it is. It requires some amount of discipline. It requires some level of training. But is it worth it? Oh, yes, absolutely worth it, both from a cost perspective, but like even the engagement of working with the model to generate the code is pretty awesome.

Sam Witteveen: Obviously with that, with your whole new harness and with your techniques that you're using, you've been able to ship a lot of new stuff.

AK: I would be remiss if I said that all of the things that we've done on the AI side of the equation for development has only been in the code generation side.

I think one of the other things that's been awesome is on the ML side of the equation. Since you spoke about speed and quantity, some of the massive set of changes that we've seen is in ML engineers being able to put out models that are ridiculously efficient and fast at getting out as quickly as possible.

So we've been huge proponents of AutoML, where basically at this point in time, through Bayesian statistics, basically all of our coefficients for supervised learning is now being generated on the fly by both LLMs, both SLMs and LLMs that are basically building those out. So now overnight, you get the starting point of a model and the optimized path of a model, and this has allowed us to get models out very rapidly.

So I would say even more than development, this has seen the faster growth of getting models out. So yeah, it's across the board, it's been awesome.

Sam Witteveen: In relation to that, are you using the decision models, the JEV models, and that kind of thing?

AK: A combination. So in some cases, it's a feed into a GBM model.

Some cases it's a feed into a neural net. So it depends on the type of model that's running. The other thing that we've done, which has been pretty sweet, is we've always had this problem where models get extremely bloated. So it's very easy to say "Hey, we've got a crazy amount of data," obviously with 13 years of history and understanding buying patterns.

We'd love to pull all of that information and then make decisions off of that. But there is a part of it that's too much data is also a problem portion of it both in terms of diluting the answer that is provided, but also in terms of being able to fit in memory. So one of the things that we've been able to do now is to be able to use the ML part of it to get the responses as quickly as possible for the top end percentile.

Usually it's somewhere between 92% and 95%. But then the long tail side of it, now we're basically plugging in an SLM that's been tuned for it and then stored in some kind of a metaform, like some kind of a memory cache, which now combines the results of both the ML and that portion of it, and that's also something that we've plugged in.

So that entire process of the iteration that takes or iteration that's necessary to both solve the model, as in make the model available and tuned, but also solve the long tail, has become so much more pleasant at Instacart over this year.

Sam Witteveen: So tell us a little bit about how you've put that into products and stuff.

So Clementine, right? Perhaps tell our listeners what actually is Clementine, and then let's break it down.

AK: Yeah. Clementine is Instacart's think of it as cart assistant, but Instacart's ability for users to very flexibly interact with the system, like in a chat-like model, but it's a lot richer than chat, and be able to do whatever you want to do to then generate a purchase that ends up happening.

So for instance, if you think about the old-style way of purchasing products, especially in e-commerce, it literally is like you get a feed or you start with a search and you say, "Hey, I know I wanna make, let's say, a chocolate cake today." If I wanna make a chocolate cake today, I know that these are the ingredients that I have to purchase.

I gotta buy the powder, I gotta buy the eggs, the milk, XYZ in the equation, and then you go into Instacart saying, "I want to go get milk, and I wanna get eggs, and I want to do this part of it." What we are allowing users to now do is to start with an intent. Tell us you wanna make a chocolate cake. Tell us that it needs to be vegan.

Tell us that it needs to be for three people, for instance. Just say whatever you wanna say with it. Clementine will basically use your history, its understanding of you and say that, "Oh yeah, you know what? You're generally vegetarian by nature, so we should recommend things that are more vegan or vegetarian by nature," or these are the kind of things that you always optimize for, like you tend to like more sugary things versus more savory things.

It's using your entire history, allowing you to specify an intent and then allow you to build your cart in a very flexible way. One of the things that I love about Clementine is over time, if you go to any search page that exists in pretty much most e-commerce sites, filters exist, but the filters are filters that have been chosen by the product team based on a bunch of A/B tests that have been done over a while, which is which ones increase conversion, which ones do people engage with more, all of those things.

But those may not be pertinent or relevant to me or in this particular case, and so then I'm stuck. I'm doing the filtering as the results are coming on saying, "Oh, I want this particular thing," and so then I'll choose it. With Clementine, you don't have to do anything, just be flexible and say, "I want this controlled by this thing."

Or even say something to the fact of, "Yeah, you know what? I'm only doing this for the weekend. Don't get me like a large basket on this particular thing," or, "Give me the full-size version of it for instance." All of those things it can understand and control and then allow you to go through that smooth buying experience.

But along with that, what we've done is we've taken the language models as far as understanding the intent, but made it super fast through those model improvements that we've talked about. So that search is not a search that's going through an LLM that's ridiculously slow. It's instant in terms of giving you results, giving you accurate results based on what you've purchased before.

But not only that, it's immediately looking at what's possible to fulfill in your particular area. So it may be that you like X, but X is not available, here's a substitute for X. It's doing all of that in real time. So it's a tremendously slick experience that allows you to go through intent-based buying rather than knowing the entities of what you wanna buy, give you the capability to define those constraints and not be limited by the UX at all.

Sam Witteveen: How do you... when you're planning that, are you thinking, do you have a prompt rewriting system and then an embedding system that you're basically then doing semantic lookups on a database? Can you tell us a little bit about the sort of how you do that kind of thing?

AK: Over the last year, a couple things have changed at Instacart. So the first one which is, in the past, search was primarily a retrieving system out of an inverted index, think of it that way, with an associated connection with product mappings and then an associated mapping of availabilities in your particular zone, whatever that zone may have been.

Over the last year, we added support for semantic understanding on top of that. So as much as possible, now semantic understanding is slow. But what we've done over a period of time is through two layers, think of it as an N-layer understanding system that basically goes and says, "What are you trying to say?

If you're trying to say stuff that's inverted index based, then I can retrieve it off the system and just feed it to you." But on the other part of the equation, let me try to understand what's the domain that you're trying to feed into. Are you a long tail in the some types of query that you're making?

Understand those queries and then get into serving it out from the LLM that feeds out a series of pre-tokenized set of results that have been created for that particular topic, and then build it up over a period of time. So it's taken a little while to build out that corpus of understanding and then feeding that into a cache, which then exposes it to the semantic side of it.

So that's number one. The number two side of it is we've created this semantic map of the entire product collection that exists, so both in ads and in our organic capabilities. Now, when we find similar products, they're not similar products based on just buying behaviors of it in the past, but it's also understanding of similar products based on full catalog understanding.

And so we've created this thing called semantic IDs that allows us to understand relationships between products, which then reduces the set of products that we end up looking at any given point in time very rapidly. And the semantic IDs are based off of embeddings.

Sam Witteveen: This series is brought to you by Outshift, Cisco's incubation engine.

By creating an open interoperable infrastructure, Outshift is enabling agents and humans to share intent, context, and reasoning. The cognitive evolution for agents is here. Explore the internet of cognition at outshift.com.

Sam Witteveen: From a business perspective, it sounds like this is really working, right?

AK: For what it is, absolutely.

Sam Witteveen: From what I understand, your early numbers are saying that people are putting more in the basket. Can you tell us a little bit about that?

AK: Yeah. So a couple things that we've seen major progress. Independently, all the things that I mentioned about search have grown search conversion for the first time by a pretty significant amount.

The conversion as far as understanding the long tail has definitely improved significantly. But in addition to that, what has happened is the move towards intent. So what's interesting that we've found about Clementine users is Clementine users come in assuming that the way you shop is exactly the same way as you've shopped before, which is I already know that I wanna make a chocolate cake, and so I need to get eggs and the flour and the sugar, and things of that sort.

So I start off by asking for that. So that's the first type of things that people come in and ask. They also come in with "Hey, can you just fill my basket up from last time?" Which by the way, Clementine's exceptionally good at 'cause it saves you a lot of time and money. But what they slowly understand is, "Wait, hold on.

I did not have to go through that cognitive model of trying to figure out how to make a chocolate cake. I can just ask Clementine that I wanna make a chocolate cake. I wanna make it for four people. I wanna make it vegan based. Figure it out." And what's happening is we've noticed this transition that started happening for active users on Clementine.

Within the third session, they're beginning to make this transition going, "Oh, I can do that." And that's what's causing this rather ridiculous change in basket size that we're beginning to notice over multiple sessions because intent-based purchases are in addition to the purchases that people would make in terms of ingredients or buy it again or things of that sort.

So their regular purchases versus buy it again. So what we're seeing is the grocery space and the grocery set of purchases that people end up doing is still so under-penetrated that the more we can move towards intent-based understanding and do it well, you'll start having this movement that is, yeah, you have your regular things that you have to buy, whether it's your toilet paper and your milk and your sugar and that kind of stuff, but then you also have this intent-based thing that was never met before 'cause we didn't have the capabilities.

Those two, marrying the two of them together has been phenomenal and we've seen that in the results over the last few months.

Sam Witteveen: So another area of personalization that we're seeing taking off now is the whole area of sort of personal agents. Claude has just rolled out Claude Duet, Dots from OpenAI and probably the big one I would say has been Muse, right?

With their... from Meta. You guys built your own assistant, if I understand correctly. How do you sorta deal in this almost sort of frenemy world of where you've got your own thing, but you're having to deal with other connectors that are also bringing in customers?

AK: So a couple things over there.

We've always had the perspective that we wanna be there where our customers wanna engage with Instacart on. That's always been like a very predominant model of how we think about it and a tenet that we care about. So that's number one, so which automatically means that we've got to be where people wanna engage with the system.

Muse is obviously doing tremendously well. It's a very popular system. Instinct is another one. We're actually engaging with them at the moment. We know we have to be in those places. So then the second question that comes up is, can we ensure that the experience that we end up making available to our customers meets the expectations of what we have, whether that's on this platform or somewhere else?

What we've noticed based on some of the prior connectors that we've had, is people start their journey on those connectors, whatever that may be, whether that's Gemini or OpenAI or even Muse, and then they slowly migrate over to Instacart where you have the flexibility of two things, one which is the entire knowledge of what you have bought in the past, what do you wanna rebuy, things of that sort.

But on top of that, the fluidity of the experience, the speed of the experience and understanding that the fulfillment is actually looking at live data at this point in time. So there is this transition, so we wanna be where users want to engage with Instacart. We wanna make as much of that possible within that infrastructure.

But there is a movement that ends up happening towards our agents over a period of time to give us that flexibility and things of that sort.

Sam Witteveen: I can imagine though that must be pretty challenging though, because if whichever personal agent, if it's not as clued in how to make a chocolate cake and it's leaving ingredients out and stuff and you're just being told an ingredient list, it probably reflects worse on you than it does on them.

How do you deal with things like that?

AK: There are things that we are working with those teams directly to ensure that certain parts of the experience meet a certain standard before we end up opening it up. It's actually one of the reasons why we have not been particularly fascinated by the browser-based experiences that some folks have already built in with Instacart because it's slow and it doesn't cover all the cases that you think about, and it doesn't allow or support the ability to do the intent-based purchases that you end up having.

So we've encouraged people to move more into the MCP-based model where we've given certain accessors and certain capabilities over there, and the capabilities include the understanding of intent and feeding back results off of that. So we've shied away from the browser-based ones which are slow, and it doesn't allow the model that we are trying to push forward on in terms of how people interact with grocery-based purchases, and moved more into work with us and we'll build the connectors with you and try to expose that experience to the users, both in terms of speed but also in quality.

Sam Witteveen: When you are dealing with this kind of frenemy situation, you've got a lot of ability to design your product, to design your APIs, to design your MCPs. Are you thinking about it in a way of okay, are we trying to get them off Muse onto Instacart as quickly as possible, or whichever agent, it doesn't have to be Muse?

AK: One of the things to definitely recognize is that Muse or any of these other agents do not have the history of the purchases that have happened and your buying behaviors that have happened within the Instacart platform. And so to get that purely personalized version, which is like, I'm vegan and I will always be having these sets of products, or I'm looking for healthy kind of entities rather than something that's high caloric or things of that sort, those are things that Clementine has access to, so you don't have to repeat your purchases and repeat your intents every single time.

And so over time there is this movement that we have seen that users want to enjoy the benefits of that and end up moving over to Clementine to make their purchases.

Sam Witteveen: In things like Clementine going forward, as a large element of what you're dealing with too is dealing with brands, I'm sure.

Do you open up advertising? Do you open up sort of personal recommendations to people, "Hey, I notice you're vegan. Here's this new product on the market," or something like that?

AK: Instacart, for a very long time, has experimented and tried to understand how to play this blend between organic and inorganic behaviors.

In the earlier versions of Clementine and even the earlier versions prior to that, we've been more focused on what was that experience that truly gave people that idea of what Clementine could be and what should we or shouldn't we do, especially given the form factor and the supply, the ad suppliers in this particular case are different.

We have begun to start experimenting with even exposing both inorganic and organic results and blending them out into Clementine, and that will be coming out pretty soon. There is an opportunity for brands to be able to expose themselves just like they do in the context of the core product itself.

It's a matter of understanding what connects with the user from a form factor perspective and from an intent perspective. Those are the two things that we wanna be very careful about.

Sam Witteveen: You have a whole other part of your business, which is the enterprise part.

Can you tell us a little bit about Instacart's sort of enterprise offerings? How are you doing that? How's that going currently?

AK: First of all, the TL;DR on that, it's going exceptionally well. Instacart and its growth is pretty significant. What Instacart has done also is taken this consumer experience and allowed it to be democratized and made available to retailers.

And in that model, it's exposed every bit of the learnings that we have in terms of the models as far as what people connect with, what are the experiences that allow for the flow to be as seamless as possible, what are the optimizing experiences that users want, and what affects the business in a particular form.

All of those things are made available from an enterprise perspective. Now, in that world, we are also making Clementine available as this product called Cart Assistant within the enterprise. So now people can have that same level of fluid conversation that they have with Clementine in those experiences.

So for instance, a retailer that is built on top of our product called Storefront Pro, which is the enterprise product, can now get access to this and interact with their customers in that same fluid way, and then connect it with the fulfillment that basically then delivers that product to their place.

So our entire engineering stack, our entire product stack is made available over there, and the advancements that we have both in model tuning, in terms of how fast we can build, in terms of the experiments we can run, but in terms of understanding intent, is all made available to the enterprises.

Sam Witteveen: So when you build something like Clementine, basically an agent, a harness system kind of thing like that, do you actually have to think about it being different for an enterprise user versus what you've got in your marketplace? How do you think about those things through? And then do you actually make it flexible?

And to throw in another question, is that being improved by the fact that you're now able to rewrite the code base, and do you have multiple variations of the code bases?

AK: Really, really astute question, Sam. So yes, Clementine obviously has the persona and the mechanism of working with you the way that we want Instacart to be represented.

But other brands have their model in terms of how they wanna come through and what's the general persona of how they wanna present themselves to their customers. So Clementine has the ability to control, through the enterprise portal, what is it that you want to have and how do you wanna communicate with your customers.

So it has that. It also has the ability to highlight certain products, 'cause retailer brands have certain products that they market more or they push forward more. So you can even highlight those, and it will control results off of that part of it. So your enterprise persona doesn't get lost in this at all.

This is all made possible by this customization layer that I had spoken about earlier. What's really cool about this now is it's great to say that, but it's pretty complicated to do and it's pretty complicated to get right and it's even harder to maintain that over a period of time. 'Cause if you're managing so many different brands, how do you make sure that you didn't screw up something along the way?

Not on day one, but on day 180 of N number of brands becomes quite complicated. Our ability to now rebuild, not only rebuild that entire code base, but essentially partition the code base so that it can be uniquely different per brand, which would've been very much impossible before. 'Cause imagine like if you're doing like a Git merge across each one of these while you deploy them out would be impossible to manage.

Now those are beginning to become more possible. So our ability to rebuild that final layer, that customization layer that ends up going out to a retailer specifically, is something that is pretty powerful. And by the middle of next year, we're even thinking about having completely segregated instances of these things running especially for certain retailer brands and things of that sort.

But that all happens because you have this consistent output that can be generated on the fly when you deploy the code. So yeah, so this whole complexity that we've had in terms of how do you manage that set of variations that exist across the business gets even harder with Clementine because now it's more fluid by nature.

But then, on top of that, having to manage that would've been impossible without this ability to not have to think about the code, but think about the idea and the semantic idea of that particular business.

Sam Witteveen: Okay, in this world where you've built this harness, this agent system, how important are the models?

I'm really curious to hear from you. Can you talk about for Clementine, does it use open models, for example, as opposed to proprietary models?

AK: It does both. It does both. And going forward, the harness will be even more, I guess you could say, advanced, where depending on the domain that you're engaging with it, it will route it to proprietary models versus open models.

And even in some cases, it'll skip the models altogether and just go into a cached layer altogether. Today it does both. So the end result that you see is a combination of both. But over time, we wanna prune out the set of ones that go to the frontier models, except for when there are understandings that have not been imprinted into Clementine, for instance.

Sam Witteveen: Are you guys fine-tuning models?

AK: We are. So we do fine-tune. So the search results thing that I talked about before, the SLM that gets in, the small language model that gets generated, is a fine-tuned model that we build, which is the thing that takes you into the long tail of responses that end up coming out in search.

So we do fine-tune that model. And then on the Clementine side of the equation, in certain conditions, yes, we do fine-tune them.

Sam Witteveen: It sounds to me... I'm very curious if you can tell us a percentage, rough sort of percentages, but it sounds to me that you've definitely established a sort of moat by having open models, by fine-tuning some of those open models.

And contrary to what a lot of people think, organizations like yourself who've actually worked out some of these things are not reliant on the big model companies that are about to go public.

AK: I had mentioned earlier that we were able to reduce our cost for development by about 75%.

I would say that two things contributed to that the most. The first one, which was obviously the movement towards open source models. Right now, pretty much every interaction that ends up happening between a developer and the generation of the code through the model pretty much happens on an open source model with some caveats.

There is an automatic understanding that happens in the harness that basically says that if the developer is not getting the answer that they want, automatically upscale them to a frontier model, for instance. So there is a capability that exists of this harness that we've built that does this. We've looked at every single session that exists over the last six months of developer sessions within Instacart, and so we've begun to understand where the model works well in certain domains versus not, and so there's an automatic escalation that is done in terms of the level.

So that's the first set of things that have happened in terms of reducing that. So we don't need to have everybody learn when to apply what model, although we've seen our developers automatically pick... The model that's mostly moved towards right now is the Luna model because it's just ridiculously fast, so that's like the default choice.

The second one, which is this breakup of the planning phase from the development of the code phase side of the equation. Why does that matter? It matters because if you're engaging, if you're doing the planning as part of your development, you end up falling back into the frontier models quite a bit.

Separating those two things has helped us be able to like, create this breakup that has been very efficient. The same thing applies in the context of Clementine. So Clementine actually has a-- think of it like a pseudo query understanding layer. That query understanding layer used to be a frontier model.

Now it's an SLM that's built off of an open source model, which then feeds into the appropriate set of models based on domains that it recognizes, and that has been very important in terms of reducing our costs. So we've been able to reduce the cost of serving per session on Clementine in the last six months by, think of it, about one-eighth.

So it's about one-eighth the cost of what it was about six months ago, eight months ago.

Sam Witteveen: Where do you see this going, both from a developer, from the internal, managing the team, the development of the product, but also just what you're able to do? Because the thing I found fascinating on this call is just that whole sort of intent-based thing radically changes products, right?

You guys are probably the best example out there at the moment that is showing just how really deep understanding of a customer and their needs allows you to basically deliver so much more.

AK: 100%, what you just said, Sam, this experience and learnings that we've had is the differentiation in terms of, like, all the things we've talked about where data matters in the past, in the last 15 years, they absolutely matter, and that's the thing that differentiates to a certain degree.

But what matters now is the fluidity and the flexibility of how you make and use that to meet the customer's needs, so take away that cognitive overload from them. But in the journey of building, one of the things that we are beginning to see quite a bit right now is, I had mentioned earlier that the number of ML models that we can put out right now has gone up significantly because of the optimization speed that we've been able to get.

We've talked about how developer building of product and features has been severely sped up. The thing that we are beginning to see is we're beginning to now think about a lot of experiments that we wanna be able to run that's now feasible based on a building side of the equation. The part that we are struggling with and trying to figure out how to do is how do we create a world where the top of the funnel provides enough opportunities for us to experiment to figure out, A, understanding of intent, B, making sure that we're actually meeting the needs of the user?

So if you think about in the past, you had a very defined flow. It was like, here's discovery, here's add to cart, here's search, here's checkout, and you basically walked this flow and you said, "Hey, where did my conversion drop?" Right now there is no such flow that exists. You have to understand that part of it.

Are you doing that correctly? Because if you're not, then there's something wrong, which is why the user is bailing on you. So those experiments as far as understanding the intent itself is quite complicated. That's the part that we're spending a lot of time on. If we truly wanna be hyper-personalized per user, how do we support going beyond A/B tests?

That's the one that is something we're spending a lot of time on, which is like, how can I create multiple variations of these results and make it unique per user? So a thing that we are spending quite a bit of time on right now is synthetic users and synthetic capabilities of us understanding even before we can put it out there.

I think that over the next couple years, e-commerce companies especially will have to start thinking about if I truly wanna be hyper-personalized, that it's not A versus B anymore, how do I have an engineering system that can support so many variations, and how do I have a system that validates that before I even put it out to the user?

Sam Witteveen: It's all about verification, right? That you can do something but then you've gotta verify how did you actually... was this the right thing or not?

AK: Yeah. And I think engineering is changing into that how do you make it possible for your systems to be, to have that capability, which is allow the models to generate this non-deterministic output, non-deterministic output both in code and for your users, have so many variations of it, be able to manage that part of it, and make sure that every user is getting the right experience.

But most importantly, how do you feel confident that happened correctly? It goes right back to what you just said, which was verify that. And that, I think we're gonna have a lot of investments put into that verification part of it, whether that's synthetic users, whether that is understanding every user on their own, whether it's understanding intent and then being able to replicate it elsewhere.

So yeah.

Sam Witteveen: Okay, fascinating conversation. Really, really enjoyed it. I think we're gonna have to get you back in another six months' time to talk about synthetic users and delve into all the ways that you're doing verification then. Thank you very much for joining us on Beyond the Pilot today.