0.1s
I would have thought this would be different 13 months later, but this is still very much the case where agent is synonymous with unreliable, stochastic, demoware, vaporware, and I think something very important for us is we want to build these systems that arent just like cool examples of what is to come, but rather valuable today
0.4s
thought this would be different
1.7s
13 months later,
3.0s
but this is still very much
4.0s
the case
4.3s
where agent is synonymous
7.1s
with unreliable,
8.7s
stochastic, demoware, vaporware,
12.5s
and I think something
13.6s
very important for us is
15.5s
we want to build these systems
16.7s
that arent
17.5s
just like cool examples of
18.8s
what is to come,
19.9s
but rather valuable today
21.8s
and not just valuable for like a hacker on a side project but valuable to enterprise engineers today.
22.8s
for like a hacker on a side project
24.8s
but valuable to enterprise
26.5s
engineers today.
44.0s
Hi, and welcome to training data. We have with us today Matan Grinberg and Eno Reyes, founders of Factory. Factory is building autonomous software engineering agents, or droids, that can automate everything from the drudgery of maintaining your documentation to actually writing code for you. In doing so they are building the ultimate compound lever. Last week, Factory also announced
45.8s
We have with us today
46.8s
Matan Grinberg and Eno Reyes,
48.4s
founders of Factory.
50.0s
Factory is building
50.9s
autonomous software engineering agents,
52.9s
or droids,
54.0s
that can automate everything
55.2s
from the drudgery
56.0s
of maintaining your documentation
57.6s
to actually writing code for you.
59.7s
In doing
60.2s
so they are building
60.9s
the ultimate compound lever.
63.1s
Last week, Factory
64.2s
also announced
64.8s
some impressive results on the key AI coding benchmarks SWE-bench, beating state of the art by a wide margin. Stay tuned at the end of the episode for more context on how they built it. Were here with Matan Grinberg and Eno Reyes, founders of Factory. Gentlemen, thank you for joining us. Thank you so much for having us.
66.3s
AI coding benchmarks SWE-bench, beating
68.5s
state of the art by a wide margin.
70.4s
Stay tuned at the end of the episode
72.0s
for more context on how they built it.
75.8s
Were here with Matan
76.8s
Grinberg and Eno Reyes,
79.0s
founders of Factory.
80.7s
Gentlemen, thank you for joining us.
82.9s
Thank you so much for having us.
84.6s
Yeah, thanks for having us. Lets start. Lets start with a little bit of personal background. And Matan, maybe well start with you and then go to Eno. So Matan. One thing that I believe you and I share in common is an affinity for a well-executed cold call. I know at least two cold calls
86.5s
Lets start.
87.0s
Lets start
87.2s
with a little bit of personal background.
88.6s
And Matan, maybe well
90.1s
start with you and then go to Eno.
91.6s
So Matan.
92.8s
One thing that I believe
94.6s
you and I share in common
96.2s
is an affinity
97.1s
for a well-executed cold call.
99.8s
I know
100.1s
at least two cold calls
101.3s
that have had some bearing on your life. Why dont we start with the one that you did, as an undergrad at Princeton, to somebody who my partner, Shaun Maguire tells me is quite a famous physicist. Can we start with that cold call? Yeah, absolutely. So while I was at Princeton, I was studying string theory.
103.0s
Why dont we start with
103.6s
the one that you did,
104.8s
as an undergrad at Princeton,
107.0s
to somebody who
108.3s
my partner, Shaun Maguire tells me
109.9s
is quite a famous physicist.
111.8s
Can we start with that cold call?
113.8s
Yeah, absolutely.
114.8s
So while I was at Princeton,
117.1s
I was studying string theory.
119.6s
And the most famous string theorist happened to be working at the Institute for Advanced Study, which is an academic institution, right next to Princeton University, but not technically affiliated with it. Part of the allure of going to the IAS is that you dont have to take on graduate students, much less undergrads. That said, you know, theres
122.5s
theorist happened to be working
124.1s
at the Institute for Advanced Study,
125.8s
which is
127.0s
an academic institution,
128.2s
right next to Princeton University,
130.0s
but not technically affiliated with it.
132.4s
Part of the allure of going to the
134.3s
IAS is that you dont
135.8s
have to take on
137.1s
graduate students, much less undergrads.
140.0s
That said,
141.1s
you know, theres
142.7s
a professor there, Juan Maldacena, who is, you know, by far the kind of leader of the string theory movement. And, you know, being a young, ambitious undergrad, I, I decided, you know, might as well see if I could snag him as an advisor. And so with some advice from some graduate students, I sent him an email,
144.1s
who is, you know, by far
145.7s
the kind of leader of
147.5s
the string theory movement.
149.2s
And,
150.8s
you know, being a young,
152.2s
ambitious undergrad, I,
154.8s
I decided, you know, might as well
156.5s
see if I could snag him as an advisor.
159.5s
And so with some advice
162.1s
from some graduate students,
163.2s
I sent him an email,
165.1s
asked if we could meet. And the thing about Juan is the way he works with people, you know, hell take a meeting with anyone, basically, and will spend about two hours at the chalkboard with you. And in this two hour chalkboard session, hell subtly drop a problem that you basically have 24 hours to solve,
166.6s
And the thing about Juan
167.8s
is the way he works with people,
169.8s
you know, hell take a meeting
170.6s
with anyone, basically,
172.0s
and will spend about two hours
174.0s
at the chalkboard with you.
175.7s
And in this two hour chalkboard session,
178.5s
hell subtly drop a problem
181.7s
that you basically
182.5s
have 24 hours to solve,
183.8s
and get back to him with the solution. And then youll officially, you know, be a student of his. Luckily, I was warned about this, about this rite of passage. So yeah, I was paying close attention to any hints he was dropping. Indeed, yes, yes. So I found the problem, ended up spending, you know, basically the entire night
186.1s
And then youll officially,
187.2s
you know, be a student of his.
189.1s
Luckily,
190.0s
I was warned about this,
191.2s
about this rite of passage.
192.6s
So yeah, I was paying close attention
195.8s
to any hints he was dropping.
197.6s
Indeed, yes, yes.
200.4s
So I found the problem,
203.4s
ended up spending, you know,
205.4s
basically the entire night
206.5s
working on it, and, you know, luckily ended up having him as an advisor. We were able to then publish a paper together, which was very exciting. So, yeah. Typical undergrad experience. Yes. Yes, exactly. So theres, theres a second cold call that I want to ask you about. Before we get to that,
207.6s
you know, luckily ended up
209.0s
having him as an advisor.
210.7s
We were able to
211.5s
then publish a paper together, which was
214.2s
very exciting. So,
217.0s
yeah.
217.8s
Typical undergrad experience.
220.4s
Yes. Yes, exactly.
222.1s
So theres, theres a second cold call
223.6s
that I want to ask you about.
225.3s
Before we get to that,
226.5s
why dont we go to Eno, so Eno, you similarly went to Princeton, you have a CS degree from there, you spent some time as a machine learning engineer at Hugging Face, which is where we first intersected spent some time at Microsoft. But like a lot of great founders, your story before then started with some humble beginnings.
227.0s
you similarly went to Princeton,
230.0s
you have a CS degree from there,
231.1s
you spent some time as a machine
232.2s
learning engineer at Hugging Face,
233.5s
which is where we first intersected
235.4s
spent some time at Microsoft.
237.4s
But like a lot of great
238.2s
founders, your story before
240.2s
then started with some humble beginnings.
242.0s
Could you say a word about the stuff that doesnt appear on LinkedIn that has helped to shape who you are today? Yeah, absolutely. And I think that, you know, my family on my dads side came from Mexico in the late 60s to San Francisco. And my grandparents were both working for a bit.
243.4s
that doesnt appear on LinkedIn
244.9s
that has helped to shape
245.8s
who you are today? Yeah, absolutely.
247.9s
And I think that,
249.8s
you know, my family on my dads side
252.4s
came from Mexico
253.6s
in the late 60s to San Francisco.
256.3s
And my grandparents
258.3s
were both working for a bit.
259.5s
But when my dad was born, they started a Mexican restaurant in Los Altos. And that was in the 70s. They moved it to Haight and Cole in the 80s, and were a very kind of San Francisco immigrant story. They actually ended up leaving, to Georgia, where I grew up. But really, I think it's the drive
261.2s
they started a Mexican
263.1s
restaurant in Los Altos.
264.7s
And that was in the 70s.
268.0s
They moved it to Haight
269.2s
and Cole in the 80s,
270.3s
and were a very kind of San Francisco
273.3s
immigrant story.
274.8s
They actually ended up leaving,
276.1s
to Georgia, where I grew up.
277.9s
But really, I think it's the drive
281.5s
that they had to give my dad a successful life in America. And it was my dad and my mom that drove that same kind of mentality and to me growing up, and I think its really cool because this story is one that I think a lot of Americans share and something that makes it really exciting
283.0s
a successful life in America.
285.5s
And it was my dad
287.6s
and my mom
288.1s
that drove that same kind of mentality
290.2s
and to me growing up,
291.2s
and I think its really cool
293.5s
because this story is one
294.8s
that I think a lot of Americans share
296.4s
and something
297.1s
that makes it really exciting
298.3s
to be back in San Francisco, kind of building, building something to potentially make the world a better place for everyone. Very cool. That is the dream. Matan, I want to get back to that other cold call, because I think it leads directly into the founding or the forming of Factory. So our partner, Shaun Maguire,
300.0s
kind of building, building something
302.1s
to potentially make the world
304.0s
a better place for everyone.
305.9s
Very cool.
307.3s
That is the dream.
308.8s
Matan,
310.1s
I want to get back
310.7s
to that other cold call,
312.1s
because I think it leads
313.0s
directly into the founding
314.9s
or the forming of Factory.
316.9s
So our partner, Shaun Maguire,
318.8s
who I mentioned earlier, who I believe shares a similar academic background to your own, Ireceived an email from you a year or so ago, that led to a walk. And very shortly thereafter, Factory was formed. So Im curious. What caused you to cold call Shaun Maguire and this is less of a Shaun Maguire question because we
320.2s
who I believe shares
321.1s
a similar academic
322.5s
background to your own,
324.4s
Ireceived an email from you
326.8s
a year or so ago, that led to a walk.
329.9s
And very shortly thereafter,
331.5s
Factory was formed.
332.8s
So Im curious.
334.8s
What caused you to cold call
336.8s
Shaun Maguire
337.4s
and this is less of a
338.1s
Shaun Maguire question
339.1s
because we
339.8s
know plenty about Shaun Maguire. This is more of like, youre on a very good path, you do really good research, youre on track to get a PhD in physics. And something inspired you to go in a different direction. And Im curious, what was it that inspired you that led to that cold call?
341.2s
This is more of like,
342.5s
youre on a very good path,
344.7s
you do
345.2s
really good research,
346.2s
youre on track to get a PhD in physics.
349.0s
And something inspired you
350.9s
to go in a different direction.
352.4s
And Im curious,
353.2s
what was it that inspired you
354.8s
that led to that cold call?
355.9s
And maybe tell us a quick story about what happened shortly thereafter? Yeah, absolutely. So, you know, I was like you said, I was doing my PhD at Berkeley. About a year in though I realized that I was only doing theoretical physics, and string theory, because it was hard. And not because I actually loved it,
357.2s
about what happened shortly thereafter?
359.5s
Yeah, absolutely.
360.4s
So, you know, I was like you said,
362.3s
I was doing my PhD at Berkeley.
364.4s
About a year in though I realized that
368.5s
I was only doing
369.6s
theoretical physics, and string theory,
371.4s
because it was hard.
372.4s
And not because I actually loved it,
374.5s
which is obviously a bad reason, bad reason to do anything. And, you know, I had such tunnel vision on this path that, you know, when I came to this realization, it was kind of earth shattering, and I looked at the paths ahead of me. And there were basically three options that seemed realistic. And so
375.9s
bad reason, bad reason to do anything.
377.1s
And,
378.5s
you know,
378.8s
I had such tunnel vision
380.5s
on this path that,
382.3s
you know,
382.5s
when I came to this realization,
383.8s
it was kind of earth shattering,
385.6s
and I looked at the paths ahead of me.
387.6s
And there were basically three options
389.6s
that seemed realistic.
391.2s
And so
392.0s
it was either going into quantitative finance, going into big tech, or going into startups. And by this time, Id already kind of switched my research at Berkeley from being purely physics to ML in physics, and then slowly more ML, and then mostly AI. So it was kind of quickly cascading there. At the time,
392.8s
into quantitative finance,
394.4s
going into big tech,
395.5s
or going into startups.
397.1s
And by this time, Id already
399.4s
kind of switched my research at Berkeley
401.3s
from being purely physics to
403.8s
ML in physics, and then slowly more ML,
406.5s
and then mostly AI.
407.5s
So it was kind of
408.9s
quickly cascading there.
410.5s
At the time,
411.4s
I think I saw a video of Shaun speaking, I think over Zoom to some founders at Stanford or something. And I recognize his name from string theory research, because I had read his papers way back in the day. And it was particularly shocking to me, because Im not sure how much time you spent with string theorists.
414.4s
I think over Zoom to some founders
417.2s
at Stanford or something.
418.8s
And I recognize his name
420.2s
from string theory research,
421.4s
because I had read his papers
422.7s
way back in the day.
424.0s
And it was particularly shocking to me,
425.5s
because Im not sure
427.1s
how much time you spent
428.1s
with string theorists.
428.9s
But normally, theyre quite introverted. Not, Not, Not... not the most socialized. Yeah, and so Shaun is, you know, this very different example. And so, to me, I kind of like, like, I looked at his background, and it was just it was just shocking to see someone who was like, so deep and like a bonafide string theorist,
431.9s
Not, Not, Not... not
433.8s
the most socialized.
435.0s
Yeah, and so Shaun is,
436.7s
you know, this very different example.
439.2s
And so, to me, I kind of like, like,
441.2s
I looked at his background,
442.6s
and it was just
443.1s
it was just shocking
443.9s
to see someone
445.1s
who was like, so deep
446.3s
and like a bonafide string theorist,
448.6s
then go in and like, start his own companies, invest in some of the best companies, join Sequoia and be, you know, a partner there. And to me, it was just like, Oh, my God, this seems like someone who is of my kind of background, of my nurturing, I guess. And so I sent him an email.
450.0s
start his own companies, invest
451.5s
in some of the best
452.1s
companies, join Sequoia
453.6s
and be, you know, a partner there.
455.9s
And
457.9s
to me, it was just like, Oh, my God,
458.9s
this seems like someone who is of my
462.0s
kind of background, of my nurturing,
465.7s
I guess. And so I sent him an email.
468.1s
And I was just like, hey, you know, we both were string theorists, I dont want to do string theory anymore. Im thinking about AI, wed love to get your advice, like you mentioned that then turn into a walk, it actually was supposed to be a 30 minute walk. We ended up going
469.0s
you know, we both were string theorists,
470.6s
I dont want to do string theory anymore.
472.7s
Im thinking about AI,
473.8s
wed love to get your advice,
475.2s
like you mentioned that
476.1s
then turn into a walk,
477.5s
it actually was supposed to be
478.6s
a 30 minute walk.
479.5s
We ended up going
481.0s
from the Sequoia offices in Menlo Park, all the way to Stanford, and then back. And so it ended up in three hours. He missed a lot of his meetings that day. So it was pretty amusing. And basically at the conclusion of the walk, he said, the one thing was for sure, he was saying
483.7s
all the way to Stanford, and then back.
485.7s
And so it ended up in three hours.
487.5s
He missed a lot of his meetings that day.
489.4s
So it was pretty amusing.
492.6s
And basically
493.3s
at the conclusion of the walk,
494.9s
he said,
495.8s
the one thing was for sure,
497.1s
he was saying
498.0s
you must drop out of your PhD, theres way too many exciting things to do. And he kind of left me with the advice of, you should either join Twitter right now, because this is just after Elon took over. And he was saying, you know, only the most badass people are going to join Twitter right now, too.
499.6s
theres way
500.0s
too many exciting things to do.
501.8s
And he kind of left me
503.0s
with the advice of,
504.7s
you should either join Twitter right now,
507.1s
because this is just after Elon
508.7s
took over.
509.5s
And he was saying,
510.5s
you know,
511.0s
only the most badass people
512.5s
are going to join Twitter right now, too.
514.9s
You could join a company of mine as just like a general glue is what he said, his is Foundry by the way. Yep. Or three, if theres some ideas that youve been thinking about, you should start a company. And I was like, you know, very grateful for all the time that he spent and,
516.7s
as just like a general
518.9s
glue is what he said,
520.2s
his is Foundry by the way.
521.8s
Yep.
522.1s
Or three,
523.1s
if theres some ideas
524.0s
that youve been thinking about,
525.1s
you should start a company.
526.6s
And I was like, you know, very grateful
529.4s
for all the time that he spent and,
530.8s
you know, we kind of left off there. Beautifully, in parallel, Eno and I had just reconnected at a LangChain hackathon. And he was in Atlanta the weekend prior, and he basically got back the next day. So that next day, Eno and I got coffee, and I think we got coffee at noon. And then basically every hour
533.2s
Beautifully, in parallel,
535.4s
Eno and I had just
536.9s
reconnected at a LangChain hackathon.
539.8s
And he was in Atlanta the weekend prior,
543.0s
and he basically got back the next day.
545.3s
So that next day,
546.5s
Eno and I got coffee,
548.0s
and I think we got coffee at noon.
550.8s
And then basically every hour
553.0s
since then, until now, Eno and Ive been working together, talking constantly about code generation and what became Factory. Did you and Eno know each other in undergrad? We had like the maximal overlap of mutual friends without ever having had a one-to-one conversation. Yeah, its pretty funny. We were in eating clubs at the time opposite from each other.
554.8s
Eno and Ive been working
555.8s
together, talking
557.0s
constantly about code generation
560.5s
and what became Factory.
562.7s
Did you and Eno
564.2s
know each other in undergrad?
567.1s
We had like the maximal
568.3s
overlap of mutual friends
569.8s
without ever having
570.8s
had a one-to-one conversation.
573.7s
Yeah, its pretty funny.
575.2s
We were in eating clubs
577.6s
at the time opposite from each other.
579.3s
And we had just so many mutual friends and it really wasnt until I moved to the Bay Area that we had a face to face convo and it was a very fruitful conversation for sure. It was intellectual love at first sight, you could say. I love that and its so serendipitous about the LangChain connection.
580.6s
so many mutual friends
581.9s
and it really wasnt
583.8s
until
584.7s
I moved to the Bay Area
585.8s
that we had a face to face convo and
588.5s
it was a very fruitful conversation
590.8s
for sure.
591.7s
It was intellectual
592.5s
love at first sight, you could say.
596.6s
I love that and its so serendipitous
598.3s
about the LangChain connection.
599.8s
How did you guys decide on, Im curious, I mean, youre both brilliant. And I think for a lot of founders starting out in AI right now, a lot of them find it hard to resist the siren call of training a foundation model. So like, how did you decide to, you know, build in the application layer? Im curious.
602.8s
I mean, youre both brilliant.
603.9s
And I think for a lot of founders
605.3s
starting out in AI right now,
607.8s
a lot of them
608.2s
find it hard to resist the siren call of
610.8s
training a foundation model.
612.0s
So like, how did you decide to,
613.7s
you know, build in the application layer?
616.0s
Im curious.
616.6s
And then why software engineering? Yeah. So I think from my perspective, like going deep from academia, I think, throughout all the years of spending time on math and physics, the theory of beauty that I learned to be drawn to was things being fundamental. And, you know, spending, you know, time doing AI research, it was so clear
620.3s
Yeah.
620.5s
So I think from my perspective,
622.2s
like going deep from academia,
623.9s
I think,
625.0s
throughout all the years
626.1s
of spending time on math and physics,
628.3s
the theory of beauty that I
631.2s
learned to
632.4s
be drawn to was things being fundamental.
635.2s
And, you know, spending,
637.0s
you know, time doing AI research,
639.3s
it was so clear
640.2s
that code is fundamental to machine intelligence. And so I was just naturally attracted to the role that it plays there. And I think that kind of joined quite well with Enos attraction to the space. Youve referred to it a couple of times as a compound lever. Can you unpack that for us?
642.3s
to machine intelligence.
643.5s
And so I was just naturally attracted to
646.8s
the role that it plays there.
648.6s
And I think that kind of joined
650.6s
quite well
651.1s
with Enos attraction to the space.
654.9s
Youve referred to it
655.8s
a couple of times as a compound lever.
658.2s
Can you unpack that for us?
660.0s
And let us know what that means? Yeah, I mean, so there's the famous Archimedes quote about, you know, software, or well, his quote is rather, you know, if you have a large enough lever, you can move the world. And then I think thats been co-opted for software engineering, right, that software is a lever upon the world.
662.0s
Yeah, I mean,
662.7s
so there's the famous Archimedes
665.3s
quote about,
666.5s
you know,
667.0s
software, or well, his quote is rather,
669.2s
you know,
669.5s
if you have a large enough lever,
670.9s
you can move the world.
672.1s
And then I think thats been co-opted
673.8s
for software engineering, right,
675.1s
that software is a lever upon the world.
677.9s
And for us, we see AI and in particular AI code generation, as a lever on software, the impacts of that being, you know, compounding, exponential. And sorry, Eno, I think I cut you off, I think you were maybe mentioning how you got to the founding inspiration for Factory? Oh, yeah, absolutely, I mean,
680.4s
AI code generation,
681.8s
as a lever on software,
684.8s
the impacts of that
685.6s
being,
686.5s
you know, compounding, exponential.
688.8s
And sorry, Eno,
689.5s
I think I cut you off, I think you were
691.8s
maybe mentioning how you got to
695.4s
the founding inspiration for Factory?
697.9s
Oh, yeah,
698.3s
absolutely, I mean,
699.0s
I think Matan story is really indicative of kind of the energy at the time I was at Hugging Face working on, you know, training, optimizing, deploying LLMs, for enterprise kind of use cases, was actually working with Harrison on early like LangChain kind of integrations. And it was so clear that the work that was happening in open
702.2s
of kind of the energy at the time
704.0s
I was
704.9s
at Hugging Face working on,
706.8s
you know,
707.1s
training, optimizing,
708.5s
deploying LLMs, for enterprise
710.7s
kind of use cases,
711.8s
was actually working
712.5s
with Harrison
713.2s
on early like LangChain
715.1s
kind of integrations.
716.6s
And it was so clear
717.8s
that the work
718.8s
that was happening in open
720.0s
source was directionally moving towards modeling human cognition with LLMs, where the LLM was just one piece of the system. The idea of chains, and I think Harrison calls them cognitive architectures and the LangChain folks call it that. And seeing that happening, and seeing that, within the code gen space, the most advanced players were basically looking at autocomplete,
723.1s
modeling human cognition with LLMs,
726.6s
where the
727.2s
LLM was just one piece of the system.
730.4s
The idea of chains, and
732.7s
I think Harrison calls them
733.7s
cognitive architectures
734.8s
and the LangChain folks call it that.
736.4s
And seeing that happening,
739.2s
and seeing that,
740.5s
within the code gen space,
742.2s
the most advanced players
743.5s
were basically looking at autocomplete,
745.7s
it felt like there was a huge opportunity to take that to the next step. And take some of those lessons that were happening both in the kind of the fringe research and open source communities and applying them towards a kind of massive organization. I realize we havent set explicitly yet, what is Factory? So Matan, what is Factory?
746.4s
there was a huge opportunity
748.0s
to take that to the next step.
750.5s
And take some of those lessons
752.6s
that were happening both
753.9s
in the kind of the fringe research
755.5s
and open source communities
756.8s
and applying them towards
758.0s
a kind of massive organization.
761.9s
I realize we havent set explicitly yet,
764.0s
what is Factory?
765.2s
So Matan, what is Factory?
768.2s
And then maybe what are a couple of the key decisions that youve made about the way Factory is built. And you know, for example, one of them is to start by benefiting from all the ongoing improvements in the foundation model layer. You know, one of them might be the product itself,
770.0s
what are a couple of the key
771.2s
decisions that youve made
772.6s
about the way Factory is built.
774.2s
And you know, for example,
775.2s
one of them is to start by benefiting
777.8s
from all the ongoing
778.8s
improvements in the foundation
780.0s
model layer.
781.0s
You know,
781.2s
one of them might be the product itself,
783.3s
but can you just say what is Factory? And what are some of the key decisions youve made that have shaped Factory today? Yeah, absolutely. So Factory is a cutting edge AI startup. Our mission is to bring autonomy to software engineering. What that means more concretely, we are automating tasks in the software development lifecycle. And in particular
784.9s
And what are some of the key decisions
786.5s
youve made that have shaped
787.7s
Factory today?
789.0s
Yeah, absolutely.
790.3s
So Factory is a cutting edge AI startup.
794.5s
Our mission is to bring autonomy
797.6s
to software engineering.
799.4s
What that means more concretely,
801.7s
we are automating
802.6s
tasks in the software
803.9s
development lifecycle.
805.8s
And in particular
807.0s
tasks like code review, documentation, testing, debugging, refactoring. And, you know, as I list these off, youll kind of hear quickly that these are the tasks that engineers dont particularly enjoy doing. And thats very much intentional, right? Like, obviously, we are doing code generation. And thats really important. But I think, an equally important thing, too, you know,
810.0s
testing, debugging, refactoring.
813.3s
And, you know, as I list these off,
815.0s
youll kind of hear quickly
816.4s
that these are
817.0s
the tasks that engineers dont
818.3s
particularly enjoy doing.
820.9s
And thats very much intentional, right?
823.0s
Like, obviously,
824.4s
we are doing code generation.
825.8s
And thats really important.
827.1s
But I think,
828.1s
an equally important thing, too,
829.9s
you know,
831.0s
generating some inspirational and like forward looking demos, its also important to understand what engineers are actually spending their time on. And in most organizations, its not actually fun development work. In most organizations, theyre spending a lot of their time on things like review and testing and documentation. Normally, theyll do these things way too late. And theyre suffering
833.5s
and like forward
834.2s
looking demos,
835.8s
its also important to understand
836.9s
what engineers
837.7s
are actually spending their time on.
839.7s
And in most organizations, its
841.8s
not actually fun development work.
844.3s
In most organizations, theyre
845.5s
spending a lot of their time
846.5s
on things like review
848.0s
and testing and documentation.
850.4s
Normally, theyll do
851.0s
these things way too late.
852.2s
And theyre suffering
853.1s
because theyre missing deadlines, right. And so our approach is, we want these tools to be useful in the enterprise. And so to do that, we need to kind of meet engineers where they are with the tasks that they are very eager to automate away. We call these autonomous systems droids, and like Eno was alluding to earlier,
854.9s
And so our approach is, we want
858.0s
these tools
859.4s
to be useful in the enterprise.
861.8s
And so to do that,
862.6s
we need to kind of meet engineers
864.0s
where they are
864.9s
with the tasks that they are very eager
867.1s
to automate away.
868.5s
We call these autonomous systems
870.8s
droids,
872.0s
and like Eno was alluding to earlier,
874.2s
these are kind of, theres a droid for each category of task. And in this kind of a paradigm where we want to frame these problems as games. Its very convenient that software development has a clearly defined software development lifecycle. And so for each kind of category of tasks, where each step in the software development lifecycle,
876.3s
theres a droid for each
877.5s
category of task.
879.2s
And in this kind of a paradigm where
882.5s
we want to frame these problems
884.5s
as games.
885.8s
Its very convenient
887.0s
that software
888.1s
development
888.6s
has a clearly defined software
890.6s
development lifecycle.
891.9s
And so for each
893.1s
kind of category of tasks, where each
894.8s
step in the software
895.9s
development lifecycle,
897.0s
we have a corresponding droid. So that's kind of a first pass there. I guess there were, I think there was a second part of your question that I missed. Oh, well get into the rest of it. Where did the name droid come from? It's a pretty catchy name. It's very memorable and distinct to Factory.
900.1s
So that's kind of a first pass there.
902.4s
I guess there were,
903.2s
I think there was a second part
904.2s
of your question that I missed.
907.0s
Oh, well get into the rest of it.
908.8s
Where did the name droid come from?
911.4s
It's a pretty catchy name.
912.4s
It's very memorable
913.8s
and distinct to Factory.
915.0s
Whered that come from? Yeah, yeah. So I mean, keep in mind, you know, when Factory started, this was, like you mentioned about a year and a month ago. And you know, actually, I would have thought this would be different 13 months later, but this is still very much the case where agent is synonymous with unreliable,
916.5s
So I mean, keep in mind,
918.4s
you know, when Factory started, this was,
921.2s
like you mentioned about a year
923.2s
and a month ago.
924.1s
And you know, actually,
925.5s
I would have thought
926.2s
this would be different
927.3s
13 months later,
928.7s
but this is still very much the case
930.0s
where agent is synonymous
932.8s
with unreliable,
934.5s
stochastic, demoware vaporware. And I think something very important for us is we want to build these systems that arent just like cool examples of what is to come, but rather valuable today. And not just valuable for like a hacker on a side project but valuable to enterprise engineers today. We felt very strongly
938.2s
And I think something
939.2s
very important for us is
941.2s
we want to build these systems
942.3s
that arent
943.2s
just like cool examples of
944.5s
what is to come,
945.5s
but rather valuable today.
947.5s
And not just valuable
948.5s
for like a hacker on a side project
950.4s
but valuable to enterprise
952.1s
engineers today.
953.5s
We felt very strongly
955.2s
that agents just don't really capture what were trying to deliver. And so fun fact, we were originally incorporated as the San Francisco Droid Company. But upon legal advice and given I guess the eagerness with which Lucasfilm pursues its trademarks, we changed our name to Factory. Fair enough. So is it? Is it fair
957.6s
what were trying to deliver.
959.2s
And so fun fact,
960.6s
we were
961.2s
originally incorporated
962.4s
as the San Francisco Droid Company.
965.2s
But upon legal advice
967.9s
and given
968.5s
I guess
970.3s
the eagerness
971.0s
with which Lucasfilm pursues
972.6s
its trademarks,
973.3s
we changed our name to Factory.
976.8s
Fair enough.
977.4s
So is it?
978.2s
Is it fair
978.7s
to say that a droid is sort of like a job specific autonomous agent that actually works? Is that a reasonable way to think about it? Yeah. Okay, exactly. You just said the words cognitive architecture. And I know my partner, Sonya Huang well enough to know that this is her love language. So Im sure that Sonias mind
980.5s
like a job specific autonomous agent
983.7s
that actually works?
985.0s
Is that a reasonable way to
986.1s
think about it?
987.2s
Yeah. Okay, exactly.
989.5s
You just said the words
990.4s
cognitive architecture.
991.8s
And I know my partner, Sonya Huang
993.8s
well enough to know
994.6s
that this is her love language.
996.0s
So Im sure that Sonias mind
997.5s
just lit up with a whole bunch of questions for you. So I dont want to get in the way. Sonya, have at it. We just had Harrison on the podcast, who talked about custom cognitive architectures as well, I guess, what are you doing on that front? And how do your implementations dovetail with the multi-droid strategy
998.1s
with a whole bunch of questions for you.
999.4s
So I dont want to get in the way.
1001.0s
Sonya, have at it.
1002.6s
We just had Harrison on the podcast,
1004.2s
who talked about custom
1005.1s
cognitive architectures as well,
1006.8s
I guess,
1007.1s
what are you doing on that front?
1008.5s
And how do your implementations
1010.2s
dovetail with the multi-droid strategy
1012.2s
that youre talking about? Yeah, absolutely. I mean, its a great question. And I think the way that we think about reasoning, and, and cognition within the standpoint of these systems, there are clearly huge innovations happening on both layers, the foundation model layer, as well as on the kind of orchestration or kind of application layer.
1013.5s
Yeah, absolutely.
1014.7s
I mean, its a great question.
1016.3s
And I think the way that we think about
1021.0s
reasoning, and, and cognition
1024.0s
within the standpoint of these systems,
1027.4s
there are clearly huge
1029.8s
innovations happening on
1032.0s
both layers, the foundation model layer,
1034.3s
as well
1034.8s
as on the kind of orchestration
1036.8s
or kind of application layer.
1039.4s
The way that you can kind of think of our technical approach on this is that, you know, traditionally labs like DeepMind, and kind of some of these, these orgs that are really focused on solving problems, that you can model like a game, where you have rules and an environment and feedback loops, you can build out systems
1042.1s
of our technical approach on this
1044.7s
is that,
1045.6s
you know, traditionally labs
1047.5s
like DeepMind,
1048.7s
and kind of some of these,
1050.1s
these orgs that are really focused
1051.7s
on solving problems,
1053.3s
that you can model like a game,
1056.0s
where you have rules and an environment
1059.0s
and feedback loops,
1060.6s
you can build out systems
1062.5s
which model the reasoning of humans and even outperform them, they did this with the Alpha series of models, protein folding, Go, code. And for us, most of the reasoning work we do is similarly focused on kind of inference time reasoning, search through kind of decisions, and, and in what we kind of think of as you know,
1064.7s
and even outperform them,
1066.0s
they did this
1066.5s
with the Alpha series of models,
1068.3s
protein folding, Go, code.
1070.5s
And for us, most of the reasoning work
1072.8s
we do
1073.5s
is similarly focused
1074.9s
on kind of inference time
1077.5s
reasoning, search
1078.7s
through kind of decisions, and,
1081.1s
and in what we kind of
1081.8s
think of as you know,
1083.2s
maybe its something of intuition. Maybe its something of planning, but we arent training foundation models yet. And I think that a lot of the innovation thats going to happen at the foundation model, there will be things like latency and context window and kind of performance on some subset of tasks. But anytime that you need action
1085.4s
Maybe its something of planning, but
1089.2s
we arent training foundation models yet.
1091.3s
And I think that a lot of the innovation
1094.6s
thats going to happen
1095.1s
at the foundation model,
1096.4s
there will be things like latency
1099.1s
and context window
1100.8s
and kind of performance
1102.5s
on some subset of tasks.
1104.3s
But anytime that you need action
1107.7s
and environmental feedback, and kind of long-term planning, its going to be really difficult to build a single foundation model that does that. And I think its really the application layer where those types of innovations are going to happen. Yeah, I thought the Princeton SWE-agent paper that came out last week or so. It was really interesting
1109.3s
and kind of long-term planning, its
1113.5s
going to be really difficult
1114.6s
to build a single foundation model
1117.0s
that does that.
1118.0s
And I think its really
1119.5s
the application layer
1121.2s
where those types of innovations
1122.7s
are going to happen.
1123.6s
Yeah,
1124.2s
I thought the Princeton
1125.3s
SWE-agent paper that came out
1128.0s
last week or so.
1129.0s
It was really interesting
1130.0s
as an example, that if like, you can get incredible agentic reasoning performance on code tasks from small open source models. I thought that was really nice, a proof point of what youre saying. We love the whole team that put that together. And the SWE-bench work, I think, is a popular benchmark in the space.
1130.5s
that if like,
1131.2s
you can get incredible agentic reasoning
1134.3s
performance on code tasks
1136.7s
from small open source models.
1138.5s
I thought that was really nice,
1139.9s
a proof point of what youre saying.
1141.8s
We love the whole team
1143.2s
that put that together.
1144.1s
And the SWE-bench work,
1145.5s
I think,
1146.5s
is a popular benchmark in the space.
1149.0s
I think its, you know, its clear that a lot of the efforts towards building these systems relies on not just like any one, benchmark, or eval or set of tasks, but rather collaboration across a bunch of different areas, whether its the model layer, whether its the tasks themselves, its how, what data are you using to evaluate?
1152.9s
a lot of the efforts towards
1154.7s
building these systems relies on
1157.1s
not just like any one, benchmark, or eval
1160.4s
or set of tasks,
1161.5s
but rather
1162.2s
collaboration across
1164.0s
a bunch of different areas,
1165.1s
whether its the model layer,
1166.6s
whether its
1167.2s
the tasks themselves, its how,
1168.8s
what data are you using to evaluate?
1171.0s
And ultimately, like the overall architecture, and yeah, theyre, theyre a really great team. Were super pumped to see their work. Okay, last question on this? And then and then well, I will, I will pause myself. Any favorite cognitive architectures? Like is it the tree of thought stuff, chain of thought stuff like any favorite cognitive architectures
1171.6s
like the overall architecture,
1173.4s
and yeah, theyre, theyre
1174.7s
a really great team.
1175.7s
Were super pumped to see their work.
1177.7s
Okay, last question on this?
1178.8s
And then and then well, I will,
1182.3s
I will pause myself.
1183.4s
Any favorite cognitive architectures?
1186.2s
Like is it
1186.6s
the tree of thought
1188.4s
stuff, chain of thought stuff
1189.5s
like any favorite cognitive architectures
1191.9s
that you think are especially promising or fruitful in your field? Yeah, I think thats, thats a great question. I mean, I think kind of what I alluded to Previously, when you have almost like the game-like problem space where there are kind of simulatable, analyzable and optimizable boundaries, then that means that you can search through those decisions.
1193.9s
or fruitful in your field?
1195.5s
Yeah, I think thats, thats
1196.9s
a great question.
1197.6s
I mean,
1198.2s
I think kind of what I alluded
1199.7s
to Previously,
1200.6s
when you have
1202.2s
almost like the game-like problem space
1205.2s
where there are kind of simulatable,
1209.2s
analyzable and optimizable boundaries,
1213.0s
then that means that you can search
1215.2s
through those decisions.
1217.1s
And theres a bunch of techniques like Monte Carlo tree search, language agent tree search, that people have talked about in research papers that I think are interesting approaches here. I think that, in my mind, there isnt a singular cognitive architecture that makes sense for all tasks. And a lot of the benefit of breaking down the
1218.5s
like Monte Carlo tree search, language
1220.9s
agent tree
1221.3s
search,
1222.6s
that people have talked about
1223.8s
in research papers
1225.2s
that I think
1225.7s
are interesting approaches here.
1227.8s
I think that, in my mind, there isnt
1231.6s
a singular cognitive architecture
1234.0s
that makes sense for all tasks.
1236.7s
And a lot of the benefit
1238.4s
of breaking down the
1240.5s
software development lifecycle into kind of semantically meaningful segments is that developers, when they have these workflows that move from one step to the next, theyve kind of defined the boundaries of the game, so to speak. And so a lot of the work we do is figuring out which cognitive architecture or what design
1242.9s
kind of
1243.5s
semantically meaningful segments
1245.8s
is that developers,
1247.3s
when they have these workflows
1248.8s
that move from one
1249.8s
step to the next,
1251.1s
theyve
1251.4s
kind of defined
1252.2s
the boundaries of the game, so to speak.
1254.4s
And so a lot of the work
1255.9s
we do is figuring out
1256.9s
which cognitive architecture
1258.0s
or what design
1259.4s
makes sense for a given task. Youre reminding me of the Rich Sutton, Bitter Lesson, search and learning are the two techniques that scale. Yeah, absolutely. And I think you definitely need both. And then, Eno, youre talking about this a bit, how the sort of the reasoning layer on top of the foundation model is really the focus
1262.2s
Youre reminding me of the Rich Sutton,
1264.8s
Bitter Lesson, search and learning
1267.4s
are the two techniques that scale.
1269.2s
Yeah, absolutely.
1270.4s
And I think you definitely need both.
1273.2s
And then, Eno, youre
1274.5s
talking about this a bit,
1275.5s
how the sort of the reasoning layer
1278.1s
on top of the foundation model
1280.0s
is really the focus
1281.8s
for a lot of the fundamental research and a lot of the fundamental work that you guys are doing. Matan you had a line a couple of months ago, when we were talking. That was and and hopefully this doesnt come across as snarky, because its not meant to, but if something to the effect of their interest in engineers
1283.3s
and a lot of the fundamental work
1284.4s
that you guys are doing.
1285.5s
Matan you had a line
1287.3s
a couple of months ago,
1288.6s
when we were talking.
1289.9s
That was and and hopefully this doesnt
1292.5s
come across as snarky,
1293.5s
because its not meant to,
1294.8s
but if something to the effect of
1296.6s
their interest
1297.4s
in engineers
1298.2s
at OpenAI working on my margins for me, can you say a word about that? Because I thought that was, first it was incredibly well put. And then second, a pretty good insight in terms of how youre building the business and really benefiting from the work of the foundation models? Can you just say a couple words
1301.7s
can you say a word about that?
1302.8s
Because I thought that was, first it was
1304.5s
incredibly well put.
1305.9s
And then second, a pretty good insight
1307.9s
in terms
1308.3s
of how youre building the business
1309.5s
and really benefiting
1310.8s
from the work of the foundation models?
1312.2s
Can you just say a couple words
1313.1s
about that? Yeah, absolutely. So you know, there are a lot of companies, a lot of startups that are, you know, pursuing training foundational models or fine tuning models. And then there are a lot of huge research labs like OpenAI, and Anthropic, they're also putting a ton of resources behind making these foundational models better, cheaper, faster.
1315.0s
So you know,
1315.9s
there are a lot of companies,
1318.5s
a lot of startups
1320.2s
that are, you know, pursuing
1321.4s
training foundational models
1323.0s
or fine tuning models.
1324.0s
And then there are a lot of huge
1325.0s
research labs like OpenAI,
1327.1s
and Anthropic,
1328.5s
they're also putting a ton of resources
1330.4s
behind making these foundational models
1332.8s
better, cheaper, faster.
1336.0s
And from our perspective, right, like, we dont want to run a race thats, you know, not suited to our abilities, right, or we dont want to, we dont want to fight a battle that we know we wont win. Training foundational models, were not going to win that battle. And similarly, I also dont
1338.7s
we dont want to run a race
1340.8s
thats, you know,
1341.5s
not suited to our abilities, right, or
1344.3s
we dont want to,
1344.8s
we dont want to fight a battle
1345.8s
that we know we wont win.
1347.2s
Training foundational models, were
1348.7s
not going to win that battle.
1350.0s
And similarly,
1351.0s
I also dont
1351.5s
think its a particularly unique battle, at this point. I think these companies were incredibly unique and innovative, clearly based on what theyre delivering. But now, I think the stage is set in terms of training foundation models. And I think similarly, with a lot of the infrastructure for fine tuning and that sort of thing.
1353.7s
at this point.
1354.6s
I think these companies
1355.8s
were incredibly unique and innovative,
1357.3s
clearly based on what theyre delivering.
1359.0s
But now, I think the stage is set
1361.2s
in terms of training foundation models.
1363.5s
And I think similarly,
1364.3s
with a lot of the infrastructure
1366.2s
for fine tuning and that sort of thing.
1368.2s
What has not really come to fruition yet is actually making products with AI that people are using, theres so much talk about all these foundational models, all this infrastructure, and theres still very few real products that use this AI. You know, in the analogy that, you know, VCs like to talk about a lot,
1371.5s
yet is actually making products with
1374.6s
AI that people are using,
1376.5s
theres so much talk
1377.2s
about all these foundational models,
1378.6s
all this infrastructure,
1379.7s
and theres still very few real products
1382.8s
that use this AI.
1384.2s
You know, in the analogy that, you know,
1386.6s
VCs like to talk about a lot,
1388.0s
we have a ton of picks and shovels, and no ones actually going for gold. And so the thesis behind how were building this company is, lets first you know, use these beautiful black boxes that OpenAI, Anthropic and Google are spending billions of dollars and you know, hundreds of engineers to make, lets use these black boxes,
1390.2s
and no ones actually going for gold.
1392.1s
And so the thesis behind
1393.8s
how were building this company
1395.0s
is, lets first
1396.5s
you know, use these beautiful black boxes
1399.0s
that OpenAI, Anthropic and Google
1401.2s
are spending billions of dollars
1403.0s
and you know,
1403.5s
hundreds of engineers to make, lets
1405.5s
use these black boxes,
1406.8s
and build a product that people are actually using. And once we do that, then we can earn the right to do the fancier things like fine tuning and training. If youre unable to build a product that people are actually using, with these incredible models, then chances are fine tuning and training will not save you. And its
1407.8s
that people are actually using.
1409.7s
And once we do that,
1410.8s
then we can earn the right to do
1412.7s
the fancier things like fine
1414.0s
tuning and training.
1415.3s
If youre unable to build a product
1417.6s
that people are actually using,
1419.1s
with these incredible models,
1421.0s
then chances are fine
1422.5s
tuning and training will not save you.
1424.3s
And its
1424.9s
probably just not a good product. And so thats kind of the approach that were taking there. And so, you know, we do you know, we do get a lot of improvements when new models come out. But yet, we are very much grateful for the work thats being done at at these cutting edge research labs. I want to,
1426.2s
And so thats
1427.1s
kind of the approach
1428.2s
that were taking there.
1429.0s
And so,
1429.7s
you know,
1430.4s
we do
1430.8s
you know, we do get a lot of improvements
1432.3s
when new models come out.
1433.8s
But yet,
1434.5s
we are very much grateful
1435.5s
for the work thats being done
1436.5s
at at these cutting edge research labs.
1440.7s
I want to,
1441.8s
youve said a lot about how what youre doing is kind of like making AI immediately practical for engineers and like an enterprise setting. And so I want to throw another I think its a Matan quote, and Im not sure if you were quoting somebody else, but you said you, youre telling us last time and you said If
1442.6s
about how what youre doing
1444.1s
is kind of like making
1445.7s
AI immediately practical for engineers
1447.7s
and like an enterprise setting.
1449.3s
And so I want to throw another
1450.9s
I think its a Matan quote, and Im
1452.2s
not sure if you were quoting
1453.0s
somebody else,
1453.6s
but you said you,
1455.1s
youre telling us last time
1456.2s
and you said If
1456.7s
Jeff Dean shows up in your office, and he doesnt understand your code base, he wont be productive. And unpack that for us. Like what does it take to kind of make a coding agent not just good for anybody that boots up a computer but somebody that's a full time engineer, theyre a real software company? Yeah, totally. Yeah.
1458.2s
and he doesnt understand your code base,
1460.0s
he wont be productive.
1462.0s
And unpack that for us.
1464.5s
Like what does it take
1465.7s
to kind of make a coding agent
1468.2s
not just good for anybody
1469.4s
that boots up a computer
1470.3s
but somebody
1471.2s
that's a full time engineer, theyre
1472.5s
a real software company?
1474.2s
Yeah, totally. Yeah.
1475.5s
So the analogy here is that, you know, Jeff Dean is the analog of a really, really good foundational model, lets say like GPT-6, with incredible reasoning, right? But if it comes into your engineering organization, with all your nuances, and all your engineering best practices, just having that good inference, and good reasoning is not enough to actually contribute
1478.0s
Jeff Dean is the analog of a really,
1480.3s
really good
1481.3s
foundational model,
1482.3s
lets say like GPT-6,
1483.9s
with incredible reasoning, right?
1485.6s
But if it comes into
1486.5s
your engineering organization,
1488.2s
with all your nuances,
1489.5s
and all your engineering
1490.4s
best practices,
1491.6s
just having that good inference,
1494.0s
and good reasoning is not enough
1495.6s
to actually contribute
1497.2s
and automate these tasks reliably. Some given isolated task, and sure you can solve, like, give it some, like LeetCode problem and this, give Jeff Dean a LeetCode problem, Im sure he will solve it. But if you have some, you know, 20 year old legacy code base, some part of it is dead code. The other part of it,
1499.3s
Some given isolated task,
1500.8s
and sure you can solve, like,
1502.3s
give it some,
1503.5s
like LeetCode problem
1504.6s
and this,
1505.4s
give Jeff Dean a LeetCode problem,
1507.0s
Im sure he will solve it.
1508.5s
But if you have some,
1509.8s
you know,
1510.0s
20 year old legacy code base,
1512.3s
some part of it is dead code.
1514.0s
The other part of it,
1514.9s
the last person who was contributing to it just retired. And so no one else knows whats going on there. You need a deep understanding of the engineering system, not just the code base, but like why you made certain decisions, how things are being communicated, what top of mind priorities are for the engineering organization. And its
1516.5s
to it just retired.
1517.7s
And so no one else knows
1518.8s
whats going on there.
1519.7s
You need a deep understanding
1521.5s
of the engineering system,
1522.9s
not just the code base,
1524.3s
but like why you made certain decisions,
1526.6s
how things are being communicated,
1528.3s
what top of mind priorities
1530.2s
are for the engineering organization.
1532.2s
And its
1532.8s
kind of these, less sexy, but incredibly important details that were really focused on in order to deliver this to the enterprise. What about, a lot of these AI coding companies are kind of focused on the individual developers productivity? How do you think about the individual level optimization versus maybe the system, the system wide optimization?
1535.0s
but incredibly important details
1537.0s
that were really focused on in order
1538.7s
to deliver this to the enterprise.
1540.4s
What about, a lot of
1543.7s
these AI
1545.0s
coding companies are
1546.0s
kind of focused
1546.4s
on the individual
1547.4s
developers productivity?
1549.4s
How do you think about
1550.0s
the individual level optimization versus
1552.7s
maybe the system,
1554.4s
the system wide optimization?
1556.8s
I think the the important thing to think about with respect to the the whole org is, when a VP of engineering comes into the room, that its theyre not really focused on whether or not an individual completed, like one task an hour faster, theyre concerned about how many tasks are being completed, and aggregate metrics of speed.
1559.8s
to think about with respect
1561.0s
to the the whole org
1562.5s
is, when a VP of engineering
1565.1s
comes into the room,
1566.4s
that its theyre not really focused on
1569.0s
whether or not an individual completed,
1572.1s
like one task an hour faster,
1575.1s
theyre concerned about
1576.0s
how many tasks are being completed,
1578.0s
and aggregate metrics of speed.
1580.2s
But if that person completed that task an hour faster, but its 40, worse code, right? Its churning code, where people are gonna rewrite on top of it, or that person took that task, and, you know, they did it an hour, but it took them four hours to plan that, and they were blocking five other engineers.
1581.8s
that task an hour faster,
1583.5s
but its 40, worse code, right?
1586.5s
Its churning
1587.2s
code, where people are gonna rewrite
1588.9s
on top of it,
1589.9s
or that person took that task, and,
1593.5s
you know,
1594.0s
they did it an hour,
1594.8s
but it took them four hours to plan that,
1596.9s
and they were blocking
1598.0s
five other engineers.
1599.2s
And so when you start to actually add the nuance of what does it mean to be successful? Measuring an engineering org, you start to bump into a lot of challenges with understanding what needs to be improved, and what is a bottleneck? And what is just kind of a secondary metric. I think a lot of the initial attempts
1600.8s
the nuance of what does it mean
1602.1s
to be successful?
1603.7s
Measuring an engineering org,
1605.8s
you start to bump into
1606.8s
a lot of challenges with understanding
1610.0s
what needs to be improved,
1611.9s
and what is a bottleneck?
1613.4s
And what is just kind of a
1615.2s
secondary metric.
1616.2s
I think a lot of the
1617.7s
initial attempts
1619.5s
at making AI coding tools are really focused on first order effects, how quickly is somebody tabbing to autocomplete a statement? Or, you know, how quickly is somebody completing an individual task? But I think that at Factory, a lot of what were trying to do is understand from an engineering leaders perspective, how are you measuring performance?
1620.0s
AI coding tools are really focused
1622.1s
on first order effects,
1623.6s
how quickly is somebody
1625.5s
tabbing to autocomplete a statement?
1627.5s
Or, you know, how quickly
1629.0s
is somebody completing
1630.4s
an individual task?
1632.1s
But I think that at Factory,
1633.8s
a lot of what were trying to do
1635.0s
is understand
1636.2s
from an engineering leaders perspective,
1638.8s
how are you measuring performance?
1641.2s
And what are the metrics that you look at to understand, hey, were doing really well as an org, or, hey, we need to make improvements and targeting those and I think metrics like code churn, end to end open to merge time, you know, time to first answer within the eng org, all these things are much more impactful
1642.1s
that you look at to understand, hey,
1643.9s
were doing really
1644.7s
well as an org, or,
1645.8s
hey, we need to make improvements
1647.3s
and targeting those
1648.5s
and I think metrics like code churn,
1650.8s
end to end open to merge time,
1653.3s
you know, time to first answer
1655.5s
within the eng org,
1656.5s
all these things
1657.5s
are much more impactful
1659.1s
to to an organizations speed of shipping code. And so thats kind of how we think about it. I think this really ties into what Eno was just saying quite well, which is, you know, the clearly we were talking about products earlier as well, like, clearly the AI product that has penetrated the enterprise the most is Copilot, right?
1661.7s
speed of shipping code.
1663.4s
And so thats
1664.1s
kind of how we think about it.
1665.6s
I think this really ties
1666.6s
into what Eno was just saying
1668.2s
quite well, which is,
1669.9s
you know,
1670.6s
the clearly
1671.8s
we were talking about products
1672.6s
earlier as well, like,
1673.3s
clearly the AI product
1674.5s
that has penetrated the enterprise
1675.9s
the most is Copilot, right?
1678.6s
Unfortunately, with a tool like Copilot the things that are kind of the metrics that are really held up as success are things like autocomplete acceptance rate. And the problem is exactly to your point, if youre a CTO or VP of engineering, how do you then go to the executive team and say, hey, look, our autocomplete
1680.2s
with a tool
1680.6s
like Copilot
1681.1s
the things that are kind of the metrics
1683.5s
that are really held up
1684.5s
as success
1686.0s
are things like autocomplete
1687.9s
acceptance rate.
1689.2s
And the problem is exactly to your point,
1691.0s
if youre a CTO or VP of engineering,
1693.2s
how do you then go to the executive team
1695.2s
and say, hey, look, our autocomplete
1696.5s
acceptance rate is this high. They dont know what that means. They dont understand how that translates into, like business objectives, right. And also, you know, Eno was alluding to this, theres kind of a hidden danger to some of these autocomplete tools, which is, orgs that use tools like this end up increasing their code
1698.3s
They dont know what that means.
1699.3s
They dont understand
1700.0s
how that translates into,
1701.2s
like business objectives, right.
1703.1s
And also,
1703.8s
you know,
1704.0s
Eno was alluding to this, theres
1705.5s
kind of
1706.5s
a hidden danger
1708.6s
to some of these autocomplete tools,
1710.0s
which is,
1711.4s
orgs that use tools like this
1713.0s
end up
1713.3s
increasing their code
1714.5s
churn by anywhere from 20 to 40. Theres some studies that look into this. Theres some problems with these studies. But you know, directionally whats clear is that, as the percentage of AI generated code increases, code churn, if youre not, you know, doing anything different in your review process, code churn is going to go up.
1717.8s
Theres some studies that look into this.
1719.2s
Theres some problems with these studies.
1720.5s
But you know, directionally
1721.9s
whats clear
1722.7s
is that,
1723.5s
as the percentage of
1724.8s
AI generated code increases, code churn,
1728.4s
if youre not, you know,
1730.1s
doing anything different
1731.0s
in your review process,
1732.3s
code churn is going to go up.
1741.9s
If we look at things like how fast are you completing your cycles, what is your, you know, code churn across the org, or across these different repos that divides out these kind of like, a smaller like intermediate metrics, and gives you a sense of, hey, we are shipping faster, and were churning less code.
1742.2s
look at things like
1743.1s
how fast are you completing your cycles,
1745.4s
what is your,
1746.3s
you know, code
1747.5s
churn across the org, or across
1749.1s
these different repos that divides out
1751.5s
these
1752.5s
kind of like,
1753.3s
a smaller like intermediate metrics,
1756.4s
and gives you a sense of,
1757.3s
hey, we are shipping
1758.2s
faster, and were churning less code.
1762.1s
So thats really how we talk about this with these engineering leaders. At the end of the day, the three main axes we look at are saving engineering time, increasing speed, and improving code quality. So these are three and again, theres kind of different complexity of metrics for different parts of the org.
1765.1s
with these engineering leaders.
1766.4s
At the end of the day,
1767.1s
the three main axes we look at
1770.2s
are saving engineering
1771.6s
time, increasing
1773.5s
speed, and improving code quality.
1777.2s
So these are three and again,
1779.1s
theres kind of different complexity
1782.0s
of metrics
1782.4s
for different parts of the org.
1783.9s
These are the three that we discussed with engineering leaders, but we want to arm them with information when theyre talking to lets say, their CFO. And so really, we kind of break that down into one main metric, which is engineering velocity. And thats really what all of these droids are targeted towards is increasing engineering velocity, Let me
1785.2s
with engineering leaders,
1786.5s
but we want to arm them with information
1788.2s
when theyre talking to lets
1789.1s
say, their CFO.
1790.3s
And so really, we kind of
1792.4s
break that down into one main metric,
1795.6s
which is engineering velocity.
1797.4s
And thats really what
1798.0s
all of these droids are targeted
1799.5s
towards is increasing
1801.1s
engineering velocity,
1804.0s
Let me
1805.1s
try to recap a couple parts of the story thus far. So in some ways, this is a compound lever, meaning AI is a lever on software software is a lever on the world. And so building an autonomous software development system is one of the most impactful things you can possibly do with your lives, which is pretty cool.
1806.8s
parts of the story thus far.
1808.8s
So in some ways,
1811.2s
this is a compound lever,
1812.6s
meaning AI is a lever on software
1814.9s
software is a lever on the world.
1816.5s
And so building
1817.8s
an autonomous software
1819.2s
development system
1820.2s
is one of the most impactful things
1821.5s
you can possibly do with your lives,
1822.8s
which is pretty cool.
1824.8s
There are a few unique angles to the approach that you guys have taken and maybe not unique, but distinctive, you know, one of which is the decision to write on top of the foundation models, which means that you get to benefit from all their ongoing innovation. It also frees you up to really focus on the reasoning
1826.4s
to the approach that you guys have taken
1827.8s
and maybe not unique,
1828.5s
but distinctive,
1829.7s
you know, one of which is the decision
1831.2s
to write on top of the foundation models,
1832.8s
which means that you get to benefit
1834.0s
from all their ongoing innovation.
1835.9s
It also frees
1836.8s
you up to really focus on the reasoning
1839.0s
and the agentic behavior on top of those foundation models, which is part of the reason why you can deploy your product as a series of droids which are basically job specific, autonomous agents that do something like test or review end to end in a way that is practically useful to an engineering organization. And instead of focusing
1840.2s
on top of those foundation models,
1842.5s
which is part of the reason
1844.0s
why you can deploy
1846.0s
your product
1846.8s
as a series of droids
1848.0s
which are basically job
1849.2s
specific, autonomous agents
1851.2s
that do something like test
1852.7s
or review end to end in a way
1854.7s
that is practically useful
1856.3s
to an engineering organization.
1859.1s
And instead of focusing
1860.1s
on just producing more code, youre actually focused on kind of the system wide output, which requires you to have really detailed context around, not just the codebase, but all of the systems and processes and kind of nuanced around the entire environment. And having done so you can increase, you know, velocity for an organization.
1860.9s
more code, youre
1861.5s
actually focused on
1862.4s
kind of the system wide output,
1864.0s
which requires you
1865.4s
to have really detailed context around,
1868.8s
not just the codebase,
1870.0s
but all of the systems and processes
1872.3s
and kind of nuanced
1873.7s
around the entire environment.
1875.5s
And having done
1876.5s
so you can increase,
1877.5s
you know, velocity for an organization.
1880.1s
I think thats a bunch of the story that weve talked about so far. Let's talk a bit about the results. Are there any good sort of customer examples you can share of, you know, Factory in action and the results that youve been able to have for people? Yeah, so I think some of the
1882.1s
that weve talked about so far.
1883.6s
Let's talk a bit about the results.
1886.2s
Are there any good sort of
1888.0s
customer examples you can share of,
1889.8s
you know, Factory in action
1891.1s
and the results that youve been able
1893.2s
to have for people?
1895.8s
Yeah, so I think some of the
1897.0s
some of the main things that were seeing, like across the board, and were not super public on case studies just yet, but something that we see across the board is, you know, I think our average cycle time increase is around 22. On average, we are lowering code churn by 13. Tools, like, I guess, we
1897.7s
that were seeing,
1898.2s
like across the board, and were
1900.1s
not super public
1902.0s
on case studies just yet, but
1905.0s
something that we see
1905.8s
across the board is,
1907.7s
you know,
1908.0s
I think our average
1909.2s
cycle time increase is around 22.
1913.0s
On average, we are lowering code churn
1916.2s
by 13.
1919.0s
Tools, like, I guess, we
1920.2s
havent even gotten into the specific droids, but tools like the test droid, end up saving engineers, like around 40 minutes a day, which is pretty exciting. And yeah, I think, kind of going back to what we were talking about in terms of benchmarks, one of the most exciting things about having 1000s of developers
1922.2s
droids,
1923.1s
but tools like the test droid,
1925.5s
end up saving engineers,
1927.2s
like around 40 minutes a day,
1929.5s
which is pretty exciting.
1931.2s
And yeah,
1932.3s
I think,
1933.0s
kind of
1933.4s
going back to what we were talking about
1935.0s
in terms of benchmarks,
1936.7s
one of the most exciting things
1938.0s
about having 1000s of developers
1940.4s
who are actually using these tools, is that we get this live set of benchmarks, and we get evals and feedback from these developers about how these droids are performing. And so, you know, like I mentioned, were huge fans of SWE-bench and what thats done, kind of for the general community and giving people like an open source benchmark
1942.3s
is that
1943.1s
we get this live set of benchmarks,
1946.7s
and we get evals and feedback
1948.4s
from these developers
1949.8s
about how these droids are performing.
1951.6s
And so, you know,
1953.1s
like I mentioned, were
1954.2s
huge fans of SWE-bench
1955.9s
and what thats done,
1957.0s
kind of for the general community
1958.7s
and giving people
1960.7s
like an open source benchmark
1962.1s
to really compare these models. But you know, strategically for us, having this deployed in the real world has allowed us to dramatically increase our iteration speed in terms of quality for these droids. What, what have you guys learned a lot, since you have a bunch of people using this in the real world,
1963.8s
But you know, strategically for us,
1966.0s
having this deployed in the real world
1968.2s
has allowed us to dramatically
1970.3s
increase our iteration speed
1972.0s
in terms of quality for these droids.
1973.8s
What, what have you guys learned a lot,
1976.1s
since you have a bunch of people
1977.3s
using this in the real world,
1978.3s
what have you learned along the way? And have there been any, any big surprises? Engineers love ownership. Yeah, alright, say more. Absolutely. I mean, I think it really is that, you know, when youre building an autonomous product, and the goal is to take, you know, take over a task, you have to deal with developers who are fickle,
1979.5s
And have there been any, any
1981.1s
big surprises?
1983.1s
Engineers love ownership.
1986.4s
Yeah, alright, say more.
1988.5s
Absolutely.
1989.2s
I mean, I think it really is
1991.2s
that, you know, when youre
1992.7s
building an autonomous product,
1994.7s
and the goal is to take,
1997.3s
you know, take over a task,
1999.8s
you have to deal with developers
2001.9s
who are fickle,
2002.8s
for good reason, that theyre constantly bombarded with developer tools, and, and automations. And anything thats kind of being enforced from a top down perspective needs to be very flexible. And so making sure that, you know, when were building these products, we think about what are the different preferences or ideas that people have
2004.3s
constantly bombarded
2005.3s
with developer
2006.0s
tools, and, and automations.
2007.0s
And anything thats
2008.6s
kind of being enforced from a top down
2011.8s
perspective needs to be very flexible.
2015.5s
And so making sure
2016.4s
that, you know,
2016.8s
when were building these products,
2018.4s
we think about
2019.4s
what are the different preferences
2021.3s
or ideas
2022.9s
that people have
2023.7s
about how this task should be done, and then building as much flexibility into that. I think a great example of this is the review process. Everybody has a different idea of what they want code review to look like. Some people want superhuman linters. Some people want, you know, really deep kind of analysis of the code change.
2025.7s
and then building
2026.3s
as much flexibility into that.
2028.2s
I think a great example of this is
2031.3s
the review process.
2033.3s
Everybody has a different idea
2034.8s
of what they want
2035.6s
code review to look like.
2037.5s
Some people want superhuman linters.
2040.2s
Some people want,
2041.5s
you know, really deep
2043.0s
kind of analysis of the code change.
2046.4s
Some people dont even like code review. They get annoyed by it entirely. Matan has a great quote about what code reviews like, I dont know, if you want to share that. Yeah, yeah. So I mean, in general, we've kind of internally realized that code, the code review process is very much like going to the DMV,
2048.8s
They get annoyed by it entirely.
2051.1s
Matan has a great quote about
2054.2s
what code reviews like, I dont know,
2056.0s
if you want to share that. Yeah, yeah.
2057.5s
So I mean, in general,
2058.5s
we've kind of internally realized
2060.3s
that code, the code review process
2064.1s
is very much like going to the DMV,
2066.8s
in that no matter how clean the DMV is, no matter how fast the line is. No one loves code review, right? Because at the end of the day, someone is criticizing you. Someones going in and looking out, you know what you did, and saying better ways you could have done it. So in general, the review process, it's
2069.0s
no matter how fast the line is.
2070.7s
No one loves code review, right?
2072.9s
Because at the end of the day,
2073.5s
someone is criticizing you.
2074.9s
Someones going in and looking out,
2076.2s
you know what you did,
2077.0s
and saying better ways
2078.1s
you could have done it.
2079.1s
So in general,
2080.2s
the review process, it's
2081.5s
the type of thing that as an engineering leader, its great to see like moving the needle on these organization wide metrics. As a developer, its maybe not the most fun thing. Whereas something like the test droid, right, which is generating tests for you so you dont spend hours writing your unit tests. Thats incredible as a developer,
2082.1s
that as an engineering leader, its
2084.2s
great to see like moving the needle
2085.9s
on these organization wide metrics.
2088.0s
As a developer, its
2089.5s
maybe not the most fun thing.
2090.8s
Whereas something like
2092.0s
the test droid, right,
2093.2s
which is generating tests for you
2094.6s
so you dont spend hours
2096.1s
writing your unit tests.
2097.5s
Thats incredible as a developer,
2099.3s
but you know, for the engineering leader, its slightly less obvious how that connects directly to business metrics. So I think this is part of why its important for us to have this fleet of droids. Because were not just building this for the engineering leader, nor are we just building this for the developer, but rather
2099.9s
for the engineering leader, its
2101.9s
slightly less obvious
2103.8s
how that connects
2104.7s
directly to business metrics.
2106.5s
So I think this is part of why
2108.5s
its important for us
2109.4s
to have this fleet of droids.
2111.4s
Because were not just building this
2113.5s
for the engineering leader,
2114.7s
nor are we just building this
2116.0s
for the developer,
2116.7s
but rather
2117.1s
for the engineering organization as a whole. Part of what I heard there was that I dont have to go to the DMV anymore. You can just send me my drivers license in the mail. Yeah, basically. Yeah. Well, that's a good way to sum it up. Have you guys seen Pat drive?
2118.4s
as a whole.
2119.6s
Part of what I heard there was that
2121.1s
I dont have to go to the DMV anymore.
2122.7s
You can just send me
2123.5s
my drivers license in the mail.
2125.4s
Yeah, basically.
2126.3s
Yeah.
2126.5s
Well, that's a good way to sum it up.
2128.5s
Have you guys seen Pat drive?
2129.7s
I dont think they should be sending him a drivers license. Theres Waymo for that. Speaking of Waymo, how far out do you think we are from having fully autonomous software engineers? Like if you thought about Waymo, like it felt like it was gonna come really fast. And then it felt like we went through a valley of despair.
2130.6s
a drivers license.
2131.8s
Theres Waymo for that.
2136.9s
Speaking of Waymo,
2137.9s
how far
2140.0s
out do you think we are
2141.2s
from having fully autonomous
2143.1s
software engineers?
2144.0s
Like if you thought about Waymo,
2145.1s
like it felt like
2145.9s
it was gonna come really fast.
2147.2s
And then it felt like
2147.8s
we went through a valley of despair.
2149.8s
And now the future is coming out super fast again, like, which inning are we in for the kind of fully autonomous software engineer cycle? And when do you think well have fully autonomous Jeff Deans? This is a great question. And I think one that we get, we get a lot, I think one thing thats worth
2151.2s
super fast again, like,
2153.3s
which inning
2153.8s
are we in
2154.6s
for the kind of fully autonomous software
2156.8s
engineer cycle?
2158.8s
And when do you think well
2160.8s
have fully autonomous Jeff Deans?
2163.8s
This is a great question.
2164.7s
And I think one that we get,
2166.0s
we get a lot,
2166.6s
I think one thing thats worth
2168.2s
is kind of like reframing what a fully autonomous software engineer will do. There have been many moments where technical technical progress has led to kind of, you know, labor dynamic changes, and increases in the level of abstraction in which people work. And I think that historically, enabling people to operate or impact the construction of software with,
2171.2s
a fully autonomous software
2173.1s
engineer will do.
2174.6s
There have been many moments where
2176.6s
technical technical progress
2178.1s
has led to kind of, you know,
2181.5s
labor dynamic changes, and increases
2184.5s
in the level of abstraction
2185.8s
in which people work.
2187.8s
And I think that historically,
2189.9s
enabling people
2190.8s
to operate
2191.5s
or impact
2192.7s
the construction of software with,
2195.1s
you know, at a higher level of abstraction with less domain knowledge has generally led to huge increases in demand for software. And I think that what were seeing with the the customers were working with today, is that when you free people up from these types of kind of secondary tasks like generating unit tests
2195.6s
at a higher level of abstraction
2197.0s
with less domain knowledge
2198.5s
has generally led to huge increases
2201.4s
in demand for software.
2203.1s
And I think that what were seeing with
2207.4s
the the customers were
2208.2s
working with today,
2209.2s
is that when you free people up
2211.4s
from these types of kind
2212.7s
of secondary tasks
2214.3s
like generating unit tests
2216.2s
that map to a pull request, or, or writing and maintaining documentation on a code base that 95 of people know, but that documentation comes into play for that 5, that doesnt, they start to shift their attention to higher level thinking, they think about orchestration, they think about architecture, they think about, you know,
2218.2s
or writing and maintaining
2219.8s
documentation on a code base
2221.5s
that 95 of people know,
2223.8s
but that documentation comes into play
2225.8s
for that 5, that doesnt,
2227.4s
they start to shift their attention
2229.8s
to higher level thinking,
2231.7s
they think about orchestration,
2233.6s
they think about architecture,
2236.1s
they think about, you know,
2237.1s
what is this PR actually trying to do, and less about, did they follow the style guide. did they follow the style guide. I think that what were seeing is that this is happening today, already because of AI tools. And over time, as they get better and better, well see that shift towards software engineers becoming a little bit
2239.2s
and less about,
2240.4s
did they follow the style guide.
2242.2s
did they follow the style guide.
2243.1s
I think that what were seeing is that
2244.7s
this is happening
2245.4s
today, already because of AI tools.
2248.0s
And over time,
2249.1s
as they get better and better,
2251.0s
well see that shift towards
2254.2s
software engineers becoming a
2256.3s
little bit
2256.6s
more like architects or designers of software. And so in the future, I think theres gonna be 10 times more people involved in software creation, where every individual has the impact of maybe 100 or 1000 people, it just may not look exactly like the individual steps of the development lifecycle that we see today.
2258.3s
or designers of software.
2260.2s
And so in the future,
2262.1s
I think theres gonna be 10 times
2263.2s
more people involved
2264.2s
in software creation,
2265.4s
where every individual has the impact
2267.5s
of maybe 100
2268.2s
or 1000 people,
2269.6s
it just may not look exactly like
2272.4s
the individual steps
2274.0s
of the development
2274.8s
lifecycle that we see today.
2277.0s
You know, that reminded me of a quote that you guys have on your website, which that, and Im gonna, Im gonna read this, it says, we hope to be a beacon of the coming age of creativity and freedom that on-demand intelligence will unlock. And that really resonated with me when I read it, because it sort of implies
2278.4s
that you guys have on your website,
2279.8s
which that, and Im gonna,
2281.8s
Im gonna read this,
2283.1s
it says, we
2284.5s
hope to be a beacon of the coming age
2286.3s
of creativity
2287.2s
and freedom
2287.9s
that on-demand intelligence will unlock.
2291.0s
And that really resonated with me
2293.0s
when I read it,
2293.9s
because it sort of implies
2295.3s
a very positive and optimistic view of the world that were heading into. I wonder if you guys want to say a couple more words on that or, or sort of what you think the sort of relationship between man and machine will be in the fullness of time. This kind of goes back to our original approach, which is,
2296.5s
and optimistic
2297.9s
view of the world
2298.9s
that were heading into.
2300.2s
I wonder if you guys want to say
2301.5s
a couple more words on that or,
2304.0s
or sort of what you think
2305.2s
the sort of relationship
2306.7s
between man and machine
2307.8s
will be in the fullness of time.
2309.7s
This kind of goes back to our
2311.2s
original approach,
2312.3s
which is,
2313.4s
you know, its very tempting to go after the sexiest parts of software development in particular, you know, like, building an app from scratch, right. But thats also the sort of thing that will make a developer defensive, because thats the part that they enjoy, right? And so in a world where you automate the development,
2314.0s
its very tempting to go
2315.5s
after the sexiest parts
2317.1s
of software
2317.6s
development in particular,
2318.7s
you know, like,
2319.5s
building an app from scratch, right.
2321.6s
But thats also the sort of thing
2323.5s
that will make a developer defensive,
2325.2s
because thats the part
2326.6s
that they enjoy, right?
2328.0s
And so in a world
2328.8s
where you automate the development,
2331.0s
then an engineer is just left reviewing, testing and documenting, which is like a depressing hellscape, if you were to ask any, any software engineer, right? So for us, its very important that we position ourselves aligned with the developer instead of you know, going into these organizations and being antagonistic with them, right? Like,
2333.0s
testing and documenting,
2334.5s
which is like a depressing hellscape,
2336.4s
if you were to ask any,
2337.8s
any software engineer, right?
2339.5s
So for us, its very important
2341.5s
that we position ourselves
2343.1s
aligned with the developer
2344.7s
instead of you know,
2345.7s
going into these organizations
2347.0s
and being antagonistic with them, right?
2348.8s
Like,
2349.2s
by going in and automating the things that we dont want to do, or rather, by going in there, and automating the things that developers dont want to do, we are positioning ourselves with them. Right? Five years from now, I dont think anyone really knows what software engineering will be, or even if itll be called that anymore,
2351.0s
that we dont want to do, or rather,
2353.6s
by going in there,
2354.5s
and automating the things that developers
2356.4s
dont want to do,
2357.5s
we are positioning ourselves with them.
2359.6s
Right?
2360.2s
Five years from now,
2361.4s
I dont think anyone really knows
2363.4s
what software engineering will be,
2366.0s
or even if itll be called that anymore,
2367.9s
you know, to this point, you might be, you know, youre like a software curator, or cultivator or orchestrator. But by positioning ourselves this way, with the developer, wherever that role goes, we will be there side by side, to allow them to have this higher leverage. And so yeah, completely agree, to your point,
2369.4s
you know, youre
2369.9s
like a software
2370.6s
curator, or cultivator or orchestrator.
2373.8s
But by positioning ourselves
2375.3s
this way, with the developer,
2376.9s
wherever that role goes, we will be there
2380.2s
side by side,
2381.5s
to allow them
2382.6s
to have this higher leverage.
2384.2s
And so yeah,
2385.0s
completely agree, to your point,
2386.8s
this is one of the most incredible things that is going to happen to you know, our ability as humans to create. And I think for us, its just incredibly important that we are aligned with the users of this product. and not, you know, antagonistic, trying to rule it to replace them. How far do you think we are
2389.5s
that is going to happen to
2390.9s
you know, our ability
2391.8s
as humans to create.
2393.9s
And I think for us, its
2395.0s
just incredibly important
2396.1s
that we are aligned
2398.0s
with the users of this product.
2400.0s
and not,
2400.8s
you know, antagonistic,
2401.8s
trying to rule it to replace them.
2404.2s
How far do you think we are
2405.1s
from having these reliable, kind of maybe call it intern level engineers? Is it a year out? Is it already here today? Is it a decade out? I think I mean, it depends on on on the the task first, for things like code review and testing, I think were here, were already there, where were able to operate
2407.8s
kind of
2408.5s
maybe call it intern level engineers?
2410.8s
Is it a year out?
2411.9s
Is it already here today?
2412.3s
Is it a decade out?
2415.5s
I think I mean, it depends
2417.1s
on on on the the task first, for things
2421.2s
like code
2421.6s
review and testing,
2422.7s
I think were here, were
2424.4s
already there,
2425.4s
where were able
2426.6s
to operate
2427.2s
at a level that, you know, for many, or like, theres feedback from one organization that we got, in particular, that where we brought them the review droid. And, this was pretty early on. And they said, you know, the review droid is the best reviewer on our team. And I think that every once in a while,
2428.5s
you know, for many,
2429.8s
or like, theres feedback
2431.0s
from one organization
2432.0s
that we got, in particular,
2433.4s
that where we brought them
2435.0s
the review droid.
2435.9s
And, this was pretty early on.
2437.9s
And they said,
2438.8s
you know,
2439.5s
the review droid
2440.2s
is the best reviewer on our team.
2442.6s
And I think that every once in a while,
2445.3s
you kind of hear something like that, and it gives you a lot of confidence that directionally were definitely moving towards something that is valuable. And for tasks like, you know, hey, weve got to decompose our mono repo into a ton of micro services. And you know, that type of thing that you might arm
2446.6s
and it gives you a lot of confidence
2448.5s
that directionally were
2449.6s
definitely moving towards
2450.7s
something that is valuable.
2452.4s
And for tasks like, you know, hey, weve
2456.5s
got to
2457.5s
decompose our mono repo
2459.5s
into a ton of micro services.
2461.2s
And you know, that type of thing
2462.9s
that you might arm
2463.6s
like a staff level engineer, and hat with an armed with a team of engineers under them, I think that we wont see like a binary moment of oh, well, now this is done by an AI, I think that their responsibilities will slowly start to get decomposed into the tasks of planning and implementing the refactor going,
2466.8s
with an armed
2467.6s
with a team of engineers under them,
2470.2s
I think that we wont see like
2472.8s
a binary moment of oh, well,
2474.4s
now this is done by an AI,
2476.0s
I think that their responsibilities
2477.8s
will slowly start
2478.6s
to get decomposed
2480.1s
into the tasks
2481.5s
of planning and implementing
2484.6s
the refactor going,
2486.2s
you know, one file at a time. And when they start handing off those sub tasks to AI, I think that role will will kind of start to be called something different, because when youre no longer as focused on what is the individual line of code that Im writing tomorrow, and more focused on what is our mission,
2487.7s
And when they start handing off
2489.6s
those sub tasks to AI,
2491.7s
I think
2492.5s
that role will will kind of start
2494.7s
to be called something different,
2496.0s
because when youre
2496.5s
no longer as focused on
2498.2s
what is the individual line of code
2500.1s
that Im writing tomorrow,
2501.4s
and more focused on
2502.7s
what is our mission,
2503.6s
or what is our goal, as an engineering team, you know, you really are more of an architect and less of an implementer. And concrete example of, you know, us eating, eating the, the food that were creating, right, we were dreading for months creating a GitLab integration, some of our customers use GitLab,
2504.3s
as an engineering team, you know,
2507.3s
you really are more of an architect
2508.8s
and less of an implementer.
2510.6s
And concrete example of, you know, us
2513.5s
eating, eating the,
2514.3s
the food that were creating, right,
2516.5s
we were dreading for months
2518.2s
creating a GitLab integration,
2520.0s
some of our customers use GitLab,
2521.9s
we want to build cool AI stuff, we didnt want to spend time building a GitLab integration, we had our code droid, fully spec out what the steps of building a GitLab integration would look like. And then it actually implemented every one of these sub tickets, we were of course, monitoring it just to make sure it wasnt
2523.4s
we didnt want to spend time
2524.5s
building a GitLab integration,
2526.2s
we had our code droid, fully spec out
2529.6s
what the steps of building
2530.9s
a GitLab integration would look like.
2532.8s
And then it actually implemented
2534.7s
every one of these sub tickets,
2536.3s
we were of course, monitoring it
2537.7s
just to make sure it wasnt
2538.6s
breaking anything. And we now have a GitLab integration. And so this is something that we genuinely were considering getting an intern to do. Because we just, we were just, we really didnt want to do a GitLab integration. Shout out to GitLab? Yeah. But like, materially, the droid saved us, like hours of time,
2539.7s
And we now have a GitLab integration.
2542.5s
And so
2543.2s
this is something
2543.9s
that we genuinely were considering
2545.6s
getting an intern to do.
2547.2s
Because we just,
2548.2s
we were just,
2549.0s
we really didnt
2549.5s
want to do a GitLab integration.
2551.0s
Shout out to GitLab?
2554.4s
Yeah.
2556.2s
But like,
2557.1s
materially, the droid saved us,
2559.2s
like hours of time,
2560.6s
none of us had built a Git lab integration before. And also, its just like, relatively complicated to like, abstract away the like source code manager. And so that was materially intern work that we did today. So to answer your question, it is now. Its just kind of slowly climbing up more and more
2561.7s
lab integration before.
2562.8s
And also, its just like,
2564.0s
relatively complicated
2565.3s
to like, abstract away
2566.9s
the like source code manager.
2569.3s
And so
2570.3s
that was materially intern work
2572.6s
that we did today.
2573.8s
So to answer your question, it is now.
2577.2s
Its just kind of slowly
2578.8s
climbing up more and more
2579.9s
the level of complexity of these tasks. The futures really here. I have a question about competition, and not specifically the competition in your space. But I think how you more generally think about navigating competition, I think you guys are the type of founders that a lot of companies in the application layer really look up to, because youre
2583.1s
The futures really here.
2584.9s
I have a question about competition,
2587.0s
and not specifically
2588.0s
the competition in your space.
2589.2s
But I think how you more generally
2591.3s
think about navigating competition,
2593.1s
I think you guys are the type of founders
2594.8s
that a lot of companies
2596.8s
in the application
2597.6s
layer really look up to,
2599.4s
because youre
2599.8s
insanely ambitious, building a real company of meaning. Youre making a lot of smart decisions, like, you know, riding on other peoples models. I think the obvious kind of, like, scary thing, scary, kind of other side of that is, you know, every other competitor in the space has access to the same models as you. And so Im curious
2601.7s
building a real company of meaning.
2605.0s
Youre making a lot of
2605.8s
smart decisions, like,
2606.9s
you know,
2607.3s
riding on other peoples models.
2610.8s
I think the obvious kind of, like,
2613.3s
scary thing, scary,
2615.1s
kind of other side of that is,
2616.6s
you know,
2617.3s
every other competitor in
2618.5s
the space has access
2619.4s
to the same models as you.
2621.0s
And so Im curious
2622.0s
how maybe just mentally and then I guess, overall, you think about approaching competition in this space? Do you think its more elevated in the application layer AI market, than other startup markets historically? And how do you think about navigating that? Totally. Yeah, I think thats a great question. I think thats really, you know, our approach to
2623.8s
and then I guess, overall,
2625.3s
you think about approaching
2627.2s
competition in this space?
2628.8s
Do you think its more elevated in
2631.0s
the application layer
2631.8s
AI market, than other startup
2634.8s
markets historically?
2635.7s
And how do you think
2636.1s
about navigating that?
2637.5s
Totally.
2638.0s
Yeah, I think thats a great question.
2639.1s
I think thats really,
2640.6s
you know, our approach to
2641.9s
that has defined how weve built out this team. And really, I think there are a lot of ways you can respond to competition and like mentally, kind of justify your existence versus competitors. I think for us, on the team side, were just a team of people who are more obsessed than anyone else out there.
2642.7s
how weve built out this team.
2644.6s
And really,
2645.5s
I think there are a lot of ways
2647.2s
you can respond to competition
2649.6s
and like mentally,
2650.8s
kind of
2651.5s
justify your existence
2653.4s
versus competitors.
2654.8s
I think for us, on the team side, were
2659.0s
just a team of people
2660.3s
who are more obsessed
2662.7s
than anyone else out there.
2664.8s
And I think that is like something that just has the compounding benefit of I am willing to bet everything that the people that we have assembled are just more obsessed than everyone else working in this space. I think a kind of a corollary to that is, the only way you can win is by executing faster.
2665.8s
something that just has
2667.0s
the compounding benefit of
2668.8s
I am willing to bet everything
2670.6s
that the people that we have assembled
2672.5s
are just more obsessed than everyone else
2675.0s
working in this space.
2676.4s
I think a kind of a corollary to that is,
2679.5s
the only way you can win is by
2682.2s
executing faster.
2684.6s
Everything else is all just like sprinkles on top. The only way you can really win is by executing faster and being more obsessed. And that is what our team is and I think I guess one last thing is having a group of people who respond to kind of external pressures, as more motivating. And,
2685.8s
just like sprinkles on top.
2687.4s
The only way you can really
2688.7s
win is by executing faster
2690.7s
and being more obsessed.
2692.1s
And that is what our team is and I think
2696.3s
I guess one last thing is
2698.5s
having a group of people who respond
2701.9s
to kind of external pressures,
2704.4s
as more motivating.
2706.0s
And,
2708.5s
you know, responding in that way, also being very mission driven, right? Like, if you know, a competitor that does something big and then suddenly youre deflated? Well, if youre truly obsessed with a mission, its irrelevant, right? If youre truly obsessed with our goal of bringing autonomy to software engineering, all of that is noise.
2710.5s
also being very mission driven, right?
2712.5s
Like,
2713.2s
if you know, a competitor
2714.4s
that does something big
2715.5s
and then suddenly youre deflated?
2716.7s
Well,
2717.2s
if youre truly obsessed
2718.1s
with a mission, its irrelevant, right?
2720.1s
If youre truly obsessed
2721.0s
with our goal of bringing autonomy
2722.5s
to software engineering,
2724.1s
all of that is noise.
2725.3s
What we need to do is execute as fast as possible in this direction that weve set and the rest will sort itself out. Love it. Really well said. Maybe a few final questions to close this out. If you werent solving the kind of autonomous software engineering problem, what problem would you be solving?
2726.8s
as fast as possible
2728.2s
in this direction that weve set and
2731.4s
the rest will sort itself out.
2733.3s
Love it. Really well said.
2734.7s
Maybe a few final questions
2736.9s
to close this out.
2738.2s
If you werent
2739.2s
solving the kind of autonomous software
2741.3s
engineering problem,
2742.6s
what problem would you be solving?
2744.6s
I guess Id have to be banned from coding agents for this. Perhaps robotics? I find robotics very interesting. I think a lot of the time the team here, a lot of the team comes from backgrounds working on autonomy and robotics. And we talked about how what were building really kind of resembles that in many ways. I think
2745.8s
from coding agents for this.
2748.8s
Perhaps robotics?
2750.0s
I find robotics very interesting.
2752.0s
I think a lot of the time the team here,
2753.9s
a lot of the team comes from backgrounds
2755.8s
working on autonomy and robotics.
2758.2s
And we talked about how
2759.9s
what were building
2760.5s
really kind of resembles
2761.5s
that in many ways.
2762.7s
I think
2763.2s
multimodal function-calling LLMs are here, and the robotics company with decreased hardware costs that are coming out, are clearly making progress. So it feels like a fun area. So you'd be making physical droids? Exactly. It's on the roadmap. How about you, Matan? Yeah, I think this is one of my blind spots where I just suffer
2765.0s
LLMs are here, and the robotics company
2768.7s
with decreased hardware costs
2770.2s
that are coming out,
2771.1s
are clearly making progress.
2772.5s
So it feels like a fun area.
2773.9s
So you'd be making physical droids?
2776.8s
Exactly.
2778.2s
It's on the roadmap.
2779.8s
How about you, Matan?
2781.7s
Yeah,
2783.2s
I think this is one of my blind spots
2785.7s
where I just suffer
2786.9s
from severe tunnel vision. I genuinely cannot fathom working on anything else. Im just genuinely obsessed with our mission to bring autonomy to software engineering. If I wasnt working on this, Id figure out a way to work on this. I know thats a cop out answer. But I genuinely can't, It does not compute.
2789.0s
I genuinely cannot fathom
2790.8s
working on anything else.
2791.9s
Im just genuinely obsessed
2794.8s
with our mission
2795.7s
to bring autonomy
2796.5s
to software engineering.
2797.5s
If I wasnt working on this, Id
2798.8s
figure out a way to work on this.
2800.3s
I know thats a cop out answer.
2801.7s
But I genuinely can't,
2803.3s
It does not compute.
2804.5s
So that is, in fact, a cop out answer, but it is a fantastic cop out answer. So we will take it. One of the questions that I always like to ask is who do you admire most in the world of AI, and tell you what Matan because of your background
2807.1s
but it is a fantastic cop out answer.
2809.4s
So we will take it.
2810.8s
One of the questions
2812.0s
that I always like to ask is
2813.2s
who do you admire
2814.0s
most in the world of AI,
2815.6s
and tell you what Matan
2816.6s
because of your background
2817.8s
will let you look at the superset of AI and physics, if you like. I would say theres some sort of name that comes to mind when you say that is Jeff Dean. I think we mentioned him earlier already, actually. But you know, his impact and research is one huge side of that. I think TensorFlow and
2820.8s
and physics, if you like.
2823.8s
I would say
2824.6s
theres some sort
2825.1s
of name that comes to mind
2826.1s
when you say that is Jeff Dean.
2828.3s
I think
2829.5s
we mentioned him
2830.2s
earlier already, actually.
2831.8s
But you know, his impact and research
2834.5s
is one huge side of that.
2836.0s
I think TensorFlow and
2838.2s
the kind of the work that that whole team has done at Deep Mind and related. But Ive also heard he's a nice guy. And I think that the thing is, having responsible leadership in the AI community, I think is really important, and theres a lot of folks who are on Twitter all the time, you know, clashing.
2839.5s
that that whole team has done at Deep
2841.7s
Mind and related.
2842.9s
But Ive also heard he's a nice guy.
2845.2s
And I think that the thing is, having
2848.3s
responsible leadership in the
2852.6s
AI community,
2853.2s
I think is really important,
2854.8s
and theres
2855.4s
a lot of folks who are on Twitter
2857.0s
all the time, you know, clashing.
2859.3s
And I think that seeing folks who are outside of that side of it, I think is pretty great. Yeah. And I think not to not to give you guys a double cop out. But at Factory, we very highly emphasized collaboration. And I think like in AI in particular everything has been done by groups of people.
2861.4s
who are outside of that side of it,
2864.5s
I think is pretty great.
2866.1s
Yeah.
2866.4s
And I think
2867.1s
not to
2868.1s
not to give you guys a double cop out.
2869.9s
But at Factory,
2871.3s
we very highly emphasized collaboration.
2874.8s
And I think like in AI in particular
2878.0s
everything has been done
2879.1s
by groups of people.
2880.6s
And so its hard to really think about one individual. I think in physics there are a lot more like solo geniuses doing something crazy. But I think a team like recently that I think we really admire at Factory is Mistral, and how kind of quickly they basically came into open source and brought both those models
2881.2s
hard to really think
2882.3s
about one individual.
2883.5s
I think in physics there are a lot
2884.5s
more like solo geniuses
2885.8s
doing something crazy.
2887.9s
But I think a team like
2889.1s
recently that I think
2890.2s
we really admire at Factory is Mistral,
2893.2s
and how kind of quickly they basically
2896.5s
came into open source
2897.8s
and brought both those models
2900.1s
to basically the cutting edge in a super short amount of time. And I think, yeah, I speak not just for myself, but I think all of our team really admires both the mission that they have and the speed with which they executed on that. So yeah, I would say Mistral. Awesome. Alright, last question,
2901.5s
in a super short amount of time.
2903.3s
And I think,
2904.5s
yeah, I speak not just for myself,
2906.2s
but I think all of our team
2907.3s
really admires
2908.3s
both the mission that they have
2909.5s
and the speed
2910.1s
with which they executed on that.
2912.9s
So yeah, I would say Mistral.
2915.2s
Awesome.
2916.3s
Alright, last question,
2917.2s
if you had to offer one piece of advice to founders or would-be founders, hoping to build in AI. What piece of advice would you offer them? We are in a land of picks and shovels. And no one has struck gold yet clearly. So Id say go for gold. I would say try to build something
2920.0s
to founders or would-be
2921.8s
founders, hoping to build in AI.
2924.5s
What piece of advice
2925.3s
would you offer them?
2926.9s
We are in a land of picks and shovels.
2929.8s
And no one has struck gold yet clearly.
2933.5s
So Id say go for gold.
2936.2s
I would say
2937.2s
try to build something
2938.4s
that you think is going to get 10x Better if OpenAI releases GPT-6 or 7. I think internally we think of our product as something that will multiply in value and uniqueness when new models are released. And its I think, for us, its always like were listening to the OpenAI announcement yesterday. And you know, everyone is excited,
2940.3s
10x Better if OpenAI releases GPT-6 or 7.
2944.2s
I think internally
2945.9s
we think of our product
2946.7s
as something that will multiply in value
2949.2s
and uniqueness
2950.4s
when new models are released.
2952.2s
And its
2953.7s
I think, for us, its
2954.8s
always like were
2955.8s
listening
2956.2s
to the OpenAI announcement yesterday.
2959.0s
And you know, everyone is excited,
2961.3s
everyones pumped when a new model comes out when open source does something great. If youre stressing about new product releases or demos, it might mean its worth adjusting your product strategy. Congratulations on launching Factory and beating state of the art on SWE-bench by such a wide margin last week. Its incredible. Just for our audience,
2962.1s
when a new model comes out
2963.2s
when open source does something great.
2965.1s
If youre
2965.4s
stressing about new product
2967.5s
releases or demos,
2968.8s
it might mean its worth
2970.7s
adjusting your product strategy.
2975.0s
Congratulations on launching
2976.2s
Factory and beating
2977.6s
state of the art on SWE-bench
2978.8s
by such a wide margin last week.
2980.4s
Its incredible.
2981.8s
Just for our audience,
2982.5s
can you maybe quickly recap what SWE-bench is? Yeah, absolutely. And thank you all credit goes to the Factory team for making it happen. SWE-bench is a benchmark designed to test an AI systems ability to solve real world and software engineering tasks. So its around 2300 issues which were taken from contributions made to 12, popular
2983.2s
recap what SWE-bench is?
2985.0s
Yeah, absolutely.
2986.3s
And thank you all credit
2987.8s
goes to the Factory team
2989.5s
for making it happen.
2990.9s
SWE-bench is a benchmark designed
2994.3s
to test an AI systems ability to solve
2997.9s
real world and software
2999.4s
engineering tasks.
3000.8s
So its around 2300 issues
3003.2s
which were taken from contributions
3005.9s
made to 12, popular
3007.5s
open source Python projects. And typically, these issues are bug reports or unexpected behavior that people reported on these open source projects. And the idea is, you know, all of these real world issues were addressed by other humans. And so you have the ground truth of what a human software engineer would do when faced with an issue.
3009.8s
And typically,
3011.0s
these issues are bug
3012.8s
reports or unexpected behavior
3014.9s
that people reported
3015.9s
on these open source projects.
3018.4s
And the idea is, you know, all of these
3022.2s
real world issues
3023.7s
were addressed by other humans.
3025.5s
And so you have the ground truth of what
3028.0s
a human software engineer would do
3030.2s
when faced with an issue.
3031.5s
And the benchmark is trying to test, can your AI system go through each of these issues, and generate a code change that properly addresses it, and comparing it to the human solution with tests that a human wrote. And so theres a lot of asterisks, but it is a somewhat useful approximation of your systems ability
3035.4s
can your AI system
3036.6s
go through each of these issues,
3038.6s
and generate
3039.5s
a code change that properly addresses it,
3042.9s
and comparing it to the human solution
3045.5s
with tests that a human wrote.
3048.5s
And so theres a lot of asterisks,
3050.9s
but it is a somewhat useful approximation
3054.0s
of your systems ability
3055.1s
to take natural language, and then turn that into code. And I think the previous high watermark on SWE-bench was 14 or so from cognition, Devin until last week, and you put up a really impressive new result at 19, which is such a wide margin. This is such a competitive field right now. And unlike such a competitive benchmark
3056.8s
and then turn that into code.
3058.9s
And I think the previous
3060.0s
high watermark on
3061.4s
SWE-bench was 14
3062.6s
or so from cognition, Devin
3064.3s
until last week,
3065.8s
and you put up a really impressive
3067.7s
new result at 19,
3069.4s
which is such a wide margin.
3071.2s
This is such a competitive field
3073.1s
right now.
3074.0s
And unlike such a competitive benchmark
3075.3s
that everyone is trying to beat, which makes you know, your result even more impressive. Could you maybe share a little bit about your approach? And howd you get there? Definitely. And one of the main reasons we were instead in SWE-bench is that theres a lot of companies and research labs that made submissions. So you can see Microsoft
3076.5s
which makes you know,
3077.8s
your result even more impressive.
3079.9s
Could you maybe share a little bit
3081.2s
about your approach?
3082.4s
And howd you get there? Definitely.
3084.8s
And one of the main reasons
3086.0s
we were instead in SWE-bench
3087.5s
is that theres
3088.5s
a lot of companies and research labs
3090.3s
that made submissions.
3091.4s
So you can see Microsoft
3092.4s
Research, Amazon, IBM, ByteDance. And I think thats a testament to the suite bench teams effort and making this benchmark a household name, which is great. I think one of the reasons we were able to out-compete well funded tech giants and other AI code gen startups is that were honestly not building the code droid for a benchmark,
3095.0s
And I think thats a testament
3097.8s
to the suite bench teams effort
3099.5s
and making this benchmark
3101.0s
a household name, which is great.
3103.3s
I think one of the reasons
3104.5s
we were able to out-compete
3106.4s
well funded tech giants and other AI code
3109.5s
gen startups
3110.5s
is that were
3111.1s
honestly not building the code
3112.7s
droid for a benchmark,
3114.9s
but rather to support real world customers. And weve always said customers are the best benchmark. And I think theres some great evidence for the success of that approach. Theres a few areas our technical report goes into around planning and task decomposition, environmental grounding, codebase understanding. But overall, I think that the thing that matters
3116.1s
real world customers.
3117.9s
And weve always said
3118.9s
customers are the best benchmark.
3121.2s
And I think theres
3121.8s
some great evidence
3122.7s
for the success of that approach.
3124.9s
Theres a few areas
3126.2s
our technical report goes into around
3128.3s
planning and task
3129.5s
decomposition, environmental
3130.8s
grounding, codebase understanding.
3133.0s
But overall,
3134.1s
I think that the thing that matters
3136.2s
most when your teams working on these types of general software problems, is kind of like what is the northstar? What are you iterating against? And so having kind of a real world data set can make a huge difference. And we just had Harrison on the podcast last week actually talking about cognitive architectures. To what extent did
3139.2s
these types of general software
3140.9s
problems, is kind of
3142.1s
like what is the northstar?
3143.3s
What are you iterating against?
3145.0s
And so having kind of a real world data
3147.8s
set can make a huge difference.
3150.9s
And we just had Harrison on the podcast
3152.5s
last week
3153.0s
actually talking
3153.6s
about cognitive architectures.
3155.2s
To what extent did
3156.3s
prompt engineering and cognitive architectures play a role here and your results? I would characterize our research as continuously pushing the question of how can we model each droids architecture to more closely resemble the human cognitive process that takes place during the task? Its funny, we actually have internally been referring to the flow of data
3156.9s
and cognitive architectures
3157.9s
play a role here and your results?
3159.7s
I would characterize our research
3161.5s
as continuously pushing the question of
3164.1s
how can we model
3165.2s
each droids architecture
3167.1s
to more closely resemble
3168.6s
the human cognitive process
3170.6s
that takes place during the task?
3172.6s
Its funny,
3173.8s
we actually have internally
3175.4s
been referring
3176.0s
to the flow of data
3177.2s
and LLM calls as the droid to architecture, basically, since the first droid. And when Harrison first wrote about cognitive architectures, it really became apparent that that concept, cognitive architecture, is a great mental model for how to characterize the systems that have complex LLM interactions and data flow. And so, you know, for us, I think the meta problem
3178.9s
to architecture, basically,
3180.3s
since the first droid.
3181.6s
And when Harrison first wrote
3183.2s
about cognitive architectures,
3184.9s
it really became apparent that
3187.2s
that concept, cognitive
3188.3s
architecture, is a great mental model
3190.6s
for how to characterize the systems
3194.0s
that have complex LLM
3194.8s
interactions and data flow.
3197.5s
And so,
3198.8s
you know, for us,
3199.4s
I think the meta problem
3200.8s
of designing a good cognitive architecture is balancing flexibility with rigidity in the actual workflow. You want very rigid entry points, and certain common trajectories, like error recovery need to be really consistent. But then you want the flexibility in the dynamics during the majority of the problem solving process.
3201.8s
a good cognitive
3202.8s
architecture is balancing flexibility
3206.9s
with rigidity in the actual workflow.
3210.6s
You want very rigid
3212.2s
entry points, and certain
3214.2s
common trajectories, like error recovery
3217.7s
need to be really consistent.
3219.3s
But then
3220.4s
you want the flexibility in the dynamics
3222.7s
during the majority of the
3223.8s
problem solving process.
3229.0s
But I think its one of the most interesting problems when building the droids is how do you know, kind of when to add structure and when to let the droid so to speak handle it? Really cool. So every droid has its own cognitive architecture that mirrors as closely as possible with a kind of human
3230.0s
most interesting problems
3232.0s
when building
3232.7s
the droids is
3233.3s
how do you know, kind of
3235.0s
when to add structure
3236.9s
and when to let the droid
3239.4s
so to speak handle it? Really cool.
3241.8s
So every droid has its own
3243.0s
cognitive architecture
3244.0s
that mirrors as closely as possible
3246.2s
with a kind of human
3247.0s
equivalent of that task would be doing? Yeah, exactly. 19 is amazing compared to prior state of the art. It also still feels quite far away from, you know, reliable code droids that people will just trust to run wild in their codebase. What do you think is the threshold at which engineers will actually start
3249.5s
Yeah, exactly.
3251.7s
19 is amazing
3253.9s
compared to prior state of the art.
3255.8s
It also still feels quite far away from,
3258.3s
you know,
3259.4s
reliable code droids
3260.7s
that people will just trust
3261.5s
to run wild in their codebase.
3264.4s
What do you think
3265.5s
is the threshold at which engineers
3267.2s
will actually start
3267.8s
to use these code droids reliably and just, you know, let them run? Are we there yet? Or what is the threshold? Yeah, for sure. I think that one thing to keep in mind is that the percentage on a benchmark like SWE-bench is kind of like one of many possible measures, because the answer really is
3269.0s
reliably and just,
3270.4s
you know, let them run?
3271.9s
Are we there yet?
3272.6s
Or what is the threshold?
3274.8s
Yeah, for sure.
3276.4s
I think that one thing to keep in mind
3279.0s
is that the percentage on a benchmark
3282.9s
like SWE-bench
3283.9s
is kind of like one of many
3285.7s
possible measures,
3286.9s
because
3287.6s
the answer really is
3289.0s
that they are already using it in production. But the use cases that might kind of highlight what the code droid is designed for, may not necessarily have a ton of overlap with what is tested in a given benchmark. So if you take human eval or some of the other coding benchmarks, that maybe tests your ability
3290.0s
using it in production.
3291.3s
But the use cases
3293.5s
that might kind of highlight
3295.5s
what the code droid is designed for,
3297.5s
may not necessarily have a ton of overlap
3300.3s
with what is tested in a given benchmark.
3303.8s
So if you take human eval
3305.5s
or some of the other coding benchmarks,
3307.5s
that maybe tests your ability
3309.4s
to pass a coding interview, but it doesnt really test like your real world software engineering. SWE-bench I think actually does test a lot of real world software engineering. But in the particular context of debugging or kind of, you know, unexpected behavior identification, there are some feature requests, and there are a lot of kinds of not explicitly debugging
3311.0s
but it doesnt really test
3312.0s
like your real world
3313.0s
software engineering.
3314.2s
SWE-bench I think actually does test
3316.2s
a lot of real world software engineering.
3318.1s
But in the particular context
3319.7s
of debugging or kind of,
3322.3s
you know, unexpected
3323.5s
behavior identification,
3325.2s
there are some feature requests,
3327.0s
and there are
3328.0s
a lot
3328.7s
of kinds
3329.6s
of not explicitly debugging
3331.1s
style problems. But tasks like a migration, or refactor, a modernization that takes place over multiple changes that oftentimes have humans very heavily collaborating, are really a pretty different problem. And our internal evaluations are much more focused on those customer tasks. And so we have way higher reliability rates, for like those styles of tasks.
3332.0s
But tasks like a migration, or refactor,
3335.6s
a modernization that takes place over
3338.0s
multiple changes
3339.5s
that oftentimes have humans
3341.1s
very heavily collaborating,
3344.2s
are really a pretty different problem.
3346.0s
And our internal evaluations
3347.8s
are much more focused
3349.1s
on those customer tasks.
3350.8s
And so we have way
3351.5s
higher reliability rates,
3353.3s
for like those styles of tasks.
3357.0s
And I also think that a huge part of the role of human AI interaction design is acknowledging where the systems are currently falling short, and building into your interaction pattern accommodation for the weak points of the AI system. This isnt going to 100 of the time, perfectly capture the intent of what you were doing.
3358.2s
a huge part of the role of human
3362.2s
AI interaction
3363.3s
design is acknowledging where the systems
3366.5s
are currently falling short,
3368.3s
and building into your interaction
3370.0s
pattern accommodation
3371.8s
for the weak points of the AI system.
3374.0s
This isnt going to 100 of the time,
3377.1s
perfectly capture
3378.8s
the intent of what you were doing.
3380.4s
So how do you kind of have failure trajectory handling? How do you introduce the ability to kind of edit midway, as the code droid is working to observe and have kind of some interpretability into why a code droid is making a decision. So that when it does something, the human being actually can step in,
3381.4s
have failure trajectory handling?
3384.1s
How do you introduce the ability
3385.5s
to kind of edit midway, as the code droid
3388.9s
is working to observe
3390.7s
and have kind of some interpretability
3392.7s
into why a code droid
3393.8s
is making a decision.
3394.9s
So that when it does something,
3396.6s
the human being actually can step in,
3399.8s
or at least understand what went wrong. And so I think that those allow you to say, well, we may not be at, you know, 100 on someone like SWE-bench, but we can still use this and get kind of real productive gains, in the meantime. Totally makes sense. And I hear you that SWE-bench is not the be-all, end-all.
3402.0s
And so I think that
3402.8s
those allow you to say,
3404.9s
well,
3405.2s
we may not be at,
3406.2s
you know, 100 on someone like SWE-bench,
3408.3s
but we can still use this
3410.7s
and get kind of real productive gains,
3412.8s
in the meantime. Totally makes sense.
3415.5s
And I hear you that SWE-bench
3416.8s
is not the be-all, end-all.
3418.0s
But since you have a good crystal ball into this space, do you have a prediction, at what point well get to 80 or 90 on SWE-bench? I think that the pace right now is really, really fast. Theres a kind of interesting question of, will we get to 80 to 90, on SWE-bench or will there
3420.5s
crystal ball into this space,
3422.5s
do you have a prediction,
3423.5s
at what point well
3424.6s
get to 80 or 90 on SWE-bench?
3426.6s
I think that the
3428.8s
pace right now is really, really fast.
3432.2s
Theres a kind of
3434.2s
interesting question of,
3436.4s
will we get to 80 to 90, on SWE-bench
3439.5s
or will there
3440.2s
be a better benchmark that kind of comes out before we can really meaningfully start hill climbing past like the 50-60. Theres honestly a lot of tasks and SWE-bench, which are, I wouldnt say impossible, but it almost feels like getting them right would almost only indicate that youre cheating. Its like they test for really, really specific claims,
3441.8s
that kind of comes out before
3444.7s
we can really meaningfully start
3446.5s
hill climbing past like the 50-60.
3449.0s
Theres
3450.3s
honestly a lot of tasks and SWE-bench,
3452.8s
which are, I wouldnt say impossible,
3457.4s
but it almost
3458.3s
feels like getting them right
3459.7s
would almost only indicate
3461.2s
that youre cheating.
3462.1s
Its like they test for really,
3464.3s
really specific claims,
3466.4s
or a string match. And so I think that before we see 80 to 90, on SWE-bench, what well actually see is kind of like SWE-bench two and three, bench three that focuses on trying to think deeply about how can we evaluate when, you know, a piece of code is correct, but also kind of ideal or useful
3468.2s
And so I think that before
3470.0s
we see 80 to 90, on SWE-bench,
3472.2s
what well
3472.6s
actually see is
3474.1s
kind of like SWE-bench
3475.0s
two and three, bench
3475.9s
three that focuses on
3478.2s
trying to think deeply about
3479.8s
how can we evaluate when,
3482.7s
you know, a piece of code is correct,
3485.9s
but also kind of ideal or useful
3490.2s
for a given codebase. The SWE-bench folks actually have a lot of really great thoughts about how to make these benchmarks better, but I think probably in the next two, three years, well see that. Yeah, and theyre Princeton guys as well. Right? Yeah, yeah, they are. We actually shared a thesis advisor. No way. Thats very cool.
3491.9s
The SWE-bench folks
3492.8s
actually have a lot
3493.7s
of really great thoughts
3494.7s
about how to make
3496.2s
these benchmarks better,
3497.4s
but I think probably in the next two,
3499.0s
three years, well see that.
3501.0s
Yeah, and theyre Princeton guys as well.
3502.4s
Right?
3502.9s
Yeah, yeah, they are.
3504.2s
We actually shared a thesis advisor.
3508.5s
No way.
3509.3s
Thats very cool.
3510.7s
Well, Eno, Matan, thank you so much for the conversation. Congratulations again on these results and on launching Factory. We are so excited. Thank you. Thank you very much. For.
3512.2s
thank you so much for the conversation.
3513.7s
Congratulations again
3514.8s
on these results and on
3515.9s
launching Factory. We are so excited.
3518.0s
Thank you.
3518.6s
Thank you very much.
3523.9s
For.
