Follow along on YouTube (opens in a new tab). Back to the episode
Intro
Benedikt Stemmildt
Moin moin, and welcome back to another episode of Rethink Engineering. This is a very special one for me because we discuss a lot of the topics why we renamed the podcast with our guest today. I'm Benedikt, I'm the co-founder of hackers&wizards. We teach engineering teams how to work with agents and factories and this show here is powered by us and also by re:cinq.
My guest today is Patrick Debois. I'm a huge fan. He organized the first DevOpsDays in Ghent, I think 2009. So he's been doing that for a while now. And there are a lot of similarities between DevOps and agentic engineering. So he has really interesting thoughts on what we need to do and what is happening at the moment.
We debate factories, organization, also how this changes our roles. And one thing that got me specifically was that we are currently maybe focusing too much on code and we overlook the possibilities that these factories have once we focus on outcome. And yeah, that is what we are going to discuss in the episode. I really enjoyed this one.
I hope you do too. Have fun. Hi, hello, Patrick. Super awesome to have you here. Thank you so much. I'm really honored. We're looking forward to the episode.
Patrick Debois
As they say, moin moin. No, I don't know what to say. This is how you always greet me. So thanks for having me, Benedikt, on the show.
Benedikt Stemmildt
Awesome. So let's, I don't know, let's just I think dig into the topic. I'm super interested in what's on your mind at the moment. How do you feel about engineering? I think you have a lot to share. So maybe you just start with whatever you think about at the moment the most.
Software Factories Through a DevOps Lens
Patrick Debois
I'm trying to make sense of software factories and dark factories. And given my background in DevOps, in CI/CD, trying to see what's the same, what is different. Kind of that is kind of maybe like my analysis. Where is this going and where's the adventure heading, basically? So that's kind of on my mind.
Benedikt Stemmildt
You know, we have a similar journey here, also trying to figure that out. And I'm super interested in what do you feel is the difference, though? Like, what is your, like, is there a difference? Is it the same as CI/CD? What's the current status there?
Patrick Debois
Well, I was looking kind of comparing companies that are offering factory solutions. And I was a little bit sad in a way that it was very similar. They were all kind of, hey, we're producing code, we're reviewing code, we're deploying code. And it felt like, in a way, it was the same. Obviously, the technology is new and we can do new functionalities like doing in the review.
We could have different ways instead of just running and testing the code. We can have it assess code. But we can have adversary code checks. And I think one maybe emergent topic that comes up more often is the one about risk. And that is especially if you take it up to a traditional CI/CD, you level it up step by step.
And you add more checks to it. And you assume that the checks are enough to do a certain level of deployment. But there was still a lot of human involvement. And now what we hope for is that there is a lot more autonomy on the system kind of putting things into production. And that verification layer was traditionally testing, as I mentioned.
But now it's like a whole set of verifiers, you could call them. But when do you actually merge? That decision, well, it's actually even beyond that. It's not just what you merge. If you merge, you deploy. What do you put in front of your customers? And what's the risk analysis of that? And I find that fascinating that everything that we've done is kind of reduced to what's the risk of putting something out there versus the benefit of putting something out there for a customer.
Anyway, that's interesting in the way that I draw the parallel of we automated so much with DevOps that we could instantly get a new machine, it has a new version, kind of very fluently. But it was always the discussion, you should automate, let's say, things like database schemas too. But not a lot of people dared to do that because the risk was outweighing that manual step.
And I feel now we're kind of in similar conversations. But what can you do? And and I feel there's a strong correlation to and I think that's an undersold value of the software where you can run all the tests and it passes and you can review the code and it's in a good shape. But we learned what kind of the deployment that it is your architecture of your application that will absorb a certain level of uncertainty.
The fact that I could kill my container and it starts a new one is the way we architected microservices in a way. And I'm feeling like, hey, if you compare to make like compartments of your code and pieces, they could say you can touch this, which is OK, like another color might not be the end of the world.
But if your payment system goes down, you might have a bigger issue. So kind of like finding those pieces. And it's just an example. There's there's obviously per domain, per company, the risks are different. And how do you deal with that?
Risk Is the Real Merge Decision
Benedikt Stemmildt
Interesting. So so you say basically that I mean, the idea, the idea of the factory is that you can actually automate more of your engineering work away that you were not able to to do before. Right. So the writing of the code or the reviewing of the code and so on. And you're saying that really hard to figure out how to lift that up from like being a lot in the loop and being not so much in the loop. Right. And that is this risk problem.
Patrick Debois
Yeah, correct. Yeah.
Benedikt Stemmildt
Yeah. OK. And the idea that you described is that maybe not every part of the code has the same risk problem. And that's OK. And that's why someone could kind of figure out if that is something that we could also do in the factories that that we kind of compartmentalize our code and so on different than we did it before. Is that the thing that you're saying?
Patrick Debois
Well, we're like it's one way of looking at it. Right. I'm almost already projecting. Imagine everybody is automating that same loop code review. It's not going to be that different in every factory. OK, well, maybe have some bells and whistles on kind of how that loop will go. But ultimately, what the the bigger promises of the dark factory, not just you know, kind of the factory as we know it, is that there is decisions made by machines in a way that we don't have to kind of put our attention to it.
So if the factory decides on its own to do a change, will we always have to do the review? Yes or no? And that's kind of OK, I can put a million tests on there, but still what's the risk of it doing this without you in review? And that's kind of the more of the advanced piece that I would look at in a factory.
Like, how does it help you not just like review the code, but kind of what could it do to take away pieces of that piece? So while it's not really engineering work, like it's not building it. It's not the mechanical deployment. It is more of a business centric like decision. What are you should like do it? Yes or no?
And I feel that there is a correlation between the architecture, but it could also be maybe your service level agreements or your customer size. Or I don't know what, like they're all kind of factors that go into that piece. But I also recognize this is maybe more of the advanced piece in the factory where a lot of this will accumulate too, because we want it to be autonomously. And that's kind of one of the challenges that needs to be there.
Benedikt Stemmildt
I think that that's quite on point. But for me, I didn't think that much of risk at the moment. So I think that is a really, really insightful for me. It was I always talk this story about trust, right? I trust my CI/CD pipeline because it's deterministic and because I have a lot of checks in there where I know, OK, it's not going to do anything stupid.
Kind of the question for me was always like, yeah, how do I need to build my factory so that I can trust it to make the right decisions? I don't know if that changes the idea, but maybe risk is it's a better approach because I really like this notion that you're getting at is like the architecture has some correlation there because I mean, I mean, we have dependencies, for example.
I don't I mean, I pin the version of the dependency and I'm fine with this part of the code. And I feel the risk is low then, right? So for this part of the and I could build a I could rebuild this dependency of the interface to the outside world is the same. I don't I mean, there's maybe this is like a thing which is compartmentalized where I don't have a lot of risk because the interface is quite clear.
And I could kind of do that with more of my components and then say this component is not the factory does not decide anything there because I don't trust it yet to do that or something.
Patrick Debois
There's a certain fallacy into believing that we can test all the things and that kind of is maybe an uneasy truth if you put things out there, because there's always a different piece of, OK, of course, in production is different than test, right? Because there is users, there's different things that will happen. But if we really want to like look at all the code that the thing is producing, there's going to be some cases where it just slipped.
And but it's it's similar to what kind of a human does. But they usually have this kind of feeling of, hey, this needs like, you know, a peer review and this is like, you know, that kind of thing. So analyzing that and bringing that to surface and then kind of mitigating that in your architecture is probably something that makes sense.
And I started to think about the the dark factory has gradations in a way that like, you know, for some cases, there's no it doesn't make sense with the risk you're taking to automate certain pieces. And others will be, yes, of course, that that's the. So it's a spectrum of that. And the way you decide on the spectrum is what is the impact and the risk and the likelihood kind of similar to, you know, things that you put into production and automating that will be more instrumental to.
It's saying you can go to production. Yes, no, but it's maybe additional. I'm not saying dump all the tests, but it's it's another decision that you have to see, like, hey, I looked at the piece. You didn't touch that. You didn't touch that. Seems fine. I can do this autonomously. And and that's maybe a step that we couldn't do, like something adding to something to a pipeline that like it decided on its own and then kind of saying, well, you can do it on your own.
Dark Factories Come in Gradations
Benedikt Stemmildt
There's a lot of discussion about, like, should all of these checks be deterministic, right? And what about also non deterministic checks in there? And I feel that that this is really a new task here as well as to, as you said, like to figure out what we can we do non deterministically and also still have trust or low risk in it.
Or how can this even be like a benefit that we didn't have before? Now that we have non deterministic abilities, can this even be a huge improvement than only the tests that were deterministic before?
Patrick Debois
Correct. I'm a big fan in turning whatever we can into deterministic checks. There's usually a benefit of speed and cost that we don't have to do it with like LLMs or turns or it takes too long to rediscover those pieces. Now it's a double edged things. It's like when you think about the test as a harness you put around the LLM, right, like, OK, you like you can't pause if you do you can't solve this piece.
There's a there's a fine line between micro controlling your LLM and kind of giving it some freedom that it can solve pieces. Right. So so there are like a cash on in one way of rules you could write in in the prompt. But you need to know almost you need to have a maintenance check that says like, is this still the right way?
And that's kind of the factory reflecting on itself and seeing like, is this still kind of correctly in place because it happened. And we added it because the there was an incident. But is that still true? So it's kind of doubting yourself on the architecture, on the solutions as well.
Orchestrators vs. State Machines
Benedikt Stemmildt
So like you said before that that you feel or or that that you saw that most of these loops might end up to be very similar. All right. Like you have this default expect and plan and so on. I know that there's a discussion at the moment that we might not even need these like pipelines with these checks and so on that that you can just have an orchestrator and it will figure it out all on its own. So what is your take on that?
Patrick Debois
LLMs are great, but they're also not great at certain things. Let's say you want to have a counter. Just just don't use an LLM for that. Right. So that doesn't make sense. Or I had it now multiple times. I build a tool and I say, well, read kind of use an API to get the IDs from the database and then pass that onto something else.
Now, it just hallucinates IDs after a while. So the technology inherent isn't the reliable piece all the time or it might lose context or like you can't read it as this will replace all the things. Now, there's the ultimate hope that as long as you specify everything perfectly, you know, things will go right. But I don't buy into that theory.
I'm not saying you shouldn't specify things. It definitely helps that to give context and kind of make and give better results. But I don't see any even with like exhaustive training. It can't make like understand your business decisions. They can't put those things into its own understanding. Well, it's not really understanding. It's it's more like predicting. So I don't buy into kind of it will take over.
You see, quite often, an orchestrator starts quite loosely of, hey, pick the next task. Right. You know, randomly, here's a thing to do, like do something. What you see most of people in the end building is close to a state machine. Well, first, it's here when you're in that pace. Then you can go there and they put things in in a kind of a graph system.
Why they're doing that? Because they know that it's unreliable to just have it do it on its own. Or the camp of the LLM training will say, given enough training, it will do it. But we're we're somewhere not there. Right. So like why kind of hope it and I think it's that fallacy of thou shalt not fail is kind of what almost in the last decade that we found it fought against and we say assume it's going to fail.
That's a way better approach to kind of deal with that. But that's why if you assume it's going to fail, you've got to think about the risk.
Benedikt Stemmildt
I'm also in the camp of more deterministic and more let's let's make it a state machine. I think there is also this difference between am I sitting in front of it and looking at it all the time, or do I want it to run autonomously somewhere? So that's always how I mean, if I'm or when I'm using it, myself for building apps and I'm really in front of it, I tend to do it more in this orchestrator approach because I can just follow what it does and then interrupt it or kind of figure out if it's in the right direction.
And when I build factories for our clients, it's more like, OK, we need to figure out how we can make that more trustworthy and autonomous in the cloud or everybody will just sit in front of it all the time. So maybe this is also like a notion where it's just a situation, right?
Patrick Debois
There's nothing wrong with sitting in front of it in a way that I explain that to when you were, let's say, traditionally developing, you run things and you kind of have them on your laptop and then eventually you'll move it to your CI/CD. So there is something that you do locally, maybe with a restricted environment like simplified, and you can kind of do that.
And then you put it in the bigger kind of more testable environment. The same thing with testing. There is a phase where you say run all these tests to exploratory testing. So I don't think personally that the factory will be, per se, the best way of iterating fast on like an IDE So that I would typically I keep that local.
Like one where we have a factory and I kind of like hate to do it there because the latency of getting a feedback is just saying, yeah, to insane. And so that's why I like the rapid innovation. But I can, on the other hand, if I have some kind of task that I say research this and it goes off, of course.
And then I look at the results. I don't care that much about having to wait for that whole piece. So it's kind of making it like different tools for different kind of parts of your your journey on this. But one is explorative. And I hope that's somewhere more of an emerging system of the factories that we didn't have before.
We see ourselves now delegating like, hey, you know, come back and check the code. What's kind of causing this? Okay, a very inquisitive thing, right? But it's not like I need to do coding, but it needs to do some research and digging, right? Like me digging in will take a long time. And I could just say, go ahead, come back with the results.
And I probably can kind of analyze that. But there's no things stopping us by saying, well, give me four options. Like it's another typical LLM agent thing. So why aren't we building those research loops in our factories? Like our factory is still so focused on like, take the spec and go ahead. Like, and I hope there's something more.
And this morning I had a discussion and somebody said, well, I hate the word factory because it like kills the creativity there. And then I came up with the idea. I was like, what about it's a dream factory, right? Like it should allow you to dream about ideas that you could do. And that's also how you defend, I think a lot more of the ROI.
If it's something before the factory you would never be able to get to, there's not just a multiplier of one of X it's almost to infinity because I couldn't have done this. Like, like there's no other way. And I think those cases will amplify the value of factory even more than just making sure that we're getting faster at like testing, deploying and kind of those spaces as well. Good point.
Research Loops and the Dream Factory
Benedikt Stemmildt
It's a little bit in the direction as well of like outcome versus output, right? So you could get more output at the moment or we are there maybe yet already. But in the end, what we want is more outcome and figuring out what really creates outcomes is the exploratory part, right? And the dreaming part.
Patrick Debois
Yeah. And that's also why I think there's a closer relationship now with the product owner being able to use the factory and kind of bridging the gap between them, having to build a prototype with vibe coded. And then they come to the factory that needs to be redone. I think like there's not products trying to fuse the two worlds and kind of going there.
So that's almost like the upstream that we're improving. When I was working on DevOps, I always said like there's the pipeline that we know, the DevOps pipeline, but there's a sales pipeline. There's a GTM pipeline. There's kind of a legal pipeline. Like every there's everyone has intake, triage, kind of like do something, kind of like review, okay, loop back.
All right, so and that's another opportunity. I think with the factory is that if you be able to start linking that because the bottleneck is moving from, okay, we can produce more. Oh, now we have to kind of come up with more things. Well, no, we shouldn't. But like the whole thing is actually to say no to things instead of saying yes, which we couldn't do in the past.
And then when we're producing that, like the sales next, kind of all the documentation isn't up to date. So kind of all those pieces are new opportunities that we couldn't kind of keep under control with a factory. But it's not just in code, kind of, you know, test and deploy. There's so much more we could put in a factory.
And that's maybe the imagination. The first iteration needs to settle on what we have. And then we can start adding to it. But I'm very eagerly waiting until we start adding more to it.
Smaller Teams, Fused Roles
Benedikt Stemmildt
Super inspiring. I mean, the question that immediately pops into my mind is something like that will probably change a lot also on the organizational level, right? Like how do organizations actually build like teams and departments and so on? And how do you do leadership? Do we have any idea there that what do you think about that?
Patrick Debois
So the, well, there's this whole notion of agile is dead, but it's not dead because it's the same practices or fast feedback. And you go for like, you know, the thing that matter. The cycles of a scrum, though, they, for example, they would change because instead of just talking about it, you could just do it and kind of give the results.
Another thing that changes is, you know, in the past, somebody would raise a ticket and they would have to wait for engineering capacity. It would be on the backlog. Now they could just start the factory like, okay, let's have a conversation. What is it? Like, what is it failing? Just create the PR and do almost the work.
And then the engineer is reviewing. So there is a kind of opportunity to get like to smaller teams where maybe eight was in the past between, you know, kind of that level. Get closer to four or five in a good, consistent way. But there's still the matter of understanding the code base, I feel, although AI helps with that as well.
But there's a certain level of almost like latent knowledge that somebody in the team needs to have to understand what good looks like, what is the decision of the impact of decisions, and so on. So it's almost like we're back to, we like to fuse the product and the engineering role into one now because they could do it.
But not everybody is a good product owner that is an engineer and in the other way around. It's like a full stack, you know, full stack in everything, maybe. So we're even putting more scrutiny on that. But in general, I kind of look for overlapping. So it's similar to the DevOps, the T-shaped people, they overlap, they kind of collaborate.
So somebody with more product affinity from an engineer and somebody with engineering. Obviously, that makes more sense because there's a shared language to that. I think there's still, if you start doing things at more scale, there's somewhere an attention span needed on operational sites. It's not just creating sites, but there's operational. Although AI will help us there as well.
So I'm now like with the Dev, the PM, and the Ops person. And then I would say you should almost, I wouldn't call it the trainee, but the person who's kind of the backup somewhere. Because that is still a factor that like one person, they're human, they can leave, they're not available. Yes, there's a factory, but kind of that like almost like overseer of the domain.
And so if you don't put that person in, and it could be a junior kind of learning along the others, or it could be just a backup person that you say you put in there. So I personally, I wouldn't go lower than four. You could go lower, but that's kind of forgetting the sustainable base, the risk of a person going away, that feels like the under limit to me.
Benedikt Stemmildt
Yeah, I also think that there's often also this interface to the outside world. Like there's a lot of people coming from outside that want something from you. So someone also needs to buffer that somehow, right? That the team can work productively. So lower than four is actually super small.
Patrick Debois
Yeah, but I wouldn't go lower. But there's people like I'm a solo founder. And of course, you can do it everything if you have like, and the more ties you cut on like having to have communication, there is a certain piece of going faster, but you know, kind of there's a bus factor, definitely in there. And there's a limiting factor.
But for your solo project, that might be fine. And you put it out there. But it comes back again to like, what's the risk, right? If you don't respond in time, what's the risk for your customers? If systems kind of go wrong, what's your response time? And that kind of always makes me on easy when people say, well, I'm not looking at the code anymore.
And then I dig in. Okay, you're not looking at the code anymore. Well, I have my own verifiers. Okay, that's great. And then, and then I asked them, what's the impact? Well, I'm a solo dev. I don't care. Of course, you can do it. Go for it. Like, nobody cares. Big company. It's different, right? Like, it's money, like impact.
So that's why I say there's gradations of people. And they usually don't tell the full story of what they are doing and how they are dealing with kind of that risk from their perspective.
Benedikt Stemmildt
So yeah, and I feel it's also in bigger companies or in general, really hard for a person to view them from outside on themselves. Like, what am I actually doing day to day? Also for the engineers, even, because, I mean, this is something that you would need to do to build such a factory with low risk, right?
You would need to analyze yourself. What do we do? Then take that and build. I feel there's also maybe something like a new role emerging here or our current roles are changing a little bit into some engineers are able to do that, to view on top of the, so you know what I mean, right? To reflect more on the process, how they actually engineer stuff.
Patrick Debois
It's system thinkers, right? Like, that's kind of what you're looking at. And probing the system that failed. When the pipeline of CI/CD around DevOps kind of settled, there was the emerging role of the chaos engineer, which was basically probing at the system, whether it was still working and kind of do hypothesis and kind of almost like the fire drills.
That will likely be there as well in the future as this kind of progresses. We need to have the safety checks in order. We need to make sure that the backups are there. So, I don't see that going away. But obviously, you can say, the only thing I need to do is give us back and it's taken care of.
But that's, yeah, usually people who haven't done maybe the longer running ops, things will fail and you have to deal with that, in my opinion.
Benedikt Stemmildt
In my experience, there's also a lot of engineering skill involved here, right? So, to build these state machines, to build these factories and to be able to system think on what you are doing there, you need to have all of the engineering knowledge. You can't simply just say, well, the LLM will do it somehow, right?
Patrick Debois
I think they're proficient in kind of more of the agentic engineering and they're heading towards factory and learning basically that factory and the loops and sandboxes and cost and all the things that would typically go into a platform. Or it's the other way around. People have been used to running a platform and they say, well, we could bring in the agents.
We can bring in kind of those pieces as well. Now, there is a broader spread of knowledge and as you mentioned about things changing into an organization. I feel it's not that clear cut split, but roughly I would say there's going to be people working and making the factory work and keep that humming and there's people who will likely do more spec, technical requirements and going in there.
The nice thing is that the factory as such is a little bit of an escape for the more technical people that wouldn't drive well into just writing specs. But there is also, like you say, maybe it requires some experience. It's not about age, but more about like, okay, how much of the meta pieces that you can plug into there, you see the effect.
If I change a prompt here, what will happen there? Well, what are the risks on my sandbox? Or how's the cost going up? So you got a more holistic view on the whole thing to think about that. So, but yeah, I think there's a logical platform piece to it, but it's uncertain whether it'll be more taken from the AI engineer side or more from the platform side. I'm still, the jury's out, I think, in the industry there.
Factories as Platforms
Benedikt Stemmildt
Yeah, it really reminds me of the early cloud days where it was not really clear yet where the right abstraction layer would be and how to actually, because with these AI platforms, it's more than an infrastructure. There's a lot of process or rules and so on. Is rules already part of the platform or is it something you put out from outside and so on? I feel we don't really have an answer to that yet.
Patrick Debois
No, but yeah, the platform is never just a machinery. So it's always like going into what is your business process? How do you want to deploy? What's the risk? So it's a very similar discussion if you onboard like new projects onto a platform, but it depends. Like some people just say, I built the platform. I don't care who runs on top of that, but the whole narrative of running your platform as a product is listening to your customers what they need.
And that's basically going in and listening. What do you need? Like, how do we need to adapt? What's useful? What's important for you? And bring them onto that factory as well.
Benedikt Stemmildt
Yeah, interesting. I also really like the idea of splitting these roles a little bit into people that are providing or building on the factory as more being the platform team, maybe. Something like that. We don't really know it yet, but it could be. And then the people that are maybe more awesome doing the exploratory work and the creative work and more to the product side.
And then there needs to be this handover. And both have probably also a different mental model of what they are actually currently. You mentioned this latent space. I don't know. Like, for me, both groups would probably have their own mental model of what they are building, which is both very valuable. You know? But I think there's...
Patrick Debois
Well, in your platform speak, you would say there's the feature teams and the platform team. No, you don't know yet whether that centralization is something that we'll end up with. Or in the end, it will be a very configurable SaaS solution that everybody builds their own pipeline. But it's a little bit like when, as a platform team, you put a CI/CD flow forward, it rarely stayed vanilla within a team.
They all adapted it into some form or some flavor of, well, you know, I want to run this linter. And so it needs to be configurable, but the machinery maybe, you know, we're not all in re-implementing prompt injection kind of detection or something, right? Or not a registry or kind of like an inference gateway. Like, nobody's... But we all wanted it to be adapted to a workflow to go for that.
So it's a little bit like where the GitOps thing came in. Like, we define it, but the platform is actually run centrally with those pieces as well.
Benedikt Stemmildt
I mean, as the person that would do the more exploratory stuff, I also don't want to like need to know all of that, right? I would just want to have it, okay, I can use inference and it's safe. Check. Correct. And someone else will do all the stuff that it's safe for me to use it.
Patrick Debois
Well, there's no complete abstraction. And I'm going to give the example. Let's say you go to a newer model in your factory. Okay. The impact of me writing prompts and me writing pieces. So that's something that still happens. But it's the same thing as I upgrade my version of whatever for a cell or to a new version I need to do upgrade.
So dependencies and upgrades are still going to happen even in factories. But, you know, we might get a pull. Hey, you know, we did this, we run your test with this is proactively for you. But then on a major upgrade, like, what do we do? Like, we all kind of like the nagging on security. We completely dislike it, right? So it's like probably the other way around. But yeah, probably.
Benedikt Stemmildt
I mean, your factory is also a product, right? So you would need to, as you say, you would need to have like a dependency updates for the parts of the factory in there as well. And then, yeah, if the model changes, there's going to be a higher change, right? I think this is like part of the problem why we don't really know this abstraction layer yet.
Because if the model changes, and a lot of your non-deterministic stuff changes immediately because you need to reconfigure it. Like, I don't know, like a little bit like reverse engineering, like how is the model behaving in certain scenarios and then kind of getting rules out and new rules in and so on, depending on which model you choose, right? It's like, correct.
Patrick Debois
Yeah, and usually there's a migration period of, hey, you know, central team is preparing the upgrade and they take the usual use cases and they test it and they see what the pattern is. They try to extrapolate that. They turn that into some tools. Maybe, you know, for the model, there's like some evals. You can run and see what the model gives.
But it's still a lot of engineering, like on both sides to kind of deal with the kind of complexity. So the power is great. But like a lot of people forget is, yeah, there's work still to do when that happens. And the impact is bigger on those flows, like reducing things on a prompt. Like it's a lot of easy to add things to a memory and add things to a prompt, but reducing things.
I was like, what, what am I going to remove? Like, well, let me check what the model already knows. Like, what can it do? And kind of those, yeah, very interesting questions. And I think I was chatting with you a person from Anthropic who's dealing, like every time there's a new model, they have to go to customers and they have to say like, hey, beware, do something with it.
And it's not going to be different in the factory, but it's a major cloud upgrade. It's the same thing. So it's kind of like that.
What Happens to Juniors?
Benedikt Stemmildt
To go in a little bit of a different direction, but to add up to something that you mentioned earlier, adding the junior into the organization, I really like the idea of this backup person being a junior or like someone that would like a trainee or is not there yet maybe, or like because this is like a huge discussion at the moment.
Like, how do you deal with, I mean, we mentioned this factory builders are probably some system thinkers. So you need to become a system thinker somehow, right? Like this usually, I mean, it's not age, as you said, but it's more like, yeah, you need to have like experience for that. I don't know. There's this huge discussion. How what will happen with this? Now, how do we do this? How do we do it? What's your idea about that?
Patrick Debois
It's a common question. And goes in both directions. Seniors would often tell me, well, the juniors will not have an opportunity to learn. Right. So and then the juniors tell me, we can learn faster than anybody else. Big thanks to AI. So it goes both ways. But it's kind of the exposure, some guidance, some lessons. So that's why I'm saying age is not per se, kind of the case, but bringing them on to the journey kind of makes sense.
And it's not just like that. They learn about the technicalities of the system, but also in the business. Right. So that's why I'm saying you could be senior in engineering, but still junior in the domain or the problem that you're trying to solve there as well. So it wasn't like per se, an illusion, but I do think in general, the jury's out.
On the one hand, people always say there's a risk of automation, reducing the number of people. I'm saying it now. Cloud, DevOps, every time initially, there's that discussion. Like something gets automated and it gets reduced. And my parallel would be, if you would think about photography, you have a phone, it could take a picture. So there were less professional photographers on the one hand, but there was an explosion of YouTube like people, right?
So like where does the complexity kind of reduce the number of people and what new kind of roles does it bring? And kind of maybe there's something in the exploratory phase or something in the engineering phase. So so far, the waves that I've seen is that, yes, there's a reduction, but then there's a whole new world of new roles.
And that's maybe where the juniors then eventually thrive in. And there's also a certain level of reduction of complexity. Early days, CI/CD, things were really complex and messy and you had to know what you were doing. And then it molded into that almost like very accessible way of doing and kind of defining things. And it was easy to get somebody aboard on that new flow.
And then they experienced the new flow and then they could learn about the flow. So it's a different way of dealing with that complexity.
Benedikt Stemmildt
Yeah, I mean, CI/CD is a nice example, I think. I'm a huge fan of the GitLab CI/CD. I think it's like really brought us on the next level. And if people are now starting to use it, they don't need to have the experience from the journey it took to get there to just now understand and use it, right?
And I was one of the early person to jump on Node.js. And I feel if you compare the early days of Node.js to where we are now, there's also this, yeah, you can use it now without needing to know how we got there because it was super messy. And I think, I mean, Node is still super messy in a lot of places, right?
Patrick Debois
There's a privilege about witnessing kind of a new field emerging that you say, oh, that was explored. Oh, that was that direction. But at a certain moment, that becomes useless and not useless, right? Like useless in a way that you don't need it day to day. But when there is a certain thing, like why was the decision taking?
It's like, yeah, because of that. Like, you don't know because of that. And that's useful. But I also understand people saying, we're going to skip until this matures. And then we're going to jump in and they'll have a different kind of a learning experience to that. I guess we're both probably more hooked on like the new things and kind of understand the new things.
But it's kind of a, probably a personality thing or like why we're here now in this space and in this podcast.
Benedikt Stemmildt
So yeah, I mean, my take is also, as you say, from a personality standpoint, I really like to do that. But also, I feel that it's also very beneficial to already do these things now than to wait, right? Because, I mean, it wasn't that clear in the past, like a little bit clearer. But like with CI/CD pipelines and so on, it was still a huge amount of work to do to gain a little advantage.
But now with AI, I feel that even the little stuff that you can do now already gives you a huge advantage compared to persons that are waiting and not doing it at all, right?
Patrick Debois
Yeah, that makes sense. You have to put in work. It's one of those, how do we get started? It's a question I'll often get and it's like, yeah, just pick a project, start doing it. Like, you can read books, but it will go completely past you if you don't have skin in the game, which I tell them like, pick a project that you care about, like something you want to improve yourself and just dig in.
Is it the most efficient way? There's different ways of learning. If it's kind of well laid out, you could probably absorb a lot more information that you're stumbling into the field, but there's still a lot of waste of having to watch all the videos, all the blog posts, right? Like something will settle eventually and you kind of can forget all the rest.
But I think it's a little bit like the craftsman, right? Like a craftsman is about having a lot of tools on his belt that kind of knows when to pick the right one. And it doesn't mean you can't get away with a few tools if that is the thing you do, but that's what you pick along the journey.
And I've had it many times that I go into these rabbit holes and they don't lead to anything, but like, you know, five years later, it's like, whoa. Something like that.
Benedikt Stemmildt
I mean, to wrap this up a little bit also in the direction of risk again, it's like, or like, how do companies now actually implement factories in their day-to-day work and the learning of, I feel that it's often not really a technical learning thing. It's more like the organization around it needs to change and adapt. Like when we introduce Agile into these companies, this is not really about learning in the sense that I learned what Agile is.
I mean, it's a very short book. So it's not a lot to learn in terms of like theory. And there's a lot to learn in terms of like real experience and real doing the work. So what is your take on that in terms of like, how do you do that as for the factory now? How do you, how should companies start this?
Patrick Debois
So I started seeing the difference between people working on their agent locally on their laptop and they kind of tune it. They add their own stuff and their own flavor to it. Then they started checking things like shared context and their harness into more of the repo. And then that was almost exposed through the factory to other people as well.
So that kind of exposure was probably the biggest impact. Yes, there's going to from solo to kind of that team was useful in a way that, oh, we now stick all to the same standards and there was a kind of alignment going on. The fact that you can ask tickets now to a factory and say, please do this, please give comments, please help me here.
That kind of on the one hand opened it up. So there was like became democratized in the company of doing a change. So that was definitely a change. And then the fact that it could reuse all the knowledge that was there because we're adding to the factory and making it more robust. That means there's a value of not having to reinvent things.
And I recently did a talk and I'd say that there's all the metrics of deployment still make sense. Okay, robust and it needs to go out and tests and escape rates and all is valid and delivery, the speed delivery, like whatever you value in there. But I think the number of human touches going down in processes is definitely one because that creates that efficiency.
Like we're not having to have all the people be involved. And the second one is how much of this is being reused by other people. So if you expose the factory as a service and everybody can use the same knowledge, then whatever work you did in the factory is not just getting a multiplier on your laptop of I got X better, but the whole org got better at doing this.
And then I think those are like two metrics like reduce the number of human touch points. If you can with the right risk and all kind of the things in it. So don't do it YOLO, but then kind of how much is it reused across the business? That's my second metric to think about it. And I think factory kind of and the interface, whether that's a chat interface or it's looking at our tickets or something that embodies those two kind of like values or metrics, I must say, to kind of get improvements there.
Two Metrics for a Software Factory
Benedikt Stemmildt
The way you describe it, it also sounds to me like it's a very structured and deliberate process of doing that. And that the factory also helps in that because I see a lot of companies that are like, everybody is just using it on their laptops, like super chaotic. Everybody's just, I don't know, has their own setups and I need more tokens.
I need more tokens and so on. But that seems that it needs like a program around it that says, okay, we want to build a factory, we want to get there. And what do you think?
Patrick Debois
There's stories that talk about what almost like what I would call dev factories. Indeed, they run on their laptops and they got better and the harnesses are kind of taking care of that. But what I usually would look for, the factories are the ones that are being in the, not just in the cloud because you could put your developer thing also in the cloud, but the fact that it's being centralized and being reused across.
And I think that's what the Ubers, the Ramps, kind of all these centralized engineering groups are doing in their companies. And I think that's where the benefits get multiplied. I'm not saying a dev factory is bad or anything, but it's not like it will get you up to a level, but if your colleague can't use it, the impact is limited.
So that's why I say like it's from solo to shared to multiplayer is sometime a phrase that you hear and that multiplayer is what a factory exposed as a self-servicing to the whole org could promise way more than having it solo in your on your environment. And I even started to think about like a lot of people talk about I can install skills and I started to think like, aren't we deploying skills?
Like, shouldn't we kind of like make them centrally and then reusable like kind of like we're doing that in a way because our coding agents kind of like allow us to pick that up centrally, but it's almost like I want to centralize and make it available in that sort of servicing mode that everybody can use my skill instantly with the right credentials, with the right environment, with the right things that are hooked up and going from there.
Benedikt Stemmildt
So yeah, it's interesting. It's interesting to see how I mean, the field still emerges, right? And now we have this idea of plugins and maybe there's going to become like more of an installer for this. And so we haven't really figured that out yet, right?
Patrick Debois
But it's thinking patterns, right? Okay, we got skills. They became plugins. Maybe the next thing is that will have loops that you can install in your factory, which seems to be the case. Like a lot of people talk about loops. You install your code review is a loop. Your spec to code is a loop. So that kind of will become its new entity that you work on.
And maybe then again, it becomes like interchangeable. Maybe we'll have to think about interchangeable factory configs. You know, eventually we'll get there compatibility one factory to the other. So you can like, it's, it's extravagant. We're not there yet, but that's that's that's almost the cool stuff that you can start thinking about.
Benedikt Stemmildt
So yeah, so as we, as we near near the end of our time, I have one last question. So what would you advise people to do now? Like if they hear the episode, I don't know, in a few weeks. What is the thing that you think should people should think about or should start with? I know it's not one.
Patrick Debois
Definitely depends on your maturity, of course. But if you're just starting, just start playing with it. Like kind of the more experience you can do it and do it with, like I said, with something that you care about. And if you get more experienced, think about how I would advise, like, please think beyond kind of just the coding loop and go into the company.
And I often say that senior engineers, they don't just are about like building better code. They go into the business, they think about architecture, they think about cost, they think about other values as well. So start exploring those. And maybe in that hand is like figure out like, where do you want to be? You want to be more in the platform team, you want to be more in the exploratory spec team and kind of try shaping your career there.
Of course, there's going to be new acronyms for deployed engineer, AI product engineer. And we're going to have a name for all of them. Kind of like choose your own path and kind of like, yeah, learn more, I would say. That's the motto right now.
Benedikt Stemmildt
Thank you so much, Patrick. I really enjoyed this episode. I think we had a lot of insights here. Hope we see you again. I think you are at are you at code.talks this year? Or I'm not sure if you are.
Patrick Debois
Unfortunately, I'm not. I'm curating another conference.
Benedikt Stemmildt
Okay, okay.
Patrick Debois
AI DevCon in New York.
Benedikt Stemmildt
Oh, wow.
Patrick Debois
2 to 4 November and I'm curating the program and the theme is factories. So hey, yeah, you might as well come.
Benedikt Stemmildt
Oh, maybe I'm going to go there. I was never in New York. Yeah, that would be awesome. Thank you so much. And yeah, have a great time and see you again.
Patrick Debois
Thanks for having me. I really appreciate it.
Benedikt Stemmildt
Thank you so much for listening. That was another episode of Rethink Engineering. Patrick, thank you as well. I really enjoyed having you here on the show. I hope I'm getting better at this. Let's see what the comments say. It was a pleasure taking the factory part. If you want to learn more about this, stay with us. Follow us.
Or join Patrick in Brooklyn at the AI DevCon. No, 2nd to 4th of November. You will learn a lot more about software factories there. He also curates ainativedev.io. So hop over there. You will find the links in the description. And now it has finally come to the point where we are going to have an episode with Deejay.
I'm really glad that he's joining us and we are going to find out what he's currently doing if he's going to join us more often. And yeah, I think that's it. That was Rethink Engineering with hackers&wizards and re:cinq. See you next time. Bye-bye.