Building, operating and governing real-time data products in an Agentic Software Factory requires a new approach
Software workloads that process large volumes of real-time data are becoming common. Decades working in this domain has taught me that building and operating reliable and maintainable real-time data products requires permissive access to the context of the environment. This article explains how to approach agentic engineering and apply it when building real-time data products inside an agentic software factory.
At the start of the year, Andrej Karpathy coined "agentic engineering" as the disciplined counterpart to “vibe coding”: professional engineers using AI agents to accelerate their work while maintaining human oversight, quality and rigour.
I have come to define agentic engineering as shipping and maintaining software products by treating agents as teammates, each with their own responsibilities (includes the infrastructure and configuration in addition to the code). Typically this involves multi-agent orchestration, shared memory and access controls, with a human to oversee it all.
By now, most of the engineering blogosphere have shared their take. The consensus is clear: agentic engineering works. Having detailed product specifications describing product features, behaviour and tests are the biggest differentiator. Senior engineers benefit disproportionately due to their judgment and deep understanding of fundamentals. The writing-code-is-cheap era arrived (currently subsidised and possibly the cheapest it will ever be) and human oversight became non-negotiable, especially if your work is in regulated industries.
The discipline even keeps renaming its own frontier: prompt engineering gave way to context engineering, then to harness engineering (treating the scaffolding around the model as the thing you actually engineer and evolve) and now to loop engineering (replacing prompts with loops that prompt the agents for you).
The community also agrees that context is everything. The context problem, in plain terms, is this: agents are only as good as what they can see, and what most agents see is a thin, stale slice of how the system used to look. They largely miss recent deployments, schema changes, changes in ownership and the deltas leading to the actual state of the data they're meant to operate on. Everyone claims it's a context problem, but few are discussing the context solution.
The supply chain for context
Every source on agentic engineering agrees on one thing: engineering agents are only as good as the context they receive. So where does that context come from? We know that all great software is built by engineers that have built up years of domain expertise. So how do we convey that same tribal knowledge in the right measure to agents?
Look at what the industry recommends: design documents, code repositories, wiki pages, knowledge bases, vector stores, AGENTS.md, SKILL.md, other Markdown files.
These are all important but they are static artefacts. They tell agents how the world looked when someone last wrote something down. In an organisation where business decisions are made several times an hour, deployments happen hourly and incidents fire unpredictably, the interval between when someone hits a save button and “right now” is where agents can make outdated decisions.
The solutions are centred on code, but data context is critically important
There's a second blind spot. The agentic engineering discourse is framed around coding agents for software delivery: writing code faster, reviewing pull requests, refactoring, debugging. Patterns are focused on test-driven development for agents, code review, Mean Time to Resolution (MTTR) and PR throughput. This all has value, but it's narrow.
Agentic engineering should be about adopting the mindset around the full lifecycle of software delivery, not just the coding itself (any experienced engineer will tell you it was never really about the code). We’re increasingly typing fewer lines of code, which shifts the value towards the planning and the work around the code (check out this video, where I explain this with a Formula 1 analogy).
To play this video, accept functional cookies in your privacy settings.
YouTube video
Here, I’m referring to DevOps and Infrastructure/Platform Engineers who identify as engineers but don’t write code day to day. Their work in making sure services are available, healthy, governed and that data flows to the right places is unarguably important. Making it just about the code alienates the engineers that practice agentic engineering who work in operations, infrastructure, analytics and other teams on the critical path.
As reasoning LLMs improve, so will our understanding of building better agent harnesses and meta-harnesses. In a year’s time the process of spec-to-production code could become a largely solved problem, with engineers spending less than 10% of their time looking at code. That reopens questions most teams thought were settled. Are engineering, design and product still separate roles? Does deeper technical skill automatically mean more seniority or will we lean more towards broad skills? Do you need a team at all, or one person per project? Is the trade off between features and tech debt even relevant once patching and rebuilding becomes effortless? None of this is an argument for fewer engineers. The demand for strong product builders and operators who are able to maintain production goes up, not down.
A huge portion of what engineering teams actually do is build infrastructure (pipelines, connectors, data processing applications, schemas) and operate it (govern, investigate data quality issues, manage schema evolution, troubleshoot connector failures, monitor consumer lag, handle rebalancing). We expend a lot of effort in planning how we create software products, but considerably less effort in planning how those products and the infrastructure they run on will be maintained. All this whilst knowing that software tends to exist and be used for longer than we expect before it’s eventually deprecated and sunsetted.
Extending this thinking to the AI layer, agents that write higher quality code are useful. Agents that build, operate, govern and evolve data products and infrastructure using live context from their environment are transformative.
The teams I’ve spoken to this year are aspiring to build autonomous self-improving agents. The journey to this starts with agents having access to runtime context (live events from services, recent deployments), decision traces (architectural decision records, history of what happened before and why) as well as the outcomes (what didn't work and what worked). This ensures self-discovery and reduces the need for context to be provided up front.
An agent cannot improve itself if it makes a change to a system but is then unable to see the system’s updated state or the data that it subsequently generates. This thinking has produced the new approaches I mentioned earlier: harness engineering, where guides steer the agent before it acts and feedback controls catch its mistakes, and loop engineering, where those controls run in a loop over and over until the desired outcome is reached.
The Agentic Software Factory
The shift this points towards is bigger than faster code or faster reviews. Organisations are moving from individuals using AI assistants towards an operating model where every technical role oversees a fleet of long-running, highly autonomous agents that carry out work between human checkpoints. This is the agentic software factory.
The factory metaphor makes the division of labour visible. A modern engineering organisation already has a clear role taxonomy: architects, engineers, reviewers, site reliability engineers, security engineers, platform engineers. Each of those humans now has an agent counterpart:
- Architect agents that hold the long-term plan, decompose features into specifications and schemas, evaluate trade-offs and keep system design coherent across services.
- Engineer agents that implement those specifications, write the code/config, build pipelines, manage schema evolution, monitor data quality, write tests and produce the pull requests.
- Reviewer agents that critique the work against standards, catch regressions, suggest refactors and keep style and architectural consistency.
- SRE agents that handle CI/CD, watch deployments, monitor metrics, respond to alerts, run incident triage and progressively self-heal common failure modes.
- Security agents that audit dependencies, validate access patterns, gate releases that are against policy, scan for vulnerabilities and patch them.
- Platform engineering agents that put IAM and data policies in place, audit access, ensure governance, classify and document assets.
The human in each role moves from doing the work to designing the system that does the work. Their work is to ensure that what lives in their head becomes available to the agent too. They write the rules, set the policies, grant the permissions, review the consequential decisions and let their agents handle the long tail of execution. The agents themselves run for hours or days at a time, holding state across sessions, with sandboxing, permission boundaries and audit trails that make long-running autonomy safe.
What ties these roles together is the same continuous delivery pipeline that has shaped software for the past decade. The agentic software factory is the next turn of that wheel. The software development lifecycle has always been a loop: plan, build, test, release, observe, learn, improve. Agents have collapsed how long each turn of that loop takes. Where a team has two-week sprints, an architect agent could run a series of twenty-minute design experiments. Where a product engineer might have prototyped one approach and committed to it, a fleet of engineer agents can prototype several approaches in parallel, run them against the same evaluator and let the winning candidate be promoted. Best-of-N selection, generator-evaluator loops and pattern reuse across projects turn iteration count into a competitive variable.
A few characteristics define a factory worth running:
- Parallel exploration with selection. Multiple agents make proposals, an evaluator (an agent or a human) selects one and the winning candidate is promoted. The factory treats trying several things as the default.
- Persistent state and durable execution. Multi-day workflows survive context loss. An agent that crashes mid-task resumes where it left off rather than starting from scratch. This is crash resilience applied to agents.
- Compound learning across projects. Each delivered project leaves behind patterns, traces, outcomes and important notes that the next project draws on. The factory gets faster because it has done similar work before.
- Governance built in. DORA metrics, change failure rates and lead time are emitted as first-class signals. Compliance and policy enforcement as standard.
- Human checkpoints at consequential moments. Approvals, design reviews, production releases and security exceptions stay with the human in the role. Everything else delegates.
Cost as a first class metric
The economics of running such a factory matter as much as the architecture and there is increasing pressure on teams to find ROI. Agentic engineering is useful and, right now, expensive. Frontier model API calls, parallel attempts, multi-hour autonomous runs, evaluator passes, retry cycles and the "try ten approaches and keep the best one" patterns all add up. Even what would seem like straight forward replacement of multi-step human handoffs with autonomous agentic workflows can be costly with unclear ROI.
A growing concern across the industry is that the way many teams are running agents today (loosely scoped tasks, vague specs, generic context dumps, repeated tries against an unclear goal, not switching to different sized models based on task) burns budget for very little marginal output. Model providers may eventually subsidise less, token prices may eventually rise and open-weight models running on owned GPUs in dedicated servers may eventually take more of the load, but those are multi-year structural shifts with significant capital expenditure. What every team can address today is whether each agent run is actually doing useful work.
Tracking agentic metrics belong alongside DORA metrics on the factory's instrument panel: token spend per merged PR, cost per completed task, cost per resolved incident, cost per evaluator iteration, ratio of accepted to discarded attempts, average iterations to acceptance per agent class, total time saved per workflow. These are the numbers that tell engineering leaders whether the factory is getting more efficient as it scales.
Data Governance is at the heart of Agentic Engineering
The conversations I'm having with engineering and operations teams have shifted. Six months ago, the question was "how do we give our agents access to tools?" (thank you, MCP). Now the question is “how do we trust our agents with production data?”, alluding to the identity, access management and audit trails that need to be put in place.
Data warehouses, lakehouses and transactional databases all have benefited from strong governance posture for years. Their data is refined, classified and monitored for quality. And so agents today could reach them with some level of confidence. The streaming data layer is largely missing from conversations because its governance posture tends to be less mature, yet it is the one carrying the most high-fidelity current view of what is happening in the business.
There is a broad desire for businesses to modernise their core systems to real-time (ie. connected to streams in Apache Kafka). Many businesses have 1000s of legacy systems waiting to be re-architected to streams, which may take another 10 or 15 years to complete. These are projects that would benefit most from adapting to an agentic software factory to lower the cognitive complexity held on the shoulders of engineers today.
The answer for many organisations and all our customers, who deploy different flavours of Apache Kafka, is already clear. The infrastructure that was built over the past decade to power event-driven architecture, real-time analytics and operational data pipelines (foundational for the large majority of Fortune 500 companies) is the exact setup that agents need to operate with fresh context. And it needs to be done within the governance boundaries that the organisation requires: role-based access, time-restricted access, data masking for PII, audit trails of every agent action.
Having a weak governance posture makes it difficult to give engineering agents access to build in the first place. For example, if data is not tagged and classified, an insecure engineering agent could mistakenly be exposed to PII data or take a decision to move data to another non-compliant cloud, not to mention the data exfiltration risks (the lethal trifecta for AI agents). A trained engineer would have this knowledge and be held responsible for these actions.
Teams that focus on governance will be those best equipped to connect their data to agentic systems; be it agents to build/operate an application or business agents to transact on the data. Their agents won't just write better code. They'll build infrastructure that fits the current state of the world, operate it with awareness of what's happening right now and evolve it with precision about what will be affected. They will be able to create agents specialised in improving governance, such as data classification agents, which will compound high quality context and compound their competitive advantage. They will be able to move quickly and with safety into the next iteration of the AI era.
