Erik Caldwell
Connect on LinkedIn
Menu

The Topic Catalog · 4

AI-Native Software Engineering

Last updated · 21 min read

4.1 The Software Engineering Continuum: Autocomplete → Coding Assistant → Coding Agent → Agent-Directed Engineering → Autonomous Software Factory

Priority: Must Understand

Executive Definition: AI's role in software engineering has moved through distinct stages, each requiring a different operating model, not just a better tool. Autocomplete (predicts the next few lines) gave way to coding assistants (chat-based help inside the IDE), then to coding agents (given a task, they read the repo, write code, run tests, and iterate with limited supervision), then to agent-directed engineering (a developer directs multiple agents against specifications rather than writing code line-by-line), and finally toward the autonomous "software factory" (issue-to-PR pipelines with minimal human touch). Most enterprises today sit between stage 2 and stage 3; stages 4-5 are early and unevenly proven.

Why It Matters: Each stage changes what "developer productivity" means, where risk enters the system, and what skills matter: conflating stages leads to buying the wrong tools, setting the wrong KPIs, or assuming maturity the organization doesn't have. DORA's 2025 research frames AI as an "amplifier" of existing organizational practice, not a stage-skipping shortcut: teams with weak fundamentals get faster and worse simultaneously (dora.dev, 2025). Vendor marketing routinely describes stage-4/5 capability while shipping stage-2/3 tools, so leaders need the vocabulary to tell the difference.

What I Need to Understand:

  • The five stages are not merely "more automation": each one shifts who verifies correctness, when review happens, and what unit of work the human hands off (a line, a function, a PR, a ticket, a backlog).
  • Autonomy is task- and codebase-dependent, not organization-wide: a team can be at "coding agent" maturity on a well-tested microservice and stuck at "assistant" on a legacy monolith with no test coverage: asking "where are we on the continuum" without qualifying by system is meaningless.
  • The "autonomous software factory" (issue in, PR out, no human in the loop) is still largely aspirational at enterprise scale in 2026; treat vendor claims of full autonomy as marketing until you see evidence in a comparable codebase (see Coding Agents topic).
  • Adoption of the tools (assistants, agents) is near-universal: DORA reports ~90% of respondents use AI at work (dora.dev, 2025) and GitHub reports roughly 80% of new developers use Copilot within their first week (GitHub Octoverse 2025): but tool adoption is a poor proxy for which stage of the continuum a team actually operates at.
  • Moving up the continuum requires investment unrelated to the AI tool itself: test automation, modular architecture, clear specifications, and CI/CD maturity: DORA explicitly ties AI value realization to platform engineering and loosely coupled architecture (dora.dev, 2025).

Questions I Should Be Able to Ask My Team:

  1. For our top three product lines, which stage of the continuum are we actually operating at, and what's the evidence (not the tool license)?
  2. What's blocking us from moving from "coding agent" to "agent-directed engineering" on this system: is it test coverage, architecture, specification quality, or trust?
  3. When a vendor claims "autonomous" or "agentic" capability, what specifically is still human-reviewed, and what's the failure mode when the agent is wrong?

Technologies / Standards / Companies to Know: GitHub Copilot (assistant → agent), Anthropic Claude Code, OpenAI Codex, Cursor, Google Jules, Devin (Cognition): positioned as more autonomous end of spectrum; Factory.ai and similar "software factory" platforms represent the more speculative end.

Recommended Learning:

Time Investment: 1 hour

4.2 Coding Agents & Autonomous Coding Agents

Priority: Must Understand

Executive Definition: A coding agent is given a task description rather than a single prompt, and it independently reads the codebase, plans, writes code, executes commands (build, test, lint), observes the results, and iterates: closing its own feedback loop instead of returning one suggestion for a human to accept or reject. This is qualitatively different from autocomplete or chat assistance (earlier stages of the continuum above). "Autonomous" is a spectrum of how much of that loop runs without a human checkpoint, from agents that pause for approval at each tool call to agents that run a whole task end-to-end and only surface a finished pull request.

Why It Matters: Agents generate large volumes of working code fast, but they also generate large volumes of wrong code fast, and the org's ability to safely absorb that output (not the agent's raw capability) is usually the binding constraint. GitHub reports over 1 million pull requests created by its Copilot coding agent between May and September 2025 (GitHub Octoverse 2025), showing this is now production-scale, not experimental. At the same time, a randomized controlled trial by Model Evaluation & Threat Research (METR) found experienced open-source developers took 19% longer on real tasks when using current AI tools, despite believing afterward they had been roughly 20% faster: a direct warning against trusting self-reported productivity data (METR, July 2025).

What I Need to Understand:

  • Agent effectiveness depends heavily on task shape: well-specified, well-tested, well-scoped tasks in familiar codebases succeed far more often than ambiguous, cross-cutting, or poorly tested ones: GitHub's data shows agents adopted fastest in established, well-instrumented repositories (GitHub Octoverse 2025).
  • The agent's "autonomy level" is configurable: how many tool calls it can make unsupervised, whether it can run destructive commands, whether it can push to protected branches, and that configuration is a governance decision, not just an engineering one.
  • Perceived productivity and measured productivity diverge; the METR finding that developers felt faster while being measurably slower means self-reported velocity metrics from teams adopting agents should be treated skeptically absent objective data (METR, 2025).
  • Agent output volume strains downstream processes (code review, CI, incident response) that were sized for human-paced change: see AI Code Review and CI/CD topics below.
  • "Autonomous coding agent" claims should be evaluated against a specific, checkable task class, not treated as a general capability rating.

Questions I Should Be Able to Ask My Team:

  1. On the tasks where we've deployed coding agents, what's our actual completion rate without human rework, and how do we measure that (not self-report)?
  2. What can an agent do unsupervised in our environment today (can it merge, deploy, or touch production credentials) and who approved that boundary?
  3. Have we independently measured cycle time and defect rate before/after agent adoption, or are we relying on developer sentiment surveys?

Technologies / Standards / Companies to Know: GitHub Copilot coding agent, Claude Code (Anthropic), OpenAI Codex, Cursor Agent, Google Jules, Devin (Cognition), OpenHands (open source).

Recommended Learning:

Time Investment: 1 hour

4.3 Specification-Driven Development & Context Engineering for Code

Priority: Must Understand

Executive Definition: As agents take on more of the implementation, the developer's job shifts toward writing precise specifications and curating the context an agent needs to act correctly: repository-level instruction files, architecture documentation written for machine consumption, coding standards, and test contracts. AGENTS.md is the emerging open convention for this: a standard file format where a repository states build commands, conventions, and constraints that any coding agent (not just one vendor's) can read. Anthropic uses the parallel term "context engineering": deliberately deciding what information an agent sees, in what form, at what point in its task.

Why It Matters: Agents are only as good as the context and specification they're given; vague tickets and undocumented tribal knowledge produce plausible-looking but wrong code at agent speed, which is worse than the same failure mode at human speed because it's harder to catch. AGENTS.md has become a real cross-vendor standard in under two years: over 60,000 open-source repositories use it, and it's now stewarded by the Linux Foundation's Agentic AI Foundation with support from OpenAI, Google, GitHub, Cursor, and others (agents.md, 2026): meaning "does the repo have an AGENTS.md and is it accurate" is now a legitimate engineering-maturity signal you can ask about directly.

What I Need to Understand:

  • This is a documentation and specification discipline, not a new AI capability: it requires the same rigor as writing a good design doc, and most orgs' documentation is not currently good enough to hand to an agent unsupervised.
  • AGENTS.md (and equivalents like CLAUDE.md) typically encode: build/test commands, code style, directory conventions, "do not touch" areas, and security constraints: stale or wrong instructions here actively mislead agents rather than just failing to help.
  • "Context engineering" is broader than one file: it includes what's in the agent's working memory, which tools it can call, and how much of the codebase it's shown, and getting this wrong is a common cause of agents producing code that's locally sensible but architecturally wrong.
  • Specification quality becomes a bottleneck as agent capability increases: this is the emerging discipline sometimes called "spec-driven development," and it changes what "senior engineer" work looks like (writing specs and reviewing implementations vs. writing code).
  • This is an investment with compounding returns: good repo-level context pays off across every future agent task in that codebase, unlike a one-off prompt.

Questions I Should Be Able to Ask My Team:

  1. Do our repositories have agent instruction files (AGENTS.md, CLAUDE.md, or equivalent), who owns keeping them accurate, and when were they last verified against reality?
  2. What happens when an agent's context is wrong or stale: do we have any detection for "agent acted on outdated instructions," or would we only find out from a bad PR?
  3. Whose job is it now to write specifications precise enough for an agent to implement correctly, and is that a skill we're deliberately building in the team?

Technologies / Standards / Companies to Know: AGENTS.md (Linux Foundation Agentic AI Foundation), CLAUDE.md (Anthropic convention), .cursorrules (Cursor), Model Context Protocol (MCP) for tool/data access.

Recommended Learning:

Time Investment: 1 hour

4.4 AI Code Review, AI-Generated Testing & AI Debugging

Priority: Must Understand

Executive Definition: Three related but distinct capabilities: AI code review (a model reviews a diff and flags issues before or alongside human reviewers), AI-generated testing (agents write unit/integration tests, sometimes from specifications or from the code itself), and AI debugging (agents reproduce a bug, form a hypothesis, and propose or apply a fix). Each is now a standard offering from major platforms: GitHub reports 72.6% of developers using Copilot code review found it improved their effectiveness (GitHub Octoverse 2025): but each also introduces a specific new failure mode rather than simply removing manual effort.

Why It Matters: Agents now produce PR volume that outpaces human review capacity; DORA's 2025 research names this directly as the "velocity paradox": time saved writing code is consumed by verification, and reviewers face an asymmetric cognitive burden reviewing large AI-generated changesets (dora.dev, 2025). AI-generated tests can pass while testing the wrong thing (validating the implementation's behavior rather than the intended behavior), and AI debugging can "fix" a symptom without addressing the root cause: both failure modes look like success in a green CI pipeline.

What I Need to Understand:

  • AI code review works best as a first-pass filter (style, obvious bugs, missing tests) that reduces human reviewer load, not as a replacement for a human's judgment on architecture, intent, or business logic: treat it as raising the floor, not replacing the ceiling.
  • AI-generated tests are systematically vulnerable to a specific failure: if the same or a related model writes both the code and its tests, the tests can encode the bugs of the implementation rather than catching them: test provenance and independent review of test intent (not just coverage percentage) matters more than before.
  • DORA's own recommendation is to shift AI feedback earlier (to the author while writing, not just the reviewer after the fact) and to deploy context-aware review agents for standards enforcement while reserving human review for judgment calls (dora.dev, 2025).
  • GitClear's analysis of 211 million lines of code (2020-2024, including Google/Microsoft/Meta repos) found copy-pasted code roughly doubled as a share of changes (8.3% to 12.3%) while refactoring's share fell from 25% to under 10% (the first time duplicated code exceeded refactored code in this dataset (GitClear, 2025)) a concrete quality-erosion signal to watch for, independent of any vendor's review tool.
  • Coverage metrics (percent of lines covered) become less meaningful when tests are AI-generated at volume; the question shifts to whether tests encode the actual specification/intent, which requires spot-checking, not just running the suite.

Questions I Should Be Able to Ask My Team:

  1. Are our AI-generated tests being reviewed for whether they check the right behavior, or only whether they pass and raise coverage numbers?
  2. What's our reviewer bottleneck right now: has PR volume grown faster than review capacity, and what specifically changed in our review process to compensate?
  3. When AI debugging tools "fix" an issue, do we have any process that distinguishes a root-cause fix from a symptom patch, before it ships?

Technologies / Standards / Companies to Know: GitHub Copilot code review, CodeRabbit, Greptile, Graphite, Anthropic Claude (code review and debugging use), Sentry/observability-linked debugging agents.

Recommended Learning:

Time Investment: 2-3 hours

4.5 AI-Assisted Refactoring, Modernization & Technical Debt Reduction

Priority: Monitor

Executive Definition: Using AI (assistant, agent, or a purpose-built migration tool) to modernize legacy code (language/framework upgrades, dependency updates, dead code removal, and structural refactoring) at a scale and speed manual effort couldn't match. This is one of the better-evidenced enterprise use cases: it's a bounded, verifiable task (behavior should be unchanged) rather than open-ended feature creation, which makes correctness easier to check automatically.

Why It Matters: This is where large, credible organizations report the clearest ROI. Google's engineering research group published a peer-reviewed account of an LLM-assisted migration tool used across 39 internal migration projects over twelve months: roughly 74% of the resulting code changes originated from the model, and developers reported roughly 50% time savings versus prior manual migrations, but with developers remaining firmly in the loop validating and guiding every change, not a hands-off pipeline (arXiv 2504.09691, Google, Foundations of Software Engineering (FSE) 2025). Amazon has separately publicized an AI-assisted Java modernization effort it says saved roughly 4,500 developer-years, a figure that should be treated as a vendor/leadership claim rather than an independently peer-reviewed result, but is directionally consistent with the Google data.

What I Need to Understand:

  • The strongest evidence for AI-assisted modernization is in well-scoped, mechanically-checkable migrations (language version upgrades, API replacements, dependency bumps) with automated tests as a safety net: not in ambiguous "clean up this legacy system" mandates.
  • Google's own reported model was developer-led automation, not autonomous refactoring: an algorithm found candidate locations, an LLM proposed changes, and developers validated each one: this is squarely "coding agent," not "software factory."
  • Ironically, evidence from AI Code Review/Testing (GitClear) suggests day-to-day AI-assisted coding is currently reducing the proportion of refactoring work being done, even as purpose-built migration tooling shows strong results: the gain seems concentrated in dedicated modernization initiatives, not organic day-to-day cleanup.
  • Test coverage is the precondition, not a side benefit: a codebase without strong automated tests cannot safely use AI for large-scale refactoring, because there's no automated way to confirm behavior didn't change.
  • Vendor-published productivity claims (developer-years saved, percentage faster) generally lack independent verification or methodology detail: treat them as directional, and ask what was actually measured.

Questions I Should Be Able to Ask My Team:

  1. Which parts of our tech debt backlog are mechanically-checkable migrations (good AI-refactoring candidates) versus ambiguous redesigns (poor candidates)?
  2. What test coverage do we have on the systems we're proposing to AI-refactor, and is it sufficient to catch a behavioral regression?
  3. If we adopt a migration tool or agent for this, what's the actual acceptance/validation process: who checks each change, and how much is genuinely automated versus human-reviewed?

Technologies / Standards / Companies to Know: Amazon Q Developer (transformation agents), GitHub Copilot (refactoring workflows), Google's internal LLM migration tooling (published research, not externally available), Anthropic Claude Code, OpenAI Codex: for large-scale migration, purpose-built tooling generally outperforms generic chat assistants.

Recommended Learning:

Time Investment: 1 hour

4.6 Parallel Coding Agents, Developer Supervision Ratios & Software Factories

Priority: Must Understand

Executive Definition: Rather than one developer working with one agent, some organizations are experimenting with one developer directing several agents working in parallel on different tasks: shifting the developer's role from implementer to supervisor/orchestrator. "Software factory" is the industry term for the more ambitious end-state: pipelines where issues flow to agents and PRs flow out with minimal human involvement. This is the least mature part of the continuum above: real vendor products exist, but independent, large-scale evidence of safe supervision ratios or factory-level throughput gains is thin as of late 2026.

Why It Matters: This is where the biggest headcount and org-design decisions get made prematurely on vendor claims rather than evidence: "one developer can now supervise N agents" is a governance and quality-assurance question, not a capability question, and the honest answer depends entirely on task type, codebase maturity, and review capacity, none of which vendors control for. DORA's "velocity paradox" already shows single-agent-per-developer review burden growing; multiplying the number of agents per developer without multiplying review/verification capacity increases risk faster than it increases output.

What I Need to Understand:

  • There is no established, independently-validated "supervision ratio" (agents per developer): treat any specific number a vendor gives you as a marketing claim until you've seen it hold on a comparable codebase with your own quality bar.
  • The binding constraint on parallel agents is almost never agent capability: it's the human's ability to review, integrate, and reconcile several concurrent streams of change without losing context on any of them.
  • "Software factory" claims should be checked against the same rigor as any other late-stage continuum claim: what's the actual task class, what's the failure rate, and what human checkpoint exists before code reaches production.
  • Parallelizing agents multiplies the downstream load on code review, CI, and architecture consistency: an org that hasn't solved single-agent review bottlenecks should not expect parallel agents to solve anything.
  • This area is moving fast and current best practice will likely look dated within a year: treat specific product claims as time-stamped, and revisit the evidence base periodically rather than trusting a one-time briefing.

Questions I Should Be Able to Ask My Team:

  1. If we run multiple agents per developer, what specifically increases to keep quality constant (review capacity, test automation, or something else) and have we actually provisioned that?
  2. What's the evidence behind any specific "agents per developer" ratio a vendor or consultant has proposed to us: has it been demonstrated at our scale, or only in their marketing?
  3. In a parallel-agent workflow, what happens when two agents produce conflicting changes to the same system, and who resolves that: is there a real process, or are we assuming it away?

Technologies / Standards / Companies to Know: Factory.ai, Devin (Cognition), GitHub Copilot workspace/multi-agent features, Anthropic's multi-agent orchestration patterns (published research, not a packaged product): this space consolidates quickly; expect vendor positioning to shift.

Recommended Learning:

Time Investment: 1 hour

4.7 CI/CD, Quality Gates & Architecture Validation in an Agentic Development Model

Priority: Must Understand

Executive Definition: Traditional CI/CD (build, test, lint, deploy pipelines) was designed for human-paced change volume and human-authored code. Agentic development requires the same pipelines to do more: verify not just that code compiles and passes tests, but that it conforms to architectural constraints an agent has no inherent reason to respect (module boundaries, dependency rules, security policies). "Architecture fitness functions" (automated, executable checks of architectural properties) are the emerging mechanism for this, extending a pre-AI concept (from evolutionary architecture practice) to constrain agent-generated code specifically.

Why It Matters: DORA is explicit that AI adoption has a negative relationship with software delivery stability, and that the reason is organizational, not technical: without strong automated testing, version control discipline, and fast feedback loops, increased change volume simply exposes existing weaknesses faster (dora.dev, 2025). CI/CD and architecture validation are the control systems that determine whether higher agent throughput translates into more working software or more incidents: this is the single highest-leverage investment for making the rest of this domain's topics safe.

What I Need to Understand:

  • DORA's core 2025 finding on stability is not "AI breaks things": it's that AI removes the natural rate-limiting effect of human typing speed, so whatever gaps exist in your test automation and control systems get hit harder and faster (dora.dev, 2025).
  • Quality gates need to check things beyond test pass/fail for agent-generated code specifically: dependency policy, architectural layering, security scanning, and license compliance: because an agent will happily generate code that passes tests while violating conventions no test encodes.
  • "Fitness functions" (automated architectural checks, e.g., via ArchUnit-style tooling) are how teams encode architectural intent in a form agents and pipelines can actually enforce, rather than relying on a human reviewer to notice a violation.
  • Small, frequent changes remain the safest unit of work even for agents: DORA specifically recommends enforcing small batch sizes as a countermeasure to unwieldy AI-generated changesets (dora.dev, 2025): "let the agent work for a long time and produce one huge PR" is a known anti-pattern.
  • Platform engineering maturity is a leading indicator DORA ties directly to AI value realization (90% of high performers had strong platform engineering practices): CI/CD and quality-gate investment is not a side cost of AI adoption, it's the precondition for it paying off (dora.dev, 2025).

Questions I Should Be Able to Ask My Team:

  1. What automated checks run against agent-generated PRs beyond unit tests (architectural rules, security scans, dependency policy) and what's not yet covered?
  2. Has our delivery stability (change failure rate, Mean Time to Restore (MTTR)) moved since agent adoption, and do we have the DORA metrics instrumented to actually know?
  3. Do we enforce small batch sizes for agent-generated changes, or are agents producing large changesets that overwhelm review and CI?

Technologies / Standards / Companies to Know: ArchUnit and similar fitness-function tooling, standard CI/CD platforms (GitHub Actions, GitLab CI, Jenkins) extended with agent-aware checks, DORA metrics (deployment frequency, lead time, change failure rate, MTTR) as the measurement backbone.

Recommended Learning:

Time Investment: 2-3 hours

4.8 AI-Generated Code Risk & Secure Development Practices

Priority: Must Understand

Executive Definition: AI-generated code carries security and quality risks distinct from human-authored code: models can introduce vulnerabilities even when explicitly asked to fix them, code volume can outpace security review capacity, and "vibe coding" (accepting AI output with minimal scrutiny) can bypass an organization's existing secure-development controls entirely. This topic is about the governance and technical controls needed so that increased AI-driven throughput doesn't translate into increased breach surface.

Why It Matters: DORA reports that even with 90% AI adoption, 30% of developers report little or no trust in AI-generated code (dora.dev, 2025): a meaningful trust gap at the point of production use. Peer-reviewed research on iterative AI code generation found vulnerabilities increased through refinement rather than decreased: a study of 400 code samples across 10 iterations each found critical vulnerabilities rose roughly 37.6% after five iterations, and notably, prompts explicitly requesting security improvements still introduced new vulnerabilities alongside the fixes (Institute of Electrical and Electronics Engineers (IEEE) International Symposium on Technology and Society (ISTAS) 2025 / arXiv 2506.11022). This directly contradicts the assumption that "just have the agent iterate more" improves safety.

What I Need to Understand:

  • Security review cannot be an afterthought bolted onto an accelerated pipeline: the research shows iteration itself is not self-correcting for security, so static analysis, dependency scanning, and human security review need to run on every iteration, not just the final output.
  • "Vibe coding" (a term popularized by Andrej Karpathy for accepting AI-generated code with minimal scrutiny) is a real and named enterprise governance concern; the Cloud Security Alliance and others have specifically flagged the gap between how individuals use these tools informally and what enterprise governance requires.
  • Prompt intent doesn't reliably map to secure output: security-focused prompts in the referenced study still produced vulnerabilities (notably cryptographic errors), meaning "we told the agent to be secure" is not a control, it's a suggestion.
  • GitClear's data on rising code duplication is also a security-relevant signal: duplicated code multiplies the surface area for a given vulnerability class rather than fixing it once in a shared location.
  • The trust gap DORA measured (30% low/no trust) is itself a risk signal worth tracking over time: declining trust alongside rising adoption suggests teams are shipping code they don't fully believe in, which should trigger governance attention, not be dismissed as normal friction.

Questions I Should Be Able to Ask My Team:

  1. What security scanning (Static Application Security Testing (SAST), dependency, secrets) runs specifically on AI-agent-generated PRs, and does it run per-iteration or only at final merge?
  2. Do we have any policy distinguishing what an agent is allowed to touch unsupervised (e.g., auth, payments, data access layers) versus what always requires human security review?
  3. What's our current measure of developer trust in AI-generated code, and if it's declining, what's driving that, and are we listening to it or overriding it with velocity targets?

Technologies / Standards / Companies to Know: Standard SAST/DAST and dependency-scanning tools (Snyk, Semgrep, GitHub Advanced Security) extended for agent-generated code volume; OWASP guidance on AI-assisted development; Cloud Security Alliance research on vibe-coding governance gaps.

Recommended Learning:

Time Investment: Half day