The regulatory landscape — the shape of the obligations, not legal advice.
An engineer does not need to be a lawyer, but shipping an autonomous agent without understanding the shape of AI regulation is now a way to build something you have to tear out later. This essay is a qualitative map, not legal advice: the risk-tiered structure regulation is converging on, the documentation and human-oversight duties that recur across regimes, and how to design so the agent is governable before anyone asks for paperwork. For any specific obligation, get qualified legal counsel — the value here is knowing what to ask about.
Regulation is risk-tiered: obligation scales with potential harm.
The dominant structural idea, clearest in the EU AI Act, is that obligations scale with the risk a system poses, not with how impressive it is. A small number of uses are prohibited; a defined set of high-risk uses (for example employment decisions, credit, certain biometric and critical-infrastructure uses) carry heavy duties; limited-risk uses mainly carry transparency duties; and most systems sit in minimal-risk with little obligation. The first governance question for any agent is therefore not "is this advanced" but "what is it deciding, about whom, with what consequence" — that placement, not the model size, drives everything else.
The same model is low-risk as a code helper and high-risk deciding loan eligibility. Tier the use, not the technology — this maps directly onto the governance tiering in C6.
Documentation is a duty, not an afterthought.
Across regimes, higher-risk uses come with an obligation to document: what the system is for, its known limitations, the data it was built and evaluated on, the risks assessed and how they were mitigated, and how it is monitored in production. This is why model cards and data cards matter as governance artifacts, not just ML hygiene — they are the recurring form the documentation duty takes. The practical consequence for builders: the evidence has to be generated as you build. Reconstructing a risk assessment and a data lineage after the fact, under deadline, for a system already in production is the expensive failure mode.
General-purpose models add a provider layer of obligation.
Regulation increasingly distinguishes the provider of a general-purpose model from the deployer who builds an application on it. Foundation/general-purpose model rules (the EU AI Act's GPAI provisions are the reference example) attach documentation, transparency, and — for the most capable models — systemic-risk duties to the provider. Most product teams are deployers, and the load-bearing point is that using a third-party model does not transfer your deployment-level obligations to its provider. You inherit a documentation baseline from upstream; you still own the risk assessment, oversight, and monitoring of how you actually use it.
Human oversight is a required design property for high-risk uses.
A recurring, concrete requirement for high-risk uses is effective human oversight: a person must be able to understand the system's output, decline or override it, and stop it — and that has to be designed in, not bolted on. For an autonomous agent this is a hard architectural constraint, not a UX nicety: it forces an intervention point, an override path that actually halts effects, and enough decision legibility that the human can exercise judgment rather than rubber-stamp. The human-in-the-loop and least-privilege patterns from the Safety material are how you satisfy this; here the point is that for high-risk uses it is an obligation, not an option.
"A human can see the logs afterward" is not oversight. The requirement is meaningful intervention before the consequential effect, by someone equipped to actually judge it.
Frameworks turn the law into something you can build against.
Between abstract regulation and your code sit voluntary frameworks that operationalize it. The NIST AI Risk Management Framework structures the work as govern / map / measure / manage. ISO/IEC 42001 defines an AI management system — the auditable, certifiable process layer, analogous to what ISO 27001 did for security. These are not themselves law and certification is not legal compliance, but they are the practical bridge: a defensible, recognized way to organize the risk assessments, documentation, and oversight that regulation demands, and they crosswalk onto each other so one disciplined program serves several regimes at once.
The honest tradeoff — and the disclaimer.
Timelines and details are genuinely in motion: the EU AI Act's obligations phase in on staggered dates and amendment processes have shifted some of them, and other jurisdictions are moving on different tracks. Designing for the strictest plausible tier is robust but can over-burden a low-risk product; designing for today's lightest reading risks an expensive retrofit when a deadline lands. Treat governability — auditability, an oversight point, real documentation — as an architectural default, decide the tier from the use not the tech, and get qualified legal counsel for any specific obligation; this essay is a map, not legal advice.