AI agents will soon run on our phones the way apps do today, powered by cloud LLMs and on-device micro models. A marketplace will follow. This whitepaper lays out the skeleton, template, and nine building blocks of the governance framework that ecosystem will need: manifests, trust certificates, runtime gates, risk tiers, audit, and liability.
Something new is about to run on our phones. Not another category of app, but a different kind of software altogether: AI agents that plan and act on our behalf, powered by large models in the cloud or small micro models on the device itself. They will book our travel, manage our inboxes, pay our bills, and coordinate across the apps we already use. As they mature, an ecosystem will grow around them, and with it a marketplace where third parties publish agents to millions of users, just as they publish apps today.
The app world took fifteen years to build its trust machinery: stores, review processes, permissions, code signing, and revocation. Agents inherit almost none of it cleanly. An app is a static artifact you can inspect once and trust to stay the same. An agent is a dynamic actor whose behavior shifts with its model, its instructions, and whatever content it encounters at runtime, including content planted by attackers.
This article lays out what a governance framework for that ecosystem needs to contain, and it is written to work as a template: five roles, seven layers, and nine building blocks, each with its purpose, its minimum requirements, the artifact it produces, and the honest limit of what it can guarantee.
The question is not whether this ecosystem will need governing. It is whether the framework gets designed deliberately, before the first serious incident, or assembled in a panic afterward.
01 · Background
For fifteen years the mobile experience has been organized around apps: open one, do something, close it. That model is giving way. In the near future, most routine operations on a phone will be handled by agents acting on the user's behalf. Some will run against powerful cloud-hosted LLMs; others entirely on-device against small, efficient local models; most will blend the two, keeping private data local and reaching to the cloud for heavy reasoning.
The platform groundwork is already visible. Apple exposes app capabilities to system intelligence through App Intents, backed by on-device Foundation Models and Private Cloud Compute. Google's AppFunctions let any app expose typed functions the system agent can discover and call, with on-device inference through AICore and Gemini Nano. Different companies, same primitive: declared, typed, reviewed application capabilities, surfaced to an agent, executed under OS isolation.
Once third parties can build agents against those primitives, the way they build apps against SDKs today, an agent marketplace exists whether anyone designed it or not. And that is the point at which governance stops being optional. We govern apps because strangers' code runs on our devices. Agents raise the stakes: strangers' judgment will act with our authority, our money, and our accounts.
App-store governance works because the thing that ships is the thing that was reviewed. Agents break that assumption in four ways:
The consequence is the framework's first principle: certification cannot be a one-time gate. Governance must span the moment before publication (assessment), every moment of execution (enforcement), and the whole life after release (monitoring and revocation).
02 · The skeleton
| Role | Responsibility | App-world analogy |
|---|---|---|
| Principal (User) | Delegates authority, consents, can revoke at any time | Device owner |
| Publisher | Builds the agent, declares its contract, carries first-line liability | App developer |
| Marketplace Operator | Lists, distributes, monitors, and revokes agents | App store |
| Platform | Provides capability, isolation, identity, and attestation primitives | Mobile OS |
| Assessor | Independently evaluates agents and signs attestations | App review / security lab |
One entity may wear several hats; on today's phones the Platform, Operator, and Assessor are usually the same company. The framework still keeps the roles distinct, because the accountability model depends on knowing which hat failed. A useful property falls out of this: in an enterprise deployment, the enterprise itself can be the Marketplace Operator for its managed fleet, which means the framework can be piloted without any platform's permission.
Every conforming system realizes seven logical layers, whoever implements them: identity and delegation, the capability manifest, consent and authorization, runtime enforcement, isolation and sandboxing, observability and audit, and lifecycle and revocation. Two concerns cut across all seven: certification with risk tiers, and accountability with liability.
The framework defends against nine marketplace-specific threats, mapped to public taxonomies (OWASP Top 10 for Agentic Applications, OWASP MCP Top 10, MITRE ATLAS): goal hijack via prompt injection, confused-deputy invocation across apps, over-broad manifests, silent model substitution, exfiltration through tool chains, payment abuse, background persistence, impersonation, and delegation abuse between agents.
03 · The core
Every action must trace back to a verifiable publisher, a verifiable agent instance, and an explicit chain of delegated authority. Delegation is carried in attenuable, scoped tokens: any hand-off, whether user to agent or agent to sub-agent, can only narrow authority, never widen it. Identities are non-transferable, which prevents a market in reputations.
Honest limit: identity proves who, never whether they behave. It is the precondition for accountability, not a substitute for it.
The heart of the framework, and its central template: a signed, machine-readable contract declaring everything the agent can do. Its capabilities (each bound to a native construct like an App Intents schema or AppFunction), its data scopes, every external tool it can invoke, its payment authority and limits, its network egress, and its model provenance. It is reviewed at certification, compiled into runtime policy, and rendered at consent. Default-deny is the enforcement rule: anything not declared is denied at runtime.
Honest limit: the manifest bounds the capability surface, not behavior inside it. Its job is to make the blast radius finite and known.
Before an agent receives authority the user sees a plain-language rendering of the manifest, distinguishing read from act and flagging anything irreversible or money-moving. Each capability declares a rung on the autonomy ladder: advisory, confirm-each, bounded-autonomous, or autonomous.
The subtle part is the consent budget. Confirmation fatigue is a security failure, not a UX nuisance: a user trained to approve everything protects nothing. So bounds, not prompts, are the primary consent mechanism. The user consents once, precisely, to a bound ("up to $200 a month on transit, with these two providers") rather than repeatedly and vaguely to individual actions. Prompts are reserved for actions that are irreversible, out of bound, or anomalous.
Honest limit: consent is only as good as comprehension. A technically complete disclosure nobody reads is compliance theater.
Static review cannot bind dynamic behavior, so something must stand between the agent's decision and the world's state change. The framework splits this into two layers with deliberately different guarantees, because pretending one mechanism delivers both has been the central over-claim of early agent-governance proposals.
Layer A, the deterministic scope gate, is a guarantee. A policy enforcement point sits in the path of every action; nothing reaches a native capability or tool without a decision evaluated against the manifest, the user's grant, and any delegation token in force. The check is deterministic, auditable, and made before the side-effecting call. Layer A guarantees the agent never exceeds its declared, consented scope.
Layer B, the probabilistic intent monitor, is an estimate. Inside declared scope, an injected agent can still act against the user's interest. Layer B watches for that: anomaly detection on action patterns, plausibility checks against declared purpose, and injection detection on untrusted content. Its outputs are risk signals, not verdicts; it escalates rather than silently allowing or denying, and its false-positive and false-negative rates must be measured and published.
Honest limit: Layer A is exactly as strong as the manifest is narrow. Layer B will miss novel attacks and flag legitimate behavior; budget for both.
Agents run inside OS-level isolation, and on-device inference routes through the platform's isolated compute path where one exists. The critical rule: the agent and the untrusted content it reads are separate trust domains. Instructions found in content are data, not commands; any action content induces still passes the runtime gate under the user's grant, with the content's provenance recorded.
Honest limit: isolation bounds where effects can occur, not which in-scope effects are wise.
Obligations scale with risk. Tier is a function of the most sensitive capability and highest autonomy level declared:
| Tier | Profile | Autonomy ceiling | Assessment |
|---|---|---|---|
| T0 · Informational | Read-only, non-sensitive | Advisory | Static review; self-attestation acceptable |
| T1 · Low-impact | Reversible actions; no sensitive data or payments | Confirm-each / bounded | Static review plus behavioral evaluation |
| T2 · Sensitive | Health, financial, location, or communications data | Bounded-autonomous | Full assessment incl. independent red-team; periodic re-certification |
| T3 · High-autonomy | Irreversible or high-value actions; value transfer | Autonomous within bound | Enhanced red-team; continuous monitoring; strict revocation SLAs |
Two rules keep the tiers meaningful. The composition rule: tier attaches to the effective capability of the composed system, so a publisher cannot split one T3 agent into several "T1" pieces. And borderline cases resolve by reducing scope, never by waiver. Success yields a signed trust certificate binding the manifest hash, tier, attestation set, expiry, and a revocation reference.
Model churn gets a pragmatic split: publisher-initiated changes trigger re-certification, while platform model updates within a declared lineage keep certificates valid provided the agent passes an automated regression slice. Otherwise a single OS model update would invalidate every certificate on the marketplace at once.
Honest limit: a certificate means "assessed against these named tests, on this date, at this tier." It never means "safe."
Every consequential action and delegation lands in a tamper-evident log attributing it to an agent, publisher, delegation chain, and user, enough to reconstruct who authorized what, through which chain, against which certificate. The privacy architecture is a requirement, not a nicety: the full-fidelity log lives on the device, under the user's control; the operator receives only aggregated, pseudonymized telemetry, never action contents.
Honest limit: logs deter and reconstruct; they do not prevent.
The post-market half of trust. Continuous monitoring at sensitive tiers verifies the running agent matches what was certified, using platform integrity APIs of the App Attest and Play Integrity class. Single-point revocation propagates a policy change effective at the runtime gate, not merely a store de-listing. Top-tier agents support a kill switch with defined semantics for multi-step operations: never a payment left half-executed in silence.
And because a central kill switch is also a censorship risk, revocations must be publicly logged, reason-coded, and appealable.
Honest limit: revocation speed is bounded by connectivity; an offline device enforces the last policy it saw.
Governance without a liability answer is decoration. The framework states a default allocation, disclosed to the user at consent, rebuttable, and meant to be negotiated and insured:
| Failure mode | Primarily liable | Rationale |
|---|---|---|
| Agent acts outside declared scope | Operator / Platform | The deterministic gate is their guarantee |
| Agent acts within scope but against user intent (injection) | Publisher, mitigated by conformance | They chose the model, prompts, and tool design |
| Undisclosed capability, binding, or model change | Publisher, aggravated | A revocation-grade violation |
| Certification negligence | Assessor | That is what the attestation attests |
| In-bound autonomous action the user regrets | User, with cooling-off remedies | The bound was the consent |
Honest limit: this is a default among the parties, not law. But publishing a clear default beats silence, because silence allocates liability to whoever has the worst lawyers, and that is usually the user.
04 · Proof it hangs together
Follow a travel agent, TripWeaver, through its life:
Every step names its building block. If an implementation cannot tell this story for its own agents, the missing step is the gap.
05 · Getting started
Not all nine blocks are needed on day one. A working first implementation is four pieces: the manifest with default-deny, a verified publisher identity bound into it, the deterministic scope gate in front of every action, and runtime-effective revocation. That core already delivers the property the app model cannot: a finite, enforced, revocable blast radius. Consent bounds, certification tiers, the intent monitor, privacy-architected audit, and the liability framework layer on from there.
Then deploy where nobody's permission is needed: enterprise-managed fleets first (the enterprise is the operator for its own devices, today), voluntary publisher self-declaration as a trust signal next, platform convergence after that, and cross-marketplace certificate portability last.
A governance framework earns credibility by stating what it cannot do. The manifest is static while the threat is dynamic; the architecture answers this by making the blast radius finite and labeling everything inside it as probabilistically managed. Certification risks becoming another Common Criteria, expensive and gamed; the mitigations (tiers, scope-reduction instead of waivers, named tests, the composition rule) are structural, and certification cost is a transparency metric to publish, not assume away. Confirmation fatigue is real, which is why bounds are the consent mechanism. And some problems remain genuinely open: continuous behavioral attestation at acceptable on-device cost, injection detection against adaptive attackers, and how any of this interacts with agency law across jurisdictions.
The agent ecosystem on mobiles is coming either way. The choice on the table is narrower and more urgent: whether its trust machinery is designed with the same care the ecosystem itself will demand. This skeleton, one signed contract, one runtime gate, one lifecycle loop, and an honest account of its own limits, is offered as the starting point.