Use on-device AI when privacy, offline operation, low interaction latency and predictable per-request cost dominate. Use cloud AI when the task needs a larger model, long context, current knowledge or broad device coverage. For most production Android apps, the best answer is a deliberately designed hybrid route—not an invisible cloud fallback attached at the end of development.
Key takeaways
- The decision belongs at the feature level. One app can safely use on-device inference for rewriting and cloud inference for grounded research.
- On-device does not automatically mean private. Telemetry, backups and fallbacks can still move sensitive data off the phone.
- Cloud does not automatically mean slow. Network quality, streaming, prompt size and regional infrastructure all influence perceived latency.
- Hybrid inference expands coverage, but it also creates two behaviors to test, disclose, monitor and support.
- A production plan needs an availability check, an explicit routing policy, a failure experience and separate metrics for local and cloud execution.
On-device, cloud and hybrid AI at a glance
Google’s July 2026 Android guidance frames execution location as a product decision. Gemini Nano 4, built on Gemma 4, is available to Android developers through ML Kit’s Prompt API on supported devices. Firebase AI Logic adds cloud and hybrid paths, including modes that prefer or require one execution location.
| Criterion | On-device AI | Cloud AI | Hybrid AI |
|---|---|---|---|
| Sensitive input | Can remain on the phone | Leaves the device for processing | Depends on routing and fallback |
| Offline use | Possible after model availability | Normally unavailable | Local path can continue |
| Model capability | Constrained by device resources | Larger models and longer context | Match task to the appropriate model |
| Current information | Requires supplied local context | Can use server tools or grounding | Route current-knowledge tasks to cloud |
| Device coverage | Limited to compatible devices | Broad if connectivity is available | Broadest, with more implementation work |
| Variable inference cost | No cloud charge for local request | Usage-based cloud cost | Reduced cloud volume, not zero cost |
| Operational control | Tied to device/model updates | Central model and policy changes | Two runtimes to govern |
| Typical best fit | Private transforms, offline assistance | Complex generation, grounded answers | Mixed workloads and graceful coverage |
When is on-device AI the right choice?
On-device inference is strongest when the request contains information that should not leave the phone and the task can be completed by the local model. Examples include rephrasing a private draft, extracting fields from a locally stored note, generating a short reply, classifying content or assisting where connectivity is unreliable.
The user experience can also feel immediate because a request avoids a network round trip. Google highlights privacy, offline functionality, low latency and the absence of cloud inference charges as key benefits of Gemini Nano through ML Kit. Those benefits are meaningful, but each needs a qualification.
First, availability is not universal. An app must check whether the required model and feature are supported, ready or need preparation. Never place a local-AI button in the interface and assume every Android device can complete the same request.
Second, local execution still consumes resources. Large prompts, repeated generation and sustained inference can affect memory, battery and thermal behavior. Measure the full interaction on representative low-, middle- and high-end supported devices—not only a flagship developer phone.
Third, local privacy is a system property. A prompt that stays on the device during inference can still appear in analytics events, debug logs, screenshots, cloud backups or customer-support attachments. Document every data path before marketing the feature as private.
Our Pixel 10 Tensor G5 on-device AI guide examines the hardware side of local inference. This guide focuses on the application architecture that must work across a mixed Android install base.
When should an Android app use cloud AI?
Cloud inference earns its place when task quality depends on capabilities that a compact local model or a specific phone cannot reliably supply. That includes longer context, complex reasoning, high-output workloads, server-side tools, shared enterprise knowledge and answers grounded in current information.
Cloud execution can also simplify consistency. A product team can route supported users to the same centrally managed model and update system instructions, safety controls or retrieval sources without waiting for a device model rollout. It serves phones that lack the required on-device model, provided they have suitable connectivity.
The tradeoffs are material. Prompts and context cross a network boundary. The feature acquires authentication, abuse prevention, quota, regional availability, retention and vendor-governance requirements. Latency becomes sensitive to connectivity and server load. Cost scales with request volume, prompt size, output length and model choice.
For Firebase AI Logic, Google recommends protections such as App Check with Play Integrity to reduce unauthorized access to backend resources. That is one control, not a complete threat model. Teams should also define rate limits, user entitlements, data minimization, retention rules and incident ownership.
Cloud AI should therefore be chosen because the feature needs it—not because a prototype was easiest to wire to a remote endpoint.
Why is hybrid inference often the practical answer?
Hybrid inference treats execution location as a runtime decision. A short private transformation may run locally; a request needing fresh web information may go to the cloud. If a compatible on-device model is unavailable, a permitted cloud path can preserve the feature for more users.
Firebase AI Logic documents four useful policy shapes:
PREFER_ON_DEVICE: try local inference and use the cloud when the local path is unavailable or cannot satisfy the request.PREFER_IN_CLOUD: use the cloud by default and fall back locally when appropriate.ONLY_ON_DEVICE: do not send the request to the cloud.ONLY_IN_CLOUD: require the centrally hosted path.
These modes are product promises, not merely SDK settings. If the interface says “processes only on this device,” a cloud fallback is not acceptable. If a regulated workflow requires a centrally logged model decision, silent local execution may be equally inappropriate.
How should privacy and consent change the route?
Classify the data before choosing a model. A useful design review separates public input, ordinary account data, confidential work content, highly sensitive personal data and data that policy forbids from leaving the device.
For every class, record whether it can be sent to a processor, which fields must be removed, where inference may occur, how long records are retained and whether a user must opt in. Apply the rule to attachments, retrieved context and generated output—not only the text typed into a prompt box.
If execution can switch locations, the interface should explain the meaningful difference. A compact status such as “Processed on this device” or “Uses cloud AI” can establish the boundary without forcing users to understand model architecture. A setting that disables cloud fallback is valuable only if the feature respects it in background retries and error recovery.
How do latency, quality and offline behavior interact?
Do not reduce the choice to a single benchmark. Users experience time to first visible response, total completion time, interruptions, retries and whether the result is useful enough to avoid a second attempt.
An on-device model can start quickly with no network, yet take longer to complete a demanding request on constrained hardware. A cloud model can stream an early answer over a strong connection, yet fail in a warehouse, airplane or congested event venue. A lower-quality local result that forces the user to rewrite everything has poor effective latency.
Build a test grid around real operating conditions:
| Test dimension | Minimum cases |
|---|---|
| Device | Oldest supported, median active, current flagship |
| Connectivity | Offline, weak/variable, normal Wi-Fi, low-latency cellular |
| Input | Short, typical, long, multimodal where supported |
| Thermal state | Cool start and repeated-use warm state |
| Language | Highest-volume languages and difficult edge cases |
| Outcome | Time to first token, completion, task success, retry, cancellation |
Define a quality floor for the local path. If it falls below that threshold, either ask permission for cloud processing, offer a non-AI workflow or explain that the task cannot be completed offline. Hiding a materially weaker answer behind the same label erodes trust.
What costs should the architecture review include?
On-device inference removes the cloud model charge for a locally completed request, but not the total cost of the feature. The team still owns compatibility checks, dual-path tests, device support, model preparation, analytics, customer support and release validation.
Cloud costs are easier to see because they arrive as usage. Estimate them with production-shaped prompt and output lengths, cache assumptions, retry rates and abuse scenarios. Add observability, gateway, retrieval, storage and incident-response costs where relevant.
Hybrid can reduce variable spend by moving suitable volume to devices, while increasing fixed engineering and QA effort. It is most attractive when local execution handles a large, stable class of requests—not when an elaborate router saves only a small number of cloud calls.
A feature-level decision scorecard
Score each proposed feature from 1 to 5. High scores do not automatically mean cloud or local; they show where a hard constraint exists.
| Question | If high, favor |
|---|---|
| Would sending input off-device create serious risk? | On-device or no AI |
| Must the feature work without a network? | On-device |
| Does the answer require current external information? | Cloud |
| Does the task need long context or a larger model? | Cloud |
| Is broad support across older devices essential? | Cloud or hybrid |
| Is request volume high and value per request low? | On-device or tightly optimized hybrid |
| Must behavior change centrally and immediately? | Cloud |
| Can users tolerate different capability by device? | On-device or hybrid |
Then add one decision owner for privacy, one for product quality and one for operating cost. Architecture should not be decided by whichever team first produced a demo.
How should teams implement and roll out hybrid AI?
Start with a narrow, measurable task. Build an explicit execution contract before adding a router.
- Define the task, prohibited inputs and minimum acceptable output.
- Check device/model availability before presenting the local path.
- Select the routing mode and write the fallback rule in product language.
- Minimize context independently for local and cloud requests.
- Show the execution location when it affects privacy, cost or capability.
- Instrument route, latency, completion, quality proxy, cancellation and fallback reason without logging sensitive prompt content.
- Test both paths and every transition between them.
- Release to a small cohort, compare task success and cost, then expand.
Keep non-AI recovery available. Users need a manual search, standard form, saved draft or retry option when neither model path works. The Android 17 fleet upgrade guide provides a compatible staged-release model for testing OS and device differences before broad rollout.
What should teams monitor after launch?
Separate metrics by execution route, device class, app version and network condition. A blended average can hide the exact segment receiving slow or low-quality responses.
Monitor availability rate, local preparation failures, cloud fallback frequency, time to first response, completion time, task-success proxy, user correction, retry, cancellation and cost per successful task. Add privacy and safety signals such as policy-block frequency, unexpected data egress and vendor incidents.
Review route drift. A hybrid feature can gradually become cloud-only if local compatibility deteriorates, or silently lower quality if more requests are forced onto a small local model. Set thresholds that trigger investigation.
For agentic features that act across apps, execution location is only one control. Our Android Halo AI agent visibility guide explains the separate need for visible actions, confirmation and user control.
Final recommendation
Default to on-device for bounded, privacy-sensitive, offline-friendly transformations that a supported local model can complete well. Default to cloud for capability-intensive or current-knowledge tasks with an approved data path. Choose hybrid when both workload classes are real and the product team is prepared to own routing, disclosure, observability and two complete test surfaces.
The durable architecture is not the one with the most AI options. It is the one whose execution rules a product manager can explain, a privacy reviewer can verify, an engineer can test and a user can understand when the network or device changes.
