Android Local AI Server

Run an LLM on your Android device and use it through an OpenAI-compatible localhost API.

Private by default | Built for Android | Compatible with existing clients

Android Local AI Server turns one Android device into a configurable local inference endpoint for browser apps, Termux tools, and other Android apps. Models execute on-device through LiteRT-LM. Requests never need to leave the phone.

Project status: Active development. LiteRT-LM inference works today. GGUF inference works through llama.cpp when the native JNI library is built and packaged with scripts/build-llama-jni.sh.

See it in action

Run One control for the local server	Choose Models optimized for Android hardware	Download Progress, cancellation, and resume

From install to first response

1. Install the app

Build the debug APK in Termux:

gradle --no-daemon :app:assembleDebug
termux-open app/build/outputs/apk/debug/app-debug.apk

Android will open its package installer. A packaged release flow is not available yet.

2. Add a model

Open Models and download the recommended Gemma model. You can also import an existing .litertlm file or provide a direct model URL under Settings > Import a model.

Large downloads can be cancelled and resumed. A cancelled download remains on the device so a later retry can continue when the source supports HTTP range requests.

3. Start the server

Open Home and tap the large power control. The server starts on:

http://127.0.0.1:8080/v1

It continues running as an Android foreground service when the app is closed or the screen turns off.

4. Configure your client

Copy the base URL and API key from Home. Copy a model ID by tapping an installed model on the Models page.

export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=local

The client selects a model for each request. Catalogue models use stable, readable IDs such as gemma-4-e2b-it-litert and gemma-4-e2b-it-q4-k-m-gguf; imported files prompt for a custom ID. The Android UI does not maintain a selected serving model.

5. Send a request

List the installed model IDs:

curl http://127.0.0.1:8080/v1/models

Then use one of those IDs in a chat completion:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer local" \
  -d '{
    "model": "<model-id>",
    "messages": [
      { "role": "user", "content": "Explain why on-device AI is useful in one sentence." }
    ],
    "max_tokens": 128
  }'

What works today

Area	Support
Runtime	LiteRT-LM .litertlm models
API	Models, health, chat completions, streaming chat, text responses
Tool use	OpenAI-style tool calls and tool-result follow-up
Model management	Catalogue downloads, direct URLs, file import, deletion
Runtime tuning	Per-model context window from 2K to 32K tokens and request-scoped reasoning
Downloads	Progress, cancellation, retry, HTTP range resume
Lifecycle	Preload, lazy load, one loaded model, idle unload, manual unload
Android service	Foreground operation and optional start on boot
Browser clients	Permissive CORS on the loopback server

Not implemented:

• Release-packaged llama.cpp JNI/native library artifacts
• Image generation, audio generation, and transcription
• Remote network access
• Full OpenAI API compatibility
• Multiple models loaded at the same time

Client behavior

Streaming

Set "stream": true on /v1/chat/completions. The server returns OpenAI-compatible SSE chunks. The final JSON chunk contains finish_reason (stop or tool_calls), followed by:

data: [DONE]

Run python scripts/compat-openai.py for direct wire-format checks. If the official Python client is installed, the same script also tests through OpenAI; pass --require-openai to make that check mandatory.

Tool calls

Tool execution stays with the client:

1. Send the available functions in tools.
2. Receive an assistant message containing tool_calls.
3. Execute the requested function in the client.
4. Send the original assistant tool-call message followed by a role: "tool" result.

The server preserves this conversation structure for LiteRT-LM rather than executing tools itself.

Responses API

Text requests are available through /v1/responses:

curl http://127.0.0.1:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-id>",
    "input": "Reply with a short sentence.",
    "max_tokens": 64
  }'

Health

curl http://127.0.0.1:8080/v1/health

Runtime configuration

Each installed model has its own runtime profile. Open Models, tap the settings icon beside a downloaded model, and choose its context window:

Context	Best for	Tradeoff
2K	Short prompts and constrained devices	Lowest memory use
4K	General chat and focused code excerpts	Recommended default
8K	Larger documents and coding context	Higher memory use and prefill latency
16K	Long source files and conversations	Requires substantial free memory
32K	Maximum supported Gemma 4 context	Experimental on mobile hardware

The app reloads an active model after saving a changed context setting. A larger window raises capacity, not speed: prompt processing becomes slower and the KV cache consumes more memory even when a request uses only part of the available context.

Runtime configuration is intentionally modeled above the inference adapter. LiteRT-LM receives maxNumTokens today. A future llama.cpp adapter can use the same model profile for context size while adding adapter-specific controls such as quantization metadata, thread count, batch size, and Vulkan GPU offload.

Reasoning

Reasoning is controlled by each request rather than stored in the model profile. Send an OpenAI-compatible reasoning_effort value:

{
  "model": "<model-id>",
  "reasoning_effort": "high",
  "messages": [
    { "role": "user", "content": "Find the bug in this function." }
  ],
  "max_tokens": 256
}

Supported values are none, minimal, low, medium, high, and xhigh. none disables thinking. The other values currently enable the runtime's model-defined thinking behavior. Upstream does not yet provide distinct token-budget mappings for each effort label, so this project does not pretend that low and high behave differently.

Reasoning and the final answer share max_tokens. Chat Completions return model reasoning separately as message.reasoning_content and streamed delta.reasoning_content; it is never mixed into content. Usage includes completion_tokens_details.reasoning_tokens. For llama.cpp, the JNI bridge applies a small reasoning budget so GGUF thinking models leave room for the final answer instead of spending the whole output limit on thoughts. On mobile, enable reasoning selectively for tasks that benefit from additional deliberation.

Model and request lifecycle

Server startup is intentionally cheap. Starting the foreground service does not load a model.

1. A user can preload an installed model from Models, or a request can load it on demand.
2. Later requests reuse the initialized engine.
3. Every HTTP request creates a fresh inference conversation from its supplied OpenAI messages.
4. A request naming another model unloads the current model first.
5. The idle timeout or manual unload action releases model memory.
6. Stopping the server unloads the runtime.

Only one model engine can be loaded at a time, but request history is never retained invisibly between clients. The client owns conversation history and sends it with each request, matching the OpenAI stateless request model. Set the idle timeout to 0 to keep the engine loaded until manual unload or server shutdown.

Security model

The HTTP server binds only to:

127.0.0.1

The API key value local is accepted for compatibility but is not used as authentication. This is appropriate only while the service remains loopback-only. Authentication must be implemented before any future LAN binding.

CORS is currently permissive so browser applications on the same device can connect.

Architecture

Android app
├── Material 3 user interface
│   ├── Server control and connection details
│   ├── Model catalogue and local model management
│   └── Service, memory, and import settings
├── Foreground service
│   └── OpenAI-compatible HTTP server on 127.0.0.1
├── Model runtime manager
│   ├── Preload, lazy load, and single-model policy
│   └── Runtime profile, idle unload, and manual unload
├── Runtime configuration seam
│   └── Per-model context and future backend controls
├── LiteRT-LM runtime
├── llama.cpp GGUF runtime
├── Resumable download manager
└── App-controlled model storage

The important boundaries are:

• OpenAiHttpServer handles sockets and routes.
• OpenAiProtocol parses and formats OpenAI-compatible JSON.
• ModelRuntimeManager owns model loading, runtime configuration, and generation lifecycle.
• RuntimeConfig is the persisted, runtime-neutral model configuration seam shared by inference adapters.
• GenerationConfig carries request-scoped controls such as output limit and reasoning.
• LiteRtRuntime applies LiteRT-LM configuration and creates a fresh conversation for every request.
• LlamaCppRuntime adapts GGUF models through the JNI bridge in app/src/main/cpp/llama_jni.cpp.
• DownloadManager owns resumable network downloads.
• ModelStore persists model records and configuration.

Build and verify

The project is designed to build directly on Android in Termux. It requires Java 21 because LiteRT-LM 0.16.0 contains Java 21 bytecode.

Install the Java/Android build prerequisites and ensure local.properties points at your Android SDK:

sdk.dir=/data/data/com.termux/files/home/android-sdk

This repo intentionally uses Termux-native Android build tools. Keep android.aapt2FromMavenOverride in gradle.properties and do not replace ARM-native tools with desktop SDK binaries.

Build llama.cpp JNI support

GGUF support is packaged through app/src/main/jniLibs/arm64-v8a/liblocalaiserver_llama.so. Rebuild it after changing app/src/main/cpp/, updating third_party/llama.cpp, or cloning onto a fresh device:

scripts/build-llama-jni.sh

The script:

• installs Termux packages needed for native builds (cmake, ninja, clang, shaderc, Vulkan headers/loader),
• clones third_party/llama.cpp if it is missing,
• builds with the Termux-native compiler/toolchain rather than the Android NDK desktop toolchain,
• enables Vulkan and CPU backends,
• writes liblocalaiserver_llama.so and libc++_shared.so into app/src/main/jniLibs/arm64-v8a/.

Do not run Android Studio/NDK CMake commands directly on-device unless you know the hybrid SDK layout; the official NDK binaries are usually x86_64 and the Termux setup relies on native ARM tools.

Build the APK

Then build and verify the APK:

gradle --no-daemon :app:assembleDebug
apksigner verify --print-certs app/build/outputs/apk/debug/app-debug.apk

APK output:

app/build/outputs/apk/debug/app-debug.apk

If stripDebugDebugSymbols warns that it cannot execute an NDK llvm-strip, the debug build can still succeed; Gradle packages the unstripped native library.

Validate a running server

Run the end-to-end smoke test:

python scripts/smoke-openai.py

It checks health, model listing, chat, request-scoped reasoning, streaming, tool calls, and tool-result follow-up.

Run the lightweight benchmark:

python scripts/bench-openai.py

Compare two installed model IDs, for example a Gemma LiteRT-LM CPU build and a Gemma GGUF llama.cpp build:

python scripts/bench-openai.py -m <litert-model-id> -m <gguf-model-id> -n 3

It reports cold or warm time-to-first-token, total generation time, and approximate streaming decode speed. Performance varies substantially with device hardware, model variant, backend, battery state, and thermal conditions. Use the same prompt, token limit, context window, and model family to compare LiteRT CPU, llama.cpp CPU, and llama.cpp Vulkan runs.

Before releasing

Manually verify the Android lifecycle behaviors that an HTTP smoke test cannot cover:

• Closing the UI leaves the foreground service running.
• The API responds after screen-off.
• Idle and manual unload release the model.
• Start on boot works when enabled.
• A large download can be cancelled, resumed, and removed.
• Port changes take effect only after restarting the server.

Direction

The product direction is a phone-native counterpart to desktop local model servers: model management, explicit runtime profiles, a stable OpenAI-compatible interface, and honest device capability detection.

The next major runtime milestone is making llama.cpp packaging reproducible for release builds across supported ABIs. The Java runtime seam routes GGUF records to a llama.cpp adapter and fails clearly if the native library is absent; it does not add placeholder responses that can be mistaken for model output. Its configuration builds on the existing per-model runtime profile and includes CPU/Vulkan backend selection.