Case study · 2026
Dikera: Agentic AI Voice Assistant for Mac, iPhone & CarPlay
Agentic AI · Real-time voice agent · Tool calling · Swift + FastAPI
Personal AI product · Built solo, Sep – Oct 2026 · Beta: code complete, TestFlight and notarized DMG for family testers next
A personal AI agent you talk to. It follows you from your desk to your phone to your car, plans multi-step requests, uses 35 tools to get them done, and remembers the same things on every device. It never takes an irreversible step without a human tap.

- Solo builder · AI engineer · Design to backend
- 2026
- Agentic AI
- 13 technologies
- Gemini Live API
- Gemini Flash-Lite
- Function calling
- Structured output
- Swift 6
- SwiftUI
- CarPlay
- App Intents
- FastAPI
- PostgreSQL
- WebSockets
- Google Stitch
- Railway
35
Agent tools, filtered per device
44/46
Real-model eval cases passed (from 40)
3
Platforms: macOS, iOS, CarPlay
728
Automated tests (581 Swift, 147 Python)
~34K
Lines of code (27K Swift, 6.7K Python)
0
API keys shipped inside the apps
iPhone client UI
The first screen is the running app; the rest are the Google Stitch reference designs the SwiftUI screens were rebuilt from · tap to enlarge
The problem
Voice assistants are split across devices. Each one forgets what the others know, can't act across your machines, and either does too little or acts without asking. I wanted one agent with one memory that can actually get things done (email, calendar, maps, music, full control of a Mac), works in English and Sinhala, and is safe enough to trust with real actions. That raises the hard agentic-AI questions: how the model picks tools, how you stop it doing something irreversible, how it checks its own work, and how you measure all of that.
Overview
Dikera is a tool-using AI agent with a voice. The Gemini Live API handles real-time speech in and speech out. You can talk over Dikera and he stops to listen, echo is cancelled on laptop speakers, and long conversations reconnect without a gap. On the Mac, an on-device "Hey Dikera" wake word means no audio leaves the machine until he is awake.
The agent loop is the core of the project. The model plans and emits function calls. Deterministic Swift code then decides whether each call may run: is the tool legal on this device, does this call need a human tap, and how is it undone. Dikera acts through 35 self-describing tools: memory, weather, calendar, reminders, Gmail, places and routes, YouTube Music, topic watches, and 14 Mac-only tools for controlling the computer (apps, files, settings, browser tabs, reading the screen, clicking and typing in any app, iMessage, Shortcuts, and scripts).
Multi-step requests ("What's the weather in Kandy tomorrow, and remind me to take an umbrella") run as parallel tool calls. A live step checklist shows each step as it runs. When the agent clicks something on the Mac, it reads the screen state before and after, so it reports "a Save As dialog opened" rather than claiming the file is saved.
Your devices work as one system. From the iPhone or the car, Dikera can delegate to your Mac through a WebSocket relay (ask_mac). The Mac keeps its own confirmation gate, so an irreversible action requested from the car still needs a tap on a screen.
Memory needs no vector database. The model saves facts through tools, recalls them by meaning in English or Sinhala, and after each conversation a second model (Gemini Flash-Lite with a JSON schema) picks out up to five lasting facts. Memory is stored locally in SwiftData, synced through the backend, and encrypted at rest.
I measure the agent with DikeraEval, my own evaluation harness. It runs 46 scripted cases against the real model, with the real prompt and tool schemas, and checks tool choice, argument matching, ordering, and safety using fixed rules, with no LLM judge. The first baseline passed 40/46. After fixing what it found, it passed 44/46.
Architecture
System architecture
One repository with clearly separated modules. Three Swift apps share one agent core package. The FastAPI backend holds every secret and every paid API. Audio streams from the device straight to Gemini, so the backend stays off the latency path. Each layer links to its code.
- Channels
- AI agents
- Service
- External API
- Data & infra
- Client app code
Client apps (the UIs)
Mac app
HUD with the Orb, live captions, content cards and a step checklist. Menu bar item, "Hey Dikera", ⌥Space, clipboard helper.
iPhone app
Live tab with a Voice | Chat switch, plus Memories and History tabs. Siri, Action button, Live Activity, and camera vision.
CarPlay
Voice-control template, with Maps and music handed over to the car screen. Voice first: nothing to read while driving.
Shared features
Chat, Memory, confirmation cards with an expiry countdown, content cards, the step checklist, and usage.
@Observable view models → AssistantSessionControlling (protocol) - Agent core code
Agent core
AssistantSession
Owns the conversation and its state machine. Runs tool calls off the main actor and tracks each step.
PromptBuilder
Rebuilds the system prompt at every wake from the persona, memories, legal tools, limits, and language rules.
ToolRegistry
35 self-describing tools, offered and re-checked per platform.
ConfirmationGate + UndoStack
Human tap for irreversible calls. Reversible calls record their undo before they run.
ports (VoiceModel, LiveTransport, MemoryRepository, …) implemented by the Data layer - Code
Real-time model connection
GeminiLiveModel
WebSocket session with session resumption and sliding-window context compression
VoiceAudioEngine
16 kHz mic in, 24 kHz playback out, and voice processing with echo cancellation
WakeWordListener
Apple's on-device speech recogniser, active only while Dikera sleeps
Single-use token
Fetched from the backend for each connection and locked to the model
WebSocket: audio + function calls + transcripts LLMs
Gemini Live (native audio)
The conversation: speech, function calling, Google Search grounding, and screen and camera frames
Gemini Flash-Lite
One-shot jobs with structured JSON output: memory learning, clipboard actions, and topic-watch checks
Apple on-device models
Speech recognition for the wake word, and Vision for reading text on the screen
tools that need secrets or paid APIs go through the backend- Backend code
Backend
Tokens & auth
Sign in with Apple sessions. Mints single-use Gemini tokens.
Server-side tools
Gmail (OAuth PKCE), Places, Routes, YouTube Data, Open-Meteo, and topic watches with push
Relay hub
Gives the iPhone and CarPlay access to the Mac's tools over a WebSocket
Budget guard
Per-feature cost metering, a monthly budget, and daily caps (429 before any spend)
SQLAlchemy 2 + Alembic Data
memories
Values Fernet-encrypted at rest, synced by cursor, newest edit wins
integrations
Encrypted Google refresh tokens
usage_days
Requests, tokens, and estimated cost per user, day, and feature
On device
SwiftData memory and conversations. Session token in the Keychain.
Architecture diagrams
Eight diagrams, from the big picture down to the details: the system, one request end to end, the agent loop with its guard rails, the state machine, the clean architecture of the Swift code, the cross-device relay, memory, and evaluation. They are drawn from the code and the repository's ARCHITECTURE.md. Tap any diagram to zoom.
System architecture
Four columns: the client apps, the shared agent core, the cloud (Gemini Live and the FastAPI backend), and data and external APIs. The key decision is the split of paths. Voice goes straight from the device to Gemini, while everything that needs a secret or costs money goes through the backend.
- The apps never hold an API key. They get a single-use, model-locked token from POST /v1/tokens/live
- The same DikeraCore package runs on all three surfaces, so the agent behaves the same everywhere
- The Mac's RelayAgent keeps a WebSocket to the backend, so the phone and the car can use the Mac's tools
- The budget guard sits in front of every paid endpoint ($10 a month by default, plus daily caps)
One request, end to end
A two-part request traced from the wake word to the spoken answer. The model asks for two tools in one turn. The registry checks both, they run in parallel, and Dikera answers once with everything done.
- Steps 2–3: usage limits are checked before any AI spend, then a short-lived token is issued
- Step 4: the system prompt lists only the tools legal on this device, plus what Dikera cannot do here
- Steps 8–10: parallel function calls, a step checklist on screen, and errors returned as JSON the model can explain
- After the conversation, a second model learns lasting facts in the background
The agent loop: tool calling with guard rails
This is the agentic heart of Dikera. The LLM is free to plan and choose tools, but whether a call may actually run is decided by deterministic code that the model cannot override. I use the same principle in my CrewAI work: let the model reason, but keep control flow and safety in code.
- Platform check on every call: Mac-only tools can't run on the phone even if the model asks
- Confirmation is decided per call: ui_control asks only for risky clicks such as "Send"
- The tap token is issued only by UI code. It is single-use, bound to one call, and expires in 60 seconds. The model can never confirm its own action
- Every failure goes back to the model as {"error": …}, so Dikera explains it instead of hallucinating success
Assistant state machine
A single state machine drives the audio session, the colour and motion of the Orb, the captions, and the CarPlay template, so the UI can never disagree with what the agent is doing.
- Working is a separate state from thinking, so you can see when a tool is running
- Barge-in flushes playback and goes straight back to listening
- Speaking → idle waits for both the model's turnComplete and the audio actually finishing
Clean architecture inside the Swift code
Dependencies point inwards. The domain knows only protocols, so the voice model sits behind a VoiceModel port and Gemini could be swapped for another real-time model without touching the tools, the session, or the UI.
- Each app's composition root builds the real implementations and injects them
- View models depend only on protocols, so they are tested with fakes and no audio or network
- Every tool is unit-tested, and the domain stays pure Swift
Cross-device relay (ask_mac)
Delegation between devices. The phone's agent sees the Mac's tools in its prompt and calls them through one ask_mac tool. The Mac still enforces its own safety rules, and the one-time id is kept away from the model.
- Screenshots travel only in memory and go to the phone's model as an image frame
- Only devices belonging to the same user can reach each other
- A confirmation runs only for the same tool and the same arguments within 60 seconds
Memory: save, recall by meaning, learn, sync
Long-term memory without a vector database. I measured Apple's on-device embeddings first and rejected them: they gave wrong top matches, had no usable cut-off, and didn't support Sinhala. Instead, the LLM matches by meaning over a compact list of facts.
- Learning uses structured output: a JSON schema, at most five facts, and only things the user actually stated
- Injection-safe: the conversation is wrapped in tags and treated as data, and the prompt template fills placeholders in a single pass
- Sync uses a (server_updated_at, id) cursor and newest-edit-wins, with values encrypted using Fernet
DikeraEval: testing the agent against the real model
Prompt and tool changes are measured, not guessed. Each case opens a real Gemini Live session with Dikera's real prompt and tools, answers the tool calls from a script so nothing actually runs, and checks the result with fixed rules.
- 46 cases: memory, tool choice, multi-step requests, small talk, Mac control, safety, CarPlay limits, and the relay
- --repeat gives pass rates for flaky cases, and --baseline shows what got better or worse
- A unit test makes sure every case only expects tools its device actually offers
Agent design
How Dikera is built as an agent
The same building blocks as any production agent system (role, tools, memory, planning, guard rails, and evaluation), implemented natively in Swift for real-time voice.
- 01
Role & persona
A warm Sri Lankan companion with rules for how to speak (short and heard, not read) and how to act (say one line before slow work, never invent facts, only say "done" when a result shows it).
- 02
Tools
Each tool declares its name, a one-line summary for the model, a JSON schema, its platforms, whether it can be undone, and whether it needs confirmation. The prompt is generated from the registry, and tool names are never hard-coded.
- 03
Planning
Multi-part requests become several tool calls in one turn. Dikera does every part before answering, and says which part failed if one does.
- 04
Memory
The Live session keeps context with sliding-window compression. Long-term facts sit in the prompt, can be recalled by meaning, and are learned after each conversation.
- 05
Delegation
A phone-side agent hands work to the Mac-side agent and gets results (including screenshots) back, much like one agent delegating to another.
- 06
Guard rails
Platform checks on every call, a human tap for irreversible actions, undo for reversible ones, and per-feature budgets so a runaway loop can't run up costs.
- 07
Self-checking
After every click or keypress the agent reads the screen state and reports what actually changed, or says that nothing did.
- 08
Evaluation
Regression tests against the real model with rule-based matching. They are run before and after every prompt or tool change.
UI design
Designed in Google Stitch, rebuilt in SwiftUI
The screens were designed with Google Stitch (Google's Gemini-powered UI design tool), then rebuilt by hand in SwiftUI on Dikera's own design tokens. No generated HTML or CSS is shipped.
- 01
Design rules first
One design-system prompt (palette, type, spacing, the Orb, the rules) goes into Stitch before any screen, along with the logo.
- 02
One prompt per screen
Always in the same shape: what and where, who and why, layout top to bottom, real copy, and the rules (no shadows, no gradients).
- 03
Review & refine
Only Dikera's colours, New York serif for Dikera's words, a 4-pt grid, 44 pt targets, AA contrast. Changes are made with short single-purpose edit prompts or 3 variants.
- 04
Map to tokens
Every value is mapped to a token. If a design needs a value that isn't a token, either a token is added or the design changes.
- 05
Rebuild in SwiftUI
The Orb in the mockups is only a picture. The real one is drawn and animated in code (OrbView).
Tools & tech
AI & agents
Each model has one job, and model names are server settings, so they can change without an app release.
- Gemini Live API (native audio)
- Real-time speech-to-speech over WebSocket (BidiGenerateContentConstrained), with function calling, transcription, automatic voice activity detection, session resumption, and context compression.
- Gemini Flash-Lite
- Structured-output jobs (JSON schema): learning memories, the clipboard helper (translate, summarise, explain, fix), and news checks for topic watches.
- Function calling / tool use
- 35 tools declared with JSON schemas, plus Gemini's built-in Google Search grounding for news, prices, and comparisons.
- Prompt engineering
- A templated system prompt rebuilt at every wake. It holds the persona, memory, the tools legal on this device, explicit limits, language and accent rules, and multi-part request rules.
- Ephemeral tokens
- Single-use tokens (uses: 1, a new session within 1 minute, expiry after 30) locked to the model with fieldMask, so the app's own prompt and tools still apply.
- DikeraEval
- My own evaluation harness: real model, scripted tools, rule-based EvalMatcher, baseline diffs, and repeat runs for flaky cases.
- Apple on-device ML
- SFSpeechRecognizer with requiresOnDeviceRecognition for the wake word, and Vision RecognizeTextRequest for reading the screen.
Client apps
- Swift 6 + SwiftUI
- Strict concurrency, @Observable MVVM, one view per file, built on a shared design system.
- AVAudioEngine
- Voice processing with echo cancellation, a PlaybackClock that knows when speech has really ended, and barge-in.
- SwiftData + Keychain
- Local memory and conversations. The session token stays readable after first unlock, so CarPlay works with the phone locked.
- CarPlay
- Apple granted the voice-based conversational app entitlement. The app follows Apple's rules: list → voice-control template, at most 3 levels deep.
- App Intents, WidgetKit, ActivityKit
- Siri and the Action button, a lock-screen widget, and a Live Activity for Mac tasks started from the phone.
- macOS automation
- The Accessibility API, ScreenCaptureKit, AppleScript and Apple Events, the Shortcuts CLI, and SMAppService for launch at login.
- XcodeGen
- Every target is defined in project.yml, with scripted TestFlight and notarized DMG releases.
Backend
- Python 3.12 + FastAPI
- Routers per domain with no logic in them, services that hold the logic, and Pydantic v2 schemas.
- PostgreSQL + SQLAlchemy 2 + Alembic
- Users, devices, encrypted memories, integrations, topic watches, and usage, with versioned migrations.
- WebSockets
- The relay hub between a user's Macs and their phone and car.
- cryptography (Fernet)
- Memory values and OAuth refresh tokens encrypted at rest, with a rotatable key list.
- httpx · PyJWT · uv · ruff · pytest
- Async API clients, Sign in with Apple verification, dependency management, linting, and 147 tests.
- Railway
- Deployed from a Dockerfile with a /health check. All secrets live in Railway variables.
Integrations
- Sign in with Apple
- Accounts and sessions, with no passwords.
- Gmail API (OAuth 2.0 + PKCE)
- Read, search, and summarise email. Sending and replying always need a tap.
- Google Places (New) + Routes
- Nearby places and travel time with traffic, falling back to Apple MapKit.
- YouTube Data API v3
- Finds a song, then hands it to YouTube Music on the phone, Mac, or car screen.
- Open-Meteo · APNs
- Weather with no API key, and push notifications for topic watches and budget warnings.
Code modules
Packages/DikeraCore
The shared agent: session, state machine, prompt builder, tool registry, all 35 tools, safety, memory, the Gemini Live client, audio, sync, relay, and the design system.
lahirumanulanka/dikera-assistant/tree/main/Packages/DikeraCoreDomain/Tools
Every ability as a self-describing DikeraTool, including the Mac/ and Maps/ sub-folders. A good place to see how the tool-calling layer is built.
lahirumanulanka/dikera-assistant/tree/main/Packages/DikeraCore/Sources/DikeraCore/Domain/ToolsApps/DikeraMac
The HUD, menu bar, clipboard helper, settings, and every macOS permission, distributed as a notarized Developer ID DMG.
lahirumanulanka/dikera-assistant/tree/main/Apps/DikeraMacApps/DikeraiOS
The Live, Memories, and History tabs, Devices and Settings, the CarPlay scene, App Intents, and the Live Activity.
lahirumanulanka/dikera-assistant/tree/main/Apps/DikeraiOSApps/Shared
Features compiled into both apps: Chat, Memory, confirmation cards, content cards, the step checklist, conversations, and usage.
lahirumanulanka/dikera-assistant/tree/main/Apps/Sharedbackend
Tokens, Sign in with Apple, memory sync and learning, server-side tools, Gmail OAuth, the relay hub, push, and usage metering with budgets.
lahirumanulanka/dikera-assistant/tree/main/backendDikeraEvalKit + eval/
The eval runner, scripted cases, rule-based matcher, and reports. Cases live in eval/cases.json.
lahirumanulanka/dikera-assistant/tree/main/Packages/DikeraCore/Sources/DikeraEvalKitdocs/
ARCHITECTURE.md, UI_DESIGN.md, the Stitch prompts and reference screens, the roadmap, and the release guide.
lahirumanulanka/dikera-assistant/tree/main/docs
Results
Real-model eval: pass rate by area
Latest full run, 44/46. Cases run on a real Gemini Live session, with tool calls scripted.
- Memory (8)8/8
- Mac control & UI (8)8/8
- iPhone, CarPlay & relay (6)6/6
- Small talk & multi-step (5)5/5
- Everyday tools (14)13/14
- Safety: needs a tap (5)4/5
Eval baseline → after fixes
DikeraEval found prompt and tool problems, which were then fixed and re-measured.
- All cases: baseline87%
- All cases: after96%
- Cross-device: baseline50%
- Cross-device: after100%
Codebase
Lines of code, counted from the repository.
- Swift apps + core19.2K
- Swift tests7.8K
- Python backend6.7K
Tools by platform
The registry offers each device only the tools that are legal there.
- Mac33
- iPhone (+ Mac via relay)21
- CarPlay (voice-safe)16
Key features
- Talk naturally: real-time speech with barge-in, in English or Sinhala, with a chosen male voice and a Chat mode for typing
- Wake it anywhere: "Hey Dikera" on the Mac (fully on-device), ⌥Space push-to-talk, Siri, the Action button, the lock screen, or CarPlay
- Everyday: weather, a morning briefing, calendar, reminders, "leave by" reminders, and web search with Google Search grounding
- Communication: read and summarise Gmail and iMessage; send and reply only after a tap; pre-fill WhatsApp messages
- Maps & music: nearby places, travel time with traffic, turn-by-turn hand-over to the car screen, and "play … on YouTube Music"
- Full Mac control: apps, files, settings, browser tabs, looking at and reading the screen, clicking and typing in any app, Shortcuts, and scripts
- Proactive: watch a topic and get a push notification when there's news, plus budget warnings
- Undo for every reversible action, and a human tap for every irreversible one
Agentic patterns I apply with CrewAI too
- Narrow, self-describing tools. A Dikera tool declares its schema, platforms, and safety needs, much like a CrewAI @tool with a clear description and args schema. The model chooses well because the descriptions are short and specific.
- Deterministic orchestration around the LLM. The model plans, but code decides whether a step may run, the same way my CrewAI ServiceOrchestrator in BrokeMe fixes the agent order and halts on a failure or a review flag.
- Human-in-the-loop as a hard gate, not a prompt. The confirmation token can only come from UI code, which is the same idea as human_input and review queues in a crew.
- Delegation between agents. ask_mac hands a task to another agent with different tools and returns the result, similar to a manager agent delegating to a specialist.
- Memory that works in production: short-term context, long-term facts, and a learning step after each conversation, with structured output so one chatty response can't corrupt the store.
- Evaluate, then change. Every prompt or tool change is checked against a fixed case set with rule-based scoring, the same discipline I used to measure the six-agent CrewAI pipeline (F1, recall, latency).
Safety, privacy & cost
- No secrets in the apps: the Gemini key and every third-party key live only on the backend
- No audio is stored anywhere, the mic never streams while Dikera is asleep, and the wake word is recognised on the device
- Memory values and OAuth tokens are encrypted at rest. Email and message content is never stored or logged on the server
- Location is used per question and never stored. Logs never contain audio, tokens, or memory values
- Cost control: each conversation's token usage is reported and metered, with a monthly budget and per-feature daily caps. A warning appears at 80%, and Settings shows spend per day and per feature
Engineering highlights
- 01Designed and built a production-style AI agent end to end, alone: three native clients, a shared agent core, a Python backend, and an evaluation harness
- 02A tool-calling architecture where the LLM can never approve its own irreversible action, and every reversible action can be undone
- 03Real-time voice: device-to-model audio streaming, barge-in, echo cancellation, and seamless reconnects past the roughly 10-minute connection limit
- 04Cross-device agent delegation (phone or car → Mac) that keeps safety guarantees across the relay
- 05Measured LLM behaviour with a rule-based eval on the real model, improving the pass rate from 87% to 96%
- 06Swift 6 strict concurrency, clean architecture, and 728 automated tests across Swift and Python
Status & next steps
- All six phases are code complete: the voice loop, memory and tools, Mac control, iPhone and CarPlay, the relay with proactive features, and polish
- Next: the first TestFlight build (iPhone and CarPlay) and a notarized DMG (Mac) for family testers
- Later: a custom Core ML wake-word model, Shortcuts on the iPhone, and an optional avatar with lip-sync