Selected AI Work

Dikera: Agentic AI Voice Assistant for Mac, iPhone & CarPlay

Agentic AI · Real-time voice agent · Tool calling · Swift + FastAPI

Personal AI product · Built solo, Sep – Oct 2026 · Beta: code complete, TestFlight and notarized DMG for family testers next

A personal AI agent you talk to. It follows you from your desk to your phone to your car, plans multi-step requests, uses 35 tools to get them done, and remembers the same things on every device. It never takes an irreversible step without a human tap.

Creative dark tech showcase of Dikera personal AI voice assistant with its glowing golden smiling character and voice waves
Role
Solo builder · AI engineer · Design to backend
Year
2026
Focus
Agentic AI
Stack
13 technologies
  • Gemini Live API
  • Gemini Flash-Lite
  • Function calling
  • Structured output
  • Swift 6
  • SwiftUI
  • CarPlay
  • App Intents
  • FastAPI
  • PostgreSQL
  • WebSockets
  • Google Stitch
  • Railway
  • 35

    Agent tools, filtered per device

  • 44/46

    Real-model eval cases passed (from 40)

  • 3

    Platforms: macOS, iOS, CarPlay

  • 728

    Automated tests (581 Swift, 147 Python)

  • ~34K

    Lines of code (27K Swift, 6.7K Python)

  • 0

    API keys shipped inside the apps

iPhone client UI

The first screen is the running app; the rest are the Google Stitch reference designs the SwiftUI screens were rebuilt from · tap to enlarge

Live: listening (real app)
Speaking + weather card
Chat mode
Human-in-the-loop: send email
Confirmation sheet (voice)
Irreversible: shut down Mac
Step checklist / tasks
Background Mac task
Gmail inbox summary
Memory: read, search, forget
Home: idle

Voice assistants are split across devices. Each one forgets what the others know, can't act across your machines, and either does too little or acts without asking. I wanted one agent with one memory that can actually get things done (email, calendar, maps, music, full control of a Mac), works in English and Sinhala, and is safe enough to trust with real actions. That raises the hard agentic-AI questions: how the model picks tools, how you stop it doing something irreversible, how it checks its own work, and how you measure all of that.

Dikera is a tool-using AI agent with a voice. The Gemini Live API handles real-time speech in and speech out. You can talk over Dikera and he stops to listen, echo is cancelled on laptop speakers, and long conversations reconnect without a gap. On the Mac, an on-device "Hey Dikera" wake word means no audio leaves the machine until he is awake.

The agent loop is the core of the project. The model plans and emits function calls. Deterministic Swift code then decides whether each call may run: is the tool legal on this device, does this call need a human tap, and how is it undone. Dikera acts through 35 self-describing tools: memory, weather, calendar, reminders, Gmail, places and routes, YouTube Music, topic watches, and 14 Mac-only tools for controlling the computer (apps, files, settings, browser tabs, reading the screen, clicking and typing in any app, iMessage, Shortcuts, and scripts).

Multi-step requests ("What's the weather in Kandy tomorrow, and remind me to take an umbrella") run as parallel tool calls. A live step checklist shows each step as it runs. When the agent clicks something on the Mac, it reads the screen state before and after, so it reports "a Save As dialog opened" rather than claiming the file is saved.

Your devices work as one system. From the iPhone or the car, Dikera can delegate to your Mac through a WebSocket relay (ask_mac). The Mac keeps its own confirmation gate, so an irreversible action requested from the car still needs a tap on a screen.

Memory needs no vector database. The model saves facts through tools, recalls them by meaning in English or Sinhala, and after each conversation a second model (Gemini Flash-Lite with a JSON schema) picks out up to five lasting facts. Memory is stored locally in SwiftData, synced through the backend, and encrypted at rest.

I measure the agent with DikeraEval, my own evaluation harness. It runs 46 scripted cases against the real model, with the real prompt and tool schemas, and checks tool choice, argument matching, ordering, and safety using fixed rules, with no LLM judge. The first baseline passed 40/46. After fixing what it found, it passed 44/46.

System architecture

One repository with clearly separated modules. Three Swift apps share one agent core package. The FastAPI backend holds every secret and every paid API. Audio streams from the device straight to Gemini, so the backend stays off the latency path. Each layer links to its code.

  • Channels
  • AI agents
  • Service
  • External API
  • Data & infra
  1. Client apps (the UIs)

    Apps/DikeraMac · Apps/DikeraiOS · Apps/Shared · SwiftUI
    Client app code
    • Mac app

      HUD with the Orb, live captions, content cards and a step checklist. Menu bar item, "Hey Dikera", ⌥Space, clipboard helper.

    • iPhone app

      Live tab with a Voice | Chat switch, plus Memories and History tabs. Siri, Action button, Live Activity, and camera vision.

    • CarPlay

      Voice-control template, with Maps and music handed over to the car screen. Voice first: nothing to read while driving.

    • Shared features

      Chat, Memory, confirmation cards with an expiry countdown, content cards, the step checklist, and usage.

    @Observable view models → AssistantSessionControlling (protocol)
  2. Agent core

    Packages/DikeraCore · Swift 6 strict concurrency
    Agent core code
    • AssistantSession

      Owns the conversation and its state machine. Runs tool calls off the main actor and tracks each step.

    • PromptBuilder

      Rebuilds the system prompt at every wake from the persona, memories, legal tools, limits, and language rules.

    • ToolRegistry

      35 self-describing tools, offered and re-checked per platform.

    • ConfirmationGate + UndoStack

      Human tap for irreversible calls. Reversible calls record their undo before they run.

    ports (VoiceModel, LiveTransport, MemoryRepository, …) implemented by the Data layer
  3. Real-time model connection

    Data/Live · Data/Audio
    Code
    • GeminiLiveModel

      WebSocket session with session resumption and sliding-window context compression

    • VoiceAudioEngine

      16 kHz mic in, 24 kHz playback out, and voice processing with echo cancellation

    • WakeWordListener

      Apple's on-device speech recogniser, active only while Dikera sleeps

    • Single-use token

      Fetched from the backend for each connection and locked to the model

    WebSocket: audio + function calls + transcripts
  4. LLMs

    Google Gemini family · model names are server settings
    • Gemini Live (native audio)

      The conversation: speech, function calling, Google Search grounding, and screen and camera frames

    • Gemini Flash-Lite

      One-shot jobs with structured JSON output: memory learning, clipboard actions, and topic-watch checks

    • Apple on-device models

      Speech recognition for the wake word, and Vision for reading text on the screen

    tools that need secrets or paid APIs go through the backend
  5. Backend

    backend/ · Python 3.12 · FastAPI · Railway
    Backend code
    • Tokens & auth

      Sign in with Apple sessions. Mints single-use Gemini tokens.

    • Server-side tools

      Gmail (OAuth PKCE), Places, Routes, YouTube Data, Open-Meteo, and topic watches with push

    • Relay hub

      Gives the iPhone and CarPlay access to the Mac's tools over a WebSocket

    • Budget guard

      Per-feature cost metering, a monthly budget, and daily caps (429 before any spend)

    SQLAlchemy 2 + Alembic
  6. Data

    PostgreSQL · SwiftData · Keychain
    • memories

      Values Fernet-encrypted at rest, synced by cursor, newest edit wins

    • integrations

      Encrypted Google refresh tokens

    • usage_days

      Requests, tokens, and estimated cost per user, day, and feature

    • On device

      SwiftData memory and conversations. Session token in the Keychain.

Eight diagrams, from the big picture down to the details: the system, one request end to end, the agent loop with its guard rails, the state machine, the clean architecture of the Swift code, the cross-device relay, memory, and evaluation. They are drawn from the code and the repository's ARCHITECTURE.md. Tap any diagram to zoom.

  1. Fig. 01 · Architecture diagram

    System architecture

    Four columns: the client apps, the shared agent core, the cloud (Gemini Live and the FastAPI backend), and data and external APIs. The key decision is the split of paths. Voice goes straight from the device to Gemini, while everything that needs a secret or costs money goes through the backend.

    • The apps never hold an API key. They get a single-use, model-locked token from POST /v1/tokens/live
    • The same DikeraCore package runs on all three surfaces, so the agent behaves the same everywhere
    • The Mac's RelayAgent keeps a WebSocket to the backend, so the phone and the car can use the Mac's tools
    • The budget guard sits in front of every paid endpoint ($10 a month by default, plus daily caps)
  2. Fig. 02 · UML sequence diagram

    One request, end to end

    A two-part request traced from the wake word to the spoken answer. The model asks for two tools in one turn. The registry checks both, they run in parallel, and Dikera answers once with everything done.

    • Steps 2–3: usage limits are checked before any AI spend, then a short-lived token is issued
    • Step 4: the system prompt lists only the tools legal on this device, plus what Dikera cannot do here
    • Steps 8–10: parallel function calls, a step checklist on screen, and errors returned as JSON the model can explain
    • After the conversation, a second model learns lasting facts in the background
  3. Fig. 03 · Agent workflow

    The agent loop: tool calling with guard rails

    This is the agentic heart of Dikera. The LLM is free to plan and choose tools, but whether a call may actually run is decided by deterministic code that the model cannot override. I use the same principle in my CrewAI work: let the model reason, but keep control flow and safety in code.

    • Platform check on every call: Mac-only tools can't run on the phone even if the model asks
    • Confirmation is decided per call: ui_control asks only for risky clicks such as "Send"
    • The tap token is issued only by UI code. It is single-use, bound to one call, and expires in 60 seconds. The model can never confirm its own action
    • Every failure goes back to the model as {"error": …}, so Dikera explains it instead of hallucinating success
  4. Fig. 04 · UML state machine

    Assistant state machine

    A single state machine drives the audio session, the colour and motion of the Orb, the captions, and the CarPlay template, so the UI can never disagree with what the agent is doing.

    • Working is a separate state from thinking, so you can see when a tool is running
    • Barge-in flushes playback and goes straight back to listening
    • Speaking → idle waits for both the model's turnComplete and the audio actually finishing
  5. Fig. 05 · Layer diagram

    Clean architecture inside the Swift code

    Dependencies point inwards. The domain knows only protocols, so the voice model sits behind a VoiceModel port and Gemini could be swapped for another real-time model without touching the tools, the session, or the UI.

    • Each app's composition root builds the real implementations and injects them
    • View models depend only on protocols, so they are tested with fakes and no audio or network
    • Every tool is unit-tested, and the domain stays pure Swift
  6. Fig. 06 · UML sequence diagram

    Cross-device relay (ask_mac)

    Delegation between devices. The phone's agent sees the Mac's tools in its prompt and calls them through one ask_mac tool. The Mac still enforces its own safety rules, and the one-time id is kept away from the model.

    • Screenshots travel only in memory and go to the phone's model as an image frame
    • Only devices belonging to the same user can reach each other
    • A confirmation runs only for the same tool and the same arguments within 60 seconds
  7. Fig. 07 · Data flow

    Memory: save, recall by meaning, learn, sync

    Long-term memory without a vector database. I measured Apple's on-device embeddings first and rejected them: they gave wrong top matches, had no usable cut-off, and didn't support Sinhala. Instead, the LLM matches by meaning over a compact list of facts.

    • Learning uses structured output: a JSON schema, at most five facts, and only things the user actually stated
    • Injection-safe: the conversation is wrapped in tags and treated as data, and the prompt template fills placeholders in a single pass
    • Sync uses a (server_updated_at, id) cursor and newest-edit-wins, with values encrypted using Fernet
  8. Fig. 08 · Evaluation pipeline

    DikeraEval: testing the agent against the real model

    Prompt and tool changes are measured, not guessed. Each case opens a real Gemini Live session with Dikera's real prompt and tools, answers the tool calls from a script so nothing actually runs, and checks the result with fixed rules.

    • 46 cases: memory, tool choice, multi-step requests, small talk, Mac control, safety, CarPlay limits, and the relay
    • --repeat gives pass rates for flaky cases, and --baseline shows what got better or worse
    • A unit test makes sure every case only expects tools its device actually offers

How Dikera is built as an agent

The same building blocks as any production agent system (role, tools, memory, planning, guard rails, and evaluation), implemented natively in Swift for real-time voice.

  1. 01

    Role & persona

    Resources/prompt.txt

    A warm Sri Lankan companion with rules for how to speak (short and heard, not read) and how to act (say one line before slow work, never invent facts, only say "done" when a result shows it).

  2. 02

    Tools

    DikeraTool protocol

    Each tool declares its name, a one-line summary for the model, a JSON schema, its platforms, whether it can be undone, and whether it needs confirmation. The prompt is generated from the registry, and tool names are never hard-coded.

  3. 03

    Planning

    Parallel function calls

    Multi-part requests become several tool calls in one turn. Dikera does every part before answering, and says which part failed if one does.

  4. 04

    Memory

    Short-term + long-term

    The Live session keeps context with sliding-window compression. Long-term facts sit in the prompt, can be recalled by meaning, and are learned after each conversation.

  5. 05

    Delegation

    ask_mac

    A phone-side agent hands work to the Mac-side agent and gets results (including screenshots) back, much like one agent delegating to another.

  6. 06

    Guard rails

    Human in the loop

    Platform checks on every call, a human tap for irreversible actions, undo for reversible ones, and per-feature budgets so a runaway loop can't run up costs.

  7. 07

    Self-checking

    ui_control ScreenState

    After every click or keypress the agent reads the screen state and reports what actually changed, or says that nothing did.

  8. 08

    Evaluation

    DikeraEval

    Regression tests against the real model with rule-based matching. They are run before and after every prompt or tool change.

Designed in Google Stitch, rebuilt in SwiftUI

The screens were designed with Google Stitch (Google's Gemini-powered UI design tool), then rebuilt by hand in SwiftUI on Dikera's own design tokens. No generated HTML or CSS is shipped.

  1. 01

    Design rules first

    UI_DESIGN.md

    One design-system prompt (palette, type, spacing, the Orb, the rules) goes into Stitch before any screen, along with the logo.

  2. 02

    One prompt per screen

    STITCH_PROMPTS.md

    Always in the same shape: what and where, who and why, layout top to bottom, real copy, and the rules (no shadows, no gradients).

  3. 03

    Review & refine

    Checklist

    Only Dikera's colours, New York serif for Dikera's words, a 4-pt grid, 44 pt targets, AA contrast. Changes are made with short single-purpose edit prompts or 3 variants.

  4. 04

    Map to tokens

    DikeraColor · DikeraFont · DikeraSpacing

    Every value is mapped to a token. If a design needs a value that isn't a token, either a token is added or the design changes.

  5. 05

    Rebuild in SwiftUI

    One view per file

    The Orb in the mockups is only a picture. The real one is drawn and animated in code (OrbView).

AI & agents

Each model has one job, and model names are server settings, so they can change without an app release.

Gemini Live API (native audio)
Real-time speech-to-speech over WebSocket (BidiGenerateContentConstrained), with function calling, transcription, automatic voice activity detection, session resumption, and context compression.
Gemini Flash-Lite
Structured-output jobs (JSON schema): learning memories, the clipboard helper (translate, summarise, explain, fix), and news checks for topic watches.
Function calling / tool use
35 tools declared with JSON schemas, plus Gemini's built-in Google Search grounding for news, prices, and comparisons.
Prompt engineering
A templated system prompt rebuilt at every wake. It holds the persona, memory, the tools legal on this device, explicit limits, language and accent rules, and multi-part request rules.
Ephemeral tokens
Single-use tokens (uses: 1, a new session within 1 minute, expiry after 30) locked to the model with fieldMask, so the app's own prompt and tools still apply.
DikeraEval
My own evaluation harness: real model, scripted tools, rule-based EvalMatcher, baseline diffs, and repeat runs for flaky cases.
Apple on-device ML
SFSpeechRecognizer with requiresOnDeviceRecognition for the wake word, and Vision RecognizeTextRequest for reading the screen.

Client apps

Swift 6 + SwiftUI
Strict concurrency, @Observable MVVM, one view per file, built on a shared design system.
AVAudioEngine
Voice processing with echo cancellation, a PlaybackClock that knows when speech has really ended, and barge-in.
SwiftData + Keychain
Local memory and conversations. The session token stays readable after first unlock, so CarPlay works with the phone locked.
CarPlay
Apple granted the voice-based conversational app entitlement. The app follows Apple's rules: list → voice-control template, at most 3 levels deep.
App Intents, WidgetKit, ActivityKit
Siri and the Action button, a lock-screen widget, and a Live Activity for Mac tasks started from the phone.
macOS automation
The Accessibility API, ScreenCaptureKit, AppleScript and Apple Events, the Shortcuts CLI, and SMAppService for launch at login.
XcodeGen
Every target is defined in project.yml, with scripted TestFlight and notarized DMG releases.

Backend

Python 3.12 + FastAPI
Routers per domain with no logic in them, services that hold the logic, and Pydantic v2 schemas.
PostgreSQL + SQLAlchemy 2 + Alembic
Users, devices, encrypted memories, integrations, topic watches, and usage, with versioned migrations.
WebSockets
The relay hub between a user's Macs and their phone and car.
cryptography (Fernet)
Memory values and OAuth refresh tokens encrypted at rest, with a rotatable key list.
httpx · PyJWT · uv · ruff · pytest
Async API clients, Sign in with Apple verification, dependency management, linting, and 147 tests.
Railway
Deployed from a Dockerfile with a /health check. All secrets live in Railway variables.

Integrations

Sign in with Apple
Accounts and sessions, with no passwords.
Gmail API (OAuth 2.0 + PKCE)
Read, search, and summarise email. Sending and replying always need a tap.
Google Places (New) + Routes
Nearby places and travel time with traffic, falling back to Apple MapKit.
YouTube Data API v3
Finds a song, then hands it to YouTube Music on the phone, Mac, or car screen.
Open-Meteo · APNs
Weather with no API key, and push notifications for topic watches and budget warnings.

Real-model eval: pass rate by area

Latest full run, 44/46. Cases run on a real Gemini Live session, with tool calls scripted.

  • Memory (8)8/8
  • Mac control & UI (8)8/8
  • iPhone, CarPlay & relay (6)6/6
  • Small talk & multi-step (5)5/5
  • Everyday tools (14)13/14
  • Safety: needs a tap (5)4/5

Eval baseline → after fixes

DikeraEval found prompt and tool problems, which were then fixed and re-measured.

  • All cases: baseline87%
  • All cases: after96%
  • Cross-device: baseline50%
  • Cross-device: after100%

Codebase

Lines of code, counted from the repository.

  • Swift apps + core19.2K
  • Swift tests7.8K
  • Python backend6.7K

Tools by platform

The registry offers each device only the tools that are legal there.

  • Mac33
  • iPhone (+ Mac via relay)21
  • CarPlay (voice-safe)16
  • Talk naturally: real-time speech with barge-in, in English or Sinhala, with a chosen male voice and a Chat mode for typing
  • Wake it anywhere: "Hey Dikera" on the Mac (fully on-device), ⌥Space push-to-talk, Siri, the Action button, the lock screen, or CarPlay
  • Everyday: weather, a morning briefing, calendar, reminders, "leave by" reminders, and web search with Google Search grounding
  • Communication: read and summarise Gmail and iMessage; send and reply only after a tap; pre-fill WhatsApp messages
  • Maps & music: nearby places, travel time with traffic, turn-by-turn hand-over to the car screen, and "play … on YouTube Music"
  • Full Mac control: apps, files, settings, browser tabs, looking at and reading the screen, clicking and typing in any app, Shortcuts, and scripts
  • Proactive: watch a topic and get a push notification when there's news, plus budget warnings
  • Undo for every reversible action, and a human tap for every irreversible one
  • Narrow, self-describing tools. A Dikera tool declares its schema, platforms, and safety needs, much like a CrewAI @tool with a clear description and args schema. The model chooses well because the descriptions are short and specific.
  • Deterministic orchestration around the LLM. The model plans, but code decides whether a step may run, the same way my CrewAI ServiceOrchestrator in BrokeMe fixes the agent order and halts on a failure or a review flag.
  • Human-in-the-loop as a hard gate, not a prompt. The confirmation token can only come from UI code, which is the same idea as human_input and review queues in a crew.
  • Delegation between agents. ask_mac hands a task to another agent with different tools and returns the result, similar to a manager agent delegating to a specialist.
  • Memory that works in production: short-term context, long-term facts, and a learning step after each conversation, with structured output so one chatty response can't corrupt the store.
  • Evaluate, then change. Every prompt or tool change is checked against a fixed case set with rule-based scoring, the same discipline I used to measure the six-agent CrewAI pipeline (F1, recall, latency).
  • No secrets in the apps: the Gemini key and every third-party key live only on the backend
  • No audio is stored anywhere, the mic never streams while Dikera is asleep, and the wake word is recognised on the device
  • Memory values and OAuth tokens are encrypted at rest. Email and message content is never stored or logged on the server
  • Location is used per question and never stored. Logs never contain audio, tokens, or memory values
  • Cost control: each conversation's token usage is reported and metered, with a monthly budget and per-feature daily caps. A warning appears at 80%, and Settings shows spend per day and per feature
  • 01Designed and built a production-style AI agent end to end, alone: three native clients, a shared agent core, a Python backend, and an evaluation harness
  • 02A tool-calling architecture where the LLM can never approve its own irreversible action, and every reversible action can be undone
  • 03Real-time voice: device-to-model audio streaming, barge-in, echo cancellation, and seamless reconnects past the roughly 10-minute connection limit
  • 04Cross-device agent delegation (phone or car → Mac) that keeps safety guarantees across the relay
  • 05Measured LLM behaviour with a rule-based eval on the real model, improving the pass rate from 87% to 96%
  • 06Swift 6 strict concurrency, clean architecture, and 728 automated tests across Swift and Python
  • All six phases are code complete: the voice loop, memory and tools, Mac control, iPhone and CarPlay, the relay with proactive features, and polish
  • Next: the first TestFlight build (iPhone and CarPlay) and a notarized DMG (Mac) for family testers
  • Later: a custom Core ML wake-word model, Shortcuts on the iPhone, and an optional avatar with lip-sync