Selected AI Work

Voice AI agent · Tool use · macOS · iOS · CarPlay

Batta: Personal AI Voice Assistant

Personal project · Built solo, late Sep – early Oct 2026 · Beta

A personal AI assistant you talk to. It follows you from your desk to your phone to your car, remembering the same things everywhere and keeping the same voice and character.

Role
Solo builder · Design to backend
Year
2026
Focus
Voice AI agent
Stack
8 technologies
  • Gemini Live
  • Swift 6
  • SwiftUI
  • App Intents
  • CarPlay
  • FastAPI
  • PostgreSQL
  • Railway

  • 30+

    Tools the agent can use

  • 3

    Platforms: macOS, iOS, CarPlay

  • ~40K

    Lines of Swift

  • 6.6K

    Lines of Python backend

  • 6

    Phases: voice loop to beta

  • 2

    Languages: English & Sinhala

The problem

Voice assistants are split across devices. Each one forgets what the others know, can't act across your machines, and either does too little or acts without asking. I wanted one assistant with one memory that can actually get things done, and that never takes an irreversible step on its own.

Overview

Batta streams real-time voice conversation over the Gemini Live API. You can interrupt it mid-sentence, it cancels echo from laptop speakers, and long conversations reconnect without a gap. On the Mac, an on-device "Hey Batta" wake word means no audio leaves the machine until Batta is awake, and a global ⌥Space shortcut works as push-to-talk.

It acts through 30+ tools covering everyday tasks, messaging, Mac control, maps, and music. A relay lets the iPhone or CarPlay ask the Mac to do things, with progress shown as a Live Activity. Apple granted the CarPlay conversational-app entitlement for this project.

I built every part myself: the native apps, a shared Swift framework, the Python backend, and the design system, which features an animated lamp (a flame on a clay pahana) as Batta's visual identity.

Architecture

How a conversation flows

The apps never hold API keys. Devices get a short-lived token from the backend and stream audio straight to Gemini, so the backend stays off the latency path.

  1. 01

    Wake

    On-device

    The "Hey Batta" wake word runs locally on the Mac, alongside ⌥Space push-to-talk and Siri, Shortcuts, and Action button intents.

  2. 02

    Authorise

    Backend

    The app requests a short-lived, single-use token. Usage limits are checked before any AI spend.

  3. 03

    Converse

    Gemini Live · WebSocket

    Low-latency streaming audio with interruption, echo cancellation, and seamless reconnects. The model sits behind a VoiceModel protocol so it can be swapped.

  4. 04

    Act

    Tool registry

    Self-describing tools, filtered per platform, so Mac-only tools are never offered on iPhone or CarPlay.

  5. 05

    Guard

    Safety

    Irreversible actions wait for a real tap on a confirmation banner, and reversible ones record how to undo themselves.

  6. 06

    Remember

    Shared memory

    People, preferences, and facts are stored per device and synced through the backend, encrypted at rest.

How it works

Cross-device relay

The iPhone or the car can ask the Mac to do the work.

  1. 01

    Request

    iPhone · CarPlay

    A voice request needs a Mac capability, such as files, apps, or the screen.

  2. 02

    Relay

    Backend

    The request is routed to the user's Mac.

  3. 03

    Execute

    macOS

    The Mac runs the tool through the Accessibility API, ScreenCaptureKit, Shortcuts, or scripts.

  4. 04

    Report

    Live Activity · APNs

    Progress and results stream back to the phone as a Live Activity.

Components

  • macOS app

    Wake word, push-to-talk, Mac control tools (apps, volume, brightness, files, tabs, screen, Accessibility clicks), and a notarized DMG build.

  • iOS + CarPlay app

    Voice-first iPhone app with a voice-only CarPlay experience, App Intents, and WidgetKit Live Activities.

  • Shared Swift package

    Modules for Voice, Live, Tools, Memory, Safety, Prompt, Session, and the Design System, shared by every app.

  • Backend

    Python 3.12, FastAPI, SQLAlchemy 2, Alembic, and PostgreSQL on Railway, handling tokens, memory sync, relay, push, and usage limits with a cost dashboard.

  • Integrations

    Sign in with Apple, Google OAuth with PKCE for Gmail, Google Places and Routes, and APNs push.

  • Design system

    Animated lamp identity, accent colour themes, and WCAG-AA contrast tests.

Key features

  • Everyday: weather, calendar, reminders, a morning briefing, and web search with cited sources
  • Communication: read and send Gmail, iMessage, and WhatsApp
  • Mac control: open apps, change volume, brightness, and dark mode, manage files and browser tabs, look at the screen, and run Shortcuts and scripts
  • Maps: nearby places, travel time with live traffic, "leave now" reminders, and turn-by-turn directions
  • Music: "play … on YouTube Music" on any device
  • Proactive help: scheduled briefings, topic watches with push notifications, and alerts when the Mac runs hot or the battery is low
  • Bilingual: English and native Sri Lankan Sinhala, with strict language-following rules

Architecture highlights

  • 01No API keys in the apps: short-lived, single-use tokens, with audio streamed directly to Gemini
  • 02Self-describing tools: each declares its name, schema, platforms, and whether it can be undone or needs confirmation, and the system prompt is generated from the registry
  • 03The AI can never approve its own action, and reversible actions support a plain "undo"
  • 04Swift 6 strict concurrency, @Observable MVVM, SwiftData, and unit tests for every tool
  • 05Scripted TestFlight and notarized-DMG releases via XcodeGen

Status & next steps

  • Beta code is complete
  • Next: the first TestFlight build (iPhone and CarPlay) and a notarized DMG (Mac) for family testers