How does LIA work?

LIA's main technical choices, and why I made them.

1. Overview

I designed LIA to run in your home, on modest hardware. The reference platform is a Raspberry Pi 5 (ARM64), and everything that follows comes from that: a single machine, a local PostgreSQL database and a local Redis, no database hosted elsewhere. Credentials and provider keys are stored encrypted.

Three principles guide the rest. Transparency first: observability is part of the architecture from the start, and every token consumed is tracked with its cost. Defense in depth next: quotas, confirmations and guardrails against hallucinations are independent layers. Provider independence last: each role in the system can be given to a different model, which protects me from a price rise as much as from an outage.

The stack is deliberately classic:

LayerChoice
BackendPython 3.14, FastAPI, asynchronous end to end
Agent orchestrationLangGraph 1.x
DataPostgreSQL with pgvector: data, graph checkpoints and vector search
Cache and coordinationRedis: cache, sessions, rate limits, leader election
FrontendNext.js 16 (App Router), React 19, strict TypeScript
ModelsOpenAI, Anthropic, Google Gemini, DeepSeek, Perplexity, Qwen, and any local model through Ollama
LogsStructured JSON, personal data filtered out

A local model runs with no API cost, entirely in your home. The API runs as several workers; a leader election in Redis reserves scheduled jobs for just one of them. Each optional subsystem is turned on by a flag, and configuration follows a single cascade: what the administrator sets in the app wins over the environment, which wins over the code's constants. I wanted a single cascade because a constant that silently drifts from the real configuration produces the hardest bugs to see.

A guided installer asks a few questions, creates the administrator account without writing any secret in clear text, checks that the installation really works and can reconfigure it later without asking for the keys again (installing LIA).

In orders of magnitude: more than 350 ADRs, hundreds of Prometheus metrics, tens of thousands of automated tests, an interface in six languages (French, English, German, Spanish, Italian, Chinese).

Going further: overall architecture · self-hosting · model providers · administrable capabilities

2. A backend split by domain

The backend follows Domain-Driven Design. Each business domain (agents, connectors, voice, memory, reminders, conversations…) is an isolated context in src/domains/, split the same way: router, service, repository, model, schema. Alongside them, a cross-cutting core (core) carries the configuration and the shared rules, and a shared infrastructure layer groups what they all need: the model factory, the cache, the MCP client, the browser, rate limits, the job scheduler, observability.

I chose a hexagonal architecture for external services. Tools know only Python protocols (ports), and the Google, Microsoft, Apple or Philips Hue clients implement them (adapters). Adding a provider therefore does not touch the code that calls it.

The repository is a monorepo: apps/api for the backend, apps/web for the interface, infrastructure/ for Docker and monitoring, docs/ for the documentation. An administrator who changes a model in the interface sees it take effect at once, without redeploying, because the live configuration is in the database.

Going further: creating a connector · creating an agent · creating a tool · adding a channel · adding a model provider · adding a proactive task

3. The journey of a message

I chose LangGraph to orchestrate the agents because it brought three things I needed: state persisted as checkpoints in PostgreSQL, cycles, and interrupt(), which lets a turn be suspended to ask for a confirmation and then resumed exactly where it stopped. Without this mechanism, there is no serious human confirmation.

Each message first goes through a router, which classifies it as plain conversation or as a request that calls for an action. A conversation goes straight to the response. An action follows the mode chosen in the chat header.

In Pipeline mode, the default, the planner writes an ExecutionPlan before anything runs: a JSON plan whose steps are linked by their dependencies. A second model, separate from the planner, validates this plan, so that no model judges its own work. The orchestrator then runs the steps in parallel waves. ReAct mode, by contrast, lets the model call a tool, read the result and decide what comes next, step by step.

I kept both because they meet two needs. The Pipeline plans, parallelizes, checks itself and forecasts its cost: one call to the planner and one for the response, against one call per iteration for ReAct, which means 4 to 8 times fewer tokens. ReAct adapts to each result, better for exploratory or ambiguous requests. They share the tool registry, confirmations, response node and observability; the automatic mode picks one per request.

Domains and tools are chosen by an embedding-based semantic router rather than by keywords or by the model alone: it costs less and makes fewer mistakes. Each tool declares a manifest (parameters, outputs, cost, permissions, keywords in several languages) read by the planner and the router, and each family of tools declares itself once in a central registry. Smart Services cut consumption further by caching query analysis and learning the plans that come back. After an actionable turn, an initiative node looks, read-only, at what the result implies elsewhere: forecast rain points to the outdoor activities in your calendar. Any other turn skips it and costs nothing.

A few rules keep the whole thing honest. A request with no intent to act cannot produce a plan that changes anything, and that is a deterministic rule, checked before any call to the model. A modification plan that runs out of correction attempts is not executed: LIA asks a question rather than write wrong data. A refusal or a failure is never presented as a completed action, and once the plan has run, nothing is replayed behind your back: the answer states each failure, which tool, why, and the exact total. For a mode that corrects itself along the way, there is ReAct.

The answer arrives token by token over SSE. Generation is decoupled from the connection: a detached producer publishes to a Redis stream specific to the turn, and the page is just one of its subscribers. Closing the tab therefore stops nothing, and the answer waits for you.

Going further: the two modes · the choice of LangGraph · the graph in detail · the router · the plan and its parallel execution · the planner and bulk operations · the semantic router · tool manifests · a pipeline failure is said, never replayed · reporting tool failures · turns that outlive the tab · ReAct mode

4. Acting with care

For MCP, you choose confirmations at account, server and tool level. What it will do explains the effective rule before you change it. A dedicated agent asks before its task when one of its tools requires approval; that answer covers its calls for the task. A tool’s “never ask” choice is suspended when its declaration changes. The chat may strengthen these protections; only Settings can reduce them. A routine may use the servers you explicitly pre-approved in its form.

LIA asks before what is actually going to happen: each draft (an email, an event, a deletion), each bulk operation that changes at least one item, each tool that declares it owes a confirmation. Confirmation is asked action by action, where the action is prepared, and not on the whole plan. Conversely, an action that nothing can divert asks for nothing: an email you send to yourself goes out directly.

What you confirm is exactly the last content shown, and each change runs only once. Several drafts in the same turn are reviewed one by one. You can reply in your own words: a model classifies your answer as agreement, refusal or a request for changes. Outside a conversation, a ticket asks for its confirmation on the ticket itself, and a routine running with nobody in front of the screen asks no question: it refuses the action and says so, otherwise your next message would pass for an answer. A command given to the chat ("switch my answers to cards") follows the same rule, and when it needs an account connection, a key or a file to be chosen, that step happens in the authenticated browser. LIA checks the requested outcome against what she can observe, but a delegation to a tool does not by itself prove all of its internal effects. Likewise, a lock prevents a double execution without being able to guarantee that an outside provider will act only once.

The answer itself is framed: the model writes only from the tools' real results. And everything LIA does is recorded in three separate registers: actions, consultations and turns. Their totals do not add up, since one turn can consult five sources without changing anything. An action is recorded before it happens and closed only on an explicit result. Erasing a conversation removes the files you sent in it, not the ones LIA produced for you.

Going further: confirmations · the authority chain and the registers · the registers on the user side · tickets · application commands · what a delegation proves · tool examples, email to yourself and drafts · synthesizing the answer

5. Memory and context

When you refer to a result by its sender, title or position, LIA first uses the items it actually showed. If several fit, it asks which one. A planned lookup that would act on the first of several matches also stops for your choice, then continues with that item.

A conversation's state is a MessagesState that LangGraph checkpoints in PostgreSQL after every node. That is what lets a turn resume after a confirmation, even hours later. Tool results do not live in the message history but in a separate data registry, so they survive when the history is truncated. Past a threshold, older exchanges are summarized; a badge shows how full the context is, and /resume triggers that summary on demand. In ReAct mode, what a loop has already read is checkpointed and reused within the turn; a stop before that checkpoint can cause a source to be read again.

Long-term memory, knowledge spaces and tool context live in the same database, with pgvector. After each exchange, a background job extracts memories from the last message; memories close to the question come back into the next context, and memory is also a tool that the planner and ReAct consult. A lasting instruction ("answer more briefly") is pinned and visible. When a sensitive subject comes up, a directive forbids joking or minimizing.

Around this memory, I added several layers. The journals are LIA's notebooks, in the first person; a portrait of the person emerges from them, with the sources that fed it, and it can only be corrected through indirect levers. The Psyche Engine gives LIA a psychological state that evolves with every exchange and shows in her style without being announced; it also learns, domain by domain, where she is reliable, from evidence; it is a pilot, with no gain claimed for now. It has an acknowledged limit: it almost never reports a calm, low-intensity emotion, and that part of its activation stays locked. Interests are detected in your requests and fed by several sources. Habits, finally, are learned without a trained model, through statistical tests, and I had three reasons, each sufficient on its own: a Raspberry Pi has no training budget, a formula can be published to the person it concerns, and on such small volumes a model would mostly learn noise.

Going further: checkpoints · the data registry · compaction · ReAct's checkpointed reads · memory in PostgreSQL · long-term memory · memory as a search tool · lasting rules · journals and the portrait · the Psyche Engine · its measured limit · the psyche on the user side · efficacy learned from evidence · interests · habits · searching the journals and the registers

6. The models

A single factory instantiates the model of any supported provider and applies its adapter. Each role has its model slot (conversation, planning, images, transcription, speech synthesis…), set in the administration without redeploying. The model catalogue lives in the database and declares, model by model, its capabilities, the parameters it accepts and the reasoning scale it knows; the configuration screen therefore offers only what the model's API really accepts.

Some tasks are not generations but bounded decisions: choosing a format, qualifying an email. They can be handed to a native decision engine, each behind its own switch and a general switch. An uncertain decision falls back to the usual processing, and neither rights nor confirmations are delegated to it. A qualification verdict authenticates no sender, and a gain measured on a test corpus is no promise for every request.

Every model call, in the pipeline as in background jobs, is tracked and priced from the rates in the database, then grouped by turn. On cost, the main saving comes from the providers' cache. Each prompt places its stable content before a marker and the request-specific content after it, and the infrastructure translates that marker into each provider's caching mechanism. Since every slot can change model, I do not tune this cache for any one model: I aim for the best average, and a model without a cache pays nothing extra. The studies run on other local engines (oMLX, llama.cpp, LM Studio) are qualifications, not shipped integrations: only the providers the instance offers count.

Going further: model selection · the catalogue in the database · the configuration screen · native decisions · their integration · qualifying emails · one layout for every cache · filtering the tool catalogue · the local engines studied

7. Connectors and tools

Each connector category (email, calendar, contacts, tasks, files, home) is a Python protocol. Only one provider is active at a time per category, and OAuth consent is given once per Google or Microsoft account. Services that ask nothing of the person (Wikipedia, page reading, places, weather, web search where deployed) belong to the instance and are active for everyone. Email summaries are reused, reply status is shown, drafts follow a style learned on request or keep explicitly supplied wording, attachments are read as text or images, and an unacknowledged send is never repeated.

Tools rest on five composable layers: a common connector base (OAuth, cache, injection), a decorator, per-domain formatters, the manifest the planner reads, and a catalogue loader. MCP tools are converted into ordinary LangChain tools and join the same catalogue, kept from one message to the next so that a message usually waits for no server; an MCP server must use HTTPS, its credentials are encrypted, and LIA believes an annotation that announces a modification, never one that claims to be read-only. Plugins in the Agent Plugins format install as a single block, with no local server. Skills following the agentskills.io standard can render a small app in the chat or run their own commands in a copy of their folder; a skill from elsewhere runs isolated and never receives a connector's token.

When a step needs a real calculation, ReAct mode can write a short Python script and run it in a disposable container. This script reaches the web only through a single proxy, does not see the person's key (the proxy substitutes it on the one authorized host), and an unknown host is asked about with three possible answers. In both modes, calculations, durations and currency conversions go through exact tools rather than the model. An agent can also drive a browser (Playwright) or convert a page into text.

Documents in knowledge spaces are chunked, embedded and stored in PostgreSQL rather than in an external vector database, then retrieved by a hybrid search, semantic and lexical. A space can follow a Google Drive or OneDrive folder with its subfolders, or a Gmail label, without indexing the whole mailbox. There is no character recognition: a scanned PDF is not indexed as is. Data sent from outside (SMS, step count, heart rate, from an iPhone, an Android or any system able to send JSON) comes in through endpoints authenticated by a limited-scope, revocable token, separate from the session.

For a complex task, LIA delegates to ephemeral, read-only sub-agents. She also produces files: documents, images, screenshots, filed in a gallery, kept, sent by email or, for an image, shared with a person connected to the instance.

Going further: Tools · interactive pages · connector patterns · grouped OAuth consent · the instance's services · connectors on the user side · emails and their summaries · attachments · per-slot timeouts · the tool base · MCP · the Agent Plugins standard · plugins · skills · skills on the user side · third-party skills · editing a skill · sharing a capability · ephemeral Python · the sandbox's network egress · computed answers · the browser · page reading · knowledge spaces · their guide · their use · a Drive folder is a tree · a Gmail label as a source · the built-in FAQ · attaching a document from a space · sub-agents · health data · its ingestion · health on the user side · SMS · document generation · documents on the user side · images · generated files · keeping a file · sending by email · composer attachments · kept answers · answered mail · writing style · web search

8. Voice, Live mode and phone

Classic voice is made of independent building blocks. The wake word "Dis LIA" is detected in the browser and sends nothing before it is heard; only the French model ships for now, in beta. Dictation is transcribed in the account's language, on the server and offline with Whisper, or by a remote engine depending on the setting. Speech synthesis has several providers, free Edge by default, and the answer starts being read sentence by sentence while the rest is still being written. An experimental talking avatar can be added, on the person's Simli key, which also bills silences as long as the connection stays open.

Live mode is a real speech-to-speech conversation. I gave it two intelligences and a single seam: the voice model carries the conversation, and every request for data or for an action goes to the chat engine through a single function, send_to_lia. The request becomes an ordinary chat turn, with its confirmations, its registers and its quotas. The voice owns no tool, no credential and no memory of its own.

A Live session falls asleep instead of dying: during standby, the connection to the provider is closed and nothing is billed. The Live provider's spend is paid by the person's key and is never recorded by LIA. In Live direct, the voice reads your data itself without changing anything.

The phone takes up the same two modes. LIA can call a third party for you after confirmation, or call you on a verified number, and you then have your data and all her functions on the phone, as in the chat. When LIA calls a third party, the calling agent has only a read-only free/busy tool: the protection of your calendar is structural, it does not depend on an instruction in a prompt. On the phone, Live mode exists only if the provider can reach your instance.

A meeting is recorded from the browser: the audio leaves in small segments, then it is transcribed and summarized into structured minutes, filed in a knowledge space. The personal radio, finally, produces only during a listening session and keeps nothing for a later replay.

Going further: voice on the user side · Live standby and the wake word · voices and speech synthesis · the talking avatar · Live mode · telephony on the user side · the phone as a channel · the radio · meetings · structured minutes · minutes templates

9. Initiative

LIA takes initiative in three forms. A periodic heartbeat looks at your sources and writes to you only if a signal warrants it, naming one or two verified facts. Being connected to a source and agreeing to be interrupted by it are two separate decisions. A push notification from Google can wake this heartbeat for a single person, and a moment (the end of a meeting, the arrival of an expected email) is served to the minute rather than at the next pass. Proactive discussions, finally, call for a reply, with their own policy: hours, a daily maximum, sensitive topics kept private, spend reserved before it happens.

Routines have one clock: a schedule, a condition (a public web page among them, read without a model) or another system's signed call, and a week view shows what actually happened. Reminders can repeat; their card snoozes an occurrence or stops the series after confirmation. The ticket board gives a task a lifecycle, a holder and a result.

Two rules bound all of this. Initiative only reads: a write is proposed, never executed on its own. And a new attempt at the same run is refused as soon as the effect register shows the action already done. The Activity page shows in one place everything LIA undertook on her own, with exact totals.

Going further: the heartbeat · permission to interrupt · push wake-up · moments and watches · proactive discussions · their delivery · their context · their budget · routines · the routines' week · reminders · a reminder that always has a text · read-only initiative · the ticket board · the Activity page · rating a notification · signed calls · watched pages

10. Security and privacy

Security is designed as independent layers, each type of attack having its own protection. Accounts sign in through OAuth 2.1 and rights follow roles. For sessions, I chose the BFF pattern: an HTTP-only cookie and a session in Redis rather than a JWT, which would be exposed to XSS if it lived in the browser, heavy to carry and impossible to revoke. An account can be protected with passkeys and a second factor, each connected device can be revoked, and linking a Google identity never changes an account's status. Usage limits apply at every door, routines included.

Everything that runs for the person is treated as hostile: skill code runs in a disposable container, a page the model writes runs in a frame with no origin and no host of its own, and a task on a remote server is prepared and then confirmed. Likewise, what LIA reads remains data and does not become an instruction. Provenance is a property of the data: each content type is classified once, an unknown type is treated as external, and a type nobody has classified keeps the application from starting.

Above debug level, a log line carries facts (numbers, identifiers, codes, the host of an address), not what you wrote or what a model wrote about you. Two caveats: the error message of an outside service in an unknown format can still quote a fragment of it, and the debug level, switched off in production, keeps more. The database is backed up automatically: at most one day of data can be lost with the default settings. The off-site copy and the attachments stay outside these backups.

Two limits are worth knowing. These measures do not on their own demonstrate a deployment's GDPR compliance. And deleting raw measurements, such as your health data, does not delete the memories derived from them, which have their own controls.

Going further: security · BFF authentication · passkeys · devices · usage limits · the sandbox · content provenance · privacy on the user side · logs without personal data · a log line carries facts · backups · a memory's provenance

11. Observing and measuring

Monitoring ships with the deployment: Prometheus and Grafana for metrics, Loki for logs, Tempo for traces, Langfuse for model calls, Alertmanager for alerts. The code defines hundreds of metrics, some thirty dashboards ship ready to use, and metric emission is defensive: a metric that fails does not bring down a request.

I also put debugging in the interface, because an agent is hard to debug: for each exchange, the debug panel shows the intent, the plan, each tool and each model call, and you can see why a request was expensive or why a given domain was chosen without opening Grafana.

On costs, every euro the platform pays for a person is traced, shown, attributed and counted for them, however it was spent. An instance spending cap exists alongside the per-account quotas, because several accounts multiplied by their quota are still an unbounded spend. An operator turns a capability off without redeploying: what the deployment allows and what the operator chooses are combined, and the smaller of the two bounds wins.

LIA also reads her own telemetry to diagnose herself: each vital alert is sent by email with its procedure and becomes an incident, and her recommendations remain proposals. An administrator can even drive Claude Code CLI from the chat to examine their server.

A note on performance: the measured latencies are the infrastructure's. The response time you feel depends mostly on the chain of model calls, from a few seconds to several tens of seconds depending on the request and the hardware. And scaling beyond a single machine (several nodes, replicas, clustered Redis) remains a possible evolution, not a reality. Two questions remain open: the growth of a worker's memory over several days and the retention of LangGraph's checkpoints.

Going further: the observability stack · the debug panel · its architecture · the instance cap · every spend answers to two caps · administration by workbook · targeted broadcasts · the demo instance · self-diagnosis · alerts and their resilience · Claude CLI from the chat · the anatomy of a worker · complete register extractions · how background jobs are organized

12. The interfaces

The badge in the middle of the chat header opens Main and the named discussions. A new discussion takes your title or the beginning of your first message; switch to it, rename, export or delete it from the list. Main receives automatic messages and offers Reset. The session badge reads the selected discussion's tokens, cost and context. Account preferences, memory and the journal apply across discussions. A message written while LIA works waits its turn, or joins the answer in ReAct mode.

The Tools workspace has its own REST endpoints, outside the conversation graph. It reuses the connector clients and publishes their supported controls and limits to the interface. Database sessions close before provider calls; submissions carry replay protection, and a lost answer is shown as an unknown outcome. HTML mail stays inside an isolated frame, with remote content blocked. This keeps a manual gesture separate from an action delegated to LIA; what a click asks of it from these screens goes through a door of its own, one call and one proposal, never a write at the provider.

The web interface is a Next.js application that installs like an app (PWA). The chat updates the conversation without reloading, renders math formulas and syntax-highlighted code, keeps one draft per discussion, and can be searched; the settings have their own search, a pure black theme for OLED screens that can follow your hours, three icon styles and an adjustable text size. One selection layer serves every screen: selected text is explained, translated or handed to the usual form, never to a second one, and a replacement keeps the browser's undo.

In the cards display, one of three modes to choose from along with plain HTML and Markdown, an answer that carries data is not formatted by the model: the server renders it as deterministic HTML cards, independent of the model and the mode, archived with the message. A card shows a received fact in full and invents nothing, and an action from a card prepares a request instead of acting. To keep the interface consistent, I wrote rules that tests check: a status takes its tone from a single table, an action has a level of importance, and a list row exposes its actions in a single way. Browser and accessibility tests target WCAG level AA, without certifying the rendering on a physical device or the accuracy of the translations.

The mobile apps are native shells published once per store. They load the server address entered at first launch and do not duplicate the interface, because a single app must serve everyone who hosts LIA. On Android, notifications go through your own server's Firebase project; on iOS, where only an app's publisher can send notifications, a minimal relay wakes the phone. Sharing a text or a link to LIA from the phone prepares a chat draft, which does not go out on its own.

Around the chat, a companion with expressive eyes reacts to the whole cycle of an answer, and each answer carries a small mood avatar. LIA World is another interface: a shared house and garden where you stroll around, without any room granting extra rights. Voice between people and federation between instances are not implemented there. The presentation video, for its part, shows interface replicas and demonstration data.

Two people on the same instance can link their assistants so that they pass messages to each other, and the Relations page gathers what concerns each person, with a debrief written by the model.

Going further: the chat on the user side · the settings · searching the settings · getting started · the chat that follows the server · cards · their rendering · places and routes · the known location · primitives and their contract · the Android app · native notifications · the apps on the user side · the expressive eyes · native companions · the presentation video · the capability map · LIA World · its prototype · connections between assistants · relations · a relationship's debrief · Telegram · discussions · waiting messages

13. How this code is kept in shape

Every change passes local gates (before the commit, before the push) and then continuous integration on GitHub Actions, which calls exactly the same commands as a developer. Python dependencies are intent manifests compiled into universal lockfiles that carry each package's hash: the same commit always installs the same verified versions, on every architecture.

The project's rules do not rely on the memory of whoever reviews: automated guards hold them. Several of them recompute the protection from the source of truth (the code itself, the routes actually mounted, the connector registry) instead of pinning a handwritten list, and were deliberately broken to check that they really fail. Elsewhere, reference lists can only shrink, so that no debt grows in silence. Prompts are versioned text files, one per prompt, read by a single loader. More than 350 ADRs record the decisions, the alternatives set aside and the trade-offs accepted, and tens of thousands of tests, continuous integration and strict typing let the system evolve without regressions.

A full 360-degree technical audit is published in the repository with its protocol, so that anyone can run it again. How this code was written, almost entirely by an AI that was directed and checked, is told in the field report.

Going further: continuous integration · dependency locking · versioned prompts · the public audit

14. Going further

This guide draws on the code, on more than 800 technical documents and on the ADRs. To go further:

The code is under AGPL-3.0, on GitHub.