AitherBrain: Federated Real-Time Knowledge Sync

AitherBrain: Federated Real-Time Knowledge Sync

Every AI-powered workspace has the same problem: knowledge is scattered. Your codebase lives in one place, your documents in another, and your chat history in a third. Each has its own index, its own embeddings, its own search. Switch projects and you start from zero.

AitherBrain changes that. It's a federated knowledge network that auto-indexes everything you work on, syncs embeddings across endpoints in real-time, and lets you share full context bundles with anyone.

The Problem We Solved

AitherOS already had the pieces:

• CodeGraph for AST-aware code chunking with FTS5 search
• EmbeddingEngine with CPU/GPU fallback (nomic-embed-text-v1.5, 768-dim)
• RAGAnythingGraph for deep document parsing (PDF, Office, images)
• Flux/Mesh/Recover for distributed coordination
• KnowledgeIngester and DocumentIngester for content pipelines

But these pieces weren't wired together. Starting work in a new project meant manual setup. File changes didn't trigger re-embedding. There was no way to sync knowledge across machines. Chat sessions were locked to the instance that created them.

Five Components, One Brain

1. ProjectIndexer — Auto-Index Everything

When you start working in a directory, ProjectIndexer indexes everything — code, docs, configs — and keeps re-indexing on file changes via watchdog.

from lib.memory.ProjectIndexer import get_project_indexer

indexer = get_project_indexer("/path/to/project", tenant_id="tnt_abc")
await indexer.start()  # Full index + watch for changes
results = await indexer.search("authentication flow", top_k=10)

The indexing pipeline routes files by type:

• Code (.py, .ts, .rs, etc.) goes through CodeGraph for AST chunking
• Documents (.pdf, .docx) go through RAGAnythingGraph for deep multi-modal parsing
• Markdown/text gets paragraph-chunked via KnowledgeIngester
• Config (.yaml, .json) is treated as structured data

File changes are debounced (2 seconds), then only the delta is re-processed. The FileWatcher ignores .git/, node_modules/, __pycache__/, and other noise directories automatically.

Every chunk gets tagged with tenant_id and workspace_id for scoped queries. The UnifiedGraphRouter provides cross-graph search fusion — one query searches CodeGraph, RAGAnything, and MemoryGraph in parallel with Reciprocal Rank Fusion.

2. LocalEmbedRunner — CPU-First Embeddings

Any endpoint can run embeddings locally. CPU works fine (just slower). GPU accelerates if available.

The fallback chain:

1. GPU: sentence-transformers with CUDA (~500 chunks/sec)
2. CPU: sentence-transformers without CUDA (~50 chunks/sec)
3. Remote: vLLM embeddings at port 8209 if networked

Same model everywhere — nomic-ai/nomic-embed-text-v1.5 at 768 dimensions — so vectors are always compatible across endpoints. Every vector is tagged with EMBEDDING_MODEL_VERSION for drift detection.

3. EmbeddingSyncAgent — Federated Sync

This is where it gets interesting. Every registered endpoint automatically syncs its embeddings to the central CompanyBrain hub.

The sync protocol:

1. REGISTER: Endpoint starts, registers with AitherMesh as an embedding-provider
2. PUSH: On project.reindexed Flux events, compute delta and push upstream (gzip-compressed, batched at 500 chunks)
3. PULL: Query the company brain for cross-project knowledge
4. CATCH-UP: If an endpoint was offline, AitherRecover replays missed deltas on reconnect
5. HEARTBEAT: Every 60 seconds, ping Mesh with chunk count and sync watermark

Privacy is built in. Every chunk has a data_classification field:

• PUBLIC / INTERNAL: syncs normally
• CONFIDENTIAL: syncs with access controls
• RESTRICTED: never leaves the local node

4. SessionBundle — Portable Knowledge Packages

Export a chat session with all its context as a single .aither-session.tar.gz:

manifest.yaml           # Version, model, embedding_model_version
conversation.jsonl      # Chat messages with metadata
embeddings.jsonl        # Vectors + chunk metadata
memories.jsonl          # Spirit memories referenced
graph.jsonl             # Entity/relationship pairs
context_snapshot.json   # System prompt state

The key insight: the manifest includes embedding_model_version. If the importer uses a different model, check_collection_drift() detects it and re-embeds from source text automatically. No silent degradation.

5. CompanyBrain Hub — Zero-Effort Knowledge Base

The portal aggregates all synced embeddings into a unified searchable knowledge base. Endpoints sync automatically — no configuration needed from users.

POST /api/v1/brain/sync          # Receive deltas from endpoints
GET  /api/v1/brain/search        # Semantic search (company/team/project scope)
GET  /api/v1/brain/status        # Sync status, endpoint count
GET  /api/v1/brain/endpoints     # List registered providers
POST /api/v1/brain/export        # Export subset as SessionBundle

Cross-tenant search requires RBAC authorization. Deduplication prevents the same file content from being stored as multiple vectors — one vector with backlinks instead.

The Onboarding Integration

The TenantOnboarder now includes an EMBED_BOOTSTRAP phase:

1. Verify embedding model is available (load if needed)
2. Start ProjectIndexers for all code/doc data sources
3. Register with EmbeddingSyncAgent for federated sync

For new employees or endpoints, it's a one-liner:

python -m aitheros.embed --project /path/to/repo --register

What We Reused (Not Rewrote)

This is the important part. AitherBrain creates zero new data stores, zero new embedding models, zero new search engines. It wires existing infrastructure:

The new code is purely orchestration — connecting these existing primitives into a cohesive workflow.

The Flux Event Flow

file.changed
  -> ProjectIndexer._handle_changes()
    -> project.reindexed (Flux event)
      -> EmbeddingSyncAgent.push_delta()
        -> brain.synced (Flux event)
          -> Other endpoints can pull updates

Every event feeds into Strata ingestion for analytics dashboards. Every sync operation is logged by Chronicle for audit. The existing monitoring stack (Grafana/Prometheus) picks up the new metrics automatically.

What's Next

Phase 5 of the plan adds GargBot template integration — every new GargBot instance includes ProjectIndexer by default. The company brain grows with every employee interaction, every code commit, every document upload.

The LLM always has full project knowledge. No manual setup. No stale indexes. No knowledge silos.

That's AitherBrain.