Capture → ingestion → runtime → intelligence → lifecycle → recall
MMRY system specificationLayers, contracts, and reference sketches.
This page is for engineers and researchers. It documents the layered pipeline from multimodal capture through ingestion, intelligence, adaptive resolution, and hybrid recall — with illustrative code contracts drawn from observed implementations and open canon decisions (D0, D15–D18).
Exact graph and vector index on one SQL adapter for talk; capsule index for vault
Index lookup must not require whole-file unfold (D15)
Full-stack memory pipeline.
Six functional layers. Data flows downward through ingestion and intelligence; resolution bands govern what ordinary recall may see.

What each box does.
Functional responsibilities — not product names.
Capture layer
Timestamped evidence enters with consent scope — not as asserted facts.
- Voice & conversation
- Photos, video & documents
- Apps, messages & activity
- Location, devices & sensors
Memory ingestion
Segments episodes, resolves entities, writes linked representations with epistemic status.
- Consent & privacy controls
- Multimodal extraction
- People, places & entity resolution
- SensAffect imprints
Private runtime
Edge-first processing with fail-closed writes and inspectable audit trails.
- Provenance, access & audit
- Pi 5 + Hailo-8 edge node
- Local models & storage
- Encrypted cloud bridge
MMRY intelligence
Core engine: compress paired information + affect; index for selective retrieve.
- Memory orchestrator
- Semantic compression
- Temporal & life timeline
- Semantic vector memory
- Knowledge–experience graph
Memory lifecycle
Human-like view bands. Retrieval probability decreases; compressed core may persist.
- VIVID → CLEAR → FADED → HAZY → FORGOTTEN
Recall & reasoning
Assembles inspectable context for the next action without whole-file unfold.
- Hybrid graph + vector retrieval
- Context assembly
- Personal memory twin
- Assistant / agent interface
Code contracts and examples.
Illustrative sketches — not production drop-ins. They encode the pass/fail tests from the canon: paired store, selective retrieve, separated affect, protected decay.
Episode boundary — one interval is one memory
Memory event boundaries mark start/end on a timeline. Object, face, place, and event markers share the same interval identity (RX-27).
interface MemoryEventBoundary {
eventId: string;
startTimeMs: number;
endTimeMs: number;
markers: Array<{
kind: "object" | "face" | "place" | "event";
label: string;
startTimeMs: number;
endTimeMs: number;
}>;
tracks: {
picture: string; // compressed segment ref
sound: string;
feeling: string; // MEMO / SensAffect imprint ref
pose?: string;
words?: string;
};
}
// Ingest attaches modalities to the same episode — not separate drawers.
function bindToEpisode(
boundary: MemoryEventBoundary,
imprint: AffectiveImprint
): MemoryEventBoundary {
return {
...boundary,
tracks: { ...boundary.tracks, feeling: imprint.imprintId },
};
}