ORACLE RT
Current status: ORACLE RT exists and runs. Wiring its output into live avatar conversations is still in progress, so this page describes the working engine and its intended live integration, not a fully live production connection.
ORACLE is the full cross-modal pattern recognition engine of the OPM pipeline. It runs dozens of rules across all three channels, evaluates intra-session patterns, and produces deep behavioral findings. For offline analysis and research, it's unmatched in depth.
But ORACLE was designed for depth, not speed. Running the full rule library on every frame of a live conversation would introduce latency that breaks the experience. A live avatar can't wait 2 seconds for pattern analysis when the person across from it expects natural conversational pacing.
ORACLE RT solves this. It's a real-time pattern recognition engine built specifically for live sessions, operating within the latency budgets that conversational interaction demands. Same principle as ORACLE (cross-modal behavioral intelligence), different architecture for a different problem: perception at the speed of conversation.
A realtime listening pass built for the same page, with tighter pacing and a denser control surface.
The hard part is doing the right analysis in the right time
The challenge isn't computational power. The challenge is doing the right analysis in the right time.
ORACLE's full pipeline processes session chunks, evaluates temporal patterns across minutes, and cross-references findings from different points in the conversation. This multi-pass approach produces rich, nuanced findings. But it requires holding large windows of data in memory, performing multi-step evaluations, and generating findings that reference patterns spanning minutes or even the full session.
In a live conversation, you can't wait for minutes of data to accumulate. You need pattern recognition that operates on the current frame, the last few seconds, and a running behavioral context. The analysis window is narrow. The delivery must be immediate. And the findings need to be useful enough to influence the avatar's next response.
ORACLE RT is built for exactly this constraint. It evaluates incoming signals against a focused rule set, maintains a lightweight running context, and delivers findings within milliseconds. Where ORACLE looks at the full landscape after the fact, ORACLE RT looks at what's changing right now.
Focused Rule Set
ORACLE's full rule library contains dozens of cross-modal rules covering a wide range of behavioral phenomena. ORACLE RT uses a targeted subset: the rules most relevant for real-time conversational interaction.
These rules prioritize phenomena that change quickly and that influence how a conversation should proceed: sudden divergence between channels, rapid shifts in engagement signals, hesitation markers, and significant deviations from the person's established baseline. Rules that require long temporal windows (patterns that only become visible over minutes) are handled by ORACLE in post-session analysis, not by ORACLE RT during the live session.
The focused rule set isn't arbitrary. It's based on which behavioral patterns an avatar or live system can actually respond to in real time. Detecting that someone's vocal pitch spiked in the last 3 seconds is actionable. Detecting that their overall session showed a gradual trend toward decreased engagement is valuable for a post-session report, but the avatar can't do anything with that information mid-conversation.
Frame-Level Evaluation
ORACLE evaluates session chunks. ORACLE RT evaluates frames and short signal windows.
When CYGNUS Lite (or any signal source) emits a set of values, ORACLE RT immediately evaluates those values against its rule set. The evaluation considers:
The current frame's signal values across all active channels.
A rolling buffer of recent signals (typically the last 5-15 seconds), providing short-term temporal context.
The person's CANON baseline, if available, for personal deviation detection.
This evaluation produces findings in real time: "Cross-modal divergence detected in current window" or "Significant pitch deviation from personal baseline" or "Hesitation cluster in progress." Each finding includes a confidence score and the specific signals that triggered it.
Lightweight Running Context
ORACLE RT maintains a minimal running context that tracks the session's behavioral trajectory without the memory overhead of ORACLE's full session model. This context includes:
The number and type of findings so far in the session.
Whether specific patterns have recurred (e.g., divergence detected earlier and now detected again).
The general direction of key behavioral metrics (trending up, stable, trending down).
This running context allows ORACLE RT to add temporal depth to its findings ("this is the third divergence detected in this session") without the computational cost of maintaining a full session analysis model.
Same principle. Different architecture for a different problem.
Understanding the relationship between these two is important. They aren't competing. They're complementary.
Structured AU input from any source
ORACLE RT accepts structured Action Unit data from any source. It doesn't require CYGNUS Lite. If you have your own face analysis pipeline, your own signal extraction system, or any other source of AU data, you can plug ORACLE RT in directly.
This is by design. ORACLE RT's input interface is a simple structured format: Action Unit identifiers, intensity values, and optionally a dominant emotion classification and a video frame for secondary verification. Any system that can produce these values can use ORACLE RT.
Input: { "AU04": 0.57, "AU07": 0.65, "AU45": 0.75, "dominant_emotion": "anger" }- Corrected emotion: "anxiety_tension" - Pattern match: AU04 + AU07 without AU05 = cognitive load pattern - Correction: "anger" -> "anxiety_tension" based on AU evidence - Summary: Natural language description for LLM injection
Why naive emotion labels need behavioral correction
Why does ORACLE RT need to correct emotion classifications?
Most face analysis systems use machine learning models trained on labeled datasets. The model sees a face and outputs a label: 'anger,' 'surprise,' 'sadness.' These models are trained on facial expressions that humans labeled as representing those emotions. They work reasonably well on clear, prototypical expressions.
But in real conversation, expressions aren't prototypical. They're subtle, mixed, transient, and context-dependent. A furrowed brow doesn't always mean anger. It means the brow is furrowed. Whether that indicates anger, concentration, confusion, or physical discomfort depends on what else is happening in the face, the voice, and the body.
Emotion classifiers can't make these distinctions because they only see the face and they only output labels. ORACLE RT can make them because it evaluates the full Action Unit profile against behavioral patterns documented in research. When the classifier says 'anger' but the AU combination says 'cognitive load,' ORACLE RT trusts the AUs.
This correction capability is why ORACLE RT is so valuable for avatar and agent pipelines. Language models that drive avatar responses are sensitive to their input. If the perception system tells the LLM 'the user is angry,' the LLM will respond to anger. If it's actually concentration, that response is wrong. ORACLE RT ensures the LLM gets accurate behavioral context, which produces dramatically better responses.
A complete live perception stack developers can actually integrate
ORACLE RT is available as an open-source component. Developers building AI avatars, virtual agents, interactive systems, or any application that needs real-time behavioral intelligence can integrate ORACLE RT into their pipelines.
The open-source release includes:
The core rule evaluation engine.
A base rule set covering the most common conversational patterns.
The correction logic for cross-referencing AU data against emotion classifications.
The structured output format for LLM context injection.
For teams that need deeper behavioral intelligence, the full ORACLE library (with its complete rule set, full temporal analysis, and deep cross-modal evaluation) is available as part of the enterprise OPM deployment. ORACLE RT provides a strong foundation. ORACLE provides the full depth.
The combination of CygnusLite + ORACLE RT as an open-source package gives developers a complete real-time perception pipeline. Signal extraction plus behavioral intelligence, available to anyone building systems that need to understand human behavior in real time. Nothing else like this exists as an open, integrable stack.
Behavioral findings become LLM-ready context
In the avatar pipeline, ORACLE RT's findings flow through GLUE to the language model. GLUE formats ORACLE RT's structured output into natural language context blocks that the LLM can incorporate into its response generation.
This block arrives in the LLM's context alongside the conversation history. The model can now see that the person is processing something carefully, speaking more slowly than usual, and pausing more. A well-tuned avatar will give the person more space, ask deeper questions, and avoid rushing the conversation.
Without ORACLE RT, the LLM is blind to these signals. It responds only to the words. With ORACLE RT, it responds to the words and the behavior. The difference in interaction quality is substantial.
[PERCEPTION - Current] Behavioral state: Cross-modal convergence. Face, voice, and body aligned. Notable: Speech rate 22% below personal baseline. Pauses lengthening. Pattern: Consistent with careful processing or deliberation. Confidence: 0.79
Integration, latency, and open-source depth
No. ORACLE RT accepts structured AU data from any source. CYGNUS Lite is the recommended pairing for a complete real-time perception pipeline, but ORACLE RT can work with any system that produces Action Unit data.
Real-time behavioral intelligence in one frame
ORACLE RT is the real-time pattern recognition engine of the OPM system. For the full-depth pattern recognition layer, see . For real-time signal extraction, see . For information about how perception data is used across our products, see our .