Several months ago I was flipping through game reviews when I came across an interesting new title called Whispers From the Star. The premise of the game is that you accidentally make contact with an astronaut on a far away planet, and communicate with her in order to help her survive. The creators claim the game is fully AI-driven, and reacts based on the players’ input on a number of levels that adds to the immersion of the experience. It’s an interesting study in human/AI interaction.
So, for the past few months I have been working on a fully local AI companion running on Ubuntu with an NVIDIA GPU, loosely inspired by the game interface, with one twist: no cloud services, no API keys, no data leaving the machine.
The surface layer is a 3D animated character built in Godot 4, voice-driven, with real-time lip sync and facial expressions responding to conversation state. This is really just a placeholder for me at the moment, as I am no artist, but the more interesting work is underneath it.
Aria maintains a persistent memory of personal interactions: journal entries written after each conversation, a relationship model that accumulates interests and personality observations across sessions, and an emotional state that drifts slowly based on how recent conversations have gone. The system even reading things while idle, searching topics from past discussions and occasionally surfacing something worth sharing without being asked.
Obviously this is a lot of data, so I have had to come up with a way of weighting and depreciating information so that the model isn’t overloaded. I have spent years working with data, and even then this has been a master class in representation, classification, storage, retrieval, and the technology behind it.
One of the more interesting research threads running through the project is the application of Pearl’s causal framework to human-AI relationship dynamics. After each conversation, structured observations are recorded: engagement level, depth, mood signals, conversational interventions. A pattern detector identifies statistical regularities in the accumulated data. A reasoning pass interprets those regularities as causal hypotheses with mechanisms and falsifiers. The system is currently in Phase 1 (observation only); Phase 2 introduces interventions, and Phase 3 introduces counterfactual reflection.
This extends directly from the MSc thesis work. TutorAlert detected student frustration from discussion posts — Level 1 on Pearl’s Ladder of Causation. Aria is an attempt to reach Levels 2 and 3: not just detecting states, but reasoning about what causes them and what would change them.
Built with: Ollama (gemma2:9b / phi4:14b / qwen2.5-coder:7b), faster-whisper, Piper TTS, Rhubarb Lip Sync, Godot 4, Python asyncio, DoWhy, pandas. I am interested in seeing whether the causal layer produces intervention strategies that are genuinely useful, and not just descriptions of what happened but guidance on what to do differently.
In active development, more to come.