The Brief
Details of this engagement are under a non-disclosure agreement. What can be shared is the nature of the work and the technical direction — without identifying the specific product, features, or any proprietary architecture.
The client is a venture-backed AI company, funded and operating. The product involves character-driven conversation: AI systems that maintain consistent personality, memory, and tone across extended interactions.
This is technically harder than it looks.
The Core Problem
The central challenge in character AI is not making an AI that can impersonate a personality. Modern LLMs do this adequately out of the box. The challenge is making an AI that maintains that personality consistently across:
- Multiple conversation sessions — memory and continuity across sessions that may be days apart
- Edge cases and adversarial prompts — robustness when users push against the character's limits
- Different user interaction styles — adaptability without drift; the character responds differently to different users without becoming a different character
- High-concurrency production load — scale without degradation
"The hardest problem in character AI is not generation. It is evaluation."
What Was Delivered
The specific deliverables are under NDA. The technical areas covered:
- Real-time chat infrastructure built for strict latency budgets with streaming as a first-class requirement
- Evaluation pipeline for qualitative properties — measuring whether a response feels like the character, at production scale, using a combination of automated scoring and structured human-in-the-loop workflows
- Production observability tooling designed for LLM systems: token efficiency, prompt version performance, character consistency scores over time, latency distribution by model and template
- Continuous evaluation loop: production conversations collected, scored against quality criteria, failure modes identified, prompts updated — on a same-day cadence
Why Evaluation Is the Hard Part
Standard APM tools do not capture what matters for LLM systems. Building the right observability for an LLM product requires knowing what to measure, which is not obvious.
The evaluation pipeline built here is the layer I am most proud of. Automated evaluation of qualitative properties — does this response feel like this character? — requires moving beyond simple metrics toward structured human-in-the-loop workflows combined with LLM-as-judge scoring. Getting the calibration right between automated signals and human judgment is where the real work is.
The methodology for measuring LLM system quality in production — not just benchmark performance, but operational quality — is something I now bring to every new engagement.
The Outcome
The engagement is active. Specific metrics are under NDA. The work is shipping to real users.
Lessons
Character AI exposes a problem with standard prompt engineering practices: prompts that work in development fail in production because the distribution of real user messages is nothing like the distribution of test messages you wrote yourself.
The solution is a continuous evaluation loop: collect real production conversations, score them against your quality criteria, find the failure modes, and update the prompts. This loop needs to be fast — ideally same-day — or the product degrades faster than you can fix it.
Ship the evaluation system before you need it. By the time you notice degradation in production, you are already behind.