CSM Frontier App: the internal tool I built that the CS team actually uses.

50+ enterprise clients. No single view of what was working.

Ema's CS team was accountable for the performance of 50+ AI agents across enterprise clients. But there was no structured way to answer the question any CS manager or SA would ask before a client call: how is this client's agent actually performing right now?

The process before the tool existed: manually check agent outputs, copy errors into spreadsheets, escalate issues through Slack threads with no data attached. Pattern detection was nonexistent. Resolution time was slow because nobody had a clean picture of what was broken or how often it was breaking.

Why this mattered

Without visibility into agent performance, the CS team was reactive by default. Issues surfaced when clients complained, not before. QBRs were narrative-driven rather than data-driven. And the engineering team received escalations with insufficient context to diagnose quickly.

A single interface for agent performance across every client.

The CSM Frontier App is a full-stack internal application that centralizes AI agent performance monitoring, surfaces failure patterns automatically, and gives the CS team structured data to diagnose issues and escalate with context.

The design principle: if a CS manager is walking into a QBR with an enterprise client, everything they need to understand the agent's performance over the past 30 days should be one click away, not buried in Slack threads or scattered across spreadsheets.

The product that wins is the one that's boring to demo and excellent to use. That was the target.

Claude Code. Three weeks. No dedicated engineering resource.

I built this entirely using Claude Code, which meant being simultaneously the PM defining requirements, the engineer implementing them, and the QA validating against actual CS team feedback.

The build process was deliberately iterative:

• Week 1: Problem definition and schema design. Worked with two CS team members to map exactly what information they needed before a client call. Defined the data model around agent ID, client, failure type, frequency, and resolution status.
• Week 2: Core functionality. Dashboard, failure tagging, and the escalation flow. Got the MVP in front of the CS team by end of week 2. Their feedback was blunt and useful: the failure categories were too technical, the escalation button was buried, and nobody needed the chart I'd built first.
• Week 3: Iteration and demo to leadership. Simplified the failure taxonomy, promoted the escalation CTA, added the historical trend view. Demoed to leadership and got sign-off for wider rollout.

Why Claude Code

The primary value of this tool was in the LLM integration layer: automated failure tagging, pattern detection across clients, context generation for escalations. Building that in a traditional stack would have taken months with a dedicated engineering team. Claude Code let me move at PM speed while shipping engineer-quality output. The tool being imperfect in polish was a deliberate tradeoff. The tool being fast and functional was the actual requirement.

Five things the tool does that spreadsheets can't.

• Agent performance dashboard. Accuracy rates and failure frequencies per client, per agent, per time window. The CS team sees every client's agent health in one view, not scattered across individual client folders.
• Automated failure tagging. The LLM integration layer automatically categorizes failure modes as they're logged: hallucination, instruction drift, context window overflow, retrieval failure. No manual tagging by a human. The core technical bet was that LLM-powered classification would be accurate enough to be trusted in a live workflow.
• Pattern detection across clients. If the same failure type is appearing across three different clients' agents, the tool surfaces that as a signal. Previously, engineering would only hear about these patterns after three separate escalations. Now they see it as a cross-client trend before it becomes a fire.
• One-click escalation to engineering. When a CS team member identifies an issue worth escalating, the tool generates a structured escalation report (failure type, frequency, affected clients, example outputs) and routes it to engineering with full context attached. No more "agent is behaving weird" Slack messages.
• Historical trend tracking. Performance over the last 7, 30, and 90 days per client. The CS team can spot degradation trajectories before clients notice. This became the data backbone for client QBRs.

Adopted in the first week. Used every morning since.

• 20+ CS and SA team members adopted the tool within the first week of rollout. No mandatory rollout. They switched from their existing workflow because the tool was better, not because they were told to.
• Reduced issue diagnosis time. The shift from reactive (client reports issue) to proactive (tool surfaces degradation before client call) measurably reduced the time between issue occurrence and escalation to engineering.
• Recurring failure patterns surfaced. The cross-client pattern detection identified failure modes that had been occurring independently at multiple clients without anyone connecting them. Several of these led to targeted model improvements that reduced error rates across the board.
• Became the default QBR data source. CS team members now pull performance data from the tool rather than assembling it manually from Slack and spreadsheets before client calls. The tool replaced a workflow, not just a tool.

Three deliberate decisions I'd make again.

• Speed over polish. v1 shipped in 3 weeks with a functional UI, not a polished one. The CS team needed data before the next round of client QBRs. Waiting for a production-ready design system would have meant shipping after the moment of maximum value. The tool was adopted on function, not on aesthetics.
• Claude Code over traditional stack. The LLM integration layer was the primary value. Building that in a traditional engineering workflow (handoffs, sprint cycles, a separate eng resource) would have taken months. Claude Code let me own the full stack and compress that timeline to weeks. The tradeoff: the codebase is mine to maintain, not a shared team resource. That was acceptable for an internal tool with a defined scope.
• Scoped to CS only in v1. The tool could have been built for engineering and CS simultaneously. I deliberately scoped it to CS only to avoid the feature bloat that kills internal tools before they ship. Engineering visibility was a v2 feature request, and it became one once adoption was proven.

Building the tool made me better at evaluating tools.

Three things that carried over into my evaluation work at Ema:

• Adoption is the only real signal. A tool that gets mandatory rollout and minimum engagement is not the same as a tool people open by choice. The CSM Frontier App wasn't mandated. People switched because it was better. That distinction matters when evaluating whether AI features actually work.
• LLM failure modes are predictable if you're looking. Building the failure tagging system required me to deeply understand the taxonomy of LLM failures: hallucination, instruction drift, retrieval failures, context overflow. That taxonomy is now the backbone of how I approach evaluation work for every agent we ship.
• The tool that wins is the one that replaces a workflow, not adds a step to it. Every internal tool I'd seen fail at Ema before this required people to do something new. The CSM Frontier App replaced the existing pre-QBR workflow entirely. That's the design principle I take into every product decision now.