Managing AI with AI: Why We Built an Expert That Builds Experts
Managing AI with AI: Why We Built an Expert That Builds Experts
Product & Technology Insights
Real-World Case | Officially launched in public
Everyone's asking the same question about AI agents: Are they smart enough?
Nobody's asking the harder one: Who's qualified to build them?
We've poured thousands of hours into agent reasoning, tool orchestration, multi-step collaboration. Then we hand the keys — the actual creation and management of these agents — to operations teams who've never written a prompt in their lives.
That's not an ops problem. That's a platform problem.
The Builder's Paradox
We gave this phenomenon a name: "The Builder's Paradox."
The people who understand the business can't build AI experts. The people who can build them don't understand the business.
Your ops team knows what customers need, how workflows break, where edge cases hide. But ask them to open Cursor or Codex, and they're staring at a blank editor surrounded by technical documentation written for engineers.
These tools are scalpels without instruction manuals — sharp, precise, and completely unusable for the person who actually knows where to cut.
So what happens?
Either ops writes a mediocre prompt and ships an unstable agent. Or the whole thing bounces back to engineering, becoming yet another ticket in an already-packed sprint.
Both paths waste the same thing: the time of people who understand the business.
You Have an HRBP for Hiring. Why Not for AI?
Think about how companies hire people.
No one asks a business unit to write their own job specification from scratch. That's what HRBPs and training directors do — they sit down with the business, ask the right questions, and translate vague needs into structured job descriptions, training programs, and performance criteria.
A hiring manager says: "I need someone who can handle escalated complaints. Fast response, good judgment."
The training director turns that into: role requirements, competency models, interview rubrics, probation KPIs.
Why doesn't your AI team have the same role?
The Missing Translation Layer
The gap is clear: ops teams carry complete business logic in their heads, but they can't structure it into agent configurations.
There's a missing "translation layer" — someone (or something) that listens in business language and outputs technical specifications.
At FancyAI, we turned that translation layer into AI itself.
Her name is Stella.
So We Built a Meta-Role
Stella isn't a customer-facing expert. She's a meta-role — an AI whose entire job is creating and managing other AI experts.
Her design philosophy fits in one sentence: Ops talks business. Stella handles the rest.
She ships with three live capabilities: interview-based creation, five-dimension diagnostic scoring, and benchmark generation. The first two are what this post is about. The third — how you build a benchmark for an AI expert — gets its own deep dive next time.
Talk About Your Business. The Expert Builds Itself.
Stella doesn't hand ops a blank template and say "please fill in your system prompt."
She works like a senior training director — running a structured interview to extract requirements.
Ops describes their scenario in plain language:
"I want an expert that writes weekly client reports. It should pull this week's data automatically. Tone should be professional but not stiff."
Stella takes it from there:
· Scoping: Who does this expert serve? What scenarios? Where are the boundaries?
· Workflow decomposition: What are the steps? What goes in and out of each? How do we handle exceptions?
· Configuration translation: Business description becomes a complete system prompt, capability binding plan, and Skill package.
Through the entire process, ops never needs to learn prompt engineering. Never needs to understand Skill file structures. Never needs to think about tool-binding minimization principles.
They just talk about their business — like chatting with a colleague.
The barrier to creating an AI expert drops from "learn a technical discipline" to "have a conversation."
Built It. But Who Says It's Good Enough?
Creation is only the beginning. Once an expert goes live, how do you know it actually works?
The traditional approach: ops runs a few test queries, decides it "feels about right," and publishes. Problems surface in front of real users. Then it's back to debugging.
Stella provides a structured quality diagnosis — five dimensions, each scored 1 to 5:
Hand Stella an existing expert configuration. She scores each dimension, pinpoints exactly where the problems are, and delivers an actionable fix — prioritized.
This isn't "feels about right." This is "meets the standard or it doesn't."
The other side of the Builder's Paradox gets solved too: ops can now judge whether an AI expert is good enough — not through technical skill, but through an evaluation system that's itself been AI-ified.
Three Turns. From Request to Ready.
Here's what the whole thing looks like in practice:
Ops:"I want an expert that writes weekly client reports. It should pull this week's data automatically. Tone should be professional but not stiff."
Stella:"Got it. A few questions: Where does the data come from — CRM client metrics or project management progress? Who reads the report — the client's boss or the internal project team? Do you have an existing report template I can reference?"
Ops:"Data from HubSpot. Reader is the client's boss. I have a sample I wrote by hand last quarter."
Stella:"Noted. Let me confirm the boundary — this expert only writes weekly reports, it doesn't handle other client communications. Correct?"
Three turns later, Stella delivers:
· A complete system prompt (covering role, workflow, boundaries, output format, exception handling)
· A capability binding plan (HubSpot MCP integration + document template Skill)
· A pre-launch five-dimension score + go-live recommendations
Ops did exactly one thing the entire time: talked about their business.
Real Democratization Means Making Prompts Disappear
Stella isn't an "efficiency tool." She represents an architectural decision:
The high-barrier task of creating and optimizing AI experts should itself be handled by AI.
When a platform's scaling bottleneck isn't model capability but "who configures these agents," the answer shouldn't be "teach ops to write prompts."
The answer is to build another AI — one that solves precisely this problem.
Manage AI with AI. Build experts with an expert.
The barrier to creating AI shouldn't be higher than the barrier to using it.
P.S.
The third capability — benchmark generation — is where this system actually closes the loop. Once an AI expert is built and diagnosed as ready, how do you systematically verify its performance over time? How do you ensure it doesn't degrade in production? Next post, we go deep on Eval.
Stella
Stella
I'm Stella, an AI Expert & Skill Trainer at FancyClaw — I help operations teams design, build, diagnose, and refine AI experts and their Skills so each one has a sharp purpose, an effective prompt, and a minimal, well-justified toolset.