Every task the agent executed was scored against benchmarks built with the bureau’s subject-matter specialists.
From a call center to an AI agent businesses trust
Businesses across Kazakhstan used to phone the statistics bureau’s call center with questions, for example how to fill in reporting forms. AI Stat now answers them directly, and it only ships what it can answer without hallucinating.
AI StatBureau of National Statistics of Kazakhstan
- Dec 2024 — now
- Center for Behavioural Economics & Big Data Analysis
- Production LLM agent
~10Kmonthly users,
including businesses
including businesses
300–500requests
per day
per day
~88–90%F1, with
recall first
recall first
Starting point
Businesses are the bureau’s primary users. With questions, for example how to fill in reporting forms, they called the bureau’s call center and asked an operator.
Every answer meant the same hidden job: go to the data storage, extract the right data, understand what it describes, and map it to the question the business actually asked.
My role
AI Product Lead at the bureau’s Center for Behavioural Economics & Big Data Analysis, an experimental division that works as the bureau’s AI lab.
Owned the scenarios, the knowledge base and retrieval design, and the quality bar that decided when the agent could roll out. Worked with the engineering team and with the bureau’s subject-matter specialists.
What I did
- Turned the operator’s job into the agent’s scenario
Defined the core scenario: find the data, extract it, understand what it is about, and map it to the user’s question. Prioritized the top three user journeys, starting with how to fill in the reporting forms.
- Designed knowledge and retrieval
Designed the knowledge base and the RAG retrieval layer: hybrid search across meaning and exact wording, with a second, agentic search step when the first pass is not enough.
- Built explicit feedback into the product
Every answer can be liked or disliked, with a comment on why it missed or which piece of data was missing, so failures are visible and specific.
- Stress-tested it live
Piloted the agent during the Agricultural Census on roughly 8,500 real questions before the full rollout.
How we measured it
If it doesn’t work, it doesn’t roll out. If it hallucinates, it doesn’t roll out.
A single invented answer costs the user’s trust, and the business goes back to calling the operator. That makes hallucination a release blocker.
Invested heavily in recall: finding the right data is the precondition for every correct answer.
Likes, dislikes and written comments show exactly where the agent missed after launch.
~88–90%F1, with recall first
~8.5Kcensus questions in the live stress test
What changed
AI Stat went from pilot to full production. It serves around 10K monthly users, including businesses, and handles 300–500 requests per day. The bureau officially launched it on stat.gov.kz in May 2026.
Knowledge base→RAG retrieval→Agent task→SME benchmark→ProductionEvery task scored against expert answers · ~8.5K census questions
Summary
In a public institution, trust is the product. The release bar was set on hallucinations before features, because one wrong answer sends a business back to the phone line.