While commentators debate how many agents automation removes, Manila has been building a second business — evaluating, correcting and stress-testing the systems themselves.
Somebody has to grade the machine, and increasingly that somebody sits in Manila. The fastest-growing category in Philippine BPO data services is not answering customers at all — it is judging whether the software answered them correctly, building the test sets that catch regressions, and writing the examples from which the next version learns.
Key takeaways
- The Philippine IT and business process sector employs well over a million people, and a growing slice now works on AI systems rather than on contacts.
- Evaluation work pays above entry-level support, sitting in the same published band as quality and compliance roles.
- The scarce input is domain knowledge, not labeling speed — knowing what a correct refund decision looks like.
- Every model update creates fresh evaluation demand, making this recurring work rather than a one-off project.
- Buyers rarely budget for it, then discover their automation has no regression safety net.
The shape of this work is unfamiliar enough that it often sits outside the contact center contract, bought separately and managed by a different team, which is part of why it has grown with so little commentary. Firms assembling specialist offshore data and evaluation teams are generally doing so outside their existing service agreements altogether.
What Does This Work Actually Involve?
Four distinct jobs that get lumped together under the word “annotation,” each requiring different people.
Evaluation
Reading an interaction the system handled and deciding whether the outcome was right, not merely whether the tone was pleasant. It is the judgment everything downstream depends on, and the hardest of the four to staff.
Correction
Writing what the system should have said, in a form precise enough to train on rather than merely precise enough to understand.
Adversarial testing
Deliberately constructing the awkward cases, the ambiguous phrasings and the contradictory requests that a real customer will eventually produce.
Curation
Maintaining the libraries of intents, exceptions and test cases that constitute the operational memory of an automated program.
Each is staffed separately from the customer-facing floor, because the selection criteria diverge sharply: the best evaluator is rarely the fastest agent, and the best adversarial tester is often someone with a slightly contrarian temperament.
What Makes Manila Suited To This Work?
Because the hard part is domain knowledge, and the country has spent twenty years accumulating it. Deciding whether an automated response about a disputed charge was correct requires knowing the policy, the regulatory constraint and the commercial convention — knowledge that sits in people who have handled thousands of those disputes.
Generic labeling can be done anywhere, and increasingly is — it has become a commodity priced accordingly. Judging whether a refund decision was defensible under a specific insurer’s rules cannot, and that distinction is where the margin in this category now sits.
“The clients who get the most from this are the ones who staff it with their best former agents, not with their cheapest new hires,” says John Maczynski, who heads PITON-Global, the country’s leading outsourcing advisory and supplier sourcing firm. “You are asking someone to be the standard the system is measured against. That is a senior job wearing a junior title.”
What Does It Pay?
More than frontline support, less than specialized technical roles. Published Philippine bands put evaluation and audit work in the same tier as quality and compliance positions, a few dollars clear of frontline support, and leadership above that again.
The premium reflects scarcity rather than credentialism. There is no certification for knowing when an automated apology is inadequate, so providers compete for people who have demonstrated that judgment on live work, which bids the rate up in exactly the way a genuine shortage should. Buyers expecting annotation prices are routinely surprised by the quotes.
Is This A Project Or An Operation?
An operation, though it is almost always bought as a project. The common pattern is a one-time engagement to build a training set before launch, after which the team disbands and the client assumes the work is finished.
It is not finished, because models change and customers find new ways to ask the same awkward questions. A provider updates its underlying system, behavior shifts on intents nobody touched, and the only way to detect that is to re-run evaluation against a maintained standard. Without a standing capability, the first signal is a complaint trend three weeks later.
Ralf Ellspermann, chief strategy officer at the advisory firm, puts the budgeting failure plainly: “Clients fund the build and not the upkeep. Then the model is upgraded, and nobody owns the question of whether it still behaves. That gap is where confidence in automation quietly dies.”
What Should Buyers Ask For?
Evidence of a maintained standard rather than a headcount. Three artifacts separate a serious capability from a staffing arrangement: a documented evaluation rubric that a third party could apply consistently, a versioned test library with results tracked across model releases, and a defined route by which evaluation findings change something — a prompt, a routing rule, a training set.
The third is the one most often missing. Evaluation that produces reports nobody acts on is an expensive way to document decline.
Maczynski sees it as the practical differentiator among providers now claiming AI capability: “Anyone can show you a dashboard of model accuracy. Ask instead who decides when the score is too low to ship, and whether that person has ever stopped a release. The answer tells you whether the function has authority or just a budget.”
Which leaves the Philippine industry in a position few predicted three years ago. The country that built its reputation on answering the phone is quietly becoming a place where the systems replacing the phone are judged, corrected and held to a standard — work that is harder to automate than the work it supervises, for the obvious reason that somebody has to be the judge.
