Benchmarking Third-Party Performance

Benchmarking third-party performance means comparing a provider's service, cost and resilience against relevant comparators, such as its own past performance, alternative providers, or market practice for a similar function. Under DORA, benchmarking is a useful oversight tool because it challenges complacency: a provider that meets its contract may still be falling behind what comparable arrangements achieve. Benchmarking does not, on its own, prove that a provider is safe or unsafe. It produces questions and evidence for discussion, renegotiation or redesign, and its value depends entirely on whether the comparison is fair, current and relevant to the service in question.

A market data feed illustration shows why the object of comparison must be defined. Two feeds might both claim broad coverage, yet differ in timeliness, completeness, resilience arrangements and support quality. Comparing headline price alone would miss those differences, while comparing every feature without weighting them would produce noise. Useful benchmarking therefore starts by naming the dimensions that matter for this service: reliability, recovery capability, support responsiveness, security posture in broad terms, and cost structure. Each dimension can then be compared on evidence, rather than on marketing claims or on a single headline figure.

Comparators should resemble the service being benchmarked, or the comparison will mislead. A provider supporting a small, specialised function cannot fairly be judged against a global platform serving a different market, and a provider's current performance cannot fairly be judged against its own performance in a period when the service was far smaller. Internal comparison over time is often the most reliable starting point, because the service and the expectations are constant. External comparison adds value where a genuine alternative exists and where data about that alternative is credible, such as through market engagement or independent sources, rather than through a competitor's sales material alone.

Benchmark data needs care because providers report performance differently, and because averages can hide the experiences that matter. A provider might report strong average availability while suffering repeated short outages that disrupt a particular service at peak times. A cost benchmark might compare unlike scopes, such as a service that includes support and recovery against one that charges separately for them. The entity should therefore ask how each figure was defined, over what period, and for which scope, before drawing conclusions. A benchmark built on unlike definitions is worse than no benchmark, because it gives a false sense of precision. Asking for definitions before comparing figures is therefore a basic step, not an optional refinement for specialists.

Benchmarking results are best used as an input to judgement, not as an automatic trigger. A provider that lags peers might be asked for an improvement plan, offered renegotiated terms, or given enhanced monitoring. A provider that leads peers might still warrant attention if its cost structure creates concentration or if its service has become difficult to replace. The response should be proportionate to the service: a minor gap for a routine tool calls for a conversation, while a significant gap for a provider supporting an important service calls for a documented plan with follow-up. Recording the decision and its reasons keeps benchmarking honest, since it prevents results from being cited selectively later.

Benchmarking cannot replace oversight, audit or exit planning, and it cannot answer whether a provider will perform during a severe disruption. Market comparison describes normal conditions well and extreme conditions poorly, because few benchmarks capture how a provider behaves when its own systems are under stress. Benchmarking also cannot capture relationship quality, such as whether a provider communicates openly during an incident, except indirectly through the entity's own experience. Treating benchmarking as one lens among several, alongside reporting, audit and incident experience, keeps it in proportion and prevents a favourable benchmark from becoming a substitute for assurance.