Live Exercises Without Harming Customers

A live exercise tests real systems, real people and real procedures, rather than discussing them, and its central challenge is to do that without harming customers. Harm can take many forms: a customer unable to complete a transaction, a message sent in error, data altered by a test, or a service slowed at a moment when customers need it most. A safe live exercise therefore starts with limits: what will be tested, who could be affected, how the test will be stopped, and how any effect will be undone. Those limits are agreed before the exercise begins, and they are treated as binding even if the exercise is going well. A live test that cannot be stopped safely should not start.

Ring fencing means separating the exercise from customer impact as far as possible, while keeping the test realistic enough to be useful. Methods vary by service and may include testing in a controlled segment, using test accounts, simulating a supplier response, or exercising a failover that customers should not notice. The choice of method should be explained in the exercise plan, including what remains untested because of the ring fence. Consider a trading service that wants to test failover to a secondary site. A plan might exercise the failover for internal traffic first, watch for errors, and only then consider customer traffic, with a clear decision point in between. Honesty about what was not tested matters, because a ring fenced success can otherwise be misread as proof that a full failover would work.

Stop criteria are conditions under which the exercise ends immediately, regardless of its schedule. They might include unexpected customer impact, a real incident occurring at the same time, a system behaving in a way the plan did not predict, or a rollback that fails. The authority to stop the exercise should be named in advance and should sit with someone who is watching impact, not only with the person running the test. Participants should also know how stopping will be communicated, so that a stop does not itself create confusion. An exercise that has no stop criteria is not bold; it is uncontrolled. The existence of stop criteria also reassures staff, which makes them more willing to report a problem early rather than hoping it resolves itself.

Rollback is the plan for returning to normal after the exercise, including undoing test data, restoring settings, and checking that no customer record was altered. Clean up should be treated as part of the exercise, not as an afterthought, and it should be verified rather than assumed. Staff should check that test transactions are removed or clearly marked, that monitoring shows normal behaviour, and that any customer facing message sent during the test is followed by a correction if needed. A live exercise that ends with a successful failover but leaves test data in a production store has created a new risk while testing an old one. The debrief should therefore include a clean up check, signed off by someone other than the person who ran the test where that separation is possible.

A live exercise can test timing, capacity, and the behaviour of systems under load in ways discussion cannot. It can reveal that a failover works but takes longer than the plan assumes, that a manual workaround is slower than expected, or that monitoring alerts arrive in an order that confuses the team. It can also test stamina and coordination across shifts, since a real exercise involves handovers, fatigue and competing demands. Those findings are valuable even when they are uncomfortable. A tabletop might conclude that a workaround is feasible; a live exercise might show that it is feasible only with staffing that is not available at the time the disruption is most likely. That difference is the reason live exercises exist, despite their risk.

Preparation for a live exercise includes telling affected staff what is happening, what is a test and what is real, and how to report a problem. That distinction matters, because staff who mistake a test for a real incident may escalate incorrectly, while staff who mistake a real incident for a test may delay a response. Customers should not normally be involved in a live exercise, and where any customer effect is possible, the firm needs a clear view of that effect and of how it will be explained. After the exercise, staff who took part should be debriefed while detail is fresh, and any customer effect, however small, should be recorded honestly. A live exercise that harms customers and then hides that harm has failed twice.