Why tests pass but the system is still broken

The pagination change that emptied a UI, broke three unrelated tests, and left CI green

Tests passed. CI was green. The system was broken.

Not broken in an obvious way — no 500s, no stack traces, no error banners. Just a page that quietly started showing an empty list.

The appointments page was empty

No errors in the console. No failed requests. The API returned 200. Every test in the suite was green.

But the page showed nothing. Every patient who opened their appointments list saw an empty screen — as if they'd never booked anything.

I checked the API directly. Data was there. The endpoint responded correctly. Nothing in my monitoring suggested a problem.

The API had changed recently. The pagination was added.

What the change did

The response shape changed:

// Before
[{ id: 1 }, { id: 2 }]

// After
{ data: [...], total: 10, page: 1, limit: 20 }

The API tests didn't fail. They validated the status code and the new response structure — both correct. The endpoint worked exactly as designed.

But the API tests validated that the response matched the API contract — not that it matched what the consumer expected.

In the frontend there was this line:

cachedRows = Array.isArray(rows) ? rows : []

For an object, Array.isArray returns false. The list silently became empty. No console errors. No 500. Just nothing on screen.

Then it got worse

Three tests failed. None of them had anything to do with pagination.

The fixture teardown also called the same endpoint. It expected an array, got an object — so Array.isArray returned false, it never cleaned up the appointments it was supposed to delete, never freed the slot. The next test tried to book the same slot and got SLOT_OVERLAP.

The slot conflict error pointed nowhere useful. The real cause was two layers up, in a response shape change that all the API tests had passed.

Why CI didn't catch it

CI answers one question: did the tests finish without errors?

It doesn't answer: do the tests still make sense relative to what the system actually does?

The API tests were correct — they validated the new format. But they validated that the response matched the API contract, not that it matched the consumer. That gap lived comfortably between two layers, invisible to both.

This is contract drift: the API changed, the consumer didn't know, and nothing in the pipeline connected the two.

What would have caught it

A consumer-driven contract test. Something that says: "the patient appointments page expects this endpoint to return an array it can iterate over — and if the shape changes, I want to know."

That's exactly what Pact does. After this incident, I added a contract layer between the API and its consumers. The contract now encodes the consumer's assumptions explicitly — not just "status 200" but "the response is something I can call .find() on."

I also added a visual test that checks whether the appointments list actually renders content. It would have caught the empty page immediately, even without understanding why.

The pattern behind it

Tests don't prove the system works.

They prove the system didn't fail in the specific ways the tests were looking for.

The gap between those two things is where silent bugs live. Contract drift, stale assertions, missing consumer-side tests — none of them produce a red CI. They just quietly change what the system does while the dashboard stays green.

Failure signature