Shadow study with AI: train the diagnosis that generated code does not do for you

When someone suggests comparing their own diagnosis with an artificial intelligence's, the most common reaction is: if AI generates code faster than I do, training this way is a waste of time. The objection is right about one part and wrong about another. To produce code, AI is faster, and nobody needs to train to type faster than it does. But producing code is not the bottleneck of a flaky test. The bottleneck is the diagnosis: given an intermittent failure, what is the cause? That question requires looking at logs, timing, state and execution order, and generating a cy.wait(2000) is cheap precisely because it does not answer it.

Generating is cheap and verifying is expensive. The more code AI produces, the more review decisions land on your desk, and the skill that does not go away is judging whether a fix hits the cause or merely hides the symptom. That judgment is what the shadow study trains, and the clearest case to see it is the flaky test.

A flaky test is a trust problem before it is a technical one. When the team learns that red sometimes lies, it starts ignoring red, and a real failure goes unnoticed. In UI automation the complaint is common: maintaining tests often costs more than writing them, and badly placed waits and brittle selectors come up in every conversation on the subject. I do not quote percentages because I do not want to repeat a number I have not verified; I describe the pattern anyone who works with automation recognizes.

The false green

The concrete risk is the false green. A wrong but plausible fix is worse than a failure, because a failure warns you and a wrong fix leaves the test passing without having repaired anything. Take a Cypress test that opens a list loaded by a request and clicks the first item:

it("opens the first article", () => {
  cy.visit("/blog");
  cy.get("[data-cy=post-item]").first().click();
  cy.url().should("include", "/blog/");
});

It depends on the API answering before the click. The most common suggestion, from a human or from an AI, is to add a fixed wait:

cy.visit("/blog");
cy.wait(2000); // fixed delay: hides the symptom
cy.get("[data-cy=post-item]").first().click();

This passes on your machine and fails in CI, under more load, because the fixed time has no relation to the event the test actually waits for. The root cause is that the test waits for a clock interval and not for a condition. The fix that hits the cause synchronizes on the event:

it("opens the first article", () => {
  cy.intercept("GET", "/api/posts*").as("posts");
  cy.visit("/blog");
  cy.wait("@posts"); // waits for the real event
  cy.get("[data-cy=post-item]").should("have.length.greaterThan", 0);
  cy.get("[data-cy=post-item]").first().click();
  cy.url().should("include", "/blog/");
});

Notice what the diagnosis required: knowing that Cypress retries assertions until they pass, that a time-based cy.wait is not an assertion, and that the source of the delay was the network. A model can get this right or propose the first version, which looks correct, and whoever has not trained the diagnosis cannot tell the two apart.

To reach that diagnosis quickly, it helps to classify the failure before acting. Four families cover most instabilities in UI tests:

• Synchronization: element not found or not clickable, and the test passes on retry. The question is which event it should wait for.
• Data and order: the test passes alone and fails in the full suite. The question is whether it depends on state left by another test.
• Environment: passes locally and fails in CI. The question is what differs, such as CPU, clock, network, time zone or data.
• Selector: breaks after a screen change that did not change behavior. The question is whether the selector describes intent or structure.

Classifying the failure into one of these families narrows the search space and names the error, and naming the error is what lets you measure progress, which is where the method comes in.

How the shadow study works

The idea is a controlled experiment for one person, with the AI as a blind comparison: you diagnose first, it diagnoses in parallel without seeing your conclusion, and then the two results are compared. The first part of the procedure is this:

1. Pick a genuinely flaky test, with the failure log and what changed between a green run and a red one.
2. Write your hypothesis before any help, in four fields: family, probable cause, evidence and proposed fix. If you cannot write the evidence, what you have is a guess.
3. Give the AI the same material, without your hypothesis, and ask for the diagnosis in the same format.

For the third step, a simple prompt is enough:

Here is the log of an intermittent failure, the test code and the diff between the green and the red run.
Answer in four fields:
1. Family (synchronization, data and order, environment or selector)
2. Probable cause
3. Evidence, in the material sent, that supports the cause
4. Proposed fix and why it hits the cause

With both diagnoses in hand, the rest of the process is this:

4. Compare field by field and note where you agreed and disagreed. The disagreement is the learning point.
5. Ask the AI to critique your hypothesis (what evidence you ignored, what alternative you did not rule out) and then do the reverse with its hypothesis.
6. Verify by execution: apply the fix and run the test dozens of times, not once.
7. Record the result on the scoreboard.

The sixth step is what keeps the method from turning into a debate of opinions, and it avoids accepting an explanation just because it is well written. In Cypress, a direct way to repeat the test is to generate many copies of it, preferably in an environment under load, such as CI, which raises the chance of exposing the race condition:

Cypress._.times(30, (i) => {
  it(`opens the first article (${i + 1})`, () => {
    cy.intercept("GET", "/api/posts*").as("posts");
    cy.visit("/blog");
    cy.wait("@posts");
    cy.get("[data-cy=post-item]").first().click();
    cy.url().should("include", "/blog/");
  });
});

The scoreboard turns hits and misses into data. A simple table is enough, with the columns case, family you pointed to, family the AI pointed to, real family, who got the cause right, and whether the fix stayed stable after the runs. After a few weeks, read it by column: your family hit rate shows whether the conditioning works, the family where you miss most shows what to study, and the AI's misses show which kind of failure to trust it less on. These numbers are yours, from a small set of cases, and not a market statistic.

What remains is answering the opening objection with something beyond opinion. The cost is small and bounded, because each case fits in a few minutes of writing plus the conversation with the AI. The return compounds, since you work with AI every day and each well-made diagnosis improves how you describe the problem to it and how you review its output. And the method protects against the false green, whose cost (a test that lies, a real failure masked) outweighs the time spent training. It does not replace the AI: you keep using it to generate, and the training calibrates the judgment that decides what to do with what it generated.

This principle has a parallel in aircraft embedded software. The DO-178C standard, recognized by authorities such as the FAA, EASA and Transport Canada, treats verification as an activity of its own, with review of requirements and test cases, traceability between requirement, test and result, and structural coverage up to MC/DC at the most critical levels, and one of its pillars is independence: whoever verifies is not whoever produced, so as not to inherit the same blind spot. The shadow study borrows that idea at a personal scale, with the AI as an independent second reading and repeated execution as objective evidence. It is a parallel of principle: I am not saying aviation teams use this method under this name.

The limits are worth stating. The method is a proposal, with no controlled study proving gain, and its only validator is your scoreboard. The AI can be confidently wrong, so verification by execution is mandatory. And logs can contain sensitive data, tokens or customer information, which should not go to the model. To start this week, take the three flakiest tests in your project, write the hypothesis for each before opening any tool, and only then compare.