Flaky unit tests (React Testing Library) in CI/CD (teamcity)

160 Views Asked by At

I am mostly looking for if someone else has had this experience and figured out what causes it or how to mitigate it.

I have a form with some fairly complex behavior in terms of some value in one field changing validation logic or making a "default" selection for some other field, so I have a different RTL test suite (here defining test suite as a .test.js file with a single describe) for each of the fields. These all pass locally when run individually or all together. In CI/CD (teamcity, if that makes a difference) some of the tests occasionally fail.

In total I have a little over 300 tests in 45 files on this form. The subset of tests that are "flaky" seems pretty consistent, but it's unclear if those tests are written incorrectly or if they're failing for some reason that is unique to the CI/CD environment. I honestly cannot identify anything wildly different they are doing that the other tests are not. They are all using the exact same set of helper functions defined in a separate file.

The failing assertions are of two varieties as well:

Type one; waitFor times out while waiting for some element to be rendered after an event trigger:

fireEvent.click(someElement);

await waitFor(() => {
  expect(
    screen.getByText("Element that shows up when you click someElement")
  ).toBeInTheDocument()
}, {timeout: 5000});

the {timeout: 5000} is just my latest attempt at mitigating this, but it hasn't worked. Again, this passes locally and about 80% of the time in CI/CD, but a 20% failure rate is still pretty annoying.

Type two; expect(submitMethod).toHaveBeenCalledWith(expectedParams) will fail, and the error log indicates that submitMethod was called with the expectedParams of the test that comes before this one. submitMethod is a jest spy, and I'm calling jest.resetAllMocks() in an afterEach

// example test code
const saveButton = await screen.findByText('Save');

fireEvent.click(saveButton);

await waitFor(() => {
  expect(submitMethod).toHaveBeenCalledWith(expectedParams)
});

this failure seems to indicate that the submitMethod jest spy has NOT been cleared after the previous test OR the this test has started running before the previous test ended

SW versions:

lmk if I can provide any further details

I have tried looking for any asynchronous production code that might still be running after the test has completed and looking for any asynchronous test code that I am failing to await such that it would continue running and allow "bleedover" into the next test.

I expect consistent behavior from the test suite in the local dev environment and in CI/CD

0

There are 0 best solutions below