Fuzz testing with Playwright and gremlins.js

Question

Fuzz testing with Playwright and gremlins.js

69 Views Asked by skellig At 22 October 2024 at 03:09

I am trying to set up Playwright together with gremlins.js to launch what they call horde of gremlins at a web app at a particular URL. The idea is that if the web app at this URL throws any errors during horde execution it should propagate back to Playwright and the test should fail. If no errors occur, the test should pass.

I came up with the following code for the test so far, but it marks the test as failed timed out, even though there are unhandled exceptions in the console.

There are different contexts - browser and Playwright, and some of the test code is async and I am failing to grasp when is what.

I tried to move some of this code around - expect() on page.evaluate and so on, but can't make it work.

import { test, expect } from '@playwright/test';

let url = 'http://localhost:7002'
// let url = 'https://demo.playwright.dev/todomvc/#/'

test('fuzz monkey test', async({ page }) => {
    page.on("pageerror", (err) => { throw err; });
    await page.addInitScript({
        path: './node_modules/gremlins.js/dist/gremlins.min.js',
    });
    await page.goto(url);
    await page.evaluate(() => gremlins.createHorde().unleash());
    expect(page).not.toThrow();
})

It gives - Timed out

page.evaluate
@
fuzz-monkey-test.spec.js:12
Error: page.evaluate: Test ended.

Original Q&A

There are 1 best solutions below

**ggorlen** · Answer 1

The default promise returned by unleash() takes about a minute to execute, which could cause a test to timeout if no error is thrown by the page.

But beyond that, your general setup looks more or less OK to me, so it's possible that gremlins.js isn't able to trigger an error on your particular page.

Try this reproducible example, which has a button that will trigger an error. Make it larger to practically guarantee gremlins will click it and cause the test to fail, or make it smaller to increase the likelihood of the test passing.

import {test} from "@playwright/test"; // ^1.41.2

const html = `<!DOCTYPE html><html><head><style>
button {
  /* adjust these sizes to change probability of error (larger = more likely) */
  height: 50vh;
  width: 50vw;
}
</style></head><body>
<button>click me to trigger an error</button>
<script>
document.querySelector("button").addEventListener("click", () => {
  throw Error("You triggered an error!");
});
</script></body></html>`;

test("fuzz monkey test", async ({page}) => {
  test.setTimeout(60_000);
  page.on("pageerror", err => { throw err; });
  await page.setContent(html);
  await page.addScriptTag({
    path: "node_modules/gremlins.js/dist/gremlins.min.js" // ^2.2.0
  });
  await page.evaluate(() => gremlins.createHorde().unleash());
});

Note that expect(page).not.toThrow(); doesn't make sense and will always fail, because page isn't a callable function. You don't actually need an assertion in this test--if the page throws, it'll fail, otherwise, it'll pass.

Fuzz testing with Playwright and gremlins.js

There are 1 best solutions below

Related Questions in JAVASCRIPT

Related Questions in PLAYWRIGHT

Related Questions in FUZZING

Related Questions in GREMLINJS

Trending Questions

Popular # Hahtags

Popular Questions