Running UI based selenium smoke tests against an ever-changing UI

Question

We are currently running smoke tests using Selenium Webdriver & JUnit against a B2C product. Since we are using Selenium, the scripts are totally dependent on the UI. Given that the product is out of a tech startup, the UI & workflows keep changing/evolving @ an extremely high frequency.

The Consequence: Smoke tests which are supposed to validate the sanctity of the application keeps failing. The team spends more time fixing the scripts rather than validating the build.

I am pretty sure most of the Automation folks out there would have faced similar issues esp. with rapid dev cycles. Looking forward to see some approaches undertaken by others in the industry who have faced similar problems.

Note: The frontend is developed in PHP

Answer 1

Webdriver works roughly like this: there is a start point, webdriver interacts with it (by simulating a button press for example) and then finds the next item to interact with. The next item might be on the next page or the same page. It might be found in various ways, by id or the 3rd div that is class="foo" etc.

The tests are things like does the page load with 200 OK, does the string "login" appear in a particular place and so on

The problem with a changing UI is that all the elements "move about". The ids change and the 3rd div class foo disappears. This means that the webdriver interactions fail and the tests if they are looking for particular elements will fail too

One solution is to develop and test against a set of ids. These ids will refer to fixed UI elements. All searching in webdriver should use the ids. The development team writing the PHP will put the ids in the correct places.

The set of ids can also be used as the basis for a sort of specification and can be used to explain UI flow in different ways to different stake holders.

I do not know of any specific product that handles this process of managing ids in both tests and development code but maintaining a "lexicon" like this to describe the UI items should not be a major task

Answer 2

The more versatile the System under Test is the more important it is to have a framework on top of Selenium that reduces the maintenance effort for a change. For the most common changes in a System under Test there are several known patterns that can help you to reduce the maintenance efforts:

• By using UIMaps to model the UI of the application it is extremely easy to handle changed IDs, CSS classes or similar changes

• PageObjects reduce the effort for larger UI changes (e.g. when an input field is changed from a TextBox to a Dropdown field)

• Use Keyword Driven Testing to model test cases without any knowledge of the underlying technological representation. i.e. a keyword encapsulates an action from the users point of view – a example for a keyword can be: “loginWithValidUser()”

• Don’t just utilize the UI for smoke testing if the UI / Application / Workflows change drastically and very often. Most of the time it is also helpful to test certain functionalities by calling WebServices without any Web-UI