Testing Framework vs Testing tool

Question

I am new to testing. The commonly used terms like Framework and tool confuses me a lot. Can anyone please explain me the difference between a Framework like STAF[software testing automation framework]and Tool like selenium.

Also how to select a tool for a particular framework. What are the criterias used for selection?

Brief explanations are welcomed!!

Answer 1

I will try to answer what I believe people normally use these terms to mean, lets start with the simpler term: A tool.

A tool like selenium is what actually does the automation, it has an API that will work for pretty much anything it covers (in this case websites) but it knows nothing about how the website you want to test works, this means it deals with low level constructs such as elements on a page and clicks.

A framework is normally just wrapping a tool to make it easier to make a test by imparting knowledge of your application, a standard example is login.

Say you want a test that checks when you enter a correct username and password in you get access to application. Using just selenium it would something like:

driver.findElement(By.id("username").sendKeys("MyUsername"); driver.findElement(By.id("password").sendKeys("password123"); driver.findElement(By.id("login").click();

Thats pretty simple, but as you can guess login is going to be used a lot across your tests and so it makes sense to move this into a place that makes it easier to reuse (both from a less code stand point and maintainability). This is where a framework comes into play, normally with selenium it will be page objects (see here)

Answer 2

Tool:

Simply put, a tool is a software. In case of test automation, tools are software that let you automate your tests on an application. There are many test automation tools that you can choose from depending on your requirements. Some examples are Selenium, UFT, Visual Studio CUIT, Jamo Solutions Meux Test, T-Plan Robot, Telerik Test Studio etc.

Often, you'll have to write tests in the tools using a supported programming language. For instance, testers using UFT need to code in VB while those using Visual Studio can code in both VB and C#. However, some testing tools (like Telerik Test Studio) let you write script-less tests where your tests will consist of a bunch of easily understandable keywords, not code.

Framework:

The most popular test automation tools like Selenium and Visual Studio provide all the basic features you require to build your own tests. However, they do not provide ready-made features (like Reporting and Exception Handling) for testing. This requires the creation of a 'Framework' which is nothing but a collection of code written using a tool of one's choice that makes testing an application easy. Simply put, a framework is what you create with a tool (or a collection of tools) to test your application.

A typical framework consists of two parts: test scripts and function libraries. Test Scripts are the pieces of code that need to be executed to perform actions on the application under test (AUT). Function Libraries are classes consisting of important functions that are called by your test scripts. These can include timing functions, reporting/logging functions, exception handling functions, data communication functions etc.

You can also use an external database to pass test data to your test scripts during run-time instead of hard-coding it in your test scripts. Frameworks that employ external databases are called data-driven frameworks. The external database can be of your choice, be it a SQL Server, an XML file or a simple Excel spreadsheet. Data-driven frameworks make use of APIs or include custom-made classes that let you communicate with the database to transfer data.

Another type of framework is the keyword-driven framework. These frameworks are used in long-term test automation projects that require scripting of thousands of test cases. The main objective of these frameworks is to reduce the time taken to script a test case by reusing code that has already been written. They often include very strong function libraries which enable scripting of test cases using just predefined keywords. For example, common actions on an application like login and logout are performed by single line codes like:

Actions.Login();

and

Actions.Logout();

where Actions is a Function Library that consists of the Login() and Logout() functions. This massively reduces the script size and the long-term maintenance requirements of the test script, among other benefits.

Of course, you can either build a test automation framework and use it for your own application or create a generic test automation framework and make it available to the testing community for everyone to use, which is what STAF is.

Selection of testing tools:

To address your second question, there is no straight-forward answer to it. There are a number of criteria that can affect your decision. But in the end, it is all about your requirements and the requirements of your AUT.

• If it is a Windows desktop app, you have to use Coded UI Tests in Visual Studio.
• If it is a Web application, you can use Selenium, UFT, Visual Studio or Test Studio.
• If it is a mobile app, you can use Appium, Jamo Solutions Meux Test or T-Plan Robot.
• If you want to test your mobile app over a large number of devices and platforms, you can use cloud-based tools like Sauce Labs, Perfecto Mobile or Device Anywhere.
• If you are short on budget, you'll be better off using open source tools over commercial tools, and so on.

Application Testing is a huge industry now and there is no dearth of testing tools available in the market. You will find the perfect tool for you if you know what you want and do some research on Google.

Answer 3

Base on my understanding:

TOOLS

• We "USE" tools to meet our objective (can be own self or your small groups of team).
• Example: We use Selenium IDE as a tools helps us to automate some repetitive steps to do certain verification during our smoke test.

FRAMEWORK

• We "DESIGN" a framework to meet the organization mission.
• Things to consider when we design the framework including:
• Maintainability
• Reusability
• Data Driven
• Reporting
• Schedule running through CI tools like Jenkins
• Example: We design a Test Automation Framework by using WebDriver + Java + TestNG + ANT, to meet the objective to identify our current code base stability, and the test will be trigger and run by jenkins in a daily basis, SSRS report will captured in a daily basis as well each time the test finished. Stakeholders can review the daily code stability report anytime he/she needed.

Hope that can help you :D