There are 2 ways to run TestNG tests – using Run As TestNG tests and from testng.xml.
In this tutorial, I’m explaining how to create and run tests using TestNG.xml.
Step 1 – In the below project, there is no testng.xml present.
Step 2 – Right click on class name “API_Test.java” and navigate “TestNG–>Convert to TestNG“.
Step 3 – Now a new window for testng.xml will be generated as shown below. Click the Finish Button.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
<test thread-count="5" name="Test">
<classes>
<class name="com.example.RestAssured_TestNG_Demo.API_Test"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
Step 4 – A xml file named “testng.xml” will be generated in project hierarchy.
Step 5 – Right click on this testng.xml and select Run As–> Testng Suite.
Step 6 – You will view the result in two different tabs: Console and “Results of running suite”. Below is the image of Rest of running suite tab.
Multiple Classes
Let us imagine a sceanrio where there are 3 classes and we want to run them all together, you can done that by creating a testng.xm and mention all 3 classes in that testng.xml.
Select all 3 classes and right click and navigate “TestNG–>Convert to TestNG”
This is how the testng.xml will look like for multiple classes.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
<test thread-count="5" name="Test">
<classes>
<class name="com.example.RestAssured_TestNG_Demo.API_Test1"/>
<class name="com.example.RestAssured_TestNG_Demo.Test3"/>
<class name="com.example.RestAssured_TestNG_Demo.Test2"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
Right click on this testng.xml and select Run As–> Testng Suite. You will get the result in two tabs: Console and “Results of running suite”. Rhis shows that all the tests present withing the three classes are executed.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
Listener is defined as interface that modifies the default TestNG’s behavior. There are several interfaces that allow you to modify TestNG’s behavior that are called “TestNG Listeners”. It allows customizing TestNG reports or logs. There are many types of TestNG listeners available. Here are a few listeners:
IAnnotationTransformer
IAnnotationTransformer2
IHookable
IInvokedMethodListener
IMethodInterceptor
IReporter
ISuiteListener
ITestListener
When you implement one of these interfaces, you can let TestNG know about it with either of the following ways:
Using in yourtestng.xml file.
Using the@Listeners annotation on any of your test classes.
ITestListener has the following methods
OnStart– Invoked before running all the test methods belonging to the classes inside the tag and calling all their Configuration methods.
onTestSuccess– onTestSuccess method is called on the success of any Test.
onTestFailure– onTestFailure method is called on the failure of any Test.
onTestSkipped– onTestSkippedmethod is called on skipped of any Test.
onTestFailedButWithinSuccessPercentage– method is called each time Test fails but is within success percentage. Invoked each time a method fails but has been annotated with successPercentage and this failure still keeps it within the success percentage requested.
onFinish– Invoked after all the test methods belonging to the classes inside the tag have run and all their Configuration methods have been called.
onTestStart – Invoked each time before a test will be invoked. The ITestResult is only partially filled with the references to class, method, start millis and status.
Here, I explain the use of listener – ITestListener in a program mentioned below
Step 1) Create class “ListenerDemo” that implements ‘ITestListener’. Add methods like onTestFailure, onTestSkipped, onTestStart, onTestSuccess to this class
Step 2) Create another class “ListenerTestCases” for the login process automation. Selenium will execute this ‘TestCases’ to login automatically.
Step 3) Next, implement this listener in our regular project class i.e. ” ListenerTestCases “. There are two different ways to connect to the class and interface.
The first way is to use Listeners annotation (@Listeners) as shown below:
Step 4): Execute the “ListenerTestCases” class. Methods in class “TestPass ” are called automatically according to the behavior of methods annotated as @Test.
Step 5): Verify the Output that logs displays at the console.
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
public class ListenerDemo implements ITestListener {
// When Test case get failed, this method is called.
public void onTestFailure(ITestResult Result) {
System.out.println("The name of the testcase failed is :" + Result.getName());
}
// When Test case get Skipped, this method is called.
public void onTestSkipped(ITestResult Result) {
System.out.println("The name of the testcase Skipped is :" + Result.getName());
}
// When Test case get Started, this method is called.
public void onTestStart(ITestResult Result) {
System.out.println(Result.getName() + " test case started");
}
// When Test case get passed, this method is called.
public void onTestSuccess(ITestResult Result) {
System.out.println("The name of the testcase passed is :" + Result.getName());
}
}
In the below test, there are 2 test cases. One Test passes and another fails. When we are executing ListenerTestCases, it will call the ListenersDemo internally.
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
importorg.openqa.selenium.chrome.ChromeDriver;
import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
@Listeners(com.selenium.testng.TestNGDemo.ListenerDemo.class)
public class ListenerTestCases {
static WebDriver driver;
@Test
public void TestPass() {
System.setProperty("webdriver.chrome.driver",
"C:\\Users\\Vibha\\Desktop\\SeleniumKT\\chromedriver_win32\\chromedriver.exe");
driver= new ChromeDriver();
driver.get("https://opensource-demo.orangehrmlive.com/");
driver.findElement(By.name("txtUsername")).sendKeys("Admin");
driver.findElement(By.name("txtPassword")).sendKeys("admin123");
driver.findElement(By.id("btnLogin")).submit();
String dashboardTitle = driver.findElement(By.id("welcome")).getText();
Assert.assertTrue(dashboardTitle.contains("Welcome"));
}
@Test
public void TestFail() {
System.setProperty("webdriver.chrome.driver",
"C:\\Users\\SingVi04\\Desktop\\SeleniumKT\\chromedriver_win32\\chromedriver.exe");
driver= new ChromeDriver();
driver.get("https://opensource-demo.orangehrmlive.com/");
driver.findElement(By.name("txtUsername")).sendKeys("Admin");
driver.findElement(By.name("txtPassword")).sendKeys("admin123");
driver.findElement(By.id("btnLogin")).submit();
String dashboardTitle = driver.findElement(By.id("welcome")).getText();
Assert.assertTrue(dashboardTitle.contains("Hello"));
}
}
Output
TestFail test case started
The name of the testcase failed is :TestFail
TestPass test case started
The name of the testcase passed is :TestPass
PASSED: TestPass
FAILED: TestFail
java.lang.AssertionError: did not expect to find [true] but found [false]
To execute this program, we need to Right-click and select Run as – TestNG.
There is another way to execute the Listener class, which is using testng.xml. There is a Listener class named “ListenerDemo” where the implementation of various methods of the listener is present. If we want to run the tests using testng.xml, then there is no need of mentioning Listeners in the ListenerTestCases class.
<?xml version = "1.0"encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name = "TestSuite">
<listeners>
<listener class-name ="com.selenium.testng.TestNGDemo.ListenerDemo"/>
</listeners>
<test name ="Test">
<classes>
<class name="com.selenium.testng.TestNGDemo.ListenerTestCases"/>
</classes>
</test>
</suite>
The test execution result will look like something shown below.
TestNG generates the various types of reports under the test-output folder. Open “emailable-report.html”, as this is an HTML report open it with the browser. It will look like something below.
TestNG also produce “index.html” report, and it resides under the test-output folder
There is another example of Listener –ITestResult.
We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
One of the important features of TestNG is the ability to pass different test data to a test case as arguments, which is called parametrization
There are mainly two ways through which we can provide parameter values to TestNG tests.
Through testng.xml XML configuration file
Through DataProviders
In this tutorial, we will discuss using testng.xml for parametrization. If we need to pass some simple values such as String or Integer types to the test methods at runtime, there is something called @Parameter where parameter values through TestNG XML configuration files pass to test.
@Parameters("value")
Let us explain how we can use parameters. To start with, add the below dependencies to the POM.xml in the case of the Maven project.
TestNG.xml looks like as shown below. Here, the parameter name is the browser name value for the browser is “Chrome”. So, this “Chrome” value is passed to Test as a parameter and as a result, a Google Chrome browser opens. Similarly, the same tests are run using Firefox, as it is mentioned in the testng.xml.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite ">
<test name="Chrome Test">
<parameter name="browser" value="chrome" />
<classes>
<class name="com.example.TestNGParameterizationDemo" />
</classes>
</test> <!-- Test -->
<test name="Firefox Test">
<parameter name="browser" value="firefox" />
<classes>
<class name="com.example.TestNGParameterizationDemo" />
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
Cross Browser is a technique in which a web application tests on different browsers and operating systems. Cross Browser testing, make sure that the site rendered the same in every browser.
Suppose, we have to execute 25 tests cases to test a web application on Google Chrome Browser and it takes around 4 hrs to execute these tests. However, we do not know if a user is going to access the web application on which browser. Therefore, now the same set of test cases need to executes on Firefox, Edge, Internet Explorer, Safari and Opera.
Therefore, now we need to execute 25*6=150 test cases and test execution hour changes from 4 to 24 hrs to check that the web application is working as expected. What is the best approach to handle this situation is to automate these tests and perform cross browser testing
Why do we need to perform cross browser testing?
Each website is built by anyone or combination of these technologies – HTML, CSS and Javascript. Each browser uses different rendering engines to compute these technologies. Chrome uses Blink, WebKit on iOS, V8 JavaScript engine, Firefox uses Gecko, Edge uses Chromium-based with Blink and V8 engines, Safari uses Webkit rendering engine, IE uses Trident and so on.
1) Font size, image orientation and alignment mismatch in different browsers
2) Different browser is compatible with different operating systems
3) CSS,HTML validation difference can be there
Lets see an example of Cross Browser testing using Selenium and TestNG.
Step 1 – Add the below dependencies to the POM.xml, in case of Maven project.
We need to specify the values of browser in the TestNG XML file that will pass to the test case file.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="tests" thread-count="3">
<test name="Chrome Test">
<parameter name="browser" value="chrome" />
<classes>
<class name="com.example.crossbrowser.CrossBrowserTests"/>
</classes>
</test> <!-- Test -->
<test name="firefox Test">
<parameter name="browser" value="firefox" />
<classes>
<class name="com.example.crossbrowser.CrossBrowserTests"/>
</classes>
</test> <!-- Test -->
<test name="Edge Test">
<parameter name="browser" value="edge" />
<classes>
<class name="com.example.crossbrowser.CrossBrowserTests"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
To execute this program, we need to Right click on the program and select Run as – TestNGTest
The test execution result looks like as shown below. It shows the test execution status of all the tests. As, in this program, 3 tests are executed and all 3 of them passes. The same result can be depicted from below image.
TestNG Report Generation
TestNG generates various type of reports under test-output folder like emailable-report.html, index.html, testng-results.xml
We are interested in ’emailable-report.html’ report. Open ’emailable-report.html’, as this is a html report open it with browser. Below image shows emailable-report.html.
TestNG also produce “index.html” report and it resides under test-output folder. Below image shows index.html report.
We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
Parallelism or multi-threading in software terms is defined as the ability of the software, operating system, or program to execute multiple parts or sub-components of another program simultaneously.
Parallel testing helps to reduce execution time and efforts and results in faster time to delivery. In a scenario where we have two versions of software available, and we need to check its stability and compatibility, we can run the two versions simultaneously and find issues at a much faster rate.
How to run Parallel Tests with Selenium?
TestNG provides multiple ways to execute tests in separate threads. In testng.xml, if we set ‘parallel’ attribute on the tag to ‘methods’, testNG will run all the ‘@Test’ methods in tag in a separate thread.
The project Structure looks like shown below:-
To start with, add the below mentioned dependencies to POM.xml (Maven project)
Let us create a class with multiple tests. In the below Example, four test cases or methods . We want to run these methods parallelly. To achieve this, we need to add the below command in testng.xml
parallel="methods"
import static org.testng.Assert.assertTrue;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.testng.Assert;
import org.testng.annotations.Test;
import io.github.bonigarcia.wdm.WebDriverManager;
public class ParallelTestsExample {
@Test
public void invalidLoginTest() throws InterruptedException {
System.out.println("Test Case 1 with Thread Id - "+Thread.currentThread().getId());
WebDriver driver = WebDriverManager.chromedriver().create();
driver.manage().window().maximize();
driver.get("https://opensource-demo.orangehrmlive.com/");
driver.findElement(By.name("txtUsername")).sendKeys("admin123123");
driver.findElement(By.name("txtPassword")).sendKeys("adm");
driver.findElement(By.id("btnLogin")).click();
String expectedError = driver.findElement(By.id("spanMessage")).getText();
Assert.assertTrue(expectedError.contains("Invalid credentials"));
}
@Test
public void verifyLinkedIn() {
System.out.println("Test Case 2 with Thread Id - "+Thread.currentThread().getId());
WebDriver driver = WebDriverManager.chromedriver().create();
driver.manage().window().maximize();
driver.get("https://opensource-demo.orangehrmlive.com/");
Boolean linkedInIcon = driver.findElement(By.xpath("//*[@id='social-icons']/a[1]/img")).isEnabled();
System.out.println("Actual linkedIn Text :" + linkedInIcon);
assertTrue(linkedInIcon);
}
@Test
public void validLoginTest() throws InterruptedException {
System.out.println("Test Case 3 with Thread Id - "+Thread.currentThread().getId());
WebDriver driver = WebDriverManager.chromedriver().create();
driver.manage().window().maximize();
driver.get("https://opensource-demo.orangehrmlive.com/");
driver.findElement(By.name("txtUsername")).sendKeys("Admin");
driver.findElement(By.name("txtPassword")).sendKeys("admin123");
driver.findElement(By.id("btnLogin")).click();
String expectedTitle = driver.findElement(By.xpath("//*[@id='content']/div/div[1]/h1")).getText();
Assert.assertTrue(expectedTitle.contains("Dashboard"));
}
@Test
public void forgotLinkTest() throws InterruptedException {
System.out.println("Test Case 4 with Thread Id - "+Thread.currentThread().getId());
WebDriver driver = WebDriverManager.chromedriver().create();
driver.manage().window().maximize();
driver.get("https://opensource-demo.orangehrmlive.com/");
String expectedLink = driver.findElement(By.id("forgotPasswordLink")).getText();
Assert.assertTrue(expectedLink.contains("Forgot your password?"));
}
}
Now, let us create a testng.xml. Right click on the project and select TestNG -> Convert to TestNG.
The attribute thread-count allows you to specify how many threads should be allocated for this execution.
parallel = “methods” means that the methods will run parallel
The parallel attribute can be extended for multiple values, as below:
· Methods: Helps run methods in separate threads
· Tests: Help to run all methods belonging to the same tag in the same thread, means tests will run sequentially
· Classes: Helps to run all methods belonging to a class in a single thread
· Instances: Helps run all methods in the same instance in the same thread
testng.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
<test name="Parallel Tests" parallel = "methods" thread-count="4">
<classes>
<class name="com.example.parallel.ParallelTestsExample"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->
How to run the tests?
Right-click on testng.xml and select Run AS -> TestNG Suite. If you will run the Test Class – ParallelTestDemo.java as Right click and then Run As TestNG Tests, then the methods will run sequentially.
Execution
Here it can be seen that 4 tests were running on thread no – 19, 20, 21 and 22. Out of all 4 tests, browser for only 1 test is closed and rest 3 browsers are left open.
First thread initialized a browser and set a value to static WebDriver reference. Second thread initialized another browser and set a new value to the same static WebDriver reference and this will impact value set by first thread as it is a static.All threads wanted to close same browser that is the reason there is one configuration method failure as one browser is closed another threads will not find sessions to close the browsers. Browser was closed already so last 3 tests did not able to close the browser.
To overcome this issue, will use ThreadLocal<WebDriver>. The complete program looks like as below:
First, I will create a HelperClass which contains the initialization of driver and closing the driver. I like to keep the tests only in Test Class. This is not mandatory. You can combine the code of both classes in one also.
HelperClass
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;
import io.github.bonigarcia.wdm.WebDriverManager;
public class BaseClass {
private static final ThreadLocal<WebDriver> driver = new ThreadLocal<WebDriver>();
@BeforeMethod
public void setDriver() {
WebDriverManager.chromedriver().setup();
driver.set(new ChromeDriver());
driver.get().get("https://opensource-demo.orangehrmlive.com/");
driver.get().manage().window().maximize();
System.out.println("Before method Thread Id:" + Thread.currentThread().getId());
}
public WebDriver getDriver() {
return driver.get();
}
@AfterMethod
public void closeBrowser() {
System.out.println("After method Thread Id:" + Thread.currentThread().getId());
driver.get().quit();
driver.remove();
}
}
ParallelTestsExampleWithThread
import org.openqa.selenium.By;
import org.testng.Assert;
import org.testng.annotations.Test;
public class ParallelTestsExampleWithThread extends BaseClass {
@Test
public void invalidLoginTest() {
System.out.println("Test Case 1 with Thread Id - "+Thread.currentThread().getId());
getDriver().findElement(By.name("txtUsername")).sendKeys("admin123123");
getDriver().findElement(By.name("txtPassword")).sendKeys("adm");
getDriver().findElement(By.id("btnLogin")).click();
String expectedError = getDriver().findElement(By.id("spanMessage")).getText();
Assert.assertTrue(expectedError.contains("Invalid credentials"));
}
@Test
public void blankLoginTest() {
System.out.println("Test Case 2 with Thread Id - "+Thread.currentThread().getId());
getDriver().findElement(By.name("txtUsername")).sendKeys("");
getDriver().findElement(By.name("txtPassword")).sendKeys("");
getDriver().findElement(By.id("btnLogin")).click();
String expectedError = getDriver().findElement(By.id("spanMessage")).getText();
Assert.assertTrue(expectedError.contains("Username cannot be empty"));
}
@Test
public void validLoginTest() throws InterruptedException {
System.out.println("Test Case 3 with Thread Id - "+Thread.currentThread().getId());
getDriver().findElement(By.name("txtUsername")).sendKeys("Admin");
getDriver().findElement(By.name("txtPassword")).sendKeys("admin123");
getDriver().findElement(By.id("btnLogin")).click();
Thread.sleep(5000);
String expectedTitle = getDriver().findElement(By.xpath("//*[@id='content']/div/div[1]/h1")).getText();
Assert.assertTrue(expectedTitle.contains("Dashboard"));
}
@Test
public void forgotLinkTest() {
System.out.println("Test Case 4 with Thread Id - "+Thread.currentThread().getId());
String expectedLink = getDriver().findElement(By.id("forgotPasswordLink")).getText();
Assert.assertTrue(expectedLink.contains("Forgot your password?"));
}
}
The output of the above program is
Report Generation
TestNG generates 2 reports – emailable-report.html and index.html
Emailable-Report.html
Go to test-output folder and open emailable-report.html
This report gives a summary of all the tests executed, passed, failed, skipped and retried with their respective execution time.
Index.html
This report provides the detailed description of the tests like no of tests present, no of methods, time taken by each step, total time taken by each steps, testng.xml data and soon.
Run Tests Sequentially
If you will run the Test Class – ParallelTestDemo.java as Right click and then Run As TestNG Tests, then the methods will run sequentially. Here all tests are run with Thread 1 whereas with parallel execution tests were run with different threads.
We can make parallel = none, if don’t want to run them parallel. It is shown below that all the tests are running on Thread 1 that means once a test ends then another test starts on that thread.
QA Automation Expert 