TestNG was developed by a famous programmer named “Cedric Beust”. It is distributed under Apache Software License and is easily available to download. TestNG requires JDK 7 or higher. TestNG is a testing framework inspired by JUnit and NUnit, but introduces some new functionalities that make it more powerful and easier to use.
Jenkins is a self-contained, open-source automation server that can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software.
Jenkins can be installed through native system packages, Docker, or even run standalone by any machine with a Java Runtime Environment (JRE) installed.
BDD is a set of practices that helps to reduce the rework caused by misunderstanding or vague requirements, narrow the communication gaps between the development team, testing team, and customers, and promote continuous communication among them. Cucumber is one such open-source tool, which supports Behaviour Driven Development(BDD). In simple words, Cucumber can be defined as a testing framework, driven by plain English. It serves as documentation, automated tests, and a development aid – all in one.
Cucumber Introduction, Installation, and Configuration
To generate Allure Report in Jenkins, we need to download Allure Plugin. Please refer to this tutorial to install the plugin – How to install Plugins in Jenkins
Configure Allure
Go back to the Manage Jenkins link as shown below:
When we click on the “Manage Jenkins” link, we are redirected to the Manage Jenkins page, where we can see various types of options, including the “Global Tool Configuration” option.
We need to set the Allure Commandline in Jenkinsas shown below.
Click on the Allure Command line installations button. By default, “Install Automatically” will be checked, so since we are going to use the Allure installed on our local machine, “Install automatically” will install the latest version of Allure.
Provide the Name as ALLURE_HOME because that is what is currently installed on my machine, and also provide the path of Allure in the ALLURE_HOME textbox.
Click on the Apply and Save buttons.
Create a new Maven Jenkins job
Step 1: Create a new project using the Maven project plugin
Give the Name of the project – AllureReportWithSelenium_Demo
Click on the Maven project.
Click on the OK button.
In the General section, enter the project description in the Description box – This is demo of Allure Report in Jenkins.
Select Source Code Management as None if the project is locally present on the machine.
Step 2: Build Management
Go to the Build section of the new job.
In the Root POM textbox, enter pom.xml
In the Goals and options section, enter “clean test”
Click on the Advanced button.
Step 3: Select a custom workspace
Mention the full path of the project in the directory.
Step 4: Select “Allure Reports” from “Post Build Actions”
Scroll down to ‘Post Build Actions’ and click on the ‘Add Post Build Actions’ drop-down list. Select “Allure Report“.
Enter the Result Path as “allure-results” and click on the “Save” button.
Click on the Apply and Save buttons.
We have created a new Maven project “AllureReportWithSelenium_Demo” with the configuration to run the Selenium with TestNG Tests and also to generate Allure Report after execution using Jenkins.
Step 5: Execute the tests
Let’s execute it now by clicking on the “Build Now” button.
Right-click on Build Number (here in my case it is #1).
Click on Console Output to see the result.
Step 6: View the Allure Report
Once the execution is completed, we could see a link to view the ‘Allure Report’.
Click on the Allure Report. It displays the summary of the tests.
There is another way to create Allure Report in Jenkins, which is by using the Jenkins pipeline. To know more about this, please refer to this tutorial – How to create Jenkins pipeline for Allure Report.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
In the previous tutorial, I explained the Integration of the Allure Report with Rest Assured with JUnit4. In this tutorial, I will explain how to Integrate Allure Report with Rest Assured and TestNG.
The below example covers the implementation of Allure Report for Rest API using Rest Assured, TestNG, Java, and Maven.
Step 4 – Create the Test Code for the testing of REST API under src/test/java
To see our request and response in more detail using Rest Assured, we need to add a line to our Rest Assured tests. This will provide the request and response details in the report.
.filter(new AllureRestAssured())
import io.qameta.allure.*;
import io.qameta.allure.restassured.AllureRestAssured;
import io.restassured.http.ContentType;
import org.json.JSONObject;
import org.testng.annotations.Test;
import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
@Epic("REST API Regression Testing using TestNG")
@Feature("Verify CRUID Operations on User module")
public class RestAPITests {
@Test(description = "To get the details of user with id 3", priority = 0)
@Story("GET Request with Valid User")
@Severity(SeverityLevel.NORMAL)
@Description("Test Description : Verify the details of user of id-3")
public void verifyUser() {
// Given
given()
.filter(new AllureRestAssured())
// When
.when()
.get("https://reqres.in/api/users/3")
// Then
.then()
.statusCode(200)
.statusLine("HTTP/1.1 200 OK")
// To verify user of id 3
.body("data.email", equalTo("emma.wong@reqres.in"))
.body("data.first_name", equalTo("Emma"))
.body("data.last_name", equalTo("Wong"));
}
@Test(description = "To create a new user", priority = 1)
@Story("POST Request")
@Severity(SeverityLevel.NORMAL)
@Description("Test Description : Verify the creation of a new user")
public void createUser() {
JSONObject data = new JSONObject();
data.put("name", "RestAPITest");
data.put("job", "Testing");
// GIVEN
given()
.filter(new AllureRestAssured())
.contentType(ContentType.JSON)
.body(data.toString())
// WHEN
.when()
.post("https://reqres.in/api/users")
// THEN
.then()
.statusCode(201)
.body("name", equalTo("RestAPITest"))
.body("job", equalTo("Testing"));
}
}
Step 5 – Create testng.xml for the project
<?xml version = "1.0"encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name = "Suite1">
<test name = "TestNG Test Demo">
<classes>
<class name = "org.example.RestAPITests"/>
</classes>
</test>
</suite>
Step 6 – Run the Test and Generate Allure Report
To run the tests, use the below command
mvn clean test
In the below image, we can see that all three tests are passed.
This will create the allure-results folder with all the test reports. These files will be used to generate the Allure Report.
To create an Allure Report, use the below command
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes, can be found.
View test history
Each time you run the report from the command line with the mvn clean test command, a new result JSON file will get added to the allure-results folder. Allure can use those files to include a historical view of your tests. Let’s give that a try.
To get started, run mvn clean test a few times and watch how the number of files in the allure-reports folder grows.
Now go back to view your report. Select Suites from the left nav, select one of your tests and click Retries in the right pane. You should see the history of test runs for that test:
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, and Story tags.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
The previous tutorial explained the generation of Allure Report with Cucumber5, Selenium and JUnit4 in a Maven project. In this tutorial, I will explain the steps to create an Allure Report with Cucumber7, Selenium, and JUnit5 in a Maven project.
Step 2 – Create Step Definition, feature file, and Test Runner Classes
Create Locator and Action classes and Step Definition corresponding to the feature file and Test Runner Class
There is another tutorial that explains the project structure as well as the feature file and corresponding Step Definitions, please refer to this tutorial – Integration of Cucumber7 with Selenium and JUnit5.
Step 3 – Execute the Tests
Use the below command to run the tests
mvn clean test
The output of the above program is
Step 4 – Generate the Allure Report
Once the test execution is finished, a folder named allure-results will be generated in thetarget folder.
To generate the Allure Report, first, go to the target folder.
cd target
Now, use the below command to generate the Allure Report
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Statistics – overall report statistics.
Launches – if this report represents several test launches, statistics per launch will be shown here.
Behaviours – information on results aggregated according to stories and features.
Executors – information on test executors that were used to run the tests.
History Trend – if tests accumulate some historical data, its trend will be calculated and shown on the graph.
Environment – information on the test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, Story, Test Severity, Test Description, Test Steps, and so on.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what has been tested in a neat web report form but allows everyone participating in the development process to extract maximum useful information from everyday execution of tests.
From the dev/qa perspective, Allure reports shorten common defect lifecycle: test failures can be divided into bugs and broken tests, also logs, steps, fixtures, attachments, timings, history, and integrations with TMS and bug-tracking systems can be configured, so the responsible developers and testers will have all information at hand.
Step 1.1 – Open PyCharm and create a new project. Go to File and select New Project from the main menu.
Step 1.2 – Choose the project location. Click the “Browse” button next to the Location field and specify the directory for your project.
Deselect the Create a main.py welcome script checkbox because you will create a new Python file for this tutorial.
Click on the “Create” Button.
Step 1.3 – A new dialog appears asking to open the project using any one of the given options. I have selected New Window as I like to have separate windows for each project.
Below is the image of the new project created in PyCharms.
Step 2 – Install allure-robotframework plugin
Go to the command prompt and run the below-mentioned command to download the plugin:
pip install allure-robotframework
The below image shows that the plugin is installed successfully.
Step 3 – Add allure-robotframework package to the PyCharms
Go to File->Settings ->Project:RobotFramework_Demo ->Python Interpreter.
Click on the “+” sign and enter allure-r in the search bar. It will show a list of packages. Select the “allure-robotframework” package and click on the “Install Package”.
Once the package is installed, we will see the message that the package is installed successfully.
Once the package is installed, it can be seen under the package list as shown below:
Step 4 – Create 3 new directories in the new project
Right-Click on the project, select New->Directory and provide the name as TestCases, Drivers, and Resources
We have the test case passed. The Robot Framework generates log.html, output.xml, and report.html by default.
These reports are generated in the output/robot folder as I have provided –outputdir ./output/robot in the command line.
Let us now see the report and log details.
Report
Right-click on report.html. Select Open In->Browser->Chrome(any browser of your wish).
As one of the tests purposefully failed to demonstrate how the report would look in the event of a failure, we can see that it is now a Red report. The framework generated the following report:
Log
Robot Framework has multiple log levels that control what is shown in the automatically generated log file. The default Robot Framework log level is INFO.
Right-click on log.html. Select Open In->Browser->Chrome(any browser of your wish).
Step 8 – Generate the Allure Report
To create Allure Report, use the below command
allure serve ./output/allure
This will generate the beautiful Allure Test Report as shown below.
Step 9 – Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Statistics – overall report statistics.
Launches – if this report represents several test launches, statistics per launch will be shown here.
Behaviours – information on results aggregated according to stories and features.
Executors – information on test executors that were used to run the tests.
History Trend – if tests accumulated some historical data, it’s trend will be calculated and shown on the graph.
Environment – information on the test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: statuses breakdown or severity and duration diagrams.
Timeline in Allure Report
The timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviours of Allure Report
This tab groups test results according to Epic, Feature, and Story tags.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
That’s it! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
Robot Framework is a generic open-source automation framework. It can be used for test automation and robotic process automation (RPA). RPA is extensively used for Web Application Automation, API Automation, RPA, and Database Testing.
Robot Framework has an easy syntax, utilizing human-readable keywords. Its capabilities can be extended by libraries implemented with Python, Java, or many other programming languages.
Jenkins was installed and started on the computer.
Implementation Steps
Step 1: Create a new pipeline project
Give the Name of the project – Allure_PipelineDemo
Click on the pipeline project.
Click on the OK button.
In the General section, enter the project description in the Description box.
Step 2: Scroll down to Pipeline
From the Definition field, choose the “Pipeline script from SCM” option. This option instructs Jenkins to obtain your Pipeline from Source Control Management (SCM), which will be your locally cloned Git repository.
From the SCM field, choose Git.
The Repositories section contains the Repository URL and Credentials.
In the Repository URL field, specify the directory path of the GitLab/GitHub project.
In the Credentials field, specify the username and password needed to log in to GitLab/GitHub.
In this case, I have the project is present in GitLab and using it.
Step 3: Create Jenkinsfile
Create and save a new text file with the name Jenkinsfile at the root of the project in the GitLab repository. Here, we are using the Selenium project with TestNG. To know more about the Integration of Selenium with TestNG, please refer to this tutorial –
For this tutorial, we are using Declarative syntax. The sample example is given below:
pipeline {
agent any
stages {
stage('Test') {
steps {
bat "mvn -D clean test"
}
post {
// If Maven was able to run the tests, even if some of the test
// failed, record the test results and archive the jar file.
success { allure([
includeProperties: false,
jdk: '',
properties: [],
reportBuildPolicy: 'ALWAYS',
results: [[path: 'target/allure-results']]
])
}
}
}
}
}
Step 4: Specify branches to build a section under Repositories
Branch Specifier – */master (This is my main branch)
ScriptPath – Jenkinsfile
Click on the Apply and Save buttons.
We have created a new Maven project “AllurePipelineDemo” with the configuration to run the Selenium Test with TestNG.
Step 5: Execute the tests
Let’s execute it now by clicking on the “Build Now” button.
Right-click on Build Number (here in my case it is #4) and click on Console Output to see the result.
Below is the test execution summary.
Step 6: Pipeline Steps
Once the execution is completed, and we want to see the Pipeline Steps, click on the Pipeline Steps mentioned on the left side of the page.
Step 7: View the Report
Once the execution is completed, go back to “Allure_PipelineDemo”. We can see below that the Allure Report is generated.
We could see a link to view the ‘Allure Report’. Click on the Allure Report. It displays the Report.
Tip: If you don’t see the Report UI intact, then you need to configure a simple groovy script. For that, go to Dashboard–>Manage Jenkins–>Script Console and add the script as:
Environment variables JAVA_HOME, ALLURE_HOME and GRADLE_HOME correctly configured
In this tutorial, I’ll create a BDD Framework for creating an Allure Report for Selenium WebDriver with JUnit5. This framework consists of:-
Dependency List
Java 11
JUnit– 5.8.2
Gradle – 7.3.3
Selenium – 4.3.0
Allure Report – 2.19
AspectJ Weaver – 1.9.7
Project Structure
Implementation Steps
There is another tutorial that explains the steps to create a Gradle Project with Selenium and JUnit5. Please refer to this tutorial –Gradle – Integration of Selenium and JUnit5.
Step 1 – Add dependencies to build.gradle
Add Selenium, JUnit5, and Allure-JUnit5 dependencies in build.gradle
/*
* This file was generated by the Gradle 'init' task.
*
*/
plugins {
// Apply the application plugin to add support for building a CLI application in Java.
id 'application'
id 'io.qameta.allure' version '2.11.0'
}
repositories {
// Use Maven Central for resolving dependencies.
mavenCentral()
}
java {
sourceCompatibility = 11
targetCompatibility = 11
}
dependencies {
// Use JUnit Jupiter for testing.
testImplementation 'org.junit.jupiter:junit-jupiter:5.8.2'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.8.2'
// Allure
implementation 'io.qameta.allure:allure-junit5:2.19.0'
runtimeOnly 'org.aspectj:aspectjweaver:1.9.7'
// This dependency is used by the application.
implementation 'com.google.guava:guava:30.1.1-jre'
implementation 'org.seleniumhq.selenium:selenium-java:4.4.0'
implementation 'io.github.bonigarcia:webdrivermanager:5.3.0'
}
application {
// Define the main class for the application.
mainClass = 'com.example.App'
}
tasks.named('test') {
// Use JUnit Platform for unit tests.
useJUnitPlatform() {
}
}
Step 2 – Create Pages and Test Code for the pages
As mentioned above, there is another tutorial that explains the project structure as well as the feature file and corresponding Step Definitions, please refer to this tutorial – Gradle Project with Selenium and JUnit5.
Below is the sample test code. I have added features of the allure report like @Severity, @Description.
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Disabled;
import io.qameta.allure.Description;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;
public class LoginPageTests extends BaseTests{
@Severity(SeverityLevel.NORMAL)
@ParameterizedTest
@CsvSource({
"admin$$,admin123",
"Admin,admin123!!",
"admin123,Admin",
"%%%%%,$$$$$$"})
@Description("Test Description : Login Test with invalid credentials")
public void invalidCredentials(String username, String password) {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login(username, password);
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getErrorMessage());
}
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Login Test with valid credentials")
public void validLogin() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("Admin", "admin123");
HomePage objHomePage = new HomePage(driver);
// Verify Home Page
assertEquals("Employee Information",objHomePage.getHomePageText());
}
@Severity(SeverityLevel.NORMAL)
@Test
@Description("Test Description : Login Test with missing username - Failed Test")
public void missingUsername() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("", "admin123");
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getMissingUsernameText());
}
@Severity(SeverityLevel.NORMAL)
@Test @Disabled
@Description("Test Description : Login Test with missing password")
public void missingPassword() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("admin", "");
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getMissingPasswordText());
}
}
Below is the BaseTest class, where I have shown the use of @Step of Allure Report.
import java.time.Duration;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import io.github.bonigarcia.wdm.WebDriverManager;
import io.qameta.allure.Step;
public class BaseTests {
public WebDriver driver;
public final static int TIMEOUT = 10;
@BeforeEach
@Step("Start the application")
public void setup() {
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
driver.manage().window().maximize();
driver.get("https://opensource-demo.orangehrmlive.com/");
driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(TIMEOUT));
}
@AfterEach
@Step("Stop the application")
public void tearDown() {
driver.quit();
}
}
Step 3 – Execute the Tests
Go to the app project and run the tests, using the below command
gradle clean test
The output of the test execution is
Step 4 – Generate the Allure Report
Once the test execution is finished, a folder named allure-results will be generated in thebuild folder.
Note:- Make sure that you move to the folder app, because the build folder is present in the app folder.
To generate an Allure Report, use the below command.
allure serve build/allure-results
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Statistics – overall report statistics.
Launches – if this report represents several test launches, statistics per launch will be shown here.
Behaviors – information on results aggregated according to stories and features.
Executors – information on test executors that were used to run the tests.
History Trend – if tests accumulated some historical data, its trend will be calculated and shown on the graph.
Environment – information on the test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes, can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, Story, Test Severity, Test Description, Test Steps, and so on.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
QA Automation Expert 