Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what has been tested in a neat web report form but allows everyone participating in the development process to extract the maximum useful information from the everyday execution of tests.
How Allure Report is generated?
Allure is based on standard xUnit results output but adds some supplementary data. Any report is generated in two steps. During test execution (first step), a small library called adapter attached to the testing framework saves information about executed tests to XML files. We already provide adapters for popular Java, PHP, Ruby, Python, Scala, and C# test frameworks. During report generation (second step), the XML files are transformed into an HTML report. This can be done with a command line tool, a plugin for CI, or a build tool.
Similarly, when we run our tests, every popular test framework generates junit-style XML report or testng style which will be used by Allure to generate HTML report.
In the below example, we use the maven surefire plugin which automatically generates xml test reports and stores them in target/surefire-reports. And these XML files are transformed into an HTML report by Allure.
Allure reports have provided adapters for Java, PHP, Ruby, Python, Scala, and C# test frameworks.
Allure report has the below-mentioned annotation.
@Epic @Features @Stories/@Story
We can add Epic, Feature, and Stories annotations to the test to describe the behaviour of the test.
@Severity(SeverityLevel.BLOCKER) – @Severity annotation is used in order to prioritize test methods by severity.
@Description(“Regression Testing”)– We can add a detailed description for each test method. To add such a description, use the @Description annotation.
@Step – In order to define steps in Java code, you need to annotate the respective methods with @Step annotation. When not specified, the step name is equal to the annotated method name.
@Attachment– An attachment in Java code is simply a method annotated with@Attachment that returns either a String or byte[], which should be added to the report.
@Link – We can link the tests to Defect Tracker or JIRA Ticket.
Below is an example that shows how to use various Allure Annotations in the Test.
@Epic("Web Application Regression Testing")
@Feature("Login Page Tests")
@Listeners(TestExecutionListener.class)
public class LoginTests extends BaseTest {
LoginPage objLogin;
DashboardPage objDashboardPage;
@Severity(SeverityLevel.NORMAL)
@Test(priority = 0, description = "Verify Login Page")
@Description("Test Description : Verify the title of Login Page")
@Story("Title of Login Page")
public void verifyLoginPage() {
// Create Login Page object
objLogin = new LoginPage(driver);
// Verify login page text
objLogin.verifyPageTitle();
}
}
Install Allure
For Windows, Allure is available from the Scoop command line installer.
Step 4 – Create a Test Code for the testing of REST API under src/test/java
Rest Assured and Allure Report are two popular tools for testing. Rest Assured is used for API testing and Allure Report is used for creating detailed reports about tests. To see our request and response in more detail using these tools, we need to add a line to our Rest Assured tests. This will provide the request and response details in the report.
For testing purpose, first test – Get Request one fails.
.filter(new AllureRestAssured())
Below is an example of the tests.
package org.example;
import io.qameta.allure.*;
import io.qameta.allure.restassured.AllureRestAssured;
import io.restassured.http.ContentType;
import org.json.JSONObject;
import org.junit.Test;
import static io.restassured.RestAssured.given;
import static org.hamcrest.core.IsEqual.equalTo;
@Epic("REST API Regression Testing using JUnit4")
@Feature("Verify CRUID Operations on Employee module")
public class APITests {
String BaseURL = "https://dummy.restapiexample.com/api";
@Test
@Story("GET Request")
@Severity(SeverityLevel.NORMAL)
@Description("Test Description : Verify the details of employee of id-2")
public void getUser() {
// GIVEN
given()
.filter(new AllureRestAssured())
// WHEN
.when()
.get(BaseURL + "/v1/employee/2")
// THEN
.then()
.statusCode(200)
.statusLine("HTTP/1.1 200 OK")
// To verify booking id at index 2
.body("data.employee_name", equalTo("Garrett Winters!"))
.body("message", equalTo("Successfully! Record has been fetched."));
}
@Test
@Story("POST Request")
@Severity(SeverityLevel.NORMAL)
@Description("Test Description : Verify the creation of a new employee")
public void createUser() {
JSONObject data = new JSONObject();
data.put("employee_name", "APITest");
data.put("employee_salary", "99999");
data.put("employee_age", "30");
// GIVEN
given()
.filter(new AllureRestAssured())
.contentType(ContentType.JSON)
.body(data.toString())
// WHEN
.when()
.post(BaseURL + "/v1/create")
// THEN
.then()
.statusCode(200)
.body("data.employee_name", equalTo("APITest"))
.body("message", equalTo("Successfully! Record has been added."));
}
}
Step 5 – Run the Test and Generate Allure Report
To run the tests, use the below command
mvn clean test
The output of the above program is
This will create allure-results folder with all the test reports. These files will be used to generate Allure Report.
To create Allure Report, use the below command
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
Categories in Allure Report
The categories tab gives you a way to create custom defects classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.
View test history
Each time you run the report from the command line with the mvn clean test command, a new result JSON file will get added to the allure-results folder. Allure can use those files to include a historical view of your tests. Let’s give that a try.
To get started, run mvn clean test a few times and watch how the number of files in the allure-reports folder grows.
Now go back to view your report. Select Suites from the left nav, select one of your tests and click Retries in the right pane. You should see the history of test runs for that test:
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, and Story tags.
The below image shows the request body sent and the status code of the response, its body, and header provided by API.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
Allure Framework is a lightweight, flexible multi-language test report tool that not only displays a very concise representation of what has been tested in a neat web report form, but also allows everyone involved in the development process to extract the most useful information from everyday test execution.
The below example covers the implementation of Allure Reports with Cucumber, Selenium, TestNG, Java, and Maven. Before starting, make sure to install Allure on your machine. Refer to this tutorial to install allure – What is Allure Report?.
Create a folder – features within src/test/resources to create test scenarios in the Feature file.
Feature file should be saved as an extension of .feature. Add the test scenarios in this feature file. I have added sample test scenarios. In this feature file. The test scenarios are written in Gherkinslanguage.
Feature: Login to HRM Application
Background:
Given User is on HRMLogin page "https://opensource-demo.orangehrmlive.com/"
@ValidCredentials
Scenario: Login with valid credentials
When User enters username as "Admin" and password as "admin123"
Then User should be able to login successfully and new page open
@InvalidCredentials
Scenario Outline: Login with invalid credentials
When User enters username as "<username>" and password as "<password>"
Then User should be able to see error message "<errorMessage>"
Examples:
| username | password | errorMessage |
| Admin | admin12$$ | Invalid credentials |
| admin$$ | admin123 | Invalid credentials |
| abc123 | xyz$$ | Invalid credentials |
| 234 | xyz$$ | Invalid credentials! |
Step 5 – Create the Step Definition class or Glue Code
The stepdefinition class is created in src/test/java directory.
Below is the code for the Hooks.
package com.example.definitions;
import io.cucumber.java.After;
import io.cucumber.java.Before;
import io.cucumber.java.Scenario;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import java.time.Duration;
public class Hooks {
protected static WebDriver driver;
public final static int TIMEOUT = 5;
@Before
public void setUp() {
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
driver = new ChromeDriver(options);
driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(TIMEOUT));
}
@After
public void tearDown(Scenario scenario) {
try {
String screenshotName = scenario.getName();
if (scenario.isFailed()) {
TakesScreenshot ts = (TakesScreenshot) driver;
byte[] screenshot = ts.getScreenshotAs(OutputType.BYTES);
scenario.attach(screenshot, "img/png", screenshotName);
}
} catch (Exception e) {
e.printStackTrace();
}
driver.quit();
}
}
LoginPageDefinition
package com.example.definitions;
import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import org.openqa.selenium.By;
import org.testng.Assert;
public class LoginPageDefinitions {
Hooks hooks;
@Given("User is on HRMLogin page {string}")
public void loginTest(String url) {
hooks.driver.get(url);
}
@When("User enters username as {string} and password as {string}")
public void goToHomePage(String userName, String passWord) {
// login to application
hooks.driver.findElement(By.name("username")).sendKeys(userName);
hooks.driver.findElement(By.name("password")).sendKeys(passWord);
hooks.driver.findElement(By.xpath("//*[@class='oxd-form']/div[3]/button")).submit();
// go the next page
}
@Then("User should be able to login successfully and new page open")
public void verifyLogin() {
String homePageHeading = hooks.driver.findElement(By.xpath("//*[@class='oxd-topbar-header-breadcrumb']/h6")).getText();
//Verify new page - HomePage
Assert.assertEquals(homePageHeading,"Dashboard");
}
@Then("User should be able to see error message {string}")
public void verifyErrorMessage(String expectedErrorMessage) {
String actualErrorMessage = hooks.driver.findElement(By.xpath("//*[@class='orangehrm-login-error']/div[1]/div[1]/p")).getText();
// Verify Error Message
Assert.assertEquals(actualErrorMessage, expectedErrorMessage);
}
}
Step 6 – Create a TestNG Cucumber Runner class
We need to create a class called Runner class to run the tests. This class will use the TestNG annotation @Test, which tells TestNG what is the test runner class.
package com.example.runner;
import org.testng.annotations.Test;
import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;
@Test
@CucumberOptions(tags = "", features = {"src/test/resources/features"}, glue = {"com.example.definitions"},
plugin = {"pretty","io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"})
public class CucumberRunnerTests extends AbstractTestNGCucumberTests{
}
Note:- @Test annotation marks this class as part of the test. So, if we will remove this annotation, the Allure Report executesCucumberRunnerTests as a separate test suite, so there will be duplicate results.
Step 7 – Create testng.xml for the project
<?xml version = "1.0"encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name = "Suite1">
<test name = "Test Demo">
<classes>
<class name = "com.example.runner.CucumberRunnerTests"/>
</classes>
</test>
</suite>
Step 8 – Run the Test and Generate Allure Report
To run the tests, use the below command
mvn clean test
In the below image, we can see that one test failed and four passed out of five tests.
This will create the allure-results folder with all the test reports within target folder. These files will be used to generate Allure Report.
Use the below command to generate the Allure Report
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found. Here, we have 2 suits – Feature and Surefire test. Surefire tests are executed from CucumberRunnerTests.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviours of Allure Report
This tab groups test results according to Epic, Feature, and Story tags.
Screenshot attached to the failed test case
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
When we don’t use @Test in CucumberRunnerTests.java, then as mentioned above the Allure report will have duplicate details.
Congratulations!! We have integrated an allure report with Cucumber, Selenium, and TestNG. I hope this tutorial is useful to you.
Step 4 – Create Feature file in src/test/resources
Create source folder – src/test/resources and features folder within src/test/resources to create test scenarios in Feature file. Feature file should be saved as an extension of .feature. Add the test scenarios in this feature file. I have added sample test scenarios. In this feature file, I have created a scenario for successful login and one Scenario Outline for failed login. The test scenarios are written in Gherkins language.
Feature: Login to HRM Application
Background:
Given User is on HRMLogin page "https://opensource-demo.orangehrmlive.com/"
@ValidCredentials
Scenario: Login with valid credentials
When User enters username as "Admin" and password as "admin123"
Then User should be able to login successfully and new page open
@InvalidCredentials
Scenario Outline: Login with invalid credentials
When User enters username as "<username>" and password as "<password>"
Then User should be able to see error message "<errorMessage>"
Examples:
| username | password | errorMessage |
| Admin | admin12$$ | Invalid credentials |
| admin$$ | admin123 | Invalid credentials |
| abc123 | xyz$$ | Invalid credentials |
| 234 | xyz$$ | Invalid credentials! |
Step 5 – Create the Step Definition class or Glue Code
The stepdefinition class is created in src/test/java.
Below is the code for the LoginDefinition class.
package com.example.definitions;
import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import org.junit.Assert;
import org.openqa.selenium.By;
public class LoginPageDefinitions {
Hooks hooks;
@Given("User is on HRMLogin page {string}")
public void loginTest(String url) {
hooks.driver.get(url);
}
@When("User enters username as {string} and password as {string}")
public void goToHomePage(String userName, String passWord) {
// login to application
hooks.driver.findElement(By.name("username")).sendKeys(userName);
hooks.driver.findElement(By.name("password")).sendKeys(passWord);
hooks.driver.findElement(By.xpath("//*[@class='oxd-form']/div[3]/button")).submit();
// go the next page
}
@Then("User should be able to login successfully and new page open")
public void verifyLogin() {
String homePageHeading = hooks.driver.findElement(By.xpath("//*[@class='oxd-topbar-header-breadcrumb']/h6")).getText();
//Verify new page - HomePage
Assert.assertEquals("Dashboard",homePageHeading);
}
@Then("User should be able to see error message {string}")
public void verifyErrorMessage(String expectedErrorMessage) {
String actualErrorMessage = hooks.driver.findElement(By.xpath("//*[@class='orangehrm-login-error']/div[1]/div[1]/p")).getText();
// Verify Error Message
Assert.assertEquals(expectedErrorMessage,actualErrorMessage);
}
}
Hook.java
package com.example.definitions;
import io.cucumber.java.After;
import io.cucumber.java.Before;
import io.cucumber.java.Scenario;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import java.time.Duration;
public class Hooks {
protected static WebDriver driver;
public final static int TIMEOUT = 5;
@Before
public void setUp() {
ChromeOptions options = new ChromeOptions();
options.addArguments("--start-maximized");
driver = new ChromeDriver(options);
driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(TIMEOUT));
}
@After
public void tearDown(Scenario scenario) {
try {
String screenshotName = scenario.getName();
if (scenario.isFailed()) {
TakesScreenshot ts = (TakesScreenshot) driver;
byte[] screenshot = ts.getScreenshotAs(OutputType.BYTES);
scenario.attach(screenshot, "img/png", screenshotName);
}
} catch (Exception e) {
e.printStackTrace();
}
driver.quit();
}
}
Step 6 – Create a Cucumber Runner class
We need to create a class called Runner class to run the tests. This class will use the JUnit annotation @RunWith(), which tells JUnit what is the test runner class.
package com.example.runner;
import org.junit.runner.RunWith;
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
@RunWith(Cucumber.class)
@CucumberOptions(tags = "", features = {"src/test/resources/features"}, glue = {"com.example.definitions"},
plugin = {"pretty","io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"})
public class CucumberRunnerTests {
}
Step 7 – Create allure.properties in src/test/resources
Allure, by default, saves test results in the project’s root directory. However, it is recommended to store your test results in the build output directory. To configure this, create an allure.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:
allure.results.directory=target/allure-results
Step 8 – Run the Test and Generate Allure Report
To run the tests, use the below command
mvn clean test
In the below image, we can see that one test is failed and four passed out of five tests.
This will create allure-results folder with all the test report. These files will be use to generate Allure Report.
Step 9 – Generate Allure Report
Change current directory to target directory and then use the below command to generate the Allure Report
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
It shows detail of all the test steps and the screenshot of the failed test step also as shown below.
Categories in Allure Report
Categories tab gives you the way to create custom defects classification to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: statuses breakdown or severity and duration diagrams.
Timeline in Allure Report
Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature and Story tags.
Packages in Allure Report
Packages tab represents a tree-like layout of test results, grouped by different packages.
That’s it! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
To generate Allure Report in Jenkins, we need to download Allure Plugin. Please refer to this tutorial to install the plugin – How to install Plugins in Jenkins
Configure Allure
Go back to the Manage Jenkins link as shown below:
When we click on the “Manage Jenkins” link, we are redirected to the Manage Jenkins page, where we can see various types of options, including the “Global Tool Configuration” option.
We need to set the Allure Commandline in Jenkinsas shown below.
Click on the Allure Command line installations button. By default, “Install Automatically” will be checked, so since we are going to use the Allure installed on our local machine, “Install automatically” will install the latest version of Allure.
Provide the Name as ALLURE_HOME because that is what is currently installed on my machine, and also provide the path of Allure in the ALLURE_HOME textbox.
Click on the Apply and Save buttons.
Create a new Maven Jenkins job
Step 1: Create a new project using the Maven project plugin
Give the Name of the project – AllureReportWithSelenium_Demo
Click on the Maven project.
Click on the OK button.
In the General section, enter the project description in the Description box – This is demo of Allure Report in Jenkins.
Select Source Code Management as None if the project is locally present on the machine.
Step 2: Build Management
Go to the Build section of the new job.
In the Root POM textbox, enter pom.xml
In the Goals and options section, enter “clean test”
Click on the Advanced button.
Step 3: Select a custom workspace
Mention the full path of the project in the directory.
Step 4: Select “Allure Reports” from “Post Build Actions”
Scroll down to ‘Post Build Actions’ and click on the ‘Add Post Build Actions’ drop-down list. Select “Allure Report“.
Enter the Result Path as “allure-results” and click on the “Save” button.
Click on the Apply and Save buttons.
We have created a new Maven project “AllureReportWithSelenium_Demo” with the configuration to run the Selenium with TestNG Tests and also to generate Allure Report after execution using Jenkins.
Step 5: Execute the tests
Let’s execute it now by clicking on the “Build Now” button.
Right-click on Build Number (here in my case it is #1).
Click on Console Output to see the result.
Step 6: View the Allure Report
Once the execution is completed, we could see a link to view the ‘Allure Report’.
Click on the Allure Report. It displays the summary of the tests.
There is another way to create Allure Report in Jenkins, which is by using the Jenkins pipeline. To know more about this, please refer to this tutorial – How to create Jenkins pipeline for Allure Report.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
In the previous tutorial, I explained the Integration of the Allure Report with Rest Assured with JUnit4. In this tutorial, I will explain how to Integrate Allure Report with Rest Assured and TestNG.
The below example covers the implementation of Allure Report for Rest API using Rest Assured, TestNG, Java, and Maven.
Step 4 – Create the Test Code for the testing of REST API under src/test/java
To see our request and response in more detail using Rest Assured, we need to add a line to our Rest Assured tests. This will provide the request and response details in the report.
.filter(new AllureRestAssured())
import io.qameta.allure.*;
import io.qameta.allure.restassured.AllureRestAssured;
import io.restassured.http.ContentType;
import org.json.JSONObject;
import org.testng.annotations.Test;
import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
@Epic("REST API Regression Testing using TestNG")
@Feature("Verify CRUID Operations on User module")
public class RestAPITests {
@Test(description = "To get the details of user with id 3", priority = 0)
@Story("GET Request with Valid User")
@Severity(SeverityLevel.NORMAL)
@Description("Test Description : Verify the details of user of id-3")
public void verifyUser() {
// Given
given()
.filter(new AllureRestAssured())
// When
.when()
.get("https://reqres.in/api/users/3")
// Then
.then()
.statusCode(200)
.statusLine("HTTP/1.1 200 OK")
// To verify user of id 3
.body("data.email", equalTo("emma.wong@reqres.in"))
.body("data.first_name", equalTo("Emma"))
.body("data.last_name", equalTo("Wong"));
}
@Test(description = "To create a new user", priority = 1)
@Story("POST Request")
@Severity(SeverityLevel.NORMAL)
@Description("Test Description : Verify the creation of a new user")
public void createUser() {
JSONObject data = new JSONObject();
data.put("name", "RestAPITest");
data.put("job", "Testing");
// GIVEN
given()
.filter(new AllureRestAssured())
.contentType(ContentType.JSON)
.body(data.toString())
// WHEN
.when()
.post("https://reqres.in/api/users")
// THEN
.then()
.statusCode(201)
.body("name", equalTo("RestAPITest"))
.body("job", equalTo("Testing"));
}
}
Step 5 – Create testng.xml for the project
<?xml version = "1.0"encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name = "Suite1">
<test name = "TestNG Test Demo">
<classes>
<class name = "org.example.RestAPITests"/>
</classes>
</test>
</suite>
Step 6 – Run the Test and Generate Allure Report
To run the tests, use the below command
mvn clean test
In the below image, we can see that all three tests are passed.
This will create the allure-results folder with all the test reports. These files will be used to generate the Allure Report.
To create an Allure Report, use the below command
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes, can be found.
View test history
Each time you run the report from the command line with the mvn clean test command, a new result JSON file will get added to the allure-results folder. Allure can use those files to include a historical view of your tests. Let’s give that a try.
To get started, run mvn clean test a few times and watch how the number of files in the allure-reports folder grows.
Now go back to view your report. Select Suites from the left nav, select one of your tests and click Retries in the right pane. You should see the history of test runs for that test:
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, and Story tags.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
The previous tutorial explained the generation of Allure Report with Cucumber5, Selenium and JUnit4 in a Maven project. In this tutorial, I will explain the steps to create an Allure Report with Cucumber7, Selenium, and JUnit5 in a Maven project.
Step 2 – Create Step Definition, feature file, and Test Runner Classes
Create Locator and Action classes and Step Definition corresponding to the feature file and Test Runner Class
There is another tutorial that explains the project structure as well as the feature file and corresponding Step Definitions, please refer to this tutorial – Integration of Cucumber7 with Selenium and JUnit5.
Step 3 – Execute the Tests
Use the below command to run the tests
mvn clean test
The output of the above program is
Step 4 – Generate the Allure Report
Once the test execution is finished, a folder named allure-results will be generated in thetarget folder.
To generate the Allure Report, first, go to the target folder.
cd target
Now, use the below command to generate the Allure Report
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Statistics – overall report statistics.
Launches – if this report represents several test launches, statistics per launch will be shown here.
Behaviours – information on results aggregated according to stories and features.
Executors – information on test executors that were used to run the tests.
History Trend – if tests accumulate some historical data, its trend will be calculated and shown on the graph.
Environment – information on the test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, Story, Test Severity, Test Description, Test Steps, and so on.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
Environment variables JAVA_HOME, ALLURE_HOME and GRADLE_HOME correctly configured
In this tutorial, I’ll create a BDD Framework for creating an Allure Report for Selenium WebDriver with JUnit5. This framework consists of:-
Dependency List
Java 11
JUnit– 5.8.2
Gradle – 7.3.3
Selenium – 4.3.0
Allure Report – 2.19
AspectJ Weaver – 1.9.7
Project Structure
Implementation Steps
There is another tutorial that explains the steps to create a Gradle Project with Selenium and JUnit5. Please refer to this tutorial –Gradle – Integration of Selenium and JUnit5.
Step 1 – Add dependencies to build.gradle
Add Selenium, JUnit5, and Allure-JUnit5 dependencies in build.gradle
/*
* This file was generated by the Gradle 'init' task.
*
*/
plugins {
// Apply the application plugin to add support for building a CLI application in Java.
id 'application'
id 'io.qameta.allure' version '2.11.0'
}
repositories {
// Use Maven Central for resolving dependencies.
mavenCentral()
}
java {
sourceCompatibility = 11
targetCompatibility = 11
}
dependencies {
// Use JUnit Jupiter for testing.
testImplementation 'org.junit.jupiter:junit-jupiter:5.8.2'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.8.2'
// Allure
implementation 'io.qameta.allure:allure-junit5:2.19.0'
runtimeOnly 'org.aspectj:aspectjweaver:1.9.7'
// This dependency is used by the application.
implementation 'com.google.guava:guava:30.1.1-jre'
implementation 'org.seleniumhq.selenium:selenium-java:4.4.0'
implementation 'io.github.bonigarcia:webdrivermanager:5.3.0'
}
application {
// Define the main class for the application.
mainClass = 'com.example.App'
}
tasks.named('test') {
// Use JUnit Platform for unit tests.
useJUnitPlatform() {
}
}
Step 2 – Create Pages and Test Code for the pages
As mentioned above, there is another tutorial that explains the project structure as well as the feature file and corresponding Step Definitions, please refer to this tutorial – Gradle Project with Selenium and JUnit5.
Below is the sample test code. I have added features of the allure report like @Severity, @Description.
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Disabled;
import io.qameta.allure.Description;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;
public class LoginPageTests extends BaseTests{
@Severity(SeverityLevel.NORMAL)
@ParameterizedTest
@CsvSource({
"admin$$,admin123",
"Admin,admin123!!",
"admin123,Admin",
"%%%%%,$$$$$$"})
@Description("Test Description : Login Test with invalid credentials")
public void invalidCredentials(String username, String password) {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login(username, password);
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getErrorMessage());
}
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Login Test with valid credentials")
public void validLogin() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("Admin", "admin123");
HomePage objHomePage = new HomePage(driver);
// Verify Home Page
assertEquals("Employee Information",objHomePage.getHomePageText());
}
@Severity(SeverityLevel.NORMAL)
@Test
@Description("Test Description : Login Test with missing username - Failed Test")
public void missingUsername() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("", "admin123");
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getMissingUsernameText());
}
@Severity(SeverityLevel.NORMAL)
@Test @Disabled
@Description("Test Description : Login Test with missing password")
public void missingPassword() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("admin", "");
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getMissingPasswordText());
}
}
Below is the BaseTest class, where I have shown the use of @Step of Allure Report.
import java.time.Duration;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import io.github.bonigarcia.wdm.WebDriverManager;
import io.qameta.allure.Step;
public class BaseTests {
public WebDriver driver;
public final static int TIMEOUT = 10;
@BeforeEach
@Step("Start the application")
public void setup() {
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
driver.manage().window().maximize();
driver.get("https://opensource-demo.orangehrmlive.com/");
driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(TIMEOUT));
}
@AfterEach
@Step("Stop the application")
public void tearDown() {
driver.quit();
}
}
Step 3 – Execute the Tests
Go to the app project and run the tests, using the below command
gradle clean test
The output of the test execution is
Step 4 – Generate the Allure Report
Once the test execution is finished, a folder named allure-results will be generated in thebuild folder.
Note:- Make sure that you move to the folder app, because the build folder is present in the app folder.
To generate an Allure Report, use the below command.
allure serve build/allure-results
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Statistics – overall report statistics.
Launches – if this report represents several test launches, statistics per launch will be shown here.
Behaviors – information on results aggregated according to stories and features.
Executors – information on test executors that were used to run the tests.
History Trend – if tests accumulated some historical data, its trend will be calculated and shown on the graph.
Environment – information on the test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes, can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, Story, Test Severity, Test Description, Test Steps, and so on.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
The previous tutorial explained the generation of Allure Report with Cucumber5, Selenium and JUnit4in a Maven project. In this tutorial, I will explain the steps to create an Allure Report with Cucumber, Selenium, and JUnit4 in a Gradle project.
Prerequisite:
Java 8 or above installed
Eclipse or IntelliJ IDE installed
Gradle Installed
Environment variables JAVA_HOME, ALLURE_HOME and GRADLE_HOME are correctly configured
In this tutorial, I’ll create a BDD Framework for the testing of web applications using Cucumber7, and Selenium 4 with JUnit4. This framework consists of:-
Cucumber Java- 7.6.0
Cucumber JUnit4 – 7.6.0
Java 11
JUnit4 – 4.13.2
Gradle – 7.3.3
Selenium – 4.3.0
Allure Cucumber – 2.19.0
AspectJ Weaver – 1.9.7
Project Structure
Implementation Steps
Add Cucumber, Selenium, TestNG, and Allure-JUnit dependencies in build.gradle
Create Locator and Action classes and Step Definition corresponding to the feature file and Test Runner Class
Step 1 – Add Cucumber, Selenium, TestNG, and Allure-JUnit4 dependencies in build.gradle
/*
* This file was generated by the Gradle 'init' task.
*
*/
plugins {
// Apply the application plugin to add support for building a CLI application in Java.
id 'application'
id 'io.qameta.allure' version '2.11.0'
}
repositories {
// Use Maven Central for resolving dependencies.
mavenCentral()
}
java {
sourceCompatibility = 11
targetCompatibility = 11
}
dependencies {
testImplementation 'io.cucumber:cucumber-java:7.6.0'
testImplementation 'io.cucumber:cucumber-junit:7.6.0'
// Use JUnit test framework.
testImplementation 'junit:junit:4.13.2'
// Allure
implementation 'io.qameta.allure:allure-cucumber7-jvm:2.19.0'
runtimeOnly 'org.aspectj:aspectjweaver:1.9.7'
// This dependency is used by the application.
implementation 'com.google.guava:guava:30.1.1-jre'
implementation 'org.seleniumhq.selenium:selenium-java:4.4.0'
implementation 'io.github.bonigarcia:webdrivermanager:5.3.0'
}
application {
// Define the main class for the application.
mainClass = 'com.example.App'
}
configurations {
cucumberRuntime {
extendsFrom testImplementation
}
}
task cucumber() {
dependsOn assemble, testClasses
doLast {
javaexec {
systemProperty("allure.results.directory", "build/allure-results")
main = "io.cucumber.core.cli.Main"
classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
args = ['--plugin', 'pretty',
'--glue', 'com.example.definitions', 'src/test/resources']
}
}
}
Step 2 – Create Locator and Action classes and Step Definition corresponding to the feature file and Test Runner Class
As mentioned above, there is another tutorial that explains the project structure as well as the feature file and corresponding Step Definitions, please refer to this tutorial – Gradle Project with Cucumber, Selenium, and JUnit4.
Step 3 – Execute the Tests
Go to the app project and run the tests, using the below command
gradle cucumber
The output of the test execution is
Step 4 – Generate the Allure Report
Once the test execution is finished, a folder named allure-results will be generated in thebuild folder.
Note:- Make sure that you move to the folder app because the build folder is present in the app folder.
allure serve build/allure-results
Allure Report Dashboard
The overview page hosts several default widgets representing the basic characteristics of your project and test environment.
Statistics – overall report statistics.
Launches – if this report represents several test launches, statistics per launch will be shown here.
Behaviors – information on results aggregated according to stories and features.
Executors – information on test executors that were used to run the tests.
History Trend – if tests accumulated some historical data, its trend will be calculated and shown on the graph.
Environment – information on the test environment.
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, Story, Test Severity, Test Description, Test Steps, and so on.
Packages in Allure Report
The packages tab represents a tree-like layout of test results, grouped by different packages.
BDD Features
The feature’s description appears in every scenario.
All scenario steps are automatically translated into allure steps.
Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!
QA Automation Expert 