Allure Report

vibssingh

Integrate Allure Report with Playwright with Java, Cucumber, and JUnit5

Last Updated On

HOME

In this tutorial, I will explain how to Integrate Allure Report 3 with Playwright, Java Cucumber,  and JUnit5.

Before starting, make sure to install Allure on your machine. Refer to this tutorial to install allure – What is Allure Report?.

Why Combine Playwright, Cucumber, JUnit5 and Allure? 

  • Playwright provides modern, robust automation for testing web apps across Chromium, Firefox, and WebKit. 
  • Cucumber enables BDD, allowing tests to be written in a natural language format using Gherkin syntax. 
  • Allure Report generates easy-to-understand and aesthetically pleasing test reports, enhancing the feedback loop. 
  • JUnit 5  provides powerful features like parameterized tests, extensions, and a modular architecture for organized and manageable test suites, parallel execution

Together, they empower teams to write readable tests, run them across multiple browsers, and analyze results visually. 

Project Structure

Table of Contents

  1. Why Combine Playwright, Cucumber, JUnit5 and Allure? 
  2. Project Structure
  3. System requirements
  4. Dependency List
  5. High-Level Execution Flow
  6. Implementation Steps
    1. Set Up the Environment
      1. Update the Properties section in Maven pom.xml
      2. Add dependencies to pom.xml
      3. Update the Build Section of pom.xml in the Allure Report Project
    2. Create a Feature file
    3. Create the utility package in src/test/java
    4. Create the hooks package in src/test/java
    5. Create the pages package in src/test/java
    6. Create the step package in src/test/java
    7. Create the step definition class in src/test/java
    8. Create a JUnit Cucumber Runner class
    9. Specify Allure Plugin in junit-platform.properties
    10. Specifying Allure Results location
    11. Run the Test and Generate Allure Report
    12. How to Generate a Report
    13. How to View a Report
      1. Title
      2. Tags
      3. Owner
      4. Severity
      5. Support for Scenario Outlines
      6. Attachment
      7. Graphs in Allure Report
      8. Timeline in Allure Report
      9. Generate Reports with the Allure Awesome Plugin
    14. Summary

System requirements

  1. Java 17 installed
  2. Maven installed
  3. Eclipse or IntelliJ installed
  4. Allure installed
  5. Browsers on which tests need to be run, like Chrome, Firefox, etc.

Dependency List

  1. Playwright – 1.57.0
  2. Java 17
  3. Cucumber – 7.33.0
  4. Maven – 3.9.6
  5. Allure Report – 2.32.0
  6. Allure Maven – 2.12.0
  7. Aspectj – 1.9.25.1
  8. Maven Compiler Plugin – 3.13.1
  9. Maven Surefire Plugin – 3.5.4

High-Level Execution Flow

Implementation Steps

1. Set Up the Environment

Step 1 – Update the Properties section in Maven pom.xml
<properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>17</java.version>
        <cucumber.version>7.33.0</cucumber.version>
        <playwright.version>1.57.0</playwright.version>
        <lombok.version>1.18.30</lombok.version>
        <junit.platform.version>6.1.0-M1</junit.platform.version>
        <junit.jupiter.api.version>6.1.0-M1</junit.jupiter.api.version>
        <junit.jupiter.params.version>6.1.0-M1</junit.jupiter.params.version>
        <junit.platform.launcher.version>6.1.0-M1</junit.platform.launcher.version>
        <maven.compiler.plugin.version>3.14.1</maven.compiler.plugin.version>
        <maven.compiler.source.version>17</maven.compiler.source.version>
        <maven.compiler.target.version>17</maven.compiler.target.version>
        <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
        <allure.version>2.32.0</allure.version>
        <aspectj.version>1.9.25.1</aspectj.version>
</properties>

Step 2 – Add dependencies to pom.xml

Add Cucumber, JUnit Platform, Allure-Cucumber, and Allure-JUnit-Platform dependencies to pom.xml (Maven Project).

 <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-bom</artifactId>
                <version>${allure.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>${playwright.version}</version>
        </dependency>

        <!-- JUnit 5 -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.jupiter.api.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.jupiter.params.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>${junit.platform.launcher.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Cucumber with Java -->
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-java</artifactId>
            <version>${cucumber.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Cucumber with Junit5 -->
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit-platform-engine</artifactId>
            <version>${cucumber.version}</version>
        </dependency>

        <!-- JUnit Platform -->
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-suite</artifactId>
            <version>${junit.platform.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-cucumber7-jvm</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-junit-platform</artifactId>
            <scope>test</scope>
        </dependency>

    </dependencies>

Step 3 – Update the Build Section of pom.xml in the Allure Report Project
<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source.version}</source>
                    <target>${maven.compiler.target.version}</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

The entire pom.xml is

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>Cucumber_Playwright_JUnit5</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Cucumber_Playwright_JUnit5</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>17</java.version>
        <cucumber.version>7.33.0</cucumber.version>
        <playwright.version>1.57.0</playwright.version>
        <lombok.version>1.18.30</lombok.version>
        <junit.platform.version>6.1.0-M1</junit.platform.version>
        <junit.jupiter.api.version>6.1.0-M1</junit.jupiter.api.version>
        <junit.jupiter.params.version>6.1.0-M1</junit.jupiter.params.version>
        <junit.platform.launcher.version>6.1.0-M1</junit.platform.launcher.version>
        <maven.compiler.plugin.version>3.14.1</maven.compiler.plugin.version>
        <maven.compiler.source.version>17</maven.compiler.source.version>
        <maven.compiler.target.version>17</maven.compiler.target.version>
        <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
        <allure.version>2.32.0</allure.version>
        <aspectj.version>1.9.25.1</aspectj.version>
    </properties>

    <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-bom</artifactId>
                <version>${allure.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>${playwright.version}</version>
        </dependency>

        <!-- JUnit 5 -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.jupiter.api.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.jupiter.params.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-launcher</artifactId>
            <version>${junit.platform.launcher.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Cucumber with Java -->
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-java</artifactId>
            <version>${cucumber.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Cucumber with Junit5 -->
        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit-platform-engine</artifactId>
            <version>${cucumber.version}</version>
        </dependency>

        <!-- JUnit Platform -->
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-suite</artifactId>
            <version>${junit.platform.version}</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-cucumber7-jvm</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-junit-platform</artifactId>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source.version}</source>
                    <target>${maven.compiler.target.version}</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>

2. Create a Feature file

Create a folder – features within src/test/resources to create test scenarios in the Feature file.

Feature file should be saved as an extension of .feature. Add the test scenarios in this feature file. I have added sample test scenarios. In this feature file. The test scenarios are written in Gherkins language.

@allure.label.parent_suite:WebInterface
@allure.label.sub_suite:Login Page
@allure.label.owner:Vibha

Feature: Login to HRM Application

  Background:
    Given User is on HRMLogin page "https://opensource-demo.orangehrmlive.com/"

  @ValidCredentials @Critical
  Scenario: Login with valid credentials

    When User enters username as "Admin" and password as "admin123"
    Then User should be able to login successfully and new page opens with heading "Dashboard"

  @InvalidCredentials @High
  Scenario Outline: Login with invalid credentials

    When User enters username as "<username>" and password as "<password>"
    Then User should be able to see error message "<errorMessage>"

    Examples:
      | username   | password  | errorMessage           |
      | Admin      | admin12$$ | Invalid credentials    |
      | admin$$    | admin123  | Invalid credentials    |
      | abc123     | xyz$$     | Invalid credentials    |

  @MissingUsername @Medium
  Scenario: Login with blank username

    When User enters username as " " and password as "admin123"
    Then User should be able to see a message "Required1" below Username

Please refer to this tutorial – Integration of Playwright with Java, Cucumber and JUnit5 to get the details about Step 3, 4, 5, 6,7

3. Create the utility package in src/test/java

4. Create the hooks package in src/test/java

5. Create the pages package in src/test/java

6. Create the step package in src/test/java

7. Create the step definition class in src/test/java

8. Create a JUnit Cucumber Runner class

This JUnit 5 runner configures and launches Cucumber scenarios by selecting feature files, defining glue code packages, and executing them via the Cucumber engine on the JUnit Platform.

Add Allure Report plugin in the Test Runner to generate the Allure Report.

import static io.cucumber.junit.platform.engine.Constants.GLUE_PROPERTY_NAME;

import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectClasspathResource;
import org.junit.platform.suite.api.Suite;

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
@ConfigurationParameter(
        key = GLUE_PROPERTY_NAME,
        value = "com.example"
)


public class RunnerTests {
}

1. @Suite – Marks this class as a JUnit 5 test suite. Acts as the entry point for test execution

2. @IncludeEngines(“cucumber”) – Tells JUnit Platform to use the Cucumber engine

3. @SelectClasspathResource(“features”) – Specifies the location of feature file

4. @ConfigurationParameter – Tells Cucumber where to find step definitions and hooks

9. Specify Allure Plugin in junit-platform.properties

Specify the allure plugin in junit-platform.properties which should be placed in src/test/resources:

cucumber.publish.enabled=true
cucumber.plugin=io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm

10. Specifying Allure Results location

Allure, by default, saves test results in the project’s root directory. However, it is recommended to store your test results in the build output directory.

To configure this, create an allure.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:

#allure.properties
allure.results.directory=build/allure-results

11. Run the Test and Generate Allure Report

To run the tests, use the below command

mvn clean test

In the image below, we can see that one test failed and four passed out of five tests.

This will create the allure-results folder with all the test reports within build folder. These files will be used to generate Allure Report.

12. How to Generate a Report

allure generate processes the test results and saves an HTML report into the allure-report directory. To view the report, use the allure open command.

allure serve creates the same report as allure generate, then automatically opens the main page of the report in a web browser.

Use the command below to generate the Allure Report

 allure serve build/allure-results

This will generate the beautiful Allure Test Report as shown below.

13. How to View a Report

Test reports generated with Allure Report are basically small HTML websites intended to be viewed in a web browser. 

Title

A human-readable title of the test. If not provided, the function name is used instead.

Tags

Any number of short terms the test is related to. Usually it’s a good idea to list relevant features that are being tested. Tags can then be used for filtering.

Owner

The team member who is responsible for the test’s stability. For example, this can be the test’s author, the leading developer of the feature being tested, etc.

Severity

A value indicating how important the test is. This may give the future reader an idea of how to prioritize the investigations of different test failures.

Allowed values are: “trivial”, “minor”, “normal”, “critical”, and “blocker”.

Support for Scenario Outlines

Allure Cucumber-JVM provides complete support for Scenario Outlines, a feature of Cucumber-JVM that allows for parametrized tests. No special setup is needed to take advantage of this capability within Allure.

Attachment

In Allure reports, you have the ability to attach various types of files, which can greatly enhance the comprehensibility of the report. A common practice is to attach screenshots that capture the state of the user interface at specific moments during test execution.

Allure Cucumber-JVM offers multiple methods for creating attachments, whether from pre-existing files or from content generated on-the-fly.

Graphs in Allure Report

Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.

Timeline in Allure Report

The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.

Generate Reports with the Allure Awesome Plugin

Allure 3 Report also features an advanced report generator plugin – Allure Awesome. It supports additional configuration options, such as generating the report as a single HTML file, setting the theme, custom branding and language of the generated report, and taking known issues into account.

To manually generate a customized Allure Awesome report, use the awesome command:

 allure awesome build/allure-results

This will create the allure-report folder with all the test files and index.html report.

Open “index.html“, as this is an HTML report, and open it with the browser. The below image shows index.html.

Right click on index.html->Open In ->Browser ->Edge( any browser).

Summary:

1.Add required dependencies – Include Playwright, JUnit 5, and Allure JUnit5 in pom.xml, and configure the Allure Maven plugin.

2. Write Cucumber Scenarios in Feature File – Define the feature files using Gherkin syntax (.feature) and place them in the appropriate directory (src/test/resources/features).

3. Implement Step Definitions – Create step definitions in Java to connect the Gherkin scenarios to Playwright actions. Ensure these are correctly annotated with Cucumber’s annotations.

4. Create Test Runner – Setup a TestNG runner that combines Cucumber options and ensures Allure configurations are included.

5. Configure Allure Reporting – Utilize Allure annotations like @AllureFeature and @AllureStory within your tests to categorize and logically group test cases. Create an allure.properties file in src/test/resources with necessary configurations like resultsDirectory and linkPattern.

6. Execute Tests and Generate Reports – Execute mvn clean test, then run allure serve allure-results to build and open the HTML report.

Congratulations!! We have integrated an allure report with Playwright, Java Cucumber, and TestNG. I hope this tutorial is useful to you.

Additional Tutorials on Allure Reports

Integration of Allure Report with Selenium and JUnit4
Integration of Allure Report with Selenium and TestNG
Gradle – Allure Report for Selenium and JUnit4
Gradle – Allure Report for Cucumber, Selenium and TestNG
Integration of Allure Report with Rest Assured and JUnit4

vibssingh

Integrate Allure Report with Playwright with Java, Cucumber, and TestNG

Last Updated On

HOME

In this tutorial, I will explain how to Integrate Allure Report 3 with Playwright, Java Cucumber,  and TestNG.

Before starting, make sure to install Allure on your machine. Refer to this tutorial to install allure – What is Allure Report?.

Why Combine Playwright, Cucumber, TestNG and Allure? 

  • Playwright provides modern, robust automation for testing web apps across Chromium, Firefox, and WebKit. 
  • Cucumber enables BDD, allowing tests to be written in a natural language format using Gherkin syntax. 
  • Allure Report generates easy-to-understand and aesthetically pleasing test reports, enhancing the feedback loop. 
  • TestNG provides a robust and flexible test framework with features such as dependency testing, grouping of test methods, parallel execution

Together, they empower teams to write readable tests, run them across multiple browsers, and analyze results visually. 

Project Structure

Table of Contents

  1. Why Combine Playwright, Cucumber, TestNG and Allure? 
  2. Project Structure
  3. System requirements
  4. Dependency List
  5. High-Level Execution Flow
  6. Implementation Steps
    1. Set Up the Environment
      1. Update the Properties section in Maven pom.xml
      2. Add dependencies to pom.xml
      3. Update the Build Section of pom.xml in the Allure Report Project
    2. Create a Feature file
    3. Create the utility package in src/test/java
    4. Create the hooks package in src/test/java
    5. Create the pages package in src/test/java
    6. Create the step package in src/test/java
    7. Create the step definition class in src/test/java
    8. Create a TestNG Cucumber Runner class
    9. Create testng.xml for the project
    10. Specifying Allure Results location
    11. Run the Test and Generate Allure Report
    12. How to Generate a Report
    13. How to View a Report
      1. Title
      2. Tags
      3. Owner
      4. Severity
      5. Support for Scenario Outlines
      6. Attachment
      7. Graphs in Allure Report
      8. Timeline in Allure Report
      9. Generate Reports with the Allure Awesome Plugin
    14. Summary

System requirements

  1. Java 17 installed
  2. Maven installed
  3. Eclipse or IntelliJ installed
  4. Allure installed
  5. Browsers on which tests need to be run, like Chrome, Firefox, etc.

Dependency List

  1. Playwright – 1.57.0
  2. Java 17
  3. Cucumber – 7.33.0
  4. Maven – 3.9.6
  5. TestNG – 7.11.0
  6. Allure Report – 2.32.0
  7. Aspectj – 1.9.25.1
  8. Maven Compiler Plugin – 3.13.0
  9. Maven Surefire Plugin – 3.5.4

High-Level Execution Flow

Implementation Steps

1. Set Up the Environment

Step 1 – Update the Properties section in Maven pom.xml
 <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <playwright.version>1.57.0</playwright.version>
        <testng.version>7.11.0</testng.version>
        <allure.version>2.32.0</allure.version>
        <aspectj.version>1.9.25</aspectj.version>
        <maven.compiler.plugin.version>3.13.0</maven.compiler.plugin.version>
        <maven.compiler.source.version>17</maven.compiler.source.version>
        <maven.compiler.target.version>17</maven.compiler.target.version>
        <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
</properties>

Step 2 – Add dependencies to pom.xml

Add Cucumber, TestNG, Allure-Cucumber, and Allure-TestNG dependencies to pom.xml (Maven Project).

  <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-bom</artifactId>
        <version>${allure.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>

    <!-- Cucumber with Java -->
    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-java</artifactId>
      <version>${cucumber.version}</version>
      <scope>test</scope>
    </dependency>

    <!-- Cucumber with TestNG -->
    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-testng</artifactId>
      <version>${cucumber.version}</version>
      <scope>test</scope>
    </dependency>

    <!-- TestNG -->
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>${testng.version}</version>
      <scope>test</scope>
    </dependency>

    <!-- Playwright -->
    <dependency>
      <groupId>com.microsoft.playwright</groupId>
      <artifactId>playwright</artifactId>
      <version>${playwright.version}</version>
    </dependency>

    <!-- Add necessary Allure dependencies -->
      <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-cucumber7-jvm</artifactId>
        <scope>test</scope>
      </dependency>

      <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-testng</artifactId>
        <scope>test</scope>
      </dependency>
    
  </dependencies>

Step 3 – Update the Build Section of pom.xml in the Allure Report Project
<build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven.compiler.plugin.version}</version>
        <configuration>
          <source>${maven.compiler.source.version}</source>
          <target>${maven.compiler.target.version}</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${maven.surefire.plugin.version}</version>
        <configuration>
          <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
          </suiteXmlFiles>
          <argLine>
            -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
          </argLine>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>${aspectj.version}</version>
            <scope>runtime</scope>
          </dependency>
        </dependencies>
      </plugin>
      <plugin>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-maven</artifactId>
        <version>${allure.maven}</version>
        <configuration>
          <reportVersion>${allure.maven}</reportVersion>
        </configuration>
      </plugin>
    </plugins>
  </build>

The entire pom.xml is

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>Playwright_PageObjectModel</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Playwright_PageObjectModel</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <playwright.version>1.57.0</playwright.version>
        <testng.version>7.11.0</testng.version>
        <allure.version>2.32.0</allure.version>
        <aspectj.version>1.9.25</aspectj.version>
        <maven.compiler.plugin.version>3.13.0</maven.compiler.plugin.version>
        <maven.compiler.source.version>17</maven.compiler.source.version>
        <maven.compiler.target.version>17</maven.compiler.target.version>
        <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
    </properties>

    <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-bom</artifactId>
                <version>${allure.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>

        <!-- Playwright -->
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>${playwright.version}</version>
        </dependency>

        <!-- TestNG -->
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>${testng.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Allure-TestNG -->
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-testng</artifactId>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source.version}</source>
                    <target>${maven.compiler.target.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>

2. Create a Feature file

Create a folder – features within src/test/resources to create test scenarios in the Feature file.

Feature file should be saved as an extension of .feature. Add the test scenarios in this feature file. I have added sample test scenarios. In this feature file. The test scenarios are written in Gherkins language.

@allure.label.parent_suite:WebInterface
@allure.label.sub_suite:Login Page
@allure.label.owner:Vibha

Feature: Login to HRM Application

  Background:
    Given User is on HRMLogin page "https://opensource-demo.orangehrmlive.com/"

  @ValidCredentials @Critical
  Scenario: Login with valid credentials

    When User enters username as "Admin" and password as "admin123"
    Then User should be able to login successfully and new page opens with heading "Dashboard"

  @InvalidCredentials @High
  Scenario Outline: Login with invalid credentials

    When User enters username as "<username>" and password as "<password>"
    Then User should be able to see error message "<errorMessage>"

    Examples:
      | username   | password  | errorMessage           |
      | Admin      | admin12$$ | Invalid credentials    |
      | admin$$    | admin123  | Invalid credentials    |
      | abc123     | xyz$$     | Invalid credentials    |

  @MissingUsername @Medium
  Scenario: Login with blank username

    When User enters username as " " and password as "admin123"
    Then User should be able to see a message "Required1" below Username

Please refer to this tutorial – Integration of Playwright with Java, Cucumber and TestNG to get the details about Step 3, 4, 5, 6,7

3. Create the utility package in src/test/java

4. Create the hooks package in src/test/java

5. Create the pages package in src/test/java

6. Create the step package in src/test/java

7. Create the step definition class in src/test/java

8. Create a TestNG Cucumber Runner class

We need to create a class called Runner class to run the tests. This class will use the TestNG annotation @Test, which tells TestNG what is the test runner class.

Add Allure Report plugin in the Test Runner to generate the Allure Report.

package com.example.runner;

import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;

@CucumberOptions(features = "src/test/resources/features/LoginPage.feature",
        glue = {"com.example.hooks","com.example.definitions"},
        plugin = { "io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm",
                "pretty",
        }
)

public class RunnerTests extends AbstractTestNGCucumberTests {

}

Note:- @Test annotation marks this class as part of the test. So, if we will remove this annotation, the Allure Report executes CucumberRunnerTests as a separate test suite, so there will be duplicate results.

9. Create testng.xml for the project

<?xml version = "1.0"encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name = "Suite1">
    <test name = "Test Demo">
        <classes>
            <class name = "com.example.runner.CucumberRunnerTests"/>
        </classes>
    </test>
</suite>

10. Specifying Allure Results location

Allure, by default, saves test results in the project’s root directory. Still, it is recommended to store your test results in the build output directory.

To configure this, create an allure.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:

#allure.properties
allure.results.directory=build/allure-results

11. Run the Test and Generate Allure Report

To run the tests, use the below command

mvn clean test

In the image below, we can see that one test failed and four passed out of five tests.

This will create the allure-results folder with all the test reports within build folder. These files will be used to generate Allure Report.

12. How to Generate a Report

allure generate processes the test results and saves an HTML report into the allure-report directory. To view the report, use the allure open command.

allure serve creates the same report as allure generate, then automatically opens the main page of the report in a web browser.

Use the command below to generate the Allure Report

 allure serve build/allure-results

This will generate the beautiful Allure Test Report as shown below.

13. How to View a Report

Test reports generated with Allure Report are basically small HTML websites intended to be viewed in a web browser. 

Title

A human-readable title of the test. If not provided, the function name is used instead.

Tags

Any number of short terms the test is related to. Usually it’s a good idea to list relevant features that are being tested. Tags can then be used for filtering.

Owner

The team member who is responsible for the test’s stability. For example, this can be the test’s author, the leading developer of the feature being tested, etc.

Severity

A value indicating how important the test is. This give the future reader an idea of how to prioritize the investigations of different test failures.

Allowed values are: “trivial”, “minor”, “normal”, “critical”, and “blocker”.

Support for Scenario Outlines

Allure Cucumber-JVM provides complete support for Scenario Outlines, a feature of Cucumber-JVM that allows for parametrized tests. No special setup is needed to take advantage of this capability within Allure.

Attachment

In Allure reports, you have the ability to attach various types of files, which can greatly enhance the comprehensibility of the report. A common practice is to attach screenshots that capture the state of the user interface at specific moments during test execution.

Allure Cucumber-JVM offers multiple methods for creating attachments, whether from pre-existing files or from content generated on-the-fly.

Graphs in Allure Report

Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.

Timeline in Allure Report

The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.

Generate Reports with the Allure Awesome Plugin

Allure 3 Report also features an advanced report generator plugin – Allure Awesome. It supports additional configuration options, such as generating the report as a single HTML file, setting the theme, custom branding and language of the generated report, and taking known issues into account.

To manually generate a customized Allure Awesome report, use the awesome command:

 allure awesome build/allure-results

This will create the allure-report folder with all the test files and index.html report.

Open “index.html“, as this is an HTML report, and open it with the browser. The below image shows index.html.

Right click on index.html->Open In ->Browser ->Edge( any browser).

Summary:

1.Add required dependencies – Include Playwright, TestNG, and Allure TestNG in pom.xml, and configure the Allure Maven plugin.

2. Write Cucumber Scenarios in Feature File – Define the feature files using Gherkin syntax (.feature) and place them in the appropriate directory (src/test/resources/features).

3. Implement Step Definitions – Create step definitions in Java to connect the Gherkin scenarios to Playwright actions. Ensure these are correctly annotated with Cucumber’s annotations.

4. Create Test Runner – Setup a TestNG runner that combines Cucumber options and ensures Allure configurations are included.

5. Configure Allure Reporting – Utilize Allure annotations like @AllureFeature and @AllureStory within your tests to categorize and logically group test cases. Create an allure.properties file in src/test/resources with necessary configurations like resultsDirectory and linkPattern.

6. Execute Tests and Generate Reports – Execute mvn clean test, then run allure serve allure-results to build and open the HTML report.

Congratulations!! We have integrated an allure report with Playwright, Java Cucumber, and TestNG. I hope this tutorial is useful to you.

Additional Tutorials on Allure Reports

Integration of Allure Report with Selenium and JUnit4
Integration of Allure Report with Selenium and TestNG
Gradle – Allure Report for Selenium and JUnit4
Gradle – Allure Report for Cucumber, Selenium and TestNG
Integration of Allure Report with Rest Assured and JUnit4

vibssingh

Integrate Allure Report with Playwright with Java and JUnit5

Last Updated On

HOME

In this tutorial, I will explain how to Integrate Allure Report 3 with Playwright, Java and JUnit5.

What is an Allure Report?

Allure Framework is a flexible, lightweight, multi-language test report tool. It shows a very concise representation of what has been tested in a neat web report form. It also allows everyone participating in the development process to extract maximum useful information from the everyday execution of tests.

Refer to this tutorial to install allure – What is Allure Report?.

In this tutorial, we are using Allure Report 3.

Table of Contents

  1. What is an Allure Report?
  2. System requirements
  3. Dependency List
  4. High-Level Execution Flow
  5. Implementation Steps
    1. 1. Set Up the Environment
      1. Update the Properties section in Maven pom.xml
      2. Add dependencies to pom.xml
      3. Update the Build Section of pom.xml in the Allure Report Project
    2. Project Structure for Maintainability
    3. Creating Page Object Classes in src/test/java
    4. Create the utility package in src/test/java
    5. Write the Test Scripts
    6. Specifying Allure Results location
    7. Run the Tests and Generate Allure Report
    8. How to Generate an Allure Report
    9. How to View a Report
      1. Title
      2. Owner
      3. Severity
      4. Steps
      5. Links
      6. Graphs in Allure Report
      7. Timeline in Allure Report
      8. Generate Reports with the Allure Awesome Plugin

System requirements

  1. Java 17 installed
  2. Maven installed
  3. Eclipse or IntelliJ installed
  4. Allure installed
  5. Browsers on which tests need to be run, like Chrome, Firefox, etc.

Dependency List

  1. Playwright – 1.57.0
  2. Java 17
  3. Maven – 3.9.6
  4. Allure Report – 2.32.0
  5. Aspectj – 1.9.25
  6. Maven Compiler Plugin – 3.13.0
  7. Maven Surefire Plugin – 3.5.4

High-Level Execution Flow

Implementation Steps

1. Set Up the Environment

Step 1 – Update the Properties section in Maven pom.xml
<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <playwright.version>1.57.0</playwright.version>
    <junit.jupiter.engine.version>6.1.0-M1</junit.jupiter.engine.version>
    <junit.jupiter.api.version>6.1.0-M1</junit.jupiter.api.version>
    <junit.jupiter.params.version>6.1.0-M1</junit.jupiter.params.version>
    <junit.platform.launcher.version>6.1.0-M1</junit.platform.launcher.version>
    <maven.compiler.plugin.version>3.14.1</maven.compiler.plugin.version>
    <maven.compiler.source.version>17</maven.compiler.source.version>
    <maven.compiler.target.version>17</maven.compiler.target.version>
    <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    <maven.site.plugin.version>4.0.0-M16</maven.site.plugin.version>
    <allure.version>2.32.0</allure.version>
    <aspectj.version>1.9.25.1</aspectj.version>
  </properties>

Step 2 – Add dependencies to pom.xml

Add Playwright, JUnit Jupiter Engine, JUnit Jupiter API, JUnit Param, JUnit Platform and Allure-JUnit5 dependencies to pom.xml (Maven Project).

 <dependencyManagement>
      <dependencies>
        <dependency>
          <groupId>io.qameta.allure</groupId>
          <artifactId>allure-bom</artifactId>
          <version>${allure.version}</version>
          <type>pom</type>
          <scope>import</scope>
        </dependency>
      </dependencies>
    </dependencyManagement>

  <dependencies>

    <dependency>
      <groupId>com.microsoft.playwright</groupId>
      <artifactId>playwright</artifactId>
      <version>${playwright.version}</version>
    </dependency>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-engine</artifactId>
      <version>${junit.jupiter.engine.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-api</artifactId>
      <version>${junit.jupiter.api.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-params</artifactId>
      <version>${junit.jupiter.params.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.platform</groupId>
      <artifactId>junit-platform-launcher</artifactId>
      <version>${junit.platform.launcher.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-junit5</artifactId>
      <scope>test</scope>
    </dependency>

  </dependencies>

Step 3 – Update the Build Section of pom.xml in the Allure Report Project

Allure leverages AspectJ for the functionality of @Step and @Attachment annotations.

 <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven.compiler.plugin.version}</version>
        <configuration>
          <source>${maven.compiler.source.version}</source>
          <target>${maven.compiler.target.version}</target>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${maven.surefire.plugin.version}</version>
        <configuration>
          <argLine>
            -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
          </argLine>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>${aspectj.version}</version>
          </dependency>
        </dependencies>
      </plugin>

    </plugins>
  </build>

The entire pom.xml is

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.example</groupId>
  <artifactId>PageObjectModel_Playwright_JUnit5</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>PageObjectModel_Playwright_JUnit5</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <playwright.version>1.57.0</playwright.version>
    <junit.jupiter.engine.version>6.1.0-M1</junit.jupiter.engine.version>
    <junit.jupiter.api.version>6.1.0-M1</junit.jupiter.api.version>
    <junit.jupiter.params.version>6.1.0-M1</junit.jupiter.params.version>
    <junit.platform.launcher.version>6.1.0-M1</junit.platform.launcher.version>
    <maven.compiler.plugin.version>3.14.1</maven.compiler.plugin.version>
    <maven.compiler.source.version>17</maven.compiler.source.version>
    <maven.compiler.target.version>17</maven.compiler.target.version>
    <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    <maven.site.plugin.version>4.0.0-M16</maven.site.plugin.version>
    <allure.version>2.32.0</allure.version>
    <aspectj.version>1.9.25.1</aspectj.version>
  </properties>


    <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
    <dependencyManagement>
      <dependencies>
        <dependency>
          <groupId>io.qameta.allure</groupId>
          <artifactId>allure-bom</artifactId>
          <version>${allure.version}</version>
          <type>pom</type>
          <scope>import</scope>
        </dependency>
      </dependencies>
    </dependencyManagement>

  <dependencies>

    <dependency>
      <groupId>com.microsoft.playwright</groupId>
      <artifactId>playwright</artifactId>
      <version>${playwright.version}</version>
    </dependency>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-engine</artifactId>
      <version>${junit.jupiter.engine.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-api</artifactId>
      <version>${junit.jupiter.api.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-params</artifactId>
      <version>${junit.jupiter.params.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.platform</groupId>
      <artifactId>junit-platform-launcher</artifactId>
      <version>${junit.platform.launcher.version}</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-junit5</artifactId>
      <scope>test</scope>
    </dependency>

  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven.compiler.plugin.version}</version>
        <configuration>
          <source>${maven.compiler.source.version}</source>
          <target>${maven.compiler.target.version}</target>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${maven.surefire.plugin.version}</version>
        <configuration>
          <argLine>
            -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
          </argLine>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>${aspectj.version}</version>
          </dependency>
        </dependencies>
      </plugin>
    </plugins>
  </build>

</project>

2. Project Structure for Maintainability

Creating a well-organized project structure is crucial for maintaining a scalable and efficient automation framework.

3. Creating Page Object Classes in src/test/java

Please refer to – Setting Up Page Object Model with Playwright and JUnit5 to know more about the Page Object Classes and the respective code.

LoginPage

package com.example.pages;

import com.microsoft.playwright.Locator;
import com.microsoft.playwright.Page;
import io.qameta.allure.Step;

public class LoginPage {

    private final Page page;

    // Locators
    private final Locator usernameLocator;
    private final Locator passwordLocator;
    private final Locator submitLocator;
    private final Locator invalidCredentialsLocator;
    private final Locator missingUsernameErrorMessageLocator;

    public LoginPage(Page page) {
        this.page = page;
        this.usernameLocator = page.locator("input[name='username']");
        this.passwordLocator = page.locator("input[name='password']");
        this.submitLocator = page.locator("button[type='submit']");
        this.invalidCredentialsLocator = page.locator("//p[contains(@class, \"oxd-text oxd-text--p oxd-alert-content-text\")]");
        this.missingUsernameErrorMessageLocator = page.locator("//span[contains(@class, 'oxd-text oxd-text--span oxd-input-field-error-message oxd-input-group__message')]");
    }

    @Step("Enter credentials")
    public void login(String user, String pass){
        usernameLocator.fill(user);
        passwordLocator.fill(pass);
        submitLocator.click();
    }

    @Step("Get Error Message for invalid credentials")
    public String getErrorMessage () {
        return invalidCredentialsLocator.textContent();
    }

    @Step("Get Error Message for missing username")
    public String getMissingUsernameErrorMessage () {
        return missingUsernameErrorMessageLocator.textContent();
    }
}


DashboardPage

package com.example.pages;

import com.microsoft.playwright.Locator;
import com.microsoft.playwright.Page;
import io.qameta.allure.Step;

public class DashboardPage {

    private Page page;

    private final Locator headingLocator;

    public DashboardPage(Page page){
        this.page = page;
        this.headingLocator = page.locator("//h6[contains(@class, \"oxd-topbar-header-breadcrumb-module\")]");
        this.assignLeaveLocator = page.locator("//[contains(@text, \"Assign Leave\")]");
        this.leaveListLocator = page.getByTitle("Leave List");

    }

    @Step("Get Heading of Dashboard page")
    public String getDashboardPageHeading() {
        return headingLocator.textContent();
    }

}

4. Create the utility package in src/test/java

BaseClass

This class contains the common variables and methods used throughout the project, like setup and teardown methods.

package com.example.utils;

import com.microsoft.playwright.*;
import org.junit.jupiter.api.*;

// Subclasses will inherit PER_CLASS behavior.
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class BaseTests {

    // Shared between all tests in the class.
    static Playwright playwright;
    static Browser browser;

    // New instance for each test method.
    BrowserContext context;
    protected Page page;

    @BeforeAll
    public static void launchBrowser() {
        playwright = Playwright.create();
        browser = playwright.chromium().launch(new BrowserType.LaunchOptions().setHeadless(false));

    }

    @AfterAll
    public static void closeBrowser() {
        playwright.close();
    }


    @BeforeEach
    public void createContextAndPage() {
        context = browser.newContext();
        page = context.newPage();
        page.navigate("https://opensource-demo.orangehrmlive.com/web/index.php/auth/login");
    }

    @AfterEach
    public void closeContext() {
        context.close();
    }
}

5. Write the Test Scripts

Create a Test files under src/test/java. Use these page classes in your test scripts to perform end-to-end scenarios. This will keep your tests clean and focused on logic rather than details about the UI elements.

We will be automating the following test scenario using Playwright Java and run them in Chromium.

  1. Verify the invalid username generates error message “Invalid credentials” with different set of invalid test data.
  2. Verify that the user is able to login to the application successfully.

Creating a Login Page Test Class:

package com.example.tests;

import com.example.pages.DashboardPage;
import com.example.pages.LoginPage;
import com.example.utils.BaseTests;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvFileSource;
import io.qameta.allure.Description;
import io.qameta.allure.Issue;
import io.qameta.allure.Link;
import io.qameta.allure.Owner;
import io.qameta.allure.Severity;
import io.qameta.allure.TmsLink;
import org.junit.jupiter.api.DisplayName;

import static io.qameta.allure.SeverityLevel.CRITICAL;

public class LoginTests extends BaseTests {

    @ParameterizedTest
    @CsvFileSource(files = "src/test/resources/testData/credentials.csv", numLinesToSkip = 1)
    @DisplayName("Login Authentication")
    @Description("This test attempts to log into the website using a incorrect username and a password. Fails if any error happens.")
    @Severity(CRITICAL)
    @Owner("Vibha Singh")
    @Link(name = "Website", url = "https://opensource-demo.orangehrmlive.com/web/index.php/auth/login")
    @Issue("AUTH-123")
    @TmsLink("TMS-456")
    public void unsuccessfulLogin(String username, String password, String expectedErrorMessage) {

        LoginPage loginPage = new LoginPage(page);
        loginPage.login(username,password);
        String actualErrorMessage = loginPage.getErrorMessage();
        Assertions.assertEquals(expectedErrorMessage, actualErrorMessage, "Incorrect error message is displayed");

   }

    @Test
    @DisplayName("Successful Login Authentication")
    @Description("This test attempts to log into the website using a correct username and a password. Fails if any error happens.")
    @Severity(CRITICAL)
    @Owner("Vibha Singh")
    @Link(name = "Website", url = "https://opensource-demo.orangehrmlive.com/web/index.php/auth/login")
    @Issue("AUTH-124")
    @TmsLink("TMS-460")
    public void successfulLogin() {

        LoginPage loginPage = new LoginPage(page);
        loginPage.login("Admin","admin123");
        DashboardPage dashboardPage = new DashboardPage(page);
        String actualHeading = dashboardPage.getDashboardPageHeading();
        Assertions.assertEquals("Dashboard",actualHeading, "Unable to login to the application");

    }

}

Use the @Description() annotation to set a description statically or use the description() method to set it dynamically in runtime.

@Description("This test attempts to log into the website using a invalid login and a password that result in error")

6. Specifying Allure Results location

Allure, by default, saves test results in the project’s root directory. However, it is recommended to store your test results in the build output directory.

To configure this, create an junit-platform.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:

#allure.properties
allure.results.directory=target/allure-results

7. Run the Tests and Generate Allure Report

To run the tests, use the below command

mvn clean test

The output of the above program is

This will create the allure-results folder with all the test reports within build folder. These files will be used to generate Allure Report.

8. How to Generate an Allure Report

allure generate processes the test results and saves an HTML report into the allure-report directory. To view the report, use the allure open command.

allure serve creates the same report as allure generate, then automatically opens the main page of the report in a web browser.

Use the command below to generate the Allure Report

 allure serve build/allure-results

This will generate the beautiful Allure Test Report as shown below.

9. How to View a Report

Test reports generated with Allure Report are basically small HTML websites intended to be viewed in a web browser. 

Title

A human-readable title of the test. If not provided, the function name is used instead.

Owner

The team member who is responsible for the test’s stability. For example, this can be the test’s author, the leading developer of the feature being tested, etc.

Severity

A value indicating how important the test is. This may give the future reader an idea of how to prioritize the investigations of different test failures.

Allowed values are: “trivial”, “minor”, “normal”, “critical”, and “blocker”.

Steps

Allure JUnit 5 provides three ways of creating steps and sub-steps: “annotated steps”, “lambda steps” and “no-op steps”

Links

We can provide list of links to webpages that may be useful for a reader investigating a test failure.

Graphs in Allure Report

Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.

Timeline in Allure Report

The timeline tab visualizes retrospective test execution. Allure adaptors collect precise timings of tests. Here on this tab, they are arranged according to their sequential or parallel timing structure.

Generate Reports with the Allure Awesome Plugin

Allure 3 Report also features an advanced report generator plugin – Allure Awesome. It supports additional configuration options, such as generating the report as a single HTML file, setting the theme, custom branding and language of the generated report, and taking known issues into account.

To manually generate a customized Allure Awesome report, use the awesome command:

 allure awesome build/allure-results

This will create the allure-report folder with all the test files and index.html report.

Open “index.html“, as this is an HTML report, and open it with the browser. The below image shows index.html.

Right click on index.html->Open In ->Browser ->Edge( any browser).

Summary:

  1. Add required dependencies – Include Playwright, JUnit 5, and Allure JUnit5 in pom.xml, and configure the Allure Maven plugin.
  2. Write Test Case – Implement test cases using JUnit 5 syntax, leveraging Playwright for browser-based actions. Use annotations like @Test, @BeforeEach and @AfterEach to structure your tests.
  3. Enable JUnit 5 extension auto-detection – Create src/test/resources/junit-platform.properties and turn on extension autodetection.
  4. Annotate tests & capture artifacts – Use Allure annotations (@Epic, @Feature, @Story, @Severity, @Description), log steps with Allure.step, and attach screenshots or other files.
  5. Execute Tests and Generate Reports – Execute mvn clean test, then run allure serve allure-results to build and open the HTML report.

Congratulations!! We have integrated an allure report with Playwright, Java, and JUnit5. I hope this tutorial is useful to you.

Additional Tutorials on Allure Reports

Integration of Allure Report with Selenium and JUnit4
Integration of Allure Report with Selenium and TestNG
Gradle – Allure Report for Selenium and JUnit4
Gradle – Allure Report for Cucumber, Selenium and TestNG
Integration of Allure Report with Rest Assured and JUnit4

vibssingh

Integrate Allure Report with Playwright with Java and TestNG

Last Updated On

HOME

In this tutorial, I will explain how to Integrate Allure Report 3 with Playwright, Java and TestNG.

What is an Allure Report?

Allure Framework is a flexible, lightweight, multi-language test report tool. It shows a very concise representation of what has been tested in a neat web report form. It also allows everyone participating in the development process to extract maximum useful information from the everyday execution of tests.

Refer to this tutorial to install allure – What is Allure Report?.

In this tutorial, we are using Allure Report 3.

Table of Contents

  1. What is an Allure Report?
  2. System requirements
  3. Dependency List
  4. High-Level Execution Flow
  5. Implementation Steps
    1. Set Up the Environment
      1. Update the Properties section in Maven pom.xml
      2. Add dependencies to pom.xml
      3. Update the Build Section of pom.xml in the Allure Report Project
    2. Project Structure for Maintainability
    3. Creating Page Object Classes in src/test/java
    4. Create the utility package in src/test/java
    5. Write the Test Scripts
    6. Create testng.xml for the project
    7. Specifying Allure Results location
    8. Run the Tests and Generate Allure Report
    9. How to Generate an Allure Report
    10. How to View a Report
      1. Title
      2. Owner
      3. Severity
      4. Steps
      5. Attachment
      6. Graphs in Allure Report
      7. Timeline in Allure Report
      8. Generate Reports with the Allure Awesome Plugin

System requirements

  1. Java 17 installed
  2. Maven installed
  3. Eclipse or IntelliJ installed
  4. Allure installed
  5. Browsers on which tests need to be run, like Chrome, Firefox, etc.

Dependency List

  1. Playwright – 1.57.0
  2. Java 17
  3. Maven – 3.9.6
  4. Allure Report – 2.32.0
  5. Aspectj – 1.9.25
  6. Maven Compiler Plugin – 3.13.0
  7. Maven Surefire Plugin – 3.5.4

High-Level Execution Flow

Implementation Steps

1. Set Up the Environment

Step 1 – Update the Properties section in Maven pom.xml
<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <playwright.version>1.57.0</playwright.version>
    <testng.version>7.11.0</testng.version>
    <allure.version>2.32.0</allure.version>
    <aspectj.version>1.9.25</aspectj.version>
    <maven.compiler.plugin.version>3.13.0</maven.compiler.plugin.version>
    <maven.compiler.source.version>17</maven.compiler.source.version>
    <maven.compiler.target.version>17</maven.compiler.target.version>
    <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
  </properties>

Step 2 – Add dependencies to pom.xml

Add Playwright, TestNG, and Allure-TestNG dependencies to pom.xml (Maven Project).

 <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-bom</artifactId>
                <version>${allure.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>

        <!-- Playwright -->
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>${playwright.version}</version>
        </dependency>

        <!-- TestNG -->
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>${testng.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Allure-TestNG -->
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-testng</artifactId>
            <scope>test</scope>
        </dependency>

    </dependencies>

Step 3 – Update the Build Section of pom.xml in the Allure Report Project

Allure leverages AspectJ for the functionality of @Step and @Attachment annotations.

 <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source.version}</source>
                    <target>${maven.compiler.target.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

The entire pom.xml is

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>Playwright_PageObjectModel</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Playwright_PageObjectModel</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <playwright.version>1.57.0</playwright.version>
        <testng.version>7.11.0</testng.version>
        <allure.version>2.32.0</allure.version>
        <aspectj.version>1.9.25</aspectj.version>
        <maven.compiler.plugin.version>3.13.0</maven.compiler.plugin.version>
        <maven.compiler.source.version>17</maven.compiler.source.version>
        <maven.compiler.target.version>17</maven.compiler.target.version>
        <maven.surefire.plugin.version>3.5.4</maven.surefire.plugin.version>
    </properties>

    <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-bom</artifactId>
                <version>${allure.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>

        <!-- Playwright -->
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>${playwright.version}</version>
        </dependency>

        <!-- TestNG -->
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>${testng.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Allure-TestNG -->
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-testng</artifactId>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source.version}</source>
                    <target>${maven.compiler.target.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <suiteXmlFiles>
                        <suiteXmlFile>testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>

2. Project Structure for Maintainability

Creating a well-organized project structure is crucial for maintaining a scalable and efficient automation framework.

3. Creating Page Object Classes in src/test/java

Please refer to – Mastering Page Object Model for Playwright with Java and TestNG to know more about the Page Object Classes and the respective code.

4. Create the utility package in src/test/java

AllureAttachment class is created to attach binary data (the screenshot) to the currently running test in the Allure lifecycle. This gives a central attachment library for the framework.

package com.example.utils;

import io.qameta.allure.Attachment;

public abstract class AllureAttachments {

    @Attachment(value = "Failure Screenshot", type = "image/png")
    public static byte[] attachScreenshot(byte[] screenshot) {
        return screenshot;
    }
}

BaseClass

This class contains the common variables and methods used throughout the project, like setup and teardown methods.

package com.example.utils;

import com.microsoft.playwright.*;
import org.testng.annotations.*;

public class BaseClass {

    // Shared between all tests in this class.
    Playwright playwright;
    Browser browser = null;

    // New instance for each test method.
    BrowserContext context;
    public Page page;

    public Page getPage() {
        return page;
    }

    @BeforeClass
    public void launchBrowser() {
        playwright = Playwright.create();
        browser = playwright.chromium().launch(new BrowserType.LaunchOptions().setHeadless(false));
    }

    @BeforeMethod
    public void createContextAndPage() {
        context = browser.newContext();
        page = context.newPage();
        page.navigate("https://opensource-demo.orangehrmlive.com/web/index.php/auth/login");
    }

    @AfterMethod
    public void closeContext() {
        context.close();
    }

    @AfterClass
    public void closeBrowser() {
        playwright.close();
    }
}

TestListener

This class is a TestNG failure hook. Its job is to automatically capture a Playwright screenshot when a test fails. The screenshot is then attached to the Allure report.

By implementing ITestListener, we are plugging into the TestNG test lifecycle. That means this class gets called automatically whenever a test fails.

package com.example.utils;

import com.microsoft.playwright.Page;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class TestListener implements ITestListener {

     @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("❌ onTestFailure triggered");
        Page page = BaseClass.getPage();
        if (page != null) {
            byte[] screenshot = page.screenshot();
            AllureAttachments.attachScreenshot(screenshot);
        }
    }
}

5. Write the Test Scripts

Create a Test files under src/test/java. Use these page classes in your test scripts to perform end-to-end scenarios. This will keep your tests clean and focused on logic rather than details about the UI elements.

We will be automating the following test scenario using Playwright Java and run them in Chromium.

  1. Verify the invalid username generates error message “Invalid credentials”
  2. Verify that the user is able to login to the application successfully.
  3. Verify the missing username generates error message “Required1”. This test will fail

Creating a Login Page Test Class:

package com.example.tests;

import com.example.pages.DashboardPage;
import com.example.pages.LoginPage;
import com.example.utils.BaseClass;
import com.example.utils.TestListener;
import io.qameta.allure.Description;
import io.qameta.allure.Owner;
import io.qameta.allure.Severity;
import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
import static io.qameta.allure.SeverityLevel.*;

public class LoginTests extends BaseClass {

    @Test
    @Description("This test attempts to log into the website using a invalid login and a password that result in error")
    @Severity(NORMAL)
    @Owner("Vibha Singh")
    public void unsuccessfulLogin() {
        LoginPage loginPage = new LoginPage(page);
        loginPage.login("abc","abc");
        String actualErrorMessage = loginPage.getErrorMessage();
        Assert.assertEquals(actualErrorMessage, "Invalid credentials");
   }

    @Test
    @Description("This test attempts to log into the website using a valid login and a password")
    @Severity(CRITICAL)
    @Owner("Vibha Singh")
    public void successfulLogin() {
        LoginPage loginPage = new LoginPage(page);
        loginPage.login("Admin","admin123");
        DashboardPage dashboardPage = new DashboardPage(page);
        String actualHeading = dashboardPage.getDashboardPageHeading();
        Assert.assertEquals(actualHeading, "Dashboard");
    }

    @Test
    @Description("This test attempts to log into the website using a blank login and a password that result in error")
    @Severity(MINOR)
    @Owner("Vibha Singh")
    public void missingUsername() {
        LoginPage loginPage = new LoginPage(page);
        loginPage.login("","admin123");
        String actualErrorMessage = loginPage.getMissingUsernameErrorMessage();
        Assert.assertEquals(actualErrorMessage, "Required1");
    }

}

Use the @Description() annotation to set a description statically or use the description() method to set it dynamically in runtime.

@Description("This test attempts to log into the website using a invalid login and a password that result in error")

6. Create testng.xml for the project

Mention the listener in the testng.xml. This is used for all the test classes.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Playwright test suite">

    <listeners>
        <listener class-name="com.example.utils.TestListener"/>
    </listeners>

    <test name="Integration of Playwright Java with TestNG">
    <classes>
        <class name="com.example.tests.LoginTests">

        </class>
    </classes>
    </test>
</suite>

7. Specifying Allure Results location

Allure, by default, saves test results in the project’s root directory. However, it is recommended to store your test results in the build output directory.

To configure this, create an allure.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:

#allure.properties
allure.results.directory=target/allure-results

8. Run the Tests and Generate Allure Report

To run the tests, use the below command

mvn clean test

In the image below, we can see that one test failed and four passed out of three tests.

This will create the allure-results folder with all the test reports within build folder. These files will be used to generate Allure Report.

9. How to Generate an Allure Report

allure generate processes the test results and saves an HTML report into the allure-report directory. To view the report, use the allure open command.

allure serve creates the same report as allure generate, then automatically opens the main page of the report in a web browser.

Use the command below to generate the Allure Report

 allure serve build/allure-results

This will generate the beautiful Allure Test Report as shown below.

10. How to View a Report

Test reports generated with Allure Report are basically small HTML websites intended to be viewed in a web browser. 

Title

A human-readable title of the test. If not provided, the function name is used instead.

Owner

The team member who is responsible for the test’s stability. For example, this can be the test’s author, the leading developer of the feature being tested, etc.

Severity

A value indicating how important the test is. This may give the future reader an idea of how to prioritize the investigations of different test failures.

Allowed values are: “trivial”, “minor”, “normal”, “critical”, and “blocker”.

Steps

Allure TestNG provides three ways of creating steps and sub-steps: “annotated steps”, “lambda steps” and “no-op steps”. Define a test step with the given name.

Attachment

We can attach any sorts of files to your Allure report. Allure TestNG provides various ways to create an attachment, both from existing files or generated dynamically. To create an attachment using the Annotations API, define a method that returns some data and annotate it with @Attachment. Call the method at any point during your test.

Click on the highlighted portion and we can see the screenshot of the failed test.

Graphs in Allure Report

Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.

Timeline in Allure Report

The timeline tab visualizes retrospective test execution. Allure adaptors collect precise timings of tests. Here on this tab, they are arranged according to their sequential or parallel timing structure.

Generate Reports with the Allure Awesome Plugin

Allure 3 Report also features an advanced report generator plugin – Allure Awesome. It supports additional configuration options, such as generating the report as a single HTML file, setting the theme, custom branding and language of the generated report, and taking known issues into account.

To manually generate a customized Allure Awesome report, use the awesome command:

 allure awesome build/allure-results

Congratulations!! We have integrated an allure report with Cucumber, Selenium, and TestNG. I hope this tutorial is useful to you.

Additional Tutorials on Allure Reports

Integration of Allure Report with Selenium and JUnit4
Integration of Allure Report with Selenium and TestNG
Gradle – Allure Report for Selenium and JUnit4
Gradle – Allure Report for Cucumber, Selenium and TestNG
Integration of Allure Report with Rest Assured and JUnit4

vibssingh

What is Allure Report?

Last Updated On

HOME

In this tutorial, we will understand about the Allure Report.

Table of Contents

  1. What is an Allure Report?
  2. How Allure Report is generated?
  3. Install Allure
  4. Sample Allure Report

What is an Allure Report?

Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what has been tested in a neat web report form but allows everyone participating in the development process to extract the maximum useful information from the everyday execution of tests.

How Allure Report is generated?

Allure is based on standard xUnit results output but adds some supplementary data. Any report is generated in two steps. During test execution (first step), a small library called adapter attached to the testing framework saves information about executed tests to XML files. We already provide adapters for popular Java, PHP, Ruby, Python, Scala, and C# test frameworks. During report generation (second step), the XML files are transformed into an HTML report. This can be done with a command line tool, a plugin for CI, or a build tool.

Similarly, when we run our tests, every popular test framework generates junit-style XML report or testng style which will be used by Allure to generate HTML report.

In the below example, we use the maven surefire plugin which automatically generates xml test reports and stores them in target/surefire-reports. And these XML files are transformed into an HTML report by Allure.

Allure reports have provided adapters for Java, PHP, Ruby, Python, Scala, and C# test frameworks.

Allure report has the below-mentioned annotation.

@Epic
@Features
@Stories/@Story

We can add Epic, Feature, and Stories annotations to the test to describe the behaviour of the test.

@Severity(SeverityLevel.BLOCKER) – @Severity annotation is used in order to prioritize test methods by severity.

@Description(“Regression Testing”) – We can add a detailed description for each test method. To add such a description, use the @Description annotation.

@Step – In order to define steps in Java code, you need to annotate the respective methods with @Step annotation. When not specified, the step name is equal to the annotated method name.

@Attachment – An attachment in Java code is simply a method annotated with @Attachment that returns either a String or byte[], which should be added to the report.

@Link – We can link the tests to Defect Tracker or JIRA Ticket.

Below is an example that shows how to use various Allure Annotations in the Test.

@Epic("Web Application Regression Testing")
@Feature("Login Page Tests")
@Listeners(TestExecutionListener.class)
public class LoginTests extends BaseTest {

	LoginPage objLogin;
	DashboardPage objDashboardPage;

	@Severity(SeverityLevel.NORMAL)
	@Test(priority = 0, description = "Verify Login Page")
	@Description("Test Description : Verify the title of Login Page")
	@Story("Title of Login Page")
	public void verifyLoginPage() {

		// Create Login Page object
		objLogin = new LoginPage(driver);

		// Verify login page text
		objLogin.verifyPageTitle();
	}
}

Install Allure

For Windows, Allure is available from the Scoop command line installer. 

Set-ExecutionPolicy RemoteSigned -scope CurrentUser

Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh')

To install Allure, download and install Scoop, and then execute in the Powershell:

scoop install allure

I have Allure installed, so I’m getting a message – ‘allure’ (2.14.0) is already installed.

To check if you have Allure installed or not, please use the below command in the command line or PowerShell.

allure --version

Sample Allure Report

You can find more information on Allure documentation.

Additional Tutorials on Allure Reports

Integration of Allure Report with Selenium and JUnit4
Integration of Allure Report with Selenium and TestNG
Allure Report with Cucumber5, Selenium and TestNG
Gradle – Allure Report for Cucumber, Selenium and TestNG
Integration of Allure Report with Rest Assured and JUnit4

Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! 

vibssingh

Integration of Allure Report with Rest Assured and JUnit4

Last Updated On

HOME

In the previous tutorial, I explained the Allure Report with Cucumber, Selenium and JUnit4. In this tutorial, I will explain how to Integrate Allure Report with Rest Assured and JUnit4.

The below example covers the implementation of Allure Report for Rest API using Rest Assured, JUnit4, Java, and Maven.

Table of Contents

  1. Prerequisite
  2. Dependency List
  3. Implementation Steps
    1. Update Properties section in Maven pom.xml
    2. Add the Allure-Rest Assured dependency
    3. Update the Build Section of pom.xml in Allure Report Project
    4. Create a Test Code for the testing of REST API under src/test/java
    5. Run the Test and Generate Allure Report
  4. Allure Report Dashboard
    1. Categories in Allure Report
    2. Suites in Allure Report
    3. View test history
    4. Graphs in Allure Report
    5. Timeline in Allure Report
    6. Behaviors of Allure Report
    7. Packages in Allure Report

Prerequisite

  1. Java 11 or above installed
  2. Maven installed
  3. Eclipse or IntelliJ installed
  4. Allure installed

Dependency List:

  1. Java 17
  2. Maven – 3.9.6
  3. Allure Maven – 2.12.0
  4. Rest Assured – 5.3.4
  5. Allure Rest Assured – 2.25.0
  6. Allure JUnit4 – 2.25.0
  7. Aspectj – 1.9.21
  8. JUnit – 4.13.2

Implementation Steps

Step 1 – Update Properties section in Maven pom.xml

<properties>
        <rest-assured.version>5.4.0</rest-assured.version>
        <junit.version>4.13.2</junit.version>
        <json.version>20231013</json.version>
        <hamcrest.version>1.3</hamcrest.version>
        <allure.maven.version>2.12.0</allure.maven.version>
        <allure.junit.version>2.25.0</allure.junit.version>
        <allure.rest.assured.version>2.25.0</allure.rest.assured.version>
        <maven.compiler.plugin.version>3.12.1</maven.compiler.plugin.version>
        <maven.surefire.plugin.version>3.2.3</maven.surefire.plugin.version>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <aspectj.version>1.9.21</aspectj.version>
</properties>

Step 2 – Add the Allure-Rest Assured dependency

<!--Allure Reporting Dependency-->   
<dependency>
    <groupId>io.qameta.allure</groupId>
    <artifactId>allure-rest-assured</artifactId>
    <version>${allure.rest-assured.version}</version>
</dependency>

Add other dependencies like Rest Assured and Allure-JUnit4 dependencies in POM.xml

<dependencies>

        <!-- Rest-Assured Dependency -->
        <dependency>
            <groupId>io.rest-assured</groupId>
            <artifactId>rest-assured</artifactId>
            <version>${rest-assured.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- JUnit4 Dependency -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- JSON Dependency -->
        <dependency>
            <groupId>org.json</groupId>
            <artifactId>json</artifactId>
            <version>${json.version}</version>
        </dependency>

        <!-- Allure JUnit Dependency -->
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-junit4</artifactId>
            <version>${allure.junit.version}</version>
        </dependency>

        <!-- Allure Rest-assured Dependency-->
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-rest-assured</artifactId>
            <version>${allure.rest.assured.version}</version>
        </dependency>

    </dependencies>

Step 3 – Update the Build Section of pom.xml in Allure Report Project

<build>
        <plugins>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                    <properties>
                        <property>
                            <name>listener</name>
                            <value>io.qameta.allure.junit4.AllureJunit4</value>
                        </property>
                    </properties>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                    </dependency>
                </dependencies>
            </plugin>

            <plugin>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-maven</artifactId>
                <version>${allure.maven.version}</version>
                <configuration>
                    <reportVersion>${allure.maven.version}</reportVersion>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Step 4 – Create a Test Code for the testing of REST API under src/test/java

Rest Assured and Allure Report are two popular tools for testing. Rest Assured is used for API testing and Allure Report is used for creating detailed reports about tests. To see our request and response in more detail using these tools, we need to add a line to our Rest Assured tests. This will provide the request and response details in the report.

For testing purpose, first test – Get Request one fails.

.filter(new AllureRestAssured())

Below is an example of the tests.

package org.example;

import io.qameta.allure.*;
import io.qameta.allure.restassured.AllureRestAssured;
import io.restassured.http.ContentType;
import org.json.JSONObject;
import org.junit.Test;

import static io.restassured.RestAssured.given;
import static org.hamcrest.core.IsEqual.equalTo;

@Epic("REST API Regression Testing using JUnit4")
@Feature("Verify CRUID Operations on Employee module")
public class APITests {

    String BaseURL = "https://dummy.restapiexample.com/api";


    @Test
    @Story("GET Request")
    @Severity(SeverityLevel.NORMAL)
    @Description("Test Description : Verify the details of employee of id-2")
    public void getUser() {

        // GIVEN
        given()
                .filter(new AllureRestAssured())

                // WHEN
                .when()
                .get(BaseURL + "/v1/employee/2")

                // THEN
                .then()
                .statusCode(200)
                .statusLine("HTTP/1.1 200 OK")
                // To verify booking id at index 2
                .body("data.employee_name", equalTo("Garrett Winters!"))
                .body("message", equalTo("Successfully! Record has been fetched."));

    }

    @Test
    @Story("POST Request")
    @Severity(SeverityLevel.NORMAL)
    @Description("Test Description : Verify the creation of a new employee")
    public void createUser() {

        JSONObject data = new JSONObject();

        data.put("employee_name", "APITest");
        data.put("employee_salary", "99999");
        data.put("employee_age", "30");

        // GIVEN
        given()
                .filter(new AllureRestAssured())
                .contentType(ContentType.JSON)
                .body(data.toString())

                // WHEN
                .when()
                .post(BaseURL + "/v1/create")

                // THEN
                .then()
                .statusCode(200)
                .body("data.employee_name", equalTo("APITest"))
                .body("message", equalTo("Successfully! Record has been added."));

    }
}

Step 5 – Run the Test and Generate Allure Report

To run the tests, use the below command

mvn clean test

The output of the above program is

This will create allure-results folder with all the test reports. These files will be used to generate Allure Report.

To create Allure Report, use the below command

allure serve

This will generate the beautiful Allure Test Report as shown below.

Allure Report Dashboard

Categories in Allure Report

The categories tab gives you a way to create custom defects classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).

Suites in Allure Report

On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.

View test history

Each time you run the report from the command line with the mvn clean test command, a new result JSON file will get added to the allure-results folder. Allure can use those files to include a historical view of your tests. Let’s give that a try.

To get started, run mvn clean test a few times and watch how the number of files in the allure-reports folder grows.

Now go back to view your report. Select Suites from the left nav, select one of your tests and click Retries in the right pane. You should see the history of test runs for that test:

Graphs in Allure Report

Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.

Timeline in Allure Report

Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.

Behaviors of Allure Report

This tab groups test results according to Epic, Feature, and Story tags.

The below image shows the request body sent and the status code of the response, its body, and header provided by API.

Packages in Allure Report

The packages tab represents a tree-like layout of test results, grouped by different packages.

We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!

vibssingh

Allure Reports

Last Updated On

HOME

Allure Framework is a lightweight, flexible multi-language test report tool that not only displays a very concise representation of what has been tested in a neat web report form, but also allows everyone involved in the development process to extract the most useful information from everyday test execution.

Allure Report for Maven Projects

Chapter 1 What is Allure Report?
Chapter 2 Integration of Allure Report with Selenium and JUnit4
Chapter 3 Integration of Allure Report with Selenium and JUnit5
Chapter 4 Integration of Allure Report with Selenium and TestNG
Chapter 5 Allure Report with Cucumber, Selenium and JUnit4
Chapter 6 Allure Report with Cucumber, Selenium and TestNG
Chapter 7 Integration of Allure Report with Rest Assured and JUnit4
Chapter 8 Integration of Allure Report with Rest Assured and TestNG
Chapter 9 Allure Report for Cucumber7, Selenium, and JUnit5
Chapter 10 Integration of Allure Report with Jenkins

Allure Report for Gradle Projects

Chapter 1 Gradle – Allure Report for Selenium and TestNG
Chapter 2 Gradle – Allure Report for Selenium and JUnit4
Chapter 3 Gradle – Allure Report for Cucumber, Selenium and TestNG
vibssingh

Allure Report with Cucumber, Selenium and TestNG

Last Updated On

HOME

In the previous tutorial, I explained the Integration of the Allure Report with Selenium and TestNG. In this tutorial, I will explain how to Integrate Allure Report with Cucumber, Selenium, and TestNG.

The below example covers the implementation of Allure Reports with Cucumber, Selenium, TestNG, Java, and Maven. Before starting, make sure to install Allure on your machine. Refer to this tutorial to install allure – What is Allure Report?.

Project Structure

Table of Contents

  1. Prerequisite
  2. Dependency List
  3. Implementation Steps
    1. Update the Properties section in Maven pom.xml
    2. Add Cucumber5, Selenium, TestNG, Allure-Cucumber5, and Allure-TestNG dependencies
    3. Update the Build Section of pom.xml in the Allure Report Project
    4. Create a Feature file
    5. Create the Step Definition class or Glue Code
    6. Create a TestNG Cucumber Runner class
    7. Create testng.xml for the project
    8. Run the Test and Generate Allure Report
  4. Allure Report Dashboard
    1. Categories in Allure Report
    2. Suites in Allure Report
    3. Graphs in Allure Report
    4. Timeline in Allure Report
    5. Behaviours of Allure Report
    6. Packages in Allure Report

Prerequisite

  1. Java 17 installed
  2. Maven installed
  3. Eclipse or IntelliJ installed
  4. Allure installed

Dependency List

  1. Selenium – 4.16.1
  2. Java 17
  3. Cucumber – 7.15.0
  4. Maven – 3.9.6
  5. Allure Report – 2.25.0
  6. Allure Maven – 2.12.0
  7. Aspectj – 1.9.21
  8. Maven Compiler Plugin – 3.12.1
  9. Maven Surefire Plugin – 3.2.3

Implementation Steps

Step 1 – Update the Properties section in Maven pom.xml

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <cucumber.version>7.15.0</cucumber.version>
    <selenium.version>4.16.1</selenium.version>
    <testng.version>7.9.0</testng.version>
    <maven.compiler.plugin.version>3.12.1</maven.compiler.plugin.version>
    <maven.surefire.plugin.version>3.2.3</maven.surefire.plugin.version>
    <maven.compiler.source.version>17</maven.compiler.source.version>
    <maven.compiler.target.version>17</maven.compiler.target.version>
    <allure.junit4.version>2.25.0</allure.junit4.version>
    <aspectj.version>1.9.21</aspectj.version>
    <allure.version>2.25.0</allure.version>
    <allure.maven>2.12.0</allure.maven>
</properties>

Step 2 – Add dependencies to pom.xml

Add Cucumber, Selenium, TestNG, Allure-Cucumber, and Allure-TestNG dependencies to pom.xml (Maven Project).

<dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-bom</artifactId>
        <version>${allure.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>


  <dependencies>

    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-java</artifactId>
      <version>${cucumber.version}</version>
    </dependency>

    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-testng</artifactId>
      <version>${cucumber.version}</version>
      <scope>test</scope>
    </dependency>

    <!-- Selenium -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-java</artifactId>
      <version>${selenium.version}</version>
    </dependency>

    <!-- TestNG -->
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>${testng.version}</version>
      <scope>test</scope>
    </dependency>

    <!--Allure Cucumber Dependency-->
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-cucumber7-jvm</artifactId>
      <scope>test</scope>
    </dependency>

    <!--Allure Reporting Dependency-->
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-testng</artifactId>
      <scope>test</scope>
    </dependency>

  </dependencies>

Step 3 – Update the Build Section of pom.xml in the Allure Report Project

<build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven.compiler.plugin.version}</version>
        <configuration>
          <source>${maven.compiler.source.version}</source>
          <target>${maven.compiler.target.version}</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${maven.surefire.plugin.version}</version>
        <configuration>
          <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
          </suiteXmlFiles>
          <argLine>
            -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
          </argLine>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>${aspectj.version}</version>
            <scope>runtime</scope>
          </dependency>
        </dependencies>
      </plugin>
      <plugin>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-maven</artifactId>
        <version>${allure.maven}</version>
        <configuration>
          <reportVersion>${allure.maven}</reportVersion>
        </configuration>
      </plugin>
    </plugins>
  </build>

The complete pom.xml is

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.example</groupId>
  <artifactId>AllureReport_Cucumber_TestNG</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>AllureReport_Cucumber_TestNG</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <cucumber.version>7.15.0</cucumber.version>
    <selenium.version>4.16.1</selenium.version>
    <testng.version>7.9.0</testng.version>
    <maven.compiler.plugin.version>3.12.1</maven.compiler.plugin.version>
    <maven.surefire.plugin.version>3.2.3</maven.surefire.plugin.version>
    <maven.compiler.source.version>17</maven.compiler.source.version>
    <maven.compiler.target.version>17</maven.compiler.target.version>
    <allure.junit4.version>2.25.0</allure.junit4.version>
    <aspectj.version>1.9.21</aspectj.version>
    <allure.version>2.25.0</allure.version>
    <allure.maven>2.12.0</allure.maven>
  </properties>

  <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-bom</artifactId>
        <version>${allure.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>

    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-java</artifactId>
      <version>${cucumber.version}</version>
    </dependency>

    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-testng</artifactId>
      <version>${cucumber.version}</version>
      <scope>test</scope>
    </dependency>

    <!-- Selenium -->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-java</artifactId>
      <version>${selenium.version}</version>
    </dependency>

    <!-- TestNG -->
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>${testng.version}</version>
      <scope>test</scope>
    </dependency>

    <!--Allure Cucumber Dependency-->
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-cucumber7-jvm</artifactId>
      <scope>test</scope>
    </dependency>

    <!--Allure Reporting Dependency-->
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-testng</artifactId>
      <scope>test</scope>
    </dependency>

  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven.compiler.plugin.version}</version>
        <configuration>
          <source>${maven.compiler.source.version}</source>
          <target>${maven.compiler.target.version}</target>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${maven.surefire.plugin.version}</version>
        <configuration>
          <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
          </suiteXmlFiles>
          <argLine>
            -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
          </argLine>
        </configuration>
        <dependencies>
          <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>${aspectj.version}</version>
            <scope>runtime</scope>
          </dependency>
        </dependencies>
      </plugin>
      <plugin>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-maven</artifactId>
        <version>${allure.maven}</version>
        <configuration>
          <reportVersion>${allure.maven}</reportVersion>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Step 4 – Create a Feature file

Create a folder – features within src/test/resources to create test scenarios in the Feature file.

Feature file should be saved as an extension of .feature. Add the test scenarios in this feature file. I have added sample test scenarios. In this feature file. The test scenarios are written in Gherkins language.

Feature: Login to HRM Application

  Background:
    Given User is on HRMLogin page "https://opensource-demo.orangehrmlive.com/"

  @ValidCredentials
  Scenario: Login with valid credentials

    When User enters username as "Admin" and password as "admin123"
    Then User should be able to login successfully and new page open

  @InvalidCredentials
  Scenario Outline: Login with invalid credentials

    When User enters username as "<username>" and password as "<password>"
    Then User should be able to see error message "<errorMessage>"

    Examples:
      | username   | password  | errorMessage                      |
      | Admin      | admin12$$ | Invalid credentials               |
      | admin$$    | admin123  | Invalid credentials               |
      | abc123     | xyz$$     | Invalid credentials               |
      | 234        | xyz$$     | Invalid credentials!              |

Step 5 – Create the Step Definition class or Glue Code

The stepdefinition class is created in src/test/java directory.

Below is the code for the Hooks.

package com.example.definitions;

import io.cucumber.java.After;
import io.cucumber.java.Before;
import io.cucumber.java.Scenario;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

import java.time.Duration;

public class Hooks {
    protected static WebDriver driver;
    public final static int TIMEOUT = 5;

    @Before
    public void setUp() {

        ChromeOptions options = new ChromeOptions();
        options.addArguments("--start-maximized");
        driver = new ChromeDriver(options);
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(TIMEOUT));

    }

    @After
    public void tearDown(Scenario scenario) {
        try {
            String screenshotName = scenario.getName();
            if (scenario.isFailed()) {
                TakesScreenshot ts = (TakesScreenshot) driver;
                byte[] screenshot = ts.getScreenshotAs(OutputType.BYTES);
                scenario.attach(screenshot, "img/png", screenshotName);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        driver.quit();
    }

}

LoginPageDefinition

package com.example.definitions;


import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;

import org.openqa.selenium.By;
import org.testng.Assert;

public class LoginPageDefinitions {

   Hooks hooks;


    @Given("User is on HRMLogin page {string}")
    public void loginTest(String url) {

        hooks.driver.get(url);

    }

    @When("User enters username as {string} and password as {string}")
    public void goToHomePage(String userName, String passWord) {

        // login to application
        hooks.driver.findElement(By.name("username")).sendKeys(userName);
        hooks.driver.findElement(By.name("password")).sendKeys(passWord);
        hooks.driver.findElement(By.xpath("//*[@class='oxd-form']/div[3]/button")).submit();

        // go the next page
    }

    @Then("User should be able to login successfully and new page open")
    public void verifyLogin() {

        String homePageHeading = hooks.driver.findElement(By.xpath("//*[@class='oxd-topbar-header-breadcrumb']/h6")).getText();

        //Verify new page - HomePage
        Assert.assertEquals(homePageHeading,"Dashboard");

    }

    @Then("User should be able to see error message {string}")
    public void verifyErrorMessage(String expectedErrorMessage) {

        String actualErrorMessage = hooks.driver.findElement(By.xpath("//*[@class='orangehrm-login-error']/div[1]/div[1]/p")).getText();

        // Verify Error Message
        Assert.assertEquals(actualErrorMessage, expectedErrorMessage);

    }

}

Step 6 – Create a TestNG Cucumber Runner class

We need to create a class called Runner class to run the tests. This class will use the TestNG annotation @Test, which tells TestNG what is the test runner class.

package com.example.runner;

import org.testng.annotations.Test;
import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;


@Test
@CucumberOptions(tags = "", features = {"src/test/resources/features"}, glue = {"com.example.definitions"},
        plugin = {"pretty","io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"})

public class CucumberRunnerTests extends AbstractTestNGCucumberTests{

}

Note:- @Test annotation marks this class as part of the test. So, if we will remove this annotation, the Allure Report executes CucumberRunnerTests as a separate test suite, so there will be duplicate results.

Step 7 – Create testng.xml for the project

<?xml version = "1.0"encoding = "UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name = "Suite1">
    <test name = "Test Demo">
        <classes>
            <class name = "com.example.runner.CucumberRunnerTests"/>
        </classes>
    </test>
</suite>

Step 8 – Run the Test and Generate Allure Report

To run the tests, use the below command

mvn clean test

In the below image, we can see that one test failed and four passed out of five tests.

This will create the allure-results folder with all the test reports within target folder. These files will be used to generate Allure Report.

Use the below command to generate the Allure Report

allure serve

This will generate the beautiful Allure Test Report as shown below.

Allure Report Dashboard

Categories in Allure Report

The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).

Suites in Allure Report

On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found. Here, we have 2 suits – Feature and Surefire test. Surefire tests are executed from CucumberRunnerTests.

Graphs in Allure Report

Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.

Timeline in Allure Report

The timeline tab visualizes retrospective test execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged accordingly to their sequential or parallel timing structure.

Behaviours of Allure Report

This tab groups test results according to Epic, Feature, and Story tags.

Screenshot attached to the failed test case

Packages in Allure Report

The packages tab represents a tree-like layout of test results, grouped by different packages.

When we don’t use @Test in CucumberRunnerTests.java, then as mentioned above the Allure report will have duplicate details.

Congratulations!! We have integrated an allure report with Cucumber, Selenium, and TestNG. I hope this tutorial is useful to you.

Additional Tutorials on Allure Reports

Integration of Allure Report with Selenium and JUnit4
Integration of Allure Report with Selenium and TestNG
Gradle – Allure Report for Selenium and JUnit4
Gradle – Allure Report for Cucumber, Selenium and TestNG
Integration of Allure Report with Rest Assured and JUnit4

vibssingh

Allure Report with Cucumber, Selenium and JUnit4

Last Updated On

HOME

In the previous tutorial, I have explained the Integration of the Allure Report with Selenium and JUnit4. In this tutorial, I will explain how to Integrate Allure Report with Cucumber7 and JUnit4.

Below example covers the implementation of Allure Reports in Selenium using JUnit4, Java and Maven.

Table of Contents

  1. Prerequisite
  2. Dependency List
  3. Project Structure
  4. Implementation Steps
    1. Update Properties section in Maven pom.xml
    2. Add dependencies to POM.xml
    3. Update Build Section of pom.xml in Allure Report Project
    4. Create Feature file in src/test/resources
    5. Create the Step Definition class or Glue Code
    6. Create a Cucumber Runner class
    7. Create allure.properties in src/test/resources
    8. Run the Test and Generate Allure Report
    9. Generate Allure Report

Prerequisite

  1. Java 11 or above installed
  2. Maven installed
  3. Eclipse or IntelliJ installed
  4. Allure installed and configured

Dependency List:

  1. Selenium – 4.16.1
  2. Java 17
  3. Cucumber – 7.15.0
  4. Maven – 3.9.5
  5. Allure BOM – 2.25.0
  6. Aspectj – 1.9.21
  7. Allure Maven – 2.12.0
  8. JUnit – 4.13.2

Project Structure

Implementation Steps

Step 1 – Update Properties section in Maven pom.xml

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <cucumber.version>7.15.0</cucumber.version>
    <selenium.version>4.16.1</selenium.version>
    <junit.version>4.13.2</junit.version>
    <maven.compiler.plugin.version>3.12.1</maven.compiler.plugin.version>
    <maven.surefire.plugin.version>3.2.3</maven.surefire.plugin.version>
    <maven.compiler.source.version>17</maven.compiler.source.version>
    <maven.compiler.target.version>17</maven.compiler.target.version>
    <allure.junit4.version>2.25.0</allure.junit4.version>
    <aspectj.version>1.9.21</aspectj.version>
    <allure.version>2.25.0</allure.version>
    <allure.maven>2.12.0</allure.maven>
 </properties>

Step 2 – Add dependencies to POM.xml

Add Cucumber5, Selenium, JUnit4, Allure-Cucumber5 and Allure-JUnit4 dependencies in POM.xml

 <dependencies>
     
     <!--Cucumber Dependencies-->     
    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-java</artifactId>
      <version>${cucumber.version}</version>
    </dependency>
    
    <dependency>
      <groupId>io.cucumber</groupId>
      <artifactId>cucumber-junit</artifactId>
      <version>${cucumber.version}</version>
       <scope>test</scope>
    </dependency>
 
   <!--Selenium Dependency-->
    <dependency>
      <groupId>org.seleniumhq.selenium</groupId>
      <artifactId>selenium-java</artifactId>
      <version>${selenium.version}</version>
    </dependency>   

   <!--Hamcrest Dependency-->
    <dependency>
      <groupId>org.hamcrest</groupId>
      <artifactId>hamcrest</artifactId>
      <version>2.2</version>
      <scope>test</scope>
    </dependency>
    
   <!--Allure Cucumber Dependency-->     
      <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-cucumber5-jvm</artifactId>
        <version>${allure.cucumber5.version}</version>
    </dependency>
    
     <!--Allure Reporting Dependency-->     
    <dependency>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-junit4</artifactId>
        <version>${allure.junit4.version}</version>
        <scope>test</scope>
    </dependency>

  </dependencies>

Step 3 – Update Build Section of pom.xml in Allure Report Project

<build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source.version}</source>
                    <target>${maven.compiler.target.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                        <scope>runtime</scope>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-maven</artifactId>
                <version>${allure.maven}</version>
                <configuration>
                    <reportVersion>${allure.maven}</reportVersion>
                </configuration>
            </plugin>
        </plugins>
    </build>

The complete pom.xml is shown below:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>AllureReport_Cucumber_JUnit4</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>AllureReport_Cucumber_JUnit4</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <cucumber.version>7.15.0</cucumber.version>
        <selenium.version>4.16.1</selenium.version>
        <junit.version>4.13.2</junit.version>
        <maven.compiler.plugin.version>3.12.1</maven.compiler.plugin.version>
        <maven.surefire.plugin.version>3.2.3</maven.surefire.plugin.version>
        <maven.compiler.source.version>17</maven.compiler.source.version>
        <maven.compiler.target.version>17</maven.compiler.target.version>
        <allure.junit4.version>2.25.0</allure.junit4.version>
        <aspectj.version>1.9.21</aspectj.version>
        <allure.version>2.25.0</allure.version>
        <allure.maven>2.12.0</allure.maven>
    </properties>

    <!-- Add allure-bom to dependency management to ensure correct versions of all the dependencies are used -->
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-bom</artifactId>
                <version>${allure.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>


    <dependencies>

        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-java</artifactId>
            <version>${cucumber.version}</version>
        </dependency>

        <dependency>
            <groupId>io.cucumber</groupId>
            <artifactId>cucumber-junit</artifactId>
            <version>${cucumber.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Selenium -->
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>${selenium.version}</version>
        </dependency>

        <!-- JUnit4 -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

        <!--Allure Cucumber Dependency-->
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-cucumber7-jvm</artifactId>
            <scope>test</scope>
        </dependency>

        <!--Allure Reporting Dependency-->
        <dependency>
            <groupId>io.qameta.allure</groupId>
            <artifactId>allure-junit4</artifactId>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${maven.compiler.plugin.version}</version>
                <configuration>
                    <source>${maven.compiler.source.version}</source>
                    <target>${maven.compiler.target.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${maven.surefire.plugin.version}</version>
                <configuration>
                    <argLine>
                        -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
                    </argLine>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>org.aspectj</groupId>
                        <artifactId>aspectjweaver</artifactId>
                        <version>${aspectj.version}</version>
                        <scope>runtime</scope>
                    </dependency>
                </dependencies>
            </plugin>
            <plugin>
                <groupId>io.qameta.allure</groupId>
                <artifactId>allure-maven</artifactId>
                <version>${allure.maven}</version>
                <configuration>
                    <reportVersion>${allure.maven}</reportVersion>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Step 4 – Create Feature file in src/test/resources

Create source folder – src/test/resources and features folder within src/test/resources to create test scenarios in Feature file. Feature file should be saved as an extension of .feature. Add the test scenarios in this feature file. I have added sample test scenarios. In this feature file, I have created a scenario for successful login and one Scenario Outline for failed login. The test scenarios are written in Gherkins language.

Feature: Login to HRM Application

  Background:
    Given User is on HRMLogin page "https://opensource-demo.orangehrmlive.com/"

  @ValidCredentials
  Scenario: Login with valid credentials

    When User enters username as "Admin" and password as "admin123"
    Then User should be able to login successfully and new page open

  @InvalidCredentials
  Scenario Outline: Login with invalid credentials

    When User enters username as "<username>" and password as "<password>"
    Then User should be able to see error message "<errorMessage>"

    Examples:
      | username   | password    | errorMessage                      |
      | Admin        | admin12$$ | Invalid credentials               |
      | admin$$     | admin123   | Invalid credentials               |
      | abc123       | xyz$$          | Invalid credentials               |
      | 234             | xyz$$         | Invalid credentials!              |

Step 5 – Create the Step Definition class or Glue Code

The stepdefinition class is created in src/test/java.

Below is the code for the LoginDefinition class.

package com.example.definitions;

import io.cucumber.java.en.Given;
import io.cucumber.java.en.Then;
import io.cucumber.java.en.When;
import org.junit.Assert;
import org.openqa.selenium.By;

public class LoginPageDefinitions {

   Hooks hooks;

    @Given("User is on HRMLogin page {string}")
    public void loginTest(String url) {

        hooks.driver.get(url);

    }

    @When("User enters username as {string} and password as {string}")
    public void goToHomePage(String userName, String passWord) {

        // login to application
        hooks.driver.findElement(By.name("username")).sendKeys(userName);
        hooks.driver.findElement(By.name("password")).sendKeys(passWord);
        hooks.driver.findElement(By.xpath("//*[@class='oxd-form']/div[3]/button")).submit();

        // go the next page
    }

    @Then("User should be able to login successfully and new page open")
    public void verifyLogin() {

        String homePageHeading = hooks.driver.findElement(By.xpath("//*[@class='oxd-topbar-header-breadcrumb']/h6")).getText();

        //Verify new page - HomePage
        Assert.assertEquals("Dashboard",homePageHeading);

    }

    @Then("User should be able to see error message {string}")
    public void verifyErrorMessage(String expectedErrorMessage) {

        String actualErrorMessage = hooks.driver.findElement(By.xpath("//*[@class='orangehrm-login-error']/div[1]/div[1]/p")).getText();

        // Verify Error Message
        Assert.assertEquals(expectedErrorMessage,actualErrorMessage);

    }

}

Hook.java

package com.example.definitions;

import io.cucumber.java.After;
import io.cucumber.java.Before;
import io.cucumber.java.Scenario;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

import java.time.Duration;

public class Hooks {
    protected static WebDriver driver;
    public final static int TIMEOUT = 5;

    @Before
    public void setUp() {

        ChromeOptions options = new ChromeOptions();
        options.addArguments("--start-maximized");
        driver = new ChromeDriver(options);
        driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(TIMEOUT));

    }

    @After
    public void tearDown(Scenario scenario) {
        try {
            String screenshotName = scenario.getName();
            if (scenario.isFailed()) {
                TakesScreenshot ts = (TakesScreenshot) driver;
                byte[] screenshot = ts.getScreenshotAs(OutputType.BYTES);
                scenario.attach(screenshot, "img/png", screenshotName);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        driver.quit();
    }
}

Step 6 – Create a Cucumber Runner class

We need to create a class called Runner class to run the tests. This class will use the JUnit annotation @RunWith(), which tells JUnit what is the test runner class.

package com.example.runner;

import org.junit.runner.RunWith;
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;

@RunWith(Cucumber.class)
@CucumberOptions(tags = "", features = {"src/test/resources/features"}, glue = {"com.example.definitions"},
        plugin = {"pretty","io.qameta.allure.cucumber7jvm.AllureCucumber7Jvm"})

public class CucumberRunnerTests  {

}

Step 7 – Create allure.properties in src/test/resources

Allure, by default, saves test results in the project’s root directory. However, it is recommended to store your test results in the build output directory. To configure this, create an allure.properties file and place it in the test resources directory of your project, which is typically located at src/test/resources:

allure.results.directory=target/allure-results

Step 8 – Run the Test and Generate Allure Report

To run the tests, use the below command

mvn clean test

In the below image, we can see that one test is failed and four passed out of five tests.

This will create allure-results folder with all the test report. These files will be use to generate Allure Report.

Step 9 – Generate Allure Report

Change current directory to target directory and then use the below command to generate the Allure Report

allure serve

This will generate the beautiful Allure Test Report as shown below.

Allure Report Dashboard

It shows detail of all the test steps and the screenshot of the failed test step also as shown below.

Categories in Allure Report

Categories tab gives you the way to create custom defects classification to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).

Suites in Allure Report

On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.

Graphs in Allure Report

Graphs allow you to see different statistics collected from the test data: statuses breakdown or severity and duration diagrams.

Timeline in Allure Report

Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab they are arranged accordingly to their sequential or parallel timing structure.

Behaviors of Allure Report

This tab groups test results according to Epic, Feature and Story tags.

Packages in Allure Report

Packages tab represents a tree-like layout of test results, grouped by different packages.

That’s it! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!

The complete code can be found on GitHub – vibssingh/AllureReport_Cucumber_JUnit4.

vibssingh

Integration of Allure Report with Jenkins

Last Updated On

HOME

In this tutorial, we are going to learn how we can integrate Allure Report with Jenkins. This tutorial is using a Maven project.

Before proceeding, I strongly advise you to read this tutorial to learn How to Setup Allure Report with Selenium TestNG.

Table of Contents

  1. Implementation Steps
    1. Download Allure Jenkins Plugin
    2. Configure Allure
    3. Create a new Maven Jenkins job
      1. Create a new project using the Maven project plugin
      2. Build Management
      3. Select a custom workspace
      4. Select “Allure Reports” from “Post Build Actions”
      5. Execute the tests
      6. View the Allure Report

Implementation Steps

  1. Download the Allure Jenkins Plugin
  2. Configure Allure
  3. Create a new Maven Jenkins job

Download Allure Jenkins Plugin

To generate Allure Report in Jenkins, we need to download Allure Plugin. Please refer to this tutorial to install the plugin – How to install Plugins in Jenkins

Configure Allure

Go back to the Manage Jenkins link as shown below:

When we click on the “Manage Jenkins” link, we are redirected to the Manage Jenkins page, where we can see various types of options, including the “Global Tool Configuration” option.

We need to set the Allure Commandline in Jenkins as shown below.

Click on the Allure Command line installations button. By default, “Install Automatically” will be checked, so since we are going to use the Allure installed on our local machine, “Install automatically” will install the latest version of Allure.

Provide the Name as ALLURE_HOME because that is what is currently installed on my machine, and also provide the path of Allure in the ALLURE_HOME textbox.

Click on the Apply and Save buttons.

Create a new Maven Jenkins job

Step 1: Create a new project using the Maven project plugin

  1. Give the Name of the project – AllureReportWithSelenium_Demo
  2. Click on the Maven project. 
  3. Click on the OK button.

In the General section, enter the project description in the Description box – This is demo of Allure Report in Jenkins.

Select Source Code Management as None if the project is locally present on the machine.

Step 2: Build Management

Go to the Build section of the new job.

  1. In the Root POM textbox, enter pom.xml
  2. In the Goals and options section, enter “clean test”

Click on the Advanced button.

Step 3: Select a custom workspace

Mention the full path of the project in the directory.

Step 4: Select “Allure Reports” from “Post Build Actions”

Scroll down to ‘Post Build Actions’ and click on the ‘Add Post Build Actions’ drop-down list. Select “Allure Report“. 

Enter the Result Path as “allure-results” and click on the “Save” button.

Click on the Apply and Save buttons.

We have created a new Maven project “AllureReportWithSelenium_Demo” with the configuration to run the Selenium with TestNG Tests and also to generate Allure Report after execution using Jenkins.

Step 5: Execute the tests

Let’s execute it now by clicking on the “Build Now” button. 

Right-click on Build Number (here in my case it is #1).

Click on Console Output to see the result.

Step 6: View the Allure Report

Once the execution is completed, we could see a link to view the ‘Allure Report’.

Click on the Allure Report. It displays the summary of the tests.

There is another way to create Allure Report in Jenkins, which is by using the Jenkins pipeline. To know more about this, please refer to this tutorial – How to create Jenkins pipeline for Allure Report.

Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!! Cheers!!

Additional Tutorials

Integration Of Jenkins With Selenium WebDriver
How to install Maven Plugin in Jenkins
Integrate Gradle project with Jenkins
Jenkins GitLab Integration
How to generate HTML Reports in Jenkins
Integration of Cucumber Report with TestNG in Jenkins
Serenity with Jenkins