Last Updated On
Serenity Reports are living documentation that contains the meaningful report for each Test. It illustrated narrative reports that document and describe what your application does and how it works.
JUnit5
Gradle – Integration of Selenium and JUnit5
The previous tutorial explained How to create Gradle project with Selenium and JUnit4 in a Gradle project. In this tutorial, I will explain how we can set up a Gradle project with Selenium and JUnit5.
Pre Requisite:
- Java 8 or above installed
- Eclipse or IntelliJ IDE installed
- Gradle Installed
- Environment variables JAVA_HOME and GRADLE_HOME are correctly configured
This framework consists of:
- Java 11
- JUnit Jupiter – 5.8.2
- JUnit Jupiter Engine – 5.8.2
- Gradle – 7.3.3 (Build Tool)
- Selenium – 4.3.0
Steps to set up Gradle Java Project for Selenium and JUnit5
- Download and Install Java on the system
- Download and setup Eclipse IDE on the system
- Setup Gradle on System
- Create a new Gradle Project
- Add Selenium and JUnit5 dependencies to the Gradle project
- Create Pages and Test Code for the pages
- Run the tests from Command Line
- Gradle Report generation
Project Structure
Implementation Steps
Step 1- Download and Install Java
Selenium needs Java to be installed on the system to run the tests. Click here to know How to install Java.
Step 2 – Download and setup Eclipse IDE on the system
The Eclipse IDE (integrated development environment) provides strong support for Java developers. Click here to know How to install Eclipse.
Step 3 – Setup Gradle
To build a test framework, we need to add several dependencies to the project. This can be achieved by any build tool. I have used Gradle Build Tool. Click here to know How to install Gradle.
Step 4 – Create a new Gradle Project
If you want to create the Gradle project from Eclipse IDE, click here to know How to create a Gradle Java project.
Step 5 – Add Selenium and JUnit5 dependencies to the Gradle project
/*
* This file was generated by the Gradle 'init' task.
*
*/
plugins {
// Apply the application plugin to add support for building a CLI application in Java.
id 'application'
}
repositories {
// Use Maven Central for resolving dependencies.
mavenCentral()
}
java {
sourceCompatibility = 11
targetCompatibility = 11
}
dependencies {
// Use JUnit Jupiter for testing.
testImplementation 'org.junit.jupiter:junit-jupiter:5.8.2'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.8.2'
implementation 'com.google.guava:guava:30.1.1-jre'
implementation 'org.seleniumhq.selenium:selenium-java:4.4.0'
implementation 'io.github.bonigarcia:webdrivermanager:5.3.0'
}
application {
// Define the main class for the application.
mainClass = 'com.example.App'
}
tasks.named('test') {
// Use JUnit Platform for unit tests.
useJUnitPlatform() {
}
testLogging {
events "passed", "skipped", "failed"
showStandardStreams = true
}
systemProperties System.properties
reports.html.setDestination(file("$projectDir/GradleReports"))
}
Step 6 – Create Pages and Test Code for the pages
We have used PageFactory model to build the tests. I have created a package named pages and created the page classes in that folder. Page class contains the locators of each web element present on that particular page along with the methods of performing actions using these web elements.
This is the BasePage that contains the PageFactory.initElements.
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.support.PageFactory;
public class BasePage {
public WebDriver driver;
public BasePage(WebDriver driver) {
this.driver = driver;
PageFactory.initElements(driver,this);
}
}
Below is the code for LoginPage and HomePage
LoginPage
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;
public class LoginPage extends BasePage{
public LoginPage(WebDriver driver) {
super(driver);
}
@FindBy(name = "username")
public WebElement userName;
@FindBy(name = "password")
public WebElement password;
@FindBy(xpath = "//*[@id='app']/div[1]/div/div[1]/div/div[2]/div[2]/form/div[1]/div/span")
public WebElement missingUsernameErrorMessage;
@FindBy(xpath = "//*[@id='app']/div[1]/div/div[1]/div/div[2]/div[2]/form/div[1]/div/span")
public WebElement missingPasswordErrorMessage;
@FindBy(xpath = "//*[@id='app']/div[1]/div/div[1]/div/div[2]/div[2]/form/div[3]/button")
public WebElement login;
@FindBy(xpath = "//*[@id='app']/div[1]/div/div[1]/div/div[2]/div[2]/div/div[1]/div[1]/p")
public WebElement errorMessage;
public String getMissingUsernameText() {
return missingUsernameErrorMessage.getText();
}
public String getMissingPasswordText() {
return missingPasswordErrorMessage.getText();
}
public String getErrorMessage() {
return errorMessage.getText();
}
public void login(String strUserName, String strPassword) {
userName.sendKeys(strUserName);
password.sendKeys(strPassword);
login.click();
}
}
HomePage
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.support.FindBy;
public class HomePage extends BasePage {
public HomePage(WebDriver driver) {
super(driver);
}
@FindBy(xpath = "//*[@id='app']/div[1]/div[2]/div[2]/div/div[1]/div[1]/div[1]/h5")
public WebElement homePageUserName;
public String getHomePageText() {
return homePageUserName.getText();
}
}
Here, we have BaseTests Class also which contains the common methods needed by other test pages.
import java.time.Duration;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import io.github.bonigarcia.wdm.WebDriverManager;
public class BaseTests {
public WebDriver driver;
public final static int TIMEOUT = 10;
@BeforeEach
public void setup() {
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
driver.manage().window().maximize();
driver.get("https://opensource-demo.orangehrmlive.com/");
driver.manage().timeouts().implicitlyWait(Duration.ofSeconds(TIMEOUT));
}
@AfterEach
public void tearDown() {
driver.quit();
}
}
LoginPageTests
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Disabled;
public class LoginPageTests extends BaseTests{
@ParameterizedTest
@CsvSource({
"admin$$,admin123",
"Admin,admin123!!",
"admin123,Admin",
"%%%%%,$$$$$$"})
public void invalidCredentials(String username, String password) {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login(username, password);
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getErrorMessage());
}
@Test
public void validLogin() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("Admin", "admin123");
HomePage objHomePage = new HomePage(driver);
// Verify Home Page
assertEquals("Employee Information",objHomePage.getHomePageText());
}
@Test
public void missingUsername() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("", "admin123");
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getMissingUsernameText());
}
@Test @Disabled
public void missingPassword() {
LoginPage objLoginPage = new LoginPage(driver);
objLoginPage.login("admin", "");
// Verify Error Message
assertEquals("Invalid credentials",objLoginPage.getMissingPasswordText());
}
}
Step 7 – Run the tests from Command Line
Note:- As you can see, my project has two parts – GradleSeleniumJUnit5_Demo and app.
Go to the app project and run the tests, using the below command.
gradle clean test
The output of the above program is
Step 8 – Gradle Report generation
Once the test execution is finished, refresh the project. We will see a folder – GradleReports. This report is generated when the tests are executed through the command line.
This folder contains index.html.
Right-click on index.html and select open with Web Browser. This report shows the summary of all the tests executed. As you can see that Failed tests are selected (highlighted in blue), so the name of the test failed along with the class name is displayed here.
This report contains detailed information about the failed test, which is shown below.
This shows the list of all the tests – passed, failed, or ignored.
Assumptions in JUnit5
Assumptions is a collection of utility methods that support conditional test execution based on assumptions.
In direct contrast to failed assertions, failed assumptions do not result in a test failure; rather, a failed assumption results in a test being aborted.
Assumptions are typically used whenever it does not make sense to continue execution of a given test method — for example, if the test depends on something that does not exist in the current runtime environment.
Junit 5 comes with a subset of the assumption methods that JUnit 4 provides with Java 8 lambda expressions and method references. All JUnit Jupiter assumptions are static methods in the org.junit.jupiter.api.Assumptions class.
- Assumptions.assumeTrue() – If the condition is true, then run the test, else aborting the test.
- Assumptions.false() – If the condition is false, then run the test, else aborting the test.
- Assumptions.assumingThat() – is much more flexible, If condition is true then executes, else do not abort test continue rest of code in test.
1. assumeTrue
The assumeTrue() method validates the given assumption to be true and if the assumption is true – the test proceed, otherwise, test execution is aborted.
int num1 = 4;
int num2=6;
int num3 = 24;
int num4=10;
@Test
void assumeTrueTest() {
System.setProperty("ENV", "TEST");
assumeTrue("TEST".equals(System.getProperty("ENV")));
// Since the condition is true rest of it will get executed
assertEquals((num1*num2),num3,"The product of "+num1+"and "+num2+"is equal to "+num3);
}
The output of the above program is
In the below example, assumeTrue() is false. So, the execution is skipped.
int num1 = 4;
int num2=6;
int num3 = 24;
int num4=10;
@Test
void assumeTrueTest1() {
System.setProperty("ENV", "TEST");
assumeTrue("QA".equals(System.getProperty("ENV")));
// Since the condition is true rest of it will not get executed
assertEquals((num1*num2),num3,"The product of "+num1+"and "+num2+"is equal to "+num3);
}
The output of the above program is
2. assumeFalse()
The assumeFalse() method validates the given assumption to false and if the assumption is false – test proceed, otherwise, test execution is aborted. In the below example, the test is false and we are using assumeFalse(), so the tests will be executed.
int num1 = 4;
int num2=6;
int num3 = 24;
int num4=10;
@Test
void assumeFalseTest1() {
System.setProperty("ENV", "TEST");
assumeFalse("DEV".equals(System.getProperty("ENV")));
// Since the condition is true rest of it will get executed
assertEquals((num1*num2),num3,"The product of "+num1+"and "+num2+"is equal to "+num3);
}
The output of the above program is
In the below example, the test is false and we are using assumeFalse(), so the tests will be executed.
@Test
void assumeFalseTest() {
System.setProperty("ENV", "TEST");
assumeFalse("TEST".equals(System.getProperty("ENV")));
// Since the condition is false rest of it will not get executed
assertEquals((num1*num2),num3,"The product of "+num1+"and "+num2+"is equal to "+num3);
}
The output of the above program is
3. assertThat()
This method executes the supplied Executable, but only if the supplied assumption is valid.
Unlike the other assumption methods, this method will not abort the test.
If the assumption is invalid, this method does nothing.
If the assumption is valid and the executable throws an exception, it will be treated like a regular test failure. The thrown exception will be rethrown as is but masked as an unchecked exception.
int num1 = 4;
int num2=6;
int num3 = 24;
int num4=10;
int num5=8;
int num6=2;
@Test
void assumingThatTest() {
System.setProperty("ENV", "UAT");
assumingThat(
"UAT".equals(System.getProperty("ENV")),
() -> {
// Since the condition is true, this assertion will get executed
System.out.println("Assuming that executable executed");
assertEquals((num1+num2),num4,"The product of "+ num1 +" and "+ num2 +" is not equal to "+num4);
});
// Since the condition is false rest of it will get executed
System.out.println("Loop outside");
assertEquals((num5-num2),num6,"The difference of "+ num5 +" and "+num2+" is not equal to " + num6);
}
The output of the above program is
In the below example, the condition is false , so we skip the execution of that condition. But, we execute the rest of the code.
int num1 = 4;
int num2=6;
int num3 = 24;
int num4=10;
@Test
void assumingThatTest1() {
System.setProperty("ENV", "UAT");
assumingThat(
"DEV".equals(System.getProperty("ENV")),
() -> {
// Since the condition is false, this assertion will not get executed
System.out.println("Assuming that executable executed");
assertEquals((num1+num2),num4,"The sum of "+num1+"and "+num2+"is not equal to "+num4);
});
System.out.println("Loop outside");
assertEquals((num1*num2),num3,"The product of "+num1+"and "+num2+"is not equal to "+num3);
}
The output of the above program is
Difference between Assumption and Assertion
The main difference between the assertions and assumptions is –
The assumption is use to decide whether we want to execute a section or the rest of the test method or not and if the condition is false then the test is skipped.
Whereas if a condition in an assertion fails then it fails the test and something needs to be fixed.
How to parameterize Tests in JUnit5
JUnit5 enables us to execute a single test method multiple times with a different sets of data. This is called Parameterization. Parameterized Tests are declared just like regular @Test methods but use the @ParameterizedTest annotation.
This article shows you how to run a test multiple times with different arguments, so-called ‘Parameterized Tests’, let’s see the following ways to provide arguments to the test:
- @ValueSource
- @EnumSource
- @MethodSource
- @CsvSource
- @CsvFileSource
- @ArgumentsSource
We need to add junit-jupiter-params to support parameterized tests. In the case of Maven, add the dependency to POM.xml
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.8.2</version>
<scope>test</scope>
</dependency>
In case of Gradle, add the dependency as
testCompile("org.junit.jupiter:junit-jupiter-params:5.8.2")
1. @ValueSource
Let us start with a simple example. The following example demonstrates a parameterized test that uses the @ValueSource annotation to specify an integer array as the source of arguments. The following @ParameterizedTest method will be invoked three times, with the values 5,6, and 0 respectively.
@ParameterizedTest
@ValueSource(ints = {5, 6, 0})
void test_int_arrays(int b) {
int a= 5;
int sum = a + b;
assertTrue(sum>8);
}
When executing the above-parameterized test method, each invocation will be reported separately.
The output of the above program is:
One of the limitations of value sources is that they only support these types:
- short (with the shorts attribute)
- byte (bytes attribute)
- int (ints attribute)
- long (longs attribute)
- float (floats attribute)
- double (doubles attribute)
- char (chars attribute)
- java.lang.String (strings attribute)
- java.lang.Class (classes attribute)
Also, we can only pass one argument to the test method each time.
In the below example, an array of strings is passed as the argument to the Parameterized Test.
@ParameterizedTest(name = "#{index} - Run test with args={0}")
@ValueSource(strings = {"java", "python", "javascript","php"})
void test_string_arrays(String arg) {
assertTrue(arg.length() > 1);
}
The output of the above program is:
@NullSource
It provides a single null an argument to the annotated @ParameterizedTest method.
@ParameterizedTest()
@NullSource
void nullString(String text) {
assertTrue(text == null);
}
The output of the above program is:
@EmptySource
It provides a single empty argument to the annotated @ParameterizedTest method of the following types:
- java.lang.String
- java.util.List
- java.util.Set
- java.util.Map
- primitive arrays (e.g. int[])
- object arrays (e.g. String[])
@ParameterizedTest
@EmptySource
void testMethodEmptySource(String argument) {
assertTrue(StringUtils.isEmpty(argument));
assertTrue(StringUtils.isBlank(argument));
}
The output of the above program is:
@NullAndEmptySource
We can pass empty or null values into the test via @EmptySource, @NullSource, or @NullAndEmptySource (since JUnit 5.4).
Let’s see the following example to test an isEmpty() method.
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EmptySource;
import org.junit.jupiter.params.provider.NullSource;
import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.*;
public class ParameterizedTestDemo {
@ParameterizedTest(name = "#{index} - isEmpty()? {0}")
@NullSource
@EmptySource
@ValueSource(strings = { " ", " ", "\t", "\n","a"})
void nullEmptyAndBlankStrings(String text) {
assertTrue(text == null || text.trim().isEmpty());
}
}
The parameterized test method result in seven invocations: 1 for null, 1 for the empty string, 4 for the explicit blank strings supplied via @ValueSource, and 1 non-blank string “a” supplied via @ValueSource.
The output of the above program is:
2. @EnumSource
@EnumSource provides a convenient way to use Enum constants.
public class EnumParameterizedTest {
enum Error {
Request_Invalid,
Request_Timeout,
RequestHeader_Invalid,
Concurrency_Failed,
ExternalCall_Failed,
Schema_Invalid,
Authentication_Failed;
}
@ParameterizedTest
@EnumSource(Error.class)
void test_enum(Error error) {
assertNotNull(error);
}
}
The output of the above program is:
The annotation provides an optional names attribute that lets you specify which constants shall be used, like in the following example. If omitted, all constants will be used.
@ParameterizedTest(name = "#{index} - Is Error contains {0}?")
@EnumSource(value = Error.class, names = {"Request_Invalid", "ExternalCall_Failed", "Concurrency_Failed", "Authentication_Failed"})
void test_enum_include(Error error) {
assertTrue(EnumSet.allOf(Error.class).contains(error));
}
The output of the above program is:
The @EnumSource annotation also provides an optional mode attribute that enables fine-grained control over which constants are passed to the test method. For example, you can exclude names from the enum constant pool or specify regular expressions as in the following examples.
@ParameterizedTest
@EnumSource(value = Error.class, mode = EnumSource.Mode.EXCLUDE, names = {"Request_Invalid", "Request_Timeout", "RequestHeader_Invalid"})
void test_enum_exclude(Error error) {
EnumSet<Error> excludeRequestRelatedError = EnumSet.range(Error.Concurrency_Failed, Error.Authentication_Failed);
assertTrue(excludeRequestRelatedError.contains(error));
}
The output of the above program is:
EnumSource.Mode.EXCLUDE – It selects all declared enum constants except those supplied via the names attribute.
EnumSource.Mode.MATCH_ALL – It selects only those enum constants whose names match any pattern supplied via the names attribute.
@ParameterizedTest
@EnumSource(mode = EnumSource.Mode.MATCH_ALL, names = "^.*Invalid")
void test_match(Error error) {
assertTrue(error.name().contains("Invalid"));
}
The output of the above program is
3. @MethodSource
@MethodSource allows you to refer to one or more factory methods of the test class or external classes.
Factory methods within the test class must be static unless the test class is annotated with @TestInstance(Lifecycle.PER_CLASS); whereas, factory methods in external classes must always be static. In addition, such factory methods must not accept any arguments.
If you only need a single parameter, you can return a Stream of instances of the parameter type as demonstrated in the following example.
@ParameterizedTest(name = "#{index} - Test with String : {0}")
@MethodSource("stringProvider")
void test_method_string(String arg) {
assertNotNull(arg);
}
// this need static
static Stream<String> stringProvider() {
return Stream.of("java", "junit5", null);
}
The output of the above program is
If you do not explicitly provide a factory method name via @MethodSource, JUnit Jupiter will search for a factory method that has the same name as the current @ParameterizedTest method by convention. This is demonstrated in the following example.
@ParameterizedTest(name = "#{index} - Test with String : {0}")
@MethodSource
void test_no_factory_methodname(String arg) {
assertNotNull(arg);
}
static Stream<String> test_no_factory_methodname() {
return Stream.of("java", "junit5", null);
}
The output of the above program is
Streams for primitive types (DoubleStream, IntStream, and LongStream) are also supported as demonstrated by the following example.
@ParameterizedTest(name = "#{index} - Test with Int : {0}")
@MethodSource("rangeProvider")
void test_method_int(int arg) {
assertTrue(arg < 6);
}
static IntStream rangeProvider() {
return IntStream.range(0, 6);
}
The output of the above program is
If a parameterized test method declares multiple parameters, you need to return a collection, stream, or array of Arguments instances or object arrays as shown below.
@ParameterizedTest
@MethodSource("stringIntAndListProvider")
void testWithMultiArgMethodSource(String str, int num, List<String> list) {
assertEquals(5, str.length());
assertTrue(num >=1 && num <=2);
assertEquals(2, list.size());
}
static Stream<Arguments> stringIntAndListProvider() {
return Stream.of(
arguments("apple", 1, Arrays.asList("a", "b")),
arguments("lemon", 2, Arrays.asList("x", "y"))
);
}
The output of the above program is
4. @CsvSource
@CsvSource allows you to express argument lists as comma-separated values (i.e., CSV String literals). Each string provided via the value attribute in @CsvSource represents a CSV record and results in one invocation of the parameterized test.
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class CSVParameterizedTest {
@ParameterizedTest
@CsvSource({
"java, 4",
"javascript, 7",
"python, 6",
"HTML, 4",
})
void test(String str, int length) {
assertEquals(length, str.length());
}
}
The output of the above program is
5. @CsvFileSource
@CsvFileSource lets us use comma-separated value (CSV) files from the classpath or the local file system. Each record from a CSV file results in one invocation of the parameterized test. The first record may optionally be used to supply CSV headers.
csvdemo.csv
@ParameterizedTest
@CsvFileSource(resources = "/csvdemo.csv")
void testLength(String str, int length) {
Assertions.assertEquals(length, str.length());
}
The output of the above program is
csv file with the heading
JUnit can ignore the headers via the numLinesToSkip attribute.
@ParameterizedTest
@CsvFileSource(files = "src/test/resources/csvdemo.csv",numLinesToSkip = 1)
void testStringLength(String str, int length) {
Assertions.assertEquals(length, str.length());
}
The output of the above program is
If you would like the headers to be used in the display names, you can set the useHeadersInDisplayName attribute to true. The examples below demonstrate the use of useHeadersInDisplayName.
@ParameterizedTest(name = "[{index}] {arguments}")
@CsvFileSource(files = "src/test/resources/csvdemo.csv",useHeadersInDisplayName = true)
void testStringLength1(String str, int length) {
assertEquals(length, str.length());
}
The output of the above program is
6. @ArgumentsSource
@ArgumentsSource can be used to specify a custom, reusable ArgumentsProvider. Note that an implementation of ArgumentsProvider must be declared as either a top-level class or as a static nested class.
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.ArgumentsProvider;
import java.util.stream.Stream;
public class CustomArgumentsProvider implements ArgumentsProvider {
@Override
public Stream<? extends Arguments>
provideArguments(ExtensionContext extensionContext) throws Exception {
return Stream.of("java", "junit5", "junit4", null).map(Arguments::of);
}
}
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ArgumentsSource;
import static org.junit.jupiter.api.Assertions.assertNotNull;
public class ArgumentsSourceTest {
@ParameterizedTest
@ArgumentsSource(CustomArgumentsProvider.class)
void test_argument_custom(String arg) {
assertNotNull(arg);
}
}
The output of the above program is
Congratulation. We have understood parameterization in JUnit5 tests. Happy Learning!!
How to run JUnit5 tests in order
The general practices say that automated tests should be able to run independently and with no specific order, as well as the result of the test should not depend on the results of previous tests. But there are situations where a specific order of test execution can be justified, especially in integration or end-to-end tests. The test methods don’t follow a specific order by default to execute the tests. The test cases need not necessarily execute in the order in which they have been written.
There are different ways or modes to set the order of execution for the test cases. This article shows how to control the JUnit 5 test execution order via the following MethodOrderer classes:
- DisplayName – sorts test methods alphanumerically based on their display names
- MethodName – sorts test methods alphanumerically based on their names and formal parameter lists
- Alphanumeric – sorts test methods alphanumerically based on their names and formal parameter lists. This is deprecated from JUnit Version 5.7 onwards
- OrderAnnotation – sorts test methods numerically based on values specified via the
@Orderannotation - Random – orders test methods pseudo-randomly and support the configuration of a custom seed
- Custom Order – A custom ordering sequence can be implemented by the interface MethodOrderer and providing it as the argument to @TestMethodOrder.
Let us create JUnit5 Tests and execute them.
public class OrderRandomDemo {
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 3 + 7);
}
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(10, 14 - 4);
}
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
@Test
void test_Divide() {
System.out.println("test_Divide()");
assertEquals(10, 30 / 3);
}
@Test
void test_IsEven() {
System.out.println("test_IsEven()");
assertEquals(0, 10%2);
}
}
The output of the above program
1. DisplayName
It sorts test methods alphanumerically based on their display names. Test Method can be anything annotated with @Test, @RepeatedTest, @ParameterizedTest, @TestFactory, or @TestTemplate.
@TestMethodOrder(MethodOrderer.DisplayName.class)
@TestMethodOrder is a type-level annotation that is used to configure a MethodOrderer for the test methods of the annotated test class or test interface.
MethodOrderer defines the API for ordering the test methods in a given test class.
Test Method – It is any method annotated with @Test, @RepeatedTest, @ParameterizedTest, @TestFactory, or @TestTemplate.
DisplayName.class – MethodOrderer that sorts methods alphanumerically based on their names using String.compareTo(String).
If two methods have the same name, String representations of their formal parameter lists will be used as a fallback for comparing the methods.
An example of sorting the tests based on their display names.
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestMethodOrder(MethodOrderer.DisplayName.class)
public class DisplayNameOrderedTests {
@DisplayName("C")
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 3 + 7);
}
@DisplayName("E")
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
@DisplayName("A")
@Test
void test_Divide() {
System.out.println("test_Divide()");
assertEquals(10, 30 / 3);
}
@DisplayName("D")
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(10, 18 - 8);
}
@DisplayName("B")
@Test
void test_IsEven() {
System.out.println("test_IsEven()");
assertEquals(0, 18%2);
}
}
We can see that the test methods are sorted alphanumerically based on their display name starting from A to E. The output of the above program
2. MethodName
This annotation sorts methods alphanumerically based on their names using String.compareTo(String).
If two methods have the same name, String representations of their formal parameter lists will be used as a fallback for comparing the methods.
@TestMethodOrder(MethodOrderer.MethodName.class)
Let us see an example of MethodName.
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestMethodOrder(MethodOrderer.MethodName.class)
class MethodNameOrderedTests {
@Test
void testE() {
System.out.println("testE");
assertEquals(10, 3 + 7);
}
@Test
void testA() {
System.out.println("testA");
assertEquals(10, 14 - 4);
}
@Test
void testC() {
System.out.println("testC");
assertEquals(10, 5 * 2);
}
@Test
void testB() {
System.out.println("testB");
assertEquals(10, 30 / 3);
}
@Test
void testD() {
System.out.println("testD");
assertEquals(10, 10 + 0);
}
}
The output of the above program
3. OrderAnnotation
This sorts test method numerically based on values specified via the @Order annotation.
Any methods that are assigned the same order value will be sorted arbitrarily adjacent to each other.
When any method is not annotated with @Order, it will be assigned the default order value, which will effectively cause them to appear at the end of the sorted list.
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
Let us see an example of OrderAnnotation.
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OrderAnnotationDemo {
@Test
@Order(3)
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 3 + 7);
}
@Test
@Order(4)
void test_IsOdd() {
System.out.println("test_IsOdd()");
assertEquals(1, 11%2);
}
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(10, 14 - 4);
}
@Test
@Order(1)
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
@Test
@Order(4)
void test_Divide() {
System.out.println("test_Divide()");
assertEquals(10, 30 / 3);
}
@Test
@Order(2)
void test_IsEven() {
System.out.println("test_IsEven()");
assertEquals(0, 10%2);
}
}
Here, test_Subtract() is not assigned any order value, so it is displayed as the last one in the last.
4. Random
These sorts of test methods are pseudo-randomnly.
@TestMethodOrder(MethodOrderer.Random.class)
Let us create a program to show the random order of tests in JUnit5.
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestMethodOrder(MethodOrderer.Random.class)
public class OrderRandomDemo {
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 3 + 7);
}
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(10, 14 - 4);
}
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
@Test
void test_Divide() {
System.out.println("test_Divide()");
assertEquals(10, 30 / 3);
}
@Test
void test_IsEven() {
System.out.println("test_IsEven()");
assertEquals(0, 10%2);
}
}
The output of the above program
5. Custom Order
We can define our own custom ordering sequence by implementing the interface MethodOrderer and providing it as the argument to @TestMethodOrder.
Here, the tests are arranged in descending method order.
import org.junit.jupiter.api.MethodDescriptor;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.MethodOrdererContext;
public class DescendingMethodOrder implements MethodOrderer {
@Override
public void orderMethods(MethodOrdererContext context) {
context.getMethodDescriptors().sort((MethodDescriptor m1, MethodDescriptor m2) ->
m2.getMethod().getName().compareTo(m1.getMethod().getName()));
}
}
Now, test the above custom order.
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestMethodOrder(DescendingMethodOrder.class)
public class CustomOrderTests {
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10 , 4 + 6);
}
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(10 , 17 - 7);
}
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10 , 5 * 2);
}
@Test
void test_Divide() {
System.out.println("test_Divide()");
assertEquals(10 , 20/2);
}
@Test
void test_IsEven() {
System.out.println("test_IsEven()");
assertEquals(0 , 20%2);
}
}
Notice the test output. The tests are executed in descending order. The result of the above program is
Test Classes Ordering
- ClassName: sorts test classes alphanumerically based on their fully qualified class names.
- DisplayName: sorts test classes alphanumerically based on their display names (see display name generation precedence rules).
- OrderAnnotation: sorts test classes numerically based on values specified via the @Order annotation.
- Random: orders test classes pseudo-randomly and support the configuration of a custom seed.
The configured ClassOrderer will be applied to all top-level test classes (including static nested test classes) and @Nested test classes.
1. ClassName
The Test Classes are sorted alphanumerically based on their fully qualified class names.
package JUnit5;
import org.junit.jupiter.api.ClassOrderer;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestClassOrder;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestClassOrder(ClassOrderer.ClassName.class)
public class ClassNameOrderTests {
@Nested
class Addition {
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 3 + 7);
}
}
@Nested
class IsEven {
@Test
void test_IsEven() {
System.out.println("test_IsEven()");
assertEquals(0, 10 % 2);
}
}
@Nested
class Subtraction {
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(9, 14 - 5);
}
}
@Nested
class Multiply {
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
}
}
The result of the above program is
2. DisplayName
It sorts test classes alphanumerically based on their display names.
package JUnit5;
import org.junit.jupiter.api.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestClassOrder(ClassOrderer.DisplayName.class)
public class ClassDisplayNameTests {
@Nested
@DisplayName("B")
class AppFlowTests {
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 6 + 4);
}
}
@Nested
@DisplayName("C")
class TearDownTests {
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(10, 15 - 5);
}
}
@Nested
@DisplayName("A")
class SetupTests {
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
}
}
The result of the above program is
3. OrderAnnotation in Class
The test classes are sorted numerically based on values specified via the @Order annotation.
import org.junit.jupiter.api.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestClassOrder(ClassOrderer.ClassName.class) //sorts test classes alphanumerically based on their fully qualified class names.
public class ClassOrderedTests {
@Nested
@Order(2)
class AppFlowTests {
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 3 + 7);
}
}
@Nested
@Order(3)
class TearDownTests {
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(9, 14 - 5);
}
}
@Nested
@Order(1)
class SetupTests {
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
}
}
The result of the above program is
4. Random
The test classes are sorted pseudo-randomly and support the configuration of a custom seed.
import org.junit.jupiter.api.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
@TestClassOrder(ClassOrderer.Random.class)
public class ClassRandomTests {
@Nested
class Addition {
@Test
void test_Add() {
System.out.println("test_Add()");
assertEquals(10, 3 + 7);
}
}
@Nested
class IsEven {
@Test
void test_IsEven() {
System.out.println("test_IsEven()");
assertEquals(0, 10 % 2);
}
}
@Nested
class Subtraction {
@Test
void test_Subtract() {
System.out.println("test_Subtract()");
assertEquals(9, 14 - 5);
}
}
@Nested
class Multiply {
@Test
void test_Multiply() {
System.out.println("test_Multiply()");
assertEquals(10, 5 * 2);
}
}
}
The result of the above program is
Congratulation!! We have gone through different types of ordering in JUnit5. Happy Learning!!
How to run JUnit5 tests through Command Line
The previous tutorial explains to configure Junit in IntelliJ and run the tests as JUnit Tests. This tutorial shows the steps to run the tests through command line. We can ask, why we need to run the tests through command line?? There are many reasons, one of the reason is to achieve CI/CD. To run the tests in pipeline, they need to be run through command line. Another reason is that we don’t need to open the IDE to run the tests. Third reason is that many reports are only generated (Serenity, Cucumber), if the tests run through command line.
Below is a JUnit5 test.
import io.github.bonigarcia.wdm.WebDriverManager;
import org.junit.jupiter.api.*;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
public class Demo {
WebDriver driver;
@BeforeEach
public void setUp() {
WebDriverManager.chromedriver().setup();
ChromeOptions chromeOptions = new ChromeOptions();
driver = new ChromeDriver(chromeOptions);
driver.manage().window().fullscreen();
}
@Test
public void Junit5Test() {
driver.get("http://automationpractice.com/index.php");
System.out.println("Title of Page :" + driver.getTitle());
System.out.println("Page URL : " + driver.getCurrentUrl());
Assertions.assertEquals("My Store",driver.getTitle());
}
@AfterEach
public void tearDown() {
driver.close();
}
}
Let us see what happens when we try to run the JUnit tests through Command Line. This command is used to run the tests present in Demo class.
mvn clean test -Dtest=Demo
The output generated by the test is shown below
This shows that surefire-plugin is need to be add to the project to run t he tests successfully through command line.
Add surefire-plugin to the project
Go back to the Apache Maven Project and copy the code.
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.8.2</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M5</version>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.8.2</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
Again run the tests through command line using the command mentioned above. Now, will see that the tests are executed successfully.
Congratulations. We can see that JUnit5 tests are executed through command line. Happy Learning!!
JUnit5 Assertions Example
JUnit5 contains the assertions available as in JUnit 4 Assertions as well there are a few additional new asserts too. In this post, let’s discuss each new assertion in JUnit5 works in detail with examples.
1. assertIterableEquals
The assertIterableEquals() asserts that the expected and the actual iterables are deeply equal. In order to be equal, both iterable must return equal elements in the same order and it isn’t required that the two iterables are of the same type in order to be equal.
Example 1 – In this example, the number of elements as well as the sequence of elements is in the same order in both Iterables. It is not mandatory to have Iterables of the same type, so we can see as one of the Iterable is ArrayList whereas another Iteratble is of type LinkedList.
@Test
void iterableEqualsPositive() {
Iterable<String> iterat1 = new ArrayList<>(asList("Java", "Junit", "Test"));
Iterable<String> iterat2 = new LinkedList<>(asList("Java", "Junit", "Test"));
assertIterableEquals(iterat1, iterat2);
}
The assertion passes as the sequence and number of elements in both Iterables are the same.
Example 2 – In the below example, the ordering of elements in the Iterables is different.
@Test
void iterableEqualsNegative() {
Iterable<String> iterat1 = new ArrayList<>(asList("Java", "Junit", "Test"));
Iterable<String> iterat2 = new ArrayList<>(asList("Java","Test","Junit" ));
assertIterableEquals(iterat1, iterat2);
}
Here, we can see that the sequence of elements in Iterable are different, so the assertion fails.
Example 3 – In the below example, the number of elements is different in the Iterables. Iterable 1 has 3 elements whereas Iterable 2 has 4 elements.
@Test
void iterableEqualsNegative1() {
Iterable<String> iterat1 = new ArrayList<>(asList("Java", "Junit", "Test"));
Iterable<String> iterat2 = new LinkedList<>(asList("Java", "Junit", "Test","Junit5"));
assertIterableEquals(iterat1, iterat2);
}
As both Iterables do not have same number of elements, so the Assertion has failed.
Note:- There are no assertions like assertNotIterableEquals() or assertIterableNotEquals().
2. assertLinesMatch
This Assertion asserts that the expected list of Strings matches the actual list of String.
This method differs from other assertions that effectively only check String.equals(Object), in that it uses the following staged matching algorithm:
For each pair of expected and actual lines do
a) check if expected.equals(actual) – if yes, continue with next pair
b) otherwise treat expected as a regular expression and check via String.matches(String) – if yes, continue with the next pair
c) otherwise check if an expected line is a fast-forward marker, if yes apply to fast-forward actual lines accordingly (see below) and goto 1.
Example 1 – In the below example, expected has a regular expression that matches with the elements of actual.
@Test
void linesMatchPositive() {
List<String> expected = asList("Java", "\\d+", ".*");
List<String> actual = asList("Java", "11", "JUnit");
assertLinesMatch(expected, actual);
}
As the regular expression of elements matches, the assertion passes.
Example 2 – In the below example, the elements in the lists are different.
@Test
void linesMatchNegative1() {
List<String> expected = asList("Java", "\\d+", ".*");
List<String> actual = asList("Test","Java", "11");
assertLinesMatch(expected, actual);
}
The assertion fails as the elements are different in actual and expected.
Example 3 – In the below example, the number of elements is different in expected and actual lists.
@Test
void linesMatchNegative2() {
List<String> expected = asList("Java", "\\d+", ".*");
List<String> actual = asList("Java", "11");
assertLinesMatch(expected, actual);
}
The assertion fails as the number of elements in expected is 3 whereas 2 elements are available in actual list.
3. assertThrows
The new assertThrows() assertion allows us a clear and a simple way to assert if an executable throws the specified exception type.
Example 1 – In the below example, the length of string arr is null, so it throws a NullPointerException
@Test
void exceptionTestingPositive() {
String arr = null;
Exception exception = assertThrows(NullPointerException .class, () -> arr.length() );
assertEquals(null, exception.getMessage());
}
Result
Example 2 – In the below example, the exception thrown by String arr is NullPointerException. But, we are asserting it with ArithmeticException.
@Test
void exceptionTestingNegative() {
String arr = null;
Exception exception = assertThrows(ArithmeticException .class, () -> arr.length() );
assertEquals("Arithmetic Exception", exception.getMessage());
}
Result
4. assertTimeout
When we want to assert that execution of the supplied executable completes before the given timeout, we can use assertTimeout().
Example 1 – In the below example, assertTimeout() is 2 sec, which means the assertion should be completed within 2 secs. We are waiting for 1 sec and then perform the assertion.
@Test
void assertTimeoutPositive() {
int a = 4;
int b= 5;
assertTimeout(
ofSeconds(2),
() -> {
// code that requires less then 2 seconds to execute
Thread.sleep(1000);
}
);
assertEquals(9, (a + b));
}
As the assertion is within the specified time of assertTimeout(), the timeout assertion passes and the test passes.
Example 2 – In the below example, assertTimeout() is 2 sec whereas are waiting for 5 sec and then performing the assertion.
@Test
void assertTimeoutNegative() {
int a = 4;
int b= 5;
assertTimeout(
ofSeconds(2),
() -> {
// code that requires less then 2 seconds to execute
Thread.sleep(5000);
}
);
assertEquals(9, (a + b));
}
As the assertion is outside the specified time of assertTimeout(), so the test fails. The assertion fails with an error message similar to: “execution exceeded timeout of 2000 ms by 3010 ms”.
Example 3 – In the below example, the assertion is mentioned just after
The executable will be executed in the same thread as that of the calling code. Consequently, execution of the executable will not be preemptively aborted if the timeout is exceeded.
@Test
void assertTimeoutNegative1() {
int a = 4;
int b= 5;
assertTimeout(
ofSeconds(2),
() -> {
// code that requires less then 2 seconds to execute
Thread.sleep(5000);
assertEquals(10, (a + b));
}
);
}
This shows that the assertion assertEquals() is still executed after the timeout also.
5. assertTimeoutPreemptively()
This assertion works just like assertTimeout(). When we want to assert that the execution of the supplied executable completes before the given timeout, we can use assertTimeoutPreemptively(). The only difference is that here the executable will be executed in a different thread than that of the calling code, whereas in assertTimeout() the executable will be executed in the same thread as that of the calling code. Furthermore, execution of the executable will be preemptively aborted if the timeout is exceeded here as contrary to assertTimeout() where the executable will not be preemptively aborted.
@Test
void assertPreemptiveTimeoutNegative() {
int a = 4;
int b= 5;
assertTimeoutPreemptively(
ofSeconds(2),
() -> {
// code that requires less then 2 seconds to execute
Thread.sleep(5000);
assertEquals(9, (a + b));
}
);
}
Result
There is another very famous assertion called assertAll() [Group Assertions]. This is discussed in another tutorial.
In this post, We saw that JUnit Jupiter comes with many of the assertion methods that JUnit 4 has and adds a few that lend themselves well to being used with Java 8 lambdas. All JUnit Jupiter assertions are static methods in the org.junit.jupiter.api.Assertions class.
Grouped Assertions in JUnit 5 – assertAll()
Last Updated On
Table of Content
- What is Grouped Assertions in JUnit5?
- What is JUnit5?
- Types of AssertAll assertions
What is Grouped Assertions in JUnit5?
Grouped assertions in JUnit 5 help make your tests cleaner and more readable by allowing you to group related assertions together with a single failure message if any of them fail. This is particularly useful when you want to ensure that multiple conditions are true in a given test scenario.
JUnit 5 supports an additional Assertion feature called Grouped assertions. In JUnit4, when you have to assert let’s say – 3 different conditions, then you need to write 3 different assertions and they will be executed sequentially. Imagine, 2nd assertion fails, then the program stops there and will not go to the 3rd assertion. To overcome this problem, JUnit5 has a new assertion called Grouped assertion that execute all the conditions present within it using assertAll(), irrespective of the fact if any assertion fails or not and generate a consolidated report with all the failures. To be honest, I love this feature and use very frequently.
All JUnit Jupiter assertions are static methods. They come from class:
org.junit.jupiter.api.Assertions
What is JUnit5?
JUnit 5 is composed of several different modules from three different sub-projects.
JUnit 5 = JUnit Platform + JUnit Jupiter + JUnit Vintage
The JUnit Platform serves as a foundation for launching testing frameworks on the JVM. It also defines the Test Engine API for developing a testing framework that runs on the platform.
JUnit Jupiter is the combination of the new programming model and extension model for writing tests and extensions in JUnit 5. The Jupiter sub-project provides a Test Engine for running Jupiter-based tests on the platform.
JUnit Vintage provides a Test Engine for running JUnit 3 and JUnit 4 based tests on the platform. It requires JUnit 4.12 or later to be present on the classpath or module path.
To use JUnit5, add the Junit5 maven dependency to the POM.xml
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>511.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.11.4</version>
<scope>test</scope>
</dependency>
Types of AssertAll assertions
Grouped Assertions With Heading As Parameter
Grouped Assertions with Same Assertion Type
Example 1 – The following example demonstrates positive assertions using `assertAll()` in JUnit 5. All assertions in this example are of the same type, specifically `assertEquals()`, grouped within an `assertAll()` assertion. The heading parameter for this group of assertions is “GroupedAssertionsWithSameAssertionType”.
@Test
void allPositive1() {
assertAll(
"GroupedAssertionsWithSameAssertionType",
() -> assertEquals(8, 5+3, "8 is not sum of 5 and 3"),
() -> assertEquals("java", "JAVA".toLowerCase()),
() -> assertEquals(16,4*4,"16 is not product of 4 and 4")
);
}
Result
As all 3 assertions pass, so the final result passes.
Grouped Assertions with Different Assertion Types
Example 2 – The following example demonstrates using `assertAll()` in JUnit 5 to group assertions of different types – assertEquals(), assertNotNull and assertNotEquals() within a single test. It consists of the heading parameter with the value “GroupedAssertionsWithDifferentAssertionType”.
@Test
void allPositive2() {
String str ="Spring";
assertAll(
"GroupedAssertionsWithDifferentAssertionType",
() -> assertEquals(8, 5+3, "8 is not sum of 5 and 3"),
() -> assertNotNull(str, () -> "The string should be null"),
() -> assertEquals("java", "JAVA".toLowerCase()),
() -> assertNotEquals(20,5*3,"20 is product of 5 and 3")
);
}
Result
As all 3 assertions pass, so the final result passes.
Grouped Assertions with multiple failure
Example 3 – In the below example, out of 4 assertions, 3 assertions are failing, so the output will have the detail about all 3 assertion errors.
@Test
void allNegative() {
String str ="Spring";
Iterable<String> iterat1 = new ArrayList<>(asList("Java", "Junit4", "Test"));
Iterable<String> iterat2 = new ArrayList<>(asList("Java", "Junit5", "Test"));
assertAll(
"Negative-GroupedAssertionsWithDifferentAssertionType",
() -> assertIterableEquals(iterat1, iterat2),
() -> assertNull(str, () -> "The string should be null"),
() -> assertEquals("java", "JAVA"),
() -> assertNotEquals(20,5*3,"20 is product of 5 and 3")
);
}
Result
As one of the asserts in the group fails, instead of AssertionFailureError it results in MultipleFailuresError thereby displaying the heading of the grouped assertion passed as the input parameter i.e. Negative-GroupedAssertionsWithDifferentAssertionType in this example. This image shows all the 3 assertion failures.
Assertion 1 fails as we were expecting JUnit4, but response has JUnit5
Assertion 2 fails as the string was not NULL.
Assertion 3 fails as Java is not equal to JAVA (case sensitivity).
Grouped Assertions Without Heading As Parameter
Grouped Assertions with Same Assertion Type
The assertAll () can be implemented without using the heading parameter. The below example is the same as the above one, we are just skipping the heading part.
@Test
void allNegative() {
String str ="Spring";
Iterable<String> iterat1 = new ArrayList<>(asList("Java", "Junit4", "Test"));
Iterable<String> iterat2 = new ArrayList<>(asList("Java", "Junit5", "Test"));
// In a grouped assertion all assertions are executed, and all failures will be reported together
assertAll(
() -> assertIterableEquals(iterat1, iterat2),
() -> assertNull(str, () -> "The string should be null"),
() -> assertEquals("java", "JAVA"),
() -> assertNotEquals(20,5*3,"20 is product of 5 and 3")
);
}
Result
The result displays without a heading.
Nested Or Dependent Grouped Assertions
Example with Outer assertAll() Passing and Nested assertAll() Executing
When one assertAll() includes one or more assertAll() then these are referred to as Nested or Dependent grouped assertions.
Example 1 – In this case, first, the assertAll() validates if the sum is correct or not. The sum is correct here, so the control moves to the nested assertAll() to verify all the assertions present within it.
@Test
void allDependentPositive() {
String str ="Spring";
// Within a code block, if an assertion fails the subsequent code in the same block will be skipped.
assertAll(
"DependentPositiveAssertions",
() -> {
assertEquals(8, 5 + 3, "8 is not sum of 5 and 3");
// Executed only if the previous assertion is valid.
assertAll("sub-heading",
() -> assertNotNull(str, () -> "The string should be null"),
() -> assertEquals("java", "JAVA".toLowerCase()),
() -> assertEquals(20, 5 * 4, "20 is product of 5 and 4")
); // end of inner AssertAll()
}
); // end of outer AssertAll()
}
Result
All the assertions within nested assertAll() are passes. So the final result passes.
Example of Outer AssertAll() Causing Nested AssertAll() Not to Execute
Example 2 – In the below example, outer AssertAll() fails, so all the assertions within nested/dependent assertAll() are not executed.
@Test
void allDependentNegative() {
String str ="Spring";
// Within a code block, if an assertion fails the subsequent code in the same block will be skipped.
assertAll(
"DependentPositiveAssertions",
() -> {
assertEquals(8, 5 + 4, "8 is not sum of 5 and 3");
// Executed only if the previous assertion is valid.
assertAll("sub-heading",
() -> assertNull(str, () -> "The string should be null"),
() -> assertNotEquals("java", "JAVA".toLowerCase()),
() -> assertNotEquals(20, 5 * 4, "20 is product of 5 and 4")
); // end of inner AssertAll()
}
); // end of outer AssertAll()
}
Result
Example of Outer AssertAll() allows Nested AssertAll(), but nested AssertAll fails
Example 3 – In the below example, outer AssertAll() passes, so all the assertions within nested/dependent assertAll() are executed. But due to the failure of assertNull, the nested assertions are not evaluated, and the report will indicate the failure of `assertNull` first
@Test
void allDependentNegative1() {
String str ="Spring";
// Within a code block, if an assertion passes the subsequent code in the same block will be executed.
assertAll(
"DependentNegativeAssertions",
() -> {
assertEquals(8, 5 + 3, "8 is not sum of 5 and 3");
// Executed only if the previous assertion is valid.
assertAll("sub-heading",
() -> assertNull(str, () -> "The string should be null"),
() -> assertNotEquals("java", "JAVA".toLowerCase()),
() -> assertNotEquals(20, 5 * 4, "20 is product of 5 and 4")
); // end of inner AssertAll()
}
); // end of outer AssertAll()
}
Result
The nested assertions have failed. So they can be seen in the execution status.
In short,
- When first assertAll() method passes, then all the subsequent assertions within that block will be executed and these assertions can further pass or fails.
- When the first assertAll() assertion fails, then the execution of subsequent assertions is skipped.
That’s it! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
Manual Tests in Serenity with JUnit5
In this tutorial, I will explain about Manual Tests in Serenity JUnit5.
You can annotate @Test not @Steps as @Manual.
In contrast to Junit4 a test method annotated with @Manual will actually be executed. This allows to further specify the example using @Step methods and show them the report.
Below is an example where tests are annotated with @Manual with description.
@SerenityTest
public class LoginTests {
@Managed
WebDriver driver;
@Steps
StepLoginPage loginPage;
@Steps
StepDashboardPage dashPage;
@Steps
StepForgetPasswordPage forgetpasswordPage;
@Test
@Title("Login to application should be successful")
public void sucessfulLogin() {
// Given
loginPage.open();
// When
loginPage.inputUserName("Admin");
loginPage.inputPassword("admin123");
loginPage.clickLogin();
// Then
dashPage.loginVerify();
}
@Test
@Title("Login to application should be unsuccessful with error message")
public void unsucessfulLogin() throws InterruptedException {
// Given
loginPage.open();
// When
loginPage.inputUserName("abc");
loginPage.inputPassword("abc12");
loginPage.clickLogin();
// Then
String actualErrorMessage = loginPage.errorMessage();
assertEquals("Invalid credentials", actualErrorMessage);
}
@Test
@Manual
void manualDefault() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.SUCCESS)
void manualSuccess() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.COMPROMISED)
void manualCompromised() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.ERROR)
void manualError() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.ERROR, reason = "A reason for the error")
void manualErrorWithReason() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.FAILURE)
void manualFailure() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.IGNORED)
void manualIgnored() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.PENDING)
void manualPending() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.SKIPPED)
void manualSkipped() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.UNDEFINED)
void manualUndefined() {
loginPage.manualStep();
}
@Test
@Manual(result = TestResult.UNSUCCESSFUL)
void manualUnsuccessful() {
loginPage.manualStep();
}
}
StepLoginPage.java
public class StepLoginPage extends PageObject {
@FindBy(name = "txtUsername")
WebElementFacade username;
@FindBy(name = "txtPassword")
WebElementFacade password;
@FindBy(name = "Submit")
WebElementFacade submitButton;
@FindBy(id = "spanMessage")
WebElementFacade errorMessage;
@FindBy(id = "forgotPasswordLink")
WebElementFacade linkText;
@Step("Enter Username")
public void inputUserName(String userName) {
username.sendKeys((userName));
}
@Step("Enter Password")
public void inputPassword(String passWord) {
password.sendKeys((passWord));
}
@Step("Click Submit Button")
public void clickLogin() {
submitButton.click();
}
@Step("Error Message on unsuccessful login")
public String errorMessage() {
String actualErrorMessage = errorMessage.getText();
System.out.println("Actual Error Message :" + actualErrorMessage);
return actualErrorMessage;
}
@Step("Manual Test Step")
public void manualStep() {
System.out.println("Verify various status of manual step");
}
}
StepDashboardPage.java
public class StepDashboardPage extends PageObject {
@FindBy(id = "welcome")
WebElementFacade dashboardText;
@Step("Successful login")
public void loginVerify() {
String dashboardTitle = dashboardText.getText();
assertThat(dashboardTitle, containsString("Welcome"));
}
}
Execute these tests by using the below command in commandline.
mvn clean verify
There are two automated tests and rest all are Manual tests. We have Manual Test marked as Default, SUCCESS, COMPROMISED, ERROR, FAILURE, IGNORED, PENDING, SKIPPED, UNDEFINED and UNSUCCESSFUL.
The execution status looks like as shown below.
The reports are generated under /target/site/serenity. There are 2 types of Reports are generated – index.html and serenity-summary.html. To know how to generate Serenity Reports, please refer tutorials for index.html and serenity-summary.html.
By default, @manual scenarios are marked as pending in the Serenity reports.
All scenarios highlighted by blue color are Pending ones whereas pink color are Broken ones.
Serenity-Summary.html
We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
Allure Report with Selenium and JUnit5
Last Updated On
In the previous tutorial, I explained the Integration of the Allure Report with Selenium and JUnit4. In this tutorial, I will explain how to Integrate Allure Report with Selenium and JUnit5.
Table of Contents
Prerequisite
- Java 11 is installed
- Maven is installed
- Eclipse or IntelliJ is installed
- Allure is installed
Dependency List:
- Selenium – 3.141.59
- Java 11
- JUnit – 4.13.2
- Maven – 3.8.1
- Allure Report – 2.14.0
- Allure JUnit4 – 2.14.0
Structure of Project
Step 1 – Update Properties section in Maven pom.xml
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>11</java.version>
<selenium.version>3.141.59</selenium.version>
<junit.jupiter.version>5.8.0-M1</junit.jupiter.version>
<junit.platform.version>1.8.0-M1</junit.platform.version>
<allure.maven.version>2.10.0</allure.maven.version>
<allure.junit5.version>2.14.0</allure.junit5.version>
<maven.surefire.plugin.version>3.0.0-M3</maven.surefire.plugin.version>
<maven.compiler.plugin.version>3.8.1</maven.compiler.plugin.version>
<aspectj.version>1.9.6</aspectj.version>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
Step 2 – Add Selenium, JUnit5, and Allure-JUnit5 dependencies in POM.xml
<dependencies>
<!--Selenium Dependency-->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>${selenium.version}</version>
</dependency>
<!--JUNIT 5 Dependencies-->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit.jupiter.version}</version>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>${junit.jupiter.version}</version>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-engine</artifactId>
<version>${junit.platform.version}</version>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-launcher</artifactId>
<version>${junit.platform.version}</version>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-runner</artifactId>
<version>${junit.platform.version}</version>
</dependency>
<!--Allure Reporting Dependencies-->
<dependency>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-junit5</artifactId>
<version>${allure.junit5.version}</version>
</dependency>
</dependencies>
Step 3 – Update the Build Section of pom.xml in Allure Report Project
<build>
<plugins>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven.surefire.plugin.version}</version>
<configuration>
<testFailureIgnore>false</testFailureIgnore>
<argLine>
-javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
</argLine>
<systemProperties>
<property>
<name>junit.jupiter.extensions.autodetection.enabled</name>
<value>true</value>
</property>
</systemProperties>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>io.qameta.allure</groupId>
<artifactId>allure-maven</artifactId>
<version>${allure.maven.version}</version>
<configuration>
<reportVersion>2.4.1</reportVersion>
</configuration>
</plugin>
</plugins>
</build>
Step 4 – Create Pages and Test Code for the pages
Below is the sample project which uses Selenium and JUnit4 which is used to generate an Allure Report.
We have 2 pages. Below is the code for Login Page which contains all the web elements and methods related to that web elements.
Note:- This is a sample code. There could be the probability that XPath would have changed. So, the tests won’t run as expected and please keep this in mind.
public class LoginPage {
WebDriver driver;
By userName = By.name("txtUsername");
By password = By.name("txtPassword");
By titleText = By.id("logInPanelHeading");
By login = By.id("btnLogin");
By errorMessage = By.id("spanMessage");
public LoginPage(WebDriver driver) {
this.driver = driver;
}
// Set user name in textbox
public void setUserName(String strUserName) {
driver.findElement(userName).sendKeys(strUserName);
}
// Set password in password textbox
public void setPassword(String strPassword) {
driver.findElement(password).sendKeys(strPassword);
}
// Click on login button
public void clickLogin() {
driver.findElement(login).click();
}
@Step("Verify title of Login Page")
public void verifyPageTitle() {
String loginPageTitle = driver.findElement(titleText).getText();
assertTrue(loginPageTitle.contains("LOGIN Panel"));
}
/* Failed Test */
@Step("Verify error message when invalid credentail is provided")
public void verifyErrorMessage() {
String invalidCredentialErrorMessage = driver.findElement(errorMessage).getText();
assertTrue(invalidCredentialErrorMessage.contains("Incorrect Credentials"));
}
@Step("Enter username and password")
public void login(String strUserName, String strPasword) {
// Fill user name
this.setUserName(strUserName);
// Fill password
this.setPassword(strPasword);
// Click Login button
this.clickLogin();
}
}
assertTrue() is imported from the below JUnit package for assertion.
import static org.junit.jupiter.api.Assertions.assertTrue;
DashboardPage.java
public class DashboardPage {
WebDriver driver;
By dashboardPageTitle = By.id("welcome");
By assignLeaveOption = By.cssSelector(
"#dashboard-quick-launch-panel-menu_holder > table > tbody > tr > td:nth-child(1) > div > a > span");
By leaveListOption = By.cssSelector(
"#dashboard-quick-launch-panel-menu_holder > table > tbody > tr > td:nth-child(2) > div > a > span");
By timesheetsOption = By.cssSelector(
"#dashboard-quick-launch-panel-menu_holder > table > tbody > tr > td:nth-child(3) > div > a > span");
By applyLeaveOption = By.cssSelector(
"#dashboard-quick-launch-panel-menu_holder > table > tbody > tr > td:nth-child(4) > div > a > span");
public DashboardPage(WebDriver driver) {
this.driver = driver;
}
@Step("Verify title of Dashboard page")
public void verifyDashboardPageTitle() {
String DashboardPageTitle = driver.findElement(dashboardPageTitle).getText();
assertTrue(DashboardPageTitle.contains("Welcome"));
}
@Step("Verify Assign Leave Quick Launch Options on Dashboard page")
public void verifyAssignLeaveOption() {
String QuickLaunchOptions = driver.findElement(assignLeaveOption).getText();
assertTrue(QuickLaunchOptions.contains("Assign Leave"));
}
@Step("Verify Leave List Quick Launch Options on Dashboard page")
public void verifyLeaveListOption() {
String LeaveListQuickLaunchOption = driver.findElement(leaveListOption).getText();
assertTrue(LeaveListQuickLaunchOption.contains("Leave List"));
}
@Step("Verify Assign Leave Quick Launch Options on Dashboard page")
public void verifytimesheetsOption() {
String timesheetsOptionQuickLaunchOption = driver.findElement(timesheetsOption).getText();
assertTrue(timesheetsOptionQuickLaunchOption.contains("Timesheets"));
}
@Step("Verify Leave List Quick Launch Options on Dashboard page")
public void verifyApplyLeaveOption() {
String applyLeaveQuickLaunchOptions = driver.findElement(applyLeaveOption).getText();
assertTrue(applyLeaveQuickLaunchOptions.contains("Apply Leave"));
}
}
Test Classes related to various Pages
BaseTest.java
public class BaseTest {
public static WebDriver driver;
LoginPage objLogin;
DashboardPage objDashboardPage;
@Step("Start the application")
@BeforeEach
public void setup() {
System.setProperty("webdriver.gecko.driver",
"C:\\Users\\Vibha\\Software\\geckodriver-v0.26.0-win64\\geckodriver.exe");
driver = new FirefoxDriver();
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.get("https://opensource-demo.orangehrmlive.com/");
}
@Step("Stop the application")
@AfterEach
public void close() {
driver.close();
}
}
@BeforeEach is used to signal that the annotated method should be executed before each @Test, @RepeatedTest, @ParameterizedTest, @TestFactory, and @TestTemplate method in the current test class. It is imported from:-
import org.junit.jupiter.api.BeforeEach;
@AfterEach is used to signal that the annotated method should be executed after each @Test, @RepeatedTest, @ParameterizedTest, @TestFactory, and @TestTemplate method in the current test class. It is imported from:-
import org.junit.jupiter.api.AfterEach;
LoginTests.java
@Epic("Web Application Regression Testing using JUnit5")
@Feature("Login Page Tests")
public class LoginTests extends BaseTest {
LoginPage objLogin;
DashboardPage objDashboardPage;
@Severity(SeverityLevel.NORMAL)
@Test
@Description("Test Description : Verify the title of Login Page")
@Story("Title of Login Page")
public void verifyLoginPage() {
// Create Login Page object
objLogin = new LoginPage(driver);
// Verify login page text
objLogin.verifyPageTitle();
}
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Login Test with invalid credentials")
@Story("Unsuccessful Login to Application")
public void invalidCredentialTest() {
// Create Login Page object
objLogin = new LoginPage(driver);
objLogin.login("test", "test123");
// Verify login page text
objLogin.verifyErrorMessage();
}
}
DashboardTests.java
package com.example.Junit5AllureReportDemo.tests;
import org.junit.jupiter.api.Test;
import com.example.Junit5AllureReportDemo.pages.DashboardPage;
import com.example.Junit5AllureReportDemo.pages.LoginPage;
import io.qameta.allure.Description;
import io.qameta.allure.Epic;
import io.qameta.allure.Feature;
import io.qameta.allure.Severity;
import io.qameta.allure.SeverityLevel;
import io.qameta.allure.Story;
@Epic("Web Application Regression Testing using JUnit5")
@Feature("Dashboard Page Tests")
public class DashboardTests extends BaseTest {
LoginPage objLogin;
DashboardPage objDashboardPage;
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Verify title of Dashboard page")
@Story("Title of Dashboard Page")
public void dashboardTitleTest() {
objLogin = new LoginPage(driver);
// login to application
objLogin.login("Admin", "admin123");
// go the dashboard page
objDashboardPage = new DashboardPage(driver);
objDashboardPage.verifyDashboardPageTitle();
}
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Verify Assign Leave Option in Quick Link Menu")
@Story("Validation of Assign Leave Option")
public void assignLeaveOptionTest() {
objLogin = new LoginPage(driver);
// login to application
objLogin.login("Admin", "admin123");
// go the dashboard page
objDashboardPage = new DashboardPage(driver);
objDashboardPage.verifyAssignLeaveOption();
}
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Verify Apply Leave Option in Quick Link Menu")
@Story("Validation of Apply Leave Option")
public void applyLeaveOptionTest() {
objLogin = new LoginPage(driver);
// login to application
objLogin.login("Admin", "admin123");
// go the dashboard page
objDashboardPage = new DashboardPage(driver);
objDashboardPage.verifyApplyLeaveOption();
}
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Verify Leave List Option in Quick Link Menu")
@Story("Validation of Leave List Option")
public void leaveListOptionTest() {
objLogin = new LoginPage(driver);
// login to application
objLogin.login("Admin", "admin123");
// go the dashboard page
objDashboardPage = new DashboardPage(driver);
objDashboardPage.verifyLeaveListOption();
}
@Severity(SeverityLevel.BLOCKER)
@Test
@Description("Test Description : Verify Timesheets Option in Quick Link Menu")
@Story("Validation of Timesheets Option")
public void timesheetsOptionTest() {
objLogin = new LoginPage(driver);
// login to application
objLogin.login("Admin", "admin123");
// go the dashboard page
objDashboardPage = new DashboardPage(driver);
objDashboardPage.verifyTimesheetsOption();
}
}
Step 5 – Run the Test and Generate Allure Report
To run the tests, use the below command
mvn clean test
In the below image, we can see that one test failed and six passed out of seven tests.
This will create the allure-results folder with all the test reports. These files will be used to generate Allure Report.
To create Allure Report, use the below command
allure serve
This will generate the beautiful Allure Test Report as shown below.
Allure Report Dashboard
Categories in Allure Report
The categories tab gives you a way to create custom defect classifications to apply for test results. There are two categories of defects – Product Defects (failed tests) and Test Defects (broken tests).
Suites in Allure Report
On the Suites tab a standard structural representation of executed tests, grouped by suites and classes can be found.
Graphs in Allure Report
Graphs allow you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline in Allure Report
Timeline tab visualizes retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab they are arranged accordingly to their sequential or parallel timing structure.
Behaviors of Allure Report
This tab groups test results according to Epic, Feature, and Story tags.
Packages in Allure Report
Packages tab represents a tree-like layout of test results, grouped by different packages.
We are done! Congratulations on making it through this tutorial and hope you found it useful! Happy Learning!!
QA Automation Expert 