vibssingh

Run Selenium Tests in GitLab CI/CD

Last Updated on

HOME

This tutorial explains the process to run the Selenium Tests in the GitLab pipeline. This is a very important step towards achieving CI/CD. Ideally, the tests need to run after any change (minor/major) before merging the latest change to the master branch. Suppose there are 100 changes merged to the master branch in a day and it is expected to run the tests every time before deployment. In this case, any QA won’t want to start the tests manually 100 times in a day. Now, what should be done to overcome this problem. Now, adding tests to the GitLab pipeline comes into the picture. We can add a test stage to the pipeline and the tests will run automatically when the pipeline run.

Table of Contents

  1. Prerequisite
  2. What is GitLab CI/CD Workflow?
  3. What is a headless browser?
  4. Implementation Steps
    1. Create a new Maven Project
    2. Add the dependencies to the POM.xml
    3. Create the Test Code
    4. Create testng.xml to run the tests
    5. Run the tests through command line
  5. GitLab Section
    1. Create a blank project in GitLab
    2. Push the project from local repository to Gitlab Repository
    3. Create a .gitlab-ci.yml file in the project in GitLab
    4. Run the tests in the GitLab pipeline
    5. Check the status of the pipeline
    6. Download the report

Prerequisite

  1. Selenium
  2. TestNG/JUnit (for Assertions)
  3. Java 11
  4. Maven/ Gradle
  5. GitLab Account

What is GitLab CI/CD Workflow?

Once the proposed changes are built, then push the commits to a feature branch in a remote repository that’s hosted in GitLab. The push triggers the CI/CD pipeline for your project. Then, GitLab CI/CD runs automated scripts (sequentially or in parallel) to build as well as to test the application. After a successful run of the test scripts, GitLab CI/CD deploys your changes automatically to any environment (DEV/QA/UAT/PROD). But if the test stage is failed in the pipeline, then the deployment is stopped.

After the implementation works as expected:

  • Get the code reviewed and approved.
  • Merge the feature branch into the default branch.
    • GitLab CI/CD deploys your changes automatically to a production environment.

To use GitLab CI/CD, we need to keep 2 things in mind:

a) Make sure a runner is available in GitLab to run the jobs. If there is no runner, install GitLab Runner and register a runner for your instance, project, or group.

b) Create a .gitlab-ci.yml file at the root of the repository. This file is where CI/CD jobs are defined.

The Selenium tests run on a headless browser in the pipeline.

What is a headless browser?

A headless browser is like any other browser, but without a Head/GUI (Graphical User Interface).  A headless browser is used to automate the browser without launching the browser. While the tests are running, we could not see the browser, but we can see the test results coming on the console.

To explain, I have created 2 Selenium tests and used TestNG for asserting the tests. The tests will run on a headless Chrome browser. One more thing to keep in mind is that when tests run on a headless Chrome browser, the window screen won’t be full screen. So, the screen needs to be maximized explicitly otherwise some of the tests would fail.

Implementation Steps

Step 1 – Create a new Maven Project

Step 2- Add the dependencies to the POM.xml

Add the below-mentioned dependencies that need to add to the project to the pom.xml in Maven Project.

 <dependencies>

        <dependency>
            <groupId>io.github.bonigarcia</groupId>
            <artifactId>webdrivermanager</artifactId>
            <version>5.1.0</version>
        </dependency>

        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>3.141.59</version>
        </dependency>

        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>7.5</version>
            <scope>test</scope>
        </dependency>
                   
    </dependencies>
    <build>
     <plugins>
     
     <!--  Compiler Plugin -->
    
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
           <source>11</source>
           <target>11</target>
        </configuration>
    </plugin>
     
      <!--  Plugin used to execute tests -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M5</version>
        <configuration>
          <suiteXmlFiles>
            <suiteXmlFile>testng.xml</suiteXmlFile>
          </suiteXmlFiles>
        </configuration>
      </plugin>
     </plugins>
  </build>
</project>

As explained in one of the previous tutorial, it is needed to add the maven-surefire-plugin to run the TestNG tests through the command line.

Step 3 – Create the Test Code

This is the BaseTest Class where the WebDriver is initialized, headless mode, full screen, and at the end close the WebDriver.

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeMethod;

import io.github.bonigarcia.wdm.WebDriverManager;

public class BaseTest {
	
	WebDriver driver;
	
	@BeforeMethod
	public void beforeTests() {
        WebDriverManager.chromedriver().setup();
        ChromeOptions options=new ChromeOptions();
        options.setHeadless(true);
        options.addArguments("window-size=1920,1200");
        driver=new ChromeDriver(options);
        driver.get("https://opensource-demo.orangehrmlive.com/");
    }

	@AfterMethod
    public void afterTests() {
        driver.quit();
    }

}

There are 2 different pages that need to be tested – LoginPage and ForgetPasswordPage

LoginPage contains the tests to log in to the application. After successful login, the application moves to the next webpage – HomePage. You can see that BaseTest class is extended in both the Test classes.

import org.openqa.selenium.By;
import org.testng.Assert;
import org.testng.annotations.Test;

@Test
public class LoginPage extends BaseTest{
	
	@Test
	public void validCredentials() throws InterruptedException {

	        driver.findElement(By.name("txtUsername")).sendKeys("Admin");	        
	        driver.findElement(By.name("txtPassword")).sendKeys("admin123");
	        driver.findElement(By.id("btnLogin")).click();
	         String newPageText = driver.findElement(By.xpath("//*[@id='content']/div/div[1]/h1")).getText();
	        System.out.println("newPageText :" + newPageText);
	        Assert.assertEquals(newPageText,"Dashboard");

	}

}

ForgetPasswordPage

import org.openqa.selenium.By;
import org.testng.Assert;
import org.testng.annotations.Test;

@Test
public class ForgetPasswordPage extends BaseTest{
	
	@Test
	public void getHeading() {
		
		Thread.sleep(1000);
		driver.findElement(By.xpath("//*[@id='forgotPasswordLink']/a")).click();
		String heading = driver.findElement(By.xpath("//*[@id='content']/div[1]/div[2]/h1")).getText();
		System.out.println("Title : "+ heading );
		Assert.assertEquals(heading, "Forgot Your Password?");
	}
	
}

Step 4 – Create testng.xml to run the tests

Now, let’s create a testng.xml to run the TestNG tests. If JUnit is used instead of TestNG, then this step is not needed.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
  <test name="Test">
    <classes>
      <class name="org.example.DockerDemo.LoginPage"/>
      <class name="org.example.DockerDemo.ForgetPasswordPage"/>
    </classes>
  </test> <!-- Test -->
</suite> <!-- Suite -->

Step 5 – Run the tests through command line

Now, let us execute the tests through the command line. Go to the place where the pom.xml of the project is placed and use the below command to run the tests. This step makes sure that all the tests are running as expected.

mvn compile test

GitLab Section

Step 6 – Create a blank project in GitLab

To know, how to create a blank new project in GitLab, please refer to this tutorial.

Step 7 – Push the project from local repository to Gitlab Repository

To know, how to push the changes in GitLab, please refer to this tutorial.

Step 8 – Create a .gitlab-ci.yml file in the project in GitLab

There are many ways to create a new file in GitLab. One of the ways is to create a file as shown in the below image.

It is a YAML file where you configure specific instructions for GitLab CI/CD. In the .gitlab-ci.yml, we can define:

  • The scripts you want to run.
  • Other configuration files and templates you want to include.
  • Dependencies and caches.
  • The commands you want to run in sequence and those you want to run in parallel.
  • The location to deploy your application to.
  • Whether you want to run the scripts automatically or trigger any of them manually.
image: markhobson/maven-chrome

stages:
  - test

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"

test:
  stage: test
  allow_failure: true

# Run the tests  
  script:
    - mvn $MAVEN_OPTS clean package
    - mvn compile test

# Store artifacts
  artifacts:
    when: always
    name: "report"
    paths:
    - target/surefire-reports/*
    expire_in: 1 h

Image – markhobson/maven-chrome is used in this test. It is a docker image for Java automated UI tests.

This gitlab-ci.yml has only 1 stage – a test that contains the command to run the tests as well as also create an artifact that contains all the surefire reports which can be saved as Test Evidence.

Step 9 – Run the tests in the GitLab pipeline

Now, when a new change is committed, a pipeline kicks off and it runs all the tests.

Step 10 – Check the status of the pipeline

Once the Status of the pipeline changes to either failed or passed.. that means the tests are already executed.

As you can see the Status is failed here which means that the execution is completed. Let us see the logs of the execution it shows that out of 2 tests, 1 test passed and 1 test failed. This shows that tests are running successfully in the GitLab pipeline.

As I have added an artifact also in the gitalb-ci.yml, which is highlighted in the image. This artifact creates a folder with the name “report” and the reports in this folder come from the path /target/surefire-reports. This artifact gives us the option to download the reports or browse the report. This report will be available for 1 hour only as mentioned in the gitlab-ci.yml.

Step 11 – Download the report

Once, will click on the download button, it will download “report.zip”. Unzip the folder and it looks like something as shown below:

Example of Emailable-Report.html

Example of Index.html

Congratulations. This tutorial has explained the steps to run Selenium tests in GitLab CI/CD. Happy Learning!!

Posted in GitLabTagged , , , , 36 Comments

36 thoughts on “Run Selenium Tests in GitLab CI/CD”

  1. Nice. Very detailed steps. Solved my problem of running my selenium tests in CI/CD. Can I run my API tests also in CI/CD?

    Like

    Reply

  2. Hi, I encounter this error when uploading artifacts.
    WARNING: target/surefire-reports/*: no matching files. Ensure that the artifact path is relative to the working directory
    ERROR: No files to upload

    Please help

    Like

    Reply

      1. Hi Saurabh, if you are getting this error – WARNING: target/surefire-reports/*: no matching files. Ensure that the artifact path is relative to the working directory, then I suggest you to look for that path of report in Gitlab run console. Go to the end of the logs and you will see something like – Please refer to/builds/….. and here you will see that path where the report is saved. Use this path in Artifacts.

        Like

  4. Hi Vibha, i came across your post and found it really informative, however, i have a few queries. At the moment, I have developed a Java-Selenium-Automation Project using Maven and have created a remote repo on GitLab. I later pushed the project from local repo to Remote repo .Now, i am clueless regarding running jobs/pipeline using the gitlab.yml file as i am not sure how to configure it in the first place. earlier, I used to work on GitHub and Jenkins for CI/CD purpose but I am quite new to Gitlab and not sure how to configure stuff here. Also, i tried running the “mvn clean test” command in my local but it said “Build Failure”,Is this because i have not set MAVEN_HOME is Path. Please do guide me. Kindly, Let me know if any further action is required from my side.

    Like

    Reply

    1. I appreciate you found this article informative. The first thing is, your project should be able to run from the command line using Maven locally. Setup Maven properly. If you want to know how, refer this – https://qaautomation.expert/2019/08/24/how-to-install-maven-on-windows/. To run the tests in Gitlab pipeline, the tests should run in headless mode. Once these 2 things are done, go to step 8 of the article, that it shows how to create gitlab-ci.yml file. Commit and push this file to GitLab, this will start the pipeline.

      Like

      Reply

      1. Firstly, I am completely bowled over with the way you swiftly gave your feedback and solution pertaining to the complexities i faced during installation. Secondly, I followed all the action points provided earlier and will be mentioning the, here along with a few Key observations and Queries.

        Steps Undertaken:
        1.Downloaded Maven Binary file
        2.Configured the MAVEN_HOME Variable in System Pariables
        3.Configured $MAVEN_HOME/Bin in Path Variables
        4.Verified Maven installation using mvn -version which gave me the version number along with other details
        5.Ran the Selenium test locally using mvn clean, mvn test, mvn install commands
        6.Set the chromeOptions configuration to Headless=true
        7.Executed step 5 again to verify if maven executes the test locally and it gave me “BUILD SUCCESS”
        8.Later i pushed the latest changes from local repo to Remote repo on Gitlab
        9.Edited the -gitlab-ci.yml file in Gitlab with the settings provided in your document
        10.Triggered a pipeline

        Key Observations:
        1.Pipeline gave me a result “Test passed with Warnings”
        2.Job result was “Build Failure” because of FileNotFoundException and actually this file was a config.properties file i used to just store the values of the browser and username in my POM design.So, i removed this step from my selenium code and hardcoded the values of browser and username rather than implementing the values using the config.properties file.Even after doing this step ,i got the same filenotfoundexception when i retried the job.

        Queries:
        1.I am really curious to know how should i custom configure the gitlab.yml file to just execute selected suites
        2.How can i set the jobs on gitlab using the gitlab.yml file if i have multiple testng.xml files with different suite collections
        3.If a selenium test gets executed successfully using maven via command line ,why does it fail in Gitlab even if they both have same configuration

        BONUS:
        Thanks for accepting my connect request on LinkedIn ( Truly nothing less than a FanBoy Moment for me!)

        Like

      2. Hi, I’ll try my best to answer to your questions.
        1. Answer to your 1st and 2nd point is create different testng.xml file with different test suites, and you can run it as make the suiteXmlFile options parameterized as shown below

        org.apache.maven.plugins
        maven-surefire-plugin
        3.0.0-M5

        ${SuiteXmlFile}

        and pass mvn compile test -DSuiteXmlFile=mytestng1.xml or whatever is the name in the gitlab-ci.yml file. In this case you are dynamically running the testng.xml

        3. Selenium tests will run both locally and gitlab, but locally you are using windows while the pipeline has linux os. So, if you are using \\src\\test\\resources\\propertyFiles\\project.properties to run the tests locally, in case of pipeline you need to change it to something like “//src//test//resources//propertyFiles//project.properties”.

        I would suggest to run the job without artifact option to just see if the tests are running fine or not. There can be chances that the tests ahave run, but artifact is unable to find the report, so the build is failing.

        Like

  5. Thanks for the Feedback, I took the steps you provided into consideration and had the following outcome:
    1. I parameterized the SuiteXmlFIle so i could run it dynamically on GitLab,but apparently i was able to run it successfully in my command line via Maven locally using mvn compile test -DSuiteXmlFIle=testng.xml however when i ran the same command using gitlab-ci.yml as a gitlab job ,i got the following error “[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-test) on project MyProject: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0:test failed: testSuiteXmlFiles0 has null value -> [Help 1] ” .I tried changing the values of the surefire plugin versions but got the same error.I also tried using the command -Dsurefire.SuiteXmlFIle=testng.xml but again got the same error.Again, I also removed the artifact step from Gitlab-ci.yml file but still same issue.
    2.The issue previously stated in an earlier comment regarding FIleNotFoundException during Build Failure on Gitlab job execution was due to the config.properties file, so I changed the path from WIndows path to Linux OS path but still faced error so I completely removed it along with ReadConfig Utility class and then the job was successfully executed on GitLab, but this was also because i did not parametrize the SuiteXmlFIle value but just kept it as testng.xml in the pom.xml file and just used “maven compile test” in Gitlab-ci.yml file in GitLab. Please do let me know what action can be undertaken at this point ,this issue has been literally draining me.

    Like

    Reply

    1. For parameterized issue, instead of parameterizing the testng.xml in pom.xml, add all the testng1.xml, testng2.xml files in surefire configuration.


      org.apache.maven.plugins
      maven-surefire-plugin
      3.0.0-M7

      testng1.xml
      testng2.xml

      Use this in gitlab-ci.yml – mvn compile test -Dsurefire.suiteXmlFiles=”testng1.xml” (or whatever testng.xml) you want to use. This should work.

      Like

      Reply

      2. Hey ,Just an Update. Sorry for the delayed response .It did workout after i added all the suites in the pom.xml file and selected the ones which were required while designing the gitlab.yml file.I just have a couple of queries not related to the current question but knew only you would be able to deliver a response:
        1.Integrating Appium Scripts with Gitlab CI/CD
        2.When does one get 200 while sending a POST request(and can also view the response body in JSON Format) but when I used GET request to retrieve all the records I am unable to see the record created in POST request

        Like

      3. I’m glad that this solution worked out for you. Regarding your first question, I have limited knowledge of Appium, so I can’t help with CI/CD. The answer to the second question is, when you use a public API and perform post-operation and later perform GET operation, then most of these public APIs don’t save the newly created record as it is very expensive (you need to save the new record in some DB). But, if it is a private API, then the POST operation should create a new record and should be accessed later by the GET operation.

        Like

  6. Hello. thanks for the explanation. I did everything you mentioned but getting an error regarding chromedriver. On gitlab it is not able to find the chromedriver. Locally I am using Mac and setting up the path using system.setproperty. Not sure how to configure the same for gitlab?

    Like

    Reply

    1. I appreciate that you found this article useful. As you are setting up the browser path using system.setproperty means locally, you have the driver on your machine and use it to run the tests. We can’t run the tests in GitLab like this. I would suggest you mention webdrivermanager dependency in your pom.xml and make the changes in test as shown in @@BeforeMethod in the tutorial. This will eliminate your need to have the driver physically on your machine. This dependency downloads the driver for you automatically. So, using this approach, you can run your tests in Gitlab.

      Like

      Reply

      1. Thanks again for your quick response.
        I solved the driver issue with a different appraoch (although your approach is much better and i will try that now).
        Now thr chrome is lauching on gitlab and tests are starting. However all the tests fail with element not found exception. The same tests are passing on locally on chrome running headless (mac machine). Tried putting all sorts of waits and other hacks but same result.
        I have no idea why its happening and how can i debug it.
        Can you please help here.

        Thanks,
        Atul

        Like

  7. Hey, nice article! I am almost there bit getting this error in gitlab pipelene – what could be the solution for it?

    org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Response code 500. Message: unknown error: Chrome failed to start: exited abnormally.
    (unknown error: DevToolsActivePort file doesn’t exist)
    (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

    Like

    Reply

    1. The response code 500 error is not a GitLab issue. It is related to Selenium. Update the Selenium version to 4.11 and you should be fine. Selenium version 4.10 or lower needed a different way to run the tests with Chrome Version 115. Selenium Version 4.11 has fixed this issue.

      Like

      Reply

  8. Hi Vibha, I make it a habit to revisit this page and always have an epiphany without fail.There were two things on my mind:
    1.I ‘ve seen here that you have mentioned we need a gitlab runner in the start however I am not sure where has it been implemented here.Does this mean if we use a pre-existing image on the .yml file, it might have the runner installed in it and hence we don’t have to register and install a new Gitlab runner from scratch.
    2.Will this markhobson image be able to execute the test cases on edge instances.If not,Where can i get images to run firefox , edge etc for parallel /cross browser execution

    Like

    Reply

    1. 1. If you use GitLab.com, you can run your CI/CD jobs on SaaS runners hosted by GitLab. These runners are managed by GitLab and fully integrated with GitLab.com. By default, these runners are enabled for all projects. Refer this page – https://docs.gitlab.com/runner/. So, you basically do not need to configure GitLab runner manually, until you have any very specific requirement. You can go to the Setting present on the left side of the Gitlab page and go to CI/CD option and then scroll down to see which runner is configured in your project.
      2. markhobson image only run chrome tests on docker. To perform cross-browser testing, it is always advisable to run tests on Selenium Grid as Grid contain the images for chrome, firefox and edge

      Like

      Reply

      1. Hi Vibha, Thanks a ton for the response and i can only imagine the hours of research and strenous debugging that must have gone to materialize this scenario.However,Please also elucidate a bit on how to run appium scripts/mobile automation scripts on CI/CD Pipeline

        Like

      3. Ok Cool.Btw I was facing an issue with regards to executing a suite on GitLabs via the image that you had given as an example, so here are a few observations that I came across and need your valuable suggestion regarding the same:
        1. The suite executed for sometime and then I got the message as Build Success but artifacts are not being produced, it’s stuck at the same point for more than an hour and I checked multiple times whether there was an issue with the Gitlab.yml with regards to configuration but I had clearly added that step as you had I.e. Artifacts, path ( I actually added /target/ folder to .gitignore coz I dint want to push it to Remote repo but later on realized that the Katy to the artifacts actually gets generated there , so I o committed /target/ to remote repo by removing it from .gitignore) but still faced the same issue.
        2. I could nt find any runner available so I had to create a shell-executor based custom runner for the same.
        3. The scripts run sequentially so take around 10 + mins but still no solid outcome regardless of Buikd Success so Does execution time impact the the uploading of artifacts.
        4. It’s actually a private portal so should nt be accessed remotely but I was surprised that the execution was actually happening , so does this means it’s actually getting executed on local and does the fact that the portal is private play a role in why it does not produce artifacts
        5.I can’t even see the artifacts section in My Gitlab account so is this related to the account.

        This Issue has been giving me sleepless nights and have nt been able to find a solution, kindly provide your valuable insights once available for the same.Also, Please let me know if you require further info

        Like

      4. The artifact will not be generated if the execution is not completed. Your tests are running in the Gitlab? Artifact section is not related to the account. By using the keyword – artifacts:.. you will get the artifact folder, but the execution should be finished for the same.

        Like

      5. Hi Vibha,

        Thanks for the swift response. I noticed a peculiar behaviour that the report is getting generated but not in a downloadable zip archive format but to the .jvm dump stream and I noticed this in the GitLab Run Console and so I had to manually navigate to the URL specified in the ..jvm dumpstream to verify the same. The Issue with this is that I did specify regarding the generation of the artifacts in the .yml file using the “artifact” keyword and you had said earlier that it will only get generated if the execution is complete so I am not sure why are the results just save to my local but not in a download zip archive format which can be shared or accessed with others, as anyone who has access to the repo can run the job and generate the results which I am not able to implement now as it’s only getting saved in my local.

        Like

  9. Hi,
    First thanks for wonderful article.

    However, while running test from GitLab
    I am getting mvn command not found.
    I have correctly setup MAVEN_HOME in my PATH and System Environment variables.
    when I do mvn –version in command prompt I do get maven version as well

    Like

    Reply

    1. Hi Anurag, when you are running your tests in Gitlab, it runs on a docker image. You have setup Maven in your local machine, not in docker. I suggest you to use the image I have mentioned in the script. That particular image should have maven installed in it. Your tests should work

      Like

      Reply

  10. Hi, thank you for providing this information. I have some inquiries regarding reports. Instead of downloading the report, is it feasible for us to watch it on Git Lab together with the complete logs?

    Like

    Reply

Leave a comment