This blog will give a technical walkthrough on how to create rich and interactive report portal dashboards such that data can be grouped by different dimensions using attributes supercharging your test analysis.

What is a report portal?

Suppose you don’t have a good way to measure and visualize how reliable your automation test suites are . It is easy to be overwhelmed by your suite's many broken or flaky tests.

SDET/Automation Tester  often struggle with:

1. Resorting to ad-hoc local spreadsheets to track bugs for a failing test
2. Maintain a record of which test scripts  are broken/flaky .

ReportPortal is an open-source, cloud-based platform designed for test automation reporting and analytics. It allows teams to aggregate test results from multiple testing frameworks, providing real-time insights into test execution trends, performance metrics, and overall quality of software releases. Report Portal supports integration with various testing tools and frameworks, making it easier for teams to manage and analyze test results efficiently.

that elegantly solves above these problem.

1.        Login to Report Portal and In the bottom left, click on the user profile image and click on profile  then generate “Access token” as mentioned below .

To start using ReportPortal with TestNG framework please do the following steps:

1. Configuration
• Create/update the reportportal.properties configuration file
• Build system configuration
• Add Listener
2. Logging configuration
• Loggers and their types
3. Running tests
• Build system commands

Configuration

To start using ReportPortal you need to create a file named reportportal.properties in your Java project in a source folder src/main/resources or src/test/resources (depending on where your tests are located):

reportportal.properties  configuration file

rp.endpoint = http://localhost:8080

rp.api.key = e0e541d8-b1cd-426a-ae18-b771173c545a

rp.launch = TestNG Tests

rp.project = default_personal

Property description

• rp.endpoint - the URL for the ReportPortal server (actual link).
• rp.api.key - an access token for ReportPortal which is used for user identification. It can be found on your report portal user profile page.
• rp.project - a project ID on which the agent will report test launches. Must be set to one of your assigned projects.
• rp.launch - a user-selected identifier of test launches.

Build system configuration

Maven

If your project is Maven-based you need to add dependencies to pom.xml file:

<project>

  <!-- project declaration omitted -->

  <dependency>

    <groupId>com.epam.reportportal</groupId>

    <artifactId>agent-java-testng</artifactId>

    <version>5.4.4</version>

    <scope>test</scope>

  </dependency>

  <!-- build config omitted -->

</project>

You are free to use you own version of TestNG, but not earlier than 7.1.0. If you leave just Agent dependency it will be still OK, it will use transitive TestNG version.

Gradle

For Gradle-based projects please update dependencies section in build.gradle file:

dependencies {

    testImplementation 'com.epam.reportportal:agent-java-testng:5.4.4'

}

Listener configuration

There are many ways to configure a listener in TestNG, but the most elegant and recommended way is to use a ServiceLoader file. Here is how you can do that:

1. Create folders /META-INF/services in resources folder (src/main/resources or src/test/resources)
2. Put there a file named org.testng.ITestNGListener
3. Put a default implementation reference as a single row into the file: com.epam.reportportal.testng.ReportPortalTestNGListener

Example: /META-INF/services/org.testng.ITestNGListener

com.epam.reportportal.testng.ReportPortalTestNGListener

That's it! You are all set.

Let’s run some tests

To run tests we just need to execute corresponding command in our build system.

Maven

mvn test or mvnw test if you are using Maven wrapper

Gradle

gradle test or gradlew test if you are using Gradle wrapper

Notice here I’m filtering tests that are tagged with a certain TestNG group like identity, onboarding, and performance using -DincludedGroups, and also using the -Drp.launch param to give the launch (or test run) a meaningful name

Attributes as metadata

ReportPortal supports adding additional metadata as key-value pairs either via the command line or as test annotations

At launch

If you open Launches, you’ll see the 3 launches show up for identity_tests, onboarding_tests, and performance_tests and they also have some default attributes or tags like those below which are specified in reportportal.properties for rp.attributes

group:test_infra;test_type:backend

These attributes and tags are very useful to filter launches and create informative dashboards

In a real project, you may want to add even more attributes to a given test suite.

Dashboards

Let’s unpack the power of visualisation using dashboards

There are a couple of distinct personas report portal supports

🤹 As a leader

I may want to see broad metrics for the 3 teams that I’m leading and identify hotspots that need attention.

A few widgets make a lot of sense such as:

1. Component health check (table): See component level health by splitting them using attribute team and observe overall pass/failed metrics in a tabular format
2. Component health check: See the split of the last test suite at a team level into visual cards with the option to open actually failed tests
3. Launch statistics chart - bar view: Bar chart with a summary of how many cases passed/failed/skipped
4. Overall statistics - donut view for latest launches: Donut chart with a summary of failed/passed cases and how many cases are left to investigate

You can also see additional charts like

1. Test-cases growth trend chart: to see how many tests were added or removed from the suites over time
2. Failed cases trend chart: to analyze the status of your broken tests

How can you configure these?

The magic lies in creating filters with a combination of certain attributes and then using them to create the visualisation you desire.

I promise it's pretty intuitive once you play with it.

For instance, below is how you can set up the component health check table

1. Select a filter that filters all launches having group equal to test_infra
2. Select the latest launches to see the last launch
3. Select team attribute as a way to split the data

And say you want to configure the Overall statistics widget

1. Select filter as “group as test_infra”
2. Select Donut view
3. Select Latest launches

And you are done.

You can also edit the filter by adding more conditions, like below

🧑🏻💻 As an engineer

The previous visualizations are great for getting a bird's eye view of how a given group with multiple teams or a product is doing.

Often you as an engineer would rather focus more on how are your tests behaving and may have a team-specific or flow-specific dashboard.

There are a few more dashboards that are more useful for analysis.

These are focussed mostly at a launch (or one test suite level) and can be used as lower-level dashboards to drive analysis and fixes in a focussed area.

1. Component health check: sliced for services to find the health of fine-grained services
2. Passing rate per launch: to find the overall status for the last launch
3. Flaky test cases table (top 50): identity the top flaky cases to look at
4. Most failed test cases table (top 50): identify all the broken tests

1. Launches duration chart: to see which launches take how much time in order to find slow-running suites
2. Most time-consuming test cases widget (top 20): to find slow-running tests to be optimised

Recap

To summarise

1. Report portal is an excellent test observability solution adhering to true FOSS (free and open source software) spirit that can be installed over Docker containers on your private network
2. Setup is quick and frictionless with running docker-compose, creating a project, getting the API key, and setting up logging integration
3. Launches provide you last test run status with support to analyse cases
4. Use attributes at the launch or test case level to add rich filtering capabilities
5. Create dashboards with either aggregated metrics or low-level analysis metrics suitable both for a leader or engineer persona

So what are you waiting for?

Go ahead and have a much better picture of your test suites and keep them in good health 🫶