Mastering Test Reporting with Allure

Introduction

Software testing plays a critical role in the successful delivery of a project. Test teams require an effective test reporting and visualization tool to understand test results, analyze failures, and facilitate knowledge sharing among teams. To meet these needs, we use a powerful tool called Allure. In this blog post, we will discuss how Allure is used and integrated into our project.

In Intenseye, we use popular test frameworks like Pytest and Behave, and we have integrated Allure with these frameworks. This integration allows us to present test results in a more visual and understandable format. Allure provides a set of XML files and a rich web interface for generating test reports.

1. What is Allure?

Allure is a powerful test reporting framework that simplifies the process of analyzing and visualizing test results. It provides a comprehensive and user-friendly platform for generating test reports that enhance the overall testing experience.

One of the standout features of Allure is its ability to present test outcomes in a clear and intuitive manner. The framework offers visually appealing reports with interactive graphs, charts, and statistics that make it easy to understand the overall status of tests. This visual representation enables testers and stakeholders to quickly identify trends, patterns, and potential issues, ultimately improving the decision-making process.

Another notable capability of Allure is its support for rich attachments. Testers can attach various files, such as screenshots, logs, and additional documentation, to provide further context and evidence of test results. This feature enables more comprehensive reporting and aids in root-cause analysis when failures occur.

Furthermore, Allure seamlessly integrates with popular testing frameworks and tools, making it compatible with a wide range of test environments. It supports integration with different languages and frameworks such as TestNG, JUnit, and Cucumber, allowing testers to leverage Allure’s reporting capabilities regardless of their preferred testing technology. Supported frameworks are grouped by language: Java, Python, JavaScript, Ruby, Groovy, PHP, .Net, and Scala.

2. Allure Report Structure

As mentioned above, Allure displays the test results in detail on different pages. Let’s explore each of these pages while examining the information they contain in depth.

Overview page:The combination of these widgets on the Allure report’s overview page provides a comprehensive overview of your project’s testing status, history, categorization, execution environment, and trend analysis. This enables effective monitoring, analysis, and communication of your test outcomes to stakeholders.

Categories page:The Categories tab in Allure allows users to create custom classifications for defects in test results. By default, there are two categories: Product defects for failed tests and Test defects for broken tests. However, users can define their own custom defect classifications by following the document.

Suites page:The Suites tab in Allure displays executed tests in a hierarchical manner, grouped by suites and classes. It provides a structured view of the test execution process, enabling easy navigation and analysis of the results based on suites and classes. By clicking on individual tests, users can access the corresponding test case pages. This page contains detailed information specific to each test case, such as executed steps, timings, attachments, categorization labels, descriptions, and links. They offer comprehensive insights into the details of the test case execution and associated data.

Allure allows you to collect information about test retries during a single test run and track the historical execution data over time. You can specify the number of retries for flaky tests by using language and test frameworks. As you can see below, you can detect the retried tests with the red-marked retry icon on the Suites tab.

Graphs page:Graphs in Allure enable users to visualize various statistics extracted from the test data. They provide valuable insights by presenting information such as the breakdown of test statuses (e.g., passed, failed, skipped), diagrams showcasing severity levels and their distribution, and duration graphs illustrating the time taken for test execution. These graphical representations enhance the understanding of test results and facilitate data-driven analysis.

Timeline page:The Timeline tab in Allure provides a visual representation of the chronological execution of tests. Allure adaptors collect precise timings of the tests, and this tab displays them in a sequential or parallel structure, reflecting the actual execution timeline. It allows users to track the order and timing of test execution, providing insights into the test execution flow and potential bottlenecks.

Behaviors page:For projects following a Behavior-driven approach, the Behavior tab in Allure groups tests results based on tags such as Epic, Feature, and Story. It provides a structured view where test cases are organized according to these tags, allowing users to easily navigate and analyze test results based on their corresponding behavior components. This helps in understanding the test coverage and traceability to the specific features and stories defined in the project.

Packages page:The Packages tab in Allure displays test results organized by packages in a tree-like layout. It provides a clear view of the test structure and allows users to navigate through different packages to access detailed information about the tests within each package. This feature helps in organizing and analyzing test results based on package dependencies, facilitating efficient test coverage tracking and management.

3. Customizing Allure Reports

Allure provides flexible options to customize reports according to specific project requirements. One way to customize reports is by using custom report categories, attachments, tags, and annotations. There are some features that we use on the Intenseye test repositories:

Custom report categories allow you to create custom classifications for defects or any other relevant information. By defining custom categories, you can categorize test results based on specific criteria that align with your project needs. For example, you can create categories such as “Security Issues” or “Performance Failures” to track and prioritize specific types of issues.
Tags and annotations offer a way to add additional metadata to test cases, providing further context and information. You can use tags to label test cases with specific attributes, such as severity or test environment. Annotations, on the other hand, allow you to add custom annotations to test methods, providing extra details or instructions.In the given example, the test case is annotated with a custom category “Security Issues”. It also specifies that the test will run in the “Test Environment: Production” and has a severity level set to “CRITICAL”. These annotations provide a way to effectively organize and classify test results based on specific criteria, enhancing the overall reporting and analysis process.,

import allure
import pytest

@allure.title("Test Case Title")
@allure.severity(allure.severity_level.CRITICAL)
@allure.category("Security Issues")
@allure.environment("Test Environment: Production")
def test_security_check():
    # Test implementation goes here
    pass

The attachment feature enables you to include additional files or documents within the report, providing supplemental information for test results. You can attach screenshots, log files, or any other relevant artifacts that assist in better understanding and troubleshooting test failures.

from allure import attach, attachment_type
from behave.model import Scenario, Status
import json

def after_scenario(context, scenario):
    if scenario.status == Status.failed:
        capture_screenshot(context, scenario.name)
        capture_browser_logs(context, scenario.name)

def capture_screenshot(context, name):
    screenshot = context.browser.get_screenshot_as_png()
    attach(screenshot, name, attachment_type.PNG)

def capture_browser_logs(context, name):
    browser_logs = context.browser.get_log("browser")
    if browser_logs:
        attach(json.dumps(browser_logs), name, attachment_type.JSON)

Customization possibilities are not limited to annotations and categories. You can explore other Allure features and extensions to further tailor the reports based on your project’s unique requirements. Allure’s extensive documentation provides detailed guidance and examples for customizing and enhancing the report’s appearance and content.

4. Setting up Allure

Under this topic, I will provide a step-by-step guide on how to install Allure, integrate it with pytest and behave, and report the results locally.

Setting Up Allure with pytest:

Install allure-pytest package: allure-pytest is a Python package that integrates Allure with the pytest testing framework.

$ pip install allure-pytest

Run your pytest tests: To enable Allure listener to collect results during the test execution, simply add the --alluredir option followed by the path to the folder where the results should be stored. For example, below command instructs pytest to run the tests and store the Allure results in the /tmp/my_allure_results folder.

$ pytest --alluredir=/tmp/my_allure_results

Generate Allure report: After running your tests, navigate to the root directory of your pytest project in the terminal and run the below command. This command will generate the Allure report and open it in your default browser.

$ allure serve allure-results

Setting Up Allure with Behave:

Install allure-behave package: allure-behave is a Python package that integrates Allure with the behave testing framework.

$ pip install allure-behave

Run your Behave tests: You can specify the formatter directly in the command line:

$ behave -f allure_behave.formatter:AllureFormatter -o allure_results ./features

Generate Allure report: After running your tests, navigate to the root directory of your Behave project in the terminal and run the below command. This command will generate the Allure report and open it in your default browser.

$ allure serve allure_results

5. Integrations

Additionally, Allure can be extended through plugins and third-party integrations, further enhancing its functionality. Developers can create custom plugins to add new features or extend existing ones within the Allure framework. These plugins can be used to integrate Allure with other tools and systems used in the testing ecosystem, enabling seamless data exchange and enhancing the overall testing workflow. Third-party integrations with popular tools like Jenkins, Maven, and Docker provide additional automation and integration capabilities, making it easier to incorporate Allure into existing testing pipelines.

In Intenseye, we have integrated Allure Docker Service into our test automation pipeline using Kubernetes (k8s), following the guidelines here. Allure Docker Service is an open-source project that offers diverse opportunities for contribution. During the integration process, we encountered a permission issue, which we successfully resolved by implementing a minor fix within the project. By using the provided APIs, we seamlessly send all test results to Allure once our automated tests are executed. This integration allows us to consolidate and access the test outcomes across our projects on Allure, irrespective of the programming languages used. With each test execution, we gain valuable insights into test trends and patterns, enabling us to analyze and evaluate our test results consistently since the inception of several projects.

By utilizing the Allure Docker Service, we have unlocked a host of additional features that enhance our testing experience. These include the ability to easily send test results, generate comprehensive reports, and maintain a clean history of our testing activities. We were also able to add different projects that were in different repositories. The screenshot below illustrates some of these powerful features provided by the Allure Docker Service, facilitating efficient test result management.

6. Best Practices for Allure Usage

To effectively utilize Allure and optimize your test reporting and analysis, it is essential to follow best practices and implement strategies for organizing test cases, maintaining reports, and leveraging Allure’s features. Here are some recommendations and tips to enhance your Allure experience:

Define Severity Levels: Define a set of severity levels that align with your project’s testing requirements. For example, you might have severity levels such as Critical, High, Medium, and Low. Each severity level represents the impact or importance of a test case or failure.
Enable Test Retry: Allure provides built-in support for retrying failed tests, which can be a valuable strategy to improve test reliability and stability.
Organizing Test Cases: Adopt a structured approach to organizing your test cases within Allure. Consider creating separate test suites or categories based on functional areas or modules. This helps in better categorization and navigation of test results, making it easier to locate and analyze specific tests.
Capturing Screenshots and Attachments: To provide a more comprehensive view of test execution, Allure allows the inclusion of screenshots and other attachments. We demonstrated how to capture screenshots automatically on test failures and attach them to the corresponding steps and test cases in Allure reports. Additionally, we discussed how you can include log files and other relevant data as attachments, providing a holistic view of test execution.
Analyzing Test Trends: With each new test execution, analyze the test trends and patterns visible in Allure. Pay attention to metrics such as test pass rates, failure rates, and execution times to identify areas that require improvement or further investigation. The interactive graphs and statistics provided by Allure enable quick identification of trends and help in making data-driven decisions.
Integrations and Plugins: Explore and leverage the available integrations and plugins for Allure to enhance its functionality. Integrate Allure with your preferred testing frameworks, continuous integration tools, and other systems used in your testing environment. This allows for seamless data exchange, automation, and integration, improving the overall testing workflow and efficiency.
Integration with CI pipeline: Run Allure reports for each branch that merges with the master. This allows developers to verify if their changes impact other areas. Reviewing Allure reports helps detect new test failures and assess the impact of code changes.

Conclusion

In conclusion, Allure is a powerful test reporting and visualization tool that has greatly benefitted our testing efforts at Intenseye. Its seamless integration with Pytest and Behave, along with its extensive features and customization options, have made it an indispensable part of our testing workflow. By adopting Allure, we have improved the quality of our test reports, enhanced collaboration within the team, and gained valuable insights from visualized test metrics.

We encourage you to explore Allure and leverage its capabilities to elevate your test reporting and visualization. Remember to refer to the official Allure documentation and additional resources mentioned throughout this blog post for further information and support.

Happy reporting with Allure!

References: