JournalBlog

From Flaky to Reliable: Engineering Stability into Selenium Automation

By Simi Sunny9 min read

When Automation Results Become Unpredictable

Automation testing is intended to make regression testing faster, reduce repetitive manual effort, and provide quick feedback about application quality.

But what happens when the automation suite itself becomes unreliable?

Imagine a QA team executing a Selenium automation suite through Jenkins. The pipeline starts successfully, but some test cases fail unexpectedly. Interestingly, when the same test cases are executed individually, they pass without any issues.

The team reruns the failed tests, and most of them pass the second time.

Initially, these failures may appear to be application defects. However, after investigating the execution logs and test behavior, the team discovers that the actual problems are related to synchronization, shared test data, and environment dependencies.

This is the challenge of flaky tests.

A flaky test is an automated test that produces inconsistent results without any corresponding change in the application or test code.

We encountered a similar challenge during our Project Delta QA automation activities.

During a system cleanup activity, the configuration associated with our Test QA Company - Disabled, which was used for QA automation, was removed. This caused existing automation scenarios to fail because they depended on that configuration.

Instead of repeatedly recreating the missing configuration, we investigated how to prevent the issue from happening again.

We renamed the designated QA account to Automation Company - Do Not Delete and introduced a naming convention that allowed the system cleanup scripts to automatically bypass this account.

This ensured that the required automation configurations were preserved during subsequent cleanup activities.

The incident reinforced an important engineering principle: automation reliability depends not only on the test scripts but also on the stability of the data, configurations, and environments that support them.

So, how can QA engineers identify these problems and build automation that produces consistent results?

1. Synchronizing Selenium with Application Behavior

One of the first areas to investigate when automation becomes unstable is synchronization.

Modern applications perform several operations in the background. API requests, database updates, data processing, and UI rendering may take different amounts of time.

Consider a data platform where a QA engineer triggers a pipeline execution.

Initially, the status badge displays Processing. Once the background operation finishes, the status changes to Completed and turns green.

If Selenium checks the status immediately after triggering the operation, the test may fail because the pipeline is still processing.

A common implementation uses a fixed delay:

driver.findElement(By.id("runPipeline")).click();

Thread.sleep(5000);

String status = driver.findElement(

        By.id("pipelineStatus")

).getText();

Assert.assertEquals(status, "Completed");

The problem is that the script assumes the operation will always finish within five seconds.

If processing takes eight seconds, the test fails. If it takes two seconds, the remaining three seconds are wasted.

What we changed: Wait for a specific condition

Instead of using fixed delays, Selenium provides explicit waits that allow us to wait for a particular condition.

WebDriverWait wait = new WebDriverWait(

        driver,

        Duration.ofSeconds(30)

);

driver.findElement(By.id("runPipeline")).click();

By statusLocator = By.id("pipelineStatus");

wait.until(

        ExpectedConditions.textToBePresentInElementLocated(

                statusLocator,

                "Completed"

        )

);

String status = driver.findElement(statusLocator).getText();

Assert.assertEquals(status, "Completed");

Now, Selenium waits until the pipeline status changes to Completed or the timeout is reached.

This approach is particularly useful when testing asynchronous operations, background processing, dynamic dashboards, and API-driven UI updates.

The important difference is that the automation script now responds to application behavior rather than depending on an assumed execution time.

2. Making Element Identification More Reliable

Another reason automation tests become unstable is the use of fragile locators.

Selenium interacts with web elements through locators such as ID, name, CSS selectors, and XPath.

However, not all locators are equally reliable.

For example, consider:

By.xpath("/html/body/div[2]/div[1]/div[3]/button");

This absolute XPath depends on the exact HTML structure. A small UI modification can change the element's position and cause the test to fail.

Instead, we should prefer stable attributes.

By.id("submitButton");

Or:

By.xpath("//button[text()='Submit']");

Why does this matter?

Consider a situation where developers update a page layout but do not change the actual functionality.

If the automation suite uses absolute XPath expressions, several test cases may fail even though the application is working correctly.

Using stable locators reduces this dependency on page structure.

Some good practices include:

  • Prefer unique IDs whenever available.
  • Use meaningful CSS selectors.
  • Avoid absolute XPath expressions.
  • Avoid frequently changing attributes.
  • Coordinate with developers to introduce stable test attributes where necessary.

The objective is to make automation resilient to minor UI changes without hiding genuine application defects.

3. Preventing Test Data Collisions During Parallel Execution

As automation suites grow, executing every test case sequentially can increase regression execution time.

TestNG provides parallel execution capabilities that allow multiple test methods or classes to run simultaneously.

However, parallel execution introduces a challenge that QA engineers need to consider: shared test data.

Consider three registration tests executing at the same time.

Test Case Email Address Expected Result
Test 1 testuser@gmail.com Successful registration
Test 2 testuser@gmail.com Successful registration
Test 3 testuser@gmail.com Successful registration

All three tests attempt to create an account using the same email address.

The first test may succeed, while the remaining tests fail because the account already exists.

These failures are caused by test data collisions rather than application defects.

Our approach: Generate unique test data

Instead of hardcoding the same email address in every test, we can generate a unique value for each execution.

String email = "testuser+" +

        UUID.randomUUID() +

        "@gmail.com";

This generates a different email address for each execution.

For example:

If the application accepts Gmail plus-addressing, verification emails can reach the same base inbox.

This approach helps:

  • Avoid duplicate registration errors.
  • Support parallel execution.
  • Reduce dependencies between test cases.
  • Minimize manual test data preparation.

For production-like test environments, we should also ensure that generated users are cleaned up appropriately after execution.

The larger lesson is that parallel execution is reliable only when test cases are designed to work independently.

4. Managing Dynamic Elements and DOM Updates

Modern web applications frequently update their content without refreshing the entire page.

For example, consider a leave management application.

A QA engineer clicks Apply Leave. The application processes the request and displays a popup containing the leave details.

If Selenium tries to interact with the popup before it becomes visible, the test may fail.

We can handle this using an explicit wait:

WebDriverWait wait = new WebDriverWait(

        driver,

        Duration.ofSeconds(10)

);



wait.until(

        ExpectedConditions.visibilityOfElementLocated(

                By.id("leavePopup")

        )

);

Similarly, a stale element reference can occur when an element is replaced during a DOM update.

For example:

WebElement leaveButton =

        driver.findElement(By.id("applyLeave"));

leaveButton.click();

leaveButton.click();

If the application refreshes the relevant DOM after the first interaction, the second click may throw a StaleElementReferenceException.

Instead, locate the element again after the DOM update:

By leaveButton = By.id("applyLeave");

WebDriverWait wait = new WebDriverWait(

        driver,

        Duration.ofSeconds(10)

);

wait.until(

        ExpectedConditions.elementToBeClickable(leaveButton)

);

driver.findElement(leaveButton).click();

This approach ensures that Selenium works with the current element rather than an outdated reference.

For applications with dynamic popups, tables, loading indicators, and search results, understanding when the DOM changes is essential for stable automation.

5. Protecting Test Configurations and Automation Environments

Even a well-written Selenium script can fail if the environment supporting it changes unexpectedly.

Our Project Delta experience demonstrated this clearly.

The cleanup activity removed a configuration required by existing automation scenarios. The test scripts had not changed, and the application functionality was not the problem. The supporting configuration had disappeared.

We addressed this by renaming the dedicated account to Automation Company - Do Not Delete and configuring the cleanup scripts to exclude it.

This prevented the required automation configuration from being removed during subsequent cleanup operations.

There are other environment-related factors that can also affect automation:

  • Browser and WebDriver version mismatches.
  • Different application configurations.
  • Missing test accounts or supporting data.
  • Network delays.
  • Insufficient CPU or memory resources.
  • Different execution behavior between local machines and Jenkins.

For example, a test may pass on a QA engineer's laptop but fail in Jenkins because the CI environment has a higher workload and the application responds more slowly.

What should QA teams consider?

Rather than treating every failure as a Selenium problem, we should also review the environment in which the test executes.

Some useful practices include:

  1. Maintain dedicated automation accounts.
  2. Protect important configurations from cleanup activities.
  3. Keep browser and driver versions compatible.
  4. Document environment dependencies.
  5. Use consistent configurations across local and CI environments.
  6. Capture execution logs and screenshots when failures occur.

A reliable automation framework requires a reliable execution environment.

6. Investigating Failures Instead of Simply Rerunning Tests

When a test fails in Jenkins, rerunning it may temporarily produce a passing result.

However, this does not necessarily mean the underlying problem has been resolved.

For example, consider a test that fails with ElementClickInterceptedException.

The actual issue might be a loading spinner covering the button. Rerunning the test may work because the spinner disappears before the second execution reaches that step.

Similarly, a registration test may fail because another parallel test has already created the same account.

In both situations, simply rerunning the test hides the symptom without addressing the cause.

A better investigation approach

When a test fails, QA engineers should examine:

  • TestNG execution reports.
  • Jenkins console logs.
  • Failure screenshots.
  • Exception messages.
  • Test data used during execution.
  • Application and environment logs.
  • Differences between individual and full-suite execution.

For example, if a test consistently passes individually but fails during parallel execution, investigate shared test data and dependencies.

If it fails only in Jenkins, compare the browser version, execution environment, and application response time.

The objective is to identify whether the failure originates from the application, automation script, test data, or execution environment.

This makes debugging more effective and helps prevent recurring failures.

7. Building a More Reliable Selenium Automation Suite

After understanding these challenges, the next step is to incorporate reliability into the automation framework itself.

Some practices that can help include:

Synchronization: Use explicit waits instead of unnecessary fixed delays.

Locator strategy: Prefer stable attributes and maintainable XPath or CSS selectors.

Test independence: Avoid execution dependencies between test cases.

Test data management: Generate unique data where required, especially during parallel execution.

DOM handling: Re-locate elements after dynamic updates.

Environment management: Maintain consistent browser versions and protected test configurations.

Failure analysis: Use screenshots, logs, and TestNG reports to identify root causes.

These practices should be part of the framework design rather than introduced only after automation starts failing.

What Improved Through These Changes?

The Project Delta configuration incident showed us that automation failures cannot always be solved by modifying Selenium code.

Sometimes the actual solution lies in improving how the testing environment and its supporting resources are managed.

By protecting dedicated automation configurations, using condition-based synchronization, generating unique test data, and improving failure investigation, QA teams can reduce unnecessary automation failures.

These improvements contribute to:

  • More consistent regression execution.
  • Fewer false failures.
  • Better Jenkins pipeline reliability.
  • Reduced debugging and investigation time.
  • More independent test cases.
  • Greater confidence in automation reports.

The ultimate objective is to make automation results meaningful so that developers and QA engineers can trust the feedback provided by the pipeline.

From Flaky Tests to Automation Confidence

Building stable Selenium automation is not a one-time activity. It requires continuous attention to the application, test scripts, test data, and execution environment.

Our experience with Project Delta reinforced an important lesson: identifying the root cause is more valuable than repeatedly fixing the symptoms.

A reliable automation suite is not simply one that executes hundreds of test cases. It is one that produces consistent results, identifies genuine application defects, and provides trustworthy feedback throughout the development lifecycle.

For QA engineers, the focus should not only be on writing scripts that pass today, but also on building automation that remains dependable as the application evolves.

Reliable Test Data → Stable Execution → Trustworthy Reports → Better Software Quality

The real success of automation is not how many tests we execute, but how much confidence we can place in their results.

Simi Sunny

Written by

Simi Sunny

QA Engineer

Simi Sunny is a QA Engineer specializing in software quality assurance and delivering reliable digital solutions. With hands-on experience in manual testing, regression testing, test case design, and automation testing, she focuses on identifying defects, enhancing product quality, and ensuring seamless user experiences. Her expertise spans web and mobile application testing, test planning, defect management, and close collaboration with cross-functional teams to deliver robust, stable, and high-quality software solutions.

Have a project in this territory?

Start a project