Spring AbstractTestNGSpringContextTests is not thread-safe regarding tracked exceptions

TestNG allows running tests in parallel, using the parallel=methods parameter. This causes tests to run in multiple threads using the same class instance.

However, the code in https://github.com/spring-projects/spring-framework/blob/main/spring-test/src/main/java/org/springframework/test/context/testng/AbstractTestNGSpringContextTests.java#L148-L162 is not thread-safe.

As an example, consider the following minimal test case:

@ContextConfiguration(classes = {SampleJavaClassTest.Configuration.class})
public class SampleJavaClassTest extends AbstractTestNGSpringContextTests {
    public static class Configuration {

    }

    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message1")
    public void message1() {
        throw new RuntimeException("Message1");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message2")
    public void message2() {
        throw new RuntimeException("Message2");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message3")
    public void message3() {
        throw new RuntimeException("Message3");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message4")
    public void message4() {
        throw new RuntimeException("Message4");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message5")
    public void message5() {
        throw new RuntimeException("Message5");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message6")
    public void message6() {
        throw new RuntimeException("Message6");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message7")
    public void message7() {
        throw new RuntimeException("Message7");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message8")
    public void message8() {
        throw new RuntimeException("Message8");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message9")
    public void message9() {
        throw new RuntimeException("Message9");
    }
    @Test(expectedExceptions = RuntimeException.class, expectedExceptionsMessageRegExp = "Message10")
    public void message10() {
        throw new RuntimeException("Message10");
    }
}

Running this test with the following TestNG configuration:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd" >
<suite name="TestSuite" verbose="2" thread-count="5">
    <test name="DemoTest" annotations="JDK" parallel="methods" verbose="2" preserve-order="true" thread-count="5">
        <packages>
            <package name="com.example.*"/>
        </packages>
    </test>
</suite>

(note thread-count=5 and parallel=methods)

Causes the tests to fail randomly:

org.testng.TestException: The exception was thrown with the wrong message: expected "Message2" but got "Message10"

The stacktrace from such failures also shows the stacktrace originating from the wrong test. The message2 test failure stack trace looks like this:

Caused by: java.lang.RuntimeException: Message10
    at com.example.SampleJavaClassTest.message10(SampleJavaClassTest.java:54)
...

In the code above, this.testException is being clobbered by a different thread, causing the issues described.

Comment From: sharky5102

That was quick! LGTM.

Comment From: github-actions[bot]

Fixed via 5cd2cb38e1b1ed3e1951499d9b5ad63372dd7e3e

Comment From: sbrannen

This has been fixed in 6.2.x and main.

It would be great it you could try out 6.2.12 snapshots to ensure that the fix addresses the issues in your project.

Cheers,

Sam