SpringBoot DynamicPropertyRegistry Values Not Set After Dev Tools Reload for RestartScope Containers

When using a combination of @RestartScope Test Containers and the DynamicPropertyRegistry registry property values are not set back after Spring Dev Tools reload the project. The issue was partially resolved in https://github.com/spring-projects/spring-boot/issues/35786 but that fix does not appear to resolve the entire issue. I've linked to a small example project demonstrating the issue. See https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue

Comment From: nosan

I did some research regarding this bug, and frankly, it is quite a tricky one.

First run:

TestcontainersLifecycleBeanPostProcessor
  -- initializeContainers()

RestartScopeInitializer.RestartScope
  -- Restarter.getOrAddAttribute('redisContainer', ObjectFactory<?> factory)
  -- 'redisContainer' does not exist in attributes, so ObjectFactory<?> is being called.

TestcontainersPropertySourceAutoConfiguration.dynamicPropertyRegistry() is being called because it is needed for 'redisContainer'
-- Attach 'testcontainersPropertySource'
-- Register bean definition named: EventPublisherRegistrar.class.getName()

TestcontainersConfiguration.redisContainer()
  -- create RedisContainer
  -- Set RedisContainer properties to the 'dynamicPropertyRegistry'

TestcontainersLifecycleBeanPostProcessor.initializeStartables()
 -- Start containers

DemoController @Value annotations are being processed.

Restart:

TestcontainersLifecycleBeanPostProcessor
  -- initializeContainers()
RestartScopeInitializer.RestartScope
  -- Restarter.getOrAddAttribute("redisContainer", ObjectFactory<?>)
  -- "redisContainer" exists in attributes, so ObjectFactory<?> **is not being called.**

DemoController @Value annotations are not being processed due to Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'MY_REDIS_HOST' in value "${MY_REDIS_HOST}"

There are two potential issues with this bug.

The first one TestcontainersPropertySourceAutoConfiguration.dynamicPropertyRegistry() is being called too late if there is no direct reference to it, so TestcontainersPropertySource will not be available during @Value annotation processing.

The second one is that methods annotated @RestartScope are being called only once. That is why dynamicPropertyRegistry will not be created earlier, and container properties are ignored. (Properties are being set inside the method).

A potential fix is: - If properties come from @RestartScope methods they should be shared across restarts. - TestcontainersPropertySource should be registered earlier.

This branch contains an implementation of the fix that I have tried to describe above. https://github.com/spring-projects/spring-boot/compare/main...nosan:spring-boot:41552

I used this example for research: https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue

Comment From: philwebb

Thanks for the detailed analysis @nosan. This is indeed a complicated issue to fix and made even more difficult by the changes we're introducing in 3.4 due to #41996.

I'm not really sure if we should attempt to fix this in 3.3 since I don't really like the fact that TestcontainersPropertySource needs to know about restart scope.

The following code does appear to work with 3.4.0-SNAPSHOT:

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

    @Bean
    @RestartScope
    public GenericContainer<?> redisContainer() {
        GenericContainer<?> redisContainer = new GenericContainer<>("redis:7");
        redisContainer.withExposedPorts(6379);
        redisContainer.withCommand("redis-server", "--requirepass redis_user", "--save 60 1", "--loglevel debug");
        return redisContainer;
    }

    @Bean
    public DynamicPropertyRegistrar redisContainerProperties(GenericContainer<?> redisContainer) {
        return (registry) -> {
            registry.add("MY_REDIS_HOST", () -> "localhost");
            registry.add("MY_REDIS_PORT", () -> redisContainer.getMappedPort(6379));
            registry.add("MY_REDIS_PASSWORD", () -> "redis_user");
        };
    }

}

This split allows the container to remain in the @RestartScope whilst the properties are always recreated.

Comment From: philwebb

The following ugly hack will work with 3.3.x:

package com.example.demo;

import java.util.List;

import org.springframework.boot.devtools.restart.RestartScope;
import org.springframework.boot.test.context.TestConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.test.context.DynamicPropertyRegistry;
import org.testcontainers.containers.ContainerState;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.lifecycle.Startable;

import com.github.dockerjava.api.command.InspectContainerResponse;

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

    @Bean
    @RestartScope
    public GenericContainer<?> redisContainer(DynamicPropertyRegistry registry) {
        GenericContainer<?> redisContainer = new GenericContainer<>("redis:7");
        redisContainer.withExposedPorts(6379);
        redisContainer.withCommand("redis-server", "--requirepass redis_user", "--save 60 1", "--loglevel debug");
        return redisContainer;
    }

    @Bean
    public RedisProperties redisProperties(GenericContainer<?> redisContainer, DynamicPropertyRegistry registry) {
        redisContainer.start();
        registry.add("MY_REDIS_HOST", () -> "localhost");
        registry.add("MY_REDIS_PORT", () -> redisContainer.getMappedPort(6379));
        registry.add("MY_REDIS_PASSWORD", () -> "redis_user");
        return new RedisProperties();
    }

    static class RedisProperties implements Startable, ContainerState {

        @Override
        public void start() {
        }

        @Override
        public void stop() {
        }

        @Override
        public List<Integer> getExposedPorts() {
            return null;
        }

        @Override
        public InspectContainerResponse getContainerInfo() {
            return null;
        }

    }

}

Comment From: philwebb

Flagging to see if the team think we should still attempt a fix in 3.2/3.3.

Comment From: nosan

I propose adding a note to the documentation explaining that DynamicPropertyRegistry does not work with restart-scoped containers (beans), along with the reason behind this limitation. For version 3.4.x, include a reference to DynamicPropertyRegistrar and mention that it works with restart-scoped containers (beans).

Fixing this issue in versions 3.2/3.3 would result in inconsistency with 3.4.x, as the latter will not receive a similar fix.

Comment From: nosan

This hack works with 3.2.x, 3.3.x and 3.4.0-M3

@TestConfiguration(proxyBeanMethods = false)
class TestcontainersConfiguration {

    @Bean
    @RestartScope
    public GenericContainer<?> redisContainer() {
        GenericContainer<?> redisContainer = new GenericContainer<>("redis:7");
        redisContainer.withExposedPorts(6379);
        redisContainer.withCommand("redis-server", "--requirepass redis_user", "--save 60 1", "--loglevel debug");
        return redisContainer;
    }

    @Autowired
    void registryRedisContainerProperties(@Lazy GenericContainer<?> redisContainer, DynamicPropertyRegistry registry) {
        registry.add("MY_REDIS_HOST", () -> "localhost");
        registry.add("MY_REDIS_PORT", () -> redisContainer.getMappedPort(6379));
        registry.add("MY_REDIS_PASSWORD", () -> "redis_user");
    }

}

Comment From: wilkinsona

It'll fail by default with 3.4.0-SNAPSHOT due to the deprecation of support for injecting DynamicPropertyRegistry.

Comment From: philwebb

We're going to leave this one open to see if we can fix it in some way, but it's hard to see us getting to it quickly. Upgrading to 3.4. when it's out is probably the best option for anyone with this issue.

Comment From: shawnweeks

I've updated to 3.4.1 and am using the new DynamicPropertyRegistrar however there are still issues namely the DynamicPropertyRegistrar bean is created early in the startup process however the initialize method is called too late in the process for all the placeholders in my data sources to resolve. On 3.3.x I could use a bean post processor on the test side to force a dependency between my data source classes and dynamicPropertyRegistry but this no longer works.

Comment From: wilkinsona

@shawnweeks can you please update https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue along those lines so that we can take a look?

Comment From: shawnweeks

I'm probably misunderstanding why the issue happens but here is an example, sorry it's a little convoluted but it's based on a real application with the same issue. See https://github.com/shawnweeks/spring_boot_testcontainers_restartscope_issue/tree/issue_v2

Comment From: philwebb

@shawnweeks I think the issue in the sample is the demoFilter bean is causing early initialization. This is a general problem, not just related to Testcotnainers. There's a small warning in the docs about this.

You can use a DelegatingFilterProxyRegistrationBean to get around the problem:

@Bean
DelegatingFilterProxyRegistrationBean demoFilterRegistration() {
    return new DelegatingFilterProxyRegistrationBean("demoFilter");
}

Comment From: shawnweeks

I've updated the example, in the real program the filters are for Spring Security and it appears that what you suggest works when used by it self but if your using FilterRegistrationBean to disable registration the original issue comes back. The filter causes everything to initialize too early. This issue didn't occur setting everything up the pre 3.4.x way.

Comment From: shawnweeks

If I change to something like this everything appears to work the new way.

    @Bean
    public DelegatingFilterProxyRegistrationBean demoFilterProxyRegistration() {
        DelegatingFilterProxyRegistrationBean delegatingFilterProxyRegistrationBean = new DelegatingFilterProxyRegistrationBean("demoFilter");
        delegatingFilterProxyRegistrationBean.setEnabled(false);
        return delegatingFilterProxyRegistrationBean;
    }