SpringBoot Provide a possibility to extend/override configuration properties binding process

We faced an issue where a property source for 700+- objects (consists of 100+- fields) binds to java objects slowly: around 6 minutes. Our services rely on spring cloud hot refresh and because of slow binding users are struggling. We could optimize this process for our use case, but most of the binding classes are closed for extension.

Comment From: philwebb

There's not really enough information here for us to be able to take any action. Could you describe what optimizations you're trying to apply and what changes you'd need to do them?

A sample application that shows the issue would also be very helpful. Perhaps if we profile it there might be some general optimizations we can apply for everyone.

Comment From: dmitriytsokolov

I'm mostly interested in extending org.springframework.boot.context.properties.bind.Binder class, but almost all classes inside org.springframework.boot.context.properties.bind are package-private.

About optimisation. I'm not sure it would work generally, but the idea is the following: I have an ConfigurationProperties class which first child field is a hashmap, so my yml files that are bond to ConfigProps looks the following way:

config-props-prefix:
  some-root-field-name:
    map-key-1:
      obj-field1-key: value1
      obj-field2-key: value2
      ...

And I have thousands of such files that are bond to the map. And I don't need a property config-props-prefix.some-root-field-name.${some-map-key}.obj-field1-key to be processed by binder multiple times, so I could cache the result and use it for binding process of other map entries. Does it make any sense to you?

I'm using spring boot 2.7.18, but I've tested on spring boot 3.2.7 and performance is the same.

Comment From: wilkinsona

@dmitriytsokolov thanks for the additional details. Can you please share a sample application that uses Spring Boot 3.2.x or later that demonstrates what you've described above?

Comment From: dmitriytsokolov

@wilkinsona sure. Should I just create an example with couple of config files, or with hundreds to emulate performance issues?

Comment From: wilkinsona

Thanks. Something that replicates the performance issue would be ideal. Perhaps you could use a script to generate a sufficient volume of synthetic test data?

Comment From: dmitriytsokolov

attaching example of spring app with 1000 configs that is started up for 40-50s locally. In my real app I have more fields and more spring placeholders, so app start time/hot refresh takes 5-6 minutes demo_config_props.zip

One more thing: initially I though that I could cache results of resolved spring placeholders (because I have a lot of them) and get some performance benefits, so I've made the following change in Binder class locally:

    private <T> Object bindProperty(Bindable<T> target, Context context, ConfigurationProperty property) {
        context.setConfigurationProperty(property);
        Object original = property.getValue();
        Object fromCache = this.resolvedPlaceholdersCache.get(original);
        if (fromCache != null) {
            return fromCache;
        }
        Object result = this.placeholdersResolver.resolvePlaceholders(original);
        result = context.getConverter().convert(result, target);
        this.resolvedPlaceholdersCache.put(original, result);
        return result;
    }

but I've got approximately minus 20 seconds out of 5-6 minutes of binding time. It's not much, but still with the ability to override/extend some binding logic I could reduce it even more knowing some specifics of my serivce

Anyway, will be waiting for your input, thanks a lot

Comment From: dmitriytsokolov

The other option I'm thinking of: As you see I have a simple hashmap on the root level. So if only a part of the binding beans were opened for extension, I could try rebinding config props partially manually (binding of each map's element asynchronously and build the map by myself). This should help, but of course I'm not sure about it. I'm going to install local version of spring with required updates and will try to implement this approach in my app. But I would appreciate your input on this idea.

Comment From: Mobe91

I am experiencing similar issues with a spring-boot application that I am working on. It is a multi-tenant application where configuration for each tenant is provided via environment variables (~6000 keys in total atm). The binding of these properties takes more than 10 minutes. This happens on startup and whenever the configuration is refreshed using Spring Cloud as suggested by @dmitriytsokolov. I also suspect this to be the cause for the issue I commented on here (at least in my case).

I have done some profiling for this and was able to identify the following performance hotspots: 1. Most of the time was spent in org.springframework.boot.context.properties.source.SystemEnvironmentPropertyMapper#isLegacyAncestorOf. I am wondering if this behavior is still needed or if it could be removed.

It was hard for me to work around this issue as the property mappers are statically constructed and assigned in `org.springframework.boot.context.properties.source.SpringConfigurationPropertySource#from`. I ended up creating a copy of `SpringConfigurationPropertySource` and `SpringIterableConfigurationPropertySource` that use a modified `SystemEnvironmentPropertyMapper` (stripped the legacy stuff from it).

Also org.springframework.core.env.MapPropertySource#getPropertyNames was a hotspot as it seems to re-construct the property names many times during binding.

I was able to work around this by wrapping all property sources in an immutable wrapper before providing them to the binder. The wrapper evaluates the property names only once at construction time.
Another hotspot was org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.Mappings#updateMappings(java.lang.String[]) which was invoked very frequently. Most of the ConfigurationPropertySources were not immutable so the Mappings were updated every time they were accessed.

I reused the workaround from (2) to solve this by implementing org.springframework.boot.origin.OriginLookup and returning true for org.springframework.boot.origin.OriginLookup#isImmutable. At that point the binding runtime was down to < 10 seconds.
The last hotspot was in org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource#containsDescendantOf. Here the descendent relationship cache implemented in org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.Mappings is only used if this.ancestorOfCheck == PropertyMapper.DEFAULT_ANCESTOR_OF_CHECK which is not the case for org.springframework.core.env.SystemEnvironmentPropertySource. So the descendent caching is not in effect for the SystemEnvironmentPropertySource.

I decided to not create a workaround for this as it would have required further modification to the Spring implementation and the performance was at an acceptable level already. However, I assume that there would be a further substantial reduction in runtime with an improved descendent caching.

Above evaluation was done using Spring Boot 3.1.6. However, I checked the latest Spring Boot sources and there seems to have been no relevant changes to the code sections mentioned above.

Comment From: philwebb

Unfortunately the relaxed binding code continues to be a source of frustration. @Mobe91 have you tried using the ConfigurationPropertyCaching interface to enable caching globally?

Comment From: Mobe91

Thank you for the helpful pointer @philwebb , I did not know about the ConfigurationPropertyCaching interface!

By enabling caching globally using the ConfigurationPropertyCaching interface I was actually able to remove the workarounds for (2) and (3) mentioned in my earlier comment without taking a performance hit.

Unfortunately, the workaround for (1) is still needed to yield acceptable performance due to the lack of descendent caching for SystemEnvironmentPropertySource mentioned in (4).

Some numbers: * With workaround (1): ~8 seconds * Without workaround (1): ~55 seconds

Comment From: quaff

Unfortunately the relaxed binding code continues to be a source of frustration.

Could it be disabled by default since next major release?

Comment From: dmitriytsokolov

Hi @Mobe91. Thanks for the comment. Would it be possible for you to somehow provide snippets of your workarounds?

@philwebb is anyone going to take a look at the example I provided? Would you suggest anything on this matter?

Comment From: wilkinsona

@dmitriytsokolov We're a small team with many different priorities. Rest assured, we will look at the sample when we have time to do so. That may take a little while as we work on higher priority issues for 3.4.0-RC1 that releases next week. Thank you for your patience in the meantime.

Comment From: dmitriytsokolov

Of course, thanks a lot @wilkinsona!

Comment From: Mobe91

@dmitriytsokolov

Would it be possible for you to somehow provide snippets of your workarounds?

Sure, this is what it comes down to:

/**
 * Mostly a copy of {@link SpringConfigurationPropertySource} to pass different {@link PropertyMapper}s to the
 * constructed {@link ConfigurationPropertySource}. Sadly, at the time of writing Spring does not provide a way
 * to do this differently.
 */
public class CustomConfigurationPropertySourceFactory {

    private static final PropertyMapper[] DEFAULT_MAPPERS = { DefaultPropertyMapper.INSTANCE };

    private static final PropertyMapper[] SYSTEM_ENVIRONMENT_MAPPERS = {
        NonLegacySystemEnvironmentPropertyMapper.INSTANCE,
        DefaultPropertyMapper.INSTANCE };

    public static ConfigurationPropertySource from(PropertySource<?> source) {
        Assert.notNull(source, "Source must not be null");
        PropertyMapper[] mappers = getPropertyMappers(source);
        if (isFullEnumerable(source)) {
            return new SpringIterableConfigurationPropertySource((EnumerablePropertySource<?>) source,
                mappers);
        }
        return new SpringConfigurationPropertySource(source, mappers);
    }

    // need to copy some more static methods here - skipped for brevity
}

/**
 * Copy of {@link SystemEnvironmentPropertyMapper} modified to remove the handling of legacy property naming
 * <a href="https://github.com/spring-projects/spring-boot/issues/42361#issuecomment-2399941504">for performance
 * reasons</a>.
 */
class NonLegacySystemEnvironmentPropertyMapper implements PropertyMapper {

    // Get rid of isLegacyAncestorOf and all uses of it
}

Use it:

        // Make a copy of the environment's PropertySources to wrap them in ConfigurationPropertySources created
        // using our CustomConfigurationPropertySourceFactory.
        Iterable<ConfigurationPropertySource> configurationPropertySources = env.getPropertySources().stream()
            .filter(Predicate.not(ConfigurationPropertySources::isAttachedConfigurationPropertySource))
            .map(CustomConfigurationPropertySourceFactory::from)
            .toList();
        // Enabling caching for the copied ConfigurationPropertySources at this point is important for the binding
        // performance.
        ConfigurationPropertyCaching.get(configurationPropertySources).enable();
        Binder binder = new Binder(configurationPropertySources, new PropertySourcesPlaceholdersResolver(env));

Comment From: wilkinsona

I have a possible optimisation for this situation that significantly improves the startup time of the provided sample. On my machine, using Spring Boot 3.3.7, the sample starts in around 30 seconds:

2024-12-19T14:57:53.740Z  INFO 16601 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 29.585 seconds (process running for 29.994)

With the optimisation in place it's less than 7 seconds:

2024-12-19T14:54:25.793Z  INFO 15157 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 6.101 seconds (process running for 6.582)

The tests all pass with the optimisation in place so I am hopeful that it doesn't break anything functionally. I am, however, a little wary that it may regress the binder's performance in other scenarios so I'd like to discuss it with the rest of the team before we proceed.

@dmitriytsokolov If you'd like to try things out in your real-world app, it would be very interesting to know the impact that the change has. You should be able to copy the modified Binder source into your app and it'll be preferred to the one in the spring-boot jar. I'd recommend doing so with Boot 3.3.7 to avoid any other incompatibilities.

Comment From: dmitriytsokolov

Wow! Thanks a lot @wilkinsona! I will definitely try it later this/next week and let you know

Comment From: dmitriytsokolov

Btw, is there any change this fix will be merged into 2.7 as well?

Comment From: wilkinsona

Definitely not in OSS as support for 2.7 ended last month. I wouldn't completely rule out something in a release for commercial support customers but we'd have to carefully weigh up the risks and benefits.

Comment From: dmitriytsokolov

@wilkinsona I can confirm performance improvement on the test app I provided on both 3.3 and 2.7 (applied same patch) versions of spring. However I don't see any performance change in my real app. I will try to debug and see where is the difference.

Comment From: wilkinsona

Thanks for trying out the prototyped optimisation.

It will only help when there are many property sources but with certain "branches" of the properties only being present in a handful of those sources. That's the case in the sample where there are 1001 (IIRC) sources and each branch of properties is only present in one of them. Perhaps the property source and property structure is different in your real app?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: dmitriytsokolov

@wilkinsona yes, it seems this is the main difference between test and real app. In my real app I have a single property source with properties for all of the objects. It seems that it makes sense to split this prop source so that each source contains properties for one object only, doesn't it?

Comment From: wilkinsona

That would only help if we applied the prototyped optimisation. I'm not sure that we should do that as it may slow down binding in some cases while only speeding it up in the case of your test app. As far as we know, having such a large number of property sources with that particular structure is unusual and possible unique to your test app.

I think we need to take a step back and re-examine the problem with property sources and property structures that more accurately mimic your real app. Could you please share an updated version of your demo app that has the same property sources and property structure as your real app?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: dmitriytsokolov

Sure, I will prepare a new example. Sorry for the delay

Comment From: dmitriytsokolov

Here is an updated version of the test app that replicates my use case. The only change I've made - I combined all 1k entries into a single file application-config.yml and included only "config" profile. So instead of 1k sources I have only one. I confirmed that the Binder patch doesn't help in this case.

I do understand that it might be dangerous to apply this fix globally, but it would be great to have a possibility to override/extend spring binder so that I can implement this fix on my side and separate configs into multiple sources.

demo_config_props_new.zip

Comment From: philwebb

Thanks very much for the updated sample @dmitriytsokolov, it provided a very useful way to profile the current code.

I've found a few areas where we can improve things and I've opened the following issues with some fixes pushed for the next 3.5.x release:

44856
44857
44858
44859
44860
44861
44862
44863

Running your demo on 3.5.0-M3 on my local laptop a few times and I get the following:

2025-03-24T13:48:30.375-07:00  INFO 50535 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 4.352 seconds (process running for 4.488)
2025-03-24T13:48:46.174-07:00  INFO 50655 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 4.36 seconds (process running for 4.505)
2025-03-24T13:49:03.525-07:00  INFO 50805 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 4.325 seconds (process running for 4.466)

With the fixes applied and using the latest SNAPSHOT I get:

2025-03-24T13:52:56.396-07:00  INFO 52376 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 2.231 seconds (process running for 2.379)
2025-03-24T13:53:10.229-07:00  INFO 52476 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 2.053 seconds (process running for 2.193)
2025-03-24T13:53:18.941-07:00  INFO 52518 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 2.087 seconds (process running for 2.226)

I'm not sure how easy it would be to upgrade your real application, but I'd be interested to know if using the latest 3.5 SNAPSHOT removes the need for the extension points your looking for?

Comment From: dmitriytsokolov

Hi @philwebb . Thanks a lot for the feedback!

I just tried 3.5.0-M3 and it works the same as 2.7/3.1 versions:

2025-03-24T22:15:34.747+01:00  INFO 45342 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 85.613 seconds (process running for 85.982)

are you sure you haven't deleted/changed app config or file(s)?

Comment From: philwebb

Damn, you're too quick! The SNAPSHOTs for the updates haven't yet been pushed to artifactory. When this CI job passes can you please try again (running mvn -U to force a SNAPSHOT update).

Comment From: dmitriytsokolov

haha, sure, I will try later 🤣 Anyway, it seems that if the fixes work as you mentioned above - there is only one way for us to fix this: bump spring to the latest. There is no way those can be added to 2.7.x, right?

Comment From: philwebb

I'm afraid the changes are too risky to backport. 2.7 is also out of OSS support so you should really be thinking about upgrading anyway.

Comment From: dmitriytsokolov

@philwebb I just checked with the new SNAPSHOT and still see +- the same results. I'm attaching a screenshot where you can see that I'm using the recent version with your latest commit.

What I'm doing wrong?

I'm using Mac M1

Comment From: philwebb

Is this the same demo application? Can you attach another zip if so. I'll check it again when I'm back at my desk.

Comment From: dmitriytsokolov

Yes, the same (as in this message). Here it is:

demo_config_props_3.zip

Comment From: philwebb

I need to add the following to make things import correctly, but after that I still see the improvement:

  <repositories>
    <repository>
      <id>spring-milestones</id>
      <name>Spring Milestones</name>
      <url>https://repo.spring.io/milestone</url>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </repository>
    <repository>
      <id>spring-snapshots</id>
      <name>Spring Snapshots</name>
      <url>https://repo.spring.io/snapshot</url>
      <releases>
        <enabled>false</enabled>
      </releases>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <id>spring-milestones</id>
      <name>Spring Milestones</name>
      <url>https://repo.spring.io/milestone</url>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </pluginRepository>
    <pluginRepository>
      <id>spring-snapshots</id>
      <name>Spring Snapshots</name>
      <url>https://repo.spring.io/snapshot</url>
      <releases>
        <enabled>false</enabled>
      </releases>
    </pluginRepository>
  </pluginRepositories>

Can you double check that your jar has a org.springframework.boot.context.properties.bind.Binder class that includes a new cache field?

Comment From: philwebb

Is that 67 seconds figure accurate as well? Even before these changes I can run the demo application in about 4-5 seconds. Is there anything specific to your mac that might slow the application down? Antivirus for example?

Comment From: philwebb

Running the original demo zip I also see some improvements, although not as dramatic as Andy's

2025-03-24T16:42:43.072-07:00  INFO 88502 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 13.555 seconds (process running for 13.728)

2025-03-24T16:44:18.182-07:00  INFO 88536 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 20.074 seconds (process running for 20.257)

Comment From: dmitriytsokolov

yes, it has both private final Map cache = new ConcurrentReferenceHashMap<>(); and private ConfigurationPropertyCaching configurationPropertyCaching;

Numbers should be accurate. Also I don't see this slowness in other apps/services. Initially, I provided a slightly different version of the app and Andy Wilkinson got the same results as me (here)

Comment From: philwebb

Oh, it might be me. I don't have Lombok installed and I think it's not binding anything because of that. Let me dig a bit more and get someone else from the team to also try the demo.

Comment From: philwebb

Well that's disappointing. It was Lombok missing from my IDE. Back to the drawing board :(

Comment From: dmitriytsokolov

I can imagine how frustrating that must be 😢

Comment From: philwebb

I've pushed 2 more updates with #44867 and #44868, with those the app (delomboked) starts in 10.285 seconds for me (down from 22.919).

I can't find much more to optimize and I think the sheer volume of entries is probably overwhelming the Binder which wasn't really designed with this in mind.

@dmitriytsokolov Can you let me know if the updates make any difference for you?

SpringBoot Provide a possibility to extend/override configuration properties binding process

44856

44857

44858

44859

44860

44861

44862

44863