I've been using Spring Boot 4 since the M2 and I have a simple web application with spring-boot-starter-web and springdoc-openapi-starter-webmvc-api.
My own endpoints use kotlinx-serialization, but SpringDoc continue to rely on jackson for their API.
In the M3 release I had some issues with my SpringDoc endpoint, my api-docs endpoint was producing a array of integers instead of the actual payload (due to it being serialized as byte array, I'd assume). I had a quick look and figured I had to add spring-boot-starter-kotlin-serialization so I could have proper kotlinx-serialization support, and that fixed the problem. Serialization would prefer kotlinx to the annotated objects, fallback to something else if not.
Just upgraded to RC1 now, and we are back to SpringDoc producing a list of bytes in the output, but this time spring-boot-starter-kotlin-serialization is present. Maybe there's some further configuration we need to do on RC1 now, but I couldn't find it, so I'm reporting this as a bug.
Let me know if you need more details.
Comment From: wilkinsona
Thanks for trying the RC. We will need some more details, please. Can you please provide a minimal sample that behaves as you have described when upgraded from M3 to RC1? You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.
Comment From: bcmedeiros
@wilkinsona please find a repro at https://github.com/bcmedeiros/spring-boot-issue-47809
If you run it and open http://localhost:8080/api-docs, you'll see just a byte array. If you go back to M3, it works as expected.
Comment From: wilkinsona
Thanks for the repro.
The change in behaviour is due to https://github.com/spring-projects/spring-boot/commit/92ee73df3047673d7f9a253b5d71ac60010401c7. A side-effect of those changes is that the ordering of the converters has changed. With 4.0.0-M3, the following order is used:
0 = {ByteArrayHttpMessageConverter@10116}
1 = {StringHttpMessageConverter@12438}
2 = {StringHttpMessageConverter@12439}
3 = {ResourceHttpMessageConverter@12440}
4 = {ResourceRegionHttpMessageConverter@12441}
5 = {AllEncompassingFormHttpMessageConverter@12442}
6 = {KotlinSerializationJsonHttpMessageConverter@12443}
7 = {JacksonJsonHttpMessageConverter@12444}
8 = {JacksonJsonHttpMessageConverter@12445}
9 = {MappingJackson2YamlHttpMessageConverter@12446}
10 = {Jaxb2RootElementHttpMessageConverter@12447}
With 4.0.0-RC1 it has changed to the following:
0 = {KotlinSerializationJsonHttpMessageConverter@10190}
1 = {ByteArrayHttpMessageConverter@12530}
2 = {StringHttpMessageConverter@12531}
3 = {ResourceHttpMessageConverter@12532}
4 = {ResourceRegionHttpMessageConverter@12533}
5 = {AllEncompassingFormHttpMessageConverter@12534}
6 = {JacksonJsonHttpMessageConverter@12535}
7 = {MappingJackson2YamlHttpMessageConverter@12536}
8 = {Jaxb2RootElementHttpMessageConverter@12537}
Crucially, KotlinSerializationJsonHttpMessageConverter is now before ByteArrayHttpMessageConverter and the byte[] response body from SpringDoc is now written using KotlinSerializationJsonHttpMessageConverter. I think a fix for this will require Framework changes as I cannot see how Boot can add KotlinSerializationJsonHttpMessageConverter and position it after ByteArrayHttpMessageConverter.
Comment From: bcmedeiros
I see, that makes sense.
Shouldn't KotlinSerializationJsonHttpMessageConverter skip any classes that are not annotated with @kotlinx.serializarion.Serializable? This is probably in the framework as well, but it makes sense to me that we skip a converter if it doesn't apply to a given payload.
Comment From: bclozel
I think that custom message converters should be ordered ahead of auto-detected ones. There are cases where applications need to contribute a custom byte/string converter. I think this is a limitation of Kotlin serialization in general. Looking for @kotlinx.serializarion.Serializable on types is not enough, and KotlinSerializationSupport#serializer is responsible for deciding whether we should involve Kotlin serialization. I'm not sure we can refine this algorithm or if we should consider it as the expected behavior here. cc @sdeleuze
Comment From: wilkinsona
@rstoyanchev mentioned these methods on the codec side of things. Perhaps something similar for HTTP message converters would help as the Kotlin Serialization converter could then become a default converter rather than a custom converter, giving more control over its ordering relative to the byte array message converter.
Comment From: bclozel
I have a local change that does this, but I'm more and more frustrated with the unspecified behavior of the Kotlin serialization JSON converter. We're fighting against conflicting constraints.
- Kotlin Serialization should not be seen as a replacement for Jackson, but more as a first pass and then fall back on Jackson (see https://github.com/spring-projects/spring-boot/issues/47178)
- Having Kotlin Serialization on the classpath is a weak signal and applications might not want to use it at all (see https://github.com/spring-projects/spring-framework/issues/32382)
- Many common Java types (not annotated with
@Serializable) are supported by default and Kotlin Serialization will take over. For example, trying to deserialize an empty response body as aMapwill result in a failure with this converter (whereas Jackson works and returnsnull):
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: Could not read kotlinx.serialization.json.Json$Default@2c9aeec6: Expected JsonObject, but had JsonNull as the serialized body of kotlinx.serialization.Polymorphic<Map> at element: $
JSON input: null
Configuring the Kotlin Serialization as a separate converter, ahead of the Jackson one fails one of our integration tests with the exception above and I suspect this case (and others with common Java types) will happen. So, to summarize:
- setting this converter ahead of other conflicts with the byte array one
- setting it in place of the jackson one fails because this converter does not support all types applications expect
- setting it beween base converters (byte array, string, resource) and the jackson one fails because of the error above
If we apply this change, the only way out of problems like the exception above would be to manually register converters (lots of boilerplate) or remove the kotlin serialization dependency from the classpath (apparently very common dependency in our ecosystem).
Comment From: bclozel
@bcmedeiros are you using Kotlin Serialization JSON support in production? How are you managing very common use cases like Map and ProblemDetail or non-@Serializable-annotated types in your application? What would be the expected behavior and the rationale for it?
Comment From: bcmedeiros
are you using Kotlin Serialization JSON support in production?
@bclozel yes, extensively, since Spring Boot 3 early days. I even reported https://github.com/spring-projects/spring-framework/issues/33138 more than an year ago and it's a very important feature for me because I don't want to rely on reflection for serialization in most of my classes.
Kotlin Serialization should not be seen as a replacement for Jackson
To be honest, I might agree with this; I was happy with the behavior on SB3 if a class can be serialized with kotlinx-serialization, go ahead and do it, otherwise, use whatever reflection framework is available.
How are you managing very common use cases like Map and ProblemDetail or non-@Serializable-annotated types in your application? What would be the expected behavior and the rationale for it?
I haven't used ProblemDetail much, but I wouldn't be worried if Jackson or anything else was used there; kotlinx-serialization requires compile-time awareness, so it's not reasonable to expect it's supported on classes we as framework users do not write ourselves.
Likewise for Map, I know this one would have a slightly stronger case in its favor, you could actually have a controller returning a Map<String, MyKotlinSerializableClass>, but really, I think this should be the exception, not the rule, so for such a rare case we could think of some way to "force" kotlinx-serialization to be used for that particular controller or method.
Many common Java types (not annotated with @Serializable) are supported by default
You meant supported by the Kotlin serialization JSON converter, right? As I said before, I wouldn't mind that. I never expected anything I didn't annotate explicitly myself to be supported anyway, so fine by me. This was never an option before, so if it is now, I think it could be under some sort of explicit kotlinx-serialization.scope=extended kind of opt-in.
Comment From: bclozel
Thanks for the feedback @bcmedeiros
You meant supported by the Kotlin serialization JSON converter, right? As I said before, I wouldn't mind that. I never expected anything I didn't annotate explicitly myself to be supported anyway, so fine by me. This was never an option before, so if it is now, I think it could be under some sort of explicit
kotlinx-serialization.scope=extendedkind of opt-in.
This is already the behavior here, but this can be quite subtle. Spring cannot introduced a kotlinx-serialization.scope=extended flag, as here Kotlin Serialization is fully in control and tells Spring whether a specific type is supported.
I have discussed this matter with @sdeleuze and we believe this approach should work: if an application has Kotlin Serialization on the classpath, it is expected to use a dedicated converter for serializing @Serializable types (and more broadly, types supported by Kotlin Serialization). Other JSON converters are considered separately, after the Kotlin Serialization converter.
Comment From: bcmedeiros
Thanks! that makes sense, it seems this is closer to the Framework 6 behaviour if I understand it correctly. I just asked a follow-up question in the Boot ticket about the newly introduced starter module for kotlinx-serialization.
Comment From: dmitrysulman
@bclozel should the WebFlux codecs configuration be aligned with this change?
Comment From: bclozel
@dmitrysulman How so? I think we have support already with #25771
Comment From: dmitrysulman
@bclozel currently, the default codec configuration considers the Kotlin Serialization codec as an alternative to Jackson (see #34410). If I understand correctly, it is now not aligned with the HttpMessageConverters default behavior.
Comment From: bclozel
@dmitrysulman this issue changes the behavior from #34410 and aligns with WebFlux. Is there anything left to align? Your previous comment seems to say so. Am I missing something?
Comment From: dmitrysulman
@bclozel from what I understand, this issue changes the behavior of HttpMessageConverters to put the Kotlin Serialization converter ahead of other JSON converters (in particular Jackson) if Kotlin Serialization is on the classpath.
However BaseDefaultCodecs behavior remains unchanged, Kotlin Serialization is still considered as an alternative to Jackson:
https://github.com/spring-projects/spring-framework/blob/ad22a99993aeb73d713f83025b26b9ec5d74c412/spring-web/src/main/java/org/springframework/http/codec/support/BaseDefaultCodecs.java#L618-L626
And:
https://github.com/spring-projects/spring-framework/blob/ad22a99993aeb73d713f83025b26b9ec5d74c412/spring-web/src/main/java/org/springframework/http/codec/support/BaseDefaultCodecs.java#L757-L765
Sorry if I'm missing something.
Comment From: bclozel
You're right. We will discuss that as a team. It might be too late to change that behavior, especially if nobody complained so far.
Comment From: sdeleuze
@dmitrysulman FYI BaseDefaultCodecs will be refined via https://github.com/spring-projects/spring-framework/issues/35761 to allow side by side usage of Kotlin Serialization and Jackson/Gson.
Comment From: dmitrysulman
Thanks! That looks like the most complete and flexible solution.