Jackson Databind Reduce/eliminate introspection on java.* classes

Is your feature request related to a problem? Please describe.

This is a follow-up to this discussion in the Quarkus tracker: https://github.com/quarkusio/quarkus/issues/45537#issuecomment-2587715601 .

In the Quarkus project, @zakkak is leading an effort to allow throwing exceptions when GraalVM/Mandrel tries to do reflection calls on elements that were not registered for reflection. This effort leads to all sort of discoveries and in the issue above we discovered that Jackson tries to collect information on java.* classes, which should probably be avoided: these classes can't be annotated with Jackson classes and they are well known so we could probably hardcode what's needed.

We are having this issue with at least AnnotatedFieldCollector/AnnotatedMethodCollector/AnnotatedCreatorCollector. There might be others.

Examples of this issue with stacktraces:

org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.util.Date.getDeclaredFields()

without it being registered for runtime reflection. Add java.util.Date.getDeclaredFields() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredFields(DynamicHub.java:1251)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector._findFields(AnnotatedFieldCollector.java:73)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector._findFields(AnnotatedFieldCollector.java:71)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collect(AnnotatedFieldCollector.java:48)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredFields()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredFields() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredFields(DynamicHub.java:1251)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector._findFields(AnnotatedFieldCollector.java:73)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collect(AnnotatedFieldCollector.java:48)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collectFields(AnnotatedFieldCollector.java:43)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredMethods()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:49)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.util.Date.getDeclaredMethods()

without it being registered for runtime reflection. Add java.util.Date.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.io.Serializable.getDeclaredMethods()

without it being registered for runtime reflection. Add java.io.Serializable.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.lang.Cloneable.getDeclaredMethods()

without it being registered for runtime reflection. Add java.lang.Cloneable.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.lang.Comparable.getDeclaredMethods()

without it being registered for runtime reflection. Add java.lang.Comparable.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredConstructors()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredConstructors() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredConstructors(DynamicHub.java:1264)
  com.fasterxml.jackson.databind.util.ClassUtil.getConstructors(ClassUtil.java:1345)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialConstructors(AnnotatedCreatorCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:70)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredMethods()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  java.base@23.0.1/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialFactories(AnnotatedCreatorCollector.java:197)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:71)

Describe the solution you'd like

Ideally, we should avoid dynamically resolving this information for java.* classes.

Usage example

No response

Additional context

No response

Comment From: cowtowncoder

Hmmmh. Ok, now I remember the challenge: although there is existing mechanism to avoid annotation introspection for JDK classes (although only applied to JDK collection types), actual introspection of declared fields etc is still performed. So that'd need to be changed and it's bigger change.

Preventing this traversal would also prevent use of mix-in annotations on JDK types. I'll probably make this a MapperFeature but one that is opt-out (i.e. default being to skip introspection).

Comment From: cowtowncoder

Another question: once I get to implement something, is there some tool I could use to see if access is violated? It'd be good for me to verify changes locally before asking others to check.

Comment From: cowtowncoder

Ok so just removing --add-opens from pom.xml will not expose these (although it does trigger 4 failures, one for EnumSet another for EnumMap type detection, 2 others for trying to force access to JDK internal collections -- I fixed 2 others that happened).

Is there maybe another compiler option to get warnings? Or is this with Graal processing?

@gsmet I'd be happy to work a bit on this now but need to see warnings to verify calls are no longer made. (ideally there'd be tests too of course but first things first).

Comment From: gsmet

@cowtowncoder sorry, things have been quite chaotic lately. Let me try to assemble a simple reproducer for you.

Comment From: gsmet

@cowtowncoder the feedback loop is not exactly fast as it requires building a native executable but the following reproducer reproduces the issue:

jackson-reproducer.zip

See the README in it for detailed instructions. Let me know if you encounter any issue.

You should be able to override Jackson in the POM to point to a SNAPSHOT.

Comment From: cowtowncoder

Thank you @gsmet !

Comment From: cowtowncoder

Ok after couple of small tweaks (figuring out dir ref to GraalVM etc), got build going. So I think this is fine.

EDIT: and after building, running service with given settings does indeed report expected warnings.

Comment From: cowtowncoder

@gsmet Trying to work on this again, now that 2.19.0 is released. Noticing that I can only build against 2.18.*; attempts to building against 2.19(.0) fail due to one thing added to jackson-annotations in 2.19.

Would it be easy to indicate if I can make local modifications to reproducer (aside from jackson-databind version in pom.xml) -- or if need be, create new reproducer zip? Apologies for extra work, but I'd like to verify if my changes help -- now that I can see what specifically is printing warnings.

Comment From: cowtowncoder

Quick note: been investigating this, learning much more about the challenges.

There are multiple ways to (try to) achieve this, f.ex:

Reduce introspection of accessors at low-level, for JDK types. Could exclude Fields without problems, but for Methods not doable without allow-listing some types (getters relied on for some serialization of StackTraceElement, ThreadGroup, as-POJO-serialization for Optional -- see #5119 )
Try to do (de)serializer lookup by providers/defaults before (most?) introspection, to prevent introspection for "known" JDK types (ones with handlers)

Of these, both seem difficult to safely do for 2.x, so I will focus more on 3.0; but if it happened that a safe set of changes were found, would backport into 2.x (2.20).

FWTW, I am thinking (2) would be most beneficial route but we'll see.

Comment From: cowtowncoder

Ok some incremental progress, for 3.0 (.0-rc5): now for deserialization side, full introspection is avoided for deserialization of "well-known" types (ones for which Jackson or modules has explicitly registered deserializer) such as java.sql.Date used in reproduction.

For serialization calls are deferred as well, with the unfortunate exception for @JsonValue introspection which must be done for accessors (Fields, Getters). To avoid that, different approach is needed; will need to think about that.

Note, though, that there is parallel effort via #5149 which would:

Allow for explicit disabling of Field/Method/Constructor introspection for specific categories of classes (like Core JDK) AND
Would be implemented for 2.x (2.20)

and which I hope to work on. But haven't had much progress yet (at idea/design stage).

Comment From: cowtowncoder

Phew! I figured out how to order things to avoid Method introspection for serialization of explicitly supported types: as I mentioned earlier this is for 3.0.0. Will mark this as closed.

Comment From: cowtowncoder

@gsmet Given that work to re-order things is based on working around specific type (java.sql.Date in this case), there are likely other paths that would still trigger same issue. So I would be interesting in finding more if you happen to have time. These can be in 2.x (whichever version); specifically thinking of Lists and Maps -- scalar types are probably covered.