Bug description

When tool response is anything else, but:

  • String, that doesn't contain a number (i.e. not "22"); more about this later
  • class/record
  • array/collection of class/record

then when VertexAiGeminiChatModel tries to encode tool response to protobuf, it throws an exception similar to this:

java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Expect a map object but found: 22
    at org.springframework.ai.vertexai.gemini.VertexAiGeminiChatModel.jsonToStruct(VertexAiGeminiChatModel.java:373) ~[spring-ai-vertex-ai-gemini-1.0.1.jar:1.0.1]
    at org.springframework.ai.vertexai.gemini.VertexAiGeminiChatModel.lambda$messageToGeminiParts$1(VertexAiGeminiChatModel.java:304) ~[spring-ai-vertex-ai-gemini-1.0.1.jar:1.0.1]
...

(Where 22 is an actual tool return value of int type)

This issue is probably related to #2849 except it is about any type (not just array) that is not an object, and about calling a tool (without mcp).

In my tests returning numeric String, String[] or List<String> from a tool doesn't work too despite the fact it is presumably handled by the code.

It looks like when a tool returns a String that is numeric-like (e.g. "123"), some code converts it to a JSON number before calling jsonToStruct. But jsonToStruct currently can handle only JSON string, array (only of object types) and object, but not JSON number, boolean or null. And not array of any type other than object.

I don't know protobuf very well, but it looks like the code JsonFormat.parser().ignoringUnknownFields().merge(elementJson, elementBuilder); accepts only JSON of object type.

Environment

JDK: 21 Spring AI: 1.0.1

Steps to reproduce

public class WeatherService {

    @Tool(description = "Get the weather in location")
    public String weatherByLocation(@ToolParam(description= "City or state name") String location) {
        return "22";
    }
}

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @Bean
    public ApplicationRunner run(ChatModel chatModel) {
        return args ->  {
            String response = ChatClient.create(chatModel)
                    .prompt("What's the weather like in Boston?")
                    .tools(new WeatherService())
                    .call()
                    .content();
            System.out.println("Response: " + response);
        };
    }

}

This is basicaly an example from the docs.

Expected behavior

Any return type of the tool, that can be converted to a correct JSON, should be supported.

Note: The same code works with Ollama.

Workaround

Wrap the tool response type into a class/record, such as public record MyResponse(String temperature) {}.

Comment From: xak2000

Found a code from LangChain4J responsible for the same task. They use a workaround with enveloping the result into a JSON object if exception is thrown. They also use a second try with an additional quoting of the result (I'm not sure why).

Anyway, it is obviously a workaround and it's better to implement it properly.