How can I capture or listen to the event when a tool is actually invoked? I would like to wrap some custom tags or markers before and after the tool execution.

such as :

findPeopleBen

Comment From: Jisprincess

For example, I want to wrap the tool’s name, input, and output with tags. Since some tools take a long time to execute, having these tags would allow me to detect when a tool is being used and implement a waiting behavior accordingly — similar to how tool events work in CrewAI.

Comment From: YunKuiLu

You can try my solution.

Inject your custom ObservationHandler Bean into your app.

@Configuration
public class MyObservationConfig {

    @Bean
    ToolExecuteObservationHandler toolExecuteObservationHandler() {
        return new ToolExecuteObservationHandler(List.of(ToolCallingObservationContext.class));
    }


    @Slf4j
    public static class ToolExecuteObservationHandler implements ObservationHandler {

        private final List<Class<? extends Observation.Context>> supportedContextTypes;

        public ToolExecuteObservationHandler(List<Class<? extends Observation.Context>> supportedContextTypes) {

            Assert.notNull(supportedContextTypes, "SupportedContextTypes must not be null");

            this.supportedContextTypes = supportedContextTypes;
        }

        @Override
        public boolean supportsContext(Observation.Context context) {
            return (context == null) ? false : this.supportedContextTypes.stream().anyMatch(clz -> clz.isInstance(context));
        }

        @Override
        public void onStart(Observation.Context context) {
            log.info("onStart: {}", context);
        }

        @Override
        public void onEvent(Observation.Event event, Observation.Context context) {
            log.info("onEvent: {} {}", event, context);
        }

        @Override
        public void onStop(Observation.Context context) {
            log.info("onStop: {}", context);
        }

        @Override
        public void onError(Observation.Context context) {
            log.error("onError: {}", context);
        }

    }

}

Now, you will receive onStart, onStop and other notifications when the tool is executed.

Comment From: Jisprincess

@YunKuiLu Thank you very much, but what I want is for the model tool invocation to be shown in the stream output, with tool usage wrapped in tags indicating the start and end of the invocation. I don't want it done via prompt constraints, because prompt constraints only summarize after the tool is finished — they don't reflect the tool invocation process in real time.

Flux<String> callTools = clientIng.build() .prompt("what today weather is ?") .stream() .content();

Output example: hello <tool><tool name>query weather</tool name><tool content>rain</tool content></tool> weather is rain today

Comment From: YunKuiLu

Sorry, I misunderstood your question.

You can try User-Controlled Tool Execution, although it might be a bit of a hassle and require some hands-on coding. (This document uses ChatModel as an example, rather than ChatClient, so you may need to make some adjustments.)

Hi @markpollack @ilayaperumalg. Is there a better approach to handle this case?

Comment From: YunKuiLu

@markpollack @ilayaperumalg I tried using ChatClient for "User-Controlled Tool Execution", and found it difficult. At least it requires writing a lot of code.😥

  1. The executeToolCalls method of ToolCallingManager requires a Prompt as a parameter, but there's no way for me to obtain it from ChatClient (which means I have to implement the tool execution logic myself).
  2. There is no simple method for me to get the result of tool execution from ToolExecutionResult (I need to iterate through it).
public Flux<String> tool2() {
    Flux<ChatClientResponse> response = chatClient.prompt()
            .options(ToolCallingChatOptions.builder().internalToolExecutionEnabled(false).build())
            .tools(this)
            .user("What is the weather in New York?")
            .stream()
            .chatClientResponse();

    return response.flatMap(chatClientResponse -> {
        ChatResponse chatResponse = chatClientResponse.chatResponse();
        if (chatResponse.hasToolCalls()) {
            Prompt prompt = null; // TODO I can't get Prompt from ChatClientResponse
            ToolExecutionResult toolExecutionResult = toolCallingManager.executeToolCalls(prompt, chatResponse);
            // TODO I need tool execution result but I can't get it easily
        }
        return Flux.just(chatResponse.getResult().getOutput().getText());
    });
}

Comment From: Jisprincess

thanks ,but i thinnk it not down my question。for example，many tools need to do 。 i wand get one result ,but need wait all tool finish。one tool lable can not show in Flex in time.

Comment From: Jisprincess

@YunKuiLu

Comment From: Jisprincess

In the default implementation, does it only start outputting the streaming response after all tool executions are completed and the response from the stream interface no longer requires any tool calls?

Comment From: markpollack

Hi @YunKuiLu - yea the answer to the question 'what tools were called' seems to be a fairly common one and I agree that the 'user controlled tool execution' gives you the raw hook to implement this, but it isn't a trivial task. Also I think @Jisprincess is asking for real time info on tool execution to be streamed back? I'll reopen this issue , as I think the observability solution is pretty lightweight and will meet the need of many users. also, it would be good to verify the user controlled tool execution in this context. Then we can create a documentation issue or perhaps an improvment to collect this informaiton.

Comment From: YunKuiLu

Thanks for the reply, @markpollack . I think @Jisprincess wants is: 1. use stream() mode to call AI, AI returns saying that multiple tools need to be executed, then the developer needs to send each tool's execution result to AI, and also process the tool execution information (including tool name and execution result) and return it to the client for page display (when sending to client, HTML tags need to be added before and after the tool name and execution result, for example <tool><tool_name>XXXX</tool_name><tool_content>YYYYY</tool_content></tool> ). 2. Finally, after AI gets all the execution results of the tools, it outputs the final response to the client in the same stream.

The client will render the HTML into a style similar to the following: Image

@Jisprincess I hope I didn't misunderstand.

For me, what I expect is that ToolCall can become a feature of ChatClient, or have a higher-level encapsulation, because the current user controlled tool execution is indeed a bit raw. 😀 This is just a pseudocode in my mind: 

public Flux<String> toolDemo() {
        DefaultChatClient.DefaultStreamResponseSpec streamResponseSpec = (DefaultChatClient.DefaultStreamResponseSpec) chatClient.prompt()
                // open "user controlled tool execution" mode
                .options(ToolCallingChatOptions.builder().internalToolExecutionEnabled(false).build())
                .tools(tools)
                .user("What is the weather in New York?")
                .stream();

        Flux<ChatClientResponse> response = streamResponseSpec.chatClientResponse();

        return response.flatMap(chatClientResponse -> {
            ChatClientResponse curResponse = chatClientResponse;
            while (curResponse.isToolCall()) { // ChatClient built-in supports toolCall
                return Flux.push(sink -> {
                    try {
                        // Usage Scene 1：Enhancement tool execution
                        ToolCallSpec toolCallSpec = chatClientResponse.toolCall();
                        sink.next("<tool>");

                        sink.next("<tool_id>");
                        sink.next(toolCallSpec.id()); // simple method to get tool id
                        sink.next("</tool_id>");

                        sink.next("<tool_name>");
                        sink.next(toolCallSpec.name());  // simple method to get tool name
                        sink.next("</tool_name>");

                        sink.next("<tool_description>");
                        sink.next(toolCallSpec.description()); // simple method to get tool description
                        sink.next("</tool_description>");

                        sink.next("<tool_content>");
                        ToolExecutionResult toolExecutionResult = toolCallSpec.execute(); // simple method to execute tool, not need params
                        sink.next(toolExecutionResult.getResult()); // simple method to get the tool execute result
                        sink.next("</tool_content>");

                        sink.next("</tool>");

                        // ==============

                        // Usage Scene 2：Return the tool results to the AI model （may not be implemented this simply.）
                        curResponse = toolCallSpec.execute(toolExecutionResult -> {
                            // modify tool execute result and send back to the AI model.
                            return "my tool execution result is " + toolExecutionResult.getResult();
                        });
                    } catch (Exception e) {
                        sink.error(e);
                    } finally {
                        sink.complete();
                    }
                });
            }
            return Flux.just(chatClientResponse.getContent());
        });
    }

Comment From: Jisprincess

@YunKuiLu @markpollack “public Flux toolDemo() “ is yes!!!
How can I get that effect in springai1.0.0 。

Comment From: Jisprincess

use low-level code in Spring AI 1.0.0

Comment From: YunKuiLu

Hi @Jisprincess Please keep this issue open, you can see the maintainers has already added labels and milestone for the current issue. We can share ideas and suggestions here to help the maintainers design new features for tool calling.

“public Flux toolDemo() “ is yes!!! How can I get that effect in springai1.0.0 。

This is just a pseudocode in my mind. The maintainers might have better designs for this.