Feature Request: Tool Calling Pattern Analysis and Optimization

Expected Behavior

I would like Spring AI to provide enhanced observability for tool calling patterns, including:

  1. Tool Call Sequence Tracking: Track which tools are called in sequence during AI conversations
  2. Pattern Analysis: Identify common tool calling patterns and combinations
  3. Redundant Call Detection: Detect when the same tool is called multiple times with similar parameters
  4. Performance Insights: Provide metrics on tool calling efficiency and optimization opportunities

Example API:

@Component
public class ToolCallPatternAnalyzer {

    // Get tool usage statistics
    public ToolUsageStatistics getToolUsageStats(TimeRange timeRange);

    // Get tool call sequences (which tools are called together)
    public List<ToolCallSequence> getToolSequences(String conversationId);

    // Get most common tool combinations
    public List<ToolCombination> getCommonCombinations();

    // Detect redundant calls
    public List<RedundantCall> detectRedundantCalls(String conversationId);
}

// Data structures
public class ToolUsageStatistics {
    private Map<String, Integer> toolCallCounts;        // tool name -> call count
    private Map<String, Double> toolSuccessRates;       // tool name -> success rate
    private Map<String, Duration> avgExecutionTimes;    // tool name -> avg time
    private List<ToolPair> frequentCombinations;        // tools used together
}

public class ToolCallSequence {
    private String conversationId;
    private List<ToolCall> sequence;
    private Duration totalTime;
}

public class ToolCombination {
    private List<String> toolNames;
    private int frequency;
    private double coOccurrenceRate;
}

// Usage example
@RestController
public class ToolAnalyticsController {

    @Autowired
    private ToolCallPatternAnalyzer analyzer;

    @GetMapping("/tool-usage-stats")
    public ToolUsageStatistics getUsageStats() {
        return analyzer.getToolUsageStats(TimeRange.LAST_7_DAYS);
    }

    @GetMapping("/tool-combinations")
    public List<ToolCombination> getCommonCombinations() {
        return analyzer.getCommonCombinations();
    }
}

Current Behavior

Spring AI currently provides basic observability through: - spring.ai.tool observations for completion time and tracing - spring.ai.chat.client observations for ChatClient calls - Basic Micrometer metrics integration

However, there's no way to: - Analyze tool calling patterns across conversations - Identify optimization opportunities in tool usage - Detect redundant or inefficient tool calls - Understand how tools are used together in real scenarios

Context

How has this issue affected you? When building AI applications with multiple tools, it's difficult to understand: - Which tools are actually being used effectively - Whether the AI is making redundant calls - How to optimize tool configurations - What tool combinations work best together

What are you trying to accomplish? - Optimize AI application performance by understanding tool usage patterns - Identify and eliminate redundant tool calls - Improve tool design based on actual usage data - Provide developers with insights into AI behavior for debugging and optimization

What other alternatives have you considered? - Custom logging and analysis (requires significant manual work) - External monitoring tools (lack Spring AI integration) - Basic metrics collection (doesn't provide pattern insights)

Are you aware of any workarounds? Currently, developers must implement custom tracking by: - Adding manual logging to each tool - Parsing logs to identify patterns - Building custom analytics dashboards - Manually correlating tool calls across conversations

This feature would provide a standardized, framework-integrated solution that all Spring AI users could benefit from.

Comment From: cwangg897

@ilayaperumalg Hi, I’d like to work on this enhancement. Is it okay if I take this?

Comment From: ilayaperumalg

@cwangg897 Thank you for your interest in contributing. Please allow us some time to review your proposal and we can engage some discussion before you start implementing it.

Comment From: cwangg897

@cwangg897 Thank you for your interest in contributing. Please allow us some time to review your proposal and we can engage some discussion before you start implementing it.

@ilayaperumalg Hello, just following up after some time — should I go ahead and share my idea here, or would it be better to wait for your response?

Comment From: ilayaperumalg

@cwangg897 You are very much welcome to share your ideas here. It is just that we don't want you to spend time working on a PR and going back and forth.

Comment From: cwangg897

I'd like to approach this step by step.

@ilayaperumalg I’d like to respectfully share an idea that I believe could enhance the project. Please let me know your thoughts or feedback.

Proposed Implementation for Tool Call Sequence Tracking

I'd like to propose an implementation approach for the Tool Call Sequence Tracking feature. Here's my plan:

Approach

I propose to integrate the sequence tracking directly into the DefaultToolCallingManager.executeToolCalls() method where the for-each loop iterates through tool calls. The idea is to capture tool call combinations like WeatherTool->LocationTool->RecommendTool and track how many times each sequence pattern occurs.

Implementation Strategy

  1. Create a dedicated ToolCallSequenceTracker component: 
@Component
@ConditionalOnClass({ObservationRegistry.class, MeterRegistry.class})
@ConditionalOnProperty(value = "spring.ai.tool.sequence.tracking.enabled", havingValue = "true", matchIfMissing = true)
public class ToolCallSequenceTracker {

    private final MeterRegistry meterRegistry;

    // In-memory storage for tool call sequences
    private final Map<String, AtomicLong> sequencePatterns = new ConcurrentHashMap<>();
    private final Map<String, List<String>> conversationSequences = new ConcurrentHashMap<>();

    public ToolCallSequenceTracker(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
    }

    /**
     * Record a tool call in the current conversation sequence
     */
    public void recordToolCall(String conversationId, String toolName) {
        conversationSequences.computeIfAbsent(conversationId, k -> new ArrayList<>())
                             .add(toolName);
    }

    /**
     * Finalize and analyze the sequence for a conversation
     */
    public void finalizeSequence(String conversationId) {
        List<String> sequence = conversationSequences.remove(conversationId);
        if (sequence != null && !sequence.isEmpty()) {
            String pattern = String.join("->", sequence);

            // Update pattern count
            sequencePatterns.computeIfAbsent(pattern, k -> new AtomicLong(0))
                           .incrementAndGet();

            // Record metric
            meterRegistry.counter("spring.ai.tool.sequence.pattern", 
                                "pattern", pattern)
                        .increment();
        }
    }

    /**
     * Get all recorded sequence patterns and their frequencies
     */
    public Map<String, Long> getSequencePatterns() {
        return sequencePatterns.entrySet().stream()
                              .collect(Collectors.toMap(
                                  Map.Entry::getKey,
                                  entry -> entry.getValue().get()
                              ));
    }
}

2. Integration point in DefaultToolCallingManager:

public class DefaultToolCallingManager implements ToolCallingManager {

    private final ToolCallSequenceTracker sequenceTracker; // Optional dependency

    public DefaultToolCallingManager(/* existing dependencies */, 
                                   @Autowired(required = false) ToolCallSequenceTracker sequenceTracker) {
        // existing initialization
        this.sequenceTracker = sequenceTracker;
    }

    private List<Message> executeToolCalls(List<ToolCall> toolCalls, String conversationId) {
        List<Message> toolCallResultMessages = new ArrayList<>();

        // Track sequence if tracker is available
        if (sequenceTracker != null) {
            for (ToolCall toolCall : toolCalls) {
                sequenceTracker.recordToolCall(conversationId, toolCall.name());
            }
        }

        // Existing tool execution logic
        for (ToolCall toolCall : toolCalls) {
            // ... existing implementation
        }

        // Finalize sequence tracking
        if (sequenceTracker != null) {
            sequenceTracker.finalizeSequence(conversationId);
        }

        return toolCallResultMessages;
    }
}

Key Benefits

  1. Non-intrusive: Only activates when Micrometer dependencies are present
  2. Memory-efficient: Stores patterns as strings with counters
  3. Observable: Integrates with existing Micrometer metrics
  4. Optional: Uses @Autowired(required = false) to avoid breaking existing setups
  5. Pattern-focused: Captures actual tool usage sequences like WeatherTool->LocationTool->RecommendTool