Package io.agentscope.harness.agent.memory.compaction

Class TokenCounterUtil

java.lang.Object
io.agentscope.harness.agent.memory.compaction.TokenCounterUtil
public class TokenCounterUtil extends Object
Utility class for estimating token count in messages.

This class provides methods to estimate the number of input tokens that would be consumed when sending messages to an LLM. The estimation uses a character-based approximation that works reasonably well for both English and Chinese text.

Token estimation strategy:

  • Text content: ~1 token per 2-4 characters (varies by language)
  • Tool calls: Includes tool name, parameters, and structure overhead
  • Tool results: Includes output content and structure overhead
  • Message structure: Role, name, and formatting overhead

  • Constructor Details

    • TokenCounterUtil

      public TokenCounterUtil()

  • Method Details

    • calculateToken

      public static int calculateToken(List<io.agentscope.core.message.Msg> messages)
      Calculates the estimated total input tokens for a list of messages.

      This method estimates tokens by:

      • Extracting all text content from messages
      • Counting characters in tool calls and results
      • Adding structure overhead for each message and content block
      Parameters:
      messages - the list of messages to estimate tokens for
      Returns:
      estimated number of input tokens