Viewing Kafka Rebalancing Logs

Notes and Constraints

• Rebalancing logs are stored for seven days by default. To retain them longer, see Modifying a Log Group.
• Enabling rebalancing logging will create a log group, log stream, and dashboard in LTS.
• When Kafka instances store rebalancing logs in one log group and log stream, the log group and stream of each instance contain the rebalancing logs of all the instances.

Prerequisites

• Ensure that you have permissions to create log groups and log streams in LTS.
• Rebalancing logging can be enabled or disabled only when the Kafka instance is in the Running state.

Enabling Rebalancing Logging

1. Log in to the console.
2. Click in the upper left corner to select the region where your instance is located.
3. Click in the upper left corner and choose Application > Distributed Message Service for Kafka to open the Kafka instance list.
4. Click the desired instance to go to the instance details page.
5. In the navigation pane, choose Analysis & Diagnosis > Rebalancing Logs.
6. Click Enable Logging. The Enable Logging dialog box is displayed.
7. Click OK. The Configure Logs dialog box is displayed.
8. Determine whether to enable this function as required. Click OK. The Background Tasks page is displayed. The rebalancing log function is enabled when the Enable logging task is in the Successful state.
   • Disabled: LTS automatically creates a log group and a log stream.
   • Enabled: Select the log group and log stream that store the coordinator.log file. To view or create a log group and log stream, click View Log Group on the right to go to the LTS console.

Viewing Rebalancing Logs

1. Log in to the console.
2. Click in the upper left corner to select the region where your instance is located.
3. Click in the upper left corner and choose Application > Distributed Message Service for Kafka to open the Kafka instance list.
4. Click the desired instance to go to the instance details page.
5. In the navigation pane, choose Analysis & Diagnosis > Rebalancing Logs.
6. On the Dashboard tab, view how many times and why consumer groups are rebalanced. On the Log Search tab, view rebalancing logs.

   To search for logs, see Accessing the Log Search Page.

   An example rebalancing log:

   {
       "level":"INFO",
       "timestamp":"2023-03-23 17:23:22,906",
       "message":{
           "leaderId":"consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635",
           "generationId":"1",
           "reason":"Assignment received from leader for group KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0 for generation 1",
           "groupId":"KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0",
           "coordinatorId":"0",
           "type":"END_REBALANCE",
           "group":"GroupMetadata(groupId=KMOffsetCache-dms-vm-fa3cf9d6-manager-shared-server-0, generation=1, protocolType=Some(consumer), currentState=CompletingRebalance, members=Map(consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635 -> MemberMetadata(memberId=consumer-1-177817b6-1f29-4717-8a83-dda8eaab1635, clientId=consumer-1, clientHost=/172.31.2.168, sessionTimeoutMs=10000, rebalanceTimeoutMs=300000, supportedProtocols=List(range), )))"
       }
   }
   
   

   Table 1 describes the parameters.

   Table 1 Rebalancing parameters

   Parameter	Description
   level	Level of the rebalancing logs. The only value is INFO.
   timestamp	Time of rebalancing.
   leaderId	Leader consumer ID.
   generationId	Generation ID of the consumer group. Generation is the number of times that a consumer group performs rebalancing. It is incremented by 1 each time a rebalancing is complete.
   reason	Reason for triggering rebalancing.
   groupId	Consumer group ID.
   coordinatorId	Broker where the Coordinator component is.
   type	Operation that triggered rebalancing. Values: JOIN_GROUP: A new consumer is added to a consumer group. OVER_CAPACITY: The group limit is exceeded. UPDATE_MEMBER: The consumer metadata is updated. PROTOCOL_CHANGE: The protocol is changed. HEARTBEAT_EXPIRED: The consumer heartbeat timed out. SYNC_GROUP: The reassignment plan is synchronized. END_REBALANCE: Rebalancing ended. LEAVE_GROUP: A consumer left a consumer group. DELETE_GROUP: A user deletes a consumer group.
   group	Information about consumers in the consumer group.

Disabling Rebalancing Logging

1. Log in to the console.
2. Click in the upper left corner to select the region where your instance is located.
3. Click in the upper left corner and choose Application > Distributed Message Service for Kafka to open the Kafka instance list.
4. Click the desired instance to go to the instance details page.
5. In the navigation pane, choose Analysis & Diagnosis > Rebalancing Logs.
6. Click Disable Logging in the upper right corner. The Disable Logging dialog box is displayed.
7. Click OK. The Background Tasks page is displayed. The rebalancing log function is disabled when the rebalancing logging task is in the Successful state.

   This only disables the rebalancing logging function. The log groups and log streams on LTS are retained and still generate fees. If you no longer need the logs, delete the log groups and log streams.