Logging FAQ

Indexes

How Do I Disable Logging?

Disabling container log and Kubernetes event collection

Method 1: Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Logging. In the upper right corner, click View Log Collection Policies. Then, locate and delete the corresponding log collection policy. By default, the default-event policy reports Kubernetes events, and the default-stdout policy reports stdout logs.

Method 2: Access the Add-ons page and uninstall the Cloud Native Log Collection add-on. Note that after you uninstall this add-on, it will no longer report Kubernetes events to AOM.

Disabling log collection for control plane components

Choose Logging > Control Plane Logs and deselect one or more components whose logs do not need to be collected.

Disabling Kubernetes audit log collection

Choose Logging > Kubernetes Audit Logs and deselect the component whose logs do not need to be collected.

What Can I Do If All Components Except log-operator Are Not Ready?

Symptom: All components except log-operator are not ready, and the volume failed to be mounted to the node.

Solution: Check the logs of log-operator. During add-on installation, the configuration files required by other components are generated by log-operator. If the configuration files are invalid, all components cannot be started.

The log information is as follows:

MountVolume.SetUp failed for volume "otel-collector-config-vol":configmap "log-agent-otel-collector-config" not found

How Do I Handle the Error in Stdout Logs of log-operator?

Symptom:

2023/05/05 12:17:20.799 [E] call 3 times failed, reason: create group failed, projectID: xxx, groupName: k8s-log-xxx, err: create groups status code: 400, response: {"error_code":"LTS.0104","error_msg":"Failed to create log group, the number of log groups exceeds the quota"}, url: https://lts.***.com/v2/xxx/groups, process will retry after 45s

Solution: On the LTS console, delete unnecessary log groups.

What Can I Do If Container File Logs Cannot Be Collected When Docker Is Used as the Container Engine?

Symptom:

A container file path is configured but is not mounted to the container, and Docker is used as the container engine. As a result, logs cannot be collected.

Solution:

Check whether Device Mapper is used for the node where the workload resides. Device Mapper does not support text log collection. (This restriction is displayed when you create a log collection policy.) To check this, perform the following operations:

Go to the node where the workload resides.
Run the docker info | grep "Storage Driver" command.
If the value of Storage Driver is Device Mapper, text logs cannot be collected.

What Can I Do If Container File Logs Cannot Be Collected Due to the Wildcard in the Collection Directory?

Troubleshooting: Check the volume mounting status in the workload configuration. If a volume is attached to the data directory of a service container, this add-on cannot collect data from the parent directory. In this case, you need to set the collection directory to a complete data directory. For example, if the data volume is attached to the /var/log/service directory, logs cannot be collected from the /var/log or /var/log/* directory. In this case, you need to set the collection directory to /var/log/service.

Solution: If the log generation directory is /application/logs/{Application name}/*.log, attach the data volume to the /application/logs directory and set the collection directory in the log collection policy to /application/logs/*/*.log.

What Can I Do If fluent-bit Pod Keeps Restarting?

Troubleshooting: Run the kubectl describe pod command. The output shows that the pod was restarted due to OOM. There are a large number of evicted pods on the node where the fluent-bit resides. As a result, resources are occupied, causing OOM.

Solution: Delete the evicted pods from the node.

What Can I Do If Logs Cannot Be Collected When the Node OS Is Ubuntu 18.04?

Troubleshooting: Restart the fluent-bit pod on the current node and check whether logs are properly collected. If the logs cannot be collected, check whether the log file to be collected already exists in the image during image packaging. In the container log collection scenario, the logs of the existing files during image packaging are invalid and cannot be collected. This issue is known in the community. For details, see Issues.

Solution: If you want to collect log files that already exist in the image during image packaging, you are advised to set Startup Command to Post-Start on the Lifestyle page when creating a workload. Before the pod of the workload is started, delete the original log files so that the log files can be regenerated.

What Can I Do If Job Logs Cannot Be Collected?

Troubleshooting: Check the job lifetime. If the job lifetime is less than 1 minute, the pod will be destroyed before logs are collected. In this case, logs cannot be collected.

Solution: Prolong the job lifetime.

What Can I Do If the Cloud Native Log Collection Add-on is Running Normally but Some Log Collection Policies Do Not Take Effect?

Solution:

If the log collection policy of the event type does not take effect or the add-on version is earlier than 1.5.0, check the stdout of the log-agent-otel-collector workload.
Go to the Add-ons page and click Cloud Native Log Collection. Then, click the Pods tab, locate log-agent-otel-collector, and choose More > View Log in the Operation column.
If the log collection policy of the other type does not take effect and the add-on version is later than 1.5.0, check the log of log-agent-fluent-bit on the node where the container to be monitored is running.
Go to the Add-ons page and click Cloud Native Log Collection. Then, click the Pods tab, locate log-agent-fluent-bit, and choose More > View Log in the Operation column.

Select the fluent-bit container, search for the keyword "fail to push {event/log} data via lts exporter" in the log, and view the error message.
1. If the error message "The log streamId does not exist." is displayed, the log group or log stream does not exist. In this case, choose Logging > View Log Collection Policies, edit or delete the log collection policy, and recreate a log collection policy to update the log group or log stream.
2. For other errors, go to LTS to search for the error code and view the cause.

What Can I Do If Some Pod Information Is Missing During Log Collection Due to Excessive Node Load?

When the Cloud Native Log Collection add-on version is later than 1.5.0, some pod information, such as the pod ID and name, is missing from container file logs or stdout logs.

Troubleshooting:

Go to the Add-ons page and click Cloud Native Log Collection. Then, click the Pods tab, locate log-agent-fluent-bit, and choose More > View Log in the Operation column.

Select the fluent-bit container and search for the keyword "cannot increase buffer: current=512000 requested=*** max=512000" in the log.

Solution:

Run the kubectl edit deploy -n monitoring log-agent-log-operator command on the node and add --kubernetes-buffer-size=20MB to the command lines of the log-operator container. The default value is 16MB. You can estimate the value based on the total size of pod information on the node. 0 indicates no limits.

Caution

If the Cloud Native Log Collection add-on is upgraded, you need to reconfigure kubernetes-buffer-size.

Figure 1 Modifying the command line parameter of the log-operator container

How Do I Change the Log Storage Period on Logging?

On the Clusters page, hover the cursor over the cluster name to view the current cluster ID.
Log in to the LTS console. In the navigation pane, choose Log Management. In Log Groups, select a search criterion. Then, query the log group and log stream by cluster ID.
Locate the log group and click Modify to configure the log storage period.

What Can I Do If the Log Group or Stream Specified in the Log Collection Policy Does Not Exist?

Scenario 1: The default log group or stream does not exist.
Take Kubernetes events as an example. If the default log group or stream does not exist, a message will be displayed on the Kubernetes events page of the console. You can click Create Log Collection Policy to create a log group or stream.

After the log group or stream is created, the ID of the default log group or stream changes, and the existing log collection policy of the default log group or stream does not take effect. In this case, you can rectify the fault by referring to Scenario 2.
Scenario 2: The default log group or stream exists but is inconsistent with that specified in the log collection policy.
- The log collection policy, for example, default-stdout, can be modified as follows:
  1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Logging.
  2. In the upper right corner, click View Log Collection Policies. Then, locate the log collection policy and click Edit in the Operation column.
  3. Select Custom and configure the default log group or stream.
- If a log collection policy cannot be modified, for example, default-event, you need to re-create a log collection policy as follows:
  1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Logging.
  2. In the upper right corner, click View Log Collection Policies. Then, locate the log collection policy and click Delete in the Operation column.
  3. Click Create Log Collection Policy. Then, select Kubernetes events and click OK.
Scenario 3: The custom log group (stream) does not exist.
CCE does not support the creation of non-default log groups (streams). You can create a non-default log group (stream) on the LTS console.

After the creation is complete, take the following steps:
1. Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Logging.
2. In the upper right corner, click View Log Collection Policies. Then, locate the log collection policy and click Edit in the Operation column.
3. Select Custom and configure a log group or stream.

Parent topic: O&M FAQ

Была ли эта статья полезна?

Поддержка Юридические документы