How Do I Troubleshoot an Unhealthy Backend Server of a Shared Load Balancer?

Symptom

Background

• If health checks are disabled, the load balancer will consider the backend servers healthy by default and still route requests to them.
• When the weight of a backend server is set to 0:
  • Dedicated load balancer: Requests are not routed to this backend server, even if the backend server is healthy.
  • Shared load balancer: Requests are not routed to this backend server, and the health check result is unhealthy.

Troubleshooting

Checking Security Group Rules

Checking Network ACL Rules

Checking the Health Check Configuration

1. In the navigation pane on the left, choose Elastic Load Balance > Backend Server Groups.
2. On the Backend Server Groups page, locate the backend server group and click its name.
3. On the Summary page, click Health Check on the right.

   Table 4 Parameters for configuring a health check

   Parameter	Description
   Domain Name	If you use HTTP for health checks and the backend server is configured to verify the Host header, enter the domain name configured for the backend server.
   Protocol	The outbound rules of the network ACL do not allow traffic over the health check protocol.
   Port	Use the backend port as the health check port.
   Path	If HTTP is used for health checks, you must check this parameter. A simple static HTML file is recommended.

   Note

   • If the health check protocol is HTTP and the health check port is normal, change the path or change the health check protocol to TCP.
   • Enter an absolute path. The following are example paths:
     • If the URL is http://www.example.com or http://192.168.63.187:9096, enter / as the health check path.
     • If the URL is http://www.example.com/chat/try/, enter /chat/try/ as the health check path.
     • If the URL is http://192.168.63.187:9096/chat/index.html, enter /chat/index.html as the health check path.

Checking Whether the Backend Server Group Is Associated with a Listener

Checking Whether an EIP or Private IP Address Is Bound to the Load Balancer

Checking the Backend Server

• Run the following command on the backend server to check whether the health check port is listened on:

  netstat -anlp | grep port
  
  

  If the health check port and LISTEN are displayed, the health check port is in the listening state. As shown in Figure 1, TCP port 880 is listened on.

  If you do not specify a health check port, backend ports are used as the health check port by default.

  Figure 1 Backend server port listened on

  
  

  Figure 2 Backend server port not listened on

  
  

  If the health check port is not in the listening state, the backend server is not listened on. You need to start the application on the backend server and check whether the health check port is listened on.

• For HTTP health checks, run the following command on the backend server to check the status code:

  curl Private IP address of the backend server:Health check port/Health check path -iv
  
  

  To perform an HTTP health check, the load balancer initiates a GET request to the backend server. If the following response status codes are displayed, the backend server is considered healthy:

  TCP listeners: 200

  200, 202, or 401 for HTTP health checks

  Figure 3 Unhealthy backend server

  
  

  Figure 4 Healthy backend server

  
  
• If HTTP is used for health checks and the backend server is detected unhealthy, take the following step to configure a TCP health check:

  On the Listeners tab, modify the target listener, select the backend server group for which TCP health check has been configured, or add a backend server group and select TCP as the health check protocol. After you complete the configuration, wait for a while and check the health check result.

Checking the Firewall on the Backend Server

Checking the Backend Server Route

Run the following command on the backend server to check whether the default route points to the gateway (For Layer 3 communications, the default route must be configured to point to the gateway of the VPC subnet where the backend server resides):

ip route

route -n

Figure 5 Example default route pointing to the gateway

Figure 6 Example default route not pointing to the gateway

Checking the Backend Server Load

Checking the hosts.deny File