Test Report Description

CPTS provides real-time and offline test reports for you to view and analyze test data anytime.

A test report includes a service report and SLA report.

The service report displays the monitored service metrics in the test case.
The SLA report displays the SLA rules configured in the test project and the rule trigger events.

Common Test Cases

Table 1 describes a test report when the case type is Normal.

This report shows the response performance of the tested system in a scenario with a large number of concurrent users. To help you understand the report, the following information is provided for reference:

Statistical dimension: In this report, RPS, response time, and concurrency are measured in a single case. If a request has multiple packets, the request is considered successful only when all the packets are responded to. The response time of the request is the sum of the response time of the packets.
Response timeout: If the corresponding TCP connection does not return the response data within the set response timeout (5s by default), the case request is counted as a response timeout. Possible causes include: the tested server is busy or crashes, or the network bandwidth is fully occupied.
Verification failure. The response packet content and response code returned from the server do not meet the expectation (the default expected response code of HTTP/HTTPS is 200), such as code 404 or 502. A possible cause is that the tested service cannot be processed normally in scenarios with a large number of concurrent users. For example, a database bottleneck happens in the distributed system or the backend application returns an error.
Parsing failure. All response packets are received, but some packets are lost. As a result, the entire case response is incomplete. This may be caused by network packet loss.
Bandwidth: This report collects statistics on the bandwidth of the execution end of CPTS. The uplink indicates the traffic sent from CPTS, and the downlink indicates the received traffic. In the external pressure test scenario, you need to check whether the EIP bandwidth of the executor meets the uplink bandwidth requirement and whether the bandwidth exceeds the downlink bandwidth of 1 GB.
RPS: RPS is the number of requests that CPTS sends to the tested server per second.
How to determine the quality of tested applications: According to the service quality definition of an application, the optimal status is that there is no response or verification failure. If there is any response or verification failure, it must be within the defined service quality range. Generally, the value does not exceed 1%. The shorter the response time is, the better. User experience is considered good if the response time is within 2s, acceptable if it is within 5s and needs optimization when it is over 5s. TP90 and TP99 can objectively reflect the response time experienced by 90% to 99% of users.

Table 1 Test report description of normal cases
Parameter	Description
Total Number of Metrics	Total number of metrics of all cases. Maximum Concurrency: indicates the maximum number of concurrent virtual users. Normal Response: indicates the number of case responses that pass the set checkpoints. If no checkpoints are set, the number of case responses returning 2XX will be calculated by default. Bandwidth: records the real-time bandwidth usage during the running of a pressure test task. Response Time: indicates the duration from the time a client sends a request until the client receives a response from the server. Abnormal Response: indicates the number of parsing failures, verification failures, response timeout, 3XX, 4XX, 5XX, and connection failures. Average RPS: indicates the average number of requests that CPTS sends to the tested server per second in a statistical period.
Response distribution statistics	Indicates the number of cases processed per second for normal return, parsing failure, verification failure, and response timeout. This metric is related to the think time, concurrent users, and server response capability. For example, if the think time is 500 ms and the response time of the last request for the current user is less than 500 ms, the user requests twice per second. TPS: indicates the number of cases that CPTS sends to the tested server per second. Normal response: indicates the number of case responses that pass the set checkpoints. If no checkpoints are set, the number of case responses with code 2XX will be calculated by default. Parsing failure: indicates the number of HTTP responses that fail to be parsed. Verification failure: indicates the number of case responses that do not pass the set checkpoints. If no checkpoints are set, the number of case responses returning contents other than 2XX will be calculated by default. Response timeout: indicates the number of case requests that do not receive responses from the server within 5 seconds after a request packet is sent. 3XX: indicates that the client needs to perform further operations to complete the request. 4XX: indicates that an error occurs on the client. As a result, the request cannot be processed. 5XX: indicates that the server cannot complete the request. Rejected: indicates the number of rejected connection requests. Others: indicates the number of other errors.
Bandwidth (kbit/s)	Records the real-time bandwidth change consumed in the running of the pressure test task. Uplink bandwidth: speed at which the CPTS execution node sends out data. Downlink bandwidth: the speed at which the CPTS execution node receives data. Current concurrency: indicates the number of current concurrent virtual users.
RPS/Average Response Time	RPS: indicates the number of requests that CPTS sends to the tested server per second. Average response time: indicates the average response time of all requests sent in a second.
Concurrent Users	Indicates the changes on the number of concurrent virtual users during testing.
Response Time Ratio	Indicates the ratio of response time of cases.
TP (ms)	If you want to calculate the top percentile XX (TPXX) for a request, collect all the response time values for the request over a time period (such as 10s) and sort them in an ascending order. Remove the top (100–XX)% from the list, and the highest value left is the value of TPXX. TP50: Remove the top 50% from the list, and the highest value left is the value of TP50. TP90: Remove the top 10% from the list, and the highest value left is the value of TP90. TP95: Remove the top 5% from the list, and the highest value left is the value of TP95. TP99: Remove the top 1% from the list, and the highest value left is the value of TP99.
Logs	Supports sampling packet capturing for requests and responses to locate problems during testing. A maximum of 100 requests and responses can be captured for each case at a time. For details, see description about logs.

SLA Reports

If you have configured SLA rules for a test task, you can check whether the SLA rules are triggered during testing in the SLA report.

Table 2 SLA report parameters
Parameter	Description
Request Name	Name of the request in the test case.
Rules	SLA rules that have been configured for the test task.
Average Value	Average metric value and the number of times that the SLA rules were triggered during testing.
Trigger Events	Request name and when and how many times the SLA rules were triggered during testing.