How to Adjust the Sensitivity of Data Equalization? What Are the Impacts After the Adjustment?
To ensure smooth operation within the Doris system, it is essential to manage data disk space effectively to prevent full disks that halt processes. This involves monitoring disk usage and available space, setting various alert levels to maintain control over system operations, and avoiding scenarios where disk space is completely utilized.
Fine-Tuning Data Equalization Sensitivity
The backend (BE) regularly reports disk usage statistics to the FE every minute. The FE then uses these statistics to regulate different operational requests.
Two critical thresholds are established within the FE: the High Watermark and the Flood Stage. The Flood Stage is a more critical level than the High Watermark. When disk usage surpasses the High Watermark, Doris limits certain tasks, such as backup balancing. Should usage exceed the Flood Stage, more severe restrictions are applied, and some processes, like imports, are halted.
Additionally, the BE sets its own Flood Stage. Since the FE cannot instantly detect the BE's disk usage or control its ongoing operations, such as Compaction, the BE uses its Flood Stage to autonomously halt certain actions to safeguard itself.
Impacts
- Setting the sensitivity too low can result in significant data volume discrepancies between disks, adversely affecting concurrent performance.
- If the interval is set too long, an excessive number of junk files may accumulate, consuming valuable disk space. In such cases, it is advisable to modify the timeout interval for junk files.
- Fine-Tuning Data Equalization Sensitivity
- Impacts