Logstash Cluster Planning Suggestions
Planning Cluster AZs
By deploying a cluster across multiple AZs, you can increase the cluster's availability, lower the likelihood of data loss, and minimize service downtime. You can select two different AZs in the same region to deploy a cluster.
If you select two AZs when creating a cluster, CSS automatically enables cross-AZ HA, ensuring cluster nodes distribute evenly across both AZs. Even distribution of cluster nodes across AZs means the difference between node quantities in different AZs does not exceed 1. For details, see Table 1.
- When creating a multi-AZ cluster, select at least two nodes. Otherwise, multi-AZ cluster deployment will fail.
- When a multi-AZ cluster is deployed, nodes are evenly distributed across different AZs.
Node Quantity | Single AZ | Two AZs | |
|---|---|---|---|
AZ1 | AZ1 | AZ2 | |
1 | 1 | Not supported | |
2 | 2 | 1 | 1 |
3 | 3 | 2 | 1 |
4 | 4 | 2 | 2 |
... | ... | ... | ... |
Planning Node Storage
- Planning node models
Logstash clusters do not need a large storage capacity, so you are advised to choose compute-intensive ECSs.
- Planning node specifications
Given the expected data handling capacities, it is always preferable to use a smaller number of nodes with larger specifications rather than a larger number of nodes with smaller specifications. For example, a cluster consisting of three nodes each with 32 CPU cores and 64 GB memory is usually better than a cluster consisting of 12 nodes each with 8 CPU cores and 16 GB memory in terms of stability and scalability.
The specific advantages are as follows:
- Cluster stability: High-specs nodes provide more powerful data processing capabilities and larger memory space, leading to higher overall cluster stability.
- Improved scalability: When a cluster consisting of high-specs nodes encounters a performance bottleneck, you simply add more of these high-specs nodes. This is easier than increasing the specifications of existing nodes.
- Easier maintenance: A smaller number of nodes means easier maintenance and less complex management.
In contrast, when a cluster consisting of low-specs nodes needs extra capacity, usually a vertical scale-up is performed, meaning to increase the specifications of existing nodes. This may entail not only more complex, challenging migration and upgrade processes, but also additional maintenance costs.
To sum up, when planning a cluster, you must fully consider performance, costs, maintenance, and scalability, and choose the node specifications that best suit your needs.
- Planning storage capacity
The default node storage capacity for a Logstash cluster is 40 GB and cannot be changed.
- Planning Cluster AZs
- Planning Node Storage