Constraints on Hudi Tables

Copy-on-write (COW) tables store data in Parquet files. Internal update operations need to be performed by rewriting the original Parquet files.

• Advantage: It is efficient because only one data file in the corresponding partition needs to be read.
• Disadvantage: During data write, a previous copy needs to be copied and then a new data file is generated based on the previous copy. This process is time-consuming. Therefore, the data read by the read request lags behind.

Merge-on-read (MOR) tables store data in a hybrid format combining columnar-based Parquet and row-based format Avro. Parquet files are used to store base data, and Avro files (also called log files) are used to store incremental data.

• Advantage: Data is written to the delta log first, and the delta log size is small. Therefore, the write cost is low.
• Disadvantage: Files need to be compacted periodically. Otherwise, there are a large number of fragment files. The read performance is poor because delta logs and old data files need to be merged.

To determine the compute engine versions supported by a queue, follow these steps: Log in to the DLI console and choose Resources > Queue Management in the navigation pane on the left. Locate the queue you want to query on the queue management page. Click the icon next to queue name to expand its details. Look for the supported versions in Supported Versions. For SQL queues, you cannot switch versions, so checking the default version will indicate the current compute engine version in use.