nav-img
Advanced

Creating a Non-Elastic Resource Pool Queue (Discarded and Not Recommended)

Queues in the non-elastic resource pool mode are the previous-gen of resource management for DLI. It involved purchasing and releasing resources based on usage demands, requiring estimation of resource needs before making purchases.

Queues in an elastic resource pool are recommended, as they offer the flexibility to use resources with high utilization as needed. For how to buy an elastic resource pool and create queues within it, see Creating an Elastic Resource Pool and Creating Queues Within It.

Note
  • If you use a sub-account to create a queue for the first time, log in to the DLI management console using the main account and keep records in the DLI database before creating a queue.
  • It takes 6 to 10 minutes for a job running on a new queue for the first time.
  • After a queue is created, if no job is run within one hour, the system releases the queue.
  • Queues with 16 CUs do not support scale-out or scale-in.
  • Queues with 64 CUs do not support scale-in.

Notes and Constraints

Table 1 Notes and constraints on queues

Item

Description

Resource type

  • Queue types:
    • default queue: A queue named default is preset in DLI, where you can use resources on demand.
    • For SQL: Spark SQL jobs can be submitted to SQL queues.
    • For general purpose: The queue is used to run Spark programs, Flink SQL jobs, and Flink Jar jobs.
  • You cannot change the queue type once a queue is purchased. To use another queue type, purchase a new queue.

Managing queues

  • The region of a queue cannot be changed.
  • DLI queues cannot access the Internet.

Queue scaling

  • Queues with 16 CUs do not support scale-out or scale-in.
  • Queues with 64 CUs do not support scale-in.
  • Newly created queues need to run jobs before they can be scaled in or out.

Procedure

  1. You can create a queue on the Overview, SQL Editor, or Queue Management page.
    • In the upper right corner of the Overview page, click Create Queue.
    • To create a queue on the Queue Management page:
      1. In the navigation pane on the left of the DLI management console, choose Resources > Queue Management.
      2. In the upper right corner of the Queue Management page, click Create Queue to create a queue.
    • To create a queue on the SQL Editor page:
      1. In the navigation pane of the DLI management console, click SQL Editor.
      2. Click Queues. On the tab page displayed, click on the right to create a queue.
  2. On the Create Queue page displayed, set the parameters according to Table 2.
    Table 2 Parameters

    Parameter

    Description

    Name

    Queue name

    • Only numbers, letters, and underscores (_) are allowed. The value cannot contain only numbers, start with an underscore (_), or be left unspecified.
    • The value can contain a maximum of 128 characters.
    NOTE:

    The queue name is case-insensitive. Uppercase letters will be automatically converted to lowercase letters.

    Type

    • For SQL: compute resources used for SQL jobs.
    • For general purpose: compute resources used for Spark and Flink jobs.
      NOTE:

      When a dedicated queue is not in use, its resources are still reserved and not released. This means that resources are constantly being held, regardless of whether the queue is being utilized. By using a dedicated queue, resources can be guaranteed to be available whenever jobs are submitted. You can create enhanced datasource connections for a dedicated queue.

    Specifications

    The compute nodes' total number of CUs. One CU equals one vCPU and 4 GB of memory. DLI automatically assigns CPU and memory resources to each compute node, and the client does not need to know how many compute nodes are being used.

    Description

    Description of the queue to be created. The value can contain a maximum of 128 characters.

    Advanced Settings

    In the Queue Type area, select Dedicated Resource Mode and then click Advanced Settings.

    • Default: The system automatically configures the parameter.
    • Custom

      CIDR Block: Enter a CIDR block range. If DLI enhanced datasource connection is used, the CIDR block of the DLI queue cannot overlap with that of the data source.

      Queue Type: When running an AI-related SQL job, select AI-enhanced. When running other jobs, select Basic.

  3. Click Create Now to create a queue.

    Once created, you can check and use it on the Queue Management page.

    Note

    It takes 6 to 10 minutes for a job running on a new queue for the first time.