Using DLI to Submit a Spark Jar Job

Scenario

Procedure

Step 1: Upload Data to OBS

1. Log in to the DLI console.
2. In the service list, click Object Storage Service under Storage.
3. Create a bucket. In this example, name it dli-test-obs01.
   1. On the displayed Buckets page, click Create Bucket in the upper right corner.
   2. On the displayed Create Bucket page, enter the Bucket Name. Retain the default values for other parameters or set them as required.
      Note

      Select a region that matches the location of the DLI console.

   3. Click Create Now.
4. In the bucket list, click the name of the dli-test-obs01 bucket you just created to access its Objects tab.
5. Click Upload Object. In the dialog box displayed, drag or add files or folders, for example, spark-examples.jar, to the upload area. Then, click Upload.

   In this example, the path after upload is obs://dli-test-obs01/spark-examples.jar.

   For more operations on the OBS console, see Object Storage Service User Guide.

Step 2: Create an Elastic Resource Pool and Add Queues to the Pool

In this example, the elastic resource pool dli_resource_pool and queue dli_queue_01 are created.

1. Log in to the DLI management console.
2. In the navigation pane on the left, choose Resources > Resource Pool.
3. On the displayed page, click Buy Resource Pool in the upper right corner.
4. On the displayed page, set the parameters.

   Table 2 describes the parameters.

   Table 2 Parameters

   Parameter	Description	Example Value
   Region	Select a region where you want to buy the elastic resource pool.	_
   Project	Project uniquely preset by the system for each region	Default
   Name	Name of the elastic resource pool	dli_resource_pool
   Specifications	Specifications of the elastic resource pool	Standard
   CU Range	The maximum and minimum CUs allowed for the elastic resource pool	64-64
   CIDR Block	CIDR block the elastic resource pool belongs to. If you use an enhanced datasource connection, this CIDR block cannot overlap that of the data source. Once set, this CIDR block cannot be changed.	172.16.0.0/19
   Enterprise Project	Select an enterprise project for the elastic resource pool.	default

5. Click Buy.
6. Click Submit.
7. In the elastic resource pool list, locate the pool you just created and click Add Queue in the Operation column.
8. Set the basic parameters listed below.

   Table 3 Basic parameters for adding a queue

   Parameter	Description	Example Value
   Name	Name of the queue to add	dli_queue_01
   Type	Type of the queue To execute SQL jobs, select For SQL. To execute Flink or Spark jobs, select For general purpose.	_
   Enterprise Project	Select an enterprise project.	default

9. Click Next and configure scaling policies for the queue.

   Click Create to add a scaling policy with varying priority, period, minimum CUs, and maximum CUs.

   Table 4 Scaling policy parameters

   Parameter	Description	Example Value
   Priority	Priority of the scaling policy in the current elastic resource pool. A larger value indicates a higher priority. In this example, only one scaling policy is configured, so its priority is set to 1 by default.	1
   Period	The first scaling policy is the default policy, and its Period parameter configuration cannot be deleted or modified. The period for the scaling policy is from 00 to 24.	00–24
   Min CU	Minimum number of CUs allowed by the scaling policy	16
   Max CU	Maximum number of CUs allowed by the scaling policy	64

10. Click OK.

Step 3: Use DEW to Manage Access Credentials

This part introduces how to create a shared secret in DEW.

1. Log in to the DEW management console.
2. In the navigation pane on the left, choose Cloud Secret Management Service > Secrets.
3. On the displayed page, click Create Secret. Set basic secret information.

   Set the AK and SK credential key-value pairs.

   • In this example, the key in the first line is the user's access key ID (AK).
   • In this example, the key in the second line is the user's secret access key (SK).
4. Set access credential parameters on the DLI Spark Jar job editing page.

   spark.hadoop.fs.obs.bucket.USER_BUCKET_NAME.dew.access.key= USER_AK_CSMS_KEY_obstest1 
   spark.hadoop.fs.obs.bucket.USER_BUCKET_NAME.dew.secret.key= USER_SK_CSMS_KEY_obstest1
   spark.hadoop.fs.obs.security.provider=com.dli.provider.UserObsBasicCredentialProvider
   spark.hadoop.fs.dew.csms.secretName=obsAkSkspark.hadoop.fs.dew.endpoint=kmsendpoint
   spark.hadoop.fs.dew.csms.version=v3spark.dli.job.agency.name=agencyname
   
   

Step 4: Create a Custom Agency to Allow DLI to Access DEW and Read Credentials

1. Log in to the management console.
2. In the upper right corner of the page, hover over the username and select Identity and Access Management.
3. In the navigation pane of the IAM console, choose Agencies.
4. On the displayed page, click Create Agency.
5. On the Create Agency page, set the following parameters:
   • Agency Name: Enter an agency name, for example, dli_dew_agency_access.
   • Agency Type: Select Cloud service.
   • Cloud Service: This parameter is available only when you select Cloud service for Agency Type. Select Data Lake Insight (DLI) from the drop-down list.
   • Validity Period: Select Unlimited.
   • Description: You can enter Agency with OBS OperateAccess permissions. This parameter is optional.
6. Click Next.
7. Click the agency name. On the displayed page, click the Permissions tab. Click Authorize. On the displayed page, click Create Policy.
8. Configure policy information.
   1. Enter a policy name, for example, dli-dew-agency.
   2. Select JSON.
   3. In the Policy Content area, paste a custom policy.

      {
          "Version": "1.1",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": [
                      "csms:secretVersion:get",
                      "csms:secretVersion:list",
                      "kms:dek:decrypt"
                  ]
              }
          ]
      }
      
      

   4. Enter a policy description as required.
9. Click Next.
10. On the Select Policy/Role page, select Custom policy from the first drop-down list and select the custom policy created in 8.
11. Click Next. On the Select Scope page, set the authorization scope. In this example, select All resources.
12. Click OK.

    It takes 15 to 30 minutes for the authorization to be in effect.

Step 5: Submit a Spark Job

1. On the DLI management console, choose Job Management > Spark Jobs in the navigation pane on the left. On the displayed page, click Create Job in the upper right corner.
2. Set the following Spark job parameters:
   • Queue: Select the queue created in Step 2: Create an Elastic Resource Pool and Add Queues to the Pool.
   • Spark Version: Select a Spark engine version. In this example, version 3.3.1 is selected.
   • Application: Select the package created in Step 1: Upload Data to OBS.
   • Agency: Select the agency created in Step 4: Create a Custom Agency to Allow DLI to Access DEW and Read Credentials, which is used to allow DLI to access the credentials stored in DEW.

   For other parameters, refer to the description about the Spark job editing page in "Creating a Spark Job" in the Data Lake Insight User Guide.

3. Click Execute in the upper right corner of the Spark job editing window, read and agree to the privacy agreement, and click OK. Submit the job. A message is displayed, indicating that the job is submitted.

4. (Optional) Switch to the Job Management > Spark Jobs page to view the status and logs of the submitted Spark job.