You can submit programs developed by yourself to MRS to execute them, and obtain the results.
This section describes how to submit a job (take a MapReduce job as an example) on the MRS console. MapReduce jobs are used to submit JAR programs to quickly process massive amounts of data in parallel and create a distributed data processing and execution environment.
If the job and file management functions are not supported on the cluster details page, submit the jobs in the background.
Before creating a job, you need to upload local data to OBS for data computing and analyzing. MRS allows exporting data from OBS to HDFS for computing and analyzing. After the data analysis and computing are completed, you can store the data in HDFS or export them to OBS. HDFS and OBS can also store the compressed data in the format of bz2 or gz.
If the IAM username contains spaces (for example, admin 01), a job cannot be created.
Submitting a Job on the GUI
- Log in to the MRS console.
- On the Active Clusters page, select a running cluster and click its name to switch to the cluster details page.
- If Kerberos authentication is enabled for the cluster, perform the following steps. If Kerberos authentication is not enabled for the cluster, skip this step.
In the Basic Information area on the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.
- Click the Jobs tab.
- Click Create. The Create Job dialog box is displayed.
- In Type, select MapReduce. Configure other job information.
Table 1 Job parameters Parameter
Description
Name
Job name. It contains 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.
NOTE:You are advised to set different names for different jobs.
Program Path
Path of the program package to be executed. The following requirements must be met:
- Contains a maximum of 1,023 characters, excluding special characters such as ;|&><'$. The parameter value cannot be empty or full of spaces.
- The path of the program to be executed can be stored in HDFS or OBS. The path varies depending on the file system.
- OBS: The path starts with obs://. Example: obs://wordcount/program/xxx.jar
- HDFS: The path must start with /user.
- For SparkScript and HiveScript, the path must end with .sql. For MapReduce, the path must end with .jar. For Flink and SparkSubmit, the path must end with .jar or .py. The .sql, .jar, and .py are case-insensitive.
Parameters
(Optional) It is the key parameter for program execution. Separate multiple parameters with space.
Configuration method: Program class name Data input path Data output path
- Program class name: It is specified by a function in your program. MRS is responsible for transferring parameters only.
- Data input path: Click HDFS or OBS to select a path or manually enter a correct path.
- Data output path: Enter a directory that does not exist.
The value can contain a maximum of 150,000 characters, including special characters (;|&'$), but cannot contain > or <. This parameter can also be left blank.
CAUTION:If you enter a parameter with sensitive information (such as the login password), the parameter may be exposed in the job details display and log printing. Exercise caution when performing this operation.
Service Parameters
(Optional) Used to modify service configuration parameters for the job to be executed. The parameter modification applies only to the job to be executed.
To add multiple parameters, click
on the right. To delete a parameter, click Delete on the right.Table 2 describes the common parameters of a service.
Command Reference
Command submitted to the background for execution when a job is submitted.
Table 2 Service configuration parameters Parameter
Description
Example Value
fs.obs.access.key
Key ID for accessing OBS.
-
fs.obs.secret.key
Key corresponding to the key ID for accessing OBS.
-
- Confirm job configuration information and click OK.
After the job is created, you can manage it.