Ingesting Data Using Open-Source OpenSearch APIs

On OpenSearch Dashboards, you can run POST commands to import data using an open-source OpenSearch API.

Applicable scenarios: development and debugging, small ad-hoc data writes, and syntax verification.

1. Log in to the OpenSearch Dashboards.
   1. Log in to the CSS management console.
   2. In the navigation pane on the left, choose Clusters > OpenSearch.
   3. In the cluster list, find the target cluster, and click Dashboards in the Operation column to log in to OpenSearch Dashboards.
   4. In the left navigation pane, choose Dev Tools.

      The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

2. (Optional) Create an index and specify a custom mapping to define field data types.

   For example, run the following command to create index my_store:

   PUT /my_store
   {
       "settings": {
           "number_of_shards": 1
       },
       "mappings": {
           "properties": {
               "productName": {
                   "type": "text"
               },
               "size": {
                   "type": "keyword"
               }
           }
       }
   }
   
   

3. Write data to the index.

   For example, run the following command to write a record to the my_store index:

   POST /my_store/_bulk  
   {"index":{}}  
   {"productName":"Latest art shirts for women in 2017 autumn","size":"L"}
   
   

4. Verify the result.

   Check the result. If the value of errors is false and the value of result in each record in the items array is created, all records are successfully written.

On an ECS server, you can run cURL commands to import files via an open-source OpenSearch API.

Applicable scenarios: batch ingestion of local JSON files using an automation script.

1. Prepare an ECS. Buy an ECS located in the same VPC as the destination OpenSearch cluster.
2. Obtain the private network address of the cluster.
   1. Log in to the CSS management console.
   2. In the navigation pane on the left, choose Clusters > OpenSearch.
   3. In the cluster list, obtain the target cluster's internal network address from the Internal Network Address column. Typical address format: <host>:<port> or <host>:<port>,<host>:<port>. Example: 10.62.179.32:9200,10.62.179.33:9200.

      If the cluster has client nodes, only the IP addresses and ports of all the client nodes are displayed. Otherwise, the IP addresses and ports of all data nodes and cold data nodes are displayed.

3. Verify connectivity. Ping the OpenSearch cluster's private network address from the ECS. If successful, the two are connected.
4. Prepare documents. Upload a JSON file to the ECS.

   For example, save the following data as a test.json file and upload it to the ECS:

   {"index": {"_index":"my_store"}}
   {"productName":"Autumn new woman blouses 2019","size":"M"}
   {"index": {"_index":"my_store"}}
   {"productName":"Autumn new woman blouses 2019","size":"L"}
   
   

5. (Optional) Create an index and specify a custom mapping to define field data types.

   For example, run the following command to create index my_store:

   • For a non-security mode cluster (HTTP): no authentication required.

     curl -X PUT "http://<host>:<port>/my_store" \
          -H 'Content-Type: application/json' \
          -d '{
            "settings": {
              "number_of_shards": 1
            },
            "mappings": {
              "properties": {
                "productName": {
                  "type": "text"
                },
                "size": {
                  "type": "keyword"
                }
              }
            }
          }'
     
     

   • For a security-mode cluster using HTTP: Authentication is required. If the password contains special characters like !, enclose the password with a pair of single quotation marks, for example, -u 'admin':'Pass!word'.

     curl -X PUT -u <user>:<password> "http://<host>:<port>/my_store" \
          -H 'Content-Type: application/json' \
          -d '{
            "settings": {
              "number_of_shards": 1
            },
            "mappings": {
              "properties": {
                "productName": {
                  "type": "text"
                },
                "size": {
                  "type": "keyword"
                }
              }
            }
          }'
     
     

   • For a security-mode cluster using HTTPS: Authentication is required. If the password contains special characters like !, enclose the password with a pair of single quotation marks, for example, -u 'admin':'Pass!word'. Furthermore, ignore SSL certificate authentication.

     curl -X PUT -k -u <user>:<password> "https://<host>:<port>/my_store" \
          -H 'Content-Type: application/json' \
          -d '{
            "settings": {
              "number_of_shards": 1
            },
            "mappings": {
              "properties": {
                "productName": {
                  "type": "text"
                },
                "size": {
                  "type": "keyword"
                }
              }
            }
          }'
     
     

6. Ingest the JSON file into the OpenSearch cluster:

   Run the following command in the ECS directory that stores the JSON file:

   • For a non-security mode cluster (HTTP): no authentication required.

     curl -X POST "http://<host>:<port>/_bulk" \
          -H 'Content-Type: application/json' \
          --data-binary @test.json
     
     

   • For a security-mode cluster using HTTP: Authentication is required. If the password contains special characters like !, enclose the password with a pair of single quotation marks, for example, -u 'admin':'Pass!word'.

     curl -X POST -u <user>:<password> "http://<host>:<port>/_bulk" \
          -H 'Content-Type: application/json' \
          --data-binary @test.json
     
     

   • For a security-mode cluster using HTTPS: Authentication is required. If the password contains special characters like !, enclose the password with a pair of single quotation marks, for example, -u 'admin':'Pass!word'. Furthermore, ignore SSL certificate authentication.

     curl -X POST -k -u <user>:<password> "https://<host>:<port>/_bulk" \
          -H 'Content-Type: application/json' \
          --data-binary @test.json
     
     

   Note

   If the designated cluster node is unavailable, the command will fail. If the cluster contains multiple nodes, replace <host> with the IP address of another node. If the cluster contains only one node, you have to wait until the node recovers.

7. Verify the result.

   After data ingestion, the following information is returned:

   {"took":204, "errors":false, "items":[...]}
   
   

   If the value of errors is false, all data has been successfully ingested.