Using the Internet to Migrate Data
Context
You can migrate data from a local NAS to SFS Turbo using the Internet.
In this solution, to migrate data from the local NAS to the cloud, a Linux server is created both on the cloud and on-premises. Inbound and outbound traffic is allowed on port 22 of these two servers. The on-premises server is used to access the local NAS, and the ECS is used to access SFS Turbo.
You can also refer to this solution to migrate data from an on-cloud NAS to SFS Turbo.
Limitations and Constraints
- Data cannot be migrated from the local NAS to SFS Capacity-Oriented using the Internet.
- Only Linux ECSs can be used to migrate data.
- The UID and GID of your file will no longer be consistent after data migration.
- The file access modes will no longer be consistent after data migration.
- Inbound and outbound traffic must be allowed on port 22.
- Incremental migration is supported, so that only changed data is migrated.
Prerequisites
- A Linux server has been created on the cloud and on-premises respectively.
- EIPs have been configured for the servers to ensure that the two servers can communicate with each other.
- You have created an SFS Turbo file system and have obtained the mount point of the file system.
- You have obtained the mount point of the local NAS.
Procedure
- Log in to the ECS console.
- Log in to the created on-premises server client1 and run the following command to access the local NAS:mount -t nfs -o vers=3,timeo=600,noresvport,nolock Mount point of the local NAS /mnt/src
- Log in to the created Linux ECS client2 and run the following command to access the SFS Turbo file system:mount -t nfs -o vers=3,timeo=600,noresvport,nolock Mount point of the SFS Turbo file system /mnt/dst
- Run the following commands on client1 to install the rclone tool:wget https://downloads.rclone.org/v1.53.4/rclone-v1.53.4-linux-amd64.zip --no-check-certificateunzip rclone-v1.53.4-linux-amd64.zipchmod 0755 ./rclone-*/rclonecp ./rclone-*/rclone /usr/bin/rm -rf ./rclone-*
- Run the following commands on client1 to configure the environment:rclone configNo remotes found - make a new onen) New remotes) Set configuration passwordq) Quit confign/s/q> nname> remote name (New name)Type of storage to configure.Enter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value24 / SSH/SFTP Connection\ "sftp"Storage> 24 (Select the SSH/SFTP number)SSH host to connect toEnter a string value. Press Enter for the default ("").Choose a number from below, or type in your own value1 / Connect to example.com\ "example.com"host> ip address (IP address of client2)SSH username, leave blank for current username, rootEnter a string value. Press Enter for the default ("").user> user name (Username of client2)SSH port, leave blank to use default (22)Enter a string value. Press Enter for the default ("").port> 22SSH password, leave blank to use ssh-agent.y) Yes type in my own passwordg) Generate random passwordn) No leave this optional password blanky/g/n> yEnter the password:password: (Password for logging in to client2)Confirm the password:password: (Confirm the password for logging in to client2)Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.Enter a string value. Press Enter for the default ("").key_file> (Press Enter)The passphrase to decrypt the PEM-encoded private key file.Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keysin the new OpenSSH format can't be used.y) Yes type in my own passwordg) Generate random passwordn) No leave this optional password blanky/g/n> nWhen set forces the usage of the ssh-agent.When key-file is also set, the ".pub" file of the specified key-file is read and only the associated key isrequested from the ssh-agent. This allows to avoid `Too many authentication failures for *username*` errorswhen the ssh-agent contains many keys.Enter a boolean value (true or false). Press Enter for the default ("false").key_use_agent> (Press Enter)Enable the use of the aes128-cbc cipher. This cipher is insecure and may allow plaintext data to be recovered by an attacker.Enter a boolean value (true or false). Press Enter for the default ("false").Choose a number from below, or type in your own value1 / Use default Cipher list.\ "false"2 / Enables the use of the aes128-cbc cipher.\ "true"use_insecure_cipher> (Press Enter)Disable the execution of SSH commands to determine if remote file hashing is available.Leave blank or set to false to enable hashing (recommended), set to true to disable hashing.Enter a boolean value (true or false). Press Enter for the default ("false").disable_hashcheck>Edit advanced config? (y/n)y) Yesn) Noy/n> nRemote config-------------------[remote_name]type = sftphost=(client2 ip)user=(client2 user name)port = 22pass = *** ENCRYPTED ***key_file_pass = *** ENCRYPTED ***--------------------y) Yes this is OKe) Edit this remoted) Delete this remotey/e/d> yCurrent remotes:Name Type==== ====remote_name sftpe) Edit existing remoten) New remoted) Delete remoter) Rename remotec) Copy remotes) Set configuration passwordq) Quit confige/n/d/r/c/s/q> q
- Run the following command to view the rclone.conf file in /root/.config/rclone/rclone.conf:cat /root/.config/rclone/rclone.conf[remote_name]type = sftphost=(client2 ip)user=(client2 user name)port = 22pass = ***key_file_pass = ***
- Run the following command on client1 to synchronize data:rclone copy /mnt/src remote_name:/mnt/dst -P --transfers 32 --checkers 64Note
- Replace remote_name in the command with the remote name in the environment.
- Set transfers and checkers based on the system specifications. The parameters are described as follows:
- transfers: number of files that can be transferred concurrently
- checkers: number of local files that can be scanned concurrently
- P: data copy progress
After data synchronization is complete, go to the SFS Turbo file system to check whether data is migrated.
Parent topic: Data Migration
- Context
- Limitations and Constraints
- Prerequisites
- Procedure