moveToCloud script readme

Use the moveToCloud.pl script to upload data files to either SoftLayer® object storage (Swift object storage) or Amazon S3. The script can split large files into smaller segments. When you load these file segments into dashDB™, the segments are assembled to reproduce the single, large, original file. To minimize bandwidth and storage, you can use the gzip utility to compress the target file. You can also specify the number of threads to use for increased performance.

The script offers a convenient way to upload large objects over 5 GB that cannot be directly loaded into cloud storage services. The script includes options to specify the number of threads for the compression process, load files in batch, and define a temporary directory for files that are generated during processing.

You can use the script to test your target credentials, the target URL, and the target container. You can also use the script to delete files that are stored on SoftLayer or Amazon S3.

If you use an SSL connection (a URL that begins with https://), the script encrypts the output files before they are sent to cloud storage. If you use a non-SSL connection (a URL that begins with http://), the output files are not encrypted.

Prerequisites

UNIX or Linux
Perl (version 5.10 or higher)
cURL
gzip

After you download the script to your computer, grant execute permission to the script by using the chmod command. For example, chmod +x moveToCloud.pl.

If you cannot call the script as an executable, call the script by using a Perl input. For example, 'perl moveToCloud.pl'.

Examples

In the following single file upload example, the script segments and compresses a file that is named sales and then uploads it to the files subdirectory of the mycontainer container on the specified SoftLayer server that is located in Dallas. The script uses a credentials file named ids and a temporary directory named temp. The process uses six concurrent threads.

moveToCloud.pl -source /home/user/sales -target softlayer::dallas::mycontainer::/files/targetfile -creds level1/ids -tmpdir /temp -threads 6

The following example uses only the required parameters for a single file upload:

moveToCloud.pl -source /home/user/sales -target softlayer::dallas::mycontainer::/files/targetfile -creds level1/ids

The following batch file example uses a batch file that is named batch and a temporary directory named temp:

moveToCloud.pl -batch /home/user/batch -tmpdir /temp

The following single file example uses SoftLayer object storage token authentication with a credentials file named token and the object storage endpoint URL https://dal05.objectstorage.softlayer.net/v1/AUTH_user1:

moveToCloud.pl -source /home/user/sales -target softlayer::https://dal05.objectstorage.softlayer.net/v1/AUTH_user1::mycontainer::/files/targetfile -creds level1/token -token

Syntax

Parameters that are surrounded by "[ ]" are optional. Values that are surrounded by "< >" are optional.

Single file syntax

moveToCloud.pl -source source_file -target service::url::container::path -creds credentials_file_path [ -threads number_of_threads ] [ -nocompression ] [ -tmpdir temp_directory ] [ -verbose < file_name > ] [ -quiet ] [ -token ]

Batch file syntax

moveToCloud.pl -batch batch_file [ -tmpdir temp_directory ] [ -verbose < file_name > ] [ -quiet ]

Delete syntax

moveToCloud.pl -delete -target service::url::container::path -creds credentials_file_path [ -verbose < file_name > ] [ -quiet ] [ -token ]

Test syntax

moveToCloud.pl -test -target service::url::container::path -creds credentials_file_path [ -token ]

Optional syntax

moveToCloud.pl [ -list ] [ -help ] [ -man ]

Parameters

-source

Required for single file processing. The path to the source file name and path. For example, /home/user/file_name.

-target

Required for single file, test, and delete processing. The target server URL and container path in the following format: service::url::container::path.

service

The cloud storage service name. (For a complete listing of supported service name values, use the -list parameter.)

softlayer: IBM® SoftLayer Object Storage
s3: Amazon AWS Simple Storage Service

url

The URL of the cloud storage service or one of the following supported service tag values. (For a complete listing of supported service tag values, use the -list parameter.)

If you are using SoftLayer token authentication, the URL corresponds to your object storage endpoint URL instead of a service tag.

If you use a non-SSL connection, you will be prompted to verify that SSL will not be used.

SoftLayer service tag values

The following service tag values use an SSL connection, and the script will encrypt files sent to these sites:

amsterdam: https://ams01.objectstorage.softlayer.net/auth/v1.0
dallas: https://dal05.objectstorage.softlayer.net/auth/v1.0
hongkong: https://hkg02.objectstorage.softlayer.net/auth/v1.0
london: https://lon02.objectstorage.softlayer.net/auth/v1.0
sanjose: https://sjc01.objectstorage.softlayer.net/auth/v1.0
singapore: https://sng01.objectstorage.softlayer.net/auth/v1.0
toronto: https://tor01.objectstorage.softlayer.net/auth/v1.0

S3 service tag values

The following service tag values use an SSL connection, and the script will encrypt files sent to these sites:

ap-northeast-1: https://s3-ap-northeast-1.amazonaws.com
ap-southeast-1: https://s3-ap-southeast-1.amazonaws.com
ap-southeast-2: https://s3-ap-southeast-2.amazonaws.com
eu-west-1: https://s3-eu-west-1.amazonaws.com
sa-east-1: https://s3-sa-east-1.amazonaws.com
us-east-1: https://s3.amazonaws.com
us-west-1: https://s3-us-west-1.amazonaws.com
us-west-2: https://s3-us-west-2.amazonaws.com

container

The name of the target container or bucket on the cloud storage server.

path

The target path of the container or bucket on the cloud storage server. To use the source file name as the target file name, include "/" at the end of the path (/put/file/here/). To specify a target file name, include the file name at the end of the path (/put/file/here/targetfile).

For example, softlayer::dallas::mycontainer::/put/file/here/targetfile.

-creds

Required for single file, test, and delete processing. Specify the credentials file name and path. Refer to the sample credentials file that is included with moveToCloud.

If you do not specify a credentials file, you will be prompted to supply the following user credentials:

Amazon S3: accesskey; secretkey
SoftLayer: username; password

-token

Required for SoftLayer token authentication. If you use this parameter with the -creds parameter to specify a credentials file, the credentials file must contain your token. Refer to the sample credentials file that is included with the moveToCloud zip file. If you do not specify a credentials file, you will be prompted to enter your token. This option is only supported by SoftLayer.

-batch

Required for batch processing, which can process multiple source files. Specify the batch file name and path. Refer to the sample batch file that is included with moveToCloud.

-test

Required for delete processing. Test your target credentials, the target URL, and target container. Use the test mode before you perform an upload to an account for the first time. If a container path is specified in -target, the script verifies that the container does not contain a file name that matches a source file name.

-delete

Required for delete processing. Deletes the target file or directory from cloud storage. If the value ends with "/", the script deletes a directory. Deleting a directory deletes all files recursively under the path. For Softlayer, if the target is a large object (manifest file), the associated parts are also deleted. You will receive a confirmation prompt unless the -quiet option is used.

-threads

The maximum number of concurrent threads to use during processing. Default is 5.

-nocompression

Disable compression that is performed by gzip. Compression is enabled by default. This option must be specified for .gz and .gzip files.

-tmpdir

Temporary work area directory. All temporary files that are created by the script will be stored in this directory and will be deleted after use.

-verbose

Displayed detailed messages during processing. To output the messages to a file, use -verbose filename, where filename is the name and path of the file.

-quiet

Do not display messages during processing.

Note: Do not use -quiet if you use a non-SSL connection for -target. If you use a non-SSL connection, you must respond to a prompt to verify that SSL will not be used. The default action for this prompt will cancel processing. The -quiet parameter prevents this prompt from displaying, which causes the script to use the default action and cancel processing.

-list

List supported services and URL endpoints for use with -target.

-yes

Automatically respond "yes" to all prompts that are displayed during processing.

-no

Automatically respond "no" to all prompts that are displayed during processing.

-help

Display a brief overview of moveToCloud syntax.

-man

Display detailed help information for moveToCloud.

Result

After the moveToCloud processing is complete, the target file will be available at the specified target location. By default, the target file is compressed and includes the extension gz. For example, for a target file that is named myfile, if -nocompression is not specified (default), the target file is named myfile.gz, and if -nocompression is specified, the target file is named myfile.

For SoftLayer storage, the process creates a pseudo-directory in the target directory that has the same name as the target file and includes the extension .parts. The pseudo-directory contains the segmented parts of the file, which is necessary for the static large object implementation of SoftLayer.

The moveToCloud process uses the following return codes:

0: Success.
1: Input error.
2: Interrupted.
n: Line of code at failure.