How do you download files from cloud storage in your custom cookbooks?

CycleCloud™’s Cluster-Init technology is a great way to copy static files onto a cloud instance.  Upload your files with the CycleCloud project feature and the automation will copy them down when your instance boots. But what happens if you need some flexibility? Perhaps you want to make the download conditional on a particular node attribute or state. Or maybe you want to use different versions of a package without having to maintain separate versions of your project.

Using a custom Chef cookbook is a great way to handle it. But how do you download the file?

Our Chef cookbooks include a lightweight resource provider (LWRP) called “Thunderball” that handles this. With Thunderball, you can use your credentials to access a cloud storage endpoint that has your packages. Thunderball provides reliability by ensuring the file is fully downloaded before completing.

To use thunderball in your cookbook, you first set up the configuration. This example reads Amazon S3 credentials from a data bag named “s3_creds”.

# Include Cycle Computing’s Thunderball recipe
include_recipe "thunderball"

# Read credentials from the databag
creds = data_bag_item('s3_creds', 'creds')
# AWS Access Key
access_key = creds['access_key']
# AWS Secret Key
secret_key = creds['secret_key']
# S3 bucket for packages
package_bucket = creds['package_bucket']
node.default[:package_bucket] = package_bucket

# Create a Thunderball config named “my_thunderball”
thunderball_config "my_thunderball" do
base "s3://#{package_bucket}"
username access_key
password secret_key
end

Now that you have created the credentials, you can use it to download packages elsewhere in your recipe. For example, if you have a Debian package of custom Python build you want to install:

# Set the custom python version from the node attribute example.python.version
python_version = node[:example][:python][:version]
python_package = "python3.4_#{python_version}_ubuntu-16.04_amd64.deb"
thunderball "download custom python package" do
url "#{python_package}"
config "my_thunderball"
dest_file "/tmp/#{python_package}"
end

dpkg_package "install custom python package" do
source "/tmp/#{python_package}"
action :install
end

Thunderball gives the flexibility you need to download files from your cloud storage on an as-needed basis. If you want to learn more about how our CycleCloud software gives you simple, managed access to cloud HPC, contact us.

Share this: