Customize your cloud instances with CycleCloud projects

CycleCloud™ 6.5.5 adds some great new functionality for supporting customizations to workflows which we call projects.

The ability to customize cloud instances with user-provided configuration, software, and data has long been one of the most popular features in CycleCloud. Using this allows for similar clusters to be easily reused by just changing the location of the customization information. With this release, we’ve made this even easier and more powerful with the project feature.

With a CycleCloud project, you can compose multiple projects. This means if instances share common configuration, you can maintain that in one place. For example, you may have two clusters that need the same users added, but each has different software packages. With the projects, you can have a project that adds users and a project for each of the software packages.

Within a project, there can be multiple “specs”, which define different applications of the project. This could be a different set of configuration for master nodes versus execute nodes. Projects also have versions, which aid composability and make it easier to test.

The cluster template for the example above might look like this:

[[node defaults]]
    [[[cluster-init addusers]]]
    Project = addusers
    Version = 1.0.0

[[node master]]
   [[[cluster-init oursoftware]]]
   Project = oursoftware
   Version = 8.1.2
   Spec = master

[[nodearray execute]]
   [[[cluster-init oursoftware]]]
   Project = oursoftware
   Version = 8.1.2
   Spec = execute

Managing a CycleCloud project

The cyclecloud command line tool has support for managing projects. The commands include:

  • cyclecloud project init myproject – Creates the basic directory structure for a project named myproject
  • cyclecloud project add_spec myspec – Adds a spec named myspec to the project in the current directory
  • cyclecloud project upload mylocker – Uploads the project to the cloud storage locker named mylocker defined in your CycleCloud installation.

For more information on creating and uploading CycleCloud Projects, see the CycleCloud documentation.

Project contents

A project contains a cluster template, the Cluster-Init configuration (this is familiar to existing CycleCloud users), and custom Chef cookbooks for advanced configuration. The directory structure is:

  • templates – your cluster templates
  • specs
    • <spec name>
      • cluster-init
        • scripts – scripts that are executed in lexicographical order on the node
        • files – raw data files to will be put on the node
        • tests – tests to be run when a cluster is started in testing mode
      • chef
        • site-cookbooks – Chef cookbooks
        • data_bags – Chef databags
        • roles – Chef role definitions

The old-style Cluster-Init is still available, and can easily be migrated into a project in order to take advantage of the advanced features. If you’d like to learn more about how CycleCloud can help you get the most from your cloud HPC and big compute, contact us.

Share this: