Cloud HPC won’t steal your job

The cloud is not coming to steal your job or the jobs of your team. But then again, what is your job? Here’s a hint: it’s furthering a particular business goal, not performing a specific and unchanging set of tasks. In our experience working with HPC users and admins from a variety of industries, there’s always more work to be done. There are new ways to compute, storage challenges, workflow challenges, scaling challenges. While our CycleCloud™ software makes simple, managed access to cloud HPC easier to provide, we’ve never seen anyone’s position get eliminated. Instead what happens is that staff are freed from mundane infrastructure labor and management tasks to do organization-specific work that adds value. The first question people ask when a new initiative comes along is “how will this affect my job?” so there’s always some trepidation to broach the topic. But avoiding the topic does our customers a disservice. Reducing the effort necessary to provide a cloud HPC environment to users is important for two reasons. The first is that HPC experience generally – and cloud HPC experience specifically – is a tight market. It can be hard to find and retain talented employees. We see two types of situations. In the first case, the staff tasked with running the cloud HPC environment do not have HPC experience. They may be talented sysadmins or user support staff who have been thrown to the wolves, so to speak. They need help getting started, managing the differences (and similarities) between cloud and in-house environments, understanding what to monitor, knowing how to help users route jobs to the best...
How do I optimize cloud HTCondor jobs for cost?

How do I optimize cloud HTCondor jobs for cost?

The most important consideration is getting your work done. The second most important consideration is doing it without wasting money. In this post, we describe how you can minimize costs in an HTCondor environment on Amazon Web Services using CycleCloud™. Our CycleCloud software is an orchestration platform for any workflow. It provides multi-user support, cost management, alerting, and automation to organizations who want to get better answers, faster. CycleCloud launches, configures, and monitors cloud resources and provides tools for managing data and workflows. With support for Microsoft Azure, Google Cloud, and Amazon Web Services, our customers use CycleCloud to power their cloud HPC and big compute workloads using HTCondor, PBS Pro, Hadoop, and other technologies. So how can you use CycleCloud’s features to get the most compute for your dollar? The HTCondor scheduler tracks the state of execution slots, including the time slots are idle. This makes it easy to identify “wasted” time in a cloud environment, but it’s not always as straightforward as it may seem. CycleCloud will wait for a user-configurable length of time before considering whether an idle instance should be shut down, and will only shut down when the node is within 5 minutes of the end of the billing hour. “Shut the instances down sooner!” is an understandable first reaction, but it isn’t necessarily beneficial. AWS bills for EC2 instances by the hour, so shutting an instance down early only reduces the appearance of idle time without lowering your bill. With any of the cloud service providers, keeping the minimum idle time too short will result in instance churn, especially with uneven work submission....
Automatic license installation for CycleCloud

Automatic license installation for CycleCloud

We know that when getting started on a new project, you want to get right to the important work.  That’s why in CycleCloud version 6.5.5, we’ve added automatic installation of your CycleCloud license. This shortens the setup time and gets you started in the cloud that much faster. When you first install the CycleCloud software and provide your Cycle Computing Portal credentials, CycleCloud will check for an unused license. If one is found, it will be immediately applied and you can begin using CycleCloud. This also makes the license renewal process easier. Once your license has been renewed, CycleCloud will automatically install the updated license. By reducing the administrative work, you can focus on what you really want: better answers, faster. See the CycleCloud documentation for more information about the automatic license installation feature. If you’d like to learn more about how CycleCloud software makes cloud HPC and big compute productive, contact...

Customize your cloud instances with CycleCloud projects

CycleCloud™ 6.5.5 adds some great new functionality for supporting customizations to workflows which we call projects. The ability to customize cloud instances with user-provided configuration, software, and data has long been one of the most popular features in CycleCloud. Using this allows for similar clusters to be easily reused by just changing the location of the customization information. With this release, we’ve made this even easier and more powerful with the project feature. With a CycleCloud project, you can compose multiple projects. This means if instances share common configuration, you can maintain that in one place. For example, you may have two clusters that need the same users added, but each has different software packages. With the projects, you can have a project that adds users and a project for each of the software packages. Within a project, there can be multiple “specs”, which define different applications of the project. This could be a different set of configuration for master nodes versus execute nodes. Projects also have versions, which aid composability and make it easier to test. The cluster template for the example above might look like this: [[node defaults]] [[[cluster-init addusers]]] Project = addusers Version = 1.0.0 [[node master]] [[[cluster-init oursoftware]]] Project = oursoftware Version = 8.1.2 Spec = master [[nodearray execute]] [[[cluster-init oursoftware]]] Project = oursoftware Version = 8.1.2 Spec = execute Managing a CycleCloud project The cyclecloud command line tool has support for managing projects. The commands include: cyclecloud project init myproject – Creates the basic directory structure for a project named myproject cyclecloud project add_spec myspec – Adds a spec named myspec to the project...
How can you save money with Preemptible VMs?

How can you save money with Preemptible VMs?

You have your workload in production on Google Cloud, so now what? The next step is to do more work without going over budget. Google Cloud offers reduced-rate cloud instances, called Preemptible VMs. Preemptible VM pricing varies by machine type, but can represent a cost savings of up to 80%. Since the prices are fixed, you can easily predict your spend. The tradeoff for the lower prices is that the instances are subject to being taken away on short notice in order to meet demand from the regularly-priced instances and they are terminated after 24 hours. Preemptible VMs also cannot have GPUs attached and do not receive sustained use discounts. For customers looking to get the most out of Preemptible VMs, CycleCloud™ makes using them easy and effective with a single click, with features like: Easy requests across multiple machine types and Zones Automatic replacement of lost instances Using Preemptible VMs in CycleCloud CycleCloud software has unique features that makes it easy to not only use Preemptible VMs, but use them effectively. Preemptible VM capacity varies by machine type and Zone. With CycleCloud, you can specify multiple machine types and Zones to make it easy to get the capacity you need. CycleCloud software automatically spreads the requests out across all combinations you choose. If you lose some instances, CycleCloud automatically requests replacement instances from the remaining combinations that have capacity. To use Preemptible VMs in CycleCloud, set Preemptible = true in the nodearray section of the cluster template. Job considerations When using Preemptible VMs, your jobs need to be interruptible. This means that they can either restart from the...
New documentation website

New documentation website

At Cycle Computing, we know how important it is for our customers to be able to get quick answers and in-depth information about using CycleCloud software to manage their cloud HPC and Big Compute environments. With that in mind, we’re pleased to share our newly-revamped documentation site at docs.cyclecomputing.com.  The new documentation site offers a few great features: Improved search provides cross-document results Content for different aspects of CycleCloud are consolidated Role-specific content for users and administrators Release summaries are featured on the front page Of course, we’re always working to make it better. In the coming months, expect to see more detail describing the settings and configuration options available. We’ll also be adding more examples to help provide guidance and inspiration for real-world...