How do I optimize cloud HTCondor jobs for cost?

How do I optimize cloud HTCondor jobs for cost?

The most important consideration is getting your work done. The second most important consideration is doing it without wasting money. In this post, we describe how you can minimize costs in an HTCondor environment on Amazon Web Services using CycleCloud™. Our CycleCloud software is an orchestration platform for any workflow. It provides multi-user support, cost management, alerting, and automation to organizations who want to get better answers, faster. CycleCloud launches, configures, and monitors cloud resources and provides tools for managing data and workflows. With support for Microsoft Azure, Google Cloud, and Amazon Web Services, our customers use CycleCloud to power their cloud HPC and big compute workloads using HTCondor, PBS Pro, Hadoop, and other technologies. So how can you use CycleCloud’s features to get the most compute for your dollar? The HTCondor scheduler tracks the state of execution slots, including the time slots are idle. This makes it easy to identify “wasted” time in a cloud environment, but it’s not always as straightforward as it may seem. CycleCloud will wait for a user-configurable length of time before considering whether an idle instance should be shut down, and will only shut down when the node is within 5 minutes of the end of the billing hour. “Shut the instances down sooner!” is an understandable first reaction, but it isn’t necessarily beneficial. AWS bills for EC2 instances by the hour, so shutting an instance down early only reduces the appearance of idle time without lowering your bill. With any of the cloud service providers, keeping the minimum idle time too short will result in instance churn, especially with uneven work submission....
Automatic license installation for CycleCloud

Automatic license installation for CycleCloud

We know that when getting started on a new project, you want to get right to the important work.  That’s why in CycleCloud version 6.5.5, we’ve added automatic installation of your CycleCloud license. This shortens the setup time and gets you started in the cloud that much faster. When you first install the CycleCloud software and provide your Cycle Computing Portal credentials, CycleCloud will check for an unused license. If one is found, it will be immediately applied and you can begin using CycleCloud. This also makes the license renewal process easier. Once your license has been renewed, CycleCloud will automatically install the updated license. By reducing the administrative work, you can focus on what you really want: better answers, faster. See the CycleCloud documentation for more information about the automatic license installation feature. If you’d like to learn more about how CycleCloud software makes cloud HPC and big compute productive, contact...

Customize your cloud instances with CycleCloud projects

CycleCloud™ 6.5.5 adds some great new functionality for supporting customizations to workflows which we call projects. The ability to customize cloud instances with user-provided configuration, software, and data has long been one of the most popular features in CycleCloud. Using this allows for similar clusters to be easily reused by just changing the location of the customization information. With this release, we’ve made this even easier and more powerful with the project feature. With a CycleCloud project, you can compose multiple projects. This means if instances share common configuration, you can maintain that in one place. For example, you may have two clusters that need the same users added, but each has different software packages. With the projects, you can have a project that adds users and a project for each of the software packages. Within a project, there can be multiple “specs”, which define different applications of the project. This could be a different set of configuration for master nodes versus execute nodes. Projects also have versions, which aid composability and make it easier to test. The cluster template for the example above might look like this: [[node defaults]] [[[cluster-init addusers]]] Project = addusers Version = 1.0.0 [[node master]] [[[cluster-init oursoftware]]] Project = oursoftware Version = 8.1.2 Spec = master [[nodearray execute]] [[[cluster-init oursoftware]]] Project = oursoftware Version = 8.1.2 Spec = execute Managing a CycleCloud project The cyclecloud command line tool has support for managing projects. The commands include: cyclecloud project init myproject – Creates the basic directory structure for a project named myproject cyclecloud project add_spec myspec – Adds a spec named myspec to the project...
How can you save money with Preemptible VMs?

How can you save money with Preemptible VMs?

You have your workload in production on Google Cloud, so now what? The next step is to do more work without going over budget. Google Cloud offers reduced-rate cloud instances, called Preemptible VMs. Preemptible VM pricing varies by machine type, but can represent a cost savings of up to 80%. Since the prices are fixed, you can easily predict your spend. The tradeoff for the lower prices is that the instances are subject to being taken away on short notice in order to meet demand from the regularly-priced instances and they are terminated after 24 hours. Preemptible VMs also cannot have GPUs attached and do not receive sustained use discounts. For customers looking to get the most out of Preemptible VMs, CycleCloud™ makes using them easy and effective with a single click, with features like: Easy requests across multiple machine types and Zones Automatic replacement of lost instances Using Preemptible VMs in CycleCloud CycleCloud software has unique features that makes it easy to not only use Preemptible VMs, but use them effectively. Preemptible VM capacity varies by machine type and Zone. With CycleCloud, you can specify multiple machine types and Zones to make it easy to get the capacity you need. CycleCloud software automatically spreads the requests out across all combinations you choose. If you lose some instances, CycleCloud automatically requests replacement instances from the remaining combinations that have capacity. To use Preemptible VMs in CycleCloud, set Preemptible = true in the nodearray section of the cluster template. Job considerations When using Preemptible VMs, your jobs need to be interruptible. This means that they can either restart from the...
New documentation website

New documentation website

At Cycle Computing, we know how important it is for our customers to be able to get quick answers and in-depth information about using CycleCloud software to manage their cloud HPC and Big Compute environments. With that in mind, we’re pleased to share our newly-revamped documentation site at docs.cyclecomputing.com.  The new documentation site offers a few great features: Improved search provides cross-document results Content for different aspects of CycleCloud are consolidated Role-specific content for users and administrators Release summaries are featured on the front page Of course, we’re always working to make it better. In the coming months, expect to see more detail describing the settings and configuration options available. We’ll also be adding more examples to help provide guidance and inspiration for real-world...

CycleCloud 6.5.3 released

Last week we pushed the button on the latest release of our CycleCloud software for managing cloud HPC and Big Compute workloads. This release has one particular feature that many customers customers asked for: Cost Alerting. This new feature will give you the ability to easily set cost alerts on a per-cluster basis. You can set the alert to be dollars per day or per month. This gives you a great way to manage consumption and assure that users aren’t blowing through budgets. After all, you want to give your users access to unlimited compute, but you don’t want to give them an unlimited budget. Clusters from any supported cloud service provider display an estimated compute cost along with the core-hour usage. Daily or monthly budgets are set from the cluster page and trigger alerts when the threshold is crossed. Because the appropriate action when a cluster goes over budget varies, the CycleCloud software does not take any automated enforcement action. We find most customers try to set it the threshold to some percentage of total budget to give them a heads up before exceeding the budget. The percentage can be a function of the type of work and size of budget. In addition to the cost alerting, we’ve added additional features to our Microsoft Azure support. CycleCloud now uses Azure Managed Disks and Images for virtual machines, simplifying management of storage and improving performance. Azure instances will automatically use CycleCloud’s standalone DNS configuration to improve the experience for Open Grid Scheduler users. Current customers can download CycleCloud 6.5.3 from the Cycle Computing Portal. If you’d like to learn...