The question isn’t cost, it’s value

Addison Snell recently wrote an article for The Next Platform called “The three great lies of cloud computing.” Snell points out that the marketing around cloud computing doesn’t always match reality. As someone who does marketing for cloud computing software, I just want to go on the record as saying Addison is absolutely….right. We’ve spent a lot of time on this blog, at conferences, etc. talking about the benefits of using public cloud services for big compute and big data. But we believe that a one-size-fits-all solution is never the right size. Public cloud services can be sized to fit many needs, but not every need. If there’s one area where Addison’s article falls short, it’s that he only considers the raw dollar amount when talking about cost. Raw dollar amount is important, of course, but it’s not the whole story. As I said in response to a question at HTCondor Week 2016, it’s all about the value that cloud resources provide, not the cost. If you spend twice as much to run a workload in the cloud, but you get three times the value (e.g. due to faster time-to-results or the ability to run simulations at a finer resolution due to adding greater capacity), that’s a net win. Customers often find value in the additional capacity or flexibility the cloud can offer: adding more compute without having to plan datacenter space or trying out new hardware by renting instead of making a large capital investment. Another part of the value discussion is the total value of your entire HPC environment: the mix of cloud plus internal resources. Many...

CycleCloud Support for Elastic File System (EFS)

Last week, Amazon released the Elastic File System (EFS) in the US East (Northern Virginia), US West (Oregon), and Europe (Ireland) regions. EFS provides a scalable, POSIX-compliant filesystem for Amazon EC2 instances without having to run a file server. This means you can grow your storage as your usage increases instead of having to pre-provision disks. Instances mount EFS just as they would any traditional NFS volume. Of course, we know that, you, our customers will want to start testing workloads against EFS, so we’ve added support for it in the next CycleCloud release. Once the EFS is created through the AWS console, cluster instances can mount it with the configuration you’re already used to. For example, the configuration below will mount EFS fs-f00cf6b8 to /mnt/efs_test: [[[configuration cyclecloud.mounts.efs_test]]] type = efs filesystem_id = fs-f00cf6b8 So what does EFS look like in the real world? We took an I/O-intensive genomics workload and ran it on a 16-instance cluster using four different configurations: c3.4xlarge filer using ephemeral storage c3.4xlarge filer using a 500 GB GP2 (solid state drive) volume c4.4xlarge filer using a 500 GB GP2 volume EFS (Basic) Each job runs without competition on a c4.4xlarge instance and pulls 25 GB of reference genome data into memory. The code performs genomic alignment in batches and at the end writes approximately 1 GB of data (per job) back to the filer. The table below shows the average runtimes for the different filer configurations with as many as 16 of such tasks simultaneously using the shared filer:   Filer Simultaneous Tasks Average runtime (seconds) c3.4xlarge (ephemeral) 1 213 c3.4xlarge (ephemeral) 16 1658...