Earlier this month, I attended the 30th USENIX LISA Conference in Boston, MA. In addition to my role co-chairing the Invited Talks tracks, I hosted a Birds of a Feather (BoF) session on cloud HPC. A group of 15 people joined to discuss the state of their cloud HPC efforts. Some attendees were actively using cloud resources — either public cloud or private cloud (i.e. OpenStack) resources. The rest know that they’ll be looking at cloud soon, so they wanted to hear what current practitioners had to say.
Among those who haven’t started using public cloud for HPC, the greatest concern was data movement. One attendee mentioned he had 63 petabytes of data on his shared filesystem. That’s clearly not a trivial amount of data to push around, even over a very fast connection. For these kinds of scenarios, internal infrastructure (whether fixed or a cloud-like solution) is often the best solution. However it could be that each job only needs a very small subset of that data, and those could be candidates for moving to public cloud.
The general agreement was that there are two main drivers for cloud-based HPC resources. The first is the ability to use correctly-sized resources. This can mean the right number of instances or even the right-sized instances. On a related note, the other main driver was the ability to isolate users from each other in a user-transparent manner. Giving users a dedicated sandbox simplifies administration and allows the IT organization to spend more time helping the user community make maximum use of the resources.
A major driver, especially for the academic and research institutions, was the fact that internal clusters are largely maintained by people with domain science expertise, not system administration. By moving to a public cloud, the basic system administration tasks (particularly hardware maintenance) get handed off to experts allowing the staff to focus on providing more direct support to the end users.
As we have always said, public cloud is a great tool for HPC, but it’s not the only tool. It was great to hear from IT practitioners about their experience in deciding when cloud is the right tool for the job.