I’m planning on offering the equivalent of 8 hours on a ~30,000-core cluster ($10,000 in free CycleCloud time) to help researchers answer questions that will help humanity. But before we get there, let’s talk about why:
Recently, Cycle got significant press coverage for using CycleCloud to create a ~30,000 core cluster on Amazon EC2 for a Top 5 Pharma to run research. It hit Ars Technica, then Slashdot, then Engadget (which had a great depiction of Nekomata, btw), then Wired's CloudLine … unreal. (Update: Now Forbes and Wired too!)
In reading all the comments and questions, like "Would this run high-res Crysis?" or "How much capacity does Amazon have?", a thought went through my mind:
I became concerned. Concerned because these shouldn't be the questions we're asking.
I worried that in all this glitter, we would miss what is truly gold: that this type of computing can speed up scientific research and solve problems we’d traditionally never dream of tackling. So I'm writing this to introduce a new concept to you, and implore you to think about how to move the human race forward through science and research.
To start, let's answer a question implied in many comments: Why is this important? For years cloud computing has been about paying for what you use, and accessing the compute power you need, when you needed it.
The problem is, today, researchers are in the long-term habit of sizing their questions to the compute cluster they have, rather than the other way around. This isn’t the way we should work. We should provision compute at the scale the questions need. We're talking about taking questions that require a million hours of computation, and answering them in a day. Securely. At reasonable cost.
I call this “utility supercomputing”, that is, provisioning resources at a scale of the Top 500 supercomputing list, for researchers to use for a few hours, then turn off. For example, take the 5 years of back data that you never were able to re-analyze with a new, innovative algorithm because it would take 2 years on your internal compute cluster, and instead get results back in 9 hours. For $12000. Wow…
Scratch the surface of this idea, and you'll see a world of research the way I see it. No more waiting. No more R&D folks task-switching for days or weeks while compute is run. Only answers at the speed of thought, at the speed of invention, at the scale of the question.
But unlike other innovations, this is not a tool for the have's and not the have-not’s. This tool is equal opportunity and an extreme competitive advantage. You pay for what you use with no up-front investment. We've already enabled individual researchers to rent computing equivalent to all of the Top 5 Pharma's HPC clusters combined, for a couple thousand dollars an hour. And all this without the researcher needing to know IT, or scripting, or APIs.
Now, on behalf of the Cycle team, I'd like to answer an additional question we’ve been asked repeatedly over the past several weeks: Will you go bigger?
Absolutely. We will. There is no more thinking that having a single, fixed-size internal cluster is the right idea. You might buy internal compute capacity for day-to-day usage. But the answer isn't there. Not anymore.
This is truly exciting. And not exciting simply because technologically Cycle can now create 30000 core clusters in minutes, leveraging robust automation, performance analytics, and moving data securely across continents to answer un-askable questions.
It's exciting because someone is going to take these clusters and cure cancer, or Alzheimer's, or my personal affliction, Type-1 diabetes. And hopefully cure them faster because they have better tools.
So there's no more running. We aim to misbehave. We aim to wreck the status quo of HPC clusters and computational science. We will enable those crazy questions from the misfit geniuses, the ones so big that you would never even ask them, you might even joke about how silly they are, and we will help answer them. Efficiently. Cheaply.
It is with this in mind that I'd like to ask you, what's your question? What is the analysis or simulation that could change your field? What is the work that would bring you, your university, your research organization, your company, or all of humanity to the next level?
Because these are the questions we should be asking.
And to put our money where our mouth is, we're going big or going home by offering the equivalent of 8 hours on a ~30000-core cluster (specifically $10000 of CycleCloud time), for free, along with free support from Cyclers, to help a single researcher with an un-askable question. It's the CycleCloud BigScience Challenge 2011.
If you have one of these questions please fill out the form, with brief entries, describing your career, the work you do, why you believe in it, and how you want to use 10000 or more cores to push us forward. Proposals are due Nov 7th, 2011, and we'll announce the Finalists at Booth #443 at Supercomputing 2011.
We want the runts, the misfits, the crazy ideas that are normally too big to ask, but might, just might, help humanity.
CEO, Cycle Computing