CloudStack for Virtualization Engineers - A New Series from ThinkVIRT
About this Series
This series of upcoming articles here at ThinkVIRT is my attempt to educate server virt admins and engineers (primarily VCPs) on an alternative Infrastructure-as-a-Service (IaaS) platform to VMware's, and one that is poised to emerge as a leader in the space. That platform is CloudStack. I'm not saying that CloudStack is the end all be all solution for every organization, but I do believe it is worth taking a look at due to its flexibility and it's adoption and inclusion in some high-profile and mega-scale cloud implementations by Zynga, KT, and Edmunds. I've been fortunate enough to mentor and career coach some virtualization folks much smarter than me, and I think it's important for the VCPs of the world to start to diversify their scope. The great news is that seasoned VMware engineers have sharpened their skills with the most feature-rich hypervisor on the market, VMware ESXi/vSphere. It's great news because all the other hypervisors are typically less feature-rich, meaning someone that can design a server virt solution with VMware vSphere, can likely design a XenServer solution once they learn the vernacular and the limitation points.
CloudStack is an IaaS platform that can leverage resources from multiple hypervisors (vSphere, Xen, KVM), something vCloud Director cannot do at the time of this writing. Imagine a seasoned VMware rockstar that can also handle a XenServer environment (for example) and manage the whole multi-tenant environment via an open platform such as CloudStack; that's market value. Also imagine having a tiered hyper-visor approach where Tier 1 or the Gold offering runs on vSphere and perhaps takes advantage of features such as FT, whereas the lesser tiers all run on KVM.
How we got to caring about IaaS in the first place
Over the last decade, VMware has successfully established itself as the leading hypervisor for server virtualization. With that success, VMware has continued to innovate in both the hypervisor market and higher up the IT stack, including datacenter automation and management.
Over the last several years, the leadership of VMware has accurately predicted the need for a more sophistaced cloud framework and as such developed and released the VMware vCloud API, acquired CloudFoundry, and developed and released a product called VMware vCloud Director. What exactly are these three pieces?
vCloud API - a proprietary API used to manipulate a VMware virtual infrastucture
CloudFoundry - an open Java framework used to develop and publish webapps
vCloud Director - a proprietary software product that manipulates a VMware virtual infrastructure as well as provides an interface for multi-tenant management and multi-tenant consumption. vCloud Director provides network isolation through the mandatory use of VMware vShield, a proprietary solution that works only with VMware virtual infrastructures.
About VMware vCloud Director
VMware acquired Akimbi in 2006 which lead to the VMware Lab Manager product. Lab Manager was a reasonably successful solution for environments with large developer footprints, classrooms, and other multi-VM environments that needed to be duplicated easily and in a way that allowed segregation from other copies of the same environment. This segregation, at a network level, is called fencing in VMware terminology. VMware was wise and took the technology used in VMware Lab Manager and developed it into their flagship Infrastructure-as-a-Service play, the now famous, Project Redwood (internal name for vCloud Director before a GA name was finalized). vCloud Director is used to provide the segregation of tenants, the financial chargeback (integrated with vCenter Chargeback), the web-based management portals, and other components necessary for an organization to turn themselves into a Service Provider.
To those that have seen the demos, the Youtube videos, the whitepapers, or the regular tweets about the product, or have actually stood up a vCloud Director environment, there is no denying that the product looks appealing. Watching a day-to-day VMware administrator (of VCP level or higher) try and design a solution on vCloud Director the first time is a painful experience, as the design of a multi-tenant virtual infrastructure is wildly complex. Design considerations include:
Management
Consumption
Network
Authentication
Storage
Compute
Cost-modeling
No why go through the exhaustive effort of designing an IaaS solution to only have it work with one vendor's platform? What happens the during the next vRam-type licensing change. What happens if an organization wants to spin up a large Hadoop cluster in the Amazon (the market leader, by the way) cloud to process data and then quickly tear it down? This likely isn't happening in a vCloud Director cloud.
It was pointed out to me by a smart VMware employee that the proprietary nature of VMware's cloud solution, to include vShield and the vNetwork Distributed Switch, is what makes the vCloud story so compelling. Without that level of proprietary integration it becomes difficult to wraps detailed policies around a WebApp or collection of virtual machines. VMware products all speak VMware, so natively they work together.
One of my many concerns with this VMware-centric story is the lock-in to vShield, vNetwork Distributed Switch/Cisco Nexus 1KV. I'd much prefer the ability to plug in the virtual/physical network solution, I prefer. Perhaps I want a software-defined network (SDN) solution that incorporates OpenFlow.
I've demo'd the vCloud Director product for customers before, and it is generally well received at first glance. However, here are some of the things to consider before going with VMware vCloud Director:
Proprietary (vCloud Director) on top of proprietary (vSphere only) requiring proprietary (vShield) integrated with proprietary (vCenter Chargeback)
Proprietary API (more on in a later article in this series)
Pricing model
Lack of flexibility (only one hypervisor)
Abstracting the Operating System
One of the emerging trends of Cloud and virtualization in general, is the increasing importance of, the apps that an organization uses and the decreasing importance (to some degree) of, the operating systems or the virtual machines themselves. Pick a random keynote from any of the forward-looking CEOs out there and one of the key messages will be, "it's all about the apps!" This is true but why?
When I'm using BankOfAmerica.com to pay bills, do I care what the underlying operating system is that the webapp is running on? I don't. I only care that it's fast.
When I'm using Twitter, or HROnline, or Paychex, or ADP, or JayCut, or SlideRocket, the only thing I care about is my user experience, the webapp's reliability, its grace in handling errors (I prefer not to see the fail whale at all), and how well it works on my device.
As PCs share more desk real estate with Macs, as mobile phones become people's primary connectivity device, and as tablets continue to propagate within the work place, an app's usability is more important now than ever. At Interop last week, Rajen Sheth from Google talked about the importance of the browser. Scott Davis (VMware) and Edwin Yuen (Microsoft) may not have all openly agreed but nobody denied this claim openly either.
If I'm developing a new app for my organization to be used by a diverse user population I have to think about how the app will run on the different devices. Writing an app to run on a Windows desktop OS isn't likely going to work for Mac users natively (would require VMware Fusion, for example) and certainly isn't going to help iPad users unless they are connecting to a Windows vDesktop (which loses it's novelty quickly - see this article).
What do all of these devices have in common? They all have a web-browser.
Great, but what's your point?
Developers are developing apps while infrastructure folks are thinking about how to scale out the application, maintain availability, provide reasonable response time, and provide recoverability.
Developers are developing apps on platforms. These platforms could live on an array of underlying supported environments but the developer doesn't or shouldn't care. Are the libraries I'm calling installed and available? Great. Is the droplet execution agent (DEA) available so new webapps can be provisioned (Cloud Foundry example)? That's about all I care about.
PaaS?
PaaS isn't about virtual machines or instances, per se, it's about applications, web-based development (typically), and the supporting environments. Google AppEngine and Force.com are two examples of PaaS. While I am not a huge fan of VMware's current IaaS play, PaaS is an area where VMware has a potentially strong cloud play.
Cloud Foundry, an open source PaaS solution developed by VMware has a few flavors:
CloudFoundry.com - Hosted PaaS environment; you (will) pay to run WebApps here (when it goes GA)
CloudFoundry.org - The (free) open source project
that you could stand up on your own infrastructure
Micro Cloud Foundry - Complete Cloud Foundry environment that is meant to run on a developer's machine
Putting it all together - Components of Cloud
So what are some of the major pieces of a cloud environment?
Developer framework + tools
Platform-as-a-Service
Infrastructure-as-a-Service
Hardware
Developers write their apps in a given language and framework and (ideally) publish their app to a PaaS provider, whether hosted or on-prem. The PaaS environment runs on an infrastructure, very likely virtualized. The management, automation, billing, etc., is managed by an IaaS solution. Again, the infrastructure could reside on-premise or in a service provider's environment.
This series will focus primarily on Infrastructure-as-a-Service, which is where both VMware vCloud Director and CloudStack reside.
Obligatory mention about Amazon Web Services
It would be remiss to not mention Amazon Web Services (AWS), as AWS, more than anyone else, has proven that it is possible to put together a cloud infrastructure at mega-scale as well as make profit on the sale of the cloud resources. There are many great success stories of companies using AWS to quickly meet the demand of a well-placed TV advertisement or spin up instances to perform some complicated number-crunching. This is not possible (quickly scaling to meet demand) in someone's datacenter unless they have an irresponsible amount of excess capacity.
AWS meets this mega-scale need for a growing number of organizations. VMware's mesage is to leverage the vCloud API + vCloud Director solution and then, "burst" to a vCloud Service Provider (e.g. Terremark). While this is certainly possible, at least in theory today, it's hard to imagine any Service Provider matching the affordability of Amazon or th rate of innovation. Not to mention, AWS is still one of the few places I can take a credit card and have compute or storage in a matter of minutes.
For the second or third time in the last year, I just took a credit card to Terremark. The first couple of times I tried to get an instance on-demand from Terremark I waited a very long time for a call back (if I got one at all). This time, the entire process took <45 minutes, which was a good improvement, but still not exactly, on-demand. I'd wager that the added length of time to the process over AWS, for example, is the way Terremark is doing security and account verification.
Congratulations...
Welcome to Terremark’s vCloud Express!
Hello Jason Langone,
Terremark has completed all tasks necessary to activate your vCloud Express account. You may now begin using your account by clicking on the Sign-In URL below and entering the user credentials you created during the Registration process.
Where could AWS fit into your cloud solution?
If the IaaS solution chosen by an organization was able to speak the AWS API, then it could manage the mundane on-prem scale-up and scale-down demands as well as the (perhaps) seasonal demand by using AWS resources.
Enter CloudStack.
About CloudStack
CloudStack is an IaaS solution developed by Cloud.com (acquired by Citrix). CloudStack is unique (when compared to vCloud Director, for example) in that it supports multiple API's, including:
VMware vCenter API (uni-directional TOvCenter, not FROM
For many organizations, multiple API support is a characteristic to look for in an IaaS solution as it provides maximum flexibility. Using the OpenStack API with CloudStack may prove to be the most feature-rich, but many organizations, especially the early adopters of public cloud, have likely standardized on the AWS API set, and for good reason. These AWS API organizations don't have to change the custom solutions they have put together, since CloudStack supports the AWS API. For those organizations looking to standardize on the vCloud API (a solid API it must be said), can still use CloudStack.
Organizations that have use-cases of seasonal, spike, or unpredictable demand, likely will incorporate AWS resources into their overall company's cloud strategy. For example, file servers, print servers, domain controllers and some middle-ware servers may reside on-prem at a given organization. At the same organization, their public facing webservers, customer-facing webapps, and required databases reside in AWS and are built to scale up and down on demand. If one IaaS framework can be used to orchestrate both the on-prem and the AWS resources, that's a powerful solution.
In closed environments with closed-networks, where the use of AWS is unlikely, vCloud may be a suitable solution for them (especially if a long-term VMware customer).
CloudStack is platform that allows an organization to manage their on-prem in the same manner they manage their service provider resources (likely to be an AWS or Rackspace).
CloudStack comes in two offerings:
F/LOSS - Free, open-source software
F/LOSS with Support - Free, open source software with PAID support
A Few Notes about OpenStack
While I originally was under the impression that these two products are headed down a path of true convergence, it has been explained to me that this is not necessarily the case. As such, the adoption of some OpenStack components, such as Swift, the scalable object storage component, may be included in a solution that uses CloudStack. As such, Swift will be covered in a later article in the series to store content such as gold VMs, images, and other relatively static data.
Conclusion
I welcome you to this new series, "CloudStack for Virtualization Engineers." While it is not meant entirely for the VCPs of the world, I will write for that audience as I've been on many ends of the ESX spectrum (admin, engineer, architect, support, etc.). The series is meant as an educational journey to showcase a platform many folks may not be familiar with; however more and more virt admins are being asked to come up with a, "cloud strategy," and CloudStack should be in strong consideration in such discussions. Welcome to the series and to open discussion!
