all things open

In a prior post on Red Hat’s Open Hybrid Cloud Architecture I discussed how IT consumers, having experienced the power of the public cloud are pressing Enterprise IT to deliver new capabilities. One of these capabilities is accelerated service delivery, or the ability to more quickly develop and release new applications that meet a businesses need. In this post I’d like to examine how the Open Hybrid Cloud Architecture provides the means to satisfy this capability and how it is different then other approaches.

There are 1000 vendors who can provide accelerated service delivery, why not just buy a product?
Many vendors will try to sell a single product as being able to accelerate service delivery. The problem with this approach is that accelerating service delivery goes far beyond a single product. This is because no single product can provide all the necessary components of application development that an IT consumer could want. Think about all the languages, frameworks, and technologies from Java, .NET, node.js to Hadoop, Casandra, Mongo to <insert your favorite technology name here>. The availability of these languages from a single product, vendor, or operating system in an optimized manner is highly unlikely. An approach that tries to accelerate service delivery within a single product or technology creates yet another silo and doesn’t solve the fundamental problem of accelerating service delivery across all an IT organization’s assets.

How can Enterprise IT provide accelerated service delivery capabilities while avoiding a silo?
By leveraging an architecture that is flexible and where each component is aware of it’s neighbors, organizations can accelerate service delivery without building a silo. Even better, having a component within your architecture that has a comprehensive understanding of every other component means virtually endless possibility for workload deployment and management. Want to deploy your workload as a VM using PXE on Red Hat Enterprise Virtualization, a template within VMWare vSphere, instances on OpenStack using Heat, or a gear in OpenShift? You can only do that if you understand each one of those technologies. Don’t build your logic for operations management into a single layer – keep it abstracted to ensure you can plug in whichever implementation of IaaS and PaaS best meets your needs. Does your application maintain too much state locally or scale vertically? Then it belongs on a traditional virtualization platform like VMware or RHEV. Is it a stateless scale out application? Then you can deploy on OpenStack. Are the languages and other dependencies available within a PaaS? Then it belongs in OpenShift. However, just deploying to each of those platforms is not enough.  What about deploying one part of your workload as gears in OpenShift and another part as instances on OpenStack at the same time? You must be able to deploy to ALL platforms within the same workload definition! The Open Hybrid Cloud Architecture is providing the foundation for such flexibility in deployment and management of workloads in the cloud.

Can you provide an example?
Let’s look at an example of a developer who would like to develop a new application for the finance team within his organization. The developer would like to utilize ruby as a web front end and utilize .NET within an IIS application server to perform some other functions. This developer expects the same capabilities that he gets using Google App Engine in that he wants to be able to push code and have it running in seconds. The user wants to request a catalog item from CloudForms which will provide them with the two components. The first is a ruby application running in the OpenShift PaaS. The second is a virtual machine running on either Red Hat Enterprise Virtualization, VMware vSphere, or Red Hat Open Stack. The service designer who designed this catalog bundle recognized that ruby applications can run in OpenShift and because OpenShift provides greater efficiencies for hosting applications then running the application within it’s own virtual machine the designer ensured that the component run in the PaaS layer. OpenShift also provides automation of the software development process which will give the end user of the designed service greater velocity in development. Since the IIS application server wasn’t available within the PaaS layer, the service designer utilized a virtual machine at the datacenter virtualization layer (vSphere) to provide this capability.

Step by Step

1. The user requests the catalog item. CloudForms could optionally provide workflow (approval, quota, etc) and best fit placement at this point.

2. CloudForms provisions the ruby application in OpenShift Enterprise. The Ruby application is running as a gear.

3. CloudForms orchestrates the adding of an action hook into the OpenShift deployment. This can be done using any configuration management utility. I used puppet and The Foreman in my demo video below.

4. The user begins developing their ruby application. They clone the repository and then commit and push the changes.

5. The action hook within OpenShift is triggered by the deploy stage of the OpenShift lifecycle and calls CloudForms API requesting a virtual machine be created.

6. CloudForms provisions the virtual machine.

This is really just the beginning of the process, but hopefully you can see where it’s going. CloudForms can perform the deployment and tear down of the virtual machines each time a developer updates their application in OpenShift. It can even tie into other continuous integration systems to deploy application code into the IIS application server. This rapid delivery of the environment is taking place across both the PaaS and IaaS. It also doesn’t try to invent a new “standard description” across all different types of models, instead it understands the models and methods of automation within each component of the architecture and orchestrates them. While the virtual machines running at the IaaS layer don’t provide the same level of density as the PaaS, CloudForms and OpenShift can be combined to provide similar operational efficiency and expand the capabilities of OpenShift’s Accelerated Service Delivery across an IT organizations entire base of assets.

I still don’t believe you, can you show me?
Want to see it in action? Check out this short video demonstration in either Ogg or Quicktime format.

You can download the action hook here.

You can download the OpenOffice Draw Diagram here.

This is cool, what would be even cooler?
If the client tools could be intercepted by CloudForms it could provide a lot of operational management capabilities to OpenShift. For example, when `rhc app create` is run CloudForms could provide approvals, workflow, quota to the OpenShift applications. Or perhaps a future command such as `rhc app promote` could utilize the approvals and automation engine inside CloudForms to provide controlled promotions of applications through a change control process.

OpenShift Enterprise, Red Hat’s Platform as a Service (PaaS), handles the management of application stacks so developers can focus on writing code. The result is faster delivery of services to organizations. OpenShift Enterprise runs on infrastructure, and that infrastructure needs to be both provisioned and managed. While provisioning OpenShift Enterprise is relatively straightforward, managing the lifecycle of the OpenShift Enterprise deployment requires the same considerations as other enterprise applications such as updates and configuration management. Moreover, while OpenShift Enterprise can scale applications running within the PaaS based on demand the OpenShift Enterprise infrastructure itself is static and unaware of the underlying infrastructure. This is by design, as the mission of the PaaS is to automate the management of application stacks and it would limit flexibility to tightly couple the PaaS with the compute resources at both the physical and virtual layer. While this architectural decision is justified given the wide array of computing platforms that OpenShift Enterprise can be deployed upon (any that Red Hat Enterprise Linux can run upon) many organizations would like to not only dynamically scale their applications running in the PaaS, but dynamically scale the infrastructure supporting the PaaS itself. Organizations that are interested in scaling infrastructure in support of OpenShift Enterprise need not look further then CloudForms, Red Hat’s Open Hybrid Cloud Management Framework. CloudForms provides the capabilities to provision, manage, and scale OpenShift Enterprise’s infrastructure automatically based on policy.

For reference, the two previous posts I authored covered deploying the OpenShift Enterprise Infrastructure via CloudForms and deploying OpenShift Enterprise Applications (along with IaaS elements such as Virtual Machines) via CloudForms. Below are two screenshots of what this looks like for background.

Operations User Deploying OpenShift Enterprise Infrastructure via CloudForms

Self-Service User Deploying OpenShift Application via CloudForms

Let’s examine how these two automations can be combined to provide auto scaling of infrastructure to meet the demands of a PaaS. Today, most IT organizations monitor applications and respond to notifications after the event has already taken place – particularly when it comes to demand upon a particular application or service. There are a number of reasons for this approach, one of which is a result of the historical “build to spec” systems that existed in historical and currently designed application architectures. As organizations transition to developing new applications on a PaaS, however, they are presented with an opportunity to reevaluate the static and often oversubscribed nature of their IT infrastructure. In short, while applications designed in the past were not [often] built to scale dynamically based on demand, the majority of new applications are, and this trend is accelerating. Inline with this accelerating trend the infrastructure underlying these new expectations must support this new requirement or much of the business value of dynamic scalability will not be realized. You could say that an organizations dynamic scalability is bounded by their least scalable layer. This also holds true for organizations that intend to run solely on a public cloud and will leverage any resources at the IaaS layer.

Here is an example of how scalability of a PaaS would currently be handled in many IT organizations.

The operations user is alerted by a monitoring tool that the PaaS has run out of capacity to host new or scale existing applications.

The operations user utilizes the IaaS manager to provision new resources (Virtual Machines) for the PaaS.

The operations user manually configures the new resources for consumption by the PaaS.

Utilizing CloudForms to deploy manage, and automatically scale OpenShift Enterprise alleviates the risk of manual configuration from the operations user while dynamically reclaiming unused capacity within the infrastructure. It also reduces the cost and complexity of maintaining a separate monitoring solution and IaaS manager. This translates to lower costs, greater uptime, and the ability to serve more end users. Here is how the process changes.

By notification from the PaaS platform or in monitoring the infrastructure for specific conditions CloudForms detects that the PaaS Infrastructure is reaching its capacity. Thresholds can be defined by a wide array of metrics already available within CloudForms, such as aggregate memory utilized, disk usage, or CPU utilization.

CloudForms examines conditions defined by the organization to determine whether or not the PaaS should receive more resources. In this case, it allows the PaaS to have more resources and provisions a new virtual machine to act as an OpenShift Node. At this point CloudForms could require approval of the scaling event before moving forward. The operations user or a third party system can receive an alert or event, but this is informational and not a request for the admin to perform any manual actions.

Upon deploying the new virtual machine CloudForms configures it appropriately. This could mean installing the VM from a provisioning system or utilizing a pre-defined template and registering it to a configuration management system such as one based on puppet or chef that configure the system.

Want to see  a prototype in action? Check out the screencast I’ve recorded.

This same problem (the ability to dynamically scale a platform) exists between the IaaS and physical layer. If the IaaS layer runs out of resources it is often not aware of the physical resources available for it to consume. This problem is not found in a large number of organizations because dynamically re-purposing physical hardware has a smaller and perhaps more specialized set of use cases (think HPC, grid, deterministic workloads). Even though this is the case it should be noted that CloudForms is able to provide a similar level of policy based automation to physical hardware to extend the capacity of the IaaS layer if required.

More and more organizations are beginning to embrace both Infrastructure as a Service (IaaS) and Platform as a Service (PaaS).  These organizations  have already begun asking why PaaS and IaaS management facilities must use different management frameworks. It only seems natural that IT organization’s customers should be able to select both IaaS and PaaS elements during their self-service workflow. Likewise, operations teams within IT organizations prefer to be able to utilize the same methods of policy, control, and automation across both IaaS and PaaS elements. In doing so operations teams could optimize workload placement both inside and outside their datacenter and reduce duplication of effort. This isn’t just a realization that customers are coming to – analysts have also been talking about the convergence of IaaS and PaaS as a natural evolutionary step in cloud computing.

Converged IaaS and PaaS

This convergence of IaaS and PaaS is something I referred to as a Hybrid Service Model in a previous post, but you may often hear it refereed to as Converged IaaS and PaaS. There are many detriments an IT organization that does not embrace the convergence of IaaS and PaaS will face. Some of the more notable detriments include the following.

• Developers: Slower delivery of services
  • Developers accessing two self-service portals in which the portals do not have knowledge of each others capabilities leads to slower development and greater risk of human error due to less automated processes on workload provisioning and management.
• Operations: Less efficient use of resources
  • Operations teams managing IaaS and PaaS with two management facilities will be unable to maximize utilization of resources.
• Management: Loss of business value
  • IT managers will be unable to capitalize efficiently without an understanding of both IaaS and PaaS models.

For these reasons and many more, it’s imperative that organizations make decisions today that will lead them to the realization of a Hybrid Service Model. There are two approaches emerging in the industry to realizing a Hybrid Service Model. The first approach is to build a completely closed or semi-open solution to allowing for a Hybrid Service Model. A good example would be a vendor offering a PaaS as long as it runs on top of a single virtualization provider (conveniently sold by them). The second approach is one in which a technology company utilizes an approach based on the tenants of an Open Hybrid Cloud to provide a fully open solution to enabling a Hybrid Service Model. I won’t go into all the reasons the second approach is better – you can read about that more here and here – but I will mention that Red Hat is committed to the Open Hybrid Cloud approach to enabling a Hybrid Service Model.

With all the background information out of the way I’d like to show you a glimpse of what will be possible due to the Open Hybrid Cloud approach at Red Hat. Red Hat is building the foundation to offer customers Hybrid Service Models alongside Hybrid Deployment Scenarios. This is possible for many reasons, but in this scenario it is primarily because of the open APIs available in OpenShift, Red Hat’s PaaS and because of the extensibility of CloudForms, Red Hat’s Hybrid Cloud Management solution. The next release of CloudForms will include a Management Engine component, based on the acquisition of ManageIQ EVM that occurred in December. Using the CloudForms Management Engine it is possible to provide self-service of applications in a PaaS along with self-service of infrastructure in IaaS from a single catalog. Here is what a possible workflow would look like.

Higher resolution viewing in quicktime format here.

In the previous post I examined how Red Hat Network (RHN) Satellite could be integrated with ManageIQ Enterprise Cloud Management (ECM). With this integration in place Satellite could provide ECM with the content required to install an operating system into a virtual machine and close the loop in ongoing systems management. This was just a first look and there is a lot of work to be done to enable discovery of RHN Satellite and best practice automation out of the box via ECM. That said, the combination of ECM and RHN Satellite provide a solid foundation for proceeding to use cases higher in the stack.

With this in mind, I decided to attempt automating a self-service deployment of OpenShift using ManageIQ ECM, RHN Satellite, and puppet.

Lucky for me, much of the heavy lifting had already been done by Krishna Raman and others who developed puppet modules for installing OpenShift Origin. There were several hurdles that had to be overcome with the existing puppet modules for my use case:

1. They were built for Fedora and OpenShift Origin and I am using RHEL6 with OpenShift Enterprise. Because of this they defaulted to using newer rubygems that weren’t available in openshift enterprise yet. It took a little time to reverse engineer the puppet modules to understand exactly what they were doing and tweak them for OpenShift Enterprise.
2. The OpenShift Origin puppet module leveraged some other puppet modules (stdlib, for example), so the puppet module tool (PMT) was needed which is not available in core puppet until > 2.7. Of course, the only version of puppet available in EPEL for RHEL 6 was puppet-2.6. I pulled an internal build of puppet-2.7 to get around this, but still required some packages from EPEL to solve dependencies.

Other then that, I was able to reuse much of what already existed and deploy OpenShift Enterprise via ManageIQ ECM. How does it work? Very similar to the Satellite use case, but with the added step of deploying puppet and a puppet master onto the deployed virtual machine and executing the puppet modules.

Workflow of OpenShift Enterprise deployment via ECM

If you are curious how the puppet modules work, here is a diagram that illustrates the flow of the openshift puppet module.

Anatomy of OpenShift Puppet Module

Here is a screencast of the self-service deployment in action.

There are a lot of areas that can be improved in the future. Here are four which were top of mind after this exercise.

First, runtime parameters should be able to be passed to the deployment of virtual machines. These parameters should ultimately be part of a service that could be composed into a deployment. One idea would be to expose puppet classes as services that could be added to a deployment. For example, layering a service of openshift_broker onto a virtual machine would instantiate the openshift_broker class on that machine upon deployment. The parameters required for openshift_broker would then be exposed to the user if they would like to customize them.

Second, gears within OpenShift – the execution area for applications – should be able to be monitored from ECM much like Virtual Machines are today. The oo-stats package provides some insight into what is running in an OpenShift environment, but more granular details could be exposed in the future. Statistics such as I/O, throughput, sessions, and more would allow ECM to further manage OpenShift in enterprise deployments and in highly dynamic environments or where elasticity of the PaaS substrate itself is a design requirement.

Third, building an upstream library of automation actions for ManageIQ ECM so that these exercises could be saved and reused in the future would be valuable. While I only focused on a simple VM deployment in this scenario, in the future I plan to use ECM’s tagging and Event, Condition, Action construct to register Brokers and Nodes to a central puppet master (possibly via Foreman). The thought is that once automatically tagged by ECM with a “Broker” or “Node” tag an action could be taken by ECM to register the systems to the puppet master which would then configure the system appropriately. All those automation actions are exportable, but no central library exists for these at the current time to promote sharing.

Fourth, and possibly most exciting, would be the ability to request applications from OpenShift via ECM alongside requests for virtual machines. This ability would lead to the realization of a hybrid service model. As far as I’m aware, this is not provided by any other vendor in the industry. Many of the analysts are coming around to the fact that the line between IaaS and PaaS will soon be gray. Driving the ability to select an application that is PaaS friendly (python for example) and traditional infrastructure components (a relational database for example) from a single catalog would provide a simplified user experience and create new opportunities for operations to drive even higher utilization at lower costs.

I hope you found this information useful. As always, if you have feedback, please leave a comment!