A Technical Overview of Red Hat Cloud Infrastructure (RHCI)

I’m often asked for a more in-depth overview of Red Hat Cloud Infrastructure (RHCI), Red Hat’s fully open source and integrated Infrastructure-as-a-Service offering. To that end I decided to write a brief technical introduction to RHCI to help those interested better understand what a typical deployment looks like, how the components interact, what Red Hat has been working on to integrate the offering, and some common use cases that RHCI solves. RHCI gives organizations access to infrastructure and management to fit their needs, whether it’s managed datacenter virtualization, a scale-up virtualization-based cloud, or a scale-out OpenStack-based cloud. Organizations can choose what they need to run and re-allocate their resources accordingly.


RHCI users can choose to deploy either Red Hat Enterprise Virtualization (RHEV) or Red Hat Enterprise Linux OpenStack Platform (RHEL-OSP) on physical systems to create a datacenter virtualization-based private cloud using RHEV or a private Infrastructure-as-a-Service cloud with RHELOSP.

RHEV comprises a hypervisor component, referred to as RHEV-H, and a manager, referred to as RHEV-M. Hypervisors leverage shared storage and common networks to provide common enterprise virtualization features such as high availability, live migration, etc.

RHEL-OSP is Red Hat’s OpenStack distribution that provides massively scalable infrastructure by providing the following projects (descriptions taken directly from the projects themselves) for use on one of the largest ecosystems of certified hardware and software vendors for OpenStack:

Nova: Implements services and associated libraries to provide massively scalable, on demand, self service access to compute resources, including bare metal, virtual machines, and containers.

Swift: Provides Object Storage.

Glance: Provides a service where users can upload and discover data assets that are meant to be used with other services, like images for Nova and templates for Heat.

Keystone: Facilitate API client authentication, service discovery, distributed multi-tenant authorization, and auditing.

Horizon: Provide an extensible unified web- based user interface for all integrated OpenStack services.

Neutron: Implements services and associated libraries to provide on-demand, scalable, and technology-agnostic network abstraction.

Cinder: Implements services and libraries to provide on-demand, self-service access to Block Storage resources via abstraction and automation on top of other block storage devices.

Ceilometer: Reliably collects measurements of the utilization of the physical and virtual resources comprising deployed clouds, persist these data for subsequent retrieval and analysis, and trigger actions when defined criteria are met.

Heat: Orchestrates composite cloud applications using a declarative template format through an OpenStack-native ReST API.

Trove: Provides scalable and reliable Cloud Database as a Service functionality  for both relational and non-relational database engines, and to continue to improve its fully-featured and extensible open source framework.

Ironic: Produces an OpenStack service and associated python libraries capable of managing and provisioning physical machines, and to do this in a security-aware and fault-tolerant manner.

Sahara: Provides a scalable data processing stack and associated management interfaces.

Red Hat CloudForms, a Cloud Management Platform based on the upstream ManageIQ project, provides hybrid cloud management of OpenStack, RHEV, Microsoft Hyper-V, VMware vSphere, and Amazon Web Services. This includes the ability to provide rich self-service with workflow and approval, discovery of systems, policy definition, capacity and utilization forecasting, and chargeback among others capabilities. CloudForms is deployed as a virtual appliance and requires no agents on the systems it manages. CloudForms has a region and zone concept that allows for complex and federated deployments across large environments and geographies.

Red Hat Satellite is a systems management solution for managing the lifecycle of RHEV, RHEL-OSP, and CloudForms as well as any tenant workloads that are running on RHEV or RHEL-OSP. It can be deployed on bare metal or, as pictured in this diagram, as a virtual machine running on either RHEV or RHEL-OSP. Satellite supports a federated model through a concept called capsules.

CloudForms is a Cloud Management Platform that is deployed as a virtual appliance and supports a federated deployment. It is fully open source just as every component in RHCI is and is based on the ManageIQ project.

One of the key technical benefits CloudForms provides is unified management of multiple providers. CloudForms splits providers into two types. First, there are infrastructure providers such as RHEV, vSphere, and Microsoft Hyper-V. CloudForms discovers and provides uniform information about these systems hosts, clusters, virtual machines, and virtual machine contents in a single interface. Second, there are cloud providers such as RHEL-OSP and Amazon Web Services. CloudForms provides discovery and uniform information for these providers about virtual machines, images, flavors similar to the infrastructure providers. All this is done by leveraging standard APIs provided from RHEV-M, SCVMM, vCenter, AWS, and OpenStack.


Red Hat Satellite provides common systems management among all aspects of RHCI.

Red Hat Satellite provides content management, allowing users to synchronize content such as RPM packages for RHEV, RHEL-OSP, and CloudForms from Red Hat’s Content Delivery Network, to an on-premises Satellite reducing bandwidth consumption and providing an on-premises control point for content management through complex environments. Satellite also allows for configuration management via Puppet to ensure compliance and enforcement of proper configuration. Finally, Red Hat Satellite allows users to account for usage of assets through entitlement reporting and controls. Satellite provides these capabilities to RHEV, RHEL-OSP, and CloudForms, allowing administrators of RHCI to maintain their environment more effectively and efficiently. Equally as important is that Satellite also extends to the tenants of RHEV and RHEL-OSP to allow for systems management of Red Hat Enterprise Linux  (RHEL) based tenants. Satellite is based on the upstream projects of Foreman, Katello, Pulp, and Candlepin.


The combination of CloudForms and Satellite is very powerful for automating not only the infrastructure, but within the operating system as well. Let’s look at an example of how CloudForms can be utilized with Satellite to provide automation of deployment and lifecycle management for tenants.

The automation engine in CloudForms is invoked when a user orders a catalog item from the CloudForms self-service catalog. CloudForms communicates with the appropriate infrastructure provider (in this case RHEV or RHEL-OSP pictured) to ensure that the infrastructure resources are created. At the same time it also ensures the appropriate records are created in Satellite so that the proper content and configuration will be applied to the system. Once the infrastructure resources are created (such as a virtual machine), they are connected to Satellite where they receive the appropriate content and configuration. Once this is completed, the service in CloudForms is updated with the appropriate information to reflect the state of the users request allowing them access to a fully compliant system with no manual interaction during configuration. Ongoing updates of the virtual machine resources can be performed by the end user or the administrator of the Satellite dependent on the customer needs.


This is another way of looking at how the functional areas of the workflow are divided in RHCI. Items such as the service catalog, quota enforcement, approvals, and workflow are handled in CloudForms, the cloud management platform. Even still, infrastructure-specific mechanisms such as heat templates, virtual machine templates, PXE, or even ISO-based deployment are utilized by the cloud management platform whenever possible. Finally, systems management is used to provide further customization within the operating system itself that is not covered by infrastructure specific provisioning systems. With this approach, users can separate operating system configuration from the infrastructure platform thus increasing portability. Likewise, operational decisions are decoupled from the infrastructure platform and placed in the cloud management platform allowing for greater flexibility and increased modularity.


Common management is a big benefit that RHCI brings to organizations, but it doesn’t stop there. RHCI is bringing together the benefits of shared services to reduce the complexity for organizations. Identity is one of the services that can be made common across RHCI through the use of Identity Management (IDM) that is included in RHEL. All components of RHCI can be configured to talk to IDM which in turn can be used to authenticate and authorize users. Alternatively, and perhaps more frequently, a trust is established between IDM and Active Directory to allow for authentication via Active Directory. By providing a common identity store between the components of RHCI, administrators can ensure compliance through the use of access controls and audit.


Similar to the benefits of shared identity, RHCI is bringing together a common network fabric for both traditional datacenter virtualization and infrastructure-as-a-service (IaaS) models. As part of the latest release of RHEV, users can now discover neutron networks and begin exposing them to guest virtual machines (in tech preview mode). By building a common network fabric organizations can simplify their architecture. No longer do they need to learn two different methods for creating and maintaining virtual networks.


Finally, Image storage can now be shared between RHEV and RHEL-OSP. This means that templates and images stored in Glance can be used by RHEV. This reduces the amount of storage required to maintain the images and allows administrators to update images in one store instead of two, increasing operational efficiency.


One often misunderstood area is around what capabilities are provided by which components of RHCI.  RHEV and OpenStack provide similar capabilities with different paradigms. These focus around compute, network, and storage virtualization. Many of the capabilities often associated with a private cloud include features found in the combination of Satellite and CloudForms. These include capabilities provided by CloudForms such as discovery, chargeback, monitoring, analytics, quota Enforcement, capacity planning, and governance. They also include capabilities that revolve around managing inside the guest operating system in areas such as content management, software distribution, configuration management, and governance.


Often organizations are not certain about the best way to view OpenStack in relation to their datacenter virtualization solution. There are two common approaches that are considered. Within one approach, datacenter virtualization is placed underneath OpenStack. This approach has several negative aspects. First, it places OpenStack, which is intended for scale out, over an architecture that is designed for scale up in RHEV, vSphere, Hyper-V, etc. This gives organizations limited scalability and, in general, an expensive infrastructure for running a scale out IaaS private cloud. Second, layering OpenStack, a Cloud Infrastructure Platform, on top of yet another infrastructure management solution makes hybrid cloud management very difficult because Cloud Management Platforms, such as CloudForms, are not designed to relate OpenStack to a virtualization manager and then to underlying hypervisors. Conversely, by using a Cloud Management Platform as the aggregator between infrastructure platforms of OpenStack, RHEV, vSphere, and others, it is possible to achieve a working approach to hybrid cloud management and use OpenStack in the massively scalable way it is designed to be used.


RHCI is meant to complement existing investments in datacenter virtualization. For example, users often utilize CloudForms and Satellite to gain efficiencies within their vSphere environment while simultaneously increasing the cloud-like capabilities of their virtualization footprints through self-service and automation. Once users are comfortable with the self-service aspects of CloudForms, it is simple to supplement vSphere with lower cost or specialized virtualization providers like RHEV or Hyper-V.

This can be done by leveraging the virt-v2v tools (shown as option 1 in the diagram above) that perform binary conversion of images in an automated fashion from vSphere to other platforms. Another approach is to standardize environment builds within Satellite (shown as option 2 in the diagram above) to allow for portability during creation of a new workload. Both of these methods are supported based on an organization’s specific requirements.


For scale-out applications running on an existing datacenter virtualization solution such as VMware vSphere RHCI can provide organizations with the tools to identify (discover), and move (automated v2v conversion), workloads to Red Hat Enterprise Linux OpenStack Platform where they can take advantage of massive scalability and reduced infrastructure costs. This again can be done through binary conversion (option 1) using CloudForms  or through standardization of environments (option 2) using Red Hat Satellite.


So far I have focused primarily on the integrations between the components of Red Hat Cloud Infrastructure to illustrate how Red Hat is bringing together a comprehensive Infrastructure-as-a-Service solution, but RHCI integrates with many existing technologies within the management domain. From integrations with configuration management solutions such as Puppet, Chef, and Ansible, and many popular Configuration Management Databases (CMDBs) as well networking providers and IPAM systems, CloudForms and Satellite are extremely extensible to ensure that they can fit into existing environments.


And of course, with Red Hat Enterprise Linux forming the basis of both Red Hat Enterprise Virtualization and Red Hat Enterprise Linux OpenStack Platform leading to one of the largest ecosystems of certified compute, network, and storage partners in the industry.

RHCI is a complete and fully open source infrastructure-as-a-service private cloud. It has industry leading integration between a datacenter virtualization and openstack based private cloud in the areas of networking, storage, and identity. A common management framework makes for efficient operations and unparallelled automation that can also span other providers. Finally, by leveraging RHEL and Systems Management and Cloud Management Platform based on upstream communities it has a large ecosystem of hardware and software partners for both infrastructure and management.

I hope this post helped you gain a better understanding of RHCI at a more technical level. Feel free to comment and be sure to follow me on twitter @jameslabocki

Tagged , , , , , , , , , ,

Docker all the OpenStack Services Presentation

Slides from the presentation Brent Holden and I gave at OpenStack Summit can be downloaded here.

A Demonstration of Kolla: Docker and Kubernetes based Deployment of OpenStack Services on Atomic

The Problem

Screen Shot 2014-10-22 at 8.39.48 AM

The Beauty of OpenStack

OpenStack is a thing of beauty, isn’t it? Just look at all those cleanly defined services, perfectly atomic, able to run standalone … it’s simply amazing. What more could developers and operators ask for in a cloud?

Screen Shot 2014-10-22 at 8.42.30 AM

The Reality of OpenStack

Except, that it’s not exactly like that. All those services heavily rely on each other and given the rate of change OpenStack is experiencing the degree of complexity only stands to increase. The problem is that OpenStack has many services that are dependent on one another and managing the lifecycle is difficult and inefficient because of this.

Let’s look at an example of updating the keystone service, OpenStack’s identity management service. It is difficult to know whether or not deploying a new version of Keystone into an existing OpenStack deployment will cause problems because of compatibility with others services. It’s also difficult to move backwards and expensive to roll back a deployment of a new keystone service with today’s tools. Operators don’t want to use extra racks of hardware to test an upgrade of a service if they can avoid it and no lifecycle management tools that try to imperatively deploy and roll back can do so as reliability as we’d like between OpenStack releases.

At this point you might conclude that I have a personal vendetta against OpenStack. Although this could be justified after the many nights I’ve spent installing, configuring, and upgrading OpenStack I can assure you that’s not the case. In truth, OpenStack is not a beautiful and unique snowflake. Lots of different infrastructure platforms face this same problem and so do many application platforms.

The Many Paths to OpenStack Lifecycle Management

Today, there are many ways to manage the lifecycle of OpenStack services, but the two most prevalent can be loosely grouped into two categories: build based and image based deployments.

Build based lifecycle management uses a build service, such as PXE, and is typically coupled together with a bunch of lifecycle management tools and  almost always uses some type of configuration management whether that’s Puppet, Chef, Ansible, or others.

Screen Shot 2014-10-22 at 9.40.38 AM

This approach is generally inefficient because each OpenStack service is placed onto a different physical piece of hardware or at least a different operating system.

Screen Shot 2014-10-22 at 9.45.48 AM

It is possible to combine multiple services on a single operating system, but this can get tricky. How does the lifecycle management tool know that OpenStack Service A in the image above won’t conflict with OpenStack Service B in terms of resources required, ports required, file systems, etc? It takes an awful lot of logic in a lifecycle management tool to know this and given the rate of change experienced in a community like OpenStack, lifecycle management tools have a hard time keeping up and delivering what users would like to deploy. Could virtual machines be used here? Possibly, but virtual machine are heavyweight and also lack rich metadata or require large infrastructures and agents loaded into those virtual machines to get metadata. In other words, VMs are too heavy and they also lack the concept of inheritance.

Screen Shot 2014-10-22 at 10.00.53 AM

Finally, build based deployments can be slow. Copying each package back and forth over the wire is not the most efficient way of deploying at scale.

Image based deployments solve the problem of slow performance that build based systems have by not requiring each package to be installed. Typically an image based system has some sort of image building tool that stores images in a repository and these images are then streamed down to physical hardware.

Screen Shot 2014-10-22 at 10.12.33 AM

However, even while using images, incremental updates can be slow due to the large size of images. Also, the expense of pushing a large image around for small incremental updates doesn’t seem appropriate.

Screen Shot 2014-10-22 at 10.12.42 AM

Even more importantly, image based deployments don’t solve the fundamental problem of complexity that understanding the relationships between OpenStack services presents. This problem is only moved earlier in the process and must be solved when building the images themselves instead of at run-time.

There is one other consideration that should be taken when looking at building a lifecycle management solution for OpenStack and that is that OpenStack doesn’t live alone. The last thing most operators want is yet another way to manage the lifecycle of a new platform. They’d like something that they can use across platforms from bare metal, to IaaS, and possibly even in a PaaS.


What Atomic, Docker, and Kubernetes Bring to the Party

Wouldn’t it be great if there was a solution for managing the lifecycle of Openstack services that was:

  • Isolated, lightweight, Portable, and Separated
  • Easily Described run-time relationships
  • Could run on something thin and easy to update
  • Worked to manage the lifecycle of services beyond OpenStack too

That’s exactly what the combination of Docker, Kubernetes, and Atomic can provide to the existing lifecycle management solutions.

Screen Shot 2014-10-22 at 10.32.57 AM

Docker provides a level of abstraction for Linux Containers through APIs and an “Engine”. It also provides an image format for sharing that supports a base and child image relationship allowing for layering. Finally, Docker provides a registry for sharing docker images. This is important because it allows developers to ship a portable image that operators can deploy on a different platform.

Screen Shot 2014-10-22 at 10.34.25 AM

Kubernetes is an open source container cluster manager. It provides scheduling of Linux Containers using a master/minion construct. It uses a declarative syntax to express desired state. This is important because it allows developers to provide a description of the relationships between different Linux Containers and let’s the cluster manager do the scheduling.

Screen Shot 2014-10-22 at 10.35.39 AM

Atomic provides just enough of an operating system to run containers in a secure, stable, and high performance manner. It includes Kubernetes and Docker and allows for users to update using newly developed update mechanisms such as OSTree. Here is a quick video that shows how easy it is to deploy atomic (in this case on OpenStack) and also how easy it is to upgrade Atomic. Watch OGG


Screen Shot 2014-10-23 at 2.20.49 PM

So when you put these pieces together what you end up with is something that looks (at a high level) like the diagram above. OpenStack developers are free to develop on a broad choice of platforms (Linux/Vagrant/Libvirt pictured) and can publish completed images to a registry. Operators on the other side would pull the kubernetes configurations into their lifecycle management tools and the tools would launch the pods and services. This would trigger Docker running on Atomic to pull the images locally and deploy containers with the OpenStack services. Services are isolated and (we are fairly certain given our experience with our OpenShift PaaS) lots and lots of containers could be run on a single operating system to maximize density of Openstack services. There are LOTS of other benefits including ease of rollback, deployment and update speed, etc, but this alone should be enough for anyone looking at running an OpenStack cloud at scale to be interested.

 Show me the Demo!

Screen Shot 2014-10-23 at 2.21.29 PM

Here are several demonstrations that illustrate the scenario above. These are a demonstration of the OpenStack Kolla project and were produced in 2 weeks time by a group of amazing developers who saw the potential these technologies had.

First there is building the images and pushing them to a registry.  Watch OGG

Second there is deploying a few pods and services manually to see how they connect and what Kubernetes and Docker are actually doing. Watch OGG

Finally, there is an example of deploying all the OpenStack services that were completed in milestone-1 all with a single command.  Watch OGG

After deploying OpenStack countless times I can say that when you see each schema automatically created in MariaDB and endpoints, services, etc automatically created all in under a minute it is an amazing feeling!

“I’m Sold, What’s Next?”

In the end, the combination of Docker, Atomic, and Kubernetes show the promise of alleviating some of the pain OpenStack developers and operators have experienced. There are still a lot of unanswered questions, but we feel that this combination of technologies shows promise and are excited that they have found a home in the TripleO project through Kolla.

If you are interested in learning more or participating please:

If you want to learn more about some of the other projects related to this post please check out the following:

Docker All The Things – OpenStack Keystone and Ceilometer Automated Builds on Docker Hub

Ok, I borrowed part of the title of this post from Nicola Paolucci at Atlassian’s blog post who likely borrowed it from a bunch of others, but it was just too good to pass up.

I decided to test Docker Hub’s automated build feature to see if I could have automated docker images created from a project relevant to Red Hat Cloud Infrastructure (RHCI), Red Hat’s private IaaS cloud solution. RHCI combines datacenter virtualization based on Red Hat Enterprise Virtualization (RHEV), scale out IaaS based on Red Hat Enterprise Linux OpenStack Platform (RHELOSP), and cloud management based on CloudForms. These come from the upstream communities of oVirt, OpenStack, and ManageIQ.

If you are interested in why containers could be so beneficial to an Infrastructure as a Service solution you could read my previous post, “Why containers for OpenStack Services?”. The bottom line is that moving more logic about the lifecycle of the IaaS services into the application layer (Think PaaS for IaaS) could solve many problems and help IaaS become much easier to manage.

Keystone Docker Image

The natural choice for the first service to attempt to containerize was the identity service, Keystone. Keystone has (relative to other openstack projects) few moving parts and is also required by most of the other services since it publishes a catalog of endpoints for the other services APIs.


Here is what I did:

1. I forked the openstack keystone project.

Screen Shot 2014-07-07 at 9.15.50 PM

2. I created a new automated build on Docker Hub.

Screen Shot 2014-07-01 at 12.34.24 PM

3. I cloned the git repository for my fork of keystone.

jlabocki@localhost# git clone https://github.com/jameslabocki/keystone.git
Cloning into 'keystone'...
remote: Counting objects: 26085, done.
remote: Compressing objects: 100% (9122/9112), done.
remote: Total 26085 (delta 16076), reused 26085 (delta 16076)
Receiving objects: 100% (1285/1285), 5.61 MiB | 2.14 MiB/s, done.
Resolving deltas: 100% (176/176), done.
Checking connectivity... done.

4. I created a Dockerfile in the root of the cloned keystone repository. Here is the contents of the Dockerfile.

FROM fedora
MAINTAINER jlabocki@redhat.com

# This Dockerfile installs the components of Keystone in a docker image as a proof of concept

#Timestamps are always useful
RUN date > /root/date

#Install required packages
RUN yum install python-pbr git python-devel python-setuptools python-pip gcc gcc-devel libxml2-python libxslt-python python-lxml sqlite python-repoze-lru -y
#RUN yum install python-sqlite2 python-lxml python-greenlet-devel python-ldap sqlite-devel openldap-devel -y

#Clone Keystone and setup
RUN git clone http://github.com/openstack/keystone.git
WORKDIR /opt/keystone
RUN python setup.py install

#Configure Keystone
RUN mkdir -p /etc/keystone
RUN cp etc/keystone.conf.sample /etc/keystone/keystone.conf
RUN cp etc/keystone-paste.ini /etc/keystone/
RUN sed -ri 's/#driver=keystone.identity.backends.sql.Identity/driver=keystone.identity.backends.sql.Identity/' /etc/keystone/keystone.conf 
RUN sed -ri 's/#connection=<None>/connection=sqlite:\/\/\/keystone.db/' /etc/keystone/keystone.conf
RUN sed -ri 's/#idle_timeout=3600/idle_timeout=200/' /etc/keystone/keystone.conf
RUN sed -ri 's/#admin_token=ADMIN/admin_token=ADMIN/' /etc/keystone/keystone.conf

# The following sections build a script that will be executed on launch via ENTRYPOINT

## Start Keystone
RUN echo "#!/bin/bash" > /root/postlaunchconfig.sh
RUN echo "/usr/bin/keystone-manage db_sync" >> /root/postlaunchconfig.sh
RUN echo "/usr/bin/keystone-all &" >> /root/postlaunchconfig.sh

## Create Services
#I'm not sure if exporting works, so I just specify these environment variables on each command, but it might be cleaner to test this
#RUN export OS_SERVICE_ENDPOINT=http://localhost:35357/v2.0
#RUN export OS_AUTH_URL=
RUN echo '/usr/bin/keystone --os_auth_url --os-token ADMIN --os-endpoint service-create --name=ceilometer --type=metering --description="Ceilometer Service"' >> /root/postlaunchconfig.sh
RUN echo '/usr/bin/keystone --os_auth_url --os-token ADMIN --os-endpoint service-create --name=keystone --type=identity --description="OpenStack Identity"' >> /root/postlaunchconfig.sh
RUN chmod 755 /root/postlaunchconfig.sh

#This you will need to substitute your values and run later - the values are:
# CEILOMETER_SERVICE = the id of the service created by the keystone service-create command
# KEYSTONE_SERVICE = the id of the service created by the keystone service-create command
# CEILOMETER_SERVICE_HOST = the host where the Ceilometer API is running
# KEYSTONE_SERVICE_HOST = the host where the Keystone API is running
RUN echo 'keystone --os_auth_url --os-token ADMIN --os-endpoint endpoint-create --region RegionOne --service_id $KEYSTONE_SERVER --publicurl "http://KEYSTONE_SERVICE_HOST:5000/v2.0" --internalurl "http://KEYSTONE_SERVICE_HOST:5000/v2.0" --adminurl "http://KEYSTONE_SERVICE_HOST:35357/v2.0"' > /root/postlaunchconfig.sh
RUN echo 'keystone --os_auth_url --os-token ADMIN --os-endpoint endpoint-create --region RegionOne --service_id $CEILOMETER_SERVICE --publicurl "http://CEILOMETER_SERVICE_HOST:8777/"  --adminurl "http://CEILOMETER_SERVICE_HOST:8777/" --internalurl "http://CEILOMETER_SERVICE_HOST:8777/"' > /root/postlaunchconfig.sh



5. I committed and pushed the change.

[root@localhost keystone]# git commit -m "testing" .
[master fe12eff] testing
1 file changed, 6 insertions(+), 7 deletions(-)
[root@localhost keystone]# git push
Username for 'https://github.com':@.com
Password for 'https://@github.com':
Counting objects: 14, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 335 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
To https://github.com/jameslabocki/keystone.git
a3de5a2..fe12eff  master -> master

6. The automated build finished successfully.

Screen Shot 2014-07-08 at 10.36.53 AM

7. I could check the details of the build, including the Dockerfile used and the output of the build.

Screen Shot 2014-07-08 at 10.38.11 AM

Assuming that the image was good and I could run it and setup Keystone rather quickly I decided to focus on another service and then attempt launching the two together (see the results section if you want to spoil the surprise).

Ceilometer Docker Image

I selected the OpenStack telemetry project, commonly known as Ceilometer, for my next test of an automated build of a docker image to take place on commit to my forked repository. Why Ceilometer? After looking at the OpenStack architecture diagram I thought it might be one of the easier services to run in a container (basically, I used a dart board), and since it only requires keystone I thought I might be able to make it happen next. Here are the components of OpenStack Ceilometer (Telemetry) at a glance taken from the OpenStack docs.

The telemetry system consists of the following basic components:

  • A compute agent (ceilometer-agent-compute). Runs on each compute node and polls for resource utilization statistics. There may be other types of agents in the future, but for now we will focus on creating the compute agent.
  • A central agent (ceilometer-agent-central). Runs on a central management server to poll for resource utilization statistics for resources not tied to instances or compute nodes.
  • A collector (ceilometer-collector). Runs on one or more central management servers to monitor the message queues (for notifications and for metering data coming from the agent). Notification messages are processed and turned into metering messages and sent back out onto the message bus using the appropriate topic. Telemetry messages are written to the data store without modification.
  • An alarm notifier (ceilometer-alarm-notifier). Runs on one or more central management servers to allow settting alarms based on threshold evaluation for a collection of samples.
  • A data store. A database capable of handling concurrent writes (from one or more collector instances) and reads (from the API server).
  • An API server (ceilometer-api). Runs on one or more central management servers to provide access to the data from the data store. These services communicate using the standard OpenStack messaging bus. Only the collector and API server have access to the data store.

Here it is in a diagram.


I decided to start with the database and ceilometer collector and then add the API. I went the route of placing all of these services in a single image. I’m aware there is a lot of debate as to whether Docker images should only run a single process or if multiple processes could be beneficial. My intention wasn’t to optimize the image for production, rather it was to test how easy or difficult it was to take a forked GitHub project and get it into an image build in an automated fashion that I could run on my Fedora 20 workstation. Also, I did not plan to add the evaluator, notifier, or any agents to this image. Since most of the agents require other components of OpenStack.

Here is what I did:

1. I forked the openstack ceilometer project.

Screen Shot 2014-07-01 at 9.21.07 AM

2. I created another new automated build on Docker Hub.

Screen Shot 2014-07-01 at 12.34.24 PM

3. I cloned the git repository for my fork of ceilometer.

jlabocki@localhost# git clone https://github.com/jameslabocki/ceilometer.git
Cloning into ‘ceilometer’…
remote: Counting objects: 26085, done.
remote: Compressing objects: 100% (7112/7112), done.
remote: Total 26085 (delta 16076), reused 26085 (delta 16076)
Receiving objects: 100% (26085/26085), 8.81 MiB | 3.14 MiB/s, done.
Resolving deltas: 100% (16076/16076), done.
Checking connectivity… done.

4. I created a Dockerfile in the root of the project following the manual installation of OpenStack Ceilometer. Here is the contents of the Dockerfile. Note I wasn’t able to run the mongod command during the build successfully. More on that later, I just created a post launch script that could be executed after the docker image is launched as a work around.

FROM fedora
MAINTAINER jlabocki@redhat.com

# This Dockerfile installs some of the components of Ceilometer in a Docker Image as a proof of concept

#Timestamps are always useful
RUN date > /root/date

#Install required packages
RUN yum install mysql-devel openssl-devel wget unzip git mongodb mongodb-server python-devel mysql-server libmysqlclient-devel libffi-devel libxml2-devel libxslt-devel python-setuptools python-pip libffi libffi-devel gcc gcc-devel python-pip python-pbr mongodb python-pymongo rabbitmq-server -y

#RUN pip install tox
#Can't run the line above because https://bugs.launchpad.net/openstack-ci/+bug/1274135, need to specify version 1.6.1
RUN pip install tox==1.6.1

#MongoDB Setup
RUN mkdir -p /data/db
RUN echo 'db.addUser("admin", "insecure", true);' > /root/mongosetup.js

#RabbitMQ Setup
RUN /usr/sbin/rabbitmq-server -detached

#Clone Ceilometer
RUN git clone https://github.com/jlabocki/ceilometer.git /opt/stack/

#Ceilometer Collector Configuration
WORKDIR /opt/stack
RUN python setup.py install
RUN mkdir -p /etc/ceilometer
RUN tox -egenconfig
RUN cp /opt/stack/etc/ceilometer/*.json /etc/ceilometer
RUN cp /opt/stack/etc/ceilometer/*.yaml /etc/ceilometer
RUN cp /opt/stack/etc/ceilometer/ceilometer.conf.sample /etc/ceilometer/ceilometer.conf

#Ceilometer Collector Configuration changes
RUN sed -ri 's/#metering_secret=change this or be hacked/metering_secret=redhat/' /etc/ceilometer/ceilometer.conf
RUN sed -ri 's/#connection=<None>/connection = mongodb:\/\/admin:insecure@localhost:27017\/ceilometer/' /etc/ceilometer/ceilometer.conf

#Ceilometer API Configuration changes
RUN cp etc/ceilometer/api_paste.ini /etc/ceilometer/api_paste.ini

##Ceilometer Post Launch Configuration
RUN echo "#!/bin/bash" > /root/postlaunch.sh

#Add Authenticate against keystone to the post launch script
# KEYSTONE_HOST = the keystone host
RUN echo "sed -ri 's/#identity_uri=<None>/identity_uri=KEYSTONE_HOST/' /etc/ceilometer/ceilometer.conf" >> /root/postlaunch.sh

#Add starting services to the postlaunch script
RUN echo "/bin/mongod --dbpath /data/db --fork --logpath /root/mongo.log --noprealloc --smallfiles" >> /root/postlaunch.sh
RUN echo "/bin/mongo mydb /root/mongosetup.js" >> /root/postlaunch.sh
RUN echo "/usr/bin/ceilometer-collector" >> /root/postlaunch.sh
RUN echo "/usr/bin/ceilometer-api" >> /root/postlaunch.sh
RUN chmod 755 /root/postlaunch.sh

5. I added and committed the change and pushed.

[root@localhost ceilometer]# git commit -m "testing" .
[master fe12eff] testing
1 file changed, 6 insertions(+), 7 deletions(-)
[root@localhost keystone]# git push
Username for 'https://github.com':@.com
Password for 'https://@github.com':
Counting objects: 14, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 335 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
To https://github.com/jameslabocki/ceilometer.git
a3de5a2..fe12eff  master -> master

6. The automated build finished successfully.

7. I was again able to view the Dockerfile and the output of the docker build.


The Results

Voila! The automated build in Docker Hub triggers builds each time I push a change to GitHub and my images were now ready to be pulled down. Here is the pull…

[jameslabocki@localhost ~]$ docker pull jameslabocki/ceilometer
Pulling repository jameslabocki/ceilometer
c6f1a8880f25: Download complete
511136ea3c5a: Download complete
fd241224e9cf: Download complete
3f2fed40e4b0: Download complete
dba879135119: Download complete
18e0ba2cede2: Download complete
f95adbc11b72: Download complete
0816f2338632: Download complete
9d5ab1dabbc6: Download complete
b9699e746fd1: Download complete
f5089b261315: Download complete
16ddaee924ac: Download complete
aa364c587e7f: Download complete
c33d108a2ed1: Download complete
a131665ee3cb: Download complete
de2e2481c394: Download complete
220014c6f68a: Download complete
291a7a267101: Download complete
e2394bfd4a3f: Download complete
6f3c660adc44: Download complete
7e32f4d4b9b3: Download complete
48eb2ed1f711: Download complete
4bdd06d07972: Download complete
e3299cecd69d: Download complete
c6ccb763dee0: Download complete
2c2de8d95775: Download complete
5d6699390133: Download complete
c6b60ee39aa4: Download complete

[jameslabocki@localhost ~]$ docker pull jameslabocki/keystone
Pulling repository jameslabocki/keystone
2653e44d0420: Download complete
511136ea3c5a: Download complete
fd241224e9cf: Download complete
3f2fed40e4b0: Download complete
ba543dd23e14: Download complete
64f9a0c3a01e: Download complete
89475c70d6b2: Download complete
541ad4ae8739: Download complete
7497b01e4a38: Download complete
68eb2b0e770d: Download complete
a5b4e9a6299c: Download complete
69804e70b9be: Download complete
061825f3a29d: Download complete
f2789b8735b4: Download complete
f25b5709610e: Download complete
133b65e73ad2: Download complete
34152ccfe1a0: Download complete
334d431c2297: Download complete
b0c278d16459: Download complete
8b4eb357e27b: Download complete
85d002913148: Download complete
a0ff34bfbe61: Download complete
e2f611facb89: Download complete
92335c389d86: Download complete
0dd39fdaf260: Download complete

I can also run them and in relatively short order have keystone and ceilometer running side by side on the same host. These containers are relatively isolated, much smaller then virtual machines, and I don’t have to worry about my local machine getting foobar’d while working on keystone or ceilometer. Some great benefits to developers and (eventually) to ops teams.

[jameslabocki@localhost ~]$ sudo docker run -i -t jameslabocki/keystone /bin/bash

On the keystone container I can execute each of the steps in /root/postlaunchconfig.sh one by one to get keystone up and running and create the services and endpoints.

bash-4.2# /usr/bin/keystone-manage db_sync
bash-4.2# /usr/bin/keystone-all &
bash-4.2# /usr/bin/keystone-manage pki_setup --keystone-user root --keystone-group root
2014-07-10 02:12:47.284 413 WARNING keystone.cli [-] keystone-manage pki_setup is not recommended for production use.
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
e is 65537 (0x10001)
Using configuration from /etc/keystone/ssl/certs/openssl.conf
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :ASN.1 12:'Unset'
localityName          :ASN.1 12:'Unset'
organizationName      :ASN.1 12:'Unset'
commonName            :ASN.1 12:'www.example.com'
Certificate is to be certified until Jul  7 02:12:47 2024 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated

bash-4.2# /usr/bin/keystone –os_auth_url –os-token ADMIN –os-endpoint service-create –name=ceilometer –type=metering –description=”Ceilometer Service”
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
|   Property  |              Value               |
| description |        Ceilometer Service        |
|   enabled   |               True               |
|      id     | 27b508729fd84be7994846873b6d7ab2 |
|     name    |            ceilometer            |
|     type    |             metering             |

bash-4.2# /usr/bin/keystone –os_auth_url –os-token ADMIN –os-endpoint service-create –name=keystone –type=identity –description=”OpenStack Identity”
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).

|   Property  |              Value               |
| description |        OpenStack Identity        |
|   enabled   |               True               |
|      id     | 8ccd55d9d6e04b5ca768a033e61db1a1 |
|     name    |             keystone             |
|     type    |             identity             |

Unfortunately, while I’m able to get tokens as a user that I created I’m not able to list users, so I am stuck. I think this might be a bug on the master branch and I’ll be digging into it.

Now, on to Ceilometer …

[jameslabocki@localhost ~]$ sudo docker run -i -t jameslabocki/ceilometer /bin/bash

bash-4.2# sed -ri 's/#identity_uri=<None>/identity_uri=' /etc/ceilometer/ceilometer.conf

bash-4.2# /bin/mongod --dbpath /data/db --fork --logpath /root/mongo.log --noprealloc --smallfiles
note: noprealloc may hurt performance in many applications
about to fork child process, waiting until server is ready for connections.
forked process: 14
all output going to: /root/mongo.log
child process started successfully, parent exiting

bash-4.2# /bin/mongo ceilometer /root/mongosetup.js
MongoDB shell version: 2.4.6
connecting to: ceilometer
"user" : "admin",
"readOnly" : true,
"pwd" : "46553e9fc2cdeada18e714cedbd05c9e",
"_id" : ObjectId("53bdff272da99d751819ff1d")

bash-4.2# /usr/bin/ceilometer-collector &
[1] 162

bash-4.2# /usr/bin/ceilometer-api &
[1] 192

I also had one more issue that I needed to work around since I was running from trunk. I had to patch a file to work around a relative pathing issue with the ceilometer API – https://review.openstack.org/#/c/99599/2/ceilometer/api/app.py

Since the keystone service was having issues I wasn’t able to run ceilometer meter-list or other commands (yet), but I do have the processes running in containers. I’ll continue to troubleshoot the keystone issue to see if I can tie these two services together.


A few thoughts came to mind while running through this exercise.

1. An area that would benefit from tooling is the ability to take an existing docker image and determine how it could be re-based on an existing parent image. For example, after I went through installing python, python-devel, mysql-devel, etc it would be nice if Docker Hub or another tool could tell me that I could save time on builds by using a parent image that already contained those components (no need to `RUN docker yum install` anything). This would save time during build processes. Call it deduplication for Docker!

2. If build times could be kept really short with such tooling it would be REALLY cool to attach an IDE to Docker Hub so that as you typed code into a project on GitHub you could instantly find out the build status. Of course syntax checking could solve some problems in a Dockerfile, but I am thinking along the lines of launching multiple docker builds and testing them with real data (system, UAT, or performance testing scenarios) and returning the result in near real-time. Building a truly integrated development experience into a continuous delivery pipeline could be really powerful (I’m imagining an IDE showing you that the line you just wrote caused a failure when run with 3-4 other docker image builds and launched on AWS, GCE, etc or that the performance was degraded).

3. Extending docker files to have pre-requisites on other docker images would allow users to reference other images required. For example, instead of installing MongoDB on the same docker image it would have been nice to be able to put some statements like this in the Dockerfile.

REFERENCE MONGODB=`docker pull mongo`
sed -ri ‘s/#connection=/connection=${MONGODB:IP_ADDR}/’

Perhaps this should live outside the Dockerfile in systemd, geard, heat, or some configuration language (puppet) and orchestration engine (Kubernetes). Whatever the case, once Docker Hub and other automated Docker build services have this functionality building images that depend on other services will be very powerful.

4. Some of the commands, such as running mongod and then adding a user during the docker hub build kept failing. I’m not sure if I am missing something, but it would seem that being able to run mongod during the build process to add users or seed data into the docker image is something that would be useful. Local docker builds also failed at this. Again, this might be something I am doing incorrectly.

One thing is certain in my mind, the future is bright for containerized IaaS services and sooner or later PaaS will drive the lifecycle of IaaS private cloud services and make the life of Ops much easier!

Here is a link to my docker hub builds for ceilometer and keystone if you want to look further.

Satellite 6

Satellite 6 Slides from the Red Hat Tech Exchange can be downloaded here

OpenStack Foreman Installer

OpenStack Foreman Installer Slides from the Red Hat Tech Exchange can be downloaded here.

Red Hat Cloud Infrastructure Update and Roadmap Slides

Red Hat Cloud Infrastructure Update and Roadmap Slides from the Red Hat Tech Exchange can be downloaded here.


Get every new post delivered to your Inbox.

Join 1,098 other followers