Docker All The Things – OpenStack Keystone and Ceilometer Automated Builds on Docker Hub

Ok, I borrowed part of the title of this post from Nicola Paolucci at Atlassian’s blog post who likely borrowed it from a bunch of others, but it was just too good to pass up.

I decided to test Docker Hub’s automated build feature to see if I could have automated docker images created from a project relevant to Red Hat Cloud Infrastructure (RHCI), Red Hat’s private IaaS cloud solution. RHCI combines datacenter virtualization based on Red Hat Enterprise Virtualization (RHEV), scale out IaaS based on Red Hat Enterprise Linux OpenStack Platform (RHELOSP), and cloud management based on CloudForms. These come from the upstream communities of oVirt, OpenStack, and ManageIQ.

If you are interested in why containers could be so beneficial to an Infrastructure as a Service solution you could read my previous post, “Why containers for OpenStack Services?”. The bottom line is that moving more logic about the lifecycle of the IaaS services into the application layer (Think PaaS for IaaS) could solve many problems and help IaaS become much easier to manage.

Keystone Docker Image

The natural choice for the first service to attempt to containerize was the identity service, Keystone. Keystone has (relative to other openstack projects) few moving parts and is also required by most of the other services since it publishes a catalog of endpoints for the other services APIs.

Here is what I did:

1. I forked the openstack keystone project.

2. I created a new automated build on Docker Hub.

3. I cloned the git repository for my fork of keystone.

jlabocki@localhost# git clone https://github.com/jameslabocki/keystone.git Cloning into 'keystone'... remote: Counting objects: 26085, done. remote: Compressing objects: 100% (9122/9112), done. remote: Total 26085 (delta 16076), reused 26085 (delta 16076) Receiving objects: 100% (1285/1285), 5.61 MiB | 2.14 MiB/s, done. Resolving deltas: 100% (176/176), done. Checking connectivity... done.

4. I created a Dockerfile in the root of the cloned keystone repository. Here is the contents of the Dockerfile.

FROM fedora
MAINTAINER jlabocki@redhat.com

# This Dockerfile installs the components of Keystone in a docker image as a proof of concept

#Timestamps are always useful
RUN date > /root/date

#Install required packages
RUN yum install python-pbr git python-devel python-setuptools python-pip gcc gcc-devel libxml2-python libxslt-python python-lxml sqlite python-repoze-lru -y
#RUN yum install python-sqlite2 python-lxml python-greenlet-devel python-ldap sqlite-devel openldap-devel -y

#Clone Keystone and setup
WORKDIR /opt
RUN git clone http://github.com/openstack/keystone.git
WORKDIR /opt/keystone
RUN python setup.py install

#Configure Keystone
RUN mkdir -p /etc/keystone
RUN cp etc/keystone.conf.sample /etc/keystone/keystone.conf
RUN cp etc/keystone-paste.ini /etc/keystone/
RUN sed -ri 's/#driver=keystone.identity.backends.sql.Identity/driver=keystone.identity.backends.sql.Identity/' /etc/keystone/keystone.conf 
RUN sed -ri 's/#connection=<None>/connection=sqlite:\/\/\/keystone.db/' /etc/keystone/keystone.conf
RUN sed -ri 's/#idle_timeout=3600/idle_timeout=200/' /etc/keystone/keystone.conf
RUN sed -ri 's/#admin_token=ADMIN/admin_token=ADMIN/' /etc/keystone/keystone.conf

# The following sections build a script that will be executed on launch via ENTRYPOINT

## Start Keystone
RUN echo "#!/bin/bash" > /root/postlaunchconfig.sh
RUN echo "/usr/bin/keystone-manage db_sync" >> /root/postlaunchconfig.sh
RUN echo "/usr/bin/keystone-all &" >> /root/postlaunchconfig.sh

## Create Services
#I'm not sure if exporting works, so I just specify these environment variables on each command, but it might be cleaner to test this
#RUN export OS_SERVICE_ENDPOINT=http://localhost:35357/v2.0
#RUN export OS_SERVICE_TOKEN=ADMIN
#RUN export OS_AUTH_URL=http://127.0.0.1:35357/v2.0/
RUN echo '/usr/bin/keystone --os_auth_url http://127.0.0.1:35357/v2.0/ --os-token ADMIN --os-endpoint http://127.0.0.1:35357/v2.0/ service-create --name=ceilometer --type=metering --description="Ceilometer Service"' >> /root/postlaunchconfig.sh
RUN echo '/usr/bin/keystone --os_auth_url http://127.0.0.1:35357/v2.0/ --os-token ADMIN --os-endpoint http://127.0.0.1:35357/v2.0/ service-create --name=keystone --type=identity --description="OpenStack Identity"' >> /root/postlaunchconfig.sh
RUN chmod 755 /root/postlaunchconfig.sh

#This you will need to substitute your values and run later - the values are:
# CEILOMETER_SERVICE = the id of the service created by the keystone service-create command
# KEYSTONE_SERVICE = the id of the service created by the keystone service-create command
# CEILOMETER_SERVICE_HOST = the host where the Ceilometer API is running
# KEYSTONE_SERVICE_HOST = the host where the Keystone API is running
RUN echo 'keystone --os_auth_url http://127.0.0.1:35357/v2.0/ --os-token ADMIN --os-endpoint http://127.0.0.1:35357/v2.0/ endpoint-create --region RegionOne --service_id $KEYSTONE_SERVER --publicurl "http://KEYSTONE_SERVICE_HOST:5000/v2.0" --internalurl "http://KEYSTONE_SERVICE_HOST:5000/v2.0" --adminurl "http://KEYSTONE_SERVICE_HOST:35357/v2.0"' > /root/postlaunchconfig.sh
RUN echo 'keystone --os_auth_url http://127.0.0.1:35357/v2.0/ --os-token ADMIN --os-endpoint http://127.0.0.1:35357/v2.0/ endpoint-create --region RegionOne --service_id $CEILOMETER_SERVICE --publicurl "http://CEILOMETER_SERVICE_HOST:8777/"  --adminurl "http://CEILOMETER_SERVICE_HOST:8777/" --internalurl "http://CEILOMETER_SERVICE_HOST:8777/"' > /root/postlaunchconfig.sh

5. I committed and pushed the change.

[root@localhost keystone]# git commit -m "testing" . [master fe12eff] testing 1 file changed, 6 insertions(+), 7 deletions(-) [root@localhost keystone]# git push Username for 'https://github.com':@.com Password for 'https://@github.com': Counting objects: 14, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 335 bytes | 0 bytes/s, done. Total 3 (delta 2), reused 0 (delta 0) To https://github.com/jameslabocki/keystone.git a3de5a2..fe12eff master -> master

6. The automated build finished successfully.

7. I could check the details of the build, including the Dockerfile used and the output of the build.

Assuming that the image was good and I could run it and setup Keystone rather quickly I decided to focus on another service and then attempt launching the two together (see the results section if you want to spoil the surprise).

Ceilometer Docker Image

I selected the OpenStack telemetry project, commonly known as Ceilometer, for my next test of an automated build of a docker image to take place on commit to my forked repository. Why Ceilometer? After looking at the OpenStack architecture diagram I thought it might be one of the easier services to run in a container (basically, I used a dart board), and since it only requires keystone I thought I might be able to make it happen next. Here are the components of OpenStack Ceilometer (Telemetry) at a glance taken from the OpenStack docs.

The telemetry system consists of the following basic components:

A compute agent (ceilometer-agent-compute). Runs on each compute node and polls for resource utilization statistics. There may be other types of agents in the future, but for now we will focus on creating the compute agent.
A central agent (ceilometer-agent-central). Runs on a central management server to poll for resource utilization statistics for resources not tied to instances or compute nodes.
A collector (ceilometer-collector). Runs on one or more central management servers to monitor the message queues (for notifications and for metering data coming from the agent). Notification messages are processed and turned into metering messages and sent back out onto the message bus using the appropriate topic. Telemetry messages are written to the data store without modification.
An alarm notifier (ceilometer-alarm-notifier). Runs on one or more central management servers to allow settting alarms based on threshold evaluation for a collection of samples.
A data store. A database capable of handling concurrent writes (from one or more collector instances) and reads (from the API server).
An API server (ceilometer-api). Runs on one or more central management servers to provide access to the data from the data store. These services communicate using the standard OpenStack messaging bus. Only the collector and API server have access to the data store.

Here it is in a diagram.

I decided to start with the database and ceilometer collector and then add the API. I went the route of placing all of these services in a single image. I’m aware there is a lot of debate as to whether Docker images should only run a single process or if multiple processes could be beneficial. My intention wasn’t to optimize the image for production, rather it was to test how easy or difficult it was to take a forked GitHub project and get it into an image build in an automated fashion that I could run on my Fedora 20 workstation. Also, I did not plan to add the evaluator, notifier, or any agents to this image. Since most of the agents require other components of OpenStack.

Here is what I did:

1. I forked the openstack ceilometer project.

2. I created another new automated build on Docker Hub.

3. I cloned the git repository for my fork of ceilometer.

jlabocki@localhost# git clone https://github.com/jameslabocki/ceilometer.git
Cloning into ‘ceilometer’…
remote: Counting objects: 26085, done.
remote: Compressing objects: 100% (7112/7112), done.
remote: Total 26085 (delta 16076), reused 26085 (delta 16076)
Receiving objects: 100% (26085/26085), 8.81 MiB | 3.14 MiB/s, done.
Resolving deltas: 100% (16076/16076), done.
Checking connectivity… done.

4. I created a Dockerfile in the root of the project following the manual installation of OpenStack Ceilometer. Here is the contents of the Dockerfile. Note I wasn’t able to run the mongod command during the build successfully. More on that later, I just created a post launch script that could be executed after the docker image is launched as a work around.

FROM fedora
MAINTAINER jlabocki@redhat.com

# This Dockerfile installs some of the components of Ceilometer in a Docker Image as a proof of concept
# 

#Timestamps are always useful
RUN date > /root/date

#Install required packages
RUN yum install mysql-devel openssl-devel wget unzip git mongodb mongodb-server python-devel mysql-server libmysqlclient-devel libffi-devel libxml2-devel libxslt-devel python-setuptools python-pip libffi libffi-devel gcc gcc-devel python-pip python-pbr mongodb python-pymongo rabbitmq-server -y

#RUN pip install tox
#Can't run the line above because https://bugs.launchpad.net/openstack-ci/+bug/1274135, need to specify version 1.6.1
RUN pip install tox==1.6.1

#MongoDB Setup
RUN mkdir -p /data/db
RUN echo 'db.addUser("admin", "insecure", true);' > /root/mongosetup.js

#RabbitMQ Setup
RUN /usr/sbin/rabbitmq-server -detached

#Clone Ceilometer
RUN git clone https://github.com/jlabocki/ceilometer.git /opt/stack/

#Ceilometer Collector Configuration
WORKDIR /opt/stack
RUN python setup.py install
RUN mkdir -p /etc/ceilometer
RUN tox -egenconfig
RUN cp /opt/stack/etc/ceilometer/*.json /etc/ceilometer
RUN cp /opt/stack/etc/ceilometer/*.yaml /etc/ceilometer
RUN cp /opt/stack/etc/ceilometer/ceilometer.conf.sample /etc/ceilometer/ceilometer.conf

#Ceilometer Collector Configuration changes
RUN sed -ri 's/#metering_secret=change this or be hacked/metering_secret=redhat/' /etc/ceilometer/ceilometer.conf
RUN sed -ri 's/#connection=<None>/connection = mongodb:\/\/admin:insecure@localhost:27017\/ceilometer/' /etc/ceilometer/ceilometer.conf

#Ceilometer API Configuration changes
RUN cp etc/ceilometer/api_paste.ini /etc/ceilometer/api_paste.ini

##Ceilometer Post Launch Configuration
RUN echo "#!/bin/bash" > /root/postlaunch.sh

#Add Authenticate against keystone to the post launch script
# KEYSTONE_HOST = the keystone host
RUN echo "sed -ri 's/#identity_uri=<None>/identity_uri=KEYSTONE_HOST/' /etc/ceilometer/ceilometer.conf" >> /root/postlaunch.sh

#Add starting services to the postlaunch script
RUN echo "/bin/mongod --dbpath /data/db --fork --logpath /root/mongo.log --noprealloc --smallfiles" >> /root/postlaunch.sh
RUN echo "/bin/mongo mydb /root/mongosetup.js" >> /root/postlaunch.sh
RUN echo "/usr/bin/ceilometer-collector" >> /root/postlaunch.sh
RUN echo "/usr/bin/ceilometer-api" >> /root/postlaunch.sh
RUN chmod 755 /root/postlaunch.sh

5. I added and committed the change and pushed.

[root@localhost ceilometer]# git commit -m "testing" . [master fe12eff] testing 1 file changed, 6 insertions(+), 7 deletions(-) [root@localhost keystone]# git push Username for 'https://github.com':@.com Password for 'https://@github.com': Counting objects: 14, done. Delta compression using up to 2 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 335 bytes | 0 bytes/s, done. Total 3 (delta 2), reused 0 (delta 0) To https://github.com/jameslabocki/ceilometer.git a3de5a2..fe12eff master -> master

6. The automated build finished successfully.

7. I was again able to view the Dockerfile and the output of the docker build.

The Results

Voila! The automated build in Docker Hub triggers builds each time I push a change to GitHub and my images were now ready to be pulled down. Here is the pull…

[jameslabocki@localhost ~]$ docker pull jameslabocki/ceilometer Pulling repository jameslabocki/ceilometer c6f1a8880f25: Download complete 511136ea3c5a: Download complete fd241224e9cf: Download complete 3f2fed40e4b0: Download complete dba879135119: Download complete 18e0ba2cede2: Download complete f95adbc11b72: Download complete 0816f2338632: Download complete 9d5ab1dabbc6: Download complete b9699e746fd1: Download complete f5089b261315: Download complete 16ddaee924ac: Download complete aa364c587e7f: Download complete c33d108a2ed1: Download complete a131665ee3cb: Download complete de2e2481c394: Download complete 220014c6f68a: Download complete 291a7a267101: Download complete e2394bfd4a3f: Download complete 6f3c660adc44: Download complete 7e32f4d4b9b3: Download complete 48eb2ed1f711: Download complete 4bdd06d07972: Download complete e3299cecd69d: Download complete c6ccb763dee0: Download complete 2c2de8d95775: Download complete 5d6699390133: Download complete c6b60ee39aa4: Download complete

[jameslabocki@localhost ~]$ docker pull jameslabocki/keystone Pulling repository jameslabocki/keystone 2653e44d0420: Download complete 511136ea3c5a: Download complete fd241224e9cf: Download complete 3f2fed40e4b0: Download complete ba543dd23e14: Download complete 64f9a0c3a01e: Download complete 89475c70d6b2: Download complete 541ad4ae8739: Download complete 7497b01e4a38: Download complete 68eb2b0e770d: Download complete a5b4e9a6299c: Download complete 69804e70b9be: Download complete 061825f3a29d: Download complete f2789b8735b4: Download complete f25b5709610e: Download complete 133b65e73ad2: Download complete 34152ccfe1a0: Download complete 334d431c2297: Download complete b0c278d16459: Download complete 8b4eb357e27b: Download complete 85d002913148: Download complete a0ff34bfbe61: Download complete e2f611facb89: Download complete 92335c389d86: Download complete 0dd39fdaf260: Download complete

I can also run them and in relatively short order have keystone and ceilometer running side by side on the same host. These containers are relatively isolated, much smaller then virtual machines, and I don’t have to worry about my local machine getting foobar’d while working on keystone or ceilometer. Some great benefits to developers and (eventually) to ops teams.

[jameslabocki@localhost ~]$ sudo docker run -i -t jameslabocki/keystone /bin/bash

On the keystone container I can execute each of the steps in /root/postlaunchconfig.sh one by one to get keystone up and running and create the services and endpoints.

bash-4.2# /usr/bin/keystone-manage db_sync bash-4.2# /usr/bin/keystone-all & bash-4.2# /usr/bin/keystone-manage pki_setup --keystone-user root --keystone-group root 2014-07-10 02:12:47.284 413 WARNING keystone.cli [-] keystone-manage pki_setup is not recommended for production use. Generating RSA private key, 2048 bit long modulus ..........+++ ...............................................................+++ e is 65537 (0x10001) Generating RSA private key, 2048 bit long modulus ............................................................................................+++ ......................................................................+++ e is 65537 (0x10001) Using configuration from /etc/keystone/ssl/certs/openssl.conf Check that the request matches the signature Signature ok The Subject's Distinguished Name is as follows countryName :PRINTABLE:'US' stateOrProvinceName :ASN.1 12:'Unset' localityName :ASN.1 12:'Unset' organizationName :ASN.1 12:'Unset' commonName :ASN.1 12:'www.example.com' Certificate is to be certified until Jul 7 02:12:47 2024 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated

bash-4.2# /usr/bin/keystone –os_auth_url http://127.0.0.1:35357/v2.0/ –os-token ADMIN –os-endpoint http://127.0.0.1:35357/v2.0/ service-create –name=keystone –type=identity –description=”OpenStack Identity”
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).

Unfortunately, while I’m able to get tokens as a user that I created I’m not able to list users, so I am stuck. I think this might be a bug on the master branch and I’ll be digging into it.

Now, on to Ceilometer …

[jameslabocki@localhost ~]$ sudo docker run -i -t jameslabocki/ceilometer /bin/bash
bash-4.2# sed -ri 's/#identity_uri=<None>/identity_uri=https://172.17.0.3:35357/' /etc/ceilometer/ceilometer.conf
bash-4.2# /bin/mongod --dbpath /data/db --fork --logpath /root/mongo.log --noprealloc --smallfiles note: noprealloc may hurt performance in many applications about to fork child process, waiting until server is ready for connections. forked process: 14 all output going to: /root/mongo.log child process started successfully, parent exiting
bash-4.2# /bin/mongo ceilometer /root/mongosetup.js MongoDB shell version: 2.4.6 connecting to: ceilometer { "user" : "admin", "readOnly" : true, "pwd" : "46553e9fc2cdeada18e714cedbd05c9e", "_id" : ObjectId("53bdff272da99d751819ff1d") }
bash-4.2# /usr/bin/ceilometer-collector & [1] 162
bash-4.2# /usr/bin/ceilometer-api & [1] 192
I also had one more issue that I needed to work around since I was running from trunk. I had to patch a file to work around a relative pathing issue with the ceilometer API – https://review.openstack.org/#/c/99599/2/ceilometer/api/app.py

Since the keystone service was having issues I wasn’t able to run ceilometer meter-list or other commands (yet), but I do have the processes running in containers. I’ll continue to troubleshoot the keystone issue to see if I can tie these two services together.

Observations

A few thoughts came to mind while running through this exercise.

1. An area that would benefit from tooling is the ability to take an existing docker image and determine how it could be re-based on an existing parent image. For example, after I went through installing python, python-devel, mysql-devel, etc it would be nice if Docker Hub or another tool could tell me that I could save time on builds by using a parent image that already contained those components (no need to `RUN docker yum install` anything). This would save time during build processes. Call it deduplication for Docker!

2. If build times could be kept really short with such tooling it would be REALLY cool to attach an IDE to Docker Hub so that as you typed code into a project on GitHub you could instantly find out the build status. Of course syntax checking could solve some problems in a Dockerfile, but I am thinking along the lines of launching multiple docker builds and testing them with real data (system, UAT, or performance testing scenarios) and returning the result in near real-time. Building a truly integrated development experience into a continuous delivery pipeline could be really powerful (I’m imagining an IDE showing you that the line you just wrote caused a failure when run with 3-4 other docker image builds and launched on AWS, GCE, etc or that the performance was degraded).

3. Extending docker files to have pre-requisites on other docker images would allow users to reference other images required. For example, instead of installing MongoDB on the same docker image it would have been nice to be able to put some statements like this in the Dockerfile.

REFERENCE MONGODB=`docker pull mongo`
sed -ri ‘s/#connection=/connection=${MONGODB:IP_ADDR}/’

Perhaps this should live outside the Dockerfile in systemd, geard, heat, or some configuration language (puppet) and orchestration engine (Kubernetes). Whatever the case, once Docker Hub and other automated Docker build services have this functionality building images that depend on other services will be very powerful.

4. Some of the commands, such as running mongod and then adding a user during the docker hub build kept failing. I’m not sure if I am missing something, but it would seem that being able to run mongod during the build process to add users or seed data into the docker image is something that would be useful. Local docker builds also failed at this. Again, this might be something I am doing incorrectly.

One thing is certain in my mind, the future is bright for containerized IaaS services and sooner or later PaaS will drive the lifecycle of IaaS private cloud services and make the life of Ops much easier!

Here is a link to my docker hub builds for ceilometer and keystone if you want to look further.

all things open

having fun with open source