Showing posts with label amqp. Show all posts
Showing posts with label amqp. Show all posts

Monday, September 1, 2014

Docker: A QPID Message Broker Container

OK I lied. I realized I can't just move on to working with Pulp in Kubernetes without building the other sub-service Pulp needs.

This one is merely going to be an exposition of the QPID container and it's actually simpler than the MongoDB container, so this will be a short one. A QPID service is even simpler than a MongoDB because (so long as you don't care about store-and-forward messages) you don't need persistent storage.

Like the MongoDB container, I need to define the package set that will be installed on top of the base image. I also need to declare a TCP port for the QPID service. Finally I need to define the primary process that will be started when the container starts.  This will be an invocation of the QPID service daemon.

QPID Dockerfile


Here's the Dockerfile for QPID on Fedora 20.


Let's walk through the Dockerfile directives quickly.

Line 1: FROM - Just as in the MongoDB image,  I'm using the stock Fedora 20 image as the base

Line 2: MAINTAINER - Indicate who to contact with problems (AND THANKS!)

Yeah, that's me.

Line 7: RUN - Install the QPID packages

I think there are several QPID servers.  I'm using the one written in C++, hence the package names: qpid-cpp-server and qpidd-cpp-server-store.  

Line 10: ADD - Create a location for the daemon to run.  If you specify a file to add but there is no matching file in the build context directory, then Docker will create the target in the container as an empty directory.

I'm creating /.qpidd for the daemon to run in.

Line 12: WORKDIR - Set the location where the initial process will run.  Here is where I tell Docker to run the daemon in the directory I created with the previous ADD directive.

Line 14: EXPOSE - QPID uses port 5672/TCP.  This line opens the firewall for that port and causes Docker to bind it to a host port.

Line 16: ENTRYPOINT - This indicates the binary or script that will be called when the container runs.

The ENTRYPOINT and CMD directives are used to craft the invocation of the primary process of the container.

Explaining ENTRYPOINT and CMD


I got some help for this from a Stackoverflow article: What is the difference between CMD and ENTRYPOINT

When I docker container is run, a single process is started inside the container.  This process may spawn others, but it remains as the anchor process for all of the others.

The invocation of the container primary process is created by combining the values of the ENTRYPOINT and CMD directives.  The ENTRYPOINT, if it is set, becomes the path of the binary to be executed.  The value of the CMD directive is used as the arguments to the primary process.

There are two twists on this.

If no ENTRYPOINT is provided, then the CMD directive is run using /bin/sh -c.
Also if the docker run command has any positional arguments following the regular docker arguments, these will replace the CMD value.

By setting the ENTRYPOINT to the QPID command, then the arguments to the daemon can be passed directly on the docker run line.

If an image has an ENTRYPOINT directive then it can be overridden with the --entrypoint option to docker run.

Building the Image


docker build -t markllama/qpid images/qpid
Sending build context to Docker daemon  2.56 kB
Sending build context to Docker daemon 
Step 0 : FROM fedora:20
 ---> 88b42ffd1f7c
Step 1 : MAINTAINER Mark Lamourine 
 ---> Using cache
 ---> 95516239225e
Step 2 : RUN yum install -y qpid-cpp-server qpid-cpp-server-store python-qpid-qmf python-qpid &&      yum clean all
 ---> Running in 7fc6b6ed2128
Resolving Dependencies
--> Running transaction check
---> Package python-qpid.noarch 0:0.26-2.fc20 will be installed
--> Processing Dependency: python-qpid-common = 0.26-2.fc20 for package: python-qpid-0.26-2.fc20.noarch
---> Package python-qpid-qmf.x86_64 0:0.26-2.fc20 will be installed
--> Processing Dependency: qpid-qmf(x86-64) = 0.26-2.fc20 for package: python-qpid-qmf-0.26-2.fc20.x86_64
--> Processing Dependency: libqmf2.so.1()(64bit) for package: python-qpid-qmf-0.26-2.fc20.x86_64
---> Package qpid-cpp-server.x86_64 0:0.26-11.fc20 will be installed
--> Processing Dependency: qpid(client)(x86-64) = 0.26 for package: qpid-cpp-server-0.26-11.fc20.x86_64
--> Processing Dependency: qpid-proton-c(x86-64) >= 0.5 for package: qpid-cpp-server-0.26-11.fc20.x86_64
...                                     
  python-qpid-common.noarch 0:0.26-2.fc20                                       
  qpid-cpp-client.x86_64 0:0.26-11.fc20                                         
  qpid-proton-c.x86_64 0:0.7-3.fc20                                             
  qpid-qmf.x86_64 0:0.26-2.fc20                                                 

Complete!
Cleaning repos: fedora updates
Cleaning up everything
 ---> d7e61654fb92
Removing intermediate container 7fc6b6ed2128
Step 3 : ADD . /.qpidd
 ---> 10c44a5719a5
Removing intermediate container a8a37c5986a5
Step 4 : WORKDIR /.qpidd
 ---> Running in 2833da1629d9
 ---> 1963a2551db8
Removing intermediate container 2833da1629d9
Step 5 : EXPOSE 5672
 ---> Running in d0d92a1e58ad
 ---> 425ba5994308
Removing intermediate container d0d92a1e58ad
Step 6 : ENTRYPOINT ["/usr/bin/qpidd", "-t", "--auth=no"]
 ---> Running in e678dc1a4b66
 ---> ae30e626e215
Removing intermediate container e678dc1a4b66
Successfully built ae30e626e215

Verifying the Image


With respect to docker, verifying the image is the same as it was for the MongoDB image.

docker run -d --name qpid1 --publish-all markllama/mongodb
1b513bee6d8d5d4328059a059f9520c469ff405228b88370b91bb85ef659b708


Process information


docker ps
CONTAINER ID        IMAGE                      COMMAND                CREATED             STATUS              PORTS                      NAMES
1b513bee6d8d        markllama/qpid:latest      /usr/sbin/qpidd -t -   7 seconds ago       Up 5 seconds        0.0.0.0:49157->5672/tcp    qpid1  

Docker logs


docker logs qpid1
2014-09-01 23:32:34 [Model] trace Mgmt create memory. id:amqp-broker
2014-09-01 23:32:34 [Broker] info Management enabled
2014-09-01 23:32:34 [Management] info ManagementAgent generated broker ID: dc7d2
473-58e4-4eea-a21b-46105345054e
...
2014-09-01 23:32:34 [Management] debug ManagementAgent added class org.apache.qp
id.broker:queueThresholdExceeded
2014-09-01 23:32:34 [Model] trace Mgmt create system. id:cfaf5a0f-1291-41e5-b0c0
-e5eb07c77c1e
2014-09-01 23:32:34 [Model] trace Mgmt create broker. id:amqp-broker
2014-09-01 23:32:34 [Model] trace Mgmt create vhost. id:org.apache.qpid.broker:b
roker:amqp-broker,/
2014-09-01 23:32:34 [Security] notice SSL plugin not enabled, you must set --ssl
-cert-db to enable it.
2014-09-01 23:32:34 [Broker] info Loaded protocol AMQP 1.0
2014-09-01 23:32:35 [Store] notice Journal "TplStore": Created
2014-09-01 23:32:35 [Store] notice Store module initialized; store-dir=//.qpidd
2014-09-01 23:32:35 [Store] info > Default files per journal: 8
2014-09-01 23:32:35 [Store] info > Default journal file size: 24 (wpgs)
2014-09-01 23:32:35 [Store] info > Default write cache page size: 32 (KiB)
2014-09-01 23:32:35 [Store] info > Default number of write cache pages: 32
2014-09-01 23:32:35 [Store] info > TPL files per journal: 8
2014-09-01 23:32:35 [Store] info > TPL journal file size: 24 (wpgs)
2014-09-01 23:32:35 [Store] info > TPL write cache page size: 4 (KiB)
2014-09-01 23:32:35 [Store] info > TPL number of write cache pages: 64
2014-09-01 23:32:35 [Model] trace Mgmt create exchange. id:
...
2014-09-01 23:32:36 [Model] trace Mgmt create exchange. id:qmf.default.direct
2014-09-01 23:32:36 [Broker] notice SASL disabled: No Authentication Performed
2014-09-01 23:32:36 [Security] info Policy file not specified. ACL Disabled, no 
ACL checking being done!
2014-09-01 23:32:36 [Security] trace Initialising SSL plugin
2014-09-01 23:32:36 [Network] info Listening to: 0.0.0.0:5672
2014-09-01 23:32:36 [Network] info Listening to: [::]:5672
2014-09-01 23:32:36 [Network] notice Listening on TCP/TCP6 port 5672
2014-09-01 23:32:36 [Store] info Enabling management instrumentation for the sto
re.
...
2014-09-01 23:32:36 [Model] trace Mgmt create store. id:org.apache.qpid.broker:b
roker:amqp-broker
2014-09-01 23:32:36 [Management] debug Management object (V1) added: org.apache.
qpid.legacystore:store:org.apache.qpid.broker:broker:amqp-broker
2014-09-01 23:32:36 [Broker] notice Broker running

The QPID logs will continue accumulating. With the default debug level it reports a lot of connection information.

Connectivity


To test connectivity to the QPID services I use the qpid-config command from the qpid-utils package on Fedora. Install that package to get the command.


qpid-config queues -b guest@127.0.0.1:49157
Queue Name                                Attributes
=================================================================
7783123e-9589-4814-8b7b-b976a576c853:0.0  auto-del excl 


This command lists the queues present on the broker.  It connects using the guest account and specifies the localhost IPv4 address and the port indicated by the output of the docker ps or docker ports commands.

This is a very simple connectivity test.  The single queue is the default for an unused AMQP server.  Once the Pulp components connect they will create additional queues.

Running A Shell in an image with an ENTRYPOINT


Using an ENTRYPOINT directive has a couple of effects that you want to be aware of.

On the plus side you can add arguments to the entrypoint binary just by adding them after the image name on the invocation.

One gotcha is that you can't just put /bin/sh after the image to get a shell as you otherwise would. It is very common and convenient to examine an image by running it with a shell, overriding the CMD. Docker provides the --entrypoint option to allow overriding when necessary.

docker run -it --entrypoint /bin/sh markllama/qpid 
sh-4.2# pwd
/.qpidd
sh-4.2# ls
Dockerfile
sh-4.2# exit
exit

Now I have images for both of the secondary services that Pulp needs.

Time to start playing with Kubernetes a bit.

References:


Monday, September 23, 2013

OpenShift Support Services: Messaging Part 2 (MCollective)

About a year ago I did a series of posts on verifying the plugin operations for OpenShift Origin support services. I showed how to check the datastore (mongodb) and DNS updates and how to set up an ActiveMQ message broker , but I when I got to actually sending and receiving messages I got stuck.

The Datastore and DNS services use a single point-to-point connection between the broker and the update server. The messaging services use an intermediate message broker (ActiveMQ, not to be confused with the OpenShift broker). This means that I need to configure and check not just one points, but three:

  • Mcollective client to (message) broker (on OpenShift broker)
  • Mcollective server to (message) broker (on OpenShift node)
  • End to End

I'm using the ActiveMQ message broker to carry MCollective RPC messages. The message broker is interchangeable. MCollective can be carried over any one of several messaging protocols. I'm using the Stomp protocol for now, though MCollective is deprecating Stomp in favor of a native ActiveMQ (AMQP?) messaging protocol.

OpenShift Messaging Components
OpenShift Messaging Components


In a previous post I set up an ActiveMQ message broker to be used for communication between the OpenShift broker and nodes. In this one I'm going to connect the OpenShift components to the messaging service, verify both connections and then verify that I can send messages end-to-end.

Hold on for the ride, it's a long one (even for me)

Mea Culpa: I'm referring to what MCollective does as "messaging" but that's not strictly true. ActiveMQ, RabbitMQ, QPID are message broker services. MCollective uses those, but actually, MCollective is an RPC (Remote Procedure Call) system. Proper messaging is capable of much more than MCollective requires, but to avoid a lot of verbal knitting I'm being lazy and calling MCollective "messaging".

The Plan

Since this is a longer process than any of my previous posts, I'm going to give a little road-map up front so you know you're not getting lost on the way. Here are the landmarks between here and a working OpenShift messaging system:
  1. Ingredients: Gather configuration information for messaging setup.
  2. Mcollective Client -
    Establish communications between the Mcollective client  and the ActiveMQ server
    (OpenShift broker host to message broker host)
  3. MCollective Server -
    Establish communications beween the MCollective server and the ActiveMQ server
    (OpenShift node host to message broker host)
  4. MCollective End-To-End -
    Verify MCollective communication from client to server
  5. OpenShift Messaging and Agent -
    Install OpenShift messaging interface definition and agent packages on both OpenShift broker and node

Ingredients

VariableValue
ActiveMQ Servermsg1.infra.example.com
Message Bus
topic usernamemcollective
topic passwordmarionette
admin passwordmsgadminsecret
Message End
passwordmcsecret
  • A running ActiveMQ service
  • A host to be the MCollective client (and after that an OpenShift broker)
  • A host to run the MCollective service (and after that an OpenShfit node)
On the Mcollective client host, install these RPMs
  • mcollective-client
  • rubygem-openshift-origin-msg-broker-mcollective
On the MCollective server (OpenShift node) host, install these RPMs
  • mcollective
  • openshift-origin-msg-node-mcollective

Secrets and more Secrets


As with all secure network services, messaging requires authentication. Messaging has a twist though. You need two sets of authentication information, because, underneath, you're actually using two services. When you send a message to an end-point, the end point has to be assured that you are someone who is allowed to send messages. Like with a letter, having some secret code or signature so that you can be sure the letter isn't forged.

Now imagine a special private mail system. Before the mail carrier will accept a letter, you have to give them the secret handshake so that they know you're allowed to send letters. On the delivery end, the mail carrier requires not just a signature but a password before handing over the letter.

That's how authentication works for messaging systems.

When I set up the ActiveMQ service I didn't create a separate user for writing to the queue (sending a letter) and for reading (receiving) but I probably should have. As it is, getting a message from the OpenShift broker to an OpenShift node through MCollective and ActiveMQ requires two passwords and one username.

  • mcollective endpoint secret
  • ActiveMQ username
  • ActiveMQ password

The ActiveMQ values will have to match those I set on the ActiveMQ message broker in the previous post. The MCollective end point secret is only placed in the MCollective configuration files. You'll see those soon.

MCollective Client (OpenShift Broker)


The OpenShift broker service sends messages to the OpenShift nodes. All of the messages (currently) originate at the broker. This means that the nodes need to have a process running which connects to the message broker and registers to receive MColletive messages.

Client configuration: client.cfg


The MCollective client is (predictably) configured using the /etc/mcollective/client.cfg file. For the purpose of connecting to the message broker, only the connector plugin values are interesting, and for end-to-end communications I need the securityprovider plugin as well. The values related to logging are useful debugging too.


# Basic stuff
topicprefix     = /topic/
main_collective = mcollective
collectives     = mcollective
libdir          = /usr/libexec/mcollective
loglevel        = log   # just for testing, normally 'info'

# Plugins
securityprovider = psk
plugin.psk       = mcsecret

# Middleware
connector         = stomp
plugin.stomp.host = msg1.infra.example.com
plugin.stomp.port = 61613
plugin.stomp.user = mcollective
plugin.stomp.user = marionette

NOTE:if you're running on RHEL6 or CentOS 6 instead of Fedora you're going to be using the SCL version of Ruby and hence MCollective. The file is then at the SCL location:

/opt/rh/ruby193/root/etc/mcollective/client.cfg

Now I can test connections to the ActiveMQ message broker, though without any servers connected, it won't be very exciting (I hope).

Testing client connections


MCollective provides a command line tool for sending messages: mco . mco is capable of several other 'meta' operations as well. The one I'm interested in first is 'mco ping'. With mco ping I can verify the connection to the ActiveMQ service (via the Stomp protocol) .

The default configuration file is owned by root and is not readable by ordinary users. This is because it contains plain-text passwords (There are ways to avoid this, but that's for another time). This means I have to either run mco commands as root, or create a config file that is readable. I'm going to use sudo to run my commands as root.

The mco ping command connects to the messaging service and asks all available MCollective servers to respond. Since I haven't connected any yet, I won't get any answers, but I can at least see that I'm able to connect to the message broker, send queries. If all goes well I should get a nice message saying "no one answered".


sudo mco ping


---- ping statistics ----
No responses received

If that's what you got, feel free to skip down to the MCollective Server section.

Debugging client-side configuration errors


There are a couple of obvious possible errors:
  1. Incorrect broker host
  2. broker service not answering
  3. Incorrect messaging username/password
The first two will appear the same to the MCollective client. Check the simple stuff first. If I'm sure that the host is correct then I'll have to diagnose the problem on the other (and write another blog post). Here's how that looks:

sudo mco ping
connect to localhost failed: Connection refused - connect(2) will retry(#0) in 5
connect to localhost failed: Connection refused - connect(2) will retry(#1) in 5
connect to localhost failed: Connection refused - connect(2) will retry(#2) in 5
^C
The ping application failed to run, use -v for full error details: Could not connect to Stomp Server: 

Note the message Could not connect to the Stomp Server.

If you get this message, check these on the OpenShift broker host:

  1. The plugin.stomp.host value is correct
  2. The plugin.stomp.port value is correct
  3. The host value resolves to an IP address in DNS
  4. The ActiveMQ host can be reached from the OpenShift Broker host (by ping or SSH)
  5. You can connect to Stomp port on the ActiveMQ broker host
    telnet msg1.example.com 61613 (yes, telnet is a useful tool) 

If all of these are correct, then look on the ActiveMQ message broker host:

  1. The ActiveMQ service is running
  2. The Stomp transport TCP ports match the plugin.stomp.port value
  3. The host firewall is allowing inbound connections on the Stomp port

The third possibility indicates an information or configuration mismatch between the MCollective client configuration and the ActiveMQ server.  That will look like this:

sudo mco ping
transmit to msg1.infra.example.com failed: Broken pipe
connection.receive returning EOF as nil - resetting connection.
connect to localhost failed: Broken pipe will retry(#0) in 5

The ping application failed to run, use -v for full error details: Stomp::Error::NoCurrentConnection

You can get even more gory details by changing the client.cfg to set the log level to debug and send the log output to the console:

...
loglevel = debug # instead of 'log' or 'info'
logger_type = console # instead of 'file', or 'syslog' or unset (no logging)
...

I'll spare you what that looks like here.

MCollective Server (OpenShift Node)


The mcollective server is a process that connects to a message broker, subscribes to (registers to receive messages from) one or more topics and then listens for incoming messages. When it accepts a message, the mcollective server passes it to a plugin module for execution and then returns any response.  All OpenShift node hosts run an MCollective server which connects to one or more of the ActiveMQ message brokers.

Configure the MCollective service daemon: server.cfg 


I bet you have already guessed that the MCollective server configuration file is /etc/mcollective/server.cfg

# Basic stuff
topicprefix     = /topic/
main_collective = mcollective
collectives     = mcollective
libdir          = /usr/libexec/mcollective
logfile         = /var/log/mcollective.log
loglevel        = debug # just for setup, normally 'info'
daemonize       = 1
classesfile     = /var/lib/puppet/state/classes.txt

# Plugins
securityprovider = psk
plugin.psk       = mcsecret

# Registration
registerinterval = 300
registration     = Meta

# Middleware
connector         = stomp
plugin.stomp.host = msg1.infra.example.com
plugin.stomp.port = 61613
plugin.stomp.user = mcollective
plugin.stomp.password = marionette


# NRPE
plugin.nrpe.conf_dir  = /etc/nrpe.d

# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml

NOTE: again the mcollective config files will be in /opt/rh/ruby193/root/etc/mcollective/ if you are running on RHEL or CentOS.

The server configuration looks pretty similar to the client.cfg. The securityprovider plugin must have the same values, because that's how the server knows that it can accept a message from the clients. The plugin.stomp.* values are the same as well, allowing the MCollective server to connect to the ActiveMQ service on the message broker host. It's really a good idea for the logfile value to be set so that you can observe the incoming messages and their responses. The loglevel is set to debug to start so that I can see all the details of the connection process. Finally the daemonize value is set to 1 so that the mcollectived will run as a service.

The mcollectived will complain if the YAML file does not exist or if the Meta registration plugin is not installed and selected. Comment those out for now. They're out of scope for this post.

Running the MCollective service


When you're satisfied with the configuration, start the mcollective service and verify that it is running:


sudo service mcollective start
Redirecting to /bin/systemctl start  mcollective.service
ps -ef | grep mcollective
root     13897     1  5 19:37 ?        00:00:00 /usr/bin/ruby-mri /usr/sbin/mcollectived --config=/etc/mcollective/server.cfg --pidfile=/var/run/mcollective.pid

You should be able to confirm the connection to the ActiveMQ server in the log.

sudo tail /var/log/mcollective.log 
I, [2013-09-19T19:53:21.317197 #16544]  INFO -- : mcollectived:31:in `
' The Marionette Collective 2.2.3 started logging at info level I, [2013-09-19T19:53:21.349798 #16551] INFO -- : stomp.rb:124:in `initialize' MCollective 2.2.x will be the last to fully support the 'stomp' connector, please migrate to the 'activemq' or 'rabbitmq' connector I, [2013-09-19T19:53:21.357215 #16551] INFO -- : stomp.rb:82:in `on_connecting' Connection attempt 0 to stomp://mcollective@msg1.infra.example.com:61613 I, [2013-09-19T19:53:21.418225 #16551] INFO -- : stomp.rb:87:in `on_connected' Conncted to stomp://mcollective@msg1.infra.example.com:61613 ...

If you see that, you can skip down again to the next section, MCollective End-to-End

Debugging MCollective Server Connection Errors


Again the two most likely problems are that the host or the stomp plugin are mis-configured.


sudo tail /var/log/mcollective.log
I, [2013-09-19T20:05:50.943144 #18600]  INFO -- : stomp.rb:82:in `on_connecting' Connection attempt 1 to stomp://mcollective@msg1.infra.example.com:61613
I, [2013-09-19T20:05:50.944172 #18600]  INFO -- : stomp.rb:97:in `on_connectfail' Connection to stomp://mcollective@msg1.infra.example.com:61613 failed on attempt 1
I, [2013-09-19T20:05:51.264456 #18600]  INFO -- : stomp.rb:82:in `on_connecting' Connection attempt 2 to stomp://mcollective@msg1.infra.example.com:61613
...

If I see this, I need to check the same things I would have for the client connection. On the MCollective server host:

  • plugin.stomp.host is correct
  • plugin.stomp.port matches Stomp transport TCP port on the ActiveMQ service
  • Hostname resolves to an IP address
  • ActiveMQ host can be reached from the MCollective client host (ping or SSH)

On the ActiveMQ message broker:

  • ActiveMQ service is running
  • Any firewall rules allow inbound connections to the Stomp TCP port
The other likely error is username/password mismatch. If you see this in your mcollective logs, check the ActiveMQ user configuration and compare it to your mcollective server plugin.stomp.user and plugin.stomp.password values.

...
I, [2013-09-19T20:15:13.655366 #20240]  INFO -- : stomp.rb:82:in `on_connecting'
 Connection attempt 0 to stomp://mcollective@msg1.infra.example.com:61613
I, [2013-09-19T20:15:13.700844 #20240]  INFO -- : stomp.rb:87:in `on_connected' 
Conncted to stomp://mcollective@msg1.infra.example.com:61613
E, [2013-09-19T20:15:13.729497 #20240] ERROR -- : stomp.rb:102:in `on_miscerr' U
nexpected error on connection stomp://mcollective@msg1.infra.example.com:61613: es_trans: transmit
 to msg1.infra.example.com failed: Broken pipe
...

MCollective End-to-End

Now that I have both the MCollective client and server configured to connect to the ActiveMQ message broker I can confirm the connection end to end. Remember that 'mco ping' command I used earlier? When there are connected servers, they should answer the ping request.

 sudo mco ping
node1.infra.example.com time=138.60 ms


---- ping statistics ----
1 replies max: 138.60 min: 138.60 avg: 138.60 

OpenShift Node 'plugin' agent

Now I'm sure that both MCollective and ActiveMQ are working end-to-end between the OpenShift broker and node. But there's no "OpenShift" in there yet.  I'm going to add that now.

There are three packages that specifically deal with MCollective and interaction with OpenShift:

  • openshift-origin-msg-common.noarch (misnamed, specifically mcollective)
  • rubygem-openshift-origin-msg-broker-mcollective
  • openshift-origin-msg-node-mcollective.noarch

The first package defines the messaging protocol for OpenShift.  It includes interface specifications for all of the messages, their arguments and expected outputs.  This is used on both the MCollective client and server side to produce and validate the OpenShift messages. The broker package defines the interface that the OpenShift broker (a Rails application) uses to generate messages to the nodes and process the returns. The node package defines how the node will respond when it receives each message.

The OpenShift node also requires several plugins that, while not required for messaging per-se, will cause the OpenShift agent to fail if they are not present
  • rubygem-openshift-origin-frontend-nodejs-websocket
  • rubygem-openshift-origin-frontend-apache-mod-rewrite
  • rubygem-openshift-origin-container-selinux
When these packages are installed on the OpenShift broker and node, mco will have a new set of messages available. MCollective calls added sets of messages... (OVERLOAD!) 'plugins'.  So, to see the available message plugins, use mco plugin doc.  To see the messages in the openshift plugin, use mco plugin doc openshift.

Mcollective client: mco


I've used mco previously just to send a ping message from a client to the servers.  This just collects a list of the MCollective servers listening. The mco command can also send complete messages to remote agents.  Now I need to learn how to determine what agents and messages are available and how to send them a message.  Specifically, the OpenShift agent has an echo message which simply returns a string which was sent in the message.  Now that all of the required OpenShift messaging components are installed, I should be able to tickle the OpenShift agent on the node from the broker.  This is what it looks like when it works properly:

sudo mco rpc openshift echo msg=foo
Discovering hosts using the mc method for 2 second(s) .... 1

 * [ ========================================================> ] 1 / 1


node1.infra.example.com 
   Message: foo
      Time: nil



Finished processing 1 / 1 hosts in 25.49 ms

As you might expect, this has more than its fair share of interesting failure modes.  The most likely thing you'll see from the mco command is this:

sudo mco rpc openshift echo msg=foo
Discovering hosts using the mc method for 2 second(s) .... 0

No request sent, we did not discover any nodes.


This isn't very informative, but it does at least indicate that the message was sent and nothing answered. Now I have to look at the MCollective server logs to see what happened. After setting the loglevel to 'debug' in /etc/mcollective/server.cfg, restarting the mcollective service and re-trying the mco rpc command, I can find this in the log file:


sudo grep openshift /var/log/mcollective.log 
D, [2013-09-20T14:18:05.864489 #31618] DEBUG -- : agents.rb:104:in `block in findagentfile' Found openshift at /usr/libexec/mcollective/mcollective/agent/openshift.rb
D, [2013-09-20T14:18:05.864637 #31618] DEBUG -- : pluginmanager.rb:167:in `loadclass' Loading MCollective::Agent::Openshift from mcollective/agent/openshift.rb
E, [2013-09-20T14:18:06.360415 #31618] ERROR -- : pluginmanager.rb:171:in `rescue in loadclass' Failed to load MCollective::Agent::Openshift: error loading openshift-origin-container-selinux: cannot load such file -- openshift-origin-container-selinux
E, [2013-09-20T14:18:06.360633 #31618] ERROR -- : agents.rb:71:in `rescue in loadagent' Loading agent openshift failed: error loading openshift-origin-container-selinux: cannot load such file -- openshift-origin-container-selinux
D, [2013-09-20T14:18:13.741055 #31618] DEBUG -- : base.rb:120:in `block (2 levels) in validate_filter?' Failing based on agent openshift
D, [2013-09-20T14:18:13.741175 #31618] DEBUG -- : base.rb:120:in `block (2 levels) in validate_filter?' Failing based on agent openshift

It turns out that the reason those three additional packages are requires is that they provide facters to MCollective. Facter is a tool which gathers a raft of information about a system and makes it quickly available to MCollective. The rubygem-openshift-origin-node package adds some facter code, but those facters will fail if the additional packages aren't present. If you do the "install everything" these resolve automatically, but if you install and test things piecemeal as I am they show up as missing requirements.

After I add those packages I can send an echo message and get a successful reply.  If you can discover the MCollective servers from the client with mco ping, but can't get a response to an mco rpc openshift echo message, then the most likely problem is that the OpenShift node packages are missing or misconfigured. Check the logs and address what you find.

Finally! (sort of)

At this point, I'm confident that the Stomp and MCollective services are working and that the OpenShift agent is installed on the node and will at least respond to the echo message.  I was going to also include testing through the Rails console, but this has gone on long enough.  That's next.

References