Tuesday, November 27, 2012

OpenShift Back End Services: DNS - Dynamic Updates

In the previous post I created an authoritative DNS server for the zone app.example.com running on ns1.example.com. The server will answer external queries and authoritative ones, but it will not yet accept update queries.  I need to add that facility before an OpenShift Origin broker service can use it.

OpenShift and DNS Updates

I don't know if I've actually posted this over and over but it feels like it since I've re-written this post at least 4 times since I started on it a week ago.  I'm going to go over it again on the off chance that someone reading this hasn't read the rest.

OpenShift depends on Dynamic DNS for one of its primary functions: Publishing developer applications.  If you can't make the applications public, OpenShift isn't very useful.

OpenShift publishes applications by creating new DNS records which bind the fully qualifed domain name (or FQDN) of the application to a specific host which contains and runs the application service.  The FQDN can then be used in a URL to allow users to access the application with a web browser.

The OpenShift broker provides a plug-in interface for a Dynamic DNS module. Currently (Nov 27 2012) there is only one published plugin.  This uses the DNS Update protocol defined in RFC 2136 along with a signed transaction defined in RFC 2845 to allow authentication of the updates.

The table below lists the information that the OpenShift Broker needs to communicate with a properly configured Bind DNS server to request updates.

OpenShift Origin Bind DNS Plugin information
Dynamic DNS Host IP address192.168.5.2ns1.example.com
Dynamic DNS Service port53/TCPDefault, but configurable
Zone to update (domain_suffix)app.example.comapps will go here.
DNSSEC Key Nameapp.example.comArbitrary name
DNSSEC Key ValueA really long stringTSIG size range: 1-512 bits

Any DNS update service would require similar information but it would be tailored to the service update protocol.

I need to generate a DNSSEC signing key for the update queries and then insert the  information into the DNS service configuration so that it will accept updates.

Generating an Update Key

The last value in the table above is missing.  I like to have all the ingredients before I start a recipe.  I'm going to generate that key value before moving on.  I'll need it for the OpenShift broker plugin configuration and for the back end DNS service setup.

I'll use dnssec-keygen to create a pair of DNSSEC key files (public and private).

dnssec-keygen -a HMAC-MD5 -b 256 -n USER app.example.com

This command translates as "Create a 256 bit USER key with the HMAC-MD5 algorithm, named example.com". A USER key is one that is used to authenticate access. The HMAC-MD5 keys can be from 1 to 512 bits. A 256 bit key fits on a single line making it easier to manage in this post.

The output indicates the filename of the resulting key files. This command produces two files:

  • Kapp.example.com.+157+45890.key
  • Kapp.example.com.+157+45890.private

The HMAC-MD5 key is a symmetric key, so the two files contain the same key value. The ".key" file is a single line and represents a DNS Resource Record. The ".private" file contains the key string and some metadata.

The key isn't actually used as a password.  Rather it is a "signing key" for a Digital Signature which will be attached to the update query.  The signature is generated using the signing key and the contents of the query. It is mathematically unlikely that someone would be able to generate the correct signature without a copy of the signing key. The DNS server generates a signature string with its copy of the key and the update query.  if the signatures match then the DNS server can be confident that the message did come from an authorize update client.

The key files look like this:

cat Kapp.example.com.+157+45890.key
app.example.com. IN KEY 0 3 157 LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=

cat Kexample.com.+157+45890.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=
Bits: AAA=
Created: 20121122011903
Publish: 20121122011903
Activate: 20121122011903

The only part I care about at the moment is the Key: line of the .private file:


I'll copy that string and save it for later.

Enable DNS Updates

Now that I have an update key and a running authoritative server I can use them to enable DNS updates. I need to provide a copy of the key string to the named and inform it that it should accept updates signed with that key.

I'm going to add some contents to the /var/named/app.example.com.conf file to provide this information.  I'm going to put the key string in its own file and use another include directive to bring it into the configuration.  This way if I need to change the key, I only need to edit a single line file rather than trying to find the key string in a larger configuration.

I can append the key information to the file fairly simply.  The key clause looks like this:

key app.example.com {
  algorithm HMAC-MD5;
  // secret "some long key string in base64 encoding";
  include "app.example.com.secret";

Append that to the /var/named/app.example.com.conf file.  The include line indicates that the secret key will be placed in /var/named/app.example.com.secret.

The secret line format is indicated in the key clause comment.  It's a single line containing the word "secret" and the key string in double quotes, terminated by a semicolon.

echo 'secret "LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=" ;' \
> /var/named/app.example.com.secret

Finally I need to inform the named that the app.example.com zone can be updated using that key. I have to add an allow-update option to the zone section. When that's done the config file will look like this:


zone "app.example.com" IN {
    type master;
    file "dynamic/app.example.com.db";
    allow-update { key app.example.com ; } ;

key app.example.com {
  algorithm HMAC-MD5;
  // secret "some long key string in base64 encoding";
  include "app.example.com.secret";

Re check the files once and try restarting the named.

Verify the Named Operation

The first thing to do as always when making a configuration change is check that I haven't broken anything.  I'll do several things to check:
  1. Observe the restart results on the CLI
  2. Search for named lines at the end of /var/log/messages and scan for errors
  3. Run queries for the SOA, NS and A glue records for the app.example.com zone (localhost)
  4. Re-run the queries from another host pointing dig at ns1.example.com
These are the same checks that I did when establishing the authoritative server in the previous post.

Verify DNS Update Operations

I'm almost finished now. The final step is to verify that I can add and remove resource records in the app.example.com zone.

The bind-utils package includes the nsupdate utility.  This is a command line tool that can send DNS update queries.

I have the server hostname and IP address.  I have the TSIG key. I know the dynamic zone.  To test dynamic updates I'm going to add a TXT record named "testrecord.app.example.com" with a value "this is a test record". The nsupdate command below expresses that.

nsupdate -k Kexample.com.+157+45890.private
update add testrecord.app.example.com 1 TXT "this is a test record"

When this command completes we should be able to send a query for that name and get an answer:

dig @ testrecord.app.example.com txt

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @ testrecord.app.example.com txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18488
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;testrecord.app.example.com. IN TXT

testrecord.app.example.com. 1 IN TXT "this is a test record"

app.example.com. 30 IN NS ns2.example.com.
app.example.com. 30 IN NS ns1.example.com.

ns1.example.com. 600 IN A
ns2.example.com. 600 IN A

;; Query time: 0 msec
;; WHEN: Wed Nov 28 02:20:33 2012
;; MSG SIZE  rcvd: 146

I can also check the named logs in /var/log/messages:

grep named /var/log/messages
Nov 28 02:12:46 ns1 named[30675]: client signer "app.example.com" approved
Nov 28 02:12:46 ns1 named[30675]: client updating zone 'app.example.com/IN': adding an RR at 'testrecord.app.example.com' TXT
Nov 28 02:12:46 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112911)

I can see that an update request arrived and the signature checked.  A record was added, and update notifications were sent (or would have been sent) to any secondary servers.

Removing the record again looks very similar:

nsupdate -k bind/Kapp.example.com.+157+13871.private 
update delete testrecord.app.example.com TXT

When I query for the record again I get a negative response now:

dig @ testrecord.app.example.com txt

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @ testrecord.app.example.com txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24061
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;testrecord.app.example.com. IN TXT

app.example.com. 10 IN SOA ns1.example.com. hostmaster.example.com. 2011112912 60 15 1800 10

;; Query time: 0 msec
;; WHEN: Wed Nov 28 02:56:01 2012
;; MSG SIZE  rcvd: 95

Note that there is no answer section.  No section.  No answer.

The logs will also show that the record was removed:

grep named /var/log/messages
Nov 28 02:50:42 ns1 named[30675]: client signer "app.example.com" approved
Nov 28 02:50:42 ns1 named[30675]: client updating zone 'app.example.com/IN': deleting rrset at 'testrecord.app.example.com' TXT
Nov 28 02:50:42 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112912)

It took four blog posts to get here, but I now have a working DNS server capable of DNS Updates.

The final steps are to create a secondary server on ns2.example.com and to submit the nameserver information to my IT department for delegation.

DNS is ready for Openshift.


  • RFC 2136 defines the DNS Update protocol.
  • RFC 2845 defines signed transactions which allow DNS updates to be authenticated.
  • dnssec-keygen Generates TSIG keys suitable for signing DNS updates
  • nsupdate is a command line tool for making DNS updates

Openshift Back End Services: DNS - An Authoritative Server

OpenShift Origin dynamic DNS requires an authoritative zone for the application records.  So the first step to creating the DNS back end service is to establish that authoritative zone.  When I'm sure it will serve records then I'll add the dynamic update feature to it.

There are a number of good resources on setting up DNS on Linux.  I'll post a few links to them in the Resources.  I've even written part of one, but each time I do this I see it a little differently and (I think) improve my understanding.  So I'm going to do it again.


I'm going to try to create a proper (ready to be delegated) zone.

Here's the list of information I'll need to set it up:

DNS Service Configuration Variables
Primary Nameserver
IP Address192.168.5.2
Hostnamens1.example.comNot in the app domain
Secondary Nameserver
IP Address192.168.5.3
Hostnamens2.example.comNot in the app domain
Update Configuration
Application Zoneapp.example.comNot the top level
Application Zone Key Nameapp.example.comArbitrary name
Application Zone Key TypeHMAC-MD5
Application Zone Key Size64 bits512 bits in real life
Application Zone Key ValueA Base64 encoded stringGenerated by dnssec-keygen

I'll be setting up both a primary and secondary server for the zone.  The update configuration information won't be needed here except to establish a static zone that will be made dynamic later.

The operations listed below will all be performed on ns1.example.com.

Bind on Linux

Installing ISC Bind 9 on most modern Linux distributions is pretty easy. It's an old and well worn tool.  For RPM based distributions you can just use yum. I generally install the bind-utils package as well just so they're handy.
yum install bind bind-utils

The DNS service binary is called named.  When I refer to Bind, I'll be talking about the software in general. When I refer to named I'll be referring to the daemon and its configuration files.

Once the package is installed there are a number of small configuration changes that are needed to enable and verify the caching service.

  • Enable listener ports
  • Enable rndc
  • Enable start on boot
  • Start the named service
  • Confirm caching service
  • Confirm rndc controls

Named Configuration and Management

The named service follows a traditional layout model on Linux (It's actually probably one of the canonical services).  Before I start making changes I want to

Named Configuration Files

The primary service configuration file is /etc/named.conf.  Additional configuration information and service data reside in the /var/named directory.  The named daemon logs to /var/log/messages through syslog.
  • /etc/named.conf - primary configuration file
  • /var/named - additional configuration and data
  • /var/log/messages - service logging

You can filter for named related log messages in /var/log/messages by grepping for (wait for it!) "named".

ISC provides a complete reference of the named configuration options on their site.

The named service adds one non-traditional feature.  It has a tool called rndc which might stand for "remote name daemon controller".  The rndc tool use used locally by the service command to provide status for the named service.  It can also be used to adjust the logging level, to dump the current zones to a file, or to force a zone reload  without restarting the daemon.

rndc does require some additional setup.  It is not enabled by default as it requires an authentication key.  A default key would present a security risk.  The bind package provides rndc-confgen to help set up the rndc access to the local named. This command produces a required key file which will reside at /etc/rndc.key.

  • /etc/rndc.key - Remote name daemon control access key

rndc also requires an addition to the /etc/named.conf file. I'll do that just before I try starting the service.

Before making any change to a configuration file I copy the original and name the new file <filename>.orig. For example, /etc/named.conf would become /etc/named.conf.orig. This way I can track my changes and revert them to the initial values if I need to.

Enable Listeners

The initial configuration of /etc/named.conf restricts queries and updates to the local IP interfaces (IPv4 and IPv6 ::1).  Since I want to allow basically anyone to find out about my zone, I have to open this up.  There are three configuration lines that control listeners and query access:

  • listen-on
  • listen-on-v6
  • allow-query

Each of these options takes a semi-colon (;)delimited list of listeners.  The list is encapsulated in a paired set of curly-braces ( {} ). Since this will be a public service, I just have to replace the appropriate localhost address entries with the keyword "any".


options {
 listen-on port 53 { any ; };
 listen-on-v6 port 53 { any; };
 directory  "/var/named";
 dump-file  "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
 allow-query     { any; };
 recursion yes;

If you want to be tricksy a sed one-liner (and a safety copy) will do the trick if you're starting from the default:

cp /etc/named.conf /etc/named.conf.orig
sed -i -e 's/\|::1\|localhost/any/' /etc/named.conf

Enable rndc

rndc is the Remote Name Daemon Control program. It's now used for all communications and control to the named on a bind server host. rndc does allow you to securely control a name server daemon remotely, but we won't be using it that way. rndc is also the program that gets and reports the status information when you run service named status so it's nice to have it configured.

rndc comes with a nice little configuration tool to help: rndc-confgen. rndc-confgen creates a unique rndc access key and places the rncd configuration in /etc/rndc.key for you.

rndc-keygen -a

This creates a new file named /etc/rndc.key which contains the access key for the named process. Both the named and the rndc command must have access to this key. rndc uses the /etc/rndc.key file by default. named must be configured for it.

If the rndc-keygen command hangs it is because there is not enough entropy (randomness) on the system. I could Log onto another window and type random commands for a bit and it would complete. If I'm impatient I could run it with -r /dev/urandom which will always complete, but which may be less secure because it will not block waiting for enough randomness to generate a good key. I often do that for lab systems.

rndc-keygen does not set the SELinux context for the key file.  It also does not set the ownership and permissions so that the named can read it. restorecon will do it for me though.

restorecon -v /etc/rndc.key
chown root:named /etc/rndc.key
chmod 640 /etc/rndc.key

Now that we have a key, we have to tell the named to use it. Append the section below to the bottom of the /etc/named.conf file.

// enable service controls via rndc
// use the default rndc key
include "/etc/rndc.key";

controls {
        inet port 953
        allow {; } keys { "rndc-key"; };

Verifying a caching DNS server

With this configuration I have a caching DNS server.  I need to check the operation now before going ahead to add a new zone.

I use the expected tools to start the named service and get status:

service named start
Starting named:                                            [  OK  ]

service named status
version: 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5
CPUs found: 16
worker threads: 16
number of zones: 19
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 0/0/1000
tcp clients: 0/100
server is up and running
named (pid  31010) is running...

This indicates that the server started and that it is responding to rndc queries. If the daemon does not start or if I got errors from the status query, I'd check /var/log/messages for error messages.

Next we want to verify that the server is actually answering queries. I use dig or host to check. host has a simpler interface and output but for this I want the verbosity of dig to help diagnose if there are any problems. I have to try from two different locations: localhost and "somewhere else". I want first to verify that the service is running and answering, and second that it is accessable from outside itself. I'll only show the localhost queries, but the remote ones are identical.

dig @ www.example.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @ www.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29408
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 4

;www.example.com.  IN A

www.example.com. 172789 IN A

example.com.  172788 IN NS a.iana-servers.net.
example.com.  172788 IN NS b.iana-servers.net.

a.iana-servers.net. 172788 IN A
a.iana-servers.net. 172788 IN AAAA 2001:500:8c::53
b.iana-servers.net. 172788 IN A
b.iana-servers.net. 172788 IN AAAA 2001:500:8d::53

;; Query time: 0 msec
;; WHEN: Wed Nov 21 20:42:21 2012
;; MSG SIZE  rcvd: 185

This is actually the real right answer.  Since RFC 2606 reserves the example.com domain, the IANA also  serves that domain.  When I build my test server I'll override that.  If I were building a real OpenShift Origin service I'd get a properly delegated sub-domain of my organization or get my own domain from a registrar.

Take a moment to look at that output. It's meaningful. The ANSWER SECTION contains the only response, an A record. The AUTHORITY SECTION lists the servers which are the designated sources for the content in the example.com zone. It contains two NS records. NS record values are fully qualified domain names (FQDN). That means those names don't have some implied suffix. They end with a dot (.) which anchors them to the root of the DNS. The ADDITIONAL SECTION provides IP address resolution for the NS record FQDNs. The last section indicates that the answer came from the IPv4 localhost address and gives the date/time stamp.

So this answer tells you not only that the IP address for www.example.com is but where the answer came from.

I'll do that again from some other host and set the server address to the public IP address of my nameserver host.

Zone Configuration

Now that I have a caching server running, it's time to start adding some content. The /etc/named.conf file syntax has a directive to include another file in line. Rather than putting the entire configuration section in the master configuration file, I'll put my configuration information in another file and include it. That just requires appending one line to /etc/named.conf

echo 'include "app.example.com.conf" ;' >> /etc/named.conf

Other than the /etc/named.conf file, all of the named configuration files reside in /var/named/. That's the default location for relative path names in the /etc/named.conf as well. I'll create a configuration file fragment there. Since I'm creating the app.example.com domain I'll call the file /var/named/app.example.com.conf.

zone "app.example.com" IN {
    type master;
    file "dynamic/app.example.com.db";

The directory option in the /etc/named.conf file determines the location of any files listed with relative path names. (see the fragment in the Enabling Listeners section)  The file directive above means that the absolute path to the zone file will be /var/named/dynamic/app.example.com.db

This will eventually be a dynamic zone.  That's why I'm putting it in "dynamic/app.example.com.db".  If it were to be a static zone I'd probably put it right at the top of the /var/named tree.

The Zone File

The last file I need to create is the zone file. This file defines the initial contents of the application zone. It also defines the NS (nameserver) records for the zone and the default TTL (time to live).


$TTL 1800 ; Default TTL: 30 Minutes
app.example.com. IN SOA ns1.example.com. hostmaster.example.com. (
                         2011112904 ; serial
                         60         ; refresh (1 minute)
                         15         ; retry (15 seconds)
                         1800       ; expire (30 minutes)
                         10         ; minimum (10 seconds)
                     NS ns1.example.com.
                     NS ns2.example.com.
;; prime the nameserver IP addresses for the app zone.
ns1.example.com.               A
ns2.example.com.               A

Verifying The App Zone

Once the configuration file and the zone database file are in place it's time to try restarting the named service. I use service named restart. and observe the results.

service named restart
Stopping named: .                     [ OK ]
Starting named:                       [ OK ]

Then I use grep named /var/log/messages to observe the typical start up messages. I look for a line indicating that the app.example.com zone has been loaded.

grep named /var/named/messages | grep loaded
Nov 22 17:40:10 ns1 named[3888]: zone 0.in-addr.arpa/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone example.com/IN: loaded serial 2011112904
Nov 22 17:40:10 ns1 named[3888]: zone app.example.com/IN: loaded serial 2011112906
Nov 22 17:40:10 ns1 named[3888]: zone localhost.localdomain/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone localhost/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: managed-keys-zone ./IN: loaded serial 53

Now that I know that the zone has been loaded successfully, I'll check that it's served properly. I first request the SOA (Start of Authority) record, and then the full zone dump.

dig @localhost app.example.com soa
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @localhost app.example.com soa
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3502
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;app.example.com.  IN SOA

app.example.com. 30 IN SOA ns1.example.com. hostmaster.example.com. 2011112906 60 15 1800 10

app.example.com. 30 IN NS ns2.example.com.
app.example.com. 30 IN NS ns1.example.com.

ns1.example.com. 600 IN A
ns2.example.com. 600 IN A

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Thu Nov 22 17:29:32 2012
;; MSG SIZE  rcvd: 148

Now test a complete zone dump, (and save it for comparison)

dig @ app.example.com axfr
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @localhost app.example.com axfr
; (2 servers found)
;; global options: +cmd
app.example.com. 30 IN SOA ns1.example.com. hostmaster.example.com. 2011112906 60 15 1800 10
app.example.com. 30 IN NS ns1.example.com.
app.example.com. 30 IN NS ns2.example.com.
app.example.com. 30 IN SOA ns1.example.com. hostmaster.example.com. 2011112906 60 15 1800 10
;; Query time: 1 msec
;; SERVER: ::1#53(::1)
;; WHEN: Thu Nov 22 17:18:28 2012
;; XFR size: 4 records (messages 1, bytes 152)


At this point I have a working authoritative server for the app.example.com zone.  To get it properly delegated I need to create a secondary server and configure zone transfers.  Then I can provide the nameserver NS and A records and a contact to my IT department and they can complete the delegation.

For now I'm going to skip delegation. The next post will describe the configuration of dynamic updates


Monday, November 26, 2012

OpenShift Back End Services: The DNS - Concepts

Most of the time, DNS just works. I have heard some people (hardware lab techs mostly) talk about how they use IP addresses for their work because DNS is so unreliable.  Then I explain to them that (at least outside the lab) DNS is probably the most reliable and fundamental service on the modern internet.  Next to the TCP/IP and the routing protocols, DNS is the most critical service.  Without DNS the rest of the net doesn't matter because no one can find anything.  But very seldom does DNS actually fail on a large scale.

For a system like OpenShift, DNS is life's blood.  The whole purpose of a PaaS is to make applications available to users.  OpenShift does that by adding a new DNS record each time an application is created.  The name portion of the record is crafted from the developer's namespace and application name.  The value portion directs a user to the node host which offers the application service.  But before the browser can find the application, the DNS resolver must find the DNS record.

If you're going to run your own DNS it's important to understand how DNS services interact.

What's so hard about DNS?

Given the ubiquity of DNS and its critical function, I've been surprised at the amount of difficulty it has caused configuring it into OpenShift Origin.  I think part of the issue is that ordinarily it works so well that few sysadmins and developers have to work with it in any depth.  Most people's experiences with DNS consist of checking and setting the /etc/resolv.conf nameserver and search lists, and the occasional dig command to check if a zone is responding.

In most companies there's one or two of the IT folks who are "The DNS guys" (or girls?).  They manage the external DNS (which shouldn't change fast) and the internal DNS (which uses lots of DHCP for desktops, laptops, wireless).  They own the DNS domains and getting new IP name/address assignments from them has a well defined process.  Getting a delegated sub-domain is generally a more involved process.  The DNS Guys don't like to do it (because they get the calls when your DNS  breaks) so people who need DNS (like lab spaces) will make do with their own or go without.

A number of geeks like me have set up split DNS in their houses.  This requires DNS forwarding features, but not delegation.  There are even tools now like Dnsmasq which implement simple split DNS and combine DNS, DHCP and Dynamic DNS all in one nice relatively simple service. These are meant for small labs or home networks where they can control the entire DNS namespace.  They provide the hostname to IP address mapping of DNS and combine that with the host resolver configuration offered by DHCP. They only work at the bottom of the namespace hierarchy and they do not require delegation, as nothing in the local database is ever published outside that bottom layer zone. Again, this removes the need for the average system administrator to think much about what's happening behind the scenes.

And there's sometimes rather a lot going on behind the scenes.

The Domain Name Service Behind the Scenes

If you're familiar with DNS operations, you can skip this part.

If you've never managed a DNS hierarchy, you might think that to install a DNS service you just install the bind package, edit the configuration file, add some zone data and start the daemon. Done.  Right?  Not quite.  Unlike nearly all other typical database services, DNS requires the participation of other servers to work properly.

The DNS  is a specialized distributed database with a hierarchical namespace.  Note something really important here.  I didn't say "Bind is..".  I said "The DNS is.." The DNS is ONE DATABASE. The data is distributed across the entire internet and the authority for portions are delegated to hundreds of thousands (or more) origanizations and individuals, but if it weren't a single unified entity it wouldn't work.  The magic of the DNS is the way in which the namespace and data are "glued" together. To see how that works I'm going to walk through an example.

DNS Queries and the /etc/resolv.conf file

When I try to access a web site from my laptop, the first thing that happens is a DNS query to resolve the URL host name to the IP address of the destination host. My computer is going to send a DNS request to some server. A computer that answers DNS requests is called a nameserver.  I have to know the address of the nameserver.  If I only knew the name of the name server, I'd have to do a look-up query for that, and since I don't have the address of someone to ask I'd be stuck in a loop.

Fortunately, the pump is primed by the /etc/resolv.conf file.

The /etc/resolv.conf file contains a list of nameserver entries. This is a list of IP addresses.  Each of the addresses corresponds to a DNS nameserver host.

; generated by /usr/sbin/dhclient-script
search westford.example.com example.com

When I make a request with a hostname, the resolver library on my laptop issues a query to the first nameserver in the list.  Say I wanted to visit openshift.example.com.  The resolver would send a query which asks essentially "tell me what you can about anything named 'openshift.example.com'"  You can simulate this with either the dig or host commands.

The dig and host commands

The dig and host commands are programs who's only purpose is to issue DNS queries and report the responses.  I tend to use host when all I need is the answer.  host has much simpler output and looks to me like it is designed for use in command-line scripts.  It responds with a single line and each field in the output is space separated.

host www.example.com
www.example.com has address
www.example.com has IPv6 address 2001:500:88:200::10

I use dig when I am verifying or diagnosing DNS operation.  By default dig prints a (mostly) human readable  report of the entire DNS response from the nameserver.  This includes not only the requested records but the authority records which indicate where the answer ultimately came from.  The format is not horribly human friendly or even string-parser-friendly, but once you learn to read it it is very concise and informative.

dig www.example.com

; <<>> DiG 9.9.2-RedHat-9.9.2-2.fc17 <<>> www.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30626
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 5

; EDNS: version: 0, flags:; udp: 4096
;www.example.com.  IN A

www.example.com. 172791 IN A

example.com.  163951 IN NS a.iana-servers.net.
example.com.  163951 IN NS b.iana-servers.net.

b.iana-servers.net. 1044 IN A
b.iana-servers.net. 1044 IN AAAA 2001:500:8d::53
a.iana-servers.net. 1044 IN A
a.iana-servers.net. 1044 IN AAAA 2001:500:8c::53

;; Query time: 43 msec
;; WHEN: Mon Nov 26 18:14:47 2012
;; MSG SIZE  rcvd: 196

In the examples that follow I'm mostly going to use dig though I may also trim them some to highlight the important parts.

A Name Lookup Example

This example is not at all contrived.  This kind of thing happens to me every day.. Really..  Sure it does.

Say I'm sitting at my desk, working on my laptop.  I'm a mid level sysadmin at Example.Com in the office in Boston, MA.   My desktop has a DNS name something like "llamadesk.boston.example.com".  My co-worker (the lucky SOB) is working from the beach outside company office on Maui.  He posts a document file on a web server in the office there.  The web server hostname is "www.maui.example.com". I want to see the document so he sends me a url for it and I dutifully paste it into my web browser address bar and hit the enter key.

My web browser is linked with the local resolver library (usually called libresolv). It has a function called getaddrinfo (used to be gethostbyname) which takes a hostname string as input and returns (among other things) an IP address associated with that name.  What happens between the call and the return is the interesting part.

The first thing the resolver library does is read /etc/resolv.conf. Then it crafts a query packet and sends it off to the first IP address in the nameserver list and waits for a response.

The nameserver is listening for query packets. I receives the packet and tries to find the best answer.

A nameserver, when looking at a query, can know one of three things:

  • I know the answer.
  • I don't know the answer, but I know the nameserver for a domain that contains the answer.
  • I don't know the answer or the domain, but I know where the root domain is.

A server which knows the answer is called the authoritative nameserver for the domain.  It will answer all queries for the contents of the domains it serves.

If the nameserver is not authoritative it then has a choice.  It can merely return a response which means "I don't know" or it can perform a recursive query.  Most of the nameservers which are at the edge of the DNS where desktops will be making queries will be configured to recurse.

So, my nearest (recursive) nameserver is in boston.example.com. It doesn't know about servers in maui.example.com. However, it does know that it's in example.com and it knows how to find the example.com nameservers.  These are servers in the NS records for example.com. So the nearby nameserver issues a query to one of the NS servers for example.com and asks for the NS records from the maui.example.com domain.  The reply will contain the names of the authoritative nameservers.  The nearby nameserver then requests the A records for the maui.example.com nameservers.  Now it knows someone who does know the answer.  It sends one final query to the maui nameserver for www.maui.example.com.  The maui nameserver returns the answer (or an error response) and the local nameserver returns the answer to my browser which can finally make a connection to the actual target host.

Did you follow all that?  See if this helps:

Or this?
  1. llamadesk -> ns1.boston.example.com
    "tell me the IP address for www.maui.example.com"
  2. ns1.boston.example.com -> ns1.example.com
    "tell me who serves maui.example.com"
  3. ns1.boston.example.com -> ns1.example.com
    "tell me the IP address for ns1.maui.example.com"
  4. ns1.boston.example.com -> ns1.maui.example.com
    "tell me the IP address for www.maui.example.com"
  5. llamadesk <- ns1.boston.redhat.com
    "here's the IP address you asked for"

Glue Records: Binding the Internet Together

The links that make the DNS work are known as glue records. The process of establishing a link between one layer in the hierarchy and the next layer down is called delegation.

The nameserver at the example.com level has to know about all of the sub-domains below example.com.  It must have two types of records for each sub-domain.  It must have a set of NS records which contain the DNS name of the authoritative servers for the sub-domain.  Since NS records return the hostname of the nameservers, the parent must also provide an A record for each nameserver.

As noted in other places, the technical aspects of delegation are much less significant than the political or organizational aspects.  Delegation requires the establishment of a relationship communication and of trust between groups that may normally be somewhat territorial. Once a sub-domain is delegated it is the responsibility of the receiving administrator to ensure that the domain is always available so that the it contains remain accessible and to be able to accept and respond to problem reports.

Development and Test Environments: Rogue DNS

"Rogue DNS" is the term I use for an undelegated DNS zone.  Some people try to soften the term but I think "rogue" carries just the right connotations.  A rogue is an independent, slightly unsavory character who none the less is capable and possibly even attractive in a "bad boy" sort of way.  A rogue is not always a bad guy and sometimes it takes a rogue to save the day.

Every NAT network which includes split DNS would be considered "rogue" under this definition.  That's pretty much every home network and most commercial business networks today.  Rogues aren't all bad.

Rogue zones are also common in testing and small or personal development environments.  They don't require any negotiation. They're pretty much required for demos or livecd try-it-out implementations.

The establishment of a rogue zone is really easy: Just create the servers and start adding resource records to the zone. The problem is that without delegation, the rogue zone is invisible to everyone else. 

The real problem with rogue DNS is that every client that wants to participate in the zone must be manually re-configured to see the rogue.  The first nameserver in the client /etc/resolv.conf must be one of the rogue name servers.  In a typical NAT environment the owner of the DNS also owns the DHCP services. Since the DHCP server also provides dynamic nameserver information, all of the DHCP clients automatically participate in the DNS as well.  In a lab setting, the lab administrators may control the DHCP as well.

Rogues and DNS Forwarding

The other problem with a Rogue DNS service is that the rogue. because it is not delegated, does not know about anything outside itself.  Bind does have a facility for "forwarding" requests.  This is commonly used in NAT environments.

When a forwarding server gets a query for which it is not authoritative, rather than trying to recurse, it will forward the request directly to one of a list of "upstream" servers.  These are usually the servers that would normally have been in the nameserver host's /etc/resolv.conf.

Dynamic DNS

The final concept to cover is Dynamic DNS.  As noted in the previous post, OpenShift depends on the ability to add and remove resource records from a DNS zone.

In most corporate DNS services the zone files are fairly static.  They are often mechanically generated from some other database on a regular basis.  It is common for DNS updates to require from 1 or 2 hours to as much as 24 hours.   OpenShift requires the updates to be applied instantly and propagation times of more than a few seconds are considered unacceptable performance.

The one exception is DNS assigned from DHCP.  Microsoft Active Directory is especially good at this.  Dnsmasq is a combined DNS/DHCP/TFTP service designed for home and small business NAT networks.  When it assigns an IP address it can also bind the address to a hostname requested by the client.  It is also possible to connect ISC Bind and ISC DHCP to do Dynamic DNS.

OpenShift does Dynamic DNS through the DNS plugin.  I want to say "plugins", but right now there is only one DNS plugin.  I've written a few posts on writing a new DNS plugin, but it needs the last few, and the sample I picked will only be useful for labs.  Personally I think we need plugins for the greatest possible variety of external DNS services, from Microsoft Active Directory DNS to commercial services.

 DNS Update services will all have similar communications requirements.  Server and access information as well as the zone to update and the new resource record content.


I'm sure you'll agree I've lectured enough on the relevant capabilities and behaviors of DNS.  I mostly went through this exercise to be sure I hadn't missed anything myself.

You'll notice that I refer a lot to RFCs (Request for Comments).  These are the official specifications for the behavior of parts of the internet.  A lot of people find the idea of the RFCs intimidating. They're dense and bland.  They're also your friend. Don't be scared to go looking for information you need.  You don't have to read them like a good novel, but it's good to scan the relevant documents and at least know where to find answers.  I think a lot of people also skip the RFCs because people are looking for "how do I do it".  The RFCs only tell you "How does it work".  I think often the latter helps illuminate the former.

When I go looking for an RFC, I usually don't know the right one to look for.  Use the search engines
Google for your topic and add "RFC" to the beginning of the query and you'll very likely get a good reference.

Scan the RFCs. You'll be glad you did.

The next post will describe the creation of an authoritative DNS server using ISC Bind 9.  As I go along I mean to include not only the configuration steps but to demonstrate some tools and resources for checking the status of the service and for diagnosing any problems that might arise.


RFCs specifically significant to OpenShift:
  • RFC 1033 - Domain Administrators Operations Guide
  • RFC 1034 - Domain Names - Concepts and Facilities
  • RFC 1035 - Domain Names - Implimentation and Specification
  • RFC 2136 - DNS Update
  • RFC 2845 - Secret Key Transaction Authentication for DNS (TSIG)

Sunday, November 25, 2012

OpenShift Back End Services: The Domain Name System - Overview

In the previous two posts I covered the OpenShift Origin back end database and the OpenShift broker to node communications system (messaging).

This time I'm going to talk about the service OpenShift Origin uses to publish user's applications: The Domain Name System or DNS.

OpenShift Origin and the DNS

One of the critical functions of OpenShift Origin is the publication of new user applications.

Each new application gets its own IP name. These names are made available universally through the DNS. The name maps directly or indirectly back to the host on which the application runs. Without the DNS, the user applications would remain invisible to the rest of the world and in particular to the application's audience.

The other three back end services (Data store, Messaging and Identification/Authentication) used by OpenShift Origin are, in some sense, self-contained.  Only the OpenShift Origin broker or node needs to access them.  In fact, access by any other means would be considered a security flaw in the design or implementation of the OpenShift Origin service.

One implication of the self-contained nature of these services is that they can be implemented and run without  the need for cooperation with other parts of the network.  Publication (and the DNS) is fundamentally different. It makes no sense to run a "self-contained DNS".

This is why I'm going to urge you not to run your own.

The best way to manage this part of the OpenShift Origin service is to ask someone else to do it. Every organization has an IT department.  They're going to control both the internal and external DNS services.  They're also going to have the knowledge and resources to add a sub-domain for your applications and to configure it for your access.  You'll need to get some information from them that you'll configure into the Bind DNS plugin and then your OpenShift Origin service will be able to publish new applications automatically.

The Openshift Bind Plugin

There's currently only one DNS plugin at the moment.  This is the Bind plugin rubygem-openshift-origin-dns-bind. This is actually a misnomer.  The plugin uses the Dynamic DNS Update protocol defined in RFC 2136. Any DNS service which implements RFC 2136 should accept updates from the Bind plugin. However, the only service I can confirm is ISC Bind, hence the name.

The OpenShift service can only add applications to a single zone. All of the resource records in a dynamic zone are potentially changed. This means that the OpenShift servers should not reside in the same zone as the applications.

Bind Plugin Requirements

If you do get your IT department to set up dynamic DNS for you, this is the information you'll need from them.

The Bind plugin configuration requires:

  • Update Server IP address
  • Update Server port
  • Update zone
  • TSIG Key Type
  • TSIG Key Name
  • TSIG Key Value

Running a Delegated Zone

If your IT department chooses to delegate your application zone, there is some information you will have to give them.

  • Primary Nameserver fully qualified domain name
  • Primary Nameserver IP address
  • Secondary Nameserver fully qualified domain name
  • Secondary Nameserver IP address
  • Zone contact: an email address for a person who can manage the domain.

Your IT department will configure this information into a Start of Authority (SOA) record, a pair of nameserver (NS) records and two A records.  These are the DNS delegation glue records which allow others to find your zone.

In this case you will be responsible for building and managing your own Dynamic DNS servers.  The next few posts will walk through how to set up your own DNS servers on ISC Bind.  This is a more involved process than either of the previous services, so I'm going to present it in several steps.

  1. Caching Server - Answers queries but doesn't contain any zone data.
  2. Primary Authoritative Name Server - Answers queries, contains the master copy of some zone data
  3. Secondary Authoritative Name Server - Answers queries, contains a slave copy of some zone data
  4. TSIG User Key - A symmetric key for user authentication 
  5. Dynamic DNS Server - accepts update requests for a zone. Serves the master copy.

Unconventional DNS Configurations

There are two additional topics I plan to cover, though they're best avoided if possible.  They deliberately break the hierarchical model of the DNS and can cause considerable confusion for unwary users and administrators.

  • Rogue DNS
    For labs and development
  • Serving DNS from dynamic IP addresses.
    For environments like Amazon Web Services or libvirt virtual machines

A Rogue DNS service is one which serves zones which are not properly delegated.  They are accessable only if every host that participates is explicitly configured to make requests from the rogue servers.

Hosts on services like Amazon are normally given IP addresses via DHCP. If the IP address of the primary public interface is re-addressed, the DNS services must also be reconfigured and possibly all of the clients must be notified.  This extremely bad for the stability of a DNS service but may be required in some environments.


Monday, November 19, 2012

OpenShift Back End Services: Messaging (ActiveMQ)

In the previous post I detailed creating a back end MongoDB service which is ready for the OpenShift Origin broker to use.

The second back end service I'm going to set up is the messaging service.  This service connects the broker to the nodes.  It carries commands from the broker to the nodes and provides a means for the broker to query the node status.


As with the previous posts there are several settings or variables for which we'll need values.

Message Server Setup Information
Message Server Hostnamemsg1.example.com
Message Server IP address192.168.5.8
ActiveMQ console admin passwordadminsecret
ActiveMQ messaging admin account passwordmsgadminsecret
OpenShift messaging account namemcollective
OpenShift messaging account passwordmarionette

Messaging 101

If you're already familiar with HPC or CMS messaging, you can skip this bit.

Most people today are familiar with a form of messaging. Whether it's SMS (cell phone text messages), commercial Instant Messaging (IM), Twitter or Internet Chat Relay (IRC), we all get the concept of writing a message, attaching an address to it and sending the message.  We expect that the message will arrive at the destination, will be read, and if necessary, the receiver will compose and send a reply back.

Most SMS or IM are addressed to a single destination, but most people are also familiar with chat rooms, a form of broadcast message.  Each of the participants connects to (subscribes) the "room" or "channel".  Any message sent to the room is forwarded to all of the participants (subscribers).  Twitter demonstrates another model. Users "follow" a topic or other user represented by a "hash tag" or "at tag". Any message containing a tag in a user's follow list regardless of who sent it, is delivered to the user.  These "everyone gets it" systems use what is called a flooding model.

The other common uses for messaging are less visible to the public.  Computers use messaging to create super computers (HPC, or High Performance Computing).  A large computation is broken down into smaller parts which are distributed across hundreds or thousands of computers.  Each of the participating computers runs an "agent" which listens for messages and can run local processes in response. The individual computers receive messages from a controller which instructs them how to process their one little part.  Then they send the result back to the controller in another message.  These distributed systems create animated films, weather reports, airline reservations and results in most sciences.

These computer messaging services have been adapted for yet another task.  They are commonly used now in computer Configuration Management Systems, such as cfengine, Puppet, bcfg2, Chef and others.  In large enterprise computer systems the agent running on the participating computers is designed to update the computer configuration on command.

All of these messaging systems (public human services and computer communications services) have a set of common elements.  All have "publishers" and "subscribers" (senders and receivers).  They also have one or more "message brokers" configured into a mesh or "grid". The message brokers are responsible for listening for new messages and distributing them to the subscribed listeners.

Computer messaging systems add a significant alternative to the "flooding" model where every message goes to every subscriber.  Message channels which use the flooding behavior are called "topics".

Computer messaging also uses a model called a "message queue".  In a message queue, the subscribers indicate the ability to handle a certain kind of message.  The message sender submits a message (or request) to the queue. Each message is delivered to exactly one subscriber who processes the message and responds when the processing is done.  The message provider doesn't know which subscriber will pick up the message and doesn't care so long as each message gets processed.  This is only significant to us because it means we actually have to define two things (a topic and a queue) and not just one in the configuration.

Messaging in Openshift

In the OpenShift service, the OpenShift broker and node are the publisher and subscriber.  The message broker(s) sit between.

OpenShift messaging is two layers deep.  The MCollective service is an abstracted RPC mechanism.  It runs on both ends of the messaging system. It relies on an underlying message passing service to do the message routing and delivery.  I'll be careful always to distinguish between the OpenShift broker (which runs the openshift service) and the message broker (which carries communications between the Openshift broker and the Openshift nodes).

OpenShift only interacts directly with the MCollective service.  It is unaware of what the underlying communications mechanism is.  MCollective can use one of several different message brokers. Since OpenShift doesn't care, you can choose which ever one suits your needs best.  The most common message broker implementations are RabbitMQ and ActiveMQ which use the Stomp protocol.  You can also use message broker which implements the AMQP protocol, such as QPID.  I'm going to use the ActiveMQ message broker service.

This diagram highlights where the messaging service sits in the Openshift Origin Service and indicates the limits of what I'm working here.

The ActiveMQ messaging service

ActiveMQ is a Java based service.  You can find it on github and it will be properly packaged for Fedora 18 and RHEL 6.4. I'm going to assume you can install it with yum.

Like most Java services, ActiveMQ configuration is formatted as XML along with a set of property files. The configuration files reside in /etc/activemq. The primary configuration file is /etc/activemq/activemq.xml.
I'm also going to configure a management interface which uses something called jetty. I'll need to modify the jetty.xml and jetty-realm.properties configuration files.

There are four things that need configuration in the activemq.xml file:
  • Set the (message) broker name that this service will report when someone connects.
  • Create user accounts for access control
  • Create message queues and topics. Assign access permissions to user accounts.
  • Enable protocol listeners
Unlike common public messaging services, you can't add topics, queues or users on-the-fly.  This is largely for security reasons. In a case like Openshift Origin it doesn't matter as we know a priori the topics we want.

ActiveMQ actually provides several baseline configurations for different protocols. Specifically they provide one for the Stomp protocol which is preferred by MCollective. The baseline configuration file for Stomp is called activemq-stomp.xml. I'm going to start configuring ActiveMQ by saving a copy of the default configuration file and replacing it with the Stomp baseline file.

cp /etc/activemq/activemq.xml /etc/activemq/activemq.xml.orig
cp /etc/activemq/activemq-stomp.xml /etc/activemq/activemq.xml

The first change to make to the activemq.xml is to set the (message) brokerName. The default value is "localhost".  We want it to be the fully qualified domain name of the message broker host; "msg1.example.com". This is an sed one liner.

sed -i -e '/<broker/s/brokerName=".*"/brokerName="msg1.example.com"/' /etc/activemq/activemq.xml

The XML schema for ActiveMQ is a bit strange. It requires that the section tags be in alphabetical order.
In yet another case of Word Overloading the authentication and authorization (topic/queue definition) sections are called "plugins". Fortunately, by using the activemq-stomp.xml file as the base, all of the changes we need to insert are confined to a single section delimited by the <plugins> tags.

  <!-- add users for mcollective -->
               <!-- change the username and password -->
               <authenticationUser username="mcollective" password="marionette" groups="mcollective,everyone"/>
               <authenticationUser username="admin" password="msgadminsecret" groups="mcollective,admin,everyone"/>
                   <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
                   <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
                   <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
                   <!-- these maybe should be "openshift" but.... -->
                   <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
                   <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />

This section must be added after the </persistenceAdaptor> close tag and before the <transportConnectors> open tag.

The <authenticationUser /> tags each define a username, password and group memberships for a messaging user. I'm defining two users; admin and mcollective, and adding the mcollective user to an mcollective group.

The <authorizationEntry /> tags define message queues and topics including the permissions and membership. The first three are the admin and control topics. The last two are used by the OpenShift Origin service.

Monitoring and Statistics Interface

The ActiveMQ service offers an HTML and REST interface for monitoring the messaging service. I'm going to enable that so I can use it to check on the status of the messaging service after I have it started. The monitoring service is configured with the jetty.xml and jetty-realm.properties files in the /etc/activemq directory. In addition to enabling the monitoring service, I want to make sure that it is secure. I'm going to restrict network access to the localhost interface and reset the admin password.

In the jetty.xml file I can make the changes with two sed one liners:

sed -i -e '/"authenticate"/s/value=".*"/value="true"/' jetty.xml
sed -i -e '/name="port"/a\ <property name="host" value="" />' jetty.xml

The final change is to reset the admin password for the monitoring service interface.  This is in the jetty-realm.properties file. Each line of the file contains a single user/password entry.  Again, a sed one liner will do the trick:

sed -e '/^admin:/s/: .*,/: adminsecret,/' jetty-realm.properties.orig

Where "adminsecret" is the new password. You pick your own value.

That should be enough to get the ActiveMQ service running and ready for OpenShift Origin. Now I have to turn it on and verify it.

Starting and Verifying the ActiveMQ service

The final steps are to try the service and verify that it is working as needed.

Starting And Enabling ActiveMQ service

The ActiveMQ is enabled and controlled just like any other standard service on RHEL or Fedora:

service activemq status
ActiveMQ Broker is not running.

service activemq start
Starting ActiveMQ Broker...

service activemq status
ActiveMQ Broker is running (18149).

chkconfig activemq on

chkconfig --list activemq
activemq        0:off 1:off 2:on 3:on 4:on 5:on 6:off

Checking the Administrative Service Interface

First, check that the admin user can fetch data from the admin interface.  The curl command below means "request the header from the root page from localhost TCP port 8161 with username 'admin' and password 'adminsecret'".  If the service is running and answering, and if the port, username and password are correct I expect to get a response like this:

curl --head -u admin:adminsecret http://localhost:8161
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 6482
Last-Modified: Wed, 02 May 2012 12:07:14 GMT
Server: Jetty(7.6.1.v20120215)

The next thing to do is to check that admin service is reporting the available queues and topics.  Until each queue or topic has been used, the list will be empty, but at least the service will tell you that.

I'll modify the curl command slightly. Instead of asking for the HTTP header, I'll ask for a specific page:

curl -u admin:adminsecret http://localhost:8161/admin/xml/topics.jsp

I can do the same thing for the queues and subscribers lists by replacing the "topics" in the command aboce with "queues" or "subscribers".

Viewing the Administrative Service in a Web Browser

I want to be able to view the ActiveMQ status console in a web browser.  Since I limited the Jetty service to the localhost interface ( in the jetty.xml file, I need to forward the admin interface port (8161) back to my workstation.

ssh -L8161:localhost:8161 msg1.example.com

When I've logged into the message host like this, I can then browse to http://localhost:8161 and I will see the ActiveMQ Administrative Console.

Verifying the Message Service Listeners

When the ActiveMQ service is running, there should be a listener  bound to port 61613.  I'll check that with the ss command (replacement for netstat)

ss -listening --tcp | grep 61613
0      128                         :::61613                        :::

The last step is to verify that the broker host and nodes will be able to connect to the service. I use the telnet client to test it.  Telnet is often not installed by default but it is an extremely useful tool for testing TCP connections.   It is NOT a recommended tool for logging into servers anymore as the contents are sent in clear text.

From the broker, I telnet to the message host on the STOMP port (61613)

telnet msg1.example.com 61613
Connected to msg1.example.com.
Escape character is '^]'.

Note that the escape character '^]' actually means "hit the ESC key and then the right bracket keys". at the telnet> prompt enter quit and carriage return and telnet will disconnect and exit.


At this point I have a running ActiveMQ service which offers the mcollective topic and queue.  Providers and Subscribers should be able to connect and communicate if they provide the correct username and password.

A note on Security

Encryption in messaging systems can be quite complex.  Depending on the needs of the service the messages may be encrypted end-to-end, or on the connections between the end points and the first message server, or between the message servers or all of these.

The complexity warrants a post of it's own.  The node completed here is not encrypted and sensitive data should not be sent though it over untrusted networks.

A later post will cover encrypting the messaging communications.


Friday, November 16, 2012

OpenShift Back End Services: Data Store (MongoDB)

In the previous post, I listed the set of back end services which underpin an OpenShift Origin service.
In this one I'm going to detail the installation of the first of these back end services and initialize it for OpenShift Origin.

Data Persistance

The OpenShift Origin service needs a backing data store to contain persistent data.  It must keep track of the meta data for user's applications, the available nodes and their capabilities and the ssh public keys provided by the user to allow them git and ssh access to their apps.

Currently OpenShift Origin uses a MongoDB back end for data persistence.  Before you can start building your OpenShift Origin service you need to have a running MongoDB that can be reached by the broker service on the Openshift Origin broker host(s).

There are lots of good resources about administration and use of MongoDB.  If you're going to run an OpenShift Origin service yourself you should keep them handy.  Check the References section at the end of this (and each) post.

In this post I'm going to walk through preparing the Data Storage host.  Except for a couple of commands to create the user accounts and an empty database, there isn't anything really databasey about this procedure.

Information Gathering

From the table in the last post, we have the hostname and IP address of the broker host and the MongoDB server.

We're going to install and configure the MongoDB service to permit the OpenShift Origin broker service to access, read and write the OpenShift database.

The MongoDB service runs on port TCP 27017.

I'm going to create a root account in the MongoDB admin database and enable authentication.  This will allow two layer access control to the data.  I'll create the empty OpenShift Origin database so that it is ready for the broker to connect. I will also create a role user in the OpenShift database for the Openshift Origin  broker to use.  I'm going to create values for those.  You should choose different passwords.

I will also need the IP address of the broker host so that I can use the iptables firewall to limit inbound connections to only that host.

So here's the information we have:

MongoDB Setup Information
FunctionHostnameIP Address
Broker Hostbroker1.example.com192.168.5.11
Data Storage Hostdata1.example.com192.168.5.8
MongoDB TCP Port27017
OpenShift Database Nameopenshift
MongoDB Privileged Userrootdbadminsecret
Openshift Database Useropenshiftdbsecret

Preparing the Base

In each of these setups I assume that the hosts have a base operating system installed and configured.  I'm working with RHEL 6.3 and Fedora 17, but some of the packages are not yet publicly available for those. (November 15 2012) They will be available with the release of Fedora 18 at the end of the month and either from EPEL or from RHEL repositories.

I start with the Base package list, ntpd, policycoreutils-python and what ever packages are needed for central user control.  These are Kerberos 5 and LDAP in my case (krb5-workstation, pam_krb5 and openldap-clients).  For me this works out at between 300 and 425 packages. Right now I have to add a set of yum repositories for the pre-release builds but these should not be neccesary once all of the packages go to release for both RHEL and Fedora.

I also generally bring my base system up to date before beginning any other work.

yum -y update

The Process

There are several steps to preparing MongoDB for the OpenShift Origin service. I have to make sure it will start correctly, that it is secured and that it is initialized so that when the broker first connects the database is ready to accept updates.  I also need to save the information that the broker will need to establish a connection as I'll need that to configure the broker plugin.

The steps look like this:
  • Install the MongoDB server software
  • Enable authentication
  • Add an administrator account
  • Add the empty OpensShift database
  • Add the OpenShift broker user account
  • Add firewall rules
  • Enable listening on external interfaces
  • Enable service restart on boot
Each of these steps is fairly small and the whole thing should be easily scriptable. It also should be fairly simple to script a set of checks to verify that the database service is properly configured and secured. These scripts can be used to check the underlying database service configuration in the event of a problem with the OpenShift service.

Note that I try to perform the steps in such a way that the service is never exposed to an external network until authentication is enable and configure and the firewalls have been established.  This prevents cracking attempts during the (admittedly tiny) window when anonymous access would succeed.

Installing the Software

The first thing to do is to install the mongodb-server software. This will pull in several pre-requisites as well.

yum -y install mongodb-server

Enable Authentication

The MongoDB configuration file is /etc/mongodb.conf. The configuration is a simple space/line delimited key/value pair format. Authentication is off by default. The auth section looks like this:

# Turn on/off security.  Off is currently the default
#noauth = true
#auth = true

The first thing to do is make a copy of the original configuration file in case I mess up.
I need to uncomment the "auth = ..." line. You can do this with an editor, but I generally do this for simple changes with a line editor like sed(1). I'm also going to sneak in one other tuning parameter here that OpenShift wants but isn't related to security or service management: The smallfiles parameter needs to be added to the end of the file and set to true.
cp /etc/mongodb.conf /etc/mongodb.conf.orig
sed -i -e '/^#auth =/auth =/' /etc/mongodb.conf
echo "smallfiles = true" >> /etc/mongodb.conf

Throughout these pages you'll see patterns like that: "save a copy, make a change or two".

Add Accounts and Empty Database.

To add accounts and initialize the database it must be running.  Once it is, I'll use the mongo(1) CLI client tool to make the changes I need.
The mongodb process needs a few seconds to start and begin listening, so there's a sleep after starting the daemon and before trying to access the database.
There are three steps here:

  • Create the administrator (root) account.
  • Create the OpenShift Origin server database
  • Create the OpenShift Origin broker role account.
MongoDB will actually create a database just by being told to use it even if it doesn't exist, so the steps are actually simpler. I'm also going to stop the database service once this is done so that I can open it (carefully) to external network access.

Note that the passwords and the OpenShift Origin database name come from the table of values a the beginning of this post. You need to change the password and you can select a different database name if you wish (you must keep track of it, you'll need it later).

# Start the mongod service (obviously)
service mongod start

# Wait for the service to be ready to listen
sleep 10

# Connect, create root account, authenticate, create database and role account.
mongo admin << EOMONGO
db.addUser('root', 'dbadminsecret');
db.auth('root', 'dbadminsecret');
use openshift;
db.addUser('openshift', 'dbsecret');

# Stop the service again.
service mongod stop

In the past I would have also created a read-only user in the admin database. This would have allowed full db backups without read-write access. Now I should probably create a read-only user in the openshift database to back up just the one database.

Open a small hole in the firewall

By default the mongod process only listens on the localhost interface. Since it also starts with no authentication this is a good thing for security. Now I want to allow the service to listen for outside connections. However I only want to allow connections from the OpenShift Origin broker host. I'll use an iptables entry to restrict inbound connections and then tell the daemon to listen on the external interface.

If you wanted to allow unrestricted inbound connections, you could just use lokkit but we want to restrict access to a single or small set of inbound hosts so we need to configure the IP tables deliberately.

It would be handy to have the iptables(8) man page handy here for reference.

I want to allow inbound connections only from the broker host. I'll have to craft an iptables line which will do that. I have the IP address of the broker host and the TCP port to which mongod listens by default. I'll use those to craft the appropriate rule.

-A INPUT-s -m state --state NEW -m tcp -p tcp --dport 27017 -j ACCEPT

This line means:

"Append an entry to the INPUT queue. Match NEW connections from Accept TCP packets destined for port 27017"

I'd like to add this line to the end of the INPUT queue, but actually not the last line. The last line is the one that says "anything not matched by now, reject it". I want that to stay at the end. I want to insert my allow line just before that.

I could craft a clever sed or awk command to line edit the /etc/sysconfig/iptables file. iptables itself provides me with a better way. I can use the iptables control commands to determine which rule I want to insert at, add the line to the running tables and then have iptables dump the result to a file. I can save that as the new /etc/sysconfig/iptables file so that it will restore my new rule set at system startup.

I can use the iptables command first to list the existing ruleset. I can count the number of rules with wc -l and use expr to subtract one. That becomes the rule number for my insert command.

# Save a copy of the original
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.orig

# Find the index of the next-to-last rule in the INPUT ruleset
INSERT_INDEX=$(expr $(iptables -S INPUT | wc -l) - 1)

# Add a new rule at N-1
iptables -I INPUT $INSERT_INDEX -s -m state --state NEW -m tcp -p tcp --dport 27017 -j ACCEPT

# dump a copy of the current rules and save them for next reboot
iptables-save > /etc/sysconfig/iptables

Listen on external interfaces

Now that the authentication and firewall are in place, I can reconfigure mongod to listen on all interfaces. We're back to the easy stuff.

The bind_ip option in the /etc/mongodb.conf file specifies where the mongod will bind. By default this is The IPv4 convention for "all addresses" is I'll replace the value of bind_ip in /etc/mongodb.conf and I'll be ready to restart the service. This is a simple sed script again.

sed -i -e '/^bind_ip =/s/=.*/=' /etc/mongodb.conf
service mongod start

Access and Security Verification

Once the service is running I want to make attempts to connect to it from at least three different sources.
  1. From the datastore host (localhost) - allowed
  2. From the datastore host (data1.example.com) - rejected
  3. From the broker host - allowed
  4. From an unauthorized host - rejected
I also want to connect both to the admin database and to the openshift on the two that should not be restricted by the firewall.

Note that, because the firewall rule did not include an allow clause for localhost as the source, you can only connect to the database using the localhost interface if you are on the datastore host.

The test commands below consist of an echo command which writes a string to standard output. The string is a single mongodb CLI command.  The string is piped as input to the mongo CLI client.  The argument to the mongo command is the host and database to be opened. 

# admin, root user on localhost
echo 'db.auth("root", "dbadminsecret");' | mongo localhost/admin
MongoDB shell version: 2.2.1
connecting to: localhost/admin

# openshift, openshift user on localhost
echo 'db.auth("openshift", "dbsecret");' | mongo localhost/openshift
MongoDB shell version: 2.2.1
connecting to: localhost/openshift

# admin, root user on external interface
echo 'db.auth("root", "dbadminsecret");' | mongo data1.example.com/admin
MongoDB shell version: 2.2.1
connecting to: data1.example.com/admin
Fri Nov 16 21:28:45 Error: couldn't connect to server data1.example.com:27017 src/mongo/shell/mongo.js:93
exception: connect failed

# openshift, openshift user on external interface.
echo 'db.auth("openshift", "dbsecret");' | mongo data1.example.com/openshift
MongoDB shell version: 2.2.1
connecting to: data1.example.com/openshift
Fri Nov 16 21:28:45 Error: couldn't connect to server data1.example.com:27017 src/mongo/shell/mongo.js:93
exception: connect failed
I would try those two tests on the broker host and some external host to verify access. I should probably add a test to each with invalid user and with valid user but invalid password to be rigorous.

Don't Do This (Without Re-Compiling with SSL)

Having gone through this exercise to try to show how the MongoDB back end can be disentangled from the Openshift Origin service configuration, I have to say: Don't Do It.

The last step should have been to establish an encrypted pipe between the broker host and the datastore database.  It turns out that MongoDB and most NoSQL databases don't think they should be doing encryption, so they don't.  While they taught the benefits of distribution and sharding and other multi-host based behaviors, I really can't recommend allowing MongoDB to play in the street with the other kids.  Both the authentication information and the data go in clear text.

In the MongoDB: The Definitive Guide  from O'Reilly the authors recommend an SSH tunnel if you really need encryption point to point.  I don't find this very satisfying.

The MongoDB documentation web site has a page on using MongoDB with SSL.  It requires recompiling the package with a -ssl option.  The documentation does not give a detailed set of instructions for re-compiling.  You have to work that out for yourself.

If you install the resulting package you then have to mark your yum repository so that it doesn't update the mongodb RPM.  Updates from the stock repositories will destroy your SSL configuration.  When an update is indicated, you'll have to download the new source RPM, rebuild again and manually update the package.

For now, if you don't recompile with SSL I'd suggest that OpenShift servers be restricted to a single broker with a single unreplicated database on a single host.

The exercise remains a good one.  You can follow the steps listed here using localhost for all of the datastore host addresses and you'll see the parts of the setup that are specific to the datastore as distinct from the OpenShift Origin service proper.

Next Up

Next up will be the Messaging service using ActiveMQ.