OpenShift Back End Services: DNS - Dynamic Updates

In the previous post I created an authoritative DNS server for the zone app.example.com running on ns1.example.com. The server will answer external queries and authoritative ones, but it will not yet accept update queries.  I need to add that facility before an OpenShift Origin broker service can use it.

OpenShift and DNS Updates

I don't know if I've actually posted this over and over but it feels like it since I've re-written this post at least 4 times since I started on it a week ago.  I'm going to go over it again on the off chance that someone reading this hasn't read the rest.

OpenShift depends on Dynamic DNS for one of its primary functions: Publishing developer applications.  If you can't make the applications public, OpenShift isn't very useful.

OpenShift publishes applications by creating new DNS records which bind the fully qualifed domain name (or FQDN) of the application to a specific host which contains and runs the application service.  The FQDN can then be used in a URL to allow users to access the application with a web browser.

The OpenShift broker provides a plug-in interface for a Dynamic DNS module. Currently (Nov 27 2012) there is only one published plugin.  This uses the DNS Update protocol defined in RFC 2136 along with a signed transaction defined in RFC 2845 to allow authentication of the updates.

The table below lists the information that the OpenShift Broker needs to communicate with a properly configured Bind DNS server to request updates.

OpenShift Origin Bind DNS Plugin information

Variable	Value	Comment
Dynamic DNS Host IP address	192.168.5.2	ns1.example.com
Dynamic DNS Service port	53/TCP	Default, but configurable
Zone to update (domain_suffix)	app.example.com	apps will go here.
DNSSEC Key Type	HMAC-MD5
DNSSEC Key Name	app.example.com	Arbitrary name
DNSSEC Key Value	A really long string	TSIG size range: 1-512 bits

Any DNS update service would require similar information but it would be tailored to the service update protocol.

I need to generate a DNSSEC signing key for the update queries and then insert the  information into the DNS service configuration so that it will accept updates.

Generating an Update Key

The last value in the table above is missing.  I like to have all the ingredients before I start a recipe.  I'm going to generate that key value before moving on.  I'll need it for the OpenShift broker plugin configuration and for the back end DNS service setup.

I'll use dnssec-keygen to create a pair of DNSSEC key files (public and private).

dnssec-keygen -a HMAC-MD5 -b 256 -n USER app.example.com
Kapp.example.com.+157+45890

This command translates as "Create a 256 bit USER key with the HMAC-MD5 algorithm, named example.com". A USER key is one that is used to authenticate access. The HMAC-MD5 keys can be from 1 to 512 bits. A 256 bit key fits on a single line making it easier to manage in this post.

The output indicates the filename of the resulting key files. This command produces two files:

• Kapp.example.com.+157+45890.key
• Kapp.example.com.+157+45890.private

The HMAC-MD5 key is a symmetric key, so the two files contain the same key value. The ".key" file is a single line and represents a DNS Resource Record. The ".private" file contains the key string and some metadata.

The key isn't actually used as a password.  Rather it is a "signing key" for a Digital Signature which will be attached to the update query.  The signature is generated using the signing key and the contents of the query. It is mathematically unlikely that someone would be able to generate the correct signature without a copy of the signing key. The DNS server generates a signature string with its copy of the key and the update query.  if the signatures match then the DNS server can be confident that the message did come from an authorize update client.

The key files look like this:

cat Kapp.example.com.+157+45890.key
app.example.com. IN KEY 0 3 157 LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=

cat Kexample.com.+157+45890.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=
Bits: AAA=
Created: 20121122011903
Publish: 20121122011903
Activate: 20121122011903

The only part I care about at the moment is the Key: line of the .private file:

LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=

I'll copy that string and save it for later.

Enable DNS Updates

Now that I have an update key and a running authoritative server I can use them to enable DNS updates. I need to provide a copy of the key string to the named and inform it that it should accept updates signed with that key.

I'm going to add some contents to the /var/named/app.example.com.conf file to provide this information.  I'm going to put the key string in its own file and use another include directive to bring it into the configuration.  This way if I need to change the key, I only need to edit a single line file rather than trying to find the key string in a larger configuration.

I can append the key information to the file fairly simply.  The key clause looks like this:

key app.example.com {
  algorithm HMAC-MD5;
  // secret "some long key string in base64 encoding";
  include "app.example.com.secret";
};

Append that to the /var/named/app.example.com.conf file.  The include line indicates that the secret key will be placed in /var/named/app.example.com.secret.

The secret line format is indicated in the key clause comment.  It's a single line containing the word "secret" and the key string in double quotes, terminated by a semicolon.

echo 'secret "LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=" ;' \
> /var/named/app.example.com.secret

Finally I need to inform the named that the app.example.com zone can be updated using that key. I have to add an allow-update option to the zone section. When that's done the config file will look like this:

/var/named/app.example.com.conf

zone "app.example.com" IN {
    type master;
    file "dynamic/app.example.com.db";
    allow-update { key app.example.com ; } ;
};

key app.example.com {
  algorithm HMAC-MD5;
  // secret "some long key string in base64 encoding";
  include "app.example.com.secret";
};

Re check the files once and try restarting the named.

Verify the Named Operation

The first thing to do as always when making a configuration change is check that I haven't broken anything.  I'll do several things to check:

1. Observe the restart results on the CLI
2. Search for named lines at the end of /var/log/messages and scan for errors
3. Run queries for the SOA, NS and A glue records for the app.example.com zone (localhost)
4. Re-run the queries from another host pointing dig at ns1.example.com

These are the same checks that I did when establishing the authoritative server in the previous post.

Verify DNS Update Operations

I'm almost finished now. The final step is to verify that I can add and remove resource records in the app.example.com zone.

The bind-utils package includes the nsupdate utility.  This is a command line tool that can send DNS update queries.

I have the server hostname and IP address.  I have the TSIG key. I know the dynamic zone.  To test dynamic updates I'm going to add a TXT record named "testrecord.app.example.com" with a value "this is a test record". The nsupdate command below expresses that.

nsupdate -k Kexample.com.+157+45890.private
server 192.168.5.2
update add testrecord.app.example.com 1 TXT "this is a test record"
send
quit

When this command completes we should be able to send a query for that name and get an answer:

dig @127.0.0.1 testrecord.app.example.com txt

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @127.0.0.1 testrecord.app.example.com txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18488
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;testrecord.app.example.com. IN TXT

;; ANSWER SECTION:
testrecord.app.example.com. 1 IN TXT "this is a test record"

;; AUTHORITY SECTION:
app.example.com. 30 IN NS ns2.example.com.
app.example.com. 30 IN NS ns1.example.com.

;; ADDITIONAL SECTION:
ns1.example.com. 600 IN A 192.168.5.2
ns2.example.com. 600 IN A 192.168.5.3

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 28 02:20:33 2012
;; MSG SIZE  rcvd: 146

I can also check the named logs in /var/log/messages:

grep named /var/log/messages
...
Nov 28 02:12:46 ns1 named[30675]: client 127.0.0.1#60808: signer "app.example.com" approved
Nov 28 02:12:46 ns1 named[30675]: client 127.0.0.1#60808: updating zone 'app.example.com/IN': adding an RR at 'testrecord.app.example.com' TXT
Nov 28 02:12:46 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112911)

I can see that an update request arrived and the signature checked.  A record was added, and update notifications were sent (or would have been sent) to any secondary servers.

Removing the record again looks very similar:

nsupdate -k bind/Kapp.example.com.+157+13871.private 
server 127.0.0.1
update delete testrecord.app.example.com TXT
send
quit

When I query for the record again I get a negative response now:

dig @127.0.0.1 testrecord.app.example.com txt

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @127.0.0.1 testrecord.app.example.com txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24061
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;testrecord.app.example.com. IN TXT

;; AUTHORITY SECTION:
app.example.com. 10 IN SOA ns1.example.com. hostmaster.example.com. 2011112912 60 15 1800 10

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 28 02:56:01 2012
;; MSG SIZE  rcvd: 95

Note that there is no answer section.  No section.  No answer.

The logs will also show that the record was removed:

grep named /var/log/messages
...
Nov 28 02:50:42 ns1 named[30675]: client 127.0.0.1#59920: signer "app.example.com" approved
Nov 28 02:50:42 ns1 named[30675]: client 127.0.0.1#59920: updating zone 'app.example.com/IN': deleting rrset at 'testrecord.app.example.com' TXT
Nov 28 02:50:42 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112912)

It took four blog posts to get here, but I now have a working DNS server capable of DNS Updates.

The final steps are to create a secondary server on ns2.example.com and to submit the nameserver information to my IT department for delegation.

DNS is ready for Openshift.

References

• RFC 2136 defines the DNS Update protocol.
• RFC 2845 defines signed transactions which allow DNS updates to be authenticated.
• dnssec-keygen Generates TSIG keys suitable for signing DNS updates
• nsupdate is a command line tool for making DNS updates