OpenShift and DNS Updates
I don't know if I've actually posted this over and over but it feels like it since I've re-written this post at least 4 times since I started on it a week ago. I'm going to go over it again on the off chance that someone reading this hasn't read the rest.
OpenShift depends on Dynamic DNS for one of its primary functions: Publishing developer applications. If you can't make the applications public, OpenShift isn't very useful.
OpenShift publishes applications by creating new DNS records which bind the fully qualifed domain name (or FQDN) of the application to a specific host which contains and runs the application service. The FQDN can then be used in a URL to allow users to access the application with a web browser.
The OpenShift broker provides a plug-in interface for a Dynamic DNS module. Currently (Nov 27 2012) there is only one published plugin. This uses the DNS Update protocol defined in RFC 2136 along with a signed transaction defined in RFC 2845 to allow authentication of the updates.
The table below lists the information that the OpenShift Broker needs to communicate with a properly configured Bind DNS server to request updates.
| Variable | Value | Comment |
|---|---|---|
| Dynamic DNS Host IP address | 192.168.5.2 | ns1.example.com |
| Dynamic DNS Service port | 53/TCP | Default, but configurable |
| Zone to update (domain_suffix) | app.example.com | apps will go here. |
| DNSSEC Key Type | HMAC-MD5 | |
| DNSSEC Key Name | app.example.com | Arbitrary name |
| DNSSEC Key Value | A really long string | TSIG size range: 1-512 bits |
Any DNS update service would require similar information but it would be tailored to the service update protocol.
I need to generate a DNSSEC signing key for the update queries and then insert the information into the DNS service configuration so that it will accept updates.
Generating an Update Key
The last value in the table above is missing. I like to have all the ingredients before I start a recipe. I'm going to generate that key value before moving on. I'll need it for the OpenShift broker plugin configuration and for the back end DNS service setup.
I'll use dnssec-keygen to create a pair of DNSSEC key files (public and private).
dnssec-keygen -a HMAC-MD5 -b 256 -n USER app.example.com Kapp.example.com.+157+45890
This command translates as "Create a 256 bit USER key with the HMAC-MD5 algorithm, named example.com". A USER key is one that is used to authenticate access. The HMAC-MD5 keys can be from 1 to 512 bits. A 256 bit key fits on a single line making it easier to manage in this post.
The output indicates the filename of the resulting key files. This command produces two files:
- Kapp.example.com.+157+45890.key
- Kapp.example.com.+157+45890.private
The HMAC-MD5 key is a symmetric key, so the two files contain the same key value. The ".key" file is a single line and represents a DNS Resource Record. The ".private" file contains the key string and some metadata.
The key isn't actually used as a password. Rather it is a "signing key" for a Digital Signature which will be attached to the update query. The signature is generated using the signing key and the contents of the query. It is mathematically unlikely that someone would be able to generate the correct signature without a copy of the signing key. The DNS server generates a signature string with its copy of the key and the update query. if the signatures match then the DNS server can be confident that the message did come from an authorize update client.
The key files look like this:
cat Kapp.example.com.+157+45890.key app.example.com. IN KEY 0 3 157 LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do= cat Kexample.com.+157+45890.private Private-key-format: v1.3 Algorithm: 157 (HMAC_MD5) Key: LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do= Bits: AAA= Created: 20121122011903 Publish: 20121122011903 Activate: 20121122011903
The only part I care about at the moment is the Key: line of the .private file:
LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=I'll copy that string and save it for later.
Enable DNS Updates
Now that I have an update key and a running authoritative server I can use them to enable DNS updates. I need to provide a copy of the key string to the named and inform it that it should accept updates signed with that key.
I'm going to add some contents to the
I can append the key information to the file fairly simply. The key clause looks like this:
Append that to the
The secret line format is indicated in the key clause comment. It's a single line containing the word "secret" and the key string in double quotes, terminated by a semicolon.
Finally I need to inform the named that the app.example.com zone can be updated using that key. I have to add an allow-update option to the zone section. When that's done the config file will look like this:
/var/named/app.example.com.conf
/var/named/app.example.com.conf file to provide this information. I'm going to put the key string in its own file and use another include directive to bring it into the configuration. This way if I need to change the key, I only need to edit a single line file rather than trying to find the key string in a larger configuration.I can append the key information to the file fairly simply. The key clause looks like this:
key app.example.com {
algorithm HMAC-MD5;
// secret "some long key string in base64 encoding";
include "app.example.com.secret";
};
Append that to the
/var/named/app.example.com.conf file. The include line indicates that the secret key will be placed in /var/named/app.example.com.secret.The secret line format is indicated in the key clause comment. It's a single line containing the word "secret" and the key string in double quotes, terminated by a semicolon.
echo 'secret "LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=" ;' \ > /var/named/app.example.com.secret
Finally I need to inform the named that the app.example.com zone can be updated using that key. I have to add an allow-update option to the zone section. When that's done the config file will look like this:
/var/named/app.example.com.conf
zone "app.example.com" IN {
type master;
file "dynamic/app.example.com.db";
allow-update { key app.example.com ; } ;
};
key app.example.com {
algorithm HMAC-MD5;
// secret "some long key string in base64 encoding";
include "app.example.com.secret";
};
Re check the files once and try restarting the named.
Verify the Named Operation
The first thing to do as always when making a configuration change is check that I haven't broken anything. I'll do several things to check:
- Observe the restart results on the CLI
- Search for named lines at the end of /var/log/messages and scan for errors
- Run queries for the SOA, NS and A glue records for the app.example.com zone (localhost)
- Re-run the queries from another host pointing dig at ns1.example.com
These are the same checks that I did when establishing the authoritative server in the previous post.
Verify DNS Update Operations
I'm almost finished now. The final step is to verify that I can add and remove resource records in the app.example.com zone.
The bind-utils package includes the nsupdate utility. This is a command line tool that can send DNS update queries.
I have the server hostname and IP address. I have the TSIG key. I know the dynamic zone. To test dynamic updates I'm going to add a TXT record named "testrecord.app.example.com" with a value "this is a test record". The nsupdate command below expresses that.
When this command completes we should be able to send a query for that name and get an answer:
I can also check the named logs in
I can see that an update request arrived and the signature checked. A record was added, and update notifications were sent (or would have been sent) to any secondary servers.
Removing the record again looks very similar:
When I query for the record again I get a negative response now:
Note that there is no answer section. No section. No answer.
The logs will also show that the record was removed:
It took four blog posts to get here, but I now have a working DNS server capable of DNS Updates.
The final steps are to create a secondary server on ns2.example.com and to submit the nameserver information to my IT department for delegation.
DNS is ready for Openshift.
The bind-utils package includes the nsupdate utility. This is a command line tool that can send DNS update queries.
I have the server hostname and IP address. I have the TSIG key. I know the dynamic zone. To test dynamic updates I'm going to add a TXT record named "testrecord.app.example.com" with a value "this is a test record". The nsupdate command below expresses that.
nsupdate -k Kexample.com.+157+45890.private server 192.168.5.2 update add testrecord.app.example.com 1 TXT "this is a test record" send quit
When this command completes we should be able to send a query for that name and get an answer:
dig @127.0.0.1 testrecord.app.example.com txt ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @127.0.0.1 testrecord.app.example.com txt ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18488 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2 ;; QUESTION SECTION: ;testrecord.app.example.com. IN TXT ;; ANSWER SECTION: testrecord.app.example.com. 1 IN TXT "this is a test record" ;; AUTHORITY SECTION: app.example.com. 30 IN NS ns2.example.com. app.example.com. 30 IN NS ns1.example.com. ;; ADDITIONAL SECTION: ns1.example.com. 600 IN A 192.168.5.2 ns2.example.com. 600 IN A 192.168.5.3 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Wed Nov 28 02:20:33 2012 ;; MSG SIZE rcvd: 146
I can also check the named logs in
/var/log/messages:grep named /var/log/messages ... Nov 28 02:12:46 ns1 named[30675]: client 127.0.0.1#60808: signer "app.example.com" approved Nov 28 02:12:46 ns1 named[30675]: client 127.0.0.1#60808: updating zone 'app.example.com/IN': adding an RR at 'testrecord.app.example.com' TXT Nov 28 02:12:46 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112911)
I can see that an update request arrived and the signature checked. A record was added, and update notifications were sent (or would have been sent) to any secondary servers.
Removing the record again looks very similar:
nsupdate -k bind/Kapp.example.com.+157+13871.private server 127.0.0.1 update delete testrecord.app.example.com TXT send quit
When I query for the record again I get a negative response now:
dig @127.0.0.1 testrecord.app.example.com txt ; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @127.0.0.1 testrecord.app.example.com txt ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24061 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;testrecord.app.example.com. IN TXT ;; AUTHORITY SECTION: app.example.com. 10 IN SOA ns1.example.com. hostmaster.example.com. 2011112912 60 15 1800 10 ;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Wed Nov 28 02:56:01 2012 ;; MSG SIZE rcvd: 95
Note that there is no answer section. No section. No answer.
The logs will also show that the record was removed:
grep named /var/log/messages ... Nov 28 02:50:42 ns1 named[30675]: client 127.0.0.1#59920: signer "app.example.com" approved Nov 28 02:50:42 ns1 named[30675]: client 127.0.0.1#59920: updating zone 'app.example.com/IN': deleting rrset at 'testrecord.app.example.com' TXT Nov 28 02:50:42 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112912)
It took four blog posts to get here, but I now have a working DNS server capable of DNS Updates.
The final steps are to create a secondary server on ns2.example.com and to submit the nameserver information to my IT department for delegation.
DNS is ready for Openshift.
References
- RFC 2136 defines the DNS Update protocol.
- RFC 2845 defines signed transactions which allow DNS updates to be authenticated.
- dnssec-keygen Generates TSIG keys suitable for signing DNS updates
- nsupdate is a command line tool for making DNS updates
