OpenShift on AWS EC2, Part 2: Being Seen (DNS)

OpenShift is, at least in part, a publication system.  Developers create applications and OpenShift tells the world about them.  This means that the very first thing you need to think about when you're considering creating an OpenShift service is "what do I call it?"

I actually created two zones when setting up the DNS for OpenShift.  The servers reside in one zone, and the user applications in another.  The broker service will be making updates to the application zone.  It doesn't seem like a good idea to have the server hostnames in the same zone where a bug or intrusion could alter or delete them. Something like this will do.

• infra.example.org - contains the server hostnames
• app.example.org - contains the application records

Picking a Domain Name (and a Registrar)

In most cases your choice is going to be constrained by what domains you own or have access too. You may need (as I did) to purchase a domain from a domain registrar. Or you will have to have your corporate IT department delegate a domain for you (whether they run it or you do).

When you register or delegate a domain your domain registrar will request a list of name servers which will be serving the content of your domain. Route53 won't tell you the nameservers until you tell them what domain they'll be serving for you. That means that creating a domain, if you don't have one, is a 3 step exchange:

1. Request domain from a registrar
2. Tell Route53 to serve the domain for you
3. Tell your registrar which Route53 nameservers will be providing your domain

These steps will happen so rarely that I haven't bothered to script them.  I just use the web interface for each step.

NOTE: there are technical differences between a zone and a domain but I'm going to treat them as synonyms for this process. When you're registering, it's called a domain. When you're going to change the contents it's called a zone.

Each registrar will have a different means for you to set your domain's nameserver records. You'll have to look them up yourself. If you're getting a domain delegated from your corporate IT department you'll have to give them the list of Route53 nameservers so that they can install the "glue records" into their service.

So, pick your Registrar, search for an available domain, request, register, and pay. Then head over to the AWS Route53 console.

Adding a zone to Route53

On the web interface, click "Create Hosted Zone" in the top tool bar. You'll see this dialog on the right side.

Fill in the values for your new domain and a comment, if you wish. Then click "Create Hosted Zone" at the bottom of the dialog and Route53 will create your zone and assign a set of nameservers.

Make a note of the "Delegation Set". This is the set of nameservers which you need to provide to your domain registrar. The registrar will provide some place to enter the nameserver list and then they will add the glue records to the top-level domain.

Make a note as well of the "Hosted Zone ID". That's what you will use to select the zone to update when you send requests to AWS Route53.

When the domain registrar completes adding the Route53 nameservers it's time to come back to the thor CLI tools installed in part one.

Viewing the Route53 DNS information

You certainly can view the DNS information on the Route53 console. If you've set up the AWS CLI tools indicated in the previous post you can also view them on the CLI.

NOTE: If you haven't followed the previous post, you should before you continue here.

First list the zones you have registered.

thor route53:zone:list
task: route53:zone:list
id: <YOURZONEID> name: app.example.org. records: 3

Now you can list the records in the zone (indicating the zone by name)

thor route53:record:list app.example.org
task: route53:record:list app.example.org
looking for zone id <YOURZONEID>
example.org. NS
  ns-131.awsdns-16.com.
  ns-860.awsdns-43.net.
  ns-2023.awsdns-60.co.uk.
  ns-1076.awsdns-06.org.
app.example.org. SOA
  ns-131.awsdns-16.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

Adding a DNS Record

The real goal in all of this is that the OpenShift broker must be able to add and remove records for applications. OpenShift uses the aws-sdk rubygem.  The Thor tasks also use that gem.  You can call them from the command line or use them to compose more complex operations.

OpenShift currently uses CNAME records to publish applications rather than A records. This is largely to allow for rapid re-naming or re-numbering of nodes within AWS. The use of CNAME records (which are aliases to another FQDN or fully qualified domain name means that the node which hosts the applications can be renumbered without the need to update every DNS record for every application.  If bulk updates of DNS are not expensive, I believe that OpenShift could use A records, though it could require significant recoding.

To verify your DNS domain has been properly configured, add a CNAME record.  Thor provides a standard help option for each query.

thor help route53:record:create
Usage:
  thor route53:record:create ZONE NAME TYPE VALUE

Options:
  [--ttl=N]    
               # Default: 300
  [--verbose]  
  [--wait]     

create a new resource record

From this you can craft a command.  This example includes the --wait and --verbose options so that you can observe the process.  Without the --wait option, the task will complete and return, but there will be a propagation delay before the name will resolve.  With the --wait option, the task polls the Route53 service until it reports that the DNS services have synched.

thor route53:record create app.example.org test1 CNAME test2.app.example.org --verbose --wait
task: route53:record:create app.example.org test1 CNAME test2.infra.example.org
update record = {:comment=>"add CNAME record test1.app.example.org", :changes=>[{:action=>"CREATE", :resource_record_set=>{:name=>"test1.app.example.org", :type=>"CNAME", :ttl=>300, :resource_records=>[{:value=>"test2.infra.example.org"}]}}]}
response = {:change_info=>{:id=>"/change/C2VQAFRSE6OXMY", :status=>"PENDING", :submitted_at=>2013-05-27 00:24:19 UTC, :comment=>"add CNAME record test1.app.example.org"}}
1) change id: /change/C2VQAFRSE6OXMY, status: UNKNOWN - sleeping 5
2) change id: /change/C2VQAFRSE6OXMY, status: PENDING - sleeping 5
3) change id: /change/C2VQAFRSE6OXMY, status: PENDING - sleeping 5
4) change id: /change/C2VQAFRSE6OXMY, status: PENDING - sleeping 5
5) change id: /change/C2VQAFRSE6OXMY, status: PENDING - sleeping 5

When this command completes the new record should resolve:

host -t cname test1.app.example.org 
test1.app.example.org is an alias for test2.infra.example.org.

Also, now if you list the zone records with thor route53:record list app.example.org you'll see the new CNAME record.

Deleting a DNS record

Deleting a DNS record is nearly identical to adding one.  To insure that you are deleting the correct record the delete task requires the same complete inputs as the create task.

thor route53:record delete app.example.org test1 CNAME test2.infra.example.org --verbose --wait
task: route53:record:delete app.example.org CNAME test1
update record = {:comment=>"delete CNAME record test1.app.example.org", :changes=>[{:action=>"DELETE", :resource_record_set=>{:name=>"test1.app.example.org", :type=>"CNAME", :ttl=>300, :resource_records=>[{:value=>"test2.infra.example.org"}]}}]}
response = {:change_info=>{:id=>"/change/C3ORAEV7FTLPBJ", :status=>"PENDING", :submitted_at=>2013-05-27 00:58:25 UTC, :comment=>"delete CNAME record test1.app.example.org"}}
1) change id: /change/C3ORAEV7FTLPBJ, status: UNKNOWN - sleeping 5
2) change id: /change/C3ORAEV7FTLPBJ, status: PENDING - sleeping 5
3) change id: /change/C3ORAEV7FTLPBJ, status: PENDING - sleeping 5
4) change id: /change/C3ORAEV7FTLPBJ, status: PENDING - sleeping 5
5) change id: /change/C3ORAEV7FTLPBJ, status: PENDING - sleeping 5

Again, with the --verbose and --wait options, the task will not complete until the DNS change has propagated. When it completes, the name will no longer resolve.

host -t cname test1.app.example.org
Host test1.app.example.org not found: 3(NXDOMAIN)

Summary

Now that we've registered a domain, and arranged to have it served by Route53, we can add and remove names.  When we configure OpenShift, it will be able to publish new application records.

Next Time

We still have to create hosts to run the OpenShift service.  On AWS that means creating instances , virtual machines in Amazon's cloud.  Amazon applies some fairly restrictive nework level packet filtering.  They use a feature called a securitygroup to define the filtering rules.  In the next post, I'll discuss how to create and manage new securitygroups, and what groups we'll need to allow OpenShift to operate.

Resources

• AWS Route53 DNS management console - https://console.aws.amazon.com/route53/home
• ICANN list of Accredited Registrars - http://www.icann.org/registrar-reports/accredited-list.html
• DNS Zones - https://en.wikipedia.org/wiki/DNS_zone

Verifying the DNS Plugin using Rails Console

Each of the OpenShift broker plugins provides an interface implementation class for the  plugin's abstract behavior.   In practical terms this means that I can fire up the rails console, create an instance of the plugin class and then use it to manipulate the service behind the plugin.

Since the DNS plugin has the simplest interface and Bind has the cleanest service logs, I'm going to demonstrate with that.  The technique is applicable to the other back-end plugin services.

Preparing Logging

To make life easy I'm going to configure logging on the DNS server host so that the logs from the named service are written to their own file.

A one line file in /etc/rsyslog.d will do the trick:

if $programname == 'named' then /var/log/named.log

Write that into /etc/rsyslog.d/00_named.conf and restart the rsyslog service.  Then restart the named service and check that the logs are appearing in the right place.

If I didn't filter out the named logs, I could still use grep on /var/log/messages to extract them.

Configuring the Bind DNS Plugin

As indicated in previous posts, the OpenShift DNS plugin is enabled by placing a file in /etc/openshift/plugins.d with the configuration information for the plugin.  The name of the file must be the name of the rubygem which implements the plugin with the suffix .conf. The Bind plugin is configured like this:

/etc/openshift/plugins.d/openshift-origin-dns-bind.conf



BIND_SERVER="192.168.5.11"
BIND_PORT=53
BIND_KEYNAME="app.example.com"
BIND_KEYVALUE="put-your-hmac-md5-key-here"
BIND_ZONE="app.example.com"

When the Rails application starts, it will import a plugin module for each .conf file and will set the config file values.

The Rails Console

Ruby on Rails has an interactive testing environment.  It it started by invoking rails console from the root directory of the application.  If I start the rails console at the top of the broker application I should be able to instantiate and work with the plugin objects.

The rails console command runs irb to offer a means of manual testing.  In addition to the ordinary ruby script environment it imports the Rails application environment which resides in the current working directory. Among other things, it processes the Gemfile which, in the case of the OpenShift broker, will load any plugin gems and initialize them. I'm going to use the Rails console to directly poke at the back end service objects.

I'm going to go to the broker application directory.  Then I'll check that bundler confirms the presence of all of the required gems.  Then I'll start the Rails console and check the plugin objects manually.

cd /var/www/openshift/broker
bundle --local
....
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
rails console
Loading production environment (Rails 3.0.13)
irb(main):001:0> 

The last line above is the Rails console prompt.

Creating a DnsService Object

The OpenShift::DnsService class is a factory class for the DNS plugin modules. It also contains an interface definition for the plugin, though Ruby and Rails don't seem to be much into formal interface specification and implementation.  The plugin interface definitions reside in the openshift-origin-controller rubygem:

https://github.com/openshift/origin-server/tree/master/controller/lib/openshift

The factory classes provide two methods by convention: provider=() sets the actual class which implements the required interface and instance() is the factory method, returning an instance of the implementing class.  They also have a private instance variable which will contain a reference to the instantiating class.  When the plugins are loaded, a reference to the instantiating class is set into the factory class.

Once the broker application is loaded using the rails console I should be able to create and work with instances of the DnsService implementation.

The first step is to check that the factory class is indeed loaded and has the right provider set. Since I can just type at the irb prompt it's easy to see what's there.

irb(main):001:0> OpenShift::DnsService
=> OpenShift::DnsService
irb(main):002:0> d = OpenShift::DnsService.instance
=> #<OpenShift::BindPlugin:0x7f540dfb9ee8 @zone="app.example.com",
 @src_port=0, @server="192.168.5.2",
 @keyvalue="GwhJNLZPghbpTya2M6N+lvcLmBQx6TYbuH7j6TPyetE=",
 @port=53, @keyname="app.example.com",
 @domain_suffix="app.example.com">

Note that the class is OpenShift::BindPlugin and the instance variables match the values I set in the plugin configuration file. I now have a variable d which refers to an instance of the DNS plugin class.

The DnsService Interface

The DNS plugin interface is the simplest of the plugins.  It contains just four methods:

• register_application(app_name, namespace, public_hostname)
• deregister_application(app_name, namespace)
• modify_application(app_name, namespace, public_hostname)
• publish()

All but the last will have a side-effect which I can check by observing the named service logs and by querying the DNS service itself.

Note that the publish() method is not included in the list with side-effects.  publish() is always called at the end of a set of change calls.  It is there to accommodate batch update processing.  Third party DNS services which use use web interfaces may require batch processing. The OpenShift::BindPlugin submits changes instantly.

Change and Check

The process of testing now will consist of three repeated steps:

1. Make a change
2. Check the DNS server logs
3. Check the DNS server response

I will repeat the steps once for each method. (though I'll only show a couple of samples here)

The logs are time-stamped.  To make it easier to find the right log entry, I'll check the time sync of the broker and DNS server hosts, and then check the time just before issuing each update command.

First I check the date and add an application record.  An application record is a DNS CNAME record which is an alias for the node which contains the application. Here goes:

irb(main):003:0> `date`
=> "Wed Dec  5 15:25:59 GMT 2012\n"
irb(main):002:0> d.register_application "testapp1", "testns1", "node1.example.com"
=> ;; Answer received from 192.168.5.11 (129 bytes)
;;
;; Security Level : UNCHECKED
;; HEADER SECTION
;; id = 25286
;; qr = true    opcode = Update    rcode = NOERROR
;; zocount = 1  prcount = 0  upcount = 0  adcount = 1

OPT pseudo-record : payloadsize 4096, xrcode 0, version 0, flags 32768

;; ZONE SECTION (1  record)
;; app.example.com. IN SOA

The register_application() method returns the Dnsruby::Message returned from the DNS server.  A little digging should indicate that the update was successful.

Next I'll examine the named service log on the DNS server host.

tail /var/log/named.log
...
Dec  5 15:26:41 ns1 named[11178]: client 10.16.137.216#54040/key app.example.com: signer "app.example.com" approved
Dec  5 15:26:41 ns1 named[11178]: client 10.16.137.216#54040/key app.example.com: updating zone 'app.example.com/IN': adding an RR at 'testapp1-testns1.app.example.com' CNAME

Finally, I'll check that the server is answering queries for that name:

dig @ns1.example.com testapp1-testns1.app.example.com CNAME

; <<>> DiG 9.9.2-rl.028.23-P1-RedHat-9.9.2-8.P1.fc18 <<>> @ns1.example.com testapp1-testns1.example.com CNAME
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 10884
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;testapp1-testns1.example.com. IN CNAME

;; AUTHORITY SECTION:
example.com.  10 IN SOA ns1.example.com. hostmaster.example.com. 2011112904 60 15 1800 10

;; Query time: 3 msec
;; SERVER: 192.168.1.11#53(192.168.1.11)
;; WHEN: Thu Dec  5 15:28:41
;; MSG SIZE  rcvd: 108

That's sufficient to confirm that the DNS Bind plugin configuration is correct and that updates are working. In a real case I'd go on and check each of the operations. for Now I'll just delete the test record and go on.

d.deregister_application "testapp1", "testns1"
=> ;; Answer received from 192.168.5.11 (129 bytes)
;;
;; Security Level : UNCHECKED
;; HEADER SECTION
;; id = 26362
;; qr = true    opcode = Update    rcode = NOERROR
;; zocount = 1  prcount = 0  upcount = 0  adcount = 1

OPT pseudo-record : payloadsize 4096, xrcode 0, version 0, flags 32768

;; ZONE SECTION (1  record)
;; app.example.com. IN SOA

dig @ns1.example.com testapp1-testns1.app.example.com CNAME
; <<>> DiG 9.9.2-rl.028.23-P1-RedHat-9.9.2-8.P1.fc18 <<>> @ns1.example.com testapp1-testns1.example.com CNAME
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 50598
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;testapp1-testns1.example.com. IN CNAME

;; AUTHORITY SECTION:
example.com.  10 IN SOA ns1.example.com. hostmaster.example.com. 2011112904 60 15 1800 10

;; Query time: 3 msec
;; SERVER: 192.168.5.11#53(192.168.5.11)
;; WHEN: Thu Dec  5 15:31:20
;; MSG SIZE  rcvd: 108

This is what a negative response looks like. There's a question section but no answer section.
Things are back where I started and I can move on to the next test.

Resources

• Rails console
• irb - Interactive Ruby

• OpenShift DNS plugin class/interface
• OpenShift Bind DNS plugin class/implementation

• dig(1) man page
• rsyslog home page
• rsyslogd man page

OpenShift Back End Services: DNS - Dynamic Updates

In the previous post I created an authoritative DNS server for the zone app.example.com running on ns1.example.com. The server will answer external queries and authoritative ones, but it will not yet accept update queries.  I need to add that facility before an OpenShift Origin broker service can use it.

OpenShift and DNS Updates

I don't know if I've actually posted this over and over but it feels like it since I've re-written this post at least 4 times since I started on it a week ago.  I'm going to go over it again on the off chance that someone reading this hasn't read the rest.

OpenShift depends on Dynamic DNS for one of its primary functions: Publishing developer applications.  If you can't make the applications public, OpenShift isn't very useful.

OpenShift publishes applications by creating new DNS records which bind the fully qualifed domain name (or FQDN) of the application to a specific host which contains and runs the application service.  The FQDN can then be used in a URL to allow users to access the application with a web browser.

The OpenShift broker provides a plug-in interface for a Dynamic DNS module. Currently (Nov 27 2012) there is only one published plugin.  This uses the DNS Update protocol defined in RFC 2136 along with a signed transaction defined in RFC 2845 to allow authentication of the updates.

The table below lists the information that the OpenShift Broker needs to communicate with a properly configured Bind DNS server to request updates.

OpenShift Origin Bind DNS Plugin information

Variable	Value	Comment
Dynamic DNS Host IP address	192.168.5.2	ns1.example.com
Dynamic DNS Service port	53/TCP	Default, but configurable
Zone to update (domain_suffix)	app.example.com	apps will go here.
DNSSEC Key Type	HMAC-MD5
DNSSEC Key Name	app.example.com	Arbitrary name
DNSSEC Key Value	A really long string	TSIG size range: 1-512 bits

Any DNS update service would require similar information but it would be tailored to the service update protocol.

I need to generate a DNSSEC signing key for the update queries and then insert the  information into the DNS service configuration so that it will accept updates.

Generating an Update Key

The last value in the table above is missing.  I like to have all the ingredients before I start a recipe.  I'm going to generate that key value before moving on.  I'll need it for the OpenShift broker plugin configuration and for the back end DNS service setup.

I'll use dnssec-keygen to create a pair of DNSSEC key files (public and private).

dnssec-keygen -a HMAC-MD5 -b 256 -n USER app.example.com
Kapp.example.com.+157+45890

This command translates as "Create a 256 bit USER key with the HMAC-MD5 algorithm, named example.com". A USER key is one that is used to authenticate access. The HMAC-MD5 keys can be from 1 to 512 bits. A 256 bit key fits on a single line making it easier to manage in this post.

The output indicates the filename of the resulting key files. This command produces two files:

• Kapp.example.com.+157+45890.key
• Kapp.example.com.+157+45890.private

The HMAC-MD5 key is a symmetric key, so the two files contain the same key value. The ".key" file is a single line and represents a DNS Resource Record. The ".private" file contains the key string and some metadata.

The key isn't actually used as a password.  Rather it is a "signing key" for a Digital Signature which will be attached to the update query.  The signature is generated using the signing key and the contents of the query. It is mathematically unlikely that someone would be able to generate the correct signature without a copy of the signing key. The DNS server generates a signature string with its copy of the key and the update query.  if the signatures match then the DNS server can be confident that the message did come from an authorize update client.

The key files look like this:

cat Kapp.example.com.+157+45890.key
app.example.com. IN KEY 0 3 157 LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=

cat Kexample.com.+157+45890.private
Private-key-format: v1.3
Algorithm: 157 (HMAC_MD5)
Key: LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=
Bits: AAA=
Created: 20121122011903
Publish: 20121122011903
Activate: 20121122011903

The only part I care about at the moment is the Key: line of the .private file:

LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=

I'll copy that string and save it for later.

Enable DNS Updates

Now that I have an update key and a running authoritative server I can use them to enable DNS updates. I need to provide a copy of the key string to the named and inform it that it should accept updates signed with that key.

I'm going to add some contents to the /var/named/app.example.com.conf file to provide this information.  I'm going to put the key string in its own file and use another include directive to bring it into the configuration.  This way if I need to change the key, I only need to edit a single line file rather than trying to find the key string in a larger configuration.

I can append the key information to the file fairly simply.  The key clause looks like this:

key app.example.com {
  algorithm HMAC-MD5;
  // secret "some long key string in base64 encoding";
  include "app.example.com.secret";
};

Append that to the /var/named/app.example.com.conf file.  The include line indicates that the secret key will be placed in /var/named/app.example.com.secret.

The secret line format is indicated in the key clause comment.  It's a single line containing the word "secret" and the key string in double quotes, terminated by a semicolon.

echo 'secret "LHKu/QNeSikkf1kob7irn816/9shxtD++mMTPYc4/do=" ;' \
> /var/named/app.example.com.secret

Finally I need to inform the named that the app.example.com zone can be updated using that key. I have to add an allow-update option to the zone section. When that's done the config file will look like this:

/var/named/app.example.com.conf

zone "app.example.com" IN {
    type master;
    file "dynamic/app.example.com.db";
    allow-update { key app.example.com ; } ;
};

key app.example.com {
  algorithm HMAC-MD5;
  // secret "some long key string in base64 encoding";
  include "app.example.com.secret";
};

Re check the files once and try restarting the named.

Verify the Named Operation

The first thing to do as always when making a configuration change is check that I haven't broken anything.  I'll do several things to check:

1. Observe the restart results on the CLI
2. Search for named lines at the end of /var/log/messages and scan for errors
3. Run queries for the SOA, NS and A glue records for the app.example.com zone (localhost)
4. Re-run the queries from another host pointing dig at ns1.example.com

These are the same checks that I did when establishing the authoritative server in the previous post.

Verify DNS Update Operations

I'm almost finished now. The final step is to verify that I can add and remove resource records in the app.example.com zone.

The bind-utils package includes the nsupdate utility.  This is a command line tool that can send DNS update queries.

I have the server hostname and IP address.  I have the TSIG key. I know the dynamic zone.  To test dynamic updates I'm going to add a TXT record named "testrecord.app.example.com" with a value "this is a test record". The nsupdate command below expresses that.

nsupdate -k Kexample.com.+157+45890.private
server 192.168.5.2
update add testrecord.app.example.com 1 TXT "this is a test record"
send
quit

When this command completes we should be able to send a query for that name and get an answer:

dig @127.0.0.1 testrecord.app.example.com txt

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @127.0.0.1 testrecord.app.example.com txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18488
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;testrecord.app.example.com. IN TXT

;; ANSWER SECTION:
testrecord.app.example.com. 1 IN TXT "this is a test record"

;; AUTHORITY SECTION:
app.example.com. 30 IN NS ns2.example.com.
app.example.com. 30 IN NS ns1.example.com.

;; ADDITIONAL SECTION:
ns1.example.com. 600 IN A 192.168.5.2
ns2.example.com. 600 IN A 192.168.5.3

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 28 02:20:33 2012
;; MSG SIZE  rcvd: 146

I can also check the named logs in /var/log/messages:

grep named /var/log/messages
...
Nov 28 02:12:46 ns1 named[30675]: client 127.0.0.1#60808: signer "app.example.com" approved
Nov 28 02:12:46 ns1 named[30675]: client 127.0.0.1#60808: updating zone 'app.example.com/IN': adding an RR at 'testrecord.app.example.com' TXT
Nov 28 02:12:46 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112911)

I can see that an update request arrived and the signature checked.  A record was added, and update notifications were sent (or would have been sent) to any secondary servers.

Removing the record again looks very similar:

nsupdate -k bind/Kapp.example.com.+157+13871.private 
server 127.0.0.1
update delete testrecord.app.example.com TXT
send
quit

When I query for the record again I get a negative response now:

dig @127.0.0.1 testrecord.app.example.com txt

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @127.0.0.1 testrecord.app.example.com txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 24061
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;testrecord.app.example.com. IN TXT

;; AUTHORITY SECTION:
app.example.com. 10 IN SOA ns1.example.com. hostmaster.example.com. 2011112912 60 15 1800 10

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 28 02:56:01 2012
;; MSG SIZE  rcvd: 95

Note that there is no answer section.  No section.  No answer.

The logs will also show that the record was removed:

grep named /var/log/messages
...
Nov 28 02:50:42 ns1 named[30675]: client 127.0.0.1#59920: signer "app.example.com" approved
Nov 28 02:50:42 ns1 named[30675]: client 127.0.0.1#59920: updating zone 'app.example.com/IN': deleting rrset at 'testrecord.app.example.com' TXT
Nov 28 02:50:42 ns1 named[30675]: zone app.example.com/IN: sending notifies (serial 2011112912)

It took four blog posts to get here, but I now have a working DNS server capable of DNS Updates.

The final steps are to create a secondary server on ns2.example.com and to submit the nameserver information to my IT department for delegation.

DNS is ready for Openshift.

References

• RFC 2136 defines the DNS Update protocol.
• RFC 2845 defines signed transactions which allow DNS updates to be authenticated.
• dnssec-keygen Generates TSIG keys suitable for signing DNS updates
• nsupdate is a command line tool for making DNS updates

Openshift Back End Services: DNS - An Authoritative Server

OpenShift Origin dynamic DNS requires an authoritative zone for the application records.  So the first step to creating the DNS back end service is to establish that authoritative zone.  When I'm sure it will serve records then I'll add the dynamic update feature to it.

There are a number of good resources on setting up DNS on Linux.  I'll post a few links to them in the Resources.  I've even written part of one, but each time I do this I see it a little differently and (I think) improve my understanding.  So I'm going to do it again.

Ingredients

I'm going to try to create a proper (ready to be delegated) zone.

Here's the list of information I'll need to set it up:

DNS Service Configuration Variables

Variable	Value	Comments
Primary Nameserver
IP Address	192.168.5.2
Hostname	ns1.example.com	Not in the app domain
Secondary Nameserver
IP Address	192.168.5.3
Hostname	ns2.example.com	Not in the app domain
Update Configuration
Application Zone	app.example.com	Not the top level
Application Zone Key Name	app.example.com	Arbitrary name
Application Zone Key Type	HMAC-MD5
Application Zone Key Size	64 bits	512 bits in real life
Application Zone Key Value	A Base64 encoded string	Generated by dnssec-keygen

I'll be setting up both a primary and secondary server for the zone.  The update configuration information won't be needed here except to establish a static zone that will be made dynamic later.

The operations listed below will all be performed on ns1.example.com.

Bind on Linux

Installing ISC Bind 9 on most modern Linux distributions is pretty easy. It's an old and well worn tool.  For RPM based distributions you can just use yum. I generally install the bind-utils package as well just so they're handy.

yum install bind bind-utils

The DNS service binary is called named.  When I refer to Bind, I'll be talking about the software in general. When I refer to named I'll be referring to the daemon and its configuration files.

Once the package is installed there are a number of small configuration changes that are needed to enable and verify the caching service.

• Enable listener ports
• Enable rndc
• Enable start on boot
• Start the named service
• Confirm caching service
• Confirm rndc controls

Named Configuration and Management

The named service follows a traditional layout model on Linux (It's actually probably one of the canonical services).  Before I start making changes I want to

Named Configuration Files

The primary service configuration file is /etc/named.conf.  Additional configuration information and service data reside in the /var/named directory.  The named daemon logs to /var/log/messages through syslog.

• /etc/named.conf - primary configuration file
• /var/named - additional configuration and data
• /var/log/messages - service logging

You can filter for named related log messages in /var/log/messages by grepping for (wait for it!) "named".

ISC provides a complete reference of the named configuration options on their site.

The named service adds one non-traditional feature.  It has a tool called rndc which might stand for "remote name daemon controller".  The rndc tool use used locally by the service command to provide status for the named service.  It can also be used to adjust the logging level, to dump the current zones to a file, or to force a zone reload  without restarting the daemon.

rndc does require some additional setup.  It is not enabled by default as it requires an authentication key.  A default key would present a security risk.  The bind package provides rndc-confgen to help set up the rndc access to the local named. This command produces a required key file which will reside at /etc/rndc.key.

• /etc/rndc.key - Remote name daemon control access key

rndc also requires an addition to the /etc/named.conf file. I'll do that just before I try starting the service.

Before making any change to a configuration file I copy the original and name the new file <filename>.orig. For example, /etc/named.conf would become /etc/named.conf.orig. This way I can track my changes and revert them to the initial values if I need to.

Enable Listeners

The initial configuration of /etc/named.conf restricts queries and updates to the local IP interfaces (IPv4 127.0.0.1 and IPv6 ::1).  Since I want to allow basically anyone to find out about my zone, I have to open this up.  There are three configuration lines that control listeners and query access:

• listen-on
• listen-on-v6
• allow-query

Each of these options takes a semi-colon (;)delimited list of listeners.  The list is encapsulated in a paired set of curly-braces ( {} ). Since this will be a public service, I just have to replace the appropriate localhost address entries with the keyword "any".

/etc/named.conf

...
options {
 listen-on port 53 { any ; };
 listen-on-v6 port 53 { any; };
 directory  "/var/named";
 dump-file  "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
 allow-query     { any; };
 recursion yes;
...

If you want to be tricksy a sed one-liner (and a safety copy) will do the trick if you're starting from the default:

cp /etc/named.conf /etc/named.conf.orig
sed -i -e 's/127.0.0.1\|::1\|localhost/any/' /etc/named.conf

Enable rndc

rndc is the Remote Name Daemon Control program. It's now used for all communications and control to the named on a bind server host. rndc does allow you to securely control a name server daemon remotely, but we won't be using it that way. rndc is also the program that gets and reports the status information when you run service named status so it's nice to have it configured.

rndc comes with a nice little configuration tool to help: rndc-confgen. rndc-confgen creates a unique rndc access key and places the rncd configuration in /etc/rndc.key for you.

rndc-keygen -a

This creates a new file named /etc/rndc.key which contains the access key for the named process. Both the named and the rndc command must have access to this key. rndc uses the /etc/rndc.key file by default. named must be configured for it.

If the rndc-keygen command hangs it is because there is not enough entropy (randomness) on the system. I could Log onto another window and type random commands for a bit and it would complete. If I'm impatient I could run it with -r /dev/urandom which will always complete, but which may be less secure because it will not block waiting for enough randomness to generate a good key. I often do that for lab systems.

rndc-keygen does not set the SELinux context for the key file.  It also does not set the ownership and permissions so that the named can read it. restorecon will do it for me though.

restorecon -v /etc/rndc.key
chown root:named /etc/rndc.key
chmod 640 /etc/rndc.key

Now that we have a key, we have to tell the named to use it. Append the section below to the bottom of the /etc/named.conf file.

// enable service controls via rndc
// use the default rndc key
include "/etc/rndc.key";

controls {
        inet 127.0.0.1 port 953
        allow { 127.0.0.1; } keys { "rndc-key"; };
};

Verifying a caching DNS server

With this configuration I have a caching DNS server.  I need to check the operation now before going ahead to add a new zone.

I use the expected tools to start the named service and get status:

service named start
Starting named:                                            [  OK  ]

service named status
version: 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5
CPUs found: 16
worker threads: 16
number of zones: 19
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 0/0/1000
tcp clients: 0/100
server is up and running
named (pid  31010) is running...

This indicates that the server started and that it is responding to rndc queries. If the daemon does not start or if I got errors from the status query, I'd check /var/log/messages for error messages.

Next we want to verify that the server is actually answering queries. I use dig or host to check. host has a simpler interface and output but for this I want the verbosity of dig to help diagnose if there are any problems. I have to try from two different locations: localhost and "somewhere else". I want first to verify that the service is running and answering, and second that it is accessable from outside itself. I'll only show the localhost queries, but the remote ones are identical.

dig @127.0.0.1 www.example.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @127.0.0.1 www.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29408
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 4

;; QUESTION SECTION:
;www.example.com.  IN A

;; ANSWER SECTION:
www.example.com. 172789 IN A 192.0.43.10

;; AUTHORITY SECTION:
example.com.  172788 IN NS a.iana-servers.net.
example.com.  172788 IN NS b.iana-servers.net.

;; ADDITIONAL SECTION:
a.iana-servers.net. 172788 IN A 199.43.132.53
a.iana-servers.net. 172788 IN AAAA 2001:500:8c::53
b.iana-servers.net. 172788 IN A 199.43.133.53
b.iana-servers.net. 172788 IN AAAA 2001:500:8d::53

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 21 20:42:21 2012
;; MSG SIZE  rcvd: 185

This is actually the real right answer.  Since RFC 2606 reserves the example.com domain, the IANA also  serves that domain.  When I build my test server I'll override that.  If I were building a real OpenShift Origin service I'd get a properly delegated sub-domain of my organization or get my own domain from a registrar.

Take a moment to look at that output. It's meaningful. The ANSWER SECTION contains the only response, an A record. The AUTHORITY SECTION lists the servers which are the designated sources for the content in the example.com zone. It contains two NS records. NS record values are fully qualified domain names (FQDN). That means those names don't have some implied suffix. They end with a dot (.) which anchors them to the root of the DNS. The ADDITIONAL SECTION provides IP address resolution for the NS record FQDNs. The last section indicates that the answer came from the IPv4 localhost address and gives the date/time stamp.

So this answer tells you not only that the IP address for www.example.com is 192.0.43.10 but where the answer came from.

I'll do that again from some other host and set the server address to the public IP address of my nameserver host.

Zone Configuration

Now that I have a caching server running, it's time to start adding some content. The /etc/named.conf file syntax has a directive to include another file in line. Rather than putting the entire configuration section in the master configuration file, I'll put my configuration information in another file and include it. That just requires appending one line to /etc/named.conf

echo 'include "app.example.com.conf" ;' >> /etc/named.conf

Other than the /etc/named.conf file, all of the named configuration files reside in /var/named/. That's the default location for relative path names in the /etc/named.conf as well. I'll create a configuration file fragment there. Since I'm creating the app.example.com domain I'll call the file /var/named/app.example.com.conf.

zone "app.example.com" IN {
    type master;
    file "dynamic/app.example.com.db";
};

The directory option in the /etc/named.conf file determines the location of any files listed with relative path names. (see the fragment in the Enabling Listeners section)  The file directive above means that the absolute path to the zone file will be /var/named/dynamic/app.example.com.db

This will eventually be a dynamic zone.  That's why I'm putting it in "dynamic/app.example.com.db".  If it were to be a static zone I'd probably put it right at the top of the /var/named tree.

The Zone File

The last file I need to create is the zone file. This file defines the initial contents of the application zone. It also defines the NS (nameserver) records for the zone and the default TTL (time to live).

/var/named/dynamic/app.example.com.db

$ORIGIN .
$TTL 1800 ; Default TTL: 30 Minutes
app.example.com. IN SOA ns1.example.com. hostmaster.example.com. (
                         2011112904 ; serial
                         60         ; refresh (1 minute)
                         15         ; retry (15 seconds)
                         1800       ; expire (30 minutes)
                         10         ; minimum (10 seconds)
                          )
                     NS ns1.example.com.
                     NS ns2.example.com.
;; prime the nameserver IP addresses for the app zone.
ns1.example.com.               A        192.168.5.2
ns2.example.com.               A        192.168.5.3

Verifying The App Zone

Once the configuration file and the zone database file are in place it's time to try restarting the named service. I use service named restart. and observe the results.

service named restart
Stopping named: .                     [ OK ]
Starting named:                       [ OK ]

Then I use grep named /var/log/messages to observe the typical start up messages. I look for a line indicating that the app.example.com zone has been loaded.

grep named /var/named/messages | grep loaded
...
Nov 22 17:40:10 ns1 named[3888]: zone 0.in-addr.arpa/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone example.com/IN: loaded serial 2011112904
Nov 22 17:40:10 ns1 named[3888]: zone app.example.com/IN: loaded serial 2011112906
Nov 22 17:40:10 ns1 named[3888]: zone localhost.localdomain/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: zone localhost/IN: loaded serial 0
Nov 22 17:40:10 ns1 named[3888]: managed-keys-zone ./IN: loaded serial 53

Now that I know that the zone has been loaded successfully, I'll check that it's served properly. I first request the SOA (Start of Authority) record, and then the full zone dump.

dig @localhost app.example.com soa
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @localhost app.example.com soa
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3502
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 2

;; QUESTION SECTION:
;app.example.com.  IN SOA

;; ANSWER SECTION:
app.example.com. 30 IN SOA ns1.example.com. hostmaster.example.com. 2011112906 60 15 1800 10

;; AUTHORITY SECTION:
app.example.com. 30 IN NS ns2.example.com.
app.example.com. 30 IN NS ns1.example.com.

;; ADDITIONAL SECTION:
ns1.example.com. 600 IN A 10.16.137.243
ns2.example.com. 600 IN A 10.16.137.244

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Thu Nov 22 17:29:32 2012
;; MSG SIZE  rcvd: 148

Now test a complete zone dump, (and save it for comparison)

dig @127.0.0.1 app.example.com axfr
; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.5 <<>> @localhost app.example.com axfr
; (2 servers found)
;; global options: +cmd
app.example.com. 30 IN SOA ns1.example.com. hostmaster.example.com. 2011112906 60 15 1800 10
app.example.com. 30 IN NS ns1.example.com.
app.example.com. 30 IN NS ns2.example.com.
app.example.com. 30 IN SOA ns1.example.com. hostmaster.example.com. 2011112906 60 15 1800 10
;; Query time: 1 msec
;; SERVER: ::1#53(::1)
;; WHEN: Thu Nov 22 17:18:28 2012
;; XFR size: 4 records (messages 1, bytes 152)

Summary

At this point I have a working authoritative server for the app.example.com zone.  To get it properly delegated I need to create a secondary server and configure zone transfers.  Then I can provide the nameserver NS and A records and a contact to my IT department and they can complete the delegation.

For now I'm going to skip delegation. The next post will describe the configuration of dynamic updates

References

• ISC Bind 9 reference
• DNS and Bind, 5th ed. Cricket Liu, O'Reilly Media
• DNS for Rocket Scientists - An online reference
• named.conf man page
• rndc man page
• rndc-confgen man page

• RFC 2606 - Reserved Domain Names

OpenShift Back End Services: The DNS - Concepts

Most of the time, DNS just works. I have heard some people (hardware lab techs mostly) talk about how they use IP addresses for their work because DNS is so unreliable.  Then I explain to them that (at least outside the lab) DNS is probably the most reliable and fundamental service on the modern internet.  Next to the TCP/IP and the routing protocols, DNS is the most critical service.  Without DNS the rest of the net doesn't matter because no one can find anything.  But very seldom does DNS actually fail on a large scale.

For a system like OpenShift, DNS is life's blood.  The whole purpose of a PaaS is to make applications available to users.  OpenShift does that by adding a new DNS record each time an application is created.  The name portion of the record is crafted from the developer's namespace and application name.  The value portion directs a user to the node host which offers the application service.  But before the browser can find the application, the DNS resolver must find the DNS record.

If you're going to run your own DNS it's important to understand how DNS services interact.

What's so hard about DNS?

Given the ubiquity of DNS and its critical function, I've been surprised at the amount of difficulty it has caused configuring it into OpenShift Origin.  I think part of the issue is that ordinarily it works so well that few sysadmins and developers have to work with it in any depth.  Most people's experiences with DNS consist of checking and setting the /etc/resolv.conf nameserver and search lists, and the occasional dig command to check if a zone is responding.

In most companies there's one or two of the IT folks who are "The DNS guys" (or girls?).  They manage the external DNS (which shouldn't change fast) and the internal DNS (which uses lots of DHCP for desktops, laptops, wireless).  They own the DNS domains and getting new IP name/address assignments from them has a well defined process.  Getting a delegated sub-domain is generally a more involved process.  The DNS Guys don't like to do it (because they get the calls when your DNS  breaks) so people who need DNS (like lab spaces) will make do with their own or go without.

A number of geeks like me have set up split DNS in their houses.  This requires DNS forwarding features, but not delegation.  There are even tools now like Dnsmasq which implement simple split DNS and combine DNS, DHCP and Dynamic DNS all in one nice relatively simple service. These are meant for small labs or home networks where they can control the entire DNS namespace.  They provide the hostname to IP address mapping of DNS and combine that with the host resolver configuration offered by DHCP. They only work at the bottom of the namespace hierarchy and they do not require delegation, as nothing in the local database is ever published outside that bottom layer zone. Again, this removes the need for the average system administrator to think much about what's happening behind the scenes.

And there's sometimes rather a lot going on behind the scenes.

The Domain Name Service Behind the Scenes

If you're familiar with DNS operations, you can skip this part.

If you've never managed a DNS hierarchy, you might think that to install a DNS service you just install the bind package, edit the configuration file, add some zone data and start the daemon. Done.  Right?  Not quite.  Unlike nearly all other typical database services, DNS requires the participation of other servers to work properly.

The DNS  is a specialized distributed database with a hierarchical namespace.  Note something really important here.  I didn't say "Bind is..".  I said "The DNS is.." The DNS is ONE DATABASE. The data is distributed across the entire internet and the authority for portions are delegated to hundreds of thousands (or more) origanizations and individuals, but if it weren't a single unified entity it wouldn't work.  The magic of the DNS is the way in which the namespace and data are "glued" together. To see how that works I'm going to walk through an example.

DNS Queries and the /etc/resolv.conf file

When I try to access a web site from my laptop, the first thing that happens is a DNS query to resolve the URL host name to the IP address of the destination host. My computer is going to send a DNS request to some server. A computer that answers DNS requests is called a nameserver.  I have to know the address of the nameserver.  If I only knew the name of the name server, I'd have to do a look-up query for that, and since I don't have the address of someone to ask I'd be stuck in a loop.

Fortunately, the pump is primed by the /etc/resolv.conf file.

The /etc/resolv.conf file contains a list of nameserver entries. This is a list of IP addresses.  Each of the addresses corresponds to a DNS nameserver host.

; generated by /usr/sbin/dhclient-script
search westford.example.com example.com
nameserver 192.168.4.2
nameserver 192.168.4.3
nameserver 172.30.41.12

When I make a request with a hostname, the resolver library on my laptop issues a query to the first nameserver in the list.  Say I wanted to visit openshift.example.com.  The resolver would send a query which asks essentially "tell me what you can about anything named 'openshift.example.com'"  You can simulate this with either the dig or host commands.

The dig and host commands

The dig and host commands are programs who's only purpose is to issue DNS queries and report the responses.  I tend to use host when all I need is the answer.  host has much simpler output and looks to me like it is designed for use in command-line scripts.  It responds with a single line and each field in the output is space separated.

host www.example.com
www.example.com has address 192.0.43.10
www.example.com has IPv6 address 2001:500:88:200::10

I use dig when I am verifying or diagnosing DNS operation.  By default dig prints a (mostly) human readable  report of the entire DNS response from the nameserver.  This includes not only the requested records but the authority records which indicate where the answer ultimately came from.  The format is not horribly human friendly or even string-parser-friendly, but once you learn to read it it is very concise and informative.

dig www.example.com

; <<>> DiG 9.9.2-RedHat-9.9.2-2.fc17 <<>> www.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30626
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 5

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.example.com.  IN A

;; ANSWER SECTION:
www.example.com. 172791 IN A 192.0.43.10

;; AUTHORITY SECTION:
example.com.  163951 IN NS a.iana-servers.net.
example.com.  163951 IN NS b.iana-servers.net.

;; ADDITIONAL SECTION:
b.iana-servers.net. 1044 IN A 199.43.133.53
b.iana-servers.net. 1044 IN AAAA 2001:500:8d::53
a.iana-servers.net. 1044 IN A 199.43.132.53
a.iana-servers.net. 1044 IN AAAA 2001:500:8c::53

;; Query time: 43 msec
;; SERVER: 10.11.255.156#53(10.11.255.156)
;; WHEN: Mon Nov 26 18:14:47 2012
;; MSG SIZE  rcvd: 196

In the examples that follow I'm mostly going to use dig though I may also trim them some to highlight the important parts.

A Name Lookup Example

This example is not at all contrived.  This kind of thing happens to me every day.. Really..  Sure it does.

Say I'm sitting at my desk, working on my laptop.  I'm a mid level sysadmin at Example.Com in the office in Boston, MA.   My desktop has a DNS name something like "llamadesk.boston.example.com".  My co-worker (the lucky SOB) is working from the beach outside company office on Maui.  He posts a document file on a web server in the office there.  The web server hostname is "www.maui.example.com". I want to see the document so he sends me a url for it and I dutifully paste it into my web browser address bar and hit the enter key.

My web browser is linked with the local resolver library (usually called libresolv). It has a function called getaddrinfo (used to be gethostbyname) which takes a hostname string as input and returns (among other things) an IP address associated with that name.  What happens between the call and the return is the interesting part.

The first thing the resolver library does is read /etc/resolv.conf. Then it crafts a query packet and sends it off to the first IP address in the nameserver list and waits for a response.

The nameserver is listening for query packets. I receives the packet and tries to find the best answer.

A nameserver, when looking at a query, can know one of three things:

• I know the answer.
• I don't know the answer, but I know the nameserver for a domain that contains the answer.
• I don't know the answer or the domain, but I know where the root domain is.

A server which knows the answer is called the authoritative nameserver for the domain.  It will answer all queries for the contents of the domains it serves.

If the nameserver is not authoritative it then has a choice.  It can merely return a response which means "I don't know" or it can perform a recursive query.  Most of the nameservers which are at the edge of the DNS where desktops will be making queries will be configured to recurse.

So, my nearest (recursive) nameserver is in boston.example.com. It doesn't know about servers in maui.example.com. However, it does know that it's in example.com and it knows how to find the example.com nameservers.  These are servers in the NS records for example.com. So the nearby nameserver issues a query to one of the NS servers for example.com and asks for the NS records from the maui.example.com domain.  The reply will contain the names of the authoritative nameservers.  The nearby nameserver then requests the A records for the maui.example.com nameservers.  Now it knows someone who does know the answer.  It sends one final query to the maui nameserver for www.maui.example.com.  The maui nameserver returns the answer (or an error response) and the local nameserver returns the answer to my browser which can finally make a connection to the actual target host.

Did you follow all that?  See if this helps:

Or this?

1. llamadesk -> ns1.boston.example.com
   "tell me the IP address for www.maui.example.com"
2. ns1.boston.example.com -> ns1.example.com
   "tell me who serves maui.example.com"
3. ns1.boston.example.com -> ns1.example.com
   "tell me the IP address for ns1.maui.example.com"
4. ns1.boston.example.com -> ns1.maui.example.com
   "tell me the IP address for www.maui.example.com"
5. llamadesk <- ns1.boston.redhat.com
   "here's the IP address you asked for"

Glue Records: Binding the Internet Together

The links that make the DNS work are known as glue records. The process of establishing a link between one layer in the hierarchy and the next layer down is called delegation.

The nameserver at the example.com level has to know about all of the sub-domains below example.com.  It must have two types of records for each sub-domain.  It must have a set of NS records which contain the DNS name of the authoritative servers for the sub-domain.  Since NS records return the hostname of the nameservers, the parent must also provide an A record for each nameserver.

As noted in other places, the technical aspects of delegation are much less significant than the political or organizational aspects.  Delegation requires the establishment of a relationship communication and of trust between groups that may normally be somewhat territorial. Once a sub-domain is delegated it is the responsibility of the receiving administrator to ensure that the domain is always available so that the it contains remain accessible and to be able to accept and respond to problem reports.

Development and Test Environments: Rogue DNS

"Rogue DNS" is the term I use for an undelegated DNS zone.  Some people try to soften the term but I think "rogue" carries just the right connotations.  A rogue is an independent, slightly unsavory character who none the less is capable and possibly even attractive in a "bad boy" sort of way.  A rogue is not always a bad guy and sometimes it takes a rogue to save the day.

Every NAT network which includes split DNS would be considered "rogue" under this definition.  That's pretty much every home network and most commercial business networks today.  Rogues aren't all bad.

Rogue zones are also common in testing and small or personal development environments.  They don't require any negotiation. They're pretty much required for demos or livecd try-it-out implementations.

The establishment of a rogue zone is really easy: Just create the servers and start adding resource records to the zone. The problem is that without delegation, the rogue zone is invisible to everyone else. 

The real problem with rogue DNS is that every client that wants to participate in the zone must be manually re-configured to see the rogue.  The first nameserver in the client /etc/resolv.conf must be one of the rogue name servers.  In a typical NAT environment the owner of the DNS also owns the DHCP services. Since the DHCP server also provides dynamic nameserver information, all of the DHCP clients automatically participate in the DNS as well.  In a lab setting, the lab administrators may control the DHCP as well.

Rogues and DNS Forwarding

The other problem with a Rogue DNS service is that the rogue. because it is not delegated, does not know about anything outside itself.  Bind does have a facility for "forwarding" requests.  This is commonly used in NAT environments.

When a forwarding server gets a query for which it is not authoritative, rather than trying to recurse, it will forward the request directly to one of a list of "upstream" servers.  These are usually the servers that would normally have been in the nameserver host's /etc/resolv.conf.

Dynamic DNS

The final concept to cover is Dynamic DNS.  As noted in the previous post, OpenShift depends on the ability to add and remove resource records from a DNS zone.

In most corporate DNS services the zone files are fairly static.  They are often mechanically generated from some other database on a regular basis.  It is common for DNS updates to require from 1 or 2 hours to as much as 24 hours.   OpenShift requires the updates to be applied instantly and propagation times of more than a few seconds are considered unacceptable performance.

The one exception is DNS assigned from DHCP.  Microsoft Active Directory is especially good at this.  Dnsmasq is a combined DNS/DHCP/TFTP service designed for home and small business NAT networks.  When it assigns an IP address it can also bind the address to a hostname requested by the client.  It is also possible to connect ISC Bind and ISC DHCP to do Dynamic DNS.

OpenShift does Dynamic DNS through the DNS plugin.  I want to say "plugins", but right now there is only one DNS plugin.  I've written a few posts on writing a new DNS plugin, but it needs the last few, and the sample I picked will only be useful for labs.  Personally I think we need plugins for the greatest possible variety of external DNS services, from Microsoft Active Directory DNS to commercial services.

 DNS Update services will all have similar communications requirements.  Server and access information as well as the zone to update and the new resource record content.

Closing

I'm sure you'll agree I've lectured enough on the relevant capabilities and behaviors of DNS.  I mostly went through this exercise to be sure I hadn't missed anything myself.

You'll notice that I refer a lot to RFCs (Request for Comments).  These are the official specifications for the behavior of parts of the internet.  A lot of people find the idea of the RFCs intimidating. They're dense and bland.  They're also your friend. Don't be scared to go looking for information you need.  You don't have to read them like a good novel, but it's good to scan the relevant documents and at least know where to find answers.  I think a lot of people also skip the RFCs because people are looking for "how do I do it".  The RFCs only tell you "How does it work".  I think often the latter helps illuminate the former.

When I go looking for an RFC, I usually don't know the right one to look for.  Use the search engines
Google for your topic and add "RFC" to the beginning of the query and you'll very likely get a good reference.

Scan the RFCs. You'll be glad you did.

The next post will describe the creation of an authoritative DNS server using ISC Bind 9.  As I go along I mean to include not only the configuration steps but to demonstrate some tools and resources for checking the status of the service and for diagnosing any problems that might arise.

References

• A list of DNS related RFCs at zoneedit.com
• DNS and Bind 5th Ed, Cricket Liu, O'Reilly Media
• Zytrax DNS for Rocket Scientists an HTML book on DNS and Bind
• ISC Bind documentation

RFCs specifically significant to OpenShift:

• RFC 1033 - Domain Administrators Operations Guide
• RFC 1034 - Domain Names - Concepts and Facilities
• RFC 1035 - Domain Names - Implimentation and Specification
• RFC 2136 - DNS Update
• RFC 2845 - Secret Key Transaction Authentication for DNS (TSIG)

OpenShift Back End Services: The Domain Name System - Overview

In the previous two posts I covered the OpenShift Origin back end database and the OpenShift broker to node communications system (messaging).

This time I'm going to talk about the service OpenShift Origin uses to publish user's applications: The Domain Name System or DNS.

OpenShift Origin and the DNS

One of the critical functions of OpenShift Origin is the publication of new user applications.

Each new application gets its own IP name. These names are made available universally through the DNS. The name maps directly or indirectly back to the host on which the application runs. Without the DNS, the user applications would remain invisible to the rest of the world and in particular to the application's audience.

The other three back end services (Data store, Messaging and Identification/Authentication) used by OpenShift Origin are, in some sense, self-contained.  Only the OpenShift Origin broker or node needs to access them.  In fact, access by any other means would be considered a security flaw in the design or implementation of the OpenShift Origin service.

One implication of the self-contained nature of these services is that they can be implemented and run without  the need for cooperation with other parts of the network.  Publication (and the DNS) is fundamentally different. It makes no sense to run a "self-contained DNS".

This is why I'm going to urge you not to run your own.

The best way to manage this part of the OpenShift Origin service is to ask someone else to do it. Every organization has an IT department.  They're going to control both the internal and external DNS services.  They're also going to have the knowledge and resources to add a sub-domain for your applications and to configure it for your access.  You'll need to get some information from them that you'll configure into the Bind DNS plugin and then your OpenShift Origin service will be able to publish new applications automatically.

The Openshift Bind Plugin

There's currently only one DNS plugin at the moment.  This is the Bind plugin rubygem-openshift-origin-dns-bind. This is actually a misnomer.  The plugin uses the Dynamic DNS Update protocol defined in RFC 2136. Any DNS service which implements RFC 2136 should accept updates from the Bind plugin. However, the only service I can confirm is ISC Bind, hence the name.

The OpenShift service can only add applications to a single zone. All of the resource records in a dynamic zone are potentially changed. This means that the OpenShift servers should not reside in the same zone as the applications.

Bind Plugin Requirements

If you do get your IT department to set up dynamic DNS for you, this is the information you'll need from them.

The Bind plugin configuration requires:

• Update Server IP address
• Update Server port
• Update zone
• TSIG Key Type
• TSIG Key Name
• TSIG Key Value

Running a Delegated Zone

If your IT department chooses to delegate your application zone, there is some information you will have to give them.

• Primary Nameserver fully qualified domain name
• Primary Nameserver IP address
• Secondary Nameserver fully qualified domain name
• Secondary Nameserver IP address
• Zone contact: an email address for a person who can manage the domain.

Your IT department will configure this information into a Start of Authority (SOA) record, a pair of nameserver (NS) records and two A records.  These are the DNS delegation glue records which allow others to find your zone.

In this case you will be responsible for building and managing your own Dynamic DNS servers.  The next few posts will walk through how to set up your own DNS servers on ISC Bind.  This is a more involved process than either of the previous services, so I'm going to present it in several steps.

1. Caching Server - Answers queries but doesn't contain any zone data.
2. Primary Authoritative Name Server - Answers queries, contains the master copy of some zone data
3. Secondary Authoritative Name Server - Answers queries, contains a slave copy of some zone data
4. TSIG User Key - A symmetric key for user authentication 
5. Dynamic DNS Server - accepts update requests for a zone. Serves the master copy.

Unconventional DNS Configurations

There are two additional topics I plan to cover, though they're best avoided if possible.  They deliberately break the hierarchical model of the DNS and can cause considerable confusion for unwary users and administrators.

• Rogue DNS
  For labs and development
• Serving DNS from dynamic IP addresses.
  For environments like Amazon Web Services or libvirt virtual machines

A Rogue DNS service is one which serves zones which are not properly delegated.  They are accessable only if every host that participates is explicitly configured to make requests from the rogue servers.

Hosts on services like Amazon are normally given IP addresses via DHCP. If the IP address of the primary public interface is re-addressed, the DNS services must also be reconfigured and possibly all of the clients must be notified.  This extremely bad for the stability of a DNS service but may be required in some environments.

References

• The Domain Name System
• Zytrax DNS for Rocket Scientists
• RFC 1591 - The Domain Name System and Delegation
• RFC 2136 - DNS Update
• RFC 2845 - Secret Key Transaction Authentication for DNS
• ISC Bind DNS server

Laying the Foundation: Broker Back End Services

An OpenShift Origin service depends on a number of moving parts to work.  The broker in particular requires four different back end services to be in place and configured before the broker itself can be set up and tested.

This post is the first of a series that will detail how to set up the back end services needed by an OpenShift Origin service.

Background

The purpose of an OpenShift Origin service is to provide a means for application developers to quickly create and deploy web applications (loosely defined).  The service can have multiple users who must be given the resources they need, but not the ability to effect other user's information or applications.  The user and application information must be stored and accessed by the OpenShift Origin service.  The controlling part of the Openshift Origin service (the broker) must be able to communicate with the parts that actually contain the user's application processes and data (the nodes).  Finally the Openshift Origin service must be able to publish the locations of the user applications as they are created.

This means that the broker must have the means to perform these abstract functions:

• User Identification and Authentication
• Data Storage
• Messaging
• Publication (DNS)

These functions are performed on behalf of the OpenShift Origin service by external or back end services. All of these services must be prepared before the OpenShift Origin broker can be configured.

The broker service integrates with each of these services through a plugin for the service.  As of the writing of this post each back end service has only one plugin.  Over time additional plugins can be added to allow integration with different back end service implementations.

Information Gathering

The Openshift Origin components and back end services can reside on different hosts on your network or even across the internet.  For this series I'm going to work with a single RFC1918 network segment.  I'm going to use two DNS domains, or more properly a top level domain and a sub-domain.  The top level domain will contain the DNS names for the service hosts.  The sub-domain will be used to publish the applications.

For this demonstration each of the services will reside on a different host.  This is to make the boundaries of each service clear.  They can co-exist on fewer hosts or even all on a single host if that suits your needs.

Openshift Origin Service Hosts

Function	Hostname	IP Address	Implementation
Broker	broker1.example.com	192.168.5.11
Name Service	ns1.example.com	192.168.5.2	Bind 9
Data Store	data1.example.com	192.168.5.5	MongoDB
Messaging	msg1.example.com	192.168.5.8	ActiveMQ/Stomp
Node	node1.example.com	192.168.5.21
	node2.example.com	192.168.5.22

You don't see a host there for authentication because the authentication is done by an apache mod-auth-* module which is configured in the apache server on the broker. Configuring the various mod-auth modules is already well documented. For this demonstration we'll use Basic-Auth and an htpasswd file on the broker host.

We're going to install the back end services first, beginning in the next post with the Bind 9 name service.