Saturday, August 18, 2012

Openshift Broker: Writing a Plugin Pt3 - Preparing to Test

FINALLY! Some Code!

The first two installments were used to create the boilerplate that goes around an Openshift Origin broker plugin. In this one we'll create the the skeleton of the plugin itself.  We'll also start writing the tests we need to verify that it's working.

MORE Boilerplate? REALLY?

There's a litte more infrastructure work to be done before we can actually start putting in implementation code.

Rubygems (or maybe just Openshift Origin rubygems?) like to have the source files arranged in a particular way.  I'm not quite sure why but I think it has to do with how Ruby require statements work.

The source files for the gem live in the lib subdirectory.  The file tree looks like this:

├── uplift-dnsmasq-plugin
│   └── uplift
│       └── dnsmasq-plugin.rb
└── uplift-dnsmasq-plugin.rb

2 directories, 2 files

The top level file just imports any other required modules and sets the superclass provider variable so that calls to the superclass instance() method will return the right sub class instance.  This is easy to reproduce:

1:  # Openshift Origin DNS plugin: DNSMasq  
2:  #  
3:  # The superclass StickShift::DnsService is defined in stickshift-controller  
4:  require "stickshift-controller"  
6:  # import the actual class definition and interface implementation  
7:  require "uplift-dnsmasq-plugin/uplift/dnsmasq_plugin.rb"  
9:  # initialize the superclass factory provider with the implementation class
10: # Shouldn't this really be a configuration value?  
11: #StickShift::DnsService.provider=Uplift::DnsMasqPlugin 

The require statement on line 7 is the one that will actually import the implementation code. Its value means that the implementation has to live at:


But what do we put there?

DNS Plugin Interface

Each plugin implements an interface to the back end service.  The DNS plugin has the simplest interface (and it's likely to get even simpler soon) so it's a good place to start.

The Openshift Origin plugin interfaces are defined in a de-facto way by the rubygem-stickshift-controller package code. Specifically, there are four ruby files down there which declare the plugin base classes and define stub methods for the interfaces.

  • application_container_proxy.rb
  • auth_service.rb
  • data_store.rb
  • dns_service.rb
(Currently the data_store "plugin" is fixed and uses the mongo_data_store.rb definition.)

As noted, the DNS service plugin interface is the simplest (which is why I'm starting there). It consists two standard class methods, an instance variable and a set of instance methods.

The class methods provide the override mechanism and factory class so that most code can just refer to a superclass instance.  The superclass is initialized with the desired subclass (using the provider=() method) during configuration and it provides subclass instances through the instance() method later.  It sounds confusing but all you really have to do is duplicate that bit of simple code.

The DnsService class definition looks like this:

1:  module StickShift  
2:   class DnsService  
3:    @ss_dns_provider = StickShift::DnsService  
5:    def self.provider=(provider_class)  
6:     @ss_dns_provider = provider_class  
7:    end  
9:    def self.instance  
11:    end  
13:    def initialize  
14:    end  
16:    def namespace_available?(namespace)  
17:     return true  
18:    end  
20:    def register_namespace(namespace)  
21:    end  
23:    def deregister_namespace(namespace)  
24:    end  
26:    def register_application(app_name, namespace, public_hostname)  
27:    end  
29:    def deregister_application(app_name, namespace)  
30:    end  
32:    def modify_application(app_name, namespace, public_hostname)  
33:    end  
35:    def publish  
36:    end  
38:    def close  
39:    end  
40:   end  
41:  end


That's all very nice and clean but it appears that the interface specs need some documentation so implementors can figure out what to do.  One thing we can do is look at a an existing implementation for clues.  What we find is this:

Two Class Methods: 

  • DnsService.provider=()  - set the subclass that will implement the interface
  • DnsService.instance() - get an instance of the implementing class (factory method) 

These two methods are used to set and then retrieve instances of the plugin class.  In current implementations  provider=() is called at the bottom of the plugin definition file.  This won't work if for some reason you want access to more than one plugin and the ability to switch from one to the other (as in for testing).  For now you can only install one plugin package.

Instance Methods: (The Meat)

The DNS Plugin is used to publish applications via the Domain Name Service.  It has a legacy use as a registry for user namespaces.  That registry function has moved to the MongoDB DataStore but the calls to the DnsService namespace* methods remain so we have to implement them.

These work on namespaces
  • namespace_available?(namespace)
  • register_namespace(namespace)
  • deregister_namespace(namespace)
These work on application DNS records
  • register_application(app_name, namespace, public_hostname)
  • deregister_application(app_name, namespace)
  • modify_application(app_name, namespace, public_hostname)
There are also two methods which are used to control the service update connection
  • publish() - called to force/allow cached updates to be sent to the DNS service
  • close() - called at the end of an update session to indicate that no more updates will be coming
These methods must at least be stubbed and may have side effects depending on the back end update protocol.  Callers need to be aware that updates may not wait until publish is called in some cases.

We're not ready to start implementing anything yet so we'll just copy the empty stubs from the interface definition file to start.

2:  #  
3:  # Implement the StickShift::DnsService interface using DNSMasq  
4:  #  
5:  require 'rubygems'  
7:  module Uplift  
8:   class DnsMasqPlugin < StickShift::DnsService  
9:    @ss_dns_provider = Uplift::DnsMasqPlugin  
12:    #[:domain_suffix]  
13:    # Rails.application.config.dns[...]  
15:    #  
16:    # Define stubs for the interface  
17:    #  
19:    def initialize(access_info = nil)  
20:    end  
22:    # Determine if a namespace is available.  
23:    # Return false if it has been reserved  
24:    # and true otherwise  
25:    def namespace_available?(namespace)  
26:     return false  
27:    end  
29:    # reserve the indicated namespace  
30:    def register_namespace(namespace)  
31:    end  
33:    # unreserve the indicated namespace  
34:    def deregister_namespace(namespace)  
35:    end  
37:    # publish the IP Name/Address of an application  
38:    def register_application(app_name, namespace, public_hostname)  
39:    end  
41:    # unpublish the IP Name/Address of an application  
42:    def deregister_application(app_name, namespace)  
43:    end  
45:    # update the IP Name/Address of an application  
46:    def modify_application(app_name, namespace, public_hostname)  
47:    end  
49:    # finalize accumulated updates (if needed)  
50:    def publish  
51:    end  
51:    end  
53:    # end communications with the server through this instance  
54:    def close  
55:    end  
57:   end  
59:  end

And Testing....

Even though this doesn't do anything yet we can still test it.  It's time.

We have an empty test file in spec/unit/uplift_dnsmasq_spec.rb.  We need to put something in it and run it and verify that the emtpy files load without errors.

There are a number of different unit testing frameworks for use with Ruby.  It comes with a test-unit, an xUnit style testing framework.  RSpec is a more recent DSL (Domain Specific Language) testing framework.   It is commonly used to test Rails applications.  You can use either one or some other, but you really should use something for unit testing.

In the previous blog entry we create a Rake task to run all RSpec tests in the spec subdirectory.  We placed a single file there so that we could observe the operation of the rake spec task.  The file is empty though. "running" the tests just indicates that there are no tests to run.  Now we can being to populate the test file.

The purpose of writing tests (at least initially) is to get them to fail.  Luckily, that's the easy part.  All we have to do is add a couple of lines to the test file:

1:  # Test each of the basic functions of the DnsMasqDnsService plugin class   
3:   # The plugin extends classes in the StickShift controller module.   
4:   require 'rubygems'   
6:   # Now load the plugin code itself (not the wrapper!)   
7:   #require 'uplift-dnsmasq-plugin/uplift/dnsmasq-plugin'   
9:   describe "DNSMasq DNS update plugin" do   
11:   it "has a forced pass" do   
12:    a = 1   
13:    a.should === 1   
14:   end   
16:   it "has a forced fail" do   
17:    a = 1   
18:    a.should === 0   
19:   end   
21: end  

The require statement for the plugin source code is still commented because we're going to hit some issues when we first try to load it.  This though will run and show 1 passing and one failing test.

 rake spec  
 /usr/bin/ruby -I ../../stickshift/common/lib -I ../../stickshift/controller/lib -I lib -S rspec ./spec/unit/uplift_dnsmasq_spec.rb  
  1) DNSMasq DNS update plugin has a forced fail  
    Failure/Error: a.should === 0  
     expected: 0  
       got: 1 (using ===)  
    # ./spec/unit/uplift_dnsmasq_spec.rb:18:in `block (2 levels) in <top (required)>'  
 Finished in 0.00285 seconds  
 2 examples, 1 failure  
 Failed examples:  
 rspec ./spec/unit/uplift_dnsmasq_spec.rb:16 # DNSMasq DNS update plugin has a forced fail  
 rake aborted!  
 /usr/bin/ruby -I ../../stickshift/common/lib -I ../../stickshift/controller/lib -I lib -S rspec ./spec/unit/uplift_dnsmasq_spec.rb failed  
 Tasks: TOP => spec  
 (See full trace by running task with --trace)  

Now we know that the test scripts are running and that both pass and fail will work - when we don't include our code.  Time to do that. If you uncomment line 7 of spec/unit/uplift_dnsmasq_spec.rb then the next run will try to load the plugin source file. And it will fail to load due to missing requirements.

 rake spec  
 /usr/bin/ruby -I ../../stickshift/common/lib -I ../../stickshift/controller/lib -I lib -S rspec ./spec/unit/uplift_dnsmasq_spec.rb  
 /home/mark/work/crankcase/uplift/dnsmasq/lib/uplift-dnsmasq-plugin/uplift/dnsmasq-plugin.rb:7:in `<module:Uplift>': uninitialized constant Uplift::StickShift (NameError)  
      from /home/mark/work/crankcase/uplift/dnsmasq/lib/uplift-dnsmasq-plugin/uplift/dnsmasq-plugin.rb:6:in `<top (required)>'  
      from /usr/share/rubygems/rubygems/custom_require.rb:36:in `require'  
      from /usr/share/rubygems/rubygems/custom_require.rb:36:in `require'  
      from /home/mark/work/crankcase/uplift/dnsmasq/spec/unit/uplift_dnsmasq_spec.rb:7:in `<top (required)>'  
      from /home/mark/.gem/ruby/1.9.1/gems/rspec-core-2.11.1/lib/rspec/core/configuration.rb:780:in `load'  
      from /home/mark/.gem/ruby/1.9.1/gems/rspec-core-2.11.1/lib/rspec/core/configuration.rb:780:in `block in load_spec_files'  
      from /home/mark/.gem/ruby/1.9.1/gems/rspec-core-2.11.1/lib/rspec/core/configuration.rb:780:in `map'  
      from /home/mark/.gem/ruby/1.9.1/gems/rspec-core-2.11.1/lib/rspec/core/configuration.rb:780:in `load_spec_files'  
      from /home/mark/.gem/ruby/1.9.1/gems/rspec-core-2.11.1/lib/rspec/core/command_line.rb:22:in `run'  
      from /home/mark/.gem/ruby/1.9.1/gems/rspec-core-2.11.1/lib/rspec/core/runner.rb:69:in `run'  
      from /home/mark/.gem/ruby/1.9.1/gems/rspec-core-2.11.1/lib/rspec/core/runner.rb:8:in `block in autorun'  
 rake aborted!  
 /usr/bin/ruby -I ../../stickshift/common/lib -I ../../stickshift/controller/lib -I lib -S rspec ./spec/unit/uplift_dnsmasq_spec.rb failed  
 Tasks: TOP => spec  
 (See full trace by running task with --trace)  

See? And that's what we'll fix next.

(No references this time, since there's really nothing new)

Thursday, August 16, 2012

Openshift Broker: Writing a Plugin Pt 2 - Skeleton

Welcome back.

In the first installment I established a build workspace for a new plugin for the Openshift Origin broker.

In this one, I'm going to create the skeleton of the new plugin in the source code tree.

Building the Scaffolding

There's a lot more to an Openshift Origin plugin than the Ruby source code. (That goes for any well produced modern software actually).  There's a raft of boilerplate for testing, packaging and documentation.
In this installment I'm going to create the skeleton of the new plugin package.  It will include everything but the source code.

The first thing to do is to identify the proper location withing the source tree.  The existing DnsService plugin is in uplift/bind.  The logical location for the new plugin is uplift/dnsmasq. Create the directory and change to it

  mkdir uplift/dnsmasq
  cd uplift/dnsmasq

The next step is to populate the initial boilerplate for a package.  Take a look in the adjacent bind plugin.

  ls ../bind
  COPYRIGHT lib    
  doc                 LICENSE  uplift-bind-plugin.gemspec
  Gemfile           Rakefile      uplift-bind-plugin.spec

doc and lib are directories. The rest of the files are typical for a package or plugin.  Several of the files can be copied as-is.  Most require some editing to reflect the new package they describe. For now, copy the normal files.  We'll customize them before committing them for the first time.

  cp ../bind/* .
  mv uplift-bind-plugin.gemspec uplift-dnsmasq-plugin.gemspec
  mv uplift-bind-plugin.spec uplift-dnsmasq-pluging.spec
And remove any of the code from the original plugin.  We'll add the real plugin code later.

rm -rf lib/*

Informational Files

There are three purely informational (or legal) files.  These can be edited to suit your needs:

  • - describe the package or plugin
  • COPYRIGHT - declare the ownership of the code
  • LICENSE - indicate the terms of use

Edit the to describe the purpose and use of your new package. (I'm not sure what to do about the US export language.  We'll see when someone puts in a pull-request without it).

The Openshift Origin source code which exists so far is all copyrighted by Red Hat Inc. and is licensed under the Apache license version 2.0.  You can keep the copyright yourself (though Red Hat has lawyers).  The license must be compatible with Apache 2.0 if you hope to have your code merged back.

Build/Test Code (Rakefile)

Rake is the Ruby equivalent of the venerable Make utility.  Rakefiles are often much simpler than Makefiles for common tasks because Rake includes a large library of pre-defined tasks.  This module will use Rake and RSpec to control unit testing.  You will need to modify the copied file to match the text below.

#require "bundler/gem_tasks"
require 'rake'
require 'rake/testtask'
require 'rspec/core/rake_task' 
desc "Run RSpec unit tests" do |t|
  t.pattern = "./spec/**/*_spec.rb" # don't need this, it's default.

You will also need to create at least the skeleton of one spec file:

mkdir spec/unit
echo "# RSpec test placeholder" > spec/unit/uplift_bind_spec.rb
Check that your simple test will run:

rake spec
/usr/bin/ruby -S rspec ./spec/unit/uplift_bind_spec.rb
No examples found.

Finished in 0.00004 seconds
0 examples, 0 failures

Package Description Files

The remainder of the files are used to control testing and packaging.

  • Gemfile - the standard Rubygem specification file: just includes the gemspec
  • uplift-dnsmasq-plugin.gemspec - the Rubygem package spec.
  • uplift-dnsmasq-plugin.spec - the RPM package spec
These last two files will require extensive editing.  Obviously you want to change the package naming.  Scan the Requires and BuildRequires. If you're not sure you need something, take it out.  You can always add it back later. Update the description, author info, sources etc. Finally you want to strip out all of the old change-logs.  If you want to get your package accepted into Fedora and or RHEL you'll want to compare it against the Fedora Packaging Guidelines.

Add and Check-In new Plugin Files

Once the package skeleton is complete, and before you can attempt to build the package, the new files must be added to your git workspace.

  git add .
  git commit -m 'create skeleton of dnsmasq plugin' .

Test Build Empty Plugin

Now that the plugin files are created you can attempt to build the (admittedly empty) plugin package for the first time. The --test --rpm  options indicate that tito should build a complete RPM and it should use the sources after the most recent tag.  When you're ready to release a new version, you'll re-run tito tag again to update the RPM version.

Be ready.  tito and rpmbuild aren't quiet about what they're doing.  If you scan though this LONG output you should be able to track the creation of a build space, the configuration and package build and buried in their the rake spec tests.  The last couple lines inform you where your new packages reside.

If what you see indicates errors, you'll have to track them down and correct them and try again.  Remember tito only works with checked in files.  Read up on it from the References at the end.
tito tag
tito build --test --rpm
 Building package [rubygem-uplift-dnsmasq-plugin-0.1.2-1]
Wrote: /tmp/tito/rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96.tar.gz
-e:1: Use RbConfig instead of obsolete and deprecated Config.
-e:1: Use RbConfig instead of obsolete and deprecated Config.
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.k2eDj1
+ umask 022
+ cd /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD
+ export LANG
+ unset DISPLAY
+ cd /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD
+ rm -rf rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ /usr/bin/gzip -dc /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/SOURCES/rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96.tar.gz
+ /usr/bin/tar -xf -
+ '[' 0 -ne 0 ']'
+ cd rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.dTur4m
+ umask 022
+ cd /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD
+ cd rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ export LANG
+ unset DISPLAY
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.WfABPI
+ umask 022
+ cd /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD
+ '[' /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64 '!=' / ']'
+ rm -rf /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64
++ dirname /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64
+ mkdir -p /home/bos/mlamouri/rpmbuild/BUILDROOT
+ mkdir /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64
+ cd rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ export LANG
+ unset DISPLAY
+ rm -rf /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64
+ mkdir -p /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64/home/bos/mlamouri/.gem/ruby/1.9.1
+ mkdir -p /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64/usr/local/share/ruby/site_ruby
+ gem build uplift-dnsmasq-plugin.gemspec
WARNING:  no homepage specified
  Successfully built RubyGem
  Name: uplift-dnsmasq-plugin
  Version: 0.1.2
  File: uplift-dnsmasq-plugin-0.1.2.gem
+ gem install --local --install-dir /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64/home/bos/mlamouri/.gem/ruby/1.9.1 --force uplift-dnsmasq-plugin-0.1.2.gem
NOTE: Gem.available? is deprecated, use Specification::find_by_name. It will be removed on or after 2011-11-01.
Gem.available? called from /home/bos/mlamouri/.rubygems/gems/sdoc-0.2.20/lib/sdoc/json_backend.rb:9.
/usr/share/rubygems/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
Successfully installed uplift-dnsmasq-plugin-0.1.2
1 gem installed
Installing ri documentation for uplift-dnsmasq-plugin-0.1.2...
Installing RDoc documentation for uplift-dnsmasq-plugin-0.1.2...
+ ln -s /home/bos/mlamouri/.gem/ruby/1.9.1/gems/uplift-dnsmasq-plugin-0.1.2/lib/uplift-dnsmasq-plugin /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64/usr/local/share/ruby/site_ruby
+ ln -s /home/bos/mlamouri/.gem/ruby/1.9.1/gems/uplift-dnsmasq-plugin-0.1.2/lib/uplift-dnsmasq-plugin.rb /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64/usr/local/share/ruby/site_ruby
+ mkdir -p /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64/usr/share/doc/rubygem-uplift-dnsmasq-plugin-0.1.2/
+ cp -r doc/README.config /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64/usr/share/doc/rubygem-uplift-dnsmasq-plugin-0.1.2/
+ /usr/lib/rpm/ --strict-build-id /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD/rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ /usr/lib/rpm/check-buildroot
+ /usr/lib/rpm/redhat/brp-compress
+ /usr/lib/rpm/redhat/brp-strip-static-archive /usr/bin/strip
+ /usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1
+ /usr/lib/rpm/redhat/brp-python-hardlink
+ /usr/lib/rpm/redhat/brp-java-repack-jars
Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.8scFt6
+ umask 022
+ cd /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD
+ cd rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ unset DISPLAY
+ rake spec
/usr/bin/ruby -S rspec ./spec/unit/uplift_bind_spec.rb
No examples found.

Finished in 0.00004 seconds
0 examples, 0 failures
+ exit 0
Processing files: rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.noarch
warning: File listed twice: /home/bos/mlamouri/.gem/ruby/1.9.1/gems/uplift-dnsmasq-plugin-0.1.2
warning: File listed twice: /home/bos/mlamouri/.gem/ruby/1.9.1/gems/uplift-dnsmasq-plugin-0.1.2/Gemfile
Provides: rubygem(uplift-dnsmasq-plugin) = 0.1.2
Requires(interp): /bin/sh
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires(post): /bin/sh
Processing files: ruby-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.noarch
Provides: ruby(uplift-dnsmasq-plugin) = 0.1.2
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64
warning: Could not canonicalize hostname: markllama
Wrote: /tmp/tito/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.src.rpm
Wrote: /tmp/tito/noarch/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.noarch.rpm
Wrote: /tmp/tito/noarch/ruby-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.noarch.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.EDFbsH
+ umask 022
+ cd /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD
+ cd rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ rm -rf /home/bos/mlamouri/rpmbuild/BUILDROOT/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.x86_64
+ exit 0
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.jVXQh6
+ umask 022
+ cd /tmp/tito/rpmbuild-rubygem-uplift-dnsmasq-plugin-c3e4f96ad4090bc7b6990725eb6972c23f28ebbfhDa71G/BUILD
+ rm -rf rubygem-uplift-dnsmasq-plugin-git-4.c3e4f96
+ exit 0
Successfully built: /tmp/tito/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.src.rpm /tmp/tito/noarch/rubygem-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.noarch.rpm /tmp/tito/noarch/ruby-uplift-dnsmasq-plugin-0.1.2-1.git.4.c3e4f96.fc17.noarch.rpm

Next Time: Putting some Meat on the Bones

In the next installment we'll start adding some real code and the tests to verify it.


Sunday, August 5, 2012

Openshift Broker: Writing a Plugin Pt 1 - Prep

Welcome Mechanics

Openshift Origin is an open source "Platform as a Service" (PaaS) system.  There's a lot of focus on the applications you can create using Openshift Origin.  This column though is dedicated to the people who need to create and run one.

Openshift Origin: Moving Parts

The Openshift Origin service consists of two major components and several minor ones.  The major components are a broker and one or more nodes.  The broker is the control center of the service.  It manages and coordinates the creation and distribution of user applications to the nodes.  The nodes are the place where the user applications run.

The broker also depends on four other services.  It maintains its own metadata in a data store.  User actions are allowed based on the authentication of the user requests. New applications are published in DNS via a name service.  The broker communicates with the nodes through a message service.

The four broker services are abstracted so that they can be implemented using any suitable back-end service.  The back end services are integrated with the broker using plugin modules.

This modular approach means that it is possible (encouraged!) for people to implement their own plugins for the back end services.

Have It Your Way

  Each service currently only has one plugin implemented:
  • Datastore: MongoDB
  • Authentication: MongoDB
  • Name Service: BIND
  • Messaging: Mcollective/QPID
I expect that there will be people who would like to user other implementations for those services.  For example, an Openshift Origin architect might choose to use one of the commercial DNS providers to publish applications.  Most of these have their own DNS update protocols and do not offer and RFC 2136 compliant mechanism. She could implement the StickShift::AuthService interface as a new plugin to interact with the commercial provider.  I expect that there will be people who want to use one of several alternate authentication mechanisms.  OAuth, LDAP, Kerberos, and Active Directory are all possible. It is common today for web services to authenticate using one of the common social media sites.  The choice is up to the service developer.

For someone who intends to implement a new plug-in there are a few considerations:  The Openshift Origin broker is a Ruby on Rails application.  The plug-ins are written in Ruby and will need access to some information from the Rails framework.

What Owner's Manual?

The Openshift Origin service is young and it is still growing and changing rapidly.  Implementing a service is still an exercise for the early adopters with both the benefits and risks that go with it.  There is a lot of room for new developers to contribute but the ecosystem around it is still a bit raw.  Both the code and the documentation need a lot of work.  Expect to do a lot of digging in the code to make it work.  Make good use of the community forums and development email and IRC resources.

The current plugins are stored in the openshift/crankcase repository on Github.  They are packaged as RPMs for inclusion in Fedora and Red Hat Enterprise Linux and their derivatives.  They are also packaged as Rubygems for use on systems which don't use RPM for software management.  Integration with the existing software management tools will be easiest if it is done from the beginning.

I'm going to implement and detail a new name service plugin for Openshift Origin.  This will implement the StickShift::DnsService interface. The most common request for development and testing has been for DNSMasq.  This is a small local service which provides a non-delegated forwarding name server.  It is useful in isolated networks. (in the sense of "individual hosts are not directly accessible from outside").  It can provide DHCP and TFTP services for network booting and dynamic addressing as well as providing dynamic hostname/IP address publication.  Properly configured it will forward requests for which it does not provide answers so that within the network it serves it provides seamless responses for both internal and external networks.

Create the Work Space

Much of the work needed to create a robust develop/build/test/package has been done already for the existing Openshift Origin modules and plugins.  The quickest way to get started right off in the right direction is to fork the openshift/crankcase project and create yourself a development branch.

You're going to need a Github account and a work station with the git client.  The development process will be easiest on Fedora or RHEL, but it's not necessary.  On a Red Hat derived system a lot of the setup will be automated.  I'm going to assume Fedora 16 for this series.  If you you're on anything else and you try one of the setup steps and it doesn't work you may have to do some digging to resolve it.

I'm going to assume you have a working knowledge of git and access to help using Github.

So here's how we get setup:

  1. Workstation: Fedora 16 (easiest), RHEL 6.x or Fedora 17 (see Idiosyncracies)
  2. On RHEL, you'll need to enable the EPEL repo
  3. Install git client
  4. Create Github account
  5. Fork openshift/crankcase
  6. Clone your project
  7. Create a development branch
There are good documents to learn how to do these tasks on the Github site and in the git documentation so I won't repeat them here. The one thing I will  mention is the hub tool which is designed specifically to add the kinds of work flow steps that are required to manage a project in Github.

In general it's a good habit not to do development on the master branch even of your own fork.  You're going to want to merge updates from the original project master branch into your master branch and then rebase your development branch against your local master. This can also make the cleanup which is often needed prior to a pull-request easier.

Because I'm writing a DnsService plugin using DNSMasq, I'm going to name the branch uplift-dnsmasq (I have no idea where uplift came from but it's the prefix in Openshift which indicates a DnsService plugin)

Initialize the Development Environment

There are two ways to prepare the build environment for a new rubygem plugin.  You can create a complete build environment for all possible packages, or you can just install the minimal requirements for your plugin.
The full environment is needed if you mean to rebuild the entire package suite.  However there are still some gotchas when creating a build environment on non-Fedora 16 systems.  A minimal build environment can be customized more easily but will likely fail when trying to build other packages.

Minimal Build Environment

For just building a Rubygem plugin, the whole build environment might be overkill. All you really need is the build tool tito and the ruby and rubygems packages (and their dependencies)

  yum install tito ruby rubygems

Complete Build Environment

The build environment is created with a rake task: build_setup.  So, to use it you have to install rubygem-rake.

  yum install rubygem-rake
  cd crankcase/build
  rake build_setup

Fedora 17

The build_setup task is currently written for Fedora 16. There is also currently no Fedora 17 repo for Openshift Origin.  The first time you run the build_setup task it will create a file called ss.repo. You will need to edit the crankcase/build/Rakefile and replace $releasever with 16 so that the installation process will  complete.

Fedora 17 also has a different version of openjdk.  You'll have to change any references to java-1.6.0-openjdk and java-1.6.0-openjdk-devel to refer to version 1.7.0

Ready to Begin

At this point you should be ready to begin work on the new plugin.  In the next installment we'll create the skeleton of a new DNS plugin.