OpenShift Service Development: Building a Build Box

I found this week that I needed to have build box so that I could repeatedly run the dev/build/install/test cycle. I've messed around with it on and off since i started working on OpenShift but I looked back and realized that I've never written a procedure for creating the build box. So here it is.

The build process takes source code from a git repository and transforms it into packages. Finally it places the packages into an install repository so that they will be available to the target hosts via yum. The yum repository is published by a small web server. It doesn't need to be fancy as it's just flat files.

The instructions here are for Fedora 18 or 19. There are some special considerations for RHEL6 or CentOS6. These are detailed in a section at the bottom of this post. There are notes inline for when the process is different for RHEL.

There are also some considerations for creating a build box in AWS EC2 (or any managed hosting service).

Install the build/publish software

On a minimal install of Fedora, install the base packages needed for the build service. Git to retrieve the source code, tito to build the RPMs, and thttpd to serve the YUM repository to the install targets.

sudo yum install git tito thttpd firewalld

(On RHEL6, enable EPEL repository and skip firewalld)

Create the YUM repository root directory

Next, create a location for the YUM repository. Place it in a space where thttpd will find it and make it writable by the build user (assumed to be the current user)

sudo mkdir /var/www/thttpd/tito
sudo chown $(id --name --user):$(id --name --group) /var/www/thttpd/tito

Enable Web Services

I have to publish the packages to the install hosts once they're built. I need a web server and on Fedora, I need the firewall daemon running and configured to allow HTTP communications.

sudo systemctl enable thttpd
sudo systemctl start thttpd

sudo systemctl enable firewalld
sudo systemctl start firewalld

# Open the port for now
sudo firewall-cmd --zone public --add-service http

# Make the change persistent across reboots
sudo firewall-cmd --zone public --add-service http --permanent

Configure Tito Output Location

Tito places the build results and RPMs in /tmp/tito by default. I can set the target location using the titorc file.

echo "RPMBUILD_BASEDIR=/var/www/thttpd/tito/" > $HOME/.titorc

Retrieve the Source Code Repository

Now that the publication and build services are prepared, it's time to actually get the software source code.

git clone https://github.com/openshift/origin-server.git

If you are doing development, substitute your own fork and branch. If you are doing your editing on the build box (not really recommended, but slightly time saving) you can also use the git: (ssh) protocol and add your github user SSH key so that you can both pull and push changes.

Install Package Build Requirements

Before you can build packages, you must also install any build requirements for the packages. The bourne shell code snippet below will walk the source code tree, find each package root and install all of the build requirements it finds using yum-builddep.

This triggers off the presence of a .spec file in the root of a package tree. It's critical as a package developer to note all build requirements in the .spec file.

for SPECPATH in $(find origin-server -name \*.spec)
do
    PKGDIR=$(dirname $SPECPATH )
    SPECFILE=$(basename $SPECPATH)
    (cd $PKGDIR ; sudo yum-builddep -y $SPECFILE )
done

Build All Packages

Now that all the build requirements are installed, it's time to build the software.

The bourne shell snippet below will walk the entire source tree and locate the root of each package tree and run tito to build the package. If you are building test packages, uncomment the TEST assignment line.

# TEST=--test
# SCL=--scl=ruby193 # for RHEL6
for SPECPATH in $(find origin-server -name \*.spec)
do
  PKGDIR=$(dirname $SPECPATH )
  SPECFILE=$(basename $SPECPATH)
  (cd $PKGDIR ; tito build --rpm $TEST $SCL)
done
createrepo /var/www/thttpd/tito

This snippet walks the source tree and runs tito to build each package.  The last line rebuilds the YUM repository metadata from the packages present.

You can build a single package by moving to the root of the package tree and running tito manually. You'll also have to re-run createrepo each time you update a package.  If you've rebuilt a package but yum claims there's no update available check that.

Which reminds me, if you're using yum for frequent updates (more than once a day), you'll also have to clear the metadata on the client machine so that it sees the updated packages

client# sudo yum clean metadata

Building Test Packages

Tito builds not from the most recent commit. That is it ignores files in the workspace which have not been committed. It also requires at least one initial tito tag to operate.

Tito builds test packages by creating a temporary commit and tag. This allows it to create a package with a unique name for each test build. Each time you make a change and rebuild, a serial number is auto-incremented so that yum will see the new package as an 'update' and accept it in preference to any currently installed version.

Configuring a yum repo on the install host

You can supersede the stock Fedora or RHEL OpenShift package repositories by placing a new repo file in /etc/yum.repos.d

/etc/yum.repos.d/openshift_buildtest.repo
[openshift_buildtest]
name=OpenShift Build/Test repository
baseurl=http://build.example.com/tito/
enabled=1
gpgcheck=0

Considerations for RHEL6

There are two significant differences between Fedora and RHEL6 when creating a build box.

Firewall and Services on RHEL6

On RHEL6 systemd and firewalld are not available. Use iptables and lokkit instead of firewalld and firewall-cmd to open the TCP port for HTTP. Use service and chkconfig instead of systemctl to control services.

# Open Firewall for HTTP
sudo lokkit --service=http

# start and enable thttpd
chkconfig thttpd on
service thttpd start

RHEL6, OpenShift, Ruby/Rails versions and Software Collections

RHEL6 is .. special. OpenShift is written in Ruby 1.9.3 and Rails 3. These didn't exist or weren't stable when RHEL6 was created. Different Ruby versions don't play nicely on a single system (there have been at least 3 attempts I can find to get Ruby 1.8 and 1.9 to co-exists like Python 2 and 3. All have thrown up their hands in frustration). Given that, heroic measures were required to get them to run on RHEL6. Those heroic measures are called Software Collections. better known as "SCL".

What SCL does is provide a means to repackage software and run it in a special environment that isolates it from the rest of the system. The SCL team has re-packaged over 500 packages to run in the ruby193 environment on RHEL6. These are all needed to run OpenShift on RHEL6. They're also needed to build OpenShift for RHEL6.

Fortunately, the SCL and OpenShift teams have kindly provided a YUM repository for them. When you add the OpenShift dependencies repository to your YUM repo configurations all of the build dependencies will resolve. They've also added a switch to tito so that it will run your builds inside the SCL environment.

/etc/yum.repos.d/openshift-dependencies.repo
[openshift-dependencies]
name=OpenShift Dependencies
baseurl=http://mirror.openshift.com/pub/openshift-origin/nightly/rhel-6/dependen
cies/$basearch/
enabled=1
gpgcheck=0

There are other things in those dependencies repositories. There are a large number of update packages which OpenShift needs but which have not yet appeared upstream. On Fedora you'll still need the dependencies YUM repository to create the runtime hosts but you don't need them to build the OpenShift packages.

Considerations for EC2 hosting

If you're placing your build box in AWS EC2 there are a couple of additional things to consider:

1. EC2 security_policy must allow HTTP (port 80/TCP)
   Your build instance must be created with a security policy which allows port 80/TCP for your install hosts.
2. Internal Hostname
   EC2 hosts have an internal and external hostname. Both names are dynamic (unless you assign ElasticIP). If your install hosts are also on EC2 you can use the internal hostname and IP address for the security_policy.
3. External Hostname
   If your install boxes are not hosted in EC2 then you must allow all hosts on port 80 TCP and note the EC2 public hostname so that the install hosts can access the build host web server.

OpenShift Source Code Repositories

• origin-server -
  http://github.com/openshift/origin-server
• rhc -
  http://githhub.com/openshift/rhc
• origin-dependencies (SRPM repository)
  http://mirror.openshift.com/pub/openshift-origin/nightly/fedora-latest/dependencies/SRPMS/

OpenShift Dependencies RPM Repositories

• Fedora -
  http://mirror.openshift.com/pub/openshift-origin/nightly/fedora-latest/dependencies/$basearch
• RHEL6 -
  http://mirror.openshift.com/pub/openshift-origin/nightly/rhel-6/dependencies/$basearch

References

• git - http://git-scm.com/
• github - https://github.com
• thttpd - http://www.acme.com/software/thttpd/
• firewalld - https://fedoraproject.org/wiki/FirewallD
• tito - http://linux.die.net/man/8/tito
• Software Collections (SCL) - https://fedorahosted.org/SoftwareCollections/