See the new one on Verifying the Mongod DataStore with the Rails Console: Mongoid Edition
In the last post I showed how I'd verify the configuration of the OpenShift Bind DNS plugin using the Rails console. In this one I'll do the same thing for the DataStore back end service. (not strictly a plugin, but hey...).
DataStore Configuration
Right now the DataStore back end service is not pluggable. The only back end service available is MongoDB. I've posted previously on how to prepare a MongoDB service for OpenShift. Now I'm going to work it from the other side and demonstrate that the communications are working.
Since the DataStore isn't pluggable, it isn't configured from the
/etc/openshift/plugins.d directory. Rather it has it's own section in the /etc/openshift/broker.conf (or broker-dev.conf).This is just the relevant fragement from
broker.conf... #Broker datastore configuration MONGO_REPLICA_SETS=false # Replica set example: "<host-1>:<port-1> <host-2>:<port-2> ..." MONGO_HOST_PORT="data1.example.com:27017" MONGO_USER="openshift" MONGO_PASSWORD="dbsecret" MONGO_DB="openshift" ...
These are the values that will be used when the broker application creates an
The
OpenShift::DataStore (and OpenShift::MongoDataStore) object.The DataStore: Abstract and Implementation
At one point the
OpenShift::DataStore was intended to be pluggable. At some point the concept of an abstracted interface was dropped and the tightly bound MongoDB interface was allowed to grow organically. The remains of the original pluggable interface are still there. Both source files live now in the openshift-origin-controller rubygem package.The
OpenShift::DataStore class still follows the plugging conventions. It implements the provider=() and instance() methods. The first takes a reference to a class that "implements the datastore interface" and the second provides an instance of the implementation class all pre-configured from the configuration file.Observing MongoDB
Unlike named MongoDB writes to its own log by default. The logs reside in
MongoDB also has a command line tool that can be used to interact with the database as well. The CLI tool is called mongo. I can invoke it like this:
This shows an initialized database, but no OpenShift data has been stored yet. The two existing collections are the system collections. OpenShift will add collections as needed to store data.
With these two mechanisms I can observe and verify access and updates from the broker to the database through the OpenShift::DataStore object.
/var/log/mongodb/mongodb.log. (as controlled by the logpath setting in /etc/mongodb.conf.) Verbose logging is controlled in the mongodb.conf as well. For this demonstration I'm going to enable that by uncommenting the line in /etc/mongodb.conf and restarting the mongod service.MongoDB also has a command line tool that can be used to interact with the database as well. The CLI tool is called mongo. I can invoke it like this:
mongo --username openshift --password dbsecret data1.example.com/openshift MongoDB shell version: 2.0.2 connecting to: data1.example.com/openshift > show collections; system.indexes system.users
This shows an initialized database, but no OpenShift data has been stored yet. The two existing collections are the system collections. OpenShift will add collections as needed to store data.
With these two mechanisms I can observe and verify access and updates from the broker to the database through the OpenShift::DataStore object.
Creating an OpenShift::DataStore Object
I'm going to create the OpenShift::DataStore object in the same way I did with the
OpenShift::DnsService object. I call the instance() method on the OpenShift::DataStore object.cd /var/www/openshift/broker
rails console
Loading production environment (Rails 3.0.13)
irb(main):001:0> store = OpenShift::DataStore.instance
=> #<OpenShift::MongoDataStore:0x7f9e42ed4918
@host_port=["data1.example.com", 27017], @db="openshift", @user="openshift",
@replica_set=false, @password="dbsecret",
@collections={:application_template=>"template", :user=>"user", :district=>"district"}>
irb(main):002:0>
Now I have a variable named db which contains a reference to an
OpenShift::MongDataStore object. I can see from the instance variables that it is configured for the right host, port, database, user etc.Checking Communications: Read
Now that I have something to work with its time to see if it will talk to the database.
The interface to the DataStore is much more complex than the DnsService interface is. Since we're only checking connectivity that's not a problem. Once I've checked connectivity, I can craft more checks of the DataStore methods themselves later.
The DataStore has a couple of methods that expose the Mongo::DB class that's underneath. With that I can force a query for the list of collections currently available in the database. If the broker service has not yet been run and users and applications created then only the system collections will exist. In the example below there are only two collections.
On the MongoDB host I can confirm that there are indeed two collections.
Finally I can check that the broker app really did issue that query and get a response:
The interface to the DataStore is much more complex than the DnsService interface is. Since we're only checking connectivity that's not a problem. Once I've checked connectivity, I can craft more checks of the DataStore methods themselves later.
The DataStore has a couple of methods that expose the Mongo::DB class that's underneath. With that I can force a query for the list of collections currently available in the database. If the broker service has not yet been run and users and applications created then only the system collections will exist. In the example below there are only two collections.
rails console
Loading production environment (Rails 3.0.13)
irb(main):001:0> store = OpenShift::DataStore.instance
=> #<OpenShift::MongoDataStore:0x7f1ac50ac698
@host_port=["data1.example.com", 27017], @db="openshift",
@user="openshift", @replica_set=false, @password="dbsecret",
@collections={:application_template=>"template", :user=>"user", :district=>"district"};gt;
irb(main):002:0> collections = store.db.collections
=> [#<Mongo::Collection:0x7f1ac5096ac8 @cache_time=300,
...
@pk_factory=BSON::ObjectId>]
irb(main):003:0> collections.size
=> 2
irb(main):004:0> collections[0].name
=> "system.users"
irb(main):005:0> collections[1].name
=> "system.indexes"
On the MongoDB host I can confirm that there are indeed two collections.
mongo --username openshift --password dbsecret data1.example.com/openshift MongoDB shell version: 2.0.2 connecting to: data1.example.com/openshift > show collections system.indexes system.users
Finally I can check that the broker app really did issue that query and get a response:
...
Thu Dec 6 14:06:29 [conn2] Accessing: openshift for the first time
Thu Dec 6 14:06:29 [conn2] authenticate: { authenticate: 1, user: "openshift",
nonce: "d2083e4185cb7d22", key: "c7c3628fe64eb1aedaaf4c87a4d5e723" }
Thu Dec 6 14:06:29 [conn2] command openshift.$cmd command: { authenticate: 1, u
ser: "openshift", nonce: "d2083e4185cb7d22", key: "c7c3628fe64eb1aedaaf4c87a4d5e
723" } ntoreturn:1 reslen:37 5ms
Thu Dec 6 14:06:29 [conn2] query openshift.system.namespaces nreturned:3 reslen
:142 0ms
...
Checking Communications: Write
Now that I'm convinced that I'm connecting to the right database and I'm able to make queries, the next check is to be sure I can write to it when needed.
Since the database has not yet been used, it's empty. I want to be careful regardless not to mess with any real OpenShift collections. I'll create a test collection, write a record to it, read it back and drop the collection again. If I do this in a consistent way I can use this test at any time to check connectivity without danger to the service data.
I'm using the ruby Mongo classes underneath the OpenShift::MongoDataStore class, so I'll have to look there for the syntax. The Mongo::DB class has a create_collection() method which will do the trick. I'll issue the command in the rails console, then check the MongoDB logs and view the list of collections using the mongo CLI tool.
Create a Collection
First, the create query (entered into an existing rails console session):
irb(main):005:0> store.db.create_collection "testcollection" => #<Mongo::Collection:0x7f76567e9ae0 @cache_time=300,... ... @name="testcollection", @logger=nil, @pk_factory=BSON::ObjectId> irb(main):006:0>
Next I'll check logs:
...
Thu Dec 6 14:37:13 [conn4] run command openshift.$cmd { authenticate: 1, user:
"openshift", nonce: "665cedb4baf82b0d", key: "eec7b08761151c858c14058c2629dee6"
}
Thu Dec 6 14:37:13 [conn4] authenticate: { authenticate: 1, user: "openshift",
nonce: "665cedb4baf82b0d", key: "eec7b08761151c858c14058c2629dee6" }
Thu Dec 6 14:37:13 [conn4] command openshift.$cmd command: { authenticate: 1, u
ser: "openshift", nonce: "665cedb4baf82b0d", key: "eec7b08761151c858c14058c2629d
ee6" } ntoreturn:1 reslen:37 0ms
Thu Dec 6 14:37:13 [conn4] query openshift.system.namespaces nreturned:3 reslen
:142 0ms
Thu Dec 6 14:37:13 [conn4] run command openshift.$cmd { create: "testcollection
" }
Thu Dec 6 14:37:13 [conn4] create collection openshift.testcollection { create:
"testcollection" }
Thu Dec 6 14:37:13 [conn4] New namespace: openshift.testcollection
Thu Dec 6 14:37:13 [conn4] adding _id index for collection openshift.testcollec
tion
Thu Dec 6 14:37:13 [conn4] build index openshift.testcollection { _id: 1 }
Thu Dec 6 14:37:13 [conn4] external sort root: /var/lib/mongodb/_tmp/esort.1354
804633.1660751058/
Thu Dec 6 14:37:13 [conn4] external sort used : 0 files in 0 secs
Thu Dec 6 14:37:13 [conn4] New namespace: openshift.testcollection.$_id_
Thu Dec 6 14:37:13 [conn4] done building bottom layer, going to commit
Thu Dec 6 14:37:13 [conn4] fastBuildIndex dupsToDrop:0
Thu Dec 6 14:37:13 [conn4] build index done 0 records 0.001 secs
Thu Dec 6 14:37:13 [conn4] command openshift.$cmd command: { create: "testcolle
ction" } ntoreturn:1 reslen:37 1ms
...
Finally I'll connect and query the database locally to check for the presence of the new collection.
This is really enough to demonstrate that the MongoDataStore object is properly configured and has the ability to read and write the database. Just for completeness I'll go one step further and create a document.
mongo --username openshift --password dbsecret data1.example.com/openshift MongoDB shell version: 2.0.2 connecting to: data1.example.com/openshift > show collections system.indexes system.users testcollection
This is really enough to demonstrate that the MongoDataStore object is properly configured and has the ability to read and write the database. Just for completeness I'll go one step further and create a document.
Add a Document to the testcollection
Since the testcollection is the most recently added, it should be the last one in the collections list in the Rails console Mongo::DB object. I can check by looking at the name attribute of that collection
irb(main):007:0> store.db.collections[2].name > "testcollection"
Now that I know I have the right one, I can add a document to it using the Mongo::Collection insert() method:
irb(main):008:0> store.db.collections[2].insert({'testdoc' => {'testkey' => 'testvalue'}})
=> BSON::ObjectId('50c0c2016892df2d56000001')
The logs show the insert like this:
Thu Dec 6 16:04:36 [conn11] run command openshift.$cmd { getnonce: 1 }
Thu Dec 6 16:04:36 [conn11] command openshift.$cmd command: { getnonce: 1 } nto
return:1 reslen:65 0ms
Thu Dec 6 16:04:36 [conn11] run command openshift.$cmd { authenticate: 1, user:
"openshift", nonce: "19ca25a92ca483ee", key: "f7ac36d2e36a3a00a91d234a59a559e3"
}
Thu Dec 6 16:04:36 [conn11] authenticate: { authenticate: 1, user: "openshift"
, nonce: "19ca25a92ca483ee", key: "f7ac36d2e36a3a00a91d234a59a559e3" }
Thu Dec 6 16:04:36 [conn11] command openshift.$cmd command: { authenticate: 1,
user: "openshift", nonce: "19ca25a92ca483ee", key: "f7ac36d2e36a3a00a91d234a59a5
59e3" } ntoreturn:1 reslen:37 0ms
Thu Dec 6 16:04:36 [conn11] query openshift.system.namespaces nreturned:5 resle
n:269 0ms
Thu Dec 6 16:04:36 [conn11] insert openshift.testcollection 0ms
And a quick CLI query to confirm that the document has been created:
mongo --username openshift --password dbsecret data1.example.com/openshift
MongoDB shell version: 2.0.2
connecting to: data1.example.com/openshift
> db.testcollection.find()
{ "_id" : ObjectId("50c0c2016892df2d56000001"), "testdoc" : { "testkey" : "testvalue" } }
Read a Document from the testcollection
In traditional database style, when you make a query, you don't get back the single thing you asked for. You get a Mongo::Cursor object which collects all of the documents which match your query. Cursors respond to a next() method which does what you would think, returning each match in turn and nil when all documents have been retrieved. The Mongo::Cursor also has a method which converts the entire response into an array. I'll use that to get just the one I want.
I won't take up space showing the log entries for this query. I know how to find them now if there's a problem.
irb(main):035:0> store.db.collections[2].find.to_a[0]
=> #<BSON::OrderedHash:0x3fbb2b27fea4
{"_id"=>BSON::ObjectId('50c0c2016892df2d56000001'),
"testdoc"=>#<BSON::OrderedHash:0x3fbb2b27fd50 {"testkey"=>"testvalue"}>}>
I won't take up space showing the log entries for this query. I know how to find them now if there's a problem.
Cleanup: Remove the testcollection
The final step in a test like this is always to remove any traces. I can drop the whole collection with a single command. This one I will confirm with the local CLI query, but the logs I'll leave for an exercise unless something goes wrong.
irb(main):037:0> store.db.collections[2].drop => true
You may notice that this was WAY too easy. Do be careful when you're working on production systems. Prepare and test backups OK?
When I look now on the CLI and ask for the list of collections, I only see two:
mongo --username openshift --password dbsecret data1.example.com/openshift MongoDB shell version: 2.0.2 connecting to: data1.example.com/openshift > show collections system.indexes system.users
Summary
In this post I showed how to access the OpenShift broker application using the Rails console. I created an OpenShift::MongoDataStore object (using the OpenShift::DataStore factory). I showed how to access the database from the CLI and where to find the MongoDB log files. With these I was able to confirm that the OpenShift broker DataStore configuration was correct and that the database was operational.
References
- MongoDB
- Ruby Mongo module
- mongo CLI database client
- Rubu on Rails console CLI
- OpenShift Origin DataStore factory class source code
- OpenShift Origin MongoDataStore class source code
This blog is apparently obsolete. The interface to the OpenShift::DataStore object has been changed and/or removed. I'm looking at how it works now and how to verify the data store connectivity with the new interface.
ReplyDeleteIt appears that the broker won't start without the MongoDB configured and will throw an error if it's not.
It turns out that, rather than decoupling the data store from Mongo, it's been integrated using the Mongiod ODM gem:
ReplyDeletehttp://mongoid.org/en/mongoid/index.html
For what it's worth, this is a great thing as it automates and abstracts the object serialization. It does mean that on start up the broker connects with the datastore so that the broker will fail to start if the datastore is not configured.
Now to figure out how to describe diagnosing it.