In a previous post I introduced you to provisioning an infrastructure with Chef using a standalone component called Chef Solo. In this post, I’m going to expand significantly on that example and cover how to use Chef in the more common scenario, which is using Chef Client to talk with a Chef Server.

Before we go too much further, I recommend you get VirtualBox and Vagrant set up (which I discuss in the post Testers Need to Virtualize). You’ll also want to get the Chef Client installed on your local workstation. To do this, go to the Chef Downloads page and click on Chef Client. From there, choose your operating system and install the correct package. To check that it worked, try this:

$ chef-client --version

With this installation, you now have the chef command line interface available to you.

In its most popular incarnation, Chef functions as a client/server web service. The server is written in Erlang while the rest is Ruby-based. Chef uses a JSON-oriented document datastore and the whole Chef framework is driven via a RESTful API.

I won’t go into all of these details here but you can get a nice high level Chef overview, with details. Essentially, however, there are three high-level components. For purposes of this article, your workstation is your local development PC. The Chef Server will be what we set up and it’s where your environment will ultimately be provisioned. Finally, there are nodes. A node is a particular virtual machine (such as one hosted by Vagrant) or a physical machine that runs the Chef Client.

The basic idea is that you develop cookbooks on your workstation. You push those cookbooks to the Chef Server. The Chef Server manages nodes, which register with the Chef Server via the Chef Client, and the nodes will pull those cookbooks and use Chef Client to run the recipes in those cookbooks.

Getting a Chef Server

You can use your own Chef Server that you set up by installing it on some machine. You can also use a Hosted Chef. This is provided by Opscode and there is a free version which you can use for learning and experimenting, as long as you are willing to be restricted to five nodes. That’s more than enough for learning purposes. You need to sign up for the service. Visit Hosted Chef and register for a free trial or the free account.

For example, I registered as the user jnyman with an organization short-name called tstories. After registering your account, you can prepare your organization to be used with your chef-repo repository. Incidentally, if you need to get back to your managed account, you can go to the Managed Login.

Within your organization, you should be able to download a Starter Kit. This will give you a chef-repo skeleton. The starter kit is Opscode’s way of providing an empty Chef repository to get you started, which is really helpful.

Configuration and Validation Files

The Chef Server provides three files that must be in the Chef repository and are required when connecting to the Chef server. These files are provided as part of the starter kit that you download.

• knife.rb
• ORGANIZATION-validator.pem
• USER.pem

These files will be in the .chef directory in our chef-repo and since the starter kit was based on your particular organization and user, that means the organization validator and user file will have the correct names. (Mine, for example, are tstories-validator.pem and jnyman.pem.) All of these files can be downloaded from the Hosted Chef pages for your organization.

Hosted Chef uses two private keys called validators: one for the organization and the other for every user. You need to tell Knife where it can find these two keys in your knife.rb file so why not open that file and take a look at it. You’ll see something like this, although, of course, your details will differ:

The validation_client_name and the validation_key tell Knife about which organization to use and where to find its private key. The client_key tells Knife about where to find your users’ private key. The chef_server_url tells Knife that you’re using Hosted Chef. You will find your organization name as the last part of the URL. So the above Knife configuration is used to connect to my tstories organization on my hosted Chef site.

Using the knife.rb file and your two validators, this means that Knife can now connect to your organization hosted by Opscode. You do not need your own, self-hosted Chef Server, nor do you need to use Chef Solo in this setup.

Connect to Hosted Chef

At this point we want to connect your local workstation with your hosted Chef. To get started, you can use Knife to verify that you can connect to your hosted Chef organization. It should only have your validator client so far.

$ knife client list

Knife is the command-line interface for the Chef Server. It uses the RESTful API exposed by the Chef Server to do its work and helps you to interact with the Chef Server. In this sense, Knife is a client of the Chef Server API just as much as is Chef Client. You can more information in the Knife documentation.

Configure a Node

Vagrant is a tool I’ve introduced and used in a efw posts now. Vagrant makes it easy to launch and manage virtual machines on your local workstation. Here we’re going to create a Vagrant-managed virtual machine to act as our Node. Vagrant manages each virtual machine as a box. Opscode makes a number of Vagrant boxes available through it’s “bento” project on github.com or you can use Vagrant provided boxes. First, make sure you install Vagrant and then, in chef-repo, initialize it:

$ vagrant init

Modify the resulting Vagrantfile so that it contains the following:

Finally, run vagrant up to launch the Vagrant box. The first time you start up the box, it has to download the file from the Vagrant file store.

Now we’re going to configure Vagrant to use the chef_client provisioner. We’ll do this similarly to how I used chef_solo in the previous post. You can find more information about the chef_client provisioner on the Vagrant website.

Change your Vagrantfile to look like this:

Here the chef_server_url setting matches that exactly of the chef_server_url in the knife.rb file. The validation_client_name likewise matches the validation_client_name found in knife.rb. The validation_key_path is, of course, the location of the organization validator file, which will be in your .chef directory. Finally, the node_name can match the node_name from knife.rb but that’s not required. Essentially the node_name is simply the “friendly name” of the node in question, which matters a lot more when you have multiple nodes.

Let’s make sure the box is re-provisioned with these changes:

$ vagrant provision

This gives you a VM running on your workstation connected to your hosted chef account. Let’s check it out:

$ knife client list

Now you have two listed: your vm and the validator client. You can also check how many nodes you have connected:

$ knife node list

This will show you one node registered with hosted chef. In my case that node is jnyman_vm, but yours will be whatever you gave to your node_name. If you want some information on that node, you can do this:

$ knife node show jnyman_vm

That will show you some information about your node. You can also see that the node is now listed in your Hosted Chef instance web interface.

So the setup here is that Vagrant will manage your virtual box The Vagrantfile was told what vagrant box to run and pointed your vagrant instance at the appropriate chef server url and keys, which are effectively the credentials for authentication. This allows chef-client on the instance (running on the VM) to know how to connect to the chef-server (which is your Hosted Chef). The validator, which operates as part of this process, is another client of the chef-server.

You have to have the validator credentials — the pem files — to register a client with the server. In fact, the first time Chef runs on a node, it looks for a client.pem file. If it doesn’t find one, and it won’t the first time it is run, then it will then try to authenticate to the chef server by using the validator credentials (in my case, tstories-validator). Assuming that goes well, the chef-server then generates a client.pem and sends that to the node. The node will then, from that time on, use that client.pem. In fact, let’s check that this file exists:

$ vagrant ssh

Once on your vm, do the following:

cd /etc/chef
ls

You should see the client.pem file. Again, this file was generated by the chef server and sent down to the node. That is what is used to authenticate to the chef server from this point forward.

Of import, when you provisioned the box with chef-client, you probably got output like this:

==> default: INFO: Run List is []
==> default: INFO: Run List expands to []
==> default: INFO: Starting Chef Run for jnyman_vm
==> default: INFO: Running start handlers
==> default: INFO: Start handlers complete.
==> default: INFO: Loading cookbooks []
==> default: WARN: Node jnyman_vm has an empty run list.

Two key lines there are that the Run List is empty as are the cookbooks. The run list would contain a list of recipes to run and those recipes would be listed in various cookbooks.

Now that we’ve got our Vagrant instance connected to Chef Server we can start managing the configuration of the VM with Chef. We’ll download a number of cookbooks from the Community Site and extract them into our Chef repository. This is going to be very similar, if not identical, to what we did in my previous post on Chef Solo. The difference here is that with Chef Solo we deployed locally to our VM, whereas here we’re going to deploy to our Hosted Chef server.

So let’s do something very similar to what we did in the previous post. We’ll grab Apache. And, as before, we’ll also grab its dependencies, which are iptables and logrotate. Incidentally, you can see the dependencies for a given cookbook by checking out its Dependencies tab in the Supermarket. For example, you can see the dependencies for apache2.

$ knife cookbook site download apache2
$ tar xzvf apache2-*.tar.gz -C cookbooks

$ knife cookbook site download iptables
$ tar xzvf iptables-*.tar.gz -C cookbooks

$ knife cookbook site download logrotate
$ tar xzvf logrotate-*.tar.gz -C cookbooks

You might be wondering if there is a better way to do this dependency management. In fact, there are various ways to help with this. Some people may point you to Librarian-Chef. Also, in many Chef tutorials you might hear about Berkshelf. However the latter has now been rolled up into the Chef Development Kit. This is an area of active development within Chef so I’m not considering it too much here, but you should definitely look into it.

Update your Vagrantfile to include port forwarding so that browsing to localhost:8080 redirects to your VM’s port 80. We’ll also add in the Chef provisioning to include Apache in the build:

Before we provision the virtual machine, however, keep in mind that unlike in the Chef Solo example, we are running this instance on the Hosted Chef or Chef Server. That means we need to make sure that our Apache cookbook is available on the server. Right now it’s just available on our local workstation. So we have to upload the apache2 cookbook as well as any other cookbooks:

$ knife cookbook upload apache2
$ knife cookbook upload iptables
$ knife cookbook upload logrotate

As a note of convenience, you could also do this:

$ knife cookbook upload --all

Now let’s provision Vagrant accordingly:

$ vagrant provision

This time you will get a lot of output, but you might notice something that has changed from the last time:

==> default: INFO: Setting the run_list to ["recipe[apache2]"] from CLI options
==> default: INFO: Run List is [recipe[apache2]]
==> default: INFO: Run List expands to [apache2]

So we now have a fresh new Ubunutu VM with Apache installed and configured and running on port 80, with our own port 8080 forwarded to the VM’s port 80. As we did with Chef Solo, let’s check it out and see if it’s working. Go to http://localhost:8080. However, you’ll find you can’t access it. What’s happening is that on Ubuntu the default site doesn’t get enabled so we have to do that ourselves. This will demonstrate passing data into the chef provisioner. Add in this to the Vagrantfile:

The chef.json section passes the specified variable values into the specified recipe file. This is exactly what I had you do in the Chef Solo post. However, to dig a little deeper this time, in your /cookbooks/apache2/recipes directory, look at the default.rb file. In there you’ll see a section like this:

This part of the cookbook handles settings for the default site, which is given a name provided by default_site_name. The status of this site is said to be equal to the value defined by default_site_enabled. For Ubuntu this defaults to false but we’ve set it to true. Don’t worry too much about the code details here, except to note that the cookbook (1) is made up of code and (2) that code is Ruby.

Now let’s provision our vm:

$ vagrant provision

Now going to http://localhost:8080 should work.

So here we’ve exactly replicated the steps we took with Chef Solo (provisioning an environment locally) but instead did this on the a remote Chef Server.

All of my posts on this topic so far have been focused on getting people familiar enough with Vagrant, VirtualBox, and Chef so that the activities themselves have started to become less mysterious. This is important for testers because there is a whole other layer to this, which is testing the infrastructure code that your developers and system administrators create as well as code you may create for your test environments.

In a future post on this topic, I’d like to explore the Chef Development Kit (CDK) which is becoming the preferred way to use Chef. Further, I want to explore the testing solutions that exist around Chef.