I interact with many testers who feel they are not relevant in their career due to various things they don’t know. Probably one of the most common of those would be virtualization. The ability to utilize virtualized environments is most definitely a key skill that testers need to have in their toolkit, so let’s talk about that a bit.

In this post, I’ll get you set up with a particular virtual environment manager. In a follow on post, we’ll talk about how to use provisioning tools with that VM manager.

The Need for Virtual Environments

Image a world where you can package the same operating system that runs in production — along with any automated configuration scripts — in a form that’s easily shared among testers and developers. This is a world where anyone can spin up a production-identical environment on their own computer simply by typing simple commands. (As you’ll see, two of those commands might be vagrant init and then vagrant up.)

Imagine a world where a tester or developer can update their environments with a new software version in minutes. In this world, not just software versions, but entire components such as databases and web servers, can be provisioned nearly instantly and with zero configuration overhead with just one command. (That command might be vagrant provision.)

This is a world with localized development and test environments. The reason this world is attractive is that centralized development and test environments create bottlenecks.

This is a world where continuous integration and continuous delivery are made flexible and consistent due to operating system agnostic synchronized environments that are sandboxed, fully provisioned and that serve as disposable computing resources. Given the mention of continuous integration and continuous deliver, you can probably see that this is a world that can support a DevOps approach very cleanly.

Virtual Environments

A virtual environment is a means of delivering computing resources that are independent of physical machines. A virtualized environment means that instead of having to fight with configurations when working on other projects, each project can simply have its own virtualized environment. What you then need is something that helps you manage these environments.

There are various tools that can help with this but testers would serve themselves well by learning Vagrant.

Vagrant allows you to build and manage lightweight and portable virtualized environments and provide a simple interface to configure those environments. This interface involves storing configuration in simple declarative text descriptions. That means these text files (and thus the environments they describe) are easy to share with colleagues, which helps keep everyone up-to-date as the project changes.

Vagrant provides a command-line interface and a common configuration language, which allows you to easily define and control virtual machines which run on your own operating systems — whatever those happened to be — but which tightly integrate, allowing you to define how your own machine and the virtual machine interact. This can involve syncing folders such that the project code on your computer is synced so that it runs on the Vagrant development environment. For example, you may have test automation code that you run on your machine but that actually executes against the virtual machine.

Vagrant uses a concept called providers to integrate with the third-party virtualization software. It’s that software that provides the virtualized machines for your virtual development environment. The default provider is for Oracle’s VirtualBox. VirtualBox is a graphical program that lets you visually create virtual machines, allocate resources, load external media such as operating system CDs, and view the screen of the virtual machine. Vagrant essentially wraps on top of this and provides a command-line interface along with integration of additional tools (including provisioners such as Puppet and Chef), so that you don’t need to worry about how VirtualBox works or what to do with it. Vagrant abstracts away those details for you.

Having brought them up, I’ll state here that Puppet and/or Chef are provisioning tools. Specifically, they are configuration management tools that use a declarative syntax to describe the desired state of a target environment and allow this description to be executed to create that state on the target machines. You’ll sometimes hear this referred to as a target topology, where “vagrant files” and “puppet/chef manifests” are used to spin up virtual environments. Rounding out the tool pantheon, continuous integration tools (like Jenkins) then listen to changes in the version control of these artifacts and can further the provisioning process based on specific actions, such as build commits.

Installing Vagrant

Vagrant can be installed on Linux, Windows, and Mac OS X and behaves identically on each OS. The tool does use the Ruby programming language for its execution, but the package includes an embedded Ruby interpreter so you don’t have to install this separately. If you do have an existing Ruby implementation set up, the Vagrant one won’t conflict with it. You will need a virtualization tool such as Oracle’s VirtualBox. The Oracle VirtualBox provider is available for free and is included built-in with Vagrant but do know that there are other provider options, such as for VMWare, Microsoft Azure, and Amazon Web Services/Elastic Compute Cloud, as well as others.

VirtualBox (https://www.virtualbox.org/) is an open source tool sponsored by Oracle, which lets you create, manage, and use virtual machines on your own computer. First download the installer from the VirtualBox downloads page (https://www.virtualbox.org/wiki/Downloads). Obviously jsut select the download that corresponds to your OS. Then install it via your OS’s installer mechanism, which will obviously differ. There are fully detailed installation instructions for all platforms on the VirtualBox website (https://www.virtualbox.org/manual/ch02.html).

Installing VirtualBox also requires setting up a kernel driver for your operating system. The kernel driver is responsible for some low-level virtualization help, makes shared folders possible, optimizes network operating, and stuff like that. Sometimes this can interrupt your network connection so be aware of that. Also note that on Linux you may need to install the Dynamic Kernel Module System and some kernel headers. VirtualBox should indicate what you need. On Windows and Mac OS X, the relevant stuff should be set up automatically.

Installing Vagrant

Now install Vagrant. You can get the downloads from their main site (https://www.vagrantup.com/downloads.html).

Verify if Vagrant has been successfully installed by opening a command prompt (terminal on Linux/OS X or cmd on Windows) and running the Vagrant tool:

$ vagrant --version

That should give you the version number. Now try this:

$ vagrant

You should get some output regarding how to use Vagrant from the command line. If so, you are good to go.

Vagrant Boxes

Each virtual machine starts with what Vagrant calls a base box. This is a specially packaged version of an operating system with some specific configurations in place. The number of configurations and packages installed on this packaged operating system is typically minimal, usually containing only a few tools which allow it to communicate with Vagrant. It’s your job to then install the additional software on your virtual machines using provisioning tools.

As stated before, provisioning is something I’ll talk about in a follow on post to this one. For now, just know that those tools will automate the process of taking a base Vagrant box and converting it into an environment suitable for your project. For example, this might install software such as a web server and a database server and configuring any appropriate programming languages. All of that could be in service to allowing some automated tests to run against a web application running on the web server and using the database, as just one example.

Boxes are basically the package format for Vagrant environments and contain already-installed operating systems. You can see a list of base boxes (http://www.vagrantbox.es/). You can also search for boxes (https://atlas.hashicorp.com/boxes/search).

Initialize a Vagrant Box

For demonstration purpose, I’ll use a standard base box here. First make sure you are in a folder/directory that you want to convert to a new Vagrant project. Just create one for playing along purposes. It doesn’t matter what you call it. Then enter the following command from within that directory:

$ vagrant init precise64 http://files.vagrantup.com/precise64.box

This runs the init subcommand within Vagrant and instructs Vagrant to create a new project with a configuration set up to use the box named precise64. If the box is not found upon start up, Vagrant will import the box located at http://files.vagrantup.com/precise64.box when the Vagrant environment is booted for the first time.

The designation “precise64” refers to Ubuntu Precise, 64-bit version, which is one of the so called LTS (Long-Term Support) editions. Do note that you can use any name that you wish. The name you specify can then be used within other new and existing projects to refer to this base box, which is an important point although not one we need to worry about for this post.

When the above command is run, assuming all goes well, you will be told that a Vagrantfile has been created.

Vagrantfile is the configuration file, which defines how Vagrant should use the project (operating system to be used, virtual machines to boot up, synced folders, forwarded ports, and so on). More specifically, the initialization of a new project always creates a file named Vagrantfile within your project directory. When we go to boot a Vagrant virtual environment, Vagrant looks for this configuration file to determine what to do. Because everything related to the Vagrant environment is either within this file or the provisioning (i.e., Puppet or Chef) files within your project, it’s very easy to maintain the environment under version control and share it with your colleagues.

Open up the Vagrantfile and get a feel for its contents. The Vagrant configuration file is written in Ruby, and the default Vagrantfile you get is mainly filled with comments that show some of the ways that you can customize the file. Your file, with the commented lines removed, likely should look something like this:

You can see two values are pre-filled for you and these are based on what you specified when you ran the init command. The first simply names the box. The second specifies where the box can be downloaded from, if need be. To explain that, if you are working with a new Vagrant project, which uses a base box you haven’t installed yet, you need a mechanism for Vagrant to find and download the base box for you. This box_url line tells Vagrant where it can download a copy of the box named in the box parameter. When the VM is booted for the first time, if the base box is not found, then it will be downloaded from the URL provided. This can either be a web address or a path to a file on the file system or network.

You might wonder about that “configure(2)” bit. Vagrant supports two versions of configuration, known as V1 and V2 configuration. Without getting into a ton of detail that you won’t need anyway, V1 configuration is the configuration syntax from Vagrant 1.0.x, the first stable release of Vagrant. V2 configuration is the configuration syntax for what will become Vagrant 2.0, which isn’t released yet. Technically the V2 configuration is experimental and actively changing while development toward 2.0 is done and that means that V2 is not “stable.” But it’s also not unstable. The point here being you can remove the 2 and you’ll use the V1 configuration if you do run into issues.

Keep in mind that the box required is specified on a per-project basis via config.vm.box in the Vagrantfile. That means you can have different project folders, all of which will have their own Vagrantfile and all of which can specify their own box. Do note, however, that multiple Vagrant environments often share the same underlying box (by referring to its name, such as “precise64”), so Vagrant manages boxes globally, unlike Vagrantfiles, which are managed on a per-project basis. You can see that file modifications in one Vagrant environment never affect another.

Manage Your Boxes

You can manage Vagrant boxes using the vagrant box subcommands. The list subcommand will list the boxes installed within Vagrant along with the provider that backs the box:

$ vagrant box list

At this point you have nothing in place. You have to add a box.

$ vagrant box add precise64 http://files.vagrantup.com/precise64.box ––force

The --force flag tells Vagrant to remove a pre-existing box with the same name if there is one. This process may take a while, as most Vagrant boxes will be somewhat large. What’s being downloaded here is a virtual hard disk. Once downloaded, the box will be extracted and available for you to use in your Vagrant projects. These virtual hard drives are installed on your local machine, of course. For example, on Windows 8 you’ll find it in /Users/jnyman/.vagrant.d/boxes (obviously with your own user name replacing mine) whereas on Mac OS X you’ll find it at ~/.vagrant.d/boxes/.

Note that technically you didn’t have to run the above command. You could have just run the vagrant up command, which would have run this fore you if the box had not already been added. I’ll talk about vagrant up in a bit. Now let’s list again:

$ vagrant box list

The precise64 box will show up and, in parentheses, the provider will be shown that backs the box. This essentially means the provider that will be hosting the box during its execution.

Because Vagrant runs the virtual machines without a user interface, it’s nice to be able to check the current status of a box. You can do that like this:

$ vagrant status

Currently you’ll be told that the environment has not yet been created.

Start the Box

Now that you have a project initialized, you need to be able to control your guest machine. The “guest” here being the virtual machine that your own machine is hosting. At the moment, all you have is a Vagrantfile file, which defines the configuration for the project. First, let’s power up the virtual machine:

$ vagrant up

A Windows Digression

Note that if you are on Windows, you are likely going to have an issue here. If you are running Windows, you may (and most likely will) get an error about VT-x is disabled in BIOS.

Actually, you’ll probably just get a generic error at the command line about how the guest machine entered an invalid state. But if you have VirtualBox open when you run the vagrant up command, you’ll see the more specific error. What this error means is you need to enable virtualization on Windows. Many (in fact, most) machines these days come with this disabled by default. This unfortunately requires you making a change to your BIOS.

I recommend getting Speccy (https://www.piriform.com/speccy). There is a free version which is all you need. Once you install it, check the CPU. You’ll see that virtualization is supported but disabled.

See the virtualization entry there? You need to make it enabled. Where these settings are in your BIOS can totally depend on the BIOS and other factors. Lately many BIOS providers put it under Security. You should see two settings like: Virtualization Technology and VT-d Feature. Make sure those are set to Enabled. Now reboot and check Speccy to make sure that your settings look like this:

I would give better instructions on this but each machine may have different ways of getting to your BIOS and specifics about where things are stored.

Back To Starting

After you run vagrant up, you won’t actually see anything since Vagrant runs the virtual machines headless — i.e., without a graphical user interface — by default. But if you look at the running processes on your machine, you should see a VBoxHeadless process running. That is your virtual machine.

Also, as part of this process, Vagrant creates a directory named .vagrant/ in your project directory that is used to maintain some state for Vagrant. Incidentally, you should make sure that this directory is ignored by your version control system.

You can check the status again with vagrant status and you should see a different response to that this time, assuming that the environment started up.

Connect To Your Box

Make sure you are able to power up your box with the vagrant up command. Vagrant will do quite a bit at this stage. It will:

• Copy the base box. If the box doesn’t exist, it will download it. If the box does exist and was in a suspended state, it will resume it.
• Create a new virtual machine with the relevant provider. The default provider will be VirtualBox.
• Forward any configured ports. By default, it will forward port 22 (SSH) on the VM to port 2222 on the host. This will allow you to connect to the VM.
• Boot the VM
• Configure and enable networking.
• Map shared folders between the host and the guest. By default, it will map the folder containing the Vagrant project to /vagrant on the guest machine.
• Run any provisioning tools that are set up.

To make sure this all worked, Vagrant can connect to the VM over SSH. Alternatively, you could use SSH to connect to localhost with port 2222, which will tunnel into the VM. To use the Vagrant approach, just do this:

$ vagrant ssh

If you are running Vagrant on Windows, we certainly won’t have a built-in SSH client. You can use a client like PuTTY to connect. PuTTY can be downloaded from http://www.chiark.greenend.org.uk/~sgtatham/putty/. More information is available on the Vagrant website for configuring PuTTY to work with Vagrant (http://docs-v1.vagrantup.com/v1/docs/getting-started/ssh.html). You could also use OpenSSH via the Git Bash shell if you have Git for Windows installed. Likewise, a Cygwin installation will also work just fine for you.

This is a full-fledged SSH prompt inside a completely sandboxed virtual machine. If this works, you have a functioning virtual machine running an Ubuntu environment. To exit the SSH session, just type exit. This will put you back into your terminal on the host machine.

Controlling the Box

At this point, let me just cover a few things you could do. You can suspend the box:

$ vagrant suspend

What this basically does is stop the virtual box from consuming any machine resources (like memory or CPU) but otherwise leaves it intact, most important in terms of the disk space it is consuming. Do keep in mind that with this option disk space is needed to store the entire contents of the guest machine’s RAM so that it can be properly resumed. This means that for a guest machine with 2 GB of RAM, two extra gigabytes of disk space is needed to suspend it. You can resume the box if you have suspended it:

$ vagrant resume

You can shut down the box entirely:

$ vagrant halt

Unlike suspending, the RAM is not preserved, since the machine was fully shut down. This means that when you begin working again, you’ll have to make sure the necessary processes such as web servers and databases are started again. Also note that hard drive space is still being consumed. In this case, you would have to use vagrant up to get the box back and functioning.

You could also destroy the machine entirely if you want:

$ vagrant destroy

With this, the host system is left in a pristine state. No extra hard disk space is taken up and no guest-related processes are running to consume CPU and RAM. This means all files will be lost — except those that are in a synced folder. This would entirely remove the base box from your machine which means it would have to be redownloaded if you then used vagrant up again.

As a note, this issue of what gets destroyed can be tricky.

For example, on a Mac OS X, you have two locations where information is stored. The virtual hard disk image that serves as the base box is stored in ~./vagrant.d/boxes however your actual implementation of that image is stored in /Users/jnyman/VirtualBox VMs. The name of this implementation will (by default) by the name of your project directory suffixed with a timestamp. When you issue the destroy command, only the full disk information in the latter directory is removed.

On Windows, you also have two locations where information is stored. The virtual hard disk image that serves as the base box is stored in Users\jnyman\.vagrant.d\boxes however your actual implementation of that image is stored in Users\jnyman\VirtualBox VMS. Again, the specific directory there will be the name of your project suffixed with a timestamp. In the case of Windows, however, it appears that sometimes the destroy command does not in fact remove the implementation and this seems to be if your server unexpectedly terminated.

Interacting Between Host and Guest

Without any form of integration between the host machine and the guest, you would have a virtual server running on top of our own operating system, which may not be particularly useful. Generally you’re going to need your own machine to be capable of integrating tightly with the guest virtual machine.

Syncing Folders

One approach is to use synced folders. Synced folders allow you to share a folder between the host and the guest. Shared folders let users of Vagrant edit files using their own editor on the host machine, and have these changes synced into the virtual machine automatically.

By default, Vagrant shares the folder containing the Vagrant project as /vagrant inside the virtual machine. Let’s test that out. After SSHing into the virtual machine, this can be verified by listing the files in that directory:

$ vagrant ssh
$ ls /vagrant/

That should show you your Vagrantfile. But if you put other files in that directory, you will see them show up as well. If a file is created on the host machine or within the Vagrant machine, the changes will be mirrored from host to guest and vice versa.

You can use the following in your Vagrantfile to sync more folders:

The first parameter is the path to the folder on your machine while the second is the mount point on the VM. If you use a relative path on your machine, it would be relative to the project folder, meaning the folder where your Vagrantfile is stored.

Forwarding Ports

Another technique is to use use port forwarding. A forwarded port exposes a port on the guest machine as a port on the host machine. There is a line in the Vagrantfile that allows for this:

If you map the web server port (80) on the VM to port 8080 on the host — which the above does — then visiting http://localhost:8080 on your own machine would show you the web service or application that you are running on the guest.

Please keep in mind how powerful this is and what this provides you. Developers and testers can continue using their own browser and development tools to access their project, while the web application code itself and all of its dependencies run isolated within the virtual machine.

You will find a commented line in your Vagrantfile that matches the above. Make sure to uncomment it. If you change any of these settings while your box is running, you can use the following command:

$ vagrant reload

That will bring in the new changes. Specifically, a reload halts the machine, and then starts the machine again with the new configuration. It skips the initial step to clone the box, since the machine is already created.

To show that this forwarded port works, let’s start a simple web server from within the virtual machine and access it from a browser on the host machine. First SSH into the box and then we’ll call up Python’s internal web server. We’ll start this web server in the vagrant directory.

$ vagrant ssh
$ cd /vagrant
$ sudo python -m SimpleHTTPServer 80

With that done, open a browser to http://localhost:8080 on the host machine — i.e., your own computer. What you should see is a directory listing of /vagrant, served from the guest machine.

Pretty cool, huh? You can stop the simple server by just doing a CTRL+Z, which stops the server. You can then logout of the SSH session.

Provisioning

Finally, one of the key concepts within Vagrant is provisioning. Puppet and Chef are both third-party tools which Vagrant supports out-of-the-box. As I mentioned earlier these provide specific languages for configuring servers in an agnostic way that can be used for different operating systems. I’ll cover provisioning in another post since it gets a little more involved.

However if you’ve never done virtualization before, you should feel good right about now! You learned how to use a key virtualization tool (Vagrant) and set up a box that you could use for a development or test environment.