Creating tested GCE images using Chef+Test Kitchen+Inspec+Packer

Recently, I’ve been playing around with the concept of a half-baked server image for a quicker application deploy via an automation pipeline.  My goal was to use test-kitchen+chef-solo+Inspec to locally build and verify my new image.  Then use packer+berks+chef-solo+Inspec in my pipeline to bake, test, and upload it.
​
To test this, I wrote a quick cookbook to do some very basic configuration.  Here is the default.rb with my 2 package resources...

Next, I wrote a simple test in Chef's Inspec to test for my new packages.  I've been using Serverspec for years, so I thought I'd try something new.  I intentionally wrote this test to break by adding in a missing package 'tree' to the list of packages to be tested.

I created a simple .kitchen.yml to work with vagrant. (For those of you on later versions of the ChefDK, you'll notice I removed the alternate locations for the tests.  I'm sticking with the test-kitchen defaults in this example.)

At this point, I could kick-off a `kitchen test` command...

Since we didn't tell chef to install the tree RPM, we expected kitchen to fail our inspec tests.  No surprises there.  The next steps would be to update our default.rb recipe to make our package resource also install the tree package, then re-run `kitchen test`.  This should not be new to anyone who has used test-kitchen as part of their cookbook development workflow.  

At this point, we could probably push our code to our source control and allow the pipeline to run packer to build our new image knowing our code works in our local vagrant environment...right?

Maybe?

What if there are major differences between our local vagrant image and the base cloud image?  (selinux, anyone?)  We should probably run our tests on the new image before packer publishes it.  The one big problem with this approach is that Packer doesn't natively support a testing framework.  BUT, we can get around that by writing a shell script to install a testing framework on our new image and then run our tests!  Ta da!

Here is my packer config to do just that.

Most of the magic happens in the provisioners section.  

• The shell-local provisioner uses berks to  "vendor" our cookbook and any dependencies it might have into a specified directory on our pipeline server.  The --delete option makes sure to clean this directory before running the command.  
• The chef-solo provisioner specifies the cookbook location which we specified in the first entry and also sets the run_list.  This is where chef configures our new image.
• The file provisioner copies everything under the test directory in our cookbook (including our inspec test) to the new image.  These would be the same tests the we used to verify the image locally in test-kitchen.
• Finally, the shell provisioner runs our "run_inspec_tests.sh" script which installs the inspec rpm and then runs the tests.

Here what the "run_inspec_tests.sh" script looks like.

So what happens when we run packer with our broken test?  Let's see!

As you can see, packer builds the new image,  runs our test and it fails...woot!  
When packer gets a non-zero exit from one of its provisioners, it deletes the instance and no image is created!  

Now let's fix the recipe so our test passes and then re-run packer...

 Success!

Now we can make sure that we are only publishing images that pass our tests!  This should remove a lot of the guess work that comes with baking your own cloud images.

I have uploaded my code to git here.
​
Download it and give it a try.  If you have any questions or ideas to make this post better, please leave a comment.

Thanks for reading.

- John