I had a situation recently where a number of live migrations failed in a truly nasty way. The live migration failed part way through, but didn’t properly back-out the changes. This left the instance running nowhere, but in a “migrate” state in the database. I tried to reboot the instance, which then left the instance in the “running” state.

Of course, the instance wasn’t actually running anywhere and the reboot command wouldn’t start the instance, because it thought it was running. The logs complained that the instance wasn’t running whether I tried to restart the migration, or reboot. What a full of fail situation.

The OpenStackManager extension is a web interface for OpenStack, and a manager for a fully integrated test and development network being written primarily for Wikimedia Foundation use.

I’ve been busy enough lately working on our OpenStack infrastructure that I haven’t made an OpenStackManager release in a while. Over the past seven months I’ve continued to make small changes to the software, and the past few weeks I’ve added features I feel deserve another release.

This is a bugfix and features release. Major changes include compatibility for cactus and diablo releases of nova, and 1.18 compatibility for MediaWiki. The changes in this release focused mainly on making workflow easier. Here’s a complete list of changes:

For the proper automation of a service using puppet, it’s necessary to ensure the service can be installed repeatedly, and that the service is fully up and ready when it is built. To ensure this, I’m using the following process, using nova:

1. Create an instance and use it to do experimentation with the service.
2. Document the service, along with the installation process on wikitech, after ensuring the service is working properly.
3. Create a second instance. Following the documentation written, puppetize the service.
4. Create a third instance. Ensure the puppetized service runs properly when initialized from scratch.

As mentioned in an older post, I’m building a test and development environment using OpenStack. The environment is intended to be fairly integrated. Part of this integration is a consistent working environment between instances in a project. Providing home directories via NFS is the easiest way of ensuring this consistent working environment.

The last day of the OpenStack Design Summit and Conference it was announced that OpenStack would now be run as a foundation, rather than as a corporate subsidiary of Rackspace. I believe this is an important step in the growth and stability of this project, and am very excited about the plans.

I recently added a Gerrit instance at Wikimedia for doing code review and git repository management for our puppet repository. Since I’m using a new tool, I need a new working environment to go with it.

Our puppet configuration is broken into two environments (production and labs), each environment having a public and private repository. Inside of these repositories our configuration is broken into three main directories: files, templates, manifests. I often jump between repositories, directories within repositories, and branches within different screen windows.

The information I need to know is this:

• The current working directory in each screen window

Really, I’ve been with the foundation for a year and a month, so this post is a little late. My first post on this was also a month late; so, at least this is an update of what I did for the past year.

Question is, did I meet the goals I set for last year? Here’s my goals from last year with the results:

1. Learn how to deal with Wikimedia infrastructure during an emergency situation, so that we don’t have to constantly wake up the same folks
• Success. I handled a number of outages without needing to page anyone.

There’s something major missing in the Debian and Ubuntu world, and that’s a decent package management system. Yes, I’m going to get responses to this like “Why don’t you use Chef, or Puppet to do package management”, and I’ll give you the quick response for this: Though Chef and Puppet do configuration management well, they do not do package management well.

As mentioned before, I’m building a test and development environment for the Wikimedia Foundation using OpenStack and MediaWiki. I wrote a MediaWiki extension for this project, and have added basic Semantic MediaWiki support to this extension. People have asked me a number of times why I chose to use MediaWiki to build the OpenStack manager, and this post will be an example of why I went this route.

Most environments I use restrict login to shell servers. This makes a lot of sense, from the perspective of limiting attack surface areas. I’ve adjusted my working environment over time to take advantage of this type of environment.

The basics

First of all, security in an environment like this is important. I restrict my login to key based login. I protect my key with a password. I don’t put my key on removable media. I keep a strong password on my key. I use full disk encryption for the system my key is on. I keep an encrypted backup of my key in a safe, located somewhere my computer isn’t kept.