Really, I’ve been with the foundation for a year and a month, so this post is a little late. My first post on this was also a month late; so, at least this is an update of what I did for the past year.
Question is, did I meet the goals I set for last year? Here’s my goals from last year with the results:
- Learn how to deal with Wikimedia infrastructure during an emergency situation, so that we don’t have to constantly wake up the same folks
- Success. I handled a number of outages without needing to page anyone.
- Partially complete. Tesla still exists and is still running a number of virtual machines that we use for test and development. A replacement is not up yet. That said, the scope of this goal changed dramatically as I started thinking of what we really needed. The plans for this are far more ambitious now, and so far it’s going well. The architecture is mostly up, we’re starting to give people accounts, and by mid-October it should be fully ready for use. We are having a hack-a-thon in New Orleans that will be primarily focused on this project. Ubuntu Ensemble will be working with us inside of this environment, and so will OpenStreetMap. I gave a keynote about the architecture of this environment at the OpenStack Design Summit and Conference, and a gave a talk at Wikimania about the future of development and operations at Wikimedia Foundation using this environment. I recently set up Gerrit, imported our puppet repository, and released it to the public, as a first major step in the opening of the Labs project.
- Partially complete. The Labs project needed a fair amount of services, and their authentication needed to be integrated. Subversion user accounts were already being stored in LDAP, so I integrated all Labs services with Subversion LDAP. I switched our puppet repo from Subversion to Git (in Gerrit), so ops code review now has integrated authentication.
- Skipped. This was one of my ambitious goals that was an “if I got time for it” goals. I didn’t get time for it. This is something we need for Labs, so it’s still on my plate, and will likely be a major goal of next year. We will be discussing this at the New Orleans hack-a-thon.
- Dropped. Support for Selenium from the Wikimedia Foundation was dropped a few months ago. We were focusing on unit tests. We were writing very few Selenium tests, and the overhead of maintaining this cluster was basically wasted effort. Because of that we decided to scrap it and support other things instead.
- Success. This is exactly why the scope of the virtualization project expanded so greatly. I think the goals of Labs address this very well. The entire environment is built around the idea of volunteer ops. Also, with the move of the puppet repository into a public Git repository, we can easily take changes from volunteers.
- I always do so poorly with this one :D. Overall I’d say this goal has been completed.
What did I do that was outside of these goals?
- Created an HTTPS cluster and enabled HTTPS in experimental mode on a couple of our larger sites.
- Moved, upgraded, and puppetized our Subversion set up. I moved it in the middle of a hack-a-thon too ;).
- Added backups for our miscellaneous database servers
- Daemonized and puppetized a bunch of things that were being started in screen sessions
- Packaged a bunch of things that were either being installed unpackaged, or new things we needed that weren’t packaged yet
- A ton of other small tasks – see our Server Admin Log for this information.
What are my goals for next year?
- Continue with the Labs project. Finish set up of test/dev Labs, and begin work and make major progress on tool Labs.
- Hire a devops contractor for work on Labs
- Build a devops community around the Wikimedia architecture
- Finish the HTTPS project. This will hopefully be complete from the ops perspective by the end of this year.
- On-board new employees
- Enable OpenID as a provider and oAuth on Wikimedia (this goal still needs consensus)