This is the second of a two-part series on moving a data center. Part one covered planning, organizing and preparing for major data center projects. This part describes getting the job done.
Given critical nature of these efforts, don’t use them for on-the-job training. If staff isn't experienced in disciplines required, bringing in consultants/contractors that have done this before, in the same region, will save money and avoid problems in the near and long term.
Whether new and old data centers are next door or around the world, you can count on a key player to be in the wrong place at an inopportune time. So arrange secure remote access between data centers and from offsite, allowing full control at the new location before the move and the old location afterward.
Site preparation is a joint exercise involving every vendor whose equipment uses electricity. Basics include checking outlets before anything is plugged in. Years ago, one vendor's equipment was burned out when the field manager didn't check power. In another case, an incorrectly wired three-phase outlet destroyed a 3211 printer by driving its print train motor backward.
Plan the Execution to Avoid Surprises
Plan execution requires rigor, flexibility, coordination, reporting and abundant and ongoing testing. Conduct status meetings at appropriate intervals with progress reviews, problem updates, and minutes widely distributed with action items noted and assigned. Ensure that nobody is left out or surprised.
Though it's unpleasant to consider, designate a few project milestones for fallback, go/no-go decisions or point-of-no-return evaluations. Don't look backward while proceeding but ensure you're never stuck unable to either finish or revert.
If using the move to upgrade storage technology, run test systems on these devices before moving real work there. Perform dry runs on your move plan as circumstances allow. If network staff can support this, do a complete rehearsal with production system at new site while isolated from the real world to prevent someone from trying to do real work there.
Track software license dates and options to ensure that plan changes or delays don't run out of processing permission. Insist that vendors flexibly and economically accommodate parallel/transition operation because production will only run in one data center at a time.
When move/build/upgrade is complete, have applications teams exercise their software before enabling anyone else, and verify all components' internal and external connectivity. Test/validation scripts created before the move should verify systems and application functionality. Compare new system benchmarks to the previous configuration's. If results fall short of expected or promised results, investigate and escalate while the project still has high visibility and vendor attention. Start by researching whether what's changed is operating correctly, then ensure that only intended changes occurred.
Achieving business continuity—for many years called disaster recovery (DR)—involves business units as much as technology functions. Keep the units involved in planning and prioritizing, but resist the temptation to change/modernize longstanding practices as part of a move/upgrade project.
Parallel operation means processing production work in both data centers. While only one set of work counts, ensure that both run everything in normal fashion, with results compared for verification. Coordinating and synchronizing that for any length of time is challenging and expensive (using data mirroring such as Extended Remote Copy solution) so either minimize the time it's necessary by making it as inclusive as possible, or do a time-point full cutover.
Be Flexible and Organized
Data center projects aren't quite like war, where every plan is good until the first shot is fired. Nevertheless, it's more surprising when everything goes according to a plan than when it doesn't. So anticipate problems and keep diagnostic tools and debugging procedures handy. Issues are most likely to arise in areas such as configuration, cabling, connections and network devices.
Consider alternatives for critical steps and be flexible. Remember MacGyver
's ingenuity and penchant for solving problems with Swiss Army knife and duct tape. VM, of course, is today's multifunction tool, being able to create configurations and devices that don't reflect reality. Encourage vendors to have contingency plans.
If you're well-practiced and successful at DR exercises, the cutover moment will be anticlimactic; you'll have seen seamless transitions before. Nevertheless, keep contemporaneous records of what's done and what happened because you'll never fully remember it later.
It's easy to be so engrossed in a major change project that usual obligations like security are forgotten. Designate a project security officer to ensure physical security (i.e., open doors monitored, data storage media encrypted for transit, in-flux network access secured, removed equipment sanitized, etc.).
Similarly, maintain normal data backup procedures, and expand them before shipping media (e.g., trucks have accidents, cargo shipments go astray) so there's no single point of catastrophic failure.
Finally, when it's done, it's not done until results are documented, successes and problems are noted and assessed for lessons learned, post mortem is distributed to everyone involved and you've written up the experience for a user group presentation or industry publication.
Gabe Goldberg has developed, worked with, and written about technology for decades. Email him at firstname.lastname@example.org.