No matter whether called conventional wisdom, best practices or wisdom of the ages—if not formalized, documented, preserved and passed on through generations of mainframers, lessons painfully learned are diluted and lost. Sometimes nuggets are brightly rediscovered with much industry fanfare—amusing anyone who's practiced them for decades—but their periodic neglect and disuse imposes costs and inefficiencies.
Maddeningly, some gems are revalidated for every technology generation—client/server, distributed computing, cloud computing, etc.—without their history and immutability being recognized. Even worse, generation gaps are innate human nature; older generations disrespect youngsters and their favorite technologies, while youngers think that the tech world started with their first system learned. So, they don’t communicate.
Not everything should be carved on stone tablets; I haven’t programmed a keypunch drum control card in quite a while. But some truths are eternal. Mainframers should remember and pass them on! Some of these truths include:
Change happens; deal with it.
Likely VMers don't long for hand-coding DMKSNT and DMKRIO, and z/OS people don’t wish for punching boxes of Stage 1 Sysgen cards. But more subtle, yet still unproductive, tendencies can lead to ignoring or resisting new technology features. Instead, remember when the mainframe—and perhaps you—were younger, and absorbing/exploiting new features was a joy. “That’s what we’ve always done” isn’t a good argument.
When designing and coding, don’t underestimate the future.
It’s inexplicable that the 1990s saw new applications developed with Y2K doom built in. And anticipating 31-bit and beyond addressing might have reserved upper bytes, rather than needing urban renewal to reclaim them. Don’t paint yourself or your successors into a corner with shortcuts or lack of imagination. The mainframe turns 50 next year—might decisions made now be burdensome in 2064?
Find, join, form and support technology communities.
Being a mainframer means having ready-made circles of colleagues. Resist inclinations to be exclusively a back-room developer; remember that there’s strength in numbers. Destination z, discussion lists, local user groups (e.g., Washington, D.C.’s Hillgang), national groups (SHARE
, VM Workshop
), and industry conferences
, all offer education, mutual aid and opportunities to influence key vendors. Though large-crowd meetings might be daunting, everyone there was once a newbie. Conference attendees and organization members understand the need to be welcoming, and there's always a friendly face.
Good news, problems, solutions, useful websites and vendor issues are more valuable when shared. And being visible—being helpful—means that people will respond when you need them. Lurking is fine when new to a community, but the way to participate fully is to contribute.
Keep the Customer in Mind
Remember your customers and what makes them happy. As problem solvers, we have strong opinions about how things should be implemented. That's good. Regardless of who we work for or the industry sector we're in, we're all responsible for producing a common product: happy customers. Your "customer" might be your immediate manager or your CEO, people at a checkout counter, or those signing checks for software license fees. However, if you're not producing happy customers, you're not optimizing your career. To help keep those customers happy, here are some suggestions:
• Communicate—yes, again. It's difficult to overdo this. Ask questions—ask users/managers and even yourself, "Is that what you mean?" Question your and others' assumptions, shortcuts and groupthink. Verifying early beats redoing later.
• Get needed decisions, approvals, resources in advance. It’s NOT better to apologize later than to ask permission!
• Design and code reviews aren’t formalities; use them to avoid blind spots.
• Don't use an editor to browse code or data—one slip can hurt.
• Use standard system services rather than developing custom interfaces.
Document first. Mike Cowlishaw, former IBM Fellow and developer of the REXX language, fully specified REXX and shaped it based on colleague comments before coding. You're building on a firmer foundation when you've agreed with your customers what will be done and how it will look. In addition:
• Document for others, not just in shorthand for yourself. Your future self will be grateful when you revisit software after a prolonged separation!
• Update documentation in parallel with code and operational changes. Letting technology pull too far ahead often means documentation never catches up.
• Draw your network before implementing it. Label and describe nodes and connections. Get buy-in from network architect/management and anything you'll connect to before committing.
It's (perhaps unfairly) said that applications people worry about how things will work, while systems people worry about how something will fail. Nothing’s perfect, expect problems. So envision fault/failure scenarios, diagnostic data gathering and recovery. This is much better baked in than thinly layered on top like icing.
• Validate input according to specifications, not by eliminating "plausible" errors (that is, just the ones you can think of).
Testing, Support, Measurement and Tuning
• Remember that thinking, "No one would think of doing that," was wrong the moment it occurred to you.
• Program modularity doesn’t mean isolationism. Define/set meaningful return codes or other semaphores; test them and act accordingly when invoking routines that set them.
• Beware local optimization consequences. Anything you're doing likely connects to a larger environment, so step back occasionally for the big picture.
• Write maintainable code—you may be maintaining it. Use ample comments and, for example, don't code "magic numbers" (use and explain named constants/equates).
• Use well-established coding conventions and idioms.
• Preserve source code generations. Keep production software under change management lest anarchy and entropy rule.
• For details and critical matters, read and refer to reference manuals rather than derivative works or memory of prior releases.
Modern systems produce abundant measurement data that sometimes presents interesting opportunities. If you aren't measuring, you can't tune—and may drive straight over a cliff you didn't see. But no amount of data lets you tune your way out of a capacity shortage; you're still sitting on the three-legged stool: balanced processor, I/O and memory. When solving user problems, don't neglect the forest (real requirements) for trees (attempted solutions).
The first step in gathering and utilizing measurement data is to conduct tests. Within reason/limits/resources/schedules—it's hard to over-test. Testing is more fun than bug fixing. So keep in mind the following recommendations regarding testing and debugging:
• Understand and respect testing—users/customers aren't there to test system programmers' system programs.
• Use test plan walk-throughs to ensure complete validation.
• For testing and entering production, prepare validated diagnostic tools for data gathering and analysis.
• Test edge/null/garbage/plausible/implausible cases. Watch for "less than," "less than or equal,” "not greater than," traps in specifications.
• Use regression tests—cumulative test buckets—to ensure that something new didn't break something old. Keep tests up-to-date with new systems/applications/development and use local test suites to complement vendor validation.
• Preserve test scaffolding. You'll need it again.
• Don't test with live data.
• Don't confuse observed symptoms with what you believe causes them; don't jump to conclusions.
• Use buddy-system debugging; explain symptoms—without mentioning suspicions—to someone else.
Long ago, Princeton University's Melinda Varian epic paper, "What Mother Never Told You About VM Service
," guided generations of VMers through maintenance and migration minefields. Though dated technically, its philosophies still ring true. In addition:
• Quick and dirty is risky; many one-time tools aren't.
• Don't pour concrete that will be dug up later. Use customization facilities but avoid changing what vendors send; that increases migration burdens and pitfalls.
• Copy, don't move. That is, don't blow away an original until you're happy with the new version. And even then, cherish the original until you're sure there are no hidden or delayed dependencies.
• Have a backout plan or Plan B. Never go anywhere you can't get back from and avoid "point of no return" decision making. You’ll never meet a backup plan you won't like. Again, question your assumptions: are you SURE the backup you were promised exists?
• Don't update the running system—not even by accident. If you're not sure why, ask a scarred veteran.
• Maintain a change log, the more detailed and comprehensive, the better. It's essential for evaluating symptoms and avoiding "death by recollection."
Remember that virtualization—running multiple system images or nested systems—introduces new hazards. Ensure—with distinctive prompts, color coded-text or the equivalent—that you're addressing the intended system image. That is, don't accidentally shut down or misconfigure the hypervisor or apply maintenance to the wrong level. Finally:
• Regularly shut down and reIPL (or reboot, or restart, or whatever's necessary on non-mainframe platforms) for practice, to make it a routine procedure and to verify documentation. Don’t learn how to do this under pressure. Wisdom and Wheels
• Of course, deal with disasters before they happen with planning, practice and drill reviews. Include networks, servers, middleware and operations.
• Periodically test backup; never-tested backup procedures are too often discovered to be "write only."
• If it's important enough to back up, back it up more than once, for multiple levels of recovery.
Many painfully learned mainframe lessons now seem so obvious as to be just common sense. But we've seen too many of them ignored or even disparaged, only to be validated again. So help other-platform colleagues not by imposing a mainframe-centric outlook, but by applying mainframe wisdom in more general situations.
Gabe Goldberg has developed, worked with and written about technology for decades. He can be contacted at firstname.lastname@example.org.