IBM Destination Z - Group home

Turn Experiences Into Wisdom

By Destination Z posted Mon December 23, 2019 03:39 PM


“Oops” or “ouch”—those are the sounds often following a mistake. “Hey, great!” is not heard so much. But don't deny it—own it, apologize for or bandage it, learn and teach from it, and move on.

Googling “mistake wisdom” results in quotes showing that the two are inextricably linked: the first leads to the second, which avoids more of the first. For example:

• "So go ahead and make mistakes. Make all you can. Because, remember, that's where you’ll find success. On the far side of failure." Thomas J. Watson Sr.
• "It is much easier to be critical than to be correct." Benjamin Disraeli
• "Anyone who has never made a mistake has never tried anything new." Albert Einstein

Good system programming, of course, requires mixing timing, tools, support and wisdom.

Avoid Technology Traps

A long-ago mistake I observed included a system programmer—brought in as a consultant—wasting time and money failing to debug a serious problem. This would have been remedied by a fix he had in hand, which he dismissed as unnecessary. He ignored a key best-practice rule: Read and follow instructions and use checklists. An excellent book, "The Checklist Manifesto: How to Get Things Right," demonstrates how all professionals can benefit from using detailed task checklists.

Specialized checklist forms can guide code and design reviews, which aren't make-work; they help catch errors and learn from others. Similarly, configuration control procedures and software are not just formalities best ignored or circumvented; when well designed and situationally appropriate, they can improve service quality, avoid small and large problems, minimize surprises and develop team thinking.

Because environments differ and change, don't assume that a cookie-cutter procedure, even from a vendor, will work unmodified. Review explicit caveats, footnotes and warnings. Before starting, check for unstated assumptions (e.g., OS or compiler versions, prerequisite software or hardware, DASD type and space available, network configuration).

Regarding financial planning, don't predict the future based on past behavior. A vendor once argued that a hardware feature specified as required by an IBM software product wasn't needed and refused to sell it to my installation. Of course, just because the salesman had never before seen his own feature used didn't mean that the IBM product—VM/Pass-Through Facility—didn't need it.

If even present, vendor verification tools often ensure that a product is minimally functional without fully testing. Build test suites to exercise your installation's key products, services and tools. Meaningful and timely regression testing avoids having users or customers discover problems. If this requires building "test scaffolding" around applications, don't remove it when transitioning into production, because then you won't be running precisely what you've tested.

Test, Question and Backup

Today’s mainframe environments are complex beyond imaginings of early days practitioners. Aside from simple-function standalone utilities, things are highly connected, and not always in obvious ways. Remember the powerful law of unintended consequences: Each change affects something else, often in unintuitive ways. Perhaps even more important, consider who will notice and who will care.

Bulletproof code requires testing return codes and handling—as gracefully as possible—all conditions, from the possible to "this can't happen." It's much better intercepting errors, problems or disasters—and capturing or providing as much diagnostic information as possible—than continuing past a problem until it can't be localized, diagnosed or remedied without being recreated. And, of course, always define and document return codes or the equivalent in code you write.

A skeptical nature is a programmer's asset. Assume that specifications may change and that supposedly verified input may have errors. If you find an error, don't assume that it’s the only one. Validate results from your own code before using them or passing them on.

Write software documentation before coding, and update documentation before changing code. Labeling both ends of cables is another form of documenting your work; someone following your footsteps will be grateful. And you will admire your work habits since you otherwise wouldn't remember how things are wired.

If it's worth backing up, it's worth creating multiple backup copies using different techniques and different media. Restores sometimes fail, and having spare copies prevents a nuisance from becoming a catastrophe. VMer Melinda Varian's long-ago refrain applies: Back it up. Back it up again.

Because backup may appear to work but not create a usable copy, or matching restore processes can fail, occasionally test all backup systems. It's much more pleasant debugging a failed test restore than learning you've run a write-only backup and have lost data.

Backups apply to more than data. When working through a complex procedure—e.g., upgrading the OS—some steps may risk corrupting what's done. So back it up and take a checkpoint. This small burden can guard against losing many hours of work. Similarly, because bad things can happen to good programmers, avoid "points of no return"—i.e., taking actions that preclude returning to your operational starting status.

Don't leave small problems hanging until someday when there's nothing more pressing; they compound into disasters. Fix them when they're encountered or in batches at your convenience, lest they become crises at the worst possible time. (But, of course, when would be a good time?)

Don't mistake locally connected test results for being good-to-go online; test end-to-end operation replicating the production user environment.

It shouldn't need to be said—but experience says otherwise—that testing and evaluation isn't a meaningless formality with preordained success. No matter how routine something is or how strongly management presses for a glowing report, be professional and fully exercise a test subject.

And no matter how familiar you are with routine tasks, be careful and let your eyes and brain confirm what your fingers are about to do. One veteran mentioned being burned by the difference between CMS commands EXEC CMS ERASE and ERASE CMS EXEC (where the former erases all files listed in CMS EXEC and the latter simply erases the EXEC).

Protect Yourself

If you bring a personal toolkit to work, document your ownership. And if those tools enter production, provide a nonexclusive license to your employer or client.

Estimating project schedules, costs and staffing could be a separate article, or perhaps a book. The saying “pick two project attributes from fast, good and cheap” is worthwhile industry wisdom. Never give a time range including an overly optimistic completion date because management might seize on that as a firm commitment.

There are often multiple ways to meet requirements: Develop applications locally, acquire—often for free—software developed by another installation and purchase a vendor solution (perhaps one among many). Each blends advantages vs. disadvantages and has obvious and hidden pitfalls. Another separate article could address the eternal build-or-buy dilemma. A common mistake is underestimating not just initial development and documentation costs but the perpetual maintenance or support burden and the challenge of recruiting staff to work on what will eventually be disdained as legacy code.

Mistakes don't happen only at the individual or installation level; history offers what some consider to be industry-shaping events.

In the early 1970s, IBM began the Future Systems project to develop a computer family incompatible with then-current mainframes, including new software models to simplify software development. It was grand, it absorbed most of IBM development resources, and it ended in 1975 without reaching the market. During its life, not many advances were made on available products.

And in the early 1980s IBM reshaped the mainframe world by reducing and removing source code distribution. Reasons given included easing migration to new software versions, reducing support burdens and dealing with competition from plug-compatible computer vendors. But many in the user community argued that doing this—a policy called object-code only (OCO)—reduced customer ability to innovate, extend and repair IBM software. There's no telling what today's mainframe landscape would be without OCO. A Computerworld snippet might bring back memories for veterans and be instructive to youngsters.

Navigate Career Pitfalls

These days, career planning is self service; no longer do workers assume they will stay at a job their entire career. That's good and bad news: There's less reason to take or stay in ill-fitting, non-rewarding jobs, but if you're change-averse nobody else will nudge you to advance yourself. Above all, have career plans in mind, including alternative strategies if things beyond your control change (e.g., the economy turning sour hindering startups or industry developments redefining markets).

Beware ego. Insisting on being right won't make it so, and it's a lonely objective. Be open to others' ideas. Be gracious about others' mistakes and encourage rather than criticize them. Be willing to be wrong and appreciate those who help you learn.

On the job and online, it's often better to try something than ask whether it will work. Your environment and approach will differ from those of others and failing may point you in a productive direction. Of course, refer to manuals, help information and online discussions. But don't believe every negative assertion; today's mainframes exploit many breakthroughs, facilities and technologies once thought impossible.

At the same time—and it's not contradictory—develop a diverse sounding-board network of colleagues, whether at your company or worldwide. SHARE and the many mainframe-related discussion lists are an extremely valuable resource.

Before writing for external publication (e.g., trade press, letters to the editor, blogs or social media), determine your employer's policy on showing your affiliation; this can be a case where it is not better apologizing than asking permission.

Remember skepticism? It needn't grow to paranoia to justify retaining copies of memos or email concerning projects or actions about which you have misgivings. Most likely, they'll simply be amusing later; occasionally, they'll be useful in reconstructing events in your favor.

Greet and Advise Your Younger Self

Who's better qualified to advise your younger self—and today's younger selves—than you? What would you do differently? People entering the industry today likely realize something that we didn't: They'll of necessity repeatedly change jobs or careers and reinvent themselves. But two tips are worth emphasizing: First, watch for opportunities that might be worthwhile even though off your current path. Historically, these might have been the PC, the rise of ISVs and the Internet. Second, don't immediately dismiss well-intentioned advice even if it sounds implausible. Get clarification to understand the perspective on which it's based.

Finally. a colleague contributed two perspectives:

"Hein's Law: Problems worthy of attack prove their worth by hitting back." —Piet Hein

The road to wisdom? — Well, it's plain and simple to express:
and err
and err again
but less
and less
and less.

Various songs also offer perspectives:

"And bad mistakes
I've made a few"

"Regrets, I've had a few;
But then again, too few to mention."

Gabe Goldberg has developed, worked with, and written about technology for decades. Email him at