In a busy messaging system, “no news” isn’t always good news.
Imagine your service goes offline for an hour because of one “poison message” — a message that keeps failing and returning to the queue. Your monitoring tools see the message, but they don’t warn you because they can’t “count” the failures.
Today, we are changing that with a simple update to how IBM MQ handles data.
A Real-World Example: The Banking “Severity 1” Incident
To understand why this matters, consider a recent case involving a major bank. Their internet banking service went down for over an hour because of a poisoned message queue.
Because the system was only recording failures as text, their monitoring tools couldn’t trigger an automatic alarm. This led to:
- Extended Downtime: Customers couldn’t access their accounts for an hour.
- Delayed Detection: The team didn’t know there was a problem until the service had already failed.
- Manual Effort: Engineers had to manually search through logs to find the root cause.
The Problem: Data You Can’t Use
IBM MQ has a counter called BackoutCount that tracks how many times a message has failed.
Until now, this was sent to monitoring tools as text. While humans can read text, computers can’t easily use it to make decisions. You couldn’t tell your system: “Alert me if this number is higher than 5” because the system didn’t see it as a number — it just saw it as a word.
The Solution: Numbers You Can Act On
By turning on a simple setting, the tracing tool now sends the BackoutCount as a numeric metric.
This small change allows monitoring platforms like Instana to understand the data. Now, you can set up automatic alerts to catch failing messages before they crash your application.
How to Turn It On
You don’t need to install anything new. You just need to add one line to your configuration file, typically found at /var/mqm/exits64/mqtracingexit.conf:
Properties
IBMMQ_FF_SEND_METRICS = "on"
Once this is set to “on”, your system starts sending numbers instead of just text.
Why It Matters
Not every failure is an emergency. A message might fail once due to a quick network glitch. But a message that fails 10 times is a “poison message” that needs your attention.
With this update, you can choose exactly when you want to be alerted:
- Critical Tasks: Alert after 2 failures.
- Normal Tasks: Alert after 5 failures.
- Large Batch Jobs: Alert after 10 failures.
A Note on the “Flag”
We specifically mention the name of this setting (IBMMQ_FF_SEND_METRICS) for two important reasons:
- Searchability: If a teammate is looking through the config file, they need to know exactly what keyword to search for.
- Control: This lets you choose. If you aren’t ready for numeric metrics yet, you can leave it “off”.
Summary
By turning on numeric metrics, you give your system an early warning sign. You can find and fix problems before your customers even notice them.
Ready to try it? Check your configuration file today and stop the silence of poison messages.
#Alerting#Tracing