In real-time systems, duplicate events are a fact of life. They can occur because of multiple retries, network glitches or overlapping sources sending the same message. If not handled carefully, they can overwhelm downstream systems and create noisy, repetitive alerts for users.
To manage this, Event Processing provides a Deduplicate node which has two deduplication modes:
- Fixed Interval
- Inactivity
Let’s explore its difference through a relatable example, a Customer Support Alerting System and see how each mode shines in its own way.
The Story: Customer Support Notifications
Alice manages an alerting stream for a customer support system. Whenever customers report a service issue like payment failures, an event is emitted to notify the support team.
For example:
{
"type": "Service Alert",
"message": "Payment gateway failure",
"timestamp": "2025-10-19T10:00:00Z"
}
Sometimes the system might generate repeated alerts for the same issue when:
Alice doesn’t want to flood the team’s dashboard with duplicate notifications. But she also doesn’t want to miss genuine new alerts.
That’s where the deduplication comes in and she needs to choose between Fixed Interval and Inactivity modes.
Scenario 1: “When a sudden flood of identical alerts appears in a short burst”
At 10:00 AM, the payment gateway suddenly goes down. Hundreds of users report the issue within seconds, producing identical alerts.
Deduplicate node configuration:
[10:00:00] Payment gateway failure → emitted
[10:00:02] Payment gateway failure → ignored
[10:00:05] Payment gateway failure → ignored
[10:00:10] Payment gateway failure → ignored
[10:00:15] Payment gateway failure → emitted
How it works: When the first alert arrives, a 10-second window starts. Any duplicate alerts that appear during this window are ignored. Once the 10 seconds pass, the next identical alert starts a new 10-second window.
In simpler terms, the first alert in each window acts as a reference point and all subsequent alerts within that time interval are compared to it.
Why Fixed Interval is suitable:
-
The duplicate reports come in burst form, right after the first event.
-
Alice doesn’t want to spam the support team every few seconds for the same incident.
-
A fixed 10-second window removes duplicates while the burst continues.
- Once the initial wave settles, new alerts can be sent again if the issue persists.
Goal: Reduce noise during initial spikes of identical reports.
Scenario 2: “When the same alert reoccurs after temporary recovery”
Later that day, the payment gateway becomes unstable, going up and down multiple times. Alerts now come at irregular intervals, sometimes just a few seconds apart.
Deduplicate node configuration:
[12:00:00] Payment gateway failure → emitted
[12:00:07] Payment gateway failure → ignored
[12:00:10] Database connection error → emitted (Unique event)
[12:00:15] Payment gateway failure → ignored
[12:00:27] Payment gateway failure → emitted
[12:00:30] Payment gateway failure → ignored
How it works:
Every time an alert arrives, whether it’s emitted or removed as a duplicate, the 10-second window resets. A new alert is only allowed through if no identical alert appears within the next 10 seconds.
Here, each alert is compared with its previous identical occurrence, unlike the fixed interval approach where every window starts from the first occurrence.
Why Inactivity is suitable:
-
The problem here is not a one-time burst, it’s intermittent.
-
Alice wants to avoid very frequent duplicates, but still alert if the issue recurs after some quiet period.
-
The Inactivity mode ensures alerts are spaced by at least 10 seconds, no matter when they happen.
Goal: Prevent over-alerting when incidents keep reappearing at irregular intervals.
Try this in Event Processing
You can easily simulate both these patterns in a flow by connecting a Deduplicate node to an Event Source node. Configure the Deduplicate node as described in the documentation, where you can choose the deduplication mode that best suits your use case.