Last week I was talking to a customer and one of their main concerns they have lately is how to troubleshoot asymmetric routing. Let me explain why…
Cisco ACI Issues
This customer is using Cisco ACI in a multipod environment and they are using identical vlans in all fabrics allowing them to extend L2 connectivity to different data centers. This is a great functionality of Cisco ACI but it can also generate problems. For example, for this customer they are having asymmetric routing issues, where some traffic that should remain internal some times try to go to another data center to reach its destination. As you may think, this looks like a relevant problem, in terms of:
· Cost: generating WAN traffic where it shouldn’t happen
· Performance: 100s of milliseconds added to every transaction impacts on the performance of every single application
· Availability: some firewalls like to work with stateful connections, if they only see ‘half’ of the conversation, they will drop the packets
Example of Next Hop Analysis
Solution
One of the (many!) cool features that we have in SevOne is the ability to monitor flow data, including the enriched and enhanced flexible netflow (aka Netflow v9 or IPFIX). With these protocols we can perform typical bandwidth analysis but also, we can do the most advanced traffic troubleshooting using the extra fields available in Netflow v9 and/or IPFIX, like, for example, ‘Next-Hop’.
Fields available in IPFIX
Fields available in NetFlow v9
With the fields ‘source IP’, ‘destination IP’ and ‘Next Hop IP’ we can build a nice flow view that shows us the direction that the traffic is taking when going from A to B. However, this is not enough to troubleshoot this situation, we need to know where this is happening and why.
Traffic from 172.24.147.115 to 172.24.147.112 only goes through 172.24.147.5, whereas traffic coming back from 172.24.147.112 goes through three different next hops.
This is when another amazing SevOne feature called ‘Flow to Flow’ allow us to drill down into the traffic and understand what applications are using a different route to reach back the source of the traffic.
With this functionality we can display the bidirectional traffic, identify the traffic that is using a different next hop to come back and see which are the applications and protocols that are being impacted by this asymmetric route.
Applications and Protocols using wrong Next Hop
With SevOne and its ability to collect, digest and report on any kind of NetFlow data, we have managed to identify when the problem occurs, which applications are impacted, and now we have the information we need to go back to the devices and fix the issue. All this in minutes rather than in weeks.