I've see some slow behavior in deployment managers, but they were typically running on VMs with unreserved resources. Other times, I've found that the NIC, network activity, or the target server was experiencing issues.
Currently, my DMGRs are not configured with a Min/ Max heap size, but they average about 600 MB.
Can you elaborate more on the topology? You mention 20 clusters, but how many nodes, are they hosted on physical or virtual hardware; do you leverage node groups; which version of ND are you running? Also, is this a new issue or something that has been ongoing since the cell was created?