A dead-letter queue is used by various parts of the queue manager (and sometimes also by applications). Channels, topic publish, and command server are some of the different entities that might place a message on a dead-letter queue if they cannot route it to the intended destination.
A message on a dead-letter queue is expected to have a DLH header which indicates the queue and queue manager that the message was supposed to be headed for (because there could be many different queues represented on the dead-letter queue) and the reason code indicating why the message couldn't be placed there.
The dead-letter queue is often processed by an automatic handler known as the dead-letter handler, which has a set of rules telling it what to do with the different reason codes it encounters. For example, just retry the put when you see MQRC_Q_FULL.
A back-out queue is an application specific resource, sometimes tied to a specific queue, sometimes a set of application queues. It is mainly used in the process of poison messages (see recent blog post on here) and the applications using it may or may not choose to place a DLH header on the front of the poison message.
A poison message is one that is going to require manual intervention to correct, and so it is less likely to be processed by any automated handler like the dead-letter queue. It is not a general purpose resource, so there won't be lots of different queue names landing up there (hence not necessarily needing a header to say which queue the message should have been on).
They serve different purposes, BUT those purposes do overlap. Some people use the back-out queue as an application specific dead-letter queue. Some people put a DLH header on messages routed to the back-out queue. And of course, the built-in behaviour in JMS uses the back-out queue, but if that fails, it uses the dead-letter queue.