How do I lock multiple mutexes in C (pthreads) and avoid the danger of deadlocks/livelocks?

Question

Suppose you have a piece of code that is run by multiple threads. Now further suppose that each of these threads wants to lock the same set of mutexes, say 5, but not in a specific order:

Thread 1: mutex1, mutex2, mutex3, mutex4, mutex5

Thread 1: mutex3, mutex2, mutex4, mutex4, mutex1

Now how would you avoid dead- and livelocks if you were to implement a function that takes these 5 mutexes and tries to lock them?

Since deadlocks shall be avoided a simple list of locking statements is out of question. Now what I thought of was something like (pseudo-code):

while(true) {
  if(!lock(m1)) { continue; }
  if(!lock(m2)) { unlock(m1); continue; }
  ...
  if(!lock(m5)) { unlock(m1);...;unlock(m4); continue; }
  break;
}

The problem with this approach is that it will certainly lead to a livelock and consume much cpu power.

So the only solution I came up with is to equip every thread with a priority number and use this number to specify the (increasing) sleeping time at the beginning of the loop:

sleepCounter = 0;
while(true) {
  sleep(sleepCounter);
  sleepCounter += threadPriorityNumber
  // aforementioned code

What do you think about this solution? Is it a suitable approach? Which alternatives do I have? Is there any literature concerning this problem in the scientific world?