From what i have observed, havok does a significantly better job for rigid simulation than Physx, especially their new Havok Physics 2013.
Im not very familiar with how state of the art physics engine works, but by testing alone i cannot get a very accurate test results.
For example, PhysX still seems to cripple CPU performance on purpose. My results shows when the simultaneous interaction rigids exceeds a certain amount (this ranges from 1024 to 8096 boxes), it's performance drops along a very unnatural steep curve and stops plumb dropping when it matches the performance of Bullet. Whereas, many other engines I've tested scale relatively linearly with scene complexity.
This goes worse if I want to measure the real world scenarios such as in games or in game-engines or even in CG making.
Undoubtedly, GPU handles particle physics far better than CPU, So i want to restrict this discussion on rigid body and soft body (including cloth) simulations.
So, is physics simulation really faster on GPU? If it is, by how much?
In general the architecure of a GPU is designed to handle massive data-parallel tasks by having a large number of stream processor cores with very wide SIMD instruction sets. So if the task can be decomposed into similarly structured independant kernels, then a GPU will be much faster (sometimes by orders of magntitude). CPUs do also have multiple cores and SIMD instructions as well, but not as many and not as wide. So it really depends on the specific properties and constraints of the specific workload, and whether or not it can take advantage of this extra parallel architecture.