I've seen DooPHP is the world fastest MVC framework and I intend to consider this for building mission critical website which is able to handle over 100,000 concurrent users.
I have to build my website from scratch on shared hosting and run CPanel on VPS which use Apache 2.0.x with PHP 5.2.14 and MySQL. Performance wise, does it make a lot of difference when coding PDO(access database)+PHP vs DooPHP?

Technically, not using a framework should always be faster because you are writing code only for what you need.
That is assuming you are well versed with PHP however, and know how to write good code.
A framework will cut down on development time.
If performance is a key issue, look into caching opcodes, server side caches, far distant expiry headers, CSS spriting, etc, etc (there is a lot).