This is when I run it with PhysX, check this
image, to the right of 50°C you see 16% that is my GPU usage, it should be like 99% but its not and that make it slow down that much.
It work great without running PhysX.
EDIT: Fixed it just reinstalled PhysX and set my PhysX processor to my GTX 470.