As well as all the other answers, there are also a few other considerations which may not affect CPU speed directly but make building anything around that CPU quite difficult;
In short, above DC, radio frequency becomes an issue. The faster you go, the more inclined everything is to act as a giant radio. This means that PCB traces suffer crosstalk, the effects of their inherent capacitance/inductance with adjacent tracks / ground plane, noise, etc. etc. etc.
The faster you go, the worse all this gets - component legs can introduce unacceptable inductance for example.
If you look at the guidelines for laying out "basic" PCB's of the sort of level of a Raspberry Pi with some DDR RAM, all the traces for the data bus etc. have to be of equal length, have correct termination etc. and that's running well below 1GHz.