Because there is bug, which according to developers has very low priority. In Cycles render settings, change your tile size to the resolution of your texture. So if you are baking 4096x4096, change the tile size to 4096x4096 (although perhaps try 2048x2048 first, just to be safe). You should get immediate increase in baking speed.
Furthermore, Cycles uses the amount of render samples for baking, which defaults at 128. For baking purposes, it makes sense to reduce it greatly. Try something like 16 samples, that should be fine. And of course, make sure you are baking on your Titan X, meaning that it’s enabled as CUDA device and used by Cycles 
EDIT: Actually, in fact, it’s not even considered a bug: ⚓ T61509 Cycles Tile Size has extreme impact on texture baking performance without any difference in baked texture quality