Sounds like the Turbo alright, what you described does not sound like it would wreck the turbo. Excessive heat or lack of lubricant are the main two things that kill turbos.
The turbo should not really have an effect on start up. Oil however does, as your injectors are hydraulically actuated.
I would check your oil, it may be old or thinned down.