It was.
But the Rooney's are loyal to a fault. And Colbert repeatedly stated that he wanted to go out like Ozzie Newsome. Who, in Colbert's opinion, pulled off a rare GM feat. He built the Ravens to contend under one QB (Flacco) and then set them up to contend again under another (Jackson). Unfortunately for Rooney and Colbert, he took his shot in a historically weak QB class.
This is probably all hindsight making things look worse than they really were.