Indeed, and even had it been otherwise, Colfax wouldn't have benefited because he wasn't Speaker in April 1865. He had been until March 4, and when the 39th Congress met in December it would make him Speaker again, but during the intervening nine months the Speakership was vacant.
In any case, however, the problem was never likely to arise, as the President of the Senate, Lafayette Foster, was at his home in Norwich, Connecticut, where the Booth conspirators had no realistic chance to harm him. When he returned to Washington after the assassination, he was given a massive military escort.