I'd guess it fractured not from being overtightened, but from being loose. Look for fretting marks where nut seated on inside of flange. Fretting means it ran loose. If it was tight, stress in shaft would be constant. Loose, stress varies with rotation and that sets up a fatigue scenario. could also mean tranny and prop shaft were misaligned.
Also, when putting it all together, pay particular attention to getting preload right on output shaft thrust bearings. I think those are opposed tapered rollers, and preload is critical.