If you bypass you intercooler, you will likely lose some power at the upper end. It might not make much difference at low power settings.
I wouldn't think that intake air temps coming directly off the turbo would damage the engine... Plenty of turbo diesels around that don't have intercoolers. Still, the guys at American Diesel would be the folks to ask
Hmm, on second thought, another possible downside to removing the intercooler occurs. The mechanical fuel injectors will be calibrated to work with the greater air charge that the intercooler gives you. So, especially at high power settings, you would likely be running pretty rich... possibly blowing smoke? I'm just swagging at this point. I would love to hear whether anyone who has done an intercooler-ectomy has experienced power loss and smoke...