RE: open transom
Many a lobster boat has survived heavy weather fishing with an open transom, and I've even seen a power cat done without the transom, to allow pulling a dinghy up on rollers. Of course you will ship water over the aft deck on occasion, and the good news is that it will empty rather quickly as well. My only concern would be a massive amount of water entering through the cabin bulkhead. I think if I were you, I'd contact one of the many lobster boat builders, such as Atlantic Boat Co. in Brooklin, Maine. They should likely have some insight.*
Another thing to consider is installation of a transom plate hinged at the bottom, which becomes a swim step when lowered. It wouldn't be a huge effort to build and install, and would limit the amount of water entering the cockpit.