A number of years ago we ended up in the middle of five gray whales who were feeding in Bellingham Bay, a very rare occurrence. We have also found ourselves in the middle of a pod of Orcas both here in Puget Sound and up the north end of Vancouver Island. In each case we shut the engine(s) down and drift and turn off the depth sounder. I was told by some whale expert at some tine or another than the clicking of a depth sounder can be an annoying sound to a whale.
The gray whales were impressive, each one considerably longer than our boat. Bellingham Bay is quite shallow, about 100 feet, and the whales were stirring up huge clouds of sand and mud. They would surface and sometimes head right for the boat, sliding under at the last second to pass just beneath us.
We weren't worried about them hitting us accidentally. Several years earlier we'd seen some underwater footage that had been taken in Hawaii of a mother humpback and her calf. Two divers were in the water with her, one of them the cameraman. The mother did not seem at all concerned about their presence but she would maneuver slowly to keep herself between the divers and the calf. At one point she did a 180-degree turn away from one of the divers, herding the calf ahead of her as she went. The other diver was filming this, and as the whale turned away the huge pectoral fin on the outside of the turn swept around like a giant sickle, knifing straight at the diver she was turning away from. But just as the fin got within a few feet of him she flicked it up a coupe of feet so it glided over him and then brought it back down on the other side. She obviously did this by feel or some other sense because her head was turned in a position where the diver was out of her line of sight.
If a whale want to hit your boat it will. If it doesn't it won't.
The only time I've ever been worried around whales was fishing some 40 miles off the north shore of Oahu in the 1970s. We were trolling along watching for birds when two sperm whales surfaced behind us and cruised past, one on either side of the boat. Both I and the boat owner were pretty concerned because a) we were fishing in a 28' Uniflite which was considerably smaller than either of the whales and, b) sperms are the ones with the reputations for attacking things (for real, not just in Moby Dick). But they cruised past like a couple of big, black subs and then submerged out ahead of us and we didn't see them again.