Many of the people saying they single hand larger boats probably have a lot of experience. If you are just getting started it may be factor. However I'd first look at what size fits your needs in terms of how long you'll be aboard, what areas you plan to visit, and what can you afford to maintain. Buying a boat is one thing, maintaining it is another. I have a 21' Wellcraft I use on Lake Tahoe and a 40' Tollycraft in the PNW. The maintenance of the Tollycraft isn't twice the Wellcraft, it's 100x.
Anyway it sounds like you're narrowing in on a range. It is easy to get caught up in the bigger is better mentality. Bigger is great if you're on it for long periods (months) but there's much more work as well. So, identify what you really need and focus on that. As to handling, hire a captain for a few days to help you learn the ropes. Frankly any new boat has learning curve, big or small.