Travelling around the coast of NJ you will be fine in most high winds if they are out of the west because if you hug the shore you will be protected. But east winds of 20 kts will make that trip tough for almost any trawler.
So my criteria is sea state, not winds. When we lived in California and regularly made the trip offshore to Catalina Island, 5-6 feet of seas was my limit.
We made one trip in 6-7' seas and it was uncomfortable but not unsafe. Anything above 8' I would consider unsafe for most medium sized trawlers.