Any marina or floating dock in Milford is cheap and walking distance to a fun collection of shops, bars, restaurants and all types of boat supplies and services.
If you go up the Thames River there is a fantastic and cheap marina in Norwich which can drive you to either of the large casinos. My wife loves the trip and the destination. At the mouth of the Thames is New London and Groton, naval history heaven.
Fishers Island Sound has a dozen anchorages and mooring fields as well as marinas including Noank, Mystic, Stonington. Oodles of quaint villages within a short walk.
Clinton and Westbrook both have huge marinas and vans that will take you to outlet malls, stores, and good eats.
Going up the Connecticut River takes you to Hamburg Cove and Essex, two bits of heaven. Essex is wonderful to visit, great moorings for cheap.
If you are clever with anchors you can overnight in the Thimble Islands. Legendary beauty and famous residents.
And I have barely scratched the surface. None of these places are anywhere near as expensive as Block or Newport or MV or Long Island (all which I like too)