The following is a pic of my 2003 Pilot 34's rudder and shoe (with the faired deadwood, which did not do any good for high speed cavitation). I am pretty sure that there was no significant clearance between the pin at the bottom of the rudder and the shoe on my boat. There certainly wasn't any play in the rudder.
If you have 1/4" of clearance, that sounds like more than there should be, but does it matter? Do you feel any prop vibration back through the helm particularly at low speeds. I know that my Island Packet sailboat with the same rudder/shoe set up would vibrate at the helm at 5 kts or so. The solution was a bushing in the shoe.
So if you do see problems, then you could drop the shoe, have a bushing made up that fits more tightly around the rudder pin, enlarge the hole in the shoe so that the bushing fits and put it all back together.
But, if it ain't broke, don't fix it.