Nice topic, it seems the "rules" in the US are more or less the same as in Europe. This is what i found.
The Flag Advisor
Flag etiquette can be arcane and boggling. We surveyed Chapman’s Piloting and our flag maker for guidance and came up with some general recommendations for boaters:
Courtesy and National Flags
As a gesture of courtesy, cruisers should fly a foreign nation’s flag when they enter and operate in its waters.
Rule No. 1—There are no real rules. Customs observed in various foreign waters differ from each other. We’ve seen cases where not flying or flying a courtesy flag improperly causes some awkward moments; you may be regarded as impolite, but nothing more. In others, it’s local law to fly the flag. Officials can—and do—impound passports or assess fines until the proper flag—which, of course, can only be purchased locally at great expense—is flying on board. If in doubt, inquire of other cruisers and observe other craft from your country for guidance.
Do not fly a courtesy flag until your vessel is properly cleared by customs and immigration. Until clearance is complete, fly the yellow Q (quarantine) flag.
On a mastless powerboat, the courtesy flag replaces any flag that is normally flown at the bow.
If a powerboat has a mast with spreaders, the courtesy flag is flown at the starboard spreader.
On a two-masted powerboat, the courtesy flag displaces any flag normally flown at the forward spreader.
On a sailboat, the courtesy flag is flown at the starboard spreader. If the sailboat has more than one mast, the courtesy flag is flown from the starboard spreader of the forward mast.
Courtesy flags are usually Civil Ensigns—not the national flag of the country. Not every country has a civil ensign. However, most former British colonies do; it is usually the red variant of the flag. It’s considered a horrible breach of etiquette to fly the blue national flag. So, if the flag that we catalog doesn’t look exactly like the national flag that you remember, it’s probably a civil ensign.
Don’t fly a foreign courtesy flag after you return to U. S. waters. It may show that you’ve
’been there,’ but it’s not proper etiquette.
Generally, the vessel’s national flag is flown from the stern (or leach) when a courtesy flag displaces it.
It’s better form for U. S. vessels to fly the U. S. flag (the “stars and stripes” with a full complement of 50 stars) at the stern or gaff or leech, rather than a Yacht Ensign. If you want to fly a Yacht or USPS Ensign, do so from the port spreader on a sailboat. If there are multiple flag halyards available on the starboard spreader, the Yacht or USPS Ensign is flown there, inboard from the courtesy ensign.
Any citizen of any state may fly the flag of that state unless doing so is specifically prohibited. It should be flown at the main masthead in place of any private, yacht club, or officer’s flag. On a mastless boat, a state flag flies from either the bow or radio antenna.
No flag—state, heritage, Confederate, pirate, gag, or otherwise—except for the vessel’s national flag, should EVER fly from the stern of your vessel. This is considered a place of honor, for the vessel’s national flag and no other.
Chapman’s recommends the following:
The flag at the stern of your boat—U. S. Ensign, Yacht Ensign, USPS Ensign, or vessel’s national flag—should be one inch on the fly for every foot of overall vessel length (e. g. 48” flag for a 48’ foot vessel).
Other flags—club burgees, private signals, or courtesy flags—should be ½” for every foot of overall vessel length.
Christine Davis differs with a somewhat more practical approach. She suggest that you let your eye be your guide. Generally, yachts up to 50’ in length look properly “dressed” with a 16” x 24” ensign and 12” x 18” courtesy flags. Size up one step for every 25-or-so additional feet in length. If you prefer the look of larger flags, go ahead—just make sure that there is a clear 360-degree fly from your halyards. Otherwise, your flags will soon be in tatters.
There are a number of flags that once were used on large yachts with professional crews (such as owner absent, cocktail, meal, etc.). Others are still common:
Diving Flags. There are two flags flown by diving operations: a red flag with a single diagonal stripe of white and International Code Flag “A”. It is generally no proper to fly dive flags on shore.
Quarantine Flag. International Code Flag “Q” is flown when entering a foreign port (except Canada and a few others) or when returning to a U. S. port from a foreign cruise. It signals to customs and immigration officials that you request clearance. Take it down and replace it with a courtesy flag after formalities are complete.
Union Jack. A rectangular blue flag with 50 stars, the Union Jack may be flown as follows:
Flying only at the jack staff—the bow staff on modern craft
Flying only during the day
Flying only when moored
Flying only on Sundays, national holidays, or when dressing ship
On national holidays, at regattas, and on other special occasions, yachts often “dress ship,” displaying a decorative collection of International Code signal flags. The following conventions are recommended:
Dress ship at 0800 and keep dress until nightfall.
Keep the dressed ship moored, except for its maiden or final voyages or for participation in parades.
Hoist the Ensign at the stern. Display the Union Jack (if desired) at the bow.
Hoist a rainbow of International Code Flags from the waterline forward to the waterline aft from stem (or bowsprit) to the masthead(s).
Bend on flags and pennants alternately. Since there are twice as many letter as numeral pennants, it is regarded as good practice to follow the following sequence:
Two flags, one pennant, two flags, one pennant, and so on
A popular example with an appealing color pattern is (from forward): AB2, UJ1, KE3, GH6, IV5, FL4, DM7, PO Third Repeater, RN First Repeater, ST Zero, CX9, WQ8, ZY Second Repeater.
Consult web page notes for signal flag set coverage.
Link: Flag etiquette