Maybe a picture will help....
This is from a Deere 4045 spec sheet, but is typical of all diesels driving props.
The red line shows the engine's max power capability at each RPM. This is the power the engine is capable of producing, but how much it actually produces depends on the load. Asking the engine to produce power anywhere above the red line creates an overload condition.
The gray line is the load required to spin a boat prop. It has a different shape than the engine output curve, starting out low, then progressively climbing as RPMs increase. Looking at it, you can see how much more power is required for relatively small increments in RPM (and speed) - something most of us know through experience.
The gray line represents a "correctly" matched prop. The key is that the prop load meets, but never exceeds the power output of the engine. Because of the difference in the shapes of the curves, the match between load and available power only occurs at max RPM.
When people look at over propping, they are looking at the space in the graph between the gray and red lines. That space represents power that the engine could produce without being overloaded, but the prop can't provide the load. Over propping relocates the prop load curve to tap more of the available power at lower rpms.
The orange crayon line draw by a child (me) shows an example of an over propped load. The load from the bigger prop grows faster as rpms increase, and utilizes more available engine power at lower rpms. You get the same power as the "correct" prop, but at lower RPMs so your engine is presumably quieter, and maybe operating at better fuel efficiency. That's what people are after when over propping.
But you can also see that now the orange line over-takes the engines max power curve shown in red. If you give the engine more throttle trying to push to higher rpms, you are now over loading the engine. That's the danger in over propping. If you are propped such that it's possible to overload like this, engine manufacturers will void your warranty. That only matters on a new engine with a warranty, but it also speaks to the hazards involved.
Proponents of over propping say to just not push past the overload point. If it hurts, don't do it. It has also been noted that exhaust gas temp (EGT) is an indication of when you are crossing over the red line, so instrumenting for that and keeping operation below max allowed EGT is a way to stay below the red line.
Hope this helps.
