Testing the bilge pump without water tells you that the pump is running but it doesn't tell you if it will actually pump water out of the bilge.
I was cleaning my bilge once and used a hose to put water into the bilge. The pump came on and ran as expected but I eventually noticed that the water level wasn't changing. It turns out. the plastic impeller broke where it attaches to the pump shaft and wasn't spinning.
Other faults could be a kinked or plugged discharge hose or debris clogging the intake screen of the pump.
My message is - check your bilge pumps with actual water from time to time. See if they actually pump the water out.