Most ballasting such as yours was a previous owners attempt to balance the boat.
Many boats have a list. Juggling equipment, tools, parts, food and so on does not always work.
I had to do that even though I moved my heavy tool boxes and any other moveable gear to one side.
Some builders will ballast boat hulls that were intended for other purposes such as a fishing boat rebuilt as a pleasure boat. THose commercial vessels were built to carry heavy loads which they in fact will never carry as pleasure vessels. Ballasting in those cases is then needed. But in those cases the ballast will be secured.
Remove it and see what happens. You may find yours is needed if the boat lists or is bow down. You may find as your boat the ballast is no longer needed, at least for now. Don't discard it as over time you may find it needed if you add/change something.
Where is yours?