"If you wanted to keep the panels easily removable a very thin stick on weather stripping might work."
I agree with Kevin. I think this is the only way you'll be able to frequently remove the panel without having to reapply sealant each time.
MDR sells 1/8" thick, closed cell, self adhesive, weather strip. I think I'd try that first.
It should compress to about half it's expanded thickness.
CLOSED-CELL FOAM TAPE SEALS OUT MOISTURE, NON-MOISTURE ABSORBING
WEAHTER, SOUND AND FUMES
PVC foam .125” thick tape is ideal for cushioning windows and doors. Absorbs shock and eliminates vibration. Prevents metal to metal contact.
Size: 1/8" x 3/8" x 10'