Hello
Many people here have barebottom tanks with marbles and rocks. The fish are fine with it. Just don't remove it all at once, as many nasties can be harboring in there. I would probably remove it in thirds just before water changes. That way you can remove anything that may get be released into the water.
If you tank is cycled, then the majority of your beneficial bacteria will be in your filter, not your gravel. If you'd feel safer with some of the gravel still in for a while, you can put some in a clean stocking and keep it in the tank for a while.
Good luck!
Debbie