Good bacteria adheres itself to the gravel,as to the muck/waste/poo under the gravel,well,that is all really bad things-especially when left to build up.It can possibly form bad pockets of toxic gas and harm your fish,or cause your fish to have more probs with bacterial and illness ,etc.
As long as you aren't taking the actual gravel out and cleaning it under the sink faucet,your good bio bugs will be fine through gravel vacs.