Do you mean uneaten food and feces or diatoms(what some people call "brown algae")? If you mean uneaten food and feces, then unless you have a lot of plants that need the mulm, there should not be any. That is the whole purpose of gravel vacuuming. If you are having problems getting it clean try this: Don't feed as much!
1. Goldfish spend most of the day going through the gravel looking for bits of food. If they are not eating it ALL, you are overfeeding. More food = more poop.

What I would do is this: For the next couple of days I would gravel vacuum daily until you get it down to a reasonable amount. I mean so little that it isn't apparent if you move something. For example: if I move the big log in my tank, nothing is stirred up. I can imagine how high your nitrAte levels are. Do you know what the reading is?
2. Just for the record, diatoms will not harm anything. They just look bad. If you mean diatoms, that is much harder to get rid of. I know because my water is high in iron which is perfect for diatoms. They coat everything in what appears to be brown algae. They are NOT a member of the algae family. Too long to explain....
Anyway basically the only way to get rid of diatoms is with good old fashioned elbow grease. Keep the nitrAte levels at below 10ppm and scrape, scrape, scrape, it off the tank and any rocks you don't want covered. It can be a big pain in the butt! With careful management of nitrate levels and strict cleaning policies though, you can keep them under control. In my tanks you can only see them if I forget, or get lazy, and don't do it for a few weeks. I have this problem in all my tanks that use my well water.
So don't feed as much and vacuum more often....Or get scraping!!!