Before coming across this website, I used to keep my fishes in small tanks and overcrowd them. Some of them did live a few years before dying of illness - ich and dropsy (not knowing and medication back then). Note that the kind of fishes that you keep are the more hardy fishes, they tend to survive on a less desirable condition and adapt to it.
When you have fancy goldfish and you have them in small tanks, you'll definitely have red streaks on your fishes' tail due to poor water quality (thanks to koko's for teaching me), thus they would live with damaged tails or fins. Definitely not a show quality fish - indicating they are not healthy.
Yes, with your experience keeping your fishes in the overcrowded tank, your fish can't grow in size. To have a huge 'palm size' goldfish, they do need a 10gallon-each space, or a pond.
I can't afford a huge tank either, as I live in an apartment, thus I stick to just 2 fishes and do water changes twice a week. Can't wait to move to a house with a pond