I hope its not stressful because all of mine would need intense therapy!

I was not happy with my decor for a long time and was always changing something. I think goldfish like the interaction of our hands in their tanks, mine do anyhow. I think its a little more stressful for tropicals to have the activity going on when we work on their tanks.
As long as your fish are acting normal I'm sure they are O.K. when you change things around. I think taking them out would be more stressful!