Very hard to tell, yes. For me personally, its only obvious during spawning season, in the early spring. Our males are rather slender, torpedo shaped fish, while the females have really deep, fat bellies, and get extremely wide. But mostly into the depth of the belly - like they are swimming with a little water melon.

Other than that, there is the thickness of the pectoral fin that gives it away, however the fish has to be bigger to tell. It wouldn't work in a 2 inch koi or something.

The pectoral in general is a bit chicker in a male, I believe. Plus, of course, male koi have breeding stars as well. That would be the easiest to spot.