An additional reason for this rule is that generally if you are not wearing your belt, then you should not be displaying it. You are also not supposed to toss your belt over your shoulder, or carry it dangling in your hand. Technically, it should be folded up and tucked discreetly under your arm (or inside your gi). If you carry your gi with the belt tied around it, and it's not in a bag, then you should also tuck the gi under your arm, so that the belt is not displayed. Showing your belt in any manner except while wearing it to train, compete, teach, or demo your art is considered bad etiquette.
There are even more unwritten belt rules that I didn't mention...if you have to fix or remove your belt, turn away from your partner and/or the class (unless you are in a competition-never turn your back to your opponent-and don't fix your belt or gi unless told to by the ref)...do not ask a black belt what dan rank they are...don't touch someone's belt without permission (except in the course of training)...etc...etc.
In the following video I broke etiquette at the beginning when I let my belt touch the floor, and at the end when I turned my back to Shihan (I was supposed to just walk backward). They still let me keep my black belt!