Clothing are not all about “Mahram and non-mahram” issues, but is sometimes from the point of view of honor and manners. Generally, when one is alone at home, you may wear simple and plain clothing. However, when hosting guests, in respecting your visitors, you may wear something more suitable to the occasion. And if you go to a wedding reception, again you would wear something suitable to that occasion. This change in the clothing is in respect and honor of the guests and surrounding environment.
Also, therefore, in the presence of Allahﷻ during your prayers, appropriate and complete clothing is required.