There is an extremely simple way to tell if a shirt or jacket is designed for a man or a woman – buttons. A male garment traditionally has buttons on the right side while female garments have them on the left side.
The reason for the male garment is that having the button on the right side is easier to button up when dressing oneself (given that he is right-handed). So why is this not the case for female garments? This is because during Victorian times, women were usually dressed by a maid or servant so it would be easier for them to button up a dress if it was on the left side. This theory is very plausible as zippers were not available then and dresses were mostly buttoned from the back. Even when women’s clothing started having buttons in the front, tailors were already accustomed to the traditional convention.
Alternate theories are quite interesting too. It is hypothesised that buttons on male clothing were modelled after a knight’s armour that would be latched on so that a right-handed opponent could not jam their pike through a seam. Contrary to the military origin of the male buttons, female buttons may be placed in such a way to allow the lady to easily expose her left breast – which is closer to her heart – to nurse her baby.
(Source: http://j-jm-m.deviantart.com/art/Button-66176952)