My mum used to say it often, with disappointment in her voice. I don't know why but I took it literally. She also used to say that my younger sister is so much cuter than me. So I made my interpretation: I'm too big and therefore disgusting. Later when I was a teenager she was worried that I won't find a boyfriend as I am too tall.
Also my cousin who was bullying me would often mention that I was older, bigger, more boring, and therefore didn't fit in.
Being older sibling was traumatic in itself for me.