A look less formal

It’s not often that I dress down, but when I do my style remains quite… preppy, I suppose.

What does a man wear with a rugby shirt if he hates jeans? Slacks are a mismatch, I don’t like shorts, and khaki pants just don’t seem to look right – are there any other options that I’m missing?

Thanks guys,