Sport coats are technically supposed to be part of a casual outfit. In the old days, it used to be that on the days that one was not wearing a suit, it would be customary to relax at home with a sports coat on instead. Boy have the times changed. Today, the moment you put on a sport coat, you'll get plenty of questions asking why you're so dressed up and formal, not dressed down. Overly casual dress has become the name of the game, and thus I see far less sport coats worn in public. Instead, a weekend outfit would more likely be comprised of a North Face jacket, distressed jeans, and Vans than a sports coat and chinos.
The traditional way to wear a sport coat would be to wear it as an odd jacket that pairs well with, but doesn't match your trousers. For example, people will often wear navy sport coats with grey flannel trousers. Or maybe heavy brown tweed sport coats with tan khakis.
Well, even though the times have changed and people aren't wearing their sport coats as traditionally as I would prefer, I still think there are plenty of ways to wear them casually, all without looking like too much effort is being put forth. Here's an example of how I would wear a sports coat fairly casually:
I'm wearing this Harris Tweed sport coat with a pair of dark denim, a casual button up, and a brown belt. Since the jeans, shirt, and belt all scream casual, I thought putting a tweed sport coat into the mix would be a good way to dress the outfit up, but still remain relaxed.
Tweed is a good material for the fall, and can look a bit less dressy when compared to a worsted wool jacket that looks like it came straight from a suit. For this reason, tweed allows you to dress down easier than other sport coats probably can.
This applies to materials like seersucker and corduroy as well (tweed and corduroy are more for the winter, cotton and seersucker are more for the summer).
Now, technically you could still dress this type of outfit down even further, going with a T-shirt or sweater instead of a button-up shirt. I say go for it as long as everything looks nice and fitted to your body. Not my first preference, but I'd say it's fair game. All I'm really trying to say is that the sport coat was, is, and always will be an amazing piece in menswear and because people consider it too formal today, it is utilized far too sparingly.
If you have to dress a sport coat down to feel comfortable wearing it out, do so, as I think it will still elevate your style game. You'll get much joy out of wearing one, I promise.