Yes, you could fish in any clothing. It's fishing, right? If you were in a 3-piece suit, and you had the chance to stop and fish for an hour, you would, wouldn't you. I would. But that's not exactly the point. If you're sweating so profusely you dehydrate yourself within an hour and then have to crawl back to the car, you miss out on fishing. If you're shivering so violently you can't tie a fly on, you miss out on fishing. Find the right clothes for the job, high-performance fabrics that will keep you from missing out on fishing.