The safest approach is to set QoS in both directions. At my company, we are concerned with prioritizing VOIP, print jobs, and SSH. VOIP is a bit easier since the VOIP server and phones automatically mark their traffic as DSCP EF, so we just trust those markings, but with the others, we do, in fact, mark them similar to the example I provided earlier where the classifier for return trip looks to the source port, not the destination.
If you knew that your remote sites, for example, had more of a problem with downloads saturating the bandwidth than uploads, you probably could set QoS in just the one direction and be fine. We set it both ways “just in case.”