Hello AZM
Now when you say that you are seeing packet discards in the trust and untrust zone, I assume these drops are occurring on interfaces found on the firewalls? Where specifically is the bandwidth saturation taking place, on the interfaces of the firewalls?
Assuming that is indeed the case, since there is no marking anywhere in this network, then using source and destination IP addresses for applying a QoS policy sounds like a good idea. If your applications are such that source and destination addresses can sufficiently identify traffic that requires QoS then it looks good to me. However you wouldn’t be able to implement this on the G0/1 and G0/2 interfaces of the L2 switch, because it is L2. This would have to be implemented on the firewalls.
It is not always best practice to have a firewall implement QoS, because it is already doing filtering and many other high CPU usage functionalities. Whenever possible, offload some of these operations from firewalls to other devices. Although it makes configuration more complicated, you could implement QoS on the Core switches for traffic destined towards the cloud based applications, and on the Internet routers for traffic coming from the cloud based applications to your internal users. All of this assumes no other sources of traffic traversing the firewalls other than those shown in the diagrams.
As for your other question, yes you are right that GLBP will not help for load balancing. However, Cisco Nexus platform switches configured in HSRP configuration will always load balance between the two (or more) available links. This is the default behaviour, and actually cannot be modified. So if the Internet routers are Nexus devices, you’re good to go. If not, the best choice is to implement BGP dual homing and have each firewall connect to both routers.
I hope this has been helpful!
Laz