Kishor,
I assume you mean the “ExStart” or “Exchange” state, so I will write about those. If OSPF is having an authentication problem, you will not see the routers stuck in ExStart or Exchange. In fact, you won’t see anything at all. The output for “show ip ospf neighbors” will just be blank (if a neighbor relationship hadn’t already formed). If a neighbor relationship already formed, and then an authentication problem is introduced, the neighbors will just drop once the dead interval is reached. The reason, in both cases, is because if there is an authentication mismatch, then the other router’s Hello message will simply be ignored. By ignoring Hello messages, the OSPF state machine will never even begin, let alone get to an ExStart phase.
To answer your other question, the most common reason by far for OSPF to be stuck in ExStart is because of an MTU mismatch between neighbors. Besides MTU mismatches, other possibilities include duplicate router IDs, access-list that block unicast packets, or NAT misconfigurations.
I recommend you check out a Cisco article that goes into great detail on OSPF getting stuck. You will find they have a very detailed 14 step explanation as to why an MTU mismatch causes this problem.
http://www.cisco.com/c/en/us/support/docs/ip/open-shortest-path-first-ospf/13684-12.html#exstart