Hello Tanmoy
The Active state can be reached via two situations. The first is when a BGP peering goes from Connect to Active. This will take place if the Connect three-way handshake fails. The other case is going from OpenSent to Active. This will take place again, if the TCP session fails. So theoretically, yes, two routers in Active state will be able to ping each other, but are in a situation where they have not yet successfully completed the TCP three way handshake.
If two routers are configured correctly to be BGP peers, and they remain in the idle state, then yes, it is most likely an L2 or L3 problem. However, it could also be a misconfiguration on one of the routers such as an incorrect neighbor IP address.
Like you said, we don’t generally see a “stuck in connected” state. This is clearly described in the BGP Neighbor Adjacency States lesson:
Connect: BGP is waiting for the TCP three-way handshake to complete. When it is successful, it will continue to the OpenSent state. In case it fails, we continue to the Active state. If the ConnectRetry timer expires then we will remain in this state. The ConnectRetry timer will be reset and BGP will try a new TCP three-way handshake. If anything else happens (for example resetting BGP) then we move back to the Idle state.
I hope this has been helpful! Stay healthy and safe!
Laz