Yes, I understand your problem. Once the failover occurs, the SLA destination is reachable again, so it thinks that connectivity has been restored so it goes back to the original connection. This connection is still down so it fails over again and so on.
In order to differentiate between which connection is down, it is possible to indicate from which specific interface your IP SLA will be tested using the following command:
icmp-echo a.b.c.d source-interface gig0/0
In this way you can set up the ping to occur on the interface that connects to ISP 1. This means when the failover occurs, the SLA pings will still be sent from the “failed” connection, thus they continue to fail until the specific connection recovers.
I hope this has been helpful!