Vladimir,
I am pretty sure that an Embedded RP will work on that platform. One key configuration step that I didn’t see you mention is that you must manually configured the chosen RP to be an RP for itself, via the ipv6 pim rp-address command.
After the RP has been statically configured to itself, your embedded RP address (assuming it is configured correctly!) should work.
You are correct in that if you are using a /128 loop back address, you would need to change the “40” to an “80” since those two digits represent the mask in hexadecimal.
If this still doesn’t work, please post your RP information, MC group info, etc., and we can double check your calculation of the embedded RP address.