Aside from ease of use of a dumb piece of RFID equipped plastic, another scenario in a busy transport hub/shop/other service location:
I drop/someone steals contactless payment card from my hands; outcome - I shrug shoulders, cancel card, get new card, in the meantime use a different card/cash.
I drop/someone steals phone from my hands; outcome - I have to work out what was on there, what might have been compromised, attempt to remotely nuke phone (need to find another handset/computer to do this ASAP), have to get phone replaced, have to restore data to new device, change passwords as a precaution, in the meantime social life is curtailed (malicious actors might even manipulate it and cause further harm at various levels), etc.
I'm reasonably happy with the dumb pieces of plastic as they permit me to independently sandbox the financial and personal data aspects.