In Transmit 5 the default unicode normalization mode has changed from NFD to NFC.

This setting changes the way UTF-8 file names are sent to the server. Linux and Windows-based servers use NFC, while macOS-based servers use NFD. (Read more about unicode normalization formats.)

If you know you are connecting to an macOS server, you need to set Text Composition to NFD in the Advanced Server Settings.