You can use a Yubikey in OTP mode for online authentication, but you cannot use U2F for online authentication.
You can use a Yubikey as U2F for offline authentication, but you cannot use OTP for offline authentication.
You can accomplish both with one device if you get something like the Yubikey 5C nano, import it to Duo as a hardware token and assign it to a user (for OTP online auth), and then the end-user can also enroll it as a U2F for local Windows logon offline access or for use with the Duo authentication prompt for web applications.
We are using Duo for Windows RDP on a server.
I want to change one user to use a Yubico.
Note that U2f for offline over RDP connections isn’t supported. Offline access over RDP must use Duo Mobile to authenticate. It’s the second limitation bullet here.