I haven't tested this lately, but in the past the takeaway was:
pam_duo doesn't work with this, because you can't choose Authentication Methods (like pubkey) AND/OR PAM.
Instead, you might try login_duo https://duo.com/docs/loginduo You may be able to add it after PubKey and Password. Also note the security concerns you need to be aware of when using login_duo -
We strongly recommend that you disable PermitTunnel and AllowTcpForwarding in your sshd_config when using login_duo to protect SSH logins. Since OpenSSH sets up port forwarding and tunneling before Duo’s two-factor challenge, an attacker may be able to access internal services via port forwarding before completing secondary authentication. Adding the following lines to your sshd_config will prevent this scenario:
note: this will likely work because you have pubkey enabled only for convenience, not security- since password is still enabled for all users. This makes it so you don't have any logic sorting if clauses for pass fail on each one, and they just move down sequentially. If you wanted to have certain users get different auth logic, you might try the Match command to send different groups down different auth paths. https://raymii.org/s/tutorials/Limit_access_to_openssh_features_with_the_Match_keyword.html