Duo request before valid password

I’ve run into an interesting problem on an externally facing server that excepts password based authentication.

The server is running Centos 7.7

If a user tried to login with a password via SSH, and types in a bad password Duo still pushes out a notification prompt to the end user. If the user accepts the prompt they are then prompted for the password again (followed by another Duo push)

Is there a way to get this so Duo only pushes on a successful password entry? This server gets quite a few password scans run against it, and even with fail2ban the users sill get several pushes that don’t need to happen before the scanning IP is banned.

PAM.d ssh config:

#%PAM-1.0
auth       substack     password-auth
auth       include      postlogin
auth       required    pam_env.so
auth       sufficient   pam_duo.so
auth       required    pam_deny.so
account    required     pam_sepermit.so
account    required     pam_nologin.so
account    include      password-auth
password   include      password-auth
# pam_selinux.so close should be the first session rule
session    required     pam_selinux.so close
session    required     pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session    required     pam_selinux.so open env_params
session    required     pam_namespace.so
session    optional     pam_keyinit.so force revoke
session    optional     pam_motd.so
session    include      password-auth
session    include      postlogin

Replying to myself again, as a workaround (in case someone else has this problem) I’ve switched to using login_duo rather than PAM. I’d still really like to use PAM, but at least on this externally facing system I don’t have to worry about spamming my admins with false requests.