AAD Primary Auth and Conditional Access

We are looking to use Azure AD as our primary auth for SSO, while also protecting M365 with Conditional Access MFA.
Has anyone else got this setup?

One issue that we are running into is the number of consecutive Duo Push requests.
1x when signing in to M365
1x for Duo Central
1x when opening a protected app