Thanks for the additional information about the steps you’re taking. Are you trying to use BOTH Duo Access Gateway and AD FS at the same time? Your process is a bit confusing.
Are you pointing DAG to AD FS as a SAML authentication source, or are you pointing DAG to the same AD used by AD FS as an LDAP authentication source? Either way, there should be no federation redirect from vCloud Director to AD FS once you have introduced DAG to the login path, so I am not sure why you would upload anything from vCloud Director to AD FS (unless I am misunderstanding your step 3).
Are you following the vCloud directions here? Did you map the attributes specified in the second paragraph? Note that the Duo generic SAML application doesn’t support sending group information today.
The answer likely lies in the vCloud logs. Somewhere it should tell you why exactly it is rejecting the SSO login, and you could take that information and work backwards to adjust the SSO config accordingly.
Since your configuration and issues are complex, you should consider contacting Duo Support. This community isn’t