How to further restrict traffic to DUO Access Gateway from the Internet?

Since the point of Office 365 is to allow access to your applications (in our case primarily email through Outlook) no matter where you are, restricting access to the DUO Access Gateway server is not really feasible based on source IP addresses. The user could literally be connecting from any IP based on what network they are coming from. It’s also not really feasible to filter on the firewall based on user group because unlike say a VPN connection with FortiClient, there isn’t going to be a way to check against an AD user group when the user tries to access https://login.microsoftonline.com and instead gets the DUO Access Gateway SSO URL.

I am curious as to how everyone is defining the firewall rule to allow access to the DUO Access Gateway. It looks like I have to define the destination as the DUO Access Gateway with port 443 and set the source field to “any” on the public WAN (Internet) interface of the firewall. Even though the server we built is on a secured DMZ network with only the necessary connections (to LDAP through read-only domain controllers, etc.), I still don’t feel too fuzzy about allowing any and all IPs 443 access to a server in our DMZ environment.