Pagination bug in admin api?

[mchyzer12]

When paging in the API, since things are index based, there are race conditions where records can be omitted or sent twice. (if deletes or inserts happen while retrieving all)

e.g.

lets say there are groups:

test1, test2, test3, test4

Then I call /groups with limit of 2. I get a next offset of 2 (since offset 0 and 1 were sent back in first call.)

Then someone else deletes “test1”. Then when calling /groups with an offset of 2, then test4 is returned. And not test3! There is a missing record!

Instead it should do paging by returning the ID of last one returned, and pass that in the next call as “return after this ID”, and it will be consistent with no race conditions.

This seems like an important bug right?

Thanks
Chris

[Dooley]

Hi Chris,
You are correct that this is a potential problem depending on how pagination is implemented. One way to mitigate this would be to start the next page one record early so that you fetch the last record of the previous page again. If the last of the previous page and the first of the current page are not the same, then you know something was deleted or inserted. Our developers understand this isn’t a perfect solution, but we are requiring pagination with an emphasis on performance and stability.

Please also note that the default and/or maximum limit values are all much greater than those used in your example scenario – for instance 100 is the limit for lists of groups. All of those values are available in our Admin API documentation: Duo Admin API | Duo Security.

[mchyzer12]

Can you please:

1. Document the API docs so it clearly says how to reliably use the existing paging in this manner (i.e. call with the next_offset-1, check to see if its the right item, if not, exit and try again, maybe loop 10 times until consistent)?
2. Open a ticket to re-implement the paging so it is easy to use and reliable.

Yes, obviously my example was just to prove the issue, not a limit that would actually be used.

thanks
Chris

[Dooley]

Hi Chris,
I have passed this feedback along to the our API Team. I would recommend you separately file a support case from an email associated with your Duo Administrator account or contact your AE or CSM if you have one to share these requests as well.