paged listServiceOffering response returns duplicated records in different pages #5316

Activity

1. borisstoyanov

   added

   component:api

   on Aug 16, 2021
2. shwstppr commented on Aug 16, 2021

   

   shwstppr

   on Aug 16, 2021

   Contributor

   More actions

   I am able to reproduce this with a Simulator env for both compute and disk offerings. Not sure if the problem happens with other resources as well.
   I created 1000 compute and disk offerings. Tried listing pages for disk offering with pagesize=500. It resulted in many duplicates. Logs for disk offerings:
   diskoffering.log

   While setting global config default.page.size to 1200 and listing 1000 items at once there were no duplicates,

   shwstppr@shwstppr-ThinkPad:~/Desktop/temp|
   ⇒  cmk list diskofferings filter=id page=1 pagesize=1000 > ./dpage3
   shwstppr@shwstppr-ThinkPad:~/Desktop/temp|
   ⇒  sort ./dpage3 | uniq -d
   +--------------------------------------+
   

   @rhtyd @weizhouapache @nvazquez could this be a caching issue with ApiServer?
3. nvazquez commented on Aug 16, 2021

   

   nvazquez

   on Aug 16, 2021

   Contributor

   More actions

   Thanks for testing @shwstppr, I checked the code for listDiskOfferings and the QueryManager is simply obtaining data from the DAO layer using a search criteria and a filter based on the start index and the page size value. Perhaps the filter can be replaced by this method: StringUtils.applyPagination?
4. weizhouapache commented on Aug 16, 2021

   

   weizhouapache

   on Aug 16, 2021

   Last edited by weizhouapache

   Member

   More actions

   @shwstppr @borisstoyanov
   is it similar as #3934 which has been fixed by #4494 ?

   try ?

   UPDATE `cloud`.`disk_offering` JOIN (SELECT COUNT(1) AS count FROM `cloud`.`disk_offering` WHERE `sort_key` != 0) AS tbl_tmp SET `sort_key` = `id` WHERE count = 0;
   

   what should the default value of sort_key be, 0 or id ?
5. shwstppr

   added a commit that references this issue on Aug 16, 2021

   db: use id column as secondary sort criteria

   ...

   This issue will close once commit 137f37f is merged into the 'main' branch.

   137f37f
6. shwstppr commented on Aug 16, 2021

   

   shwstppr

   on Aug 16, 2021

   Contributor

   More actions

   @weizhouapache @nvazquez
   By default value for sort_key field remains 0. Maybe @borisstoyanov can confirm but this issue is seen with older versions as well 4.9 probably.
   I've created a PR to use id field as secondary sort criteria which fixes the issue in my tests.
7. weizhouapache commented on Aug 16, 2021

   

   weizhouapache

   on Aug 16, 2021

   Member

   More actions

   @weizhouapache @nvazquez
   By default value for sort_key field remains 0. Maybe @borisstoyanov can confirm but this issue is seen with older versions as well 4.9 probably.
   I've created a PR to use id field as secondary sort criteria which fixes the issue in my tests.

   @shwstppr sounds like a better idea.
8. nvazquez

   added this to the 4.16.0.0 milestone on Aug 16, 2021
9. nvazquez

   added

   type:bug

   on Aug 16, 2021
10. shwstppr

    added a commit that references this issue on Aug 17, 2021

    server: use id column as secondary sort criteria with sortKey

    ...

    This issue will close once commit 169411e is merged into the 'main' branch.

    169411e
11. yadvr

    closed this as completedin #5320on Aug 18, 2021
12. yadvr

    added a commit that references this issue on Aug 18, 2021

    server: use id column as secondary sort criteria with sortKey (#5320)

    ...

    This issue will close once commit 7bfc7f0 is merged into the 'main' branch.

    7bfc7f0