Repository navigation
[bug] Search not finding all files #495
Description
Describe the bug
Hello,
We setup Sourcebot last week and connected it to our self-hosted GitLab instance.
After a couple of days it figured out, the search is not finding all occurences of a specific filename or string.
Indexing should be done for all ~600 repos. Any idea?
To reproduce
Example 1:
repo:<team1> file:pom.xml "xml version" -> 7 search results
Same search with npm tool gitlab-search lists around 50 results in the same group:
gitlab-search -g <team1> -f pom.xml "xml version "
Example 2:
repo:<team2> "test" results in 158 matches in 44 files, but only 4 repos of that group.
Same search with npm tool gitlab-search lists around 153 different files
gitlab-search -g <team2> "test"
Sourcebot deployment information
Sourcebot-Version: 4.6.6
Additional information
Configration:
{
"all": true,
"url": "https://gitlab.company.io",
"type": "gitlab",
"token": {
"secret": "SourceBot-GitLab-Token"
},
"exclude": {
"archived": true
}
}
Activity
brendan-kellam commentedon Sep 8, 2025
heya thx for raising - I suspect you may be hitting #492. Are you seeing any "Failed to sync repository" logs?
weberpatr commentedon Sep 9, 2025
Yes, I have one failed repository - which is also an empty repository:
weberpatr commentedon Sep 9, 2025
Just updated to v4.6.7.
The failed sync is now solved, but the mentioned search results did not change. Still missing a lot of results.
musabshak commentedon Sep 9, 2025
We were running into this as well. With a search query that specified a particular repository and a keyword, there would be lots of results. But searching for the same keyword without specifying repository didn't yield results, but there were also no error logs in the UI or in the server.
The solution turns out lies in two things:
The action item on Sourcebot-dev's end is to surface the Zoekt timeout errors up to the Sourcebot server logs and to the UI so the users at least know what's going on.
This is really important from an accuracy POV, so one could rely on sourcebot w.r.t questions of the form "do any references to this keyword exist in my codebase at all"?
Currently, sourcebot is like the opposite of a bloom filter: if it returns matches, you know for sure the matches exist, but if it doesn't return a match, you don't know for sure that a match does not exist, and this is not good enough.
brendan-kellam commentedon Sep 11, 2025
@weberpatr how long are these searches typically taking? Assuming they are around 10 seconds, then @musabshak theory around
ZOEKT_MAX_WALL_TIME_MSis likely correct.100% agreed
Also agreed - having some kind of "stop reason" would be helpful. Tracking in #504
weberpatr commentedon Sep 12, 2025
The search is not taking longer than 1-2 secs. Anyway I increased
ZOEKT_MAX_WALL_TIME_MSfrom 10000 to 30000.Is there any caching of search results?
Maybe I'm trying to delete the local data directory and force a re-index of all 500 repos?
brendan-kellam commentedon Sep 15, 2025
There is client side caching, but refreshing the page will flush the cache and refetch.
Yea it could be worth the shot. If the error persists, would be great to jump on a call if you have a minute this week.
weberpatr commentedon Sep 17, 2025
at the moment I'm stucked because of #509 :)
weberpatr commentedon Sep 19, 2025
After a clean re-setup on v4.7.0 it seems to work now.
The re-index succeeded without any error message and the test searches (from the first post) now gives a more plausible result than before. I also increased the
ZOEKT_MAX_WALL_TIME_MSto30000.