!d discord not working correctly

Invoking !d discord to lookup the the discord.py documentation is currently not working as intended. It causes the bot start typing and never produces an output.

It seems to be a specific issue with the discord look-up in particular, but more work into what specifically the error could be caused by is needed. The logging present currently isn't enough to trace down the specific issue in production.

It could possibly be an issue with the refresh docs method here:

bot/bot/exts/info/doc/_cog.py

Lines 211 to 229 in ad6c7f8

    
           async def refresh_inventories(self) -> None: 
        
               """Refresh internal documentation inventories.""" 
        
               self.refresh_event.clear() 
        
               await self.symbol_get_event.wait() 
        
               log.debug("Refreshing documentation inventory...") 
        
               self.inventory_scheduler.cancel_all() 
        
               self.base_urls.clear() 
        
               self.doc_symbols.clear() 
        
               self.renamed_symbols.clear() 
        
               await self.item_fetcher.clear() 
        
               coros = [ 
        
                   self.update_or_reschedule_inventory(package["package"], package["base_url"], package["inventory_url"]) 
        
                   for package in await self.bot.api_client.get("bot/documentation-links") 
        
               ] 
        
               await asyncio.gather(*coros) 
        
               log.debug("Finished inventory refresh.") 
        
               self.refresh_event.set()

That method does not check if a refresh is already underway.

Works for me here

Works for me here

It does not always produce this result, we haven't seen this behaviour in a bit I don't think but it's (as specified originally) not something that always manifests.

Happened again yesterday after I attempted to remove the disnake inventory (because it seemed to have moved). After that, all !d invocations led to this issue until the bot was restarted.

Reproduction

The stuck typing issue is reproducable by running:

!d setdoc disnake https://docs.disnake.dev/en/latest/objects.inv https://docs.disnake.dev/en/latest/api/
!d disnake.Role.color
!d refreshdoc

Issue

Due to the invalid base URL !d disnake.Role.color fails with this traceback

2025-10-07 22:39:01 2025-10-07 20:39:01 | WARNING | bot.exts.info.doc._cog | A network error has occurred when requesting parsing of DocItem(package='disnake', group='property', base_url='https://docs.disnake.dev/en/latest/api/', relative_url_path='api/roles.html', symbol_id='disnake.Role.color').
2025-10-07 22:39:01 Traceback (most recent call last):
2025-10-07 22:39:01   File "/bot/bot/exts/info/doc/_cog.py", line 258, in get_symbol_markdown
2025-10-07 22:39:01     markdown = await self.item_fetcher.get_markdown(doc_item)
2025-10-07 22:39:01                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-07 22:39:01   File "/bot/bot/exts/info/doc/_batch_parser.py", line 111, in get_markdown
2025-10-07 22:39:01     async with bot.instance.http_session.get(doc_item.url, raise_for_status=True) as response:
2025-10-07 22:39:01                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-10-07 22:39:01   File "/build/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 1488, in __aenter__
2025-10-07 22:39:01     self._resp: _RetType = await self._coro
2025-10-07 22:39:01                            ^^^^^^^^^^^^^^^^
2025-10-07 22:39:01   File "/build/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 897, in _request
2025-10-07 22:39:01     resp.raise_for_status()
2025-10-07 22:39:01     ~~~~~~~~~~~~~~~~~~~~~^^
2025-10-07 22:39:01   File "/build/.venv/lib/python3.13/site-packages/aiohttp/client_reqrep.py", line 629, in raise_for_status
2025-10-07 22:39:01     raise ClientResponseError(
2025-10-07 22:39:01     ...<5 lines>...
2025-10-07 22:39:01     )
2025-10-07 22:39:01 aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url='https://docs.disnake.dev/en/latest/api/api/roles.html'

This method is where the issue is introduced

bot/bot/exts/info/doc/_batch_parser.py

Lines 99 to 123 in f0e4d45

    
               async def get_markdown(self, doc_item: _cog.DocItem) -> str | None: 
        
                   """ 
        
                   Get the result Markdown of `doc_item`. 
        
                   If no symbols were fetched from `doc_item`s page before, 
        
                   the HTML has to be fetched and then all items from the page are put into the parse queue. 
        
                   Not safe to run while `self.clear` is running. 
        
                   """ 
        
                   if doc_item not in self._item_futures and doc_item not in self._queue: 
        
                       self._item_futures[doc_item].user_requested = True 
        
                       async with bot.instance.http_session.get(doc_item.url, raise_for_status=True) as response: 
        
                           soup = await bot.instance.loop.run_in_executor( 
        
                               None, 
        
                               BeautifulSoup, 
        
                               await response.text(encoding="utf8"), 
        
                               "lxml", 
        
                           ) 
        
                       self._queue.extendleft(QueueItem(item, soup) for item in self._page_doc_items[doc_item.url]) 
        
                       log.debug(f"Added items from {doc_item.url} to the parse queue.") 
        
                       if self._parse_task is None: 
        
                           self._parse_task = scheduling.create_task(self._parse_queue(), name="Queue parse")

Normally:

A future is created tracking the doc item and added to self._item_futures
A request is sent to the docs
The resultant html is added to a queue to be processed
A task is started to process this queue, which should in turn set_result on this future.

However, in our case, 2 failed, meaning 3 and 4 were not run, so there is a future stuck in self._item_futures that will never be resolved.

This becomes an issue when the refresh command is run, as it waits for the futures in self._item_futures to resolve, which will never happen for these stuck futures.

bot/bot/exts/info/doc/_batch_parser.py

Lines 187 to 188 in f0e4d45

    
           for future in filter(attrgetter("user_requested"), self._item_futures.values()): 
        
               await future

Fix

A simple fix would be to catch the exception from the http session and immediately set_result(None) on the future.

This code feels very fragile and complex currently though, so having a deeper look and considering a refactor may be worth looking into (though wouldn't need to be block an initial fix).

I have not looked into the code further to see if there are other related issues.

Navigation Menu

Repository navigation

`!d discord` not working correctly #3326

Reproduction

Issue

Fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	async def refresh_inventories(self) -> None:
	"""Refresh internal documentation inventories."""
	self.refresh_event.clear()
	await self.symbol_get_event.wait()
	log.debug("Refreshing documentation inventory...")
	self.inventory_scheduler.cancel_all()

	self.base_urls.clear()
	self.doc_symbols.clear()
	self.renamed_symbols.clear()
	await self.item_fetcher.clear()

	coros = [
	self.update_or_reschedule_inventory(package["package"], package["base_url"], package["inventory_url"])
	for package in await self.bot.api_client.get("bot/documentation-links")
	]
	await asyncio.gather(*coros)
	log.debug("Finished inventory refresh.")
	self.refresh_event.set()

	async def get_markdown(self, doc_item: _cog.DocItem) -> str \| None:
	"""
	Get the result Markdown of `doc_item`.

	If no symbols were fetched from `doc_item`s page before,
	the HTML has to be fetched and then all items from the page are put into the parse queue.

	Not safe to run while `self.clear` is running.
	"""
	if doc_item not in self._item_futures and doc_item not in self._queue:
	self._item_futures[doc_item].user_requested = True

	async with bot.instance.http_session.get(doc_item.url, raise_for_status=True) as response:
	soup = await bot.instance.loop.run_in_executor(
	None,
	BeautifulSoup,
	await response.text(encoding="utf8"),
	"lxml",
	)

	self._queue.extendleft(QueueItem(item, soup) for item in self._page_doc_items[doc_item.url])
	log.debug(f"Added items from {doc_item.url} to the parse queue.")

	if self._parse_task is None:
	self._parse_task = scheduling.create_task(self._parse_queue(), name="Queue parse")

	for future in filter(attrgetter("user_requested"), self._item_futures.values()):
	await future

!d discord not working correctly #3326

Description

Activity

DragonSenseiGuy commented on Sep 26, 2025

jb3 commented on Sep 26, 2025

L3viathan commented on Oct 7, 2025

wookie184 commented on Oct 7, 2025

Reproduction

Issue

Fix

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`!d discord` not working correctly #3326