Wikipeda scrapers aborting from *HTTP Error 403: Forbidden* #128

Activity

1. a2z-tech-innovations commented on Nov 27, 2025

   

   a2z-tech-innovations

   on Nov 27, 2025

   Last edited by a2z-tech-innovations

   More actions

   @royrada Wikipedia just blocks requests without headers, request via urlib excludes them by default.

   from urllib.request import urlopen, Request
   from bs4 import BeautifulSoup
   
   url = 'http://en.wikipedia.org/wiki/Kevin_Bacon'
   
   headers = {
       'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
   }
   
   req = Request(url, headers=headers)
   html = urlopen(req)
   
   bs = BeautifulSoup(html, 'lxml')
   
2. royrada commented on Nov 27, 2025

   

   royrada

   on Nov 27, 2025

   Author

   More actions

   Thank you very much. I implemented your suggestion of adding a header to pretend to be a web browser and after that Wikipedia executed without error the python program from Ryan Mitchell's book. Ryan has discussed the sophisticated strategies that web servers can employ to stop computer programs from crawling and scraping their websites. She noted that the header facade was so easily implemented that web servers were likely to go to behavioral monitoring. I assumed that Wikipedia was using behavioral monitoring. As I am learning how to scrape the web, your guidance of inserting a browser heading will be followed and each time with a virtual 'thank you' from me to you.