Repository navigation
Feeds requiring cookie support #39
Description
Hello,
Thanks for pluto, we moved the neuroscience planets over from venus a few days ago:
https://github.com/neurofedora/planet-neuroscience
https://github.com/neurofedora/planet-neuroscientists
For the neuroscience planet, which collects feeds from research journals, we're seeing errors which seem to be related to cookies:
[debug] fetch - get(_response) src: https://link.springer.com/search.rss?search-within=Journal&facet-journal-id=10827
[debug] using direct net http access; no proxy configured
[debug] GET /search.rss?search-within=Journal&facet-journal-id=10827 uri=https://link.springer.com/search.rss?search-within=Journal&facet-journal-id=10827, redirect_limit=5
[info] found cache entry for >https://link.springer.com/search.rss?search-within=Journal&facet-journal-id=10827<
[debug] 302 Found location=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/search.rss?search-within=Journal&facet-journal-id=10827&client_id=springerlink&response_type=cookie
[debug] GET /authorize?redirect_uri=https://link.springer.com/search.rss?search-within=Journal&facet-journal-id=10827&client_id=springerlink&response_type=cookie uri=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/search.rss?search-within=Journal&facet-journal-id=10827&client_id=springerlink&response_type=cookie, redirect_limit=4
[debug] 302 Found location=https://idp.springer.com/transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fsearch.rss%3Fsearch-within%3DJournal&code=5a9cf68b-e1ee-417a-8ca8-c6c202202caf
[debug] GET /transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fsearch.rss%3Fsearch-within%3DJournal&code=5a9cf68b-e1ee-417a-8ca8-c6c202202caf uri=https://idp.springer.com/transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fsearch.rss%3Fsearch-within%3DJournal&code=5a9cf68b-e1ee-417a-8ca8-c6c202202caf, redirect_limit=3
[debug] 302 Found location=https://link.springer.com/search.rss?search-within=Journal&error=cookies_not_supported&code=5a9cf68b-e1ee-417a-8ca8-c6c202202caf
[debug] GET /search.rss?search-within=Journal&error=cookies_not_supported&code=5a9cf68b-e1ee-417a-8ca8-c6c202202caf uri=https://link.springer.com/search.rss?search-within=Journal&error=cookies_not_supported&code=5a9cf68b-e1ee-417a-8ca8-c6c202202caf, redirect_limit=2
[debug] 302 Found location=null?idp_error=cookies_not_supported
[debug] url relative; try to make it absolute
[debug] GET /null?idp_error=cookies_not_supported uri=https://link.springer.com/null?idp_error=cookies_not_supported, redirect_limit=1
[debug] 302 Found location=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/null?idp_error=cookies_not_supported&client_id=springerlink&response_type=cookie
[debug] GET /authorize?redirect_uri=https://link.springer.com/null?idp_error=cookies_not_supported&client_id=springerlink&response_type=cookie uri=https://idp.springer.com/authorize?redirect_uri=https://link.springer.com/null?idp_error=cookies_not_supported&client_id=springerlink&response_type=cookie, redirect_limit=0
[debug] 302 Found location=https://idp.springer.com/transit?redirect_uri=https%3A%2F%2Flink.springer.com%2Fnull%3Fidp_error%3Dcookies_not_supported&code=e99fb052-66cc-4e41-ba89-9072db17e558
*** error: HTTP redirect too deep
/home/asinha/.local/share/gem/ruby/gems/fetcher-0.4.5/lib/fetcher/worker.rb:166:in `get_response': HTTP redirect too deep (ArgumentError)
from /home/asinha/.local/share/gem/ruby/gems/fetcher-0.4.5/lib/fetcher/worker.rb:56:in `get'
from /home/asinha/.local/share/gem/ruby/gems/pluto-feedfetcher-0.1.6/lib/pluto/feedfetcher/cond_get_with_cache.rb:34:in `fetch'
from /home/asinha/.local/share/gem/ruby/gems/pluto-update-1.6.4/lib/pluto/update/feed_refresher.rb:57:in `refresh_feed_worker'
from /home/asinha/.local/share/gem/ruby/gems/pluto-update-1.6.4/lib/pluto/update/feed_refresher.rb:49:in `block in refresh_feeds_for'
from /home/asinha/.local/share/gem/ruby/gems/activerecord-6.1.4/lib/active_record/relation/delegation.rb:88:in `each'
from /home/asinha/.local/share/gem/ruby/gems/activerecord-6.1.4/lib/active_record/relation/delegation.rb:88:in `each'
from /home/asinha/.local/share/gem/ruby/gems/pluto-update-1.6.4/lib/pluto/update/feed_refresher.rb:48:in `refresh_feeds_for'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/lib/pluto/cli/updater.rb:51:in `update_for'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/lib/pluto/cli/updater.rb:31:in `run'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/lib/pluto/cli/main.rb:289:in `block (3 levels) in <top (required)>'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/lib/pluto/cli/main.rb:270:in `each'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/lib/pluto/cli/main.rb:270:in `block (2 levels) in <top (required)>'
from /home/asinha/.local/share/gem/ruby/gems/gli-2.20.1/lib/gli/command_support.rb:131:in `execute'
from /home/asinha/.local/share/gem/ruby/gems/gli-2.20.1/lib/gli/app_support.rb:296:in `block in call_command'
from /home/asinha/.local/share/gem/ruby/gems/gli-2.20.1/lib/gli/app_support.rb:309:in `call_command'
from /home/asinha/.local/share/gem/ruby/gems/gli-2.20.1/lib/gli/app_support.rb:83:in `run'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/lib/pluto/cli/main.rb:390:in `<top (required)>'
from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /home/asinha/.local/share/gem/ruby/gems/activesupport-6.1.4/lib/active_support/dependencies.rb:332:in `block in require'
from /home/asinha/.local/share/gem/ruby/gems/activesupport-6.1.4/lib/active_support/dependencies.rb:299:in `load_dependency'
from /home/asinha/.local/share/gem/ruby/gems/activesupport-6.1.4/lib/active_support/dependencies.rb:332:in `require'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/lib/pluto.rb:24:in `main'
from /home/asinha/.local/share/gem/ruby/gems/pluto-1.3.4/bin/pluto:5:in `<top (required)>'
from /home/asinha/bin/pluto:23:in `load'
from /home/asinha/bin/pluto:23:in `<main>'
https://link.springer.com/search.rss?search-within=Journal&facet-journal-id=10827 works fine in a browser, but not on pluto---and it looks like they keep redirecting which causes pluto to error out eventually.
Is there a way to use these feeds with pluto?
Activity
chore: update springer rss feed links
geraldb commentedon Aug 5, 2021
Thanks for highlighting the neuroscience planets.
The http library used is ruby's standard Net::HTTP with a little convenience wrapper (for handling redirections and caching but not cookies), See https://github.com/rubycoco/webclient/tree/master/fetcher
If you have pluto installed locally you should also have the fetcher gem install which includes a debug command line tool called fetch. Try:
That should give you the same result, please what results you get (theoratically it's the same but "isolated" for easy testing. We can take it from there. You might check what is the spring policy on feeds and if you need a token or something.
Ok I tried:
That gets redirected to https://idp.springer.com/authorize?redirect_uri=[url] - not sure what to do here.
Not sure if you can do ruby scripts - if you can - than use the standard Net:HTTP library and make a script with a successful request with cookie and I am happy to patch / update the fetcher based on your script. This cheatsheet might help to get your started https://yukimotopress.github.io/http.
sanjayankur31 commentedon Aug 5, 2021
Thanks for looking into it @geraldb . I've dug in a little more. Using something like
wgetworks fine:However, if I stop wget from dealing with cookies, I get the same redirection loop:
So I guess the fix/workaround here would be to get fetcher etc. to support cookies. I don't do a lot of ruby, but I'll see if someone from the team can find the time to work on this.
geraldb commentedon Aug 5, 2021
Thanks for the wget analysis - that makes it look like as you say it requires to pass along the set-cookie on the redirect. Not sure if wget also has an option to show the request and response HTTP headers - it should be something like set-cookie in the first HTTP equest and cookie in the second HTTP response.
I try to put together a test script in about a week - if someone in your team can try out a request with plain ruby Net::HTTP that be great or even python if you set / manage the cookies yourself.
sanjayankur31 commentedon Aug 5, 2021
Yes, here are the outputs in debug mode which includes the headers:
With cookies:
Without cookies:
geraldb commentedon Aug 5, 2021
Thanks for the HTTP headers that makes it clear - the second request (authorize) sets the cookies and the third request than needs to pass along the cookies to succeed - so you are right it's all about cookies. I try to put together a test script next week and if it works add the cookie support to the fetcher library /gem (that will update pluto).
pacbard commentedon Aug 5, 2021
For what's worth, this script seems to work. The
response.bodyvariable returns the feed content for the springer journals that were redirecting before.geraldb commentedon Aug 5, 2021
Thanks for highlight the script on stackoverflow - I also searched / browsed some - one gotcha was:
will only work if there's only one cookie but in the springer (or more generic case) there can be 3 or multiple. I try to get to it early next week. Cheers.
Add support for redirect with cookie authentication
Add support for redirect with cookie authentication (#1)
pacbard commentedon Aug 6, 2021
tl;dr: Adding support for cookies might let pluto fetch a feed but it will probably be the wrong feed
I have bad news. I don't think that adding cookie support will completely fix the issue with Springer's feeds.
I am getting a random feed back from Springer's server after completing the request with the cookie.
Here is the trace route for the request from OP using the patched fetcher gem:
You can see that the requests succeeds with the cookie but somehow the server "eats" the search parameters that were passed to the original request. The parameters for the first request were
search-within=Journal&facet-journal-id=10827and the parameters for the final authorized request weresearch-within=Journal, which leads Pluto to parse a different feed than what was requested.I tried to trace the same request in curl and I get the same problem.
The same problem is present in the wget request from here; notice how the last request is just for
search-within=Journal, which should just be an RSS feed of links to random Springer journals instead of the article feed for the Journal of Computational Neuroscience.I am working around this issue by moving
facet-journal-id=10827as the first search parameter. That should prevent pluto from adding random papers to the feed, but the workaround might introduce other issues.Note that this is hit-or-miss depending on how the server responds. That is, if the server doesn't ask the client for authentication, the correct feed gets served. This is similar to how pluto was able to pull the feed sometimes before the cookie patch.
In fact, now I am getting this response when requesting the same feed as the one above:
The request doesn't get re-routed through the authentication server and the correct feed gets served (notice the difference in content_length). Probably the difference is that the new request is hitting a CDN cache while the original one went through Springer's servers.
Edit
After some tests, setting the three cookies from the authentication process and re-do the request to the original url, the correct feed gets returned.
If you want to test the "bugged" request, you can use an incognito session in chrome. It should get the wrong feed unless the feed it's cached on the server.
geraldb commentedon Aug 7, 2021
Thanks for your detailed report on cookies. Really appreciate your attention to detail and sharing your findings. I try to wrap up the code (in the fetcher gem) early next week. Cheers. Prost.
Add support for redirect with cookie authentication
sanjayankur31 commentedon Jan 10, 2022
Hi @geraldb , happy new year---anything we can do to help with this one (or has the code landed already)?
geraldb commentedon Jan 10, 2022
Happy new year. Prosit 2022. Thanks for the reminder. I sure try to get cookie support added / done this year in 2022. Thanks for the patience. Cheers.