HTTP user agents that don't identify fail for Zotero >= v5.0.71

odin
I understand that Zotero v5.0.71 introduces a new security mechanim that aims to block 'browsers' from accessing its API. However, when a user agent doesn’t set the `User-Agent` header, this triggers:

> Error: this.headers['user-agent'] is undefined
> Source File: chrome://zotero/content/xpcom/server.js
> Line: 434

(This is for Zotero v5.0.72.)

This, effectively, blocks HTTP requests from 'simple' user agents, most prominently Pandoc.

See:
* https://github.com/retorquere/zotero-better-bibtex/issues/1233
* https://github.com/egh/zotxt/issues/11
* https://github.com/odkr/pandoc-zotxt.lua/issues/1

From what I understand, allowing for the `User-Agent` header to *not* be set wouldn't impact the new security policy. So, would it be too much to ask to test whether that header is present and deem any user agent that doesn't set it not-a-browser?

Thanks a lot!
  • odin
    NB: The RFCs 2616 (sec. 14.43) and 7231 (sec. 5.5.3) both state that user agents "should" (not "must") set the `User-Agent` field.

    I don't want to argue over what "should" means; whether you require user agents to set this field is, of course, up to you. Still, it’s a "should" ;-).
  • dstillman
    Sorry about that — fixed in the latest Zotero beta, and the fix will be included in 5.0.73 in a day or two.
  • odin
    Thanks a lot!
  • AliNeishabouri
    Apparently the issue is back in 5.0.74. Biber and zotxt are both unable to fetch data from the local API.
  • dstillman
    @AliNeishabouri: No, there was no change to this between 5.0.73 and 5.0.74. This thread was specifically about clients that didn't pass a User-Agent at all, so unless you know for a fact that that applies to biber and zotxt, this thread doesn't apply. The primary security change — blocking certain requests made with a browser-like user-agent string unless a special header is sent — remains in effect, and those tools may need to update their code to send requests to current versions of Zotero.
  • AliNeishabouri
    edited October 3, 2019
    @dstillermann The person who created this thread did it explicitly after creating this issue (https://github.com/egh/zotxt/issues/11) about zotxt.

    As for biber, I have checked myself the following:
    - Accessing the library using the browser results in 403.
    - Accessing the library using curl works.
    - Accessing the library using biber results in 403.

    I guess I could check how biber calls remote libraries.
  • dstillman
    We would need to see a Debug ID for a request that's rejected.
  • odin
    I’ve just checked and `pandoc-zotxt.lua` appears to work fine with Zotero 5.0.74.
  • AliNeishabouri
    Sure. D108116218 is what happens when biber tries to access my library (using \addbibresource[location=remote]{http://127.0.0.1:23119/better-bibtex/library?/1/library.biblatex} in my latex file).

    Curl works fine on this url, and I get D1082902501.
  • dstillman
    edited October 3, 2019
    GET /better-bibtex/library?/1/library.biblatex HTTP/1.1
    TE: deflate,gzip;q=0.3
    Connection: TE, close
    Host: 127.0.0.1:23119
    User-Agent: Mozilla/5.0
    Right, so it's making a GET with a browser-like User-Agent, so it will be rejected from Zotero 5.0.71 on. It needs to either use a different UA string or pass Zotero-Allowed-Request: 1.
  • AliNeishabouri
    Sorry, my bad about zotxt. The issue was back in emacs (and Curl on windows not supporting --compressed). It does indeed work fine.

    I guess I'll have to open an issue with biber.

    Thanks
  • emilianoeheyns
    A temporary workaround could be a lightweight proxy which changes (or strips) the user-agent.
Sign In or Register to comment.