consensus.app looking to make results available for Zotero

christiansalem

Hello, I am one of the cofounders of Consensus (consensus.app), a new search engine that uses LLMs to find answers in scientific research.

We have many users requesting that we update our site metadata for better saving in Zotero, and we are happy to do this!

Based on my read of the Zotero documentation, here is what we need to do:
- Add Highwire press meta tags to our paper pages
- Add DOI links to paper pages
- Create a Zotero translator

I have 2 questions for the community:
1. Is there anything else we should do to make sure papers can be easily found by Zotero?
2. How do we allow the search results page to also easily be scraped? For example, I noticed when using Zotero with Semantic Scholar, you can do a search and import all results on the page into Zotero. Many users have asked for this capability in Consensus too.

Anything info at all helps! Excited to join this community to help make research easier for everyone :)

dstillman

Based on my read of the Zotero documentation, here is what we need to do:
- Add Highwire press meta tags to our paper pages
- Add DOI links to paper pages
- Create a Zotero translator

To be clear, those are mutually exclusive (or not, if you want to do them all anyway, but for Zotero specifically you'd only need to do one of them, at least for item detail pages).

How do we allow the search results page to also easily be scraped?

Search result pages are a bit tougher without a dedicated translator, given available metadata options.

If every search result has a DOI, you can just include those in the page (even in a hidden element) and Zotero will pick those up and show a selection dialog via the DOI translator.

If you have RIS or BibTeX available, you could use unAPI. As noted on Exposing Metadata, that project is dead, but it still works in Zotero, and a future version of Zotero might offer the ability to just include a <link> to RIS or BibTeX, which would allow for a trivial change from an unAPI implementation.

Or you can contribute a custom translator for your site, which would give you the best results and let you guarantee that people would always get the beta metadata on all pages. That's what Semantic Scholar has. A translator can also call other translators, so if you make metadata available in other formats already (including embedded metadata) the translator can use those on item detail pages.

christiansalem

Thank you!

If we were to make the DOI available on search results and item detail pages, what is the best way to do this to ensure that Zotero can easily pick it up?

dstillman

Zotero will detect them automatically. Only exception would be if you're updating the page via JS after the page loads, in which case you'd have to force Zotero to run detection again:

https://www.zotero.org/support/dev/exposing_metadata#force_zotero_to_refresh_metadata

christiansalem

Awesome! When you say, "Zotero will detect them automatically", do we need to label the element in the HTML something like "citation_doi"?

dstillman

Nope.

dstillman

(The benefit of non-DOI methods is that if there are only DOIs Zotero will show a folder icon that opens a selection dialog, since it can't guarantee that every DOI on the page is for the current page. With embedded metadata or a dedicated translator, Zotero will know it's for the current page and offer to save with a single click.)

christiansalem

Super helpful!

If we have embedded metadata, and one element of that metadata is DOI, will Zotero be able to use the DOI to fill in any pieces of metadata we are missing? Or will it just take our metadata we have at face value?

christiansalem

And one more question: I noticed that when you save a paper from Semantic Scholar, it saves the TLDR as a note in Zotero. How can we configure ours to save a specific field as a note?

adamsmith

among the automated methods, only unAPI via a metadata format supports that. Semantic Scholar has a custom translator (which, I'm pretty sure, they submitted the current version of).

christiansalem

Ok we implemented Highwire Press Meta Tags on our paper pages: https://consensus.app/details/creatine-supplementation-clearly-means-increase-muscle-tipton/f33ade769118584a8e79dbb5af7ebbb8/

When I save to Zotero, I notice some metadata is missing. Is there a way to make Zotero recognize the DOI and fill in the fields we don't have on our site?

Does anything else look problematic or misconfigured?

adamsmith

When I save to Zotero, I notice some metadata is missing. Is there a way to make Zotero recognize the DOI and fill in the fields we don't have on our site?

Not currently, no, though I think that may be planned. The missing data is just missing, though -- it's also not in the cites provided by your site, and you just don't have a citation_volume, citation_issue, and citation_firstpage/lastpage metatag.

christiansalem

Thank you! I will add those.

We also added the DOI element to the search results page. Here is the page: https://consensus.app/results/?q=does creatine help build muscle?&amp;synthesize=on

Here is where the DOI is on the page: https://www.awesomescreenshot.com/image/39901223?key=b0e93e2cb9c23b856e8a4b43f905edb5

However, Zotero is not picking up the papers on the search results page. Anybody know how we can fix this?

adamsmith

See that part

Zotero will detect them automatically. Only exception would be if you're updating the page via JS after the page loads, in which case you'd have to force Zotero to run detection again:

https://www.zotero.org/support/dev/exposing_metadata#force_zotero_to_refresh_metadata

The DOM element with the DOI is generated via js aver page load and so Zotero's regex runs before the DOI is on the page. If you implement the force refresh as linked after all DOM elements are loaded, that should work

christiansalem

Hey guys, we have implemented the above force Zotero to refresh metadata function: https://consensus.app/results/?q=does creatine help build muscle?&amp;synthesize=on

Yet, Zotero is still not picking up DOIs from search results page. Any other ideas what might be wrong?

christiansalem

Update: You can pick up DOI from the results page by right-clicking, going to the extension, and clicking "Save to Zotero (DOI)": https://consensus.app/results/?q=does creatine help build muscle?&amp;synthesize=on

However, when you save via clicking the Zotero extension in the Chrome bar, the default behavior is to just save a web page, rather than pick up all the DOIs.

Does anyone know how to have the Zotero Connector default to looking for the DOIs on the results page?

adamsmith

Does anyone know how to have the Zotero Connector default to looking for the DOIs on the results page?

Sorry, not possible -- the priority in which Zotero selects between generic translators is hardcoded into the translators.
I think the behavior in this type of situation will get finetuned in the future, but for now, this is the best you'll get without a dedicated translator, sorry.

christiansalem

Thanks Adam! But we could implement this priority in a dedicated translator?

adamsmith

absolutely, yes, wouldn't be hard

christiansalem

Can you point me to where in the documentation we can set this?

adamsmith

I mean -- once you write your custom translator, you explicitly tell it what to detect and what to import. Look at a couple of examples, SemanticScholar in particular is relevant https://github.com/zotero/translators/blob/master/Semantic Scholar.js