Save to Zotero (PubMed) problem

FrancisLacasse

Hi,
The "Save to Zotero (PubMed)" button does not always work properly in Pubmed.
For instance, on this result page from a search in Pubmed
https://pubmed.ncbi.nlm.nih.gov/?term=health+science+library&sort=date&size=20
Only 8 of the 20 references are "Save to Zotero"...

DWL-SDCA

Are you clicking the save-multiple (folder) icon and receiving only 8?

I tested your link and only was able to import 5 using that method.

I've given up on that method because PubMed's rate-limiting is altogether too restrictive.

This works well:
Select the PubMed items wirh the PubMed tick-boxes.
Click the Send To button
Select Citation Manager
click save file
A .nbib file will be saved to your computer
Using Zotero's File > Import select the file you downloaded.
Say yes to the default options in the next two Zotero boxes and then click Done.

The records will be imported into a new collection.

This is easy and very quick. I successfully import 120-150 PubMed records using this process every day.

adamsmith

FWIW, this imports 20 items for me fine -- was this after importing from other searches, too? I can't see Pubmed doing rate limiting after 20 requests.

dstillman

It's not 20 requests, is it? It looks like we make a single request with all the ids. So what's even happening here? Is PubMed just not returning data for all results in the EFetch query? They say "several hundred records can be downloaded using one EFetch request".

DWL-SDCA

Just now, this evening, I went to the OP's linked PubMed search page. I used the Zotero folder icon. When the Zotero box appeared I clicked the select all button. Tonight I downloaded 16 items of the 20. Early this afternoon my search and fetch using the folder Zotero icon was the first time in many months I've used the folder icon to import PubMed items into Zotero. [[As I've said before my searches with PubMed return several hundred results and I use the PubMed tick-boxes to select the usable items for my purposes as even with 50/page my searches result in many pages of items. Before the new PubMed interface, I downloaded the items in PubMed XML and used my proprietary parser to capture PubMed records and massage them before importing to my non-Zotero database. With the new PubMed database I now export in nbib format and successfully import into Zotero. FWIW some time ago I had tried to save as PMIDs and import them into Zotero but was rate-limited :
https://forums.zotero.org/discussion/86806/pmid-via-magic-wand-rate-limited-something-else#latest ]]

I can try to work with you on this issue. I find it very odd that some folks have a problem that you cannot reproduce. This sort of thing fascinates me. Clearly, I have no idea how Zotero fetches records from the NLM.

DWL-SDCA

@dstillman
I tried to reproduce the problem mentioned above after I did my usual search and retrieval. My initial PubMed search results produced 980 records. I show these in a series of 50 items/pages. After exporting and importing the resulting nbib file of 147 items; I tried to see how many of the last page of items I could import using the browser toolbar Zotero folder icon. To my surprise instead of the 30 items I expected I encountered all 980 records with all of the 147 items I had ticked on the pubmed site already pre-ticked in Zotero. I tried to import these 147 items and all records appeared in the selected Zotero collection. This was the first time I tried to import items that I had selected in PubMed via the save-to-Zotero folder icon. My problem above was encountered when I had a page of fewer than 50 items none of which I had ticked on the PubMed site. By the way, I tried to reproduce the problem I encountered 3 days ago (importing unticked PubMed records by ticking Select All in Zotero) and all records were imported. Perhaps this was an intermittent PubMed site problem.

This is __wonderful__. I didn't realize this was possible from reading the Zotero documentation.

FrancisLacasse

Hi,
Today I retried The "Save to Zotero (PubMed)" using the result page from the search in Pubmed:
https://pubmed.ncbi.nlm.nih.gov/?term=health+science+library&sort=date&size=20
This time, 13 of the 20 references were saved to Zotero.

If I understand, the more accurate way to work with Zotero in Pubmed is to 1) select the references, 2) send them in a NCBI file that will be next imported manually in Zotero (step 3).

But as a librarian teaching Zotero to my patrons, I find this 3 steps method less user friendly. And, to make matter worse, it means that I have to explain to my users that in Pubmed, they have to import one way, but in Ovid or Ebsco, they have to use another way,

Am I missing something ?

adamsmith

Ah, this is finally an example I can replicate: The top 7 items don't return anything from the API. They're all entered in the last 24hs. If you look in the nbib, you'll see at the bottom
EDAT- 2021/01/30 06:00
...
CRDT- 2021/01/29 05:56

CRDT is the creation date, i.e. this AM,
EDAT is the Entrez date, the date at which they will be in the API, i.e. tomorrow.


I think we can get the nbib data directly with some effort, but it also seems so... unnecessary. I'll see if I can find someone from NCBI

DWL-SDCA

Know that the problem goes a little deeper than that. I do a daily search for items added to the database in the last day. I use the CRDT "2021/01/29` 00:00:01"[CRDT] : "2021/01/29 23:59:59"[CRDT] (for example, knowing that I might miss records added during the 2 seconds not covered). However I found that if I run the query at 21:05 Pacific time I can miss records. Working with NCBI I learned that it can take 90 minutes (or more) for records to appear after the CRDT is established. I was told to wait "a couple of hours" before I run the search if I want to be sure of getting everything. I now wait until the next day to run my search. [edit and I've returned to using the nbib file import]

The EDAT and API appearance can require a day (or two) if there is an issue with the once-a-day automatic conversion process.

DWL-SDCA

@adamsmith

I received an updated reply from NCBI concerning when PubMed is updated with the most recent EDAT interval an how long after a record receives an EDAT before the record becomes availlable to searcheers:

Thank you for writing to the PubMed Help Desk.

PubMed is now updated 4 times each day, typically at at 8 am, 2 pm, 8 pm, and after midnight ET. The process may take 0.5 to 2 hours if everything runs smoothly.

When paging through the results it may rerun the search leading to updated results.

You may want to change the time you run your update query to somewhere between the updates times to avoid the problem you are experiencing.

In my experience with the midnight US Eastern Time update, when I make a query at 1 AM and there are many records returned, as I page through the results the number of results will increase. This causes me to see already-ticked records when I move to a new page (because additional records were added at an "earlier" point in my results as time goes forward). I now know that I must wait at least two hours after an EDAT range restriction before I can be sure that all records in that range are available. -- A longer wait is necessary if the range ending EDAT time is not "8 am, 2 pm, 8 pm, and after midnight ET".