NEW: Odd result when importing PMIDs with Magic Wand
I've noticed this problem several times over the past 8 to 10 days. The number of records imported is 1, 2, or 3 items fewer than the number of PMIDs submitted. However, if I request debug logging, the number of imported items matches the number of items submitted.
This has happened at least 10 or 12 times. It doesn't happen if I only import 5 or 6 PMID records but consistently happens if I import a substantial number of records. A couple of days ago I "wanded" 148 PMIDs but only 143 records were imported (until I activated debug logging). The omitted PMIDs are not at the very beginning or at the end of the list.
Could this be related to the debug process slowing things down a little and allowing time for every item to be imported? Is there any harm done if I routinely activate the debug-logging each time I will import PMIDs via the magic wand tool?
Zotero beta: 5.0.90-beta.3+e9473caa1
Mac 10.15.6 (19G73)
32 GB RAM
Firefox 79.0 (64-bit)
My speedtest upload averages 11.1 Mbps Southern California to Western Europe servers and a little faster if in-state. (not slow but not blazing) I don't know if my own speed has any relevance.
Each day I conduct a very sensitive PubMed search to find records meeting my criteria during the previous 00:00:01 (midnight) to 23:59:59 CRDT interval. This produces 600 to 800 PubMed items. I go through the listing, tick the relevant records and export the results to a PMID list. Typically, I have 80 to 120 relevant items to import into Zotero each day.
This has happened at least 10 or 12 times. It doesn't happen if I only import 5 or 6 PMID records but consistently happens if I import a substantial number of records. A couple of days ago I "wanded" 148 PMIDs but only 143 records were imported (until I activated debug logging). The omitted PMIDs are not at the very beginning or at the end of the list.
Could this be related to the debug process slowing things down a little and allowing time for every item to be imported? Is there any harm done if I routinely activate the debug-logging each time I will import PMIDs via the magic wand tool?
Zotero beta: 5.0.90-beta.3+e9473caa1
Mac 10.15.6 (19G73)
32 GB RAM
Firefox 79.0 (64-bit)
My speedtest upload averages 11.1 Mbps Southern California to Western Europe servers and a little faster if in-state. (not slow but not blazing) I don't know if my own speed has any relevance.
Each day I conduct a very sensitive PubMed search to find records meeting my criteria during the previous 00:00:01 (midnight) to 23:59:59 CRDT interval. This produces 600 to 800 PubMed items. I go through the listing, tick the relevant records and export the results to a PMID list. Typically, I have 80 to 120 relevant items to import into Zotero each day.
I don't see a good reason this would happen, no. Having debug enabled during import is no problem, though I'd be rather surprised if that actually ended up being the reason rather than a co-incidence. There'd be a tiny chance that a slowdown would matter, but where it'd matter would be on the pubmed end which might rate limit you for too many requests. I thought Zotero sent the search by identifier request as one single request -- if we don't do that, I wouldn't be surprised if rate limiting happens for 150 quasi-simultaneous requests.
edit: One more thing. I am not syncing these records (syncing is off).
This afternoon I had a list of 94 PMIDs each on a new line in my text editor (BBEdit). I pasted these into the magic wand tool. The first try imported to only 90 Zotero records, I deleted these. The second attempt (after copying the 94 PMID numbers from my text editor to the magic wand tool) imported 92 records to Zotero.
I tried to make started a debug log and all 94 records imported into my working collection.
This has happened several times over the past couple of weeks. Today, after editing the Zotero records and exporting them into a MODS file I, as usual, deleted the imported PubMed records from Zotero to have an empty collection for tomorrow's work. Just to test, I again pasted the 94 PMID numbers into the magic wand and imported only 91 records (without debug operative). I started a debug log and imported the records again and ended up with all 94 PubMed records.
I can only guess that the process of rapidly importing into Zotero is somehow altered (slowed down a bit?) by running a debug log. That is only a guess because I really have no knowledge of the process Zotero uses to obtain full PubMed records from NLM by submitting PMIDs.
I have a work-around so this isn't a serious problem for _me_ but the issue is so peculiar that I feel it is worth reporting. My primary use of Zotero is 1) as a really great editor of imported records from PubMed and publisher websites and 2) as a way to export the records in MODS format so that I can import them into my online database.
Might it be helpful to provide a debug log of a successful import after a partial import? I don't see how that might be useful but I can do that.
Aside: I regularly poll the NLM API with PMIDs of journals that are Medline-indexed to obtain MeSH terms (I find that the indexers are usually finished assigning terms after 5 or 6 months -- often sooner but certainly by then). I send the requests in batches of about 200 items and haven't had an issue with rate limits. I send a similar request for ePub items to get volume, issue, and pagination metadata. I do this 18 months after the CRDT (record creation date). I run the ePub / metadata fetch script about every 6 weeks and I send at least 600 or more PMID queries. I have had no rate limiting issues.
Thanks again
edit: For many years I had a license to bulk capture NLM data. My requests were accompanied by the license number string. A few years ago I received notice that my license was no longer needed and that there would be public access to the files. I seem to recall that something was mentioned about monitoring requesting IP addresses and I've used the same 2 or 3 IP addresses from the very beginning.
edit 2:
I keep close track of the total number of PubMed records returned from my query as well as the true and false positives so that I can follow closely the precision and recall stats. This close monitoring has been especially important after PubMed not only changed the interface but also changed the term mapping. It is important to keep testing my search string so that I limit the irrelevant items but capture all relevant ones. Thus, I know how many records I expect to import into Zotero.
While it's not impossible that there's some heisenbug situation here, it may just be the result of repeated requests to the PubMed API, with a couple items timing out the first time or something like that. If that's the case, trying to log the first import might still demonstrate the problem.
The debug ID is: D1606513612
I hope that this is useful.
Thank you
32907699
32909309
32908898
32907570
32907569
32907864
32907902
32909686
32907431
32906614
32909694
32909475
32909492
32909516
32907691
32907919
32907454
32907457
32909071
32907967
32907970
32909678
32907452
32908009
32907650
32907477
32907551
32909551
32909552
32906659
32906675
32906707
32906725
32909051
32909052
32909477
32909534
In theory we could automatically slow down and retry when that happened, but because of how the translator architecture currently works it might take a lot of work to implement that. I've created an issue to track it.
Are these requests of NLM arising from my local Zotero client (my IP) or is the Zotero system making the queries and then returning the results to me ('your' IP)?
Lesson: Importing PMIDs in bulk works best with batches of 3 and can be accomplished fairly quickly -- each batch imports almost instantaneously.