Upgrade StorageLarge lit search --getting started
I am just getting started w/ Zotero, so not quite sure what functionality there is and what might be in a plug-in.
I have a large lit search (hundreds of individual .ris, .bib, and PubMed nbib files ,and maybe 300 PDFs filess that *should* have a corrsponding citatiion file). I need to get the citations and the PDFs all into the app,, and married up.
Becuae I ended up using Google Scholar for a bunch of the search, rather than PubMed or Ovid, I miassing key things like PMID, MeSH headings, and often the abstract is AWOL.
I was sort of hoping that there was a way once I got the DOI (and I think there is a plugin that can fetch the PMID) that something would prompt it to fill in the missing fields. Obviously, with bibliography this large (and getting larger) I really need the abstract and MeSH just to find things.
I am sure there is kludge I can spend all day figuring out that does a search using the PubMed Ids and/or Joural name, provided they used compatable abbreviations, and the name of the paper to repeate the entire search in Ovid and then download abstract, MeSH and metadata in a single file.
Some of the PDF and citation files (eg. which comoe from the publishers' web site) seem to have some common string in thefile name, but that isn't remotely close to consistent (and Google is super hepful and calls every exported citation "scholar.ris") and more than a few publishers label their papers "download.pdf" so really just a mess w/o processing the actual content fo the files.
If this is described someone where, I am find w/ point in the right direction. But all the other times I found that someone wanted to import citations from a directory they seemed to do it one citation at a time, which is an aful lot of work w/ hundreds of citations and hundreds of PDFs.
THis isn't going to be a one-off use case either. I am probably going to end up with a couple dozen searchers with hundreds of papers in each--just the nature of the beast here.
I have a large lit search (hundreds of individual .ris, .bib, and PubMed nbib files ,and maybe 300 PDFs filess that *should* have a corrsponding citatiion file). I need to get the citations and the PDFs all into the app,, and married up.
Becuae I ended up using Google Scholar for a bunch of the search, rather than PubMed or Ovid, I miassing key things like PMID, MeSH headings, and often the abstract is AWOL.
I was sort of hoping that there was a way once I got the DOI (and I think there is a plugin that can fetch the PMID) that something would prompt it to fill in the missing fields. Obviously, with bibliography this large (and getting larger) I really need the abstract and MeSH just to find things.
I am sure there is kludge I can spend all day figuring out that does a search using the PubMed Ids and/or Joural name, provided they used compatable abbreviations, and the name of the paper to repeate the entire search in Ovid and then download abstract, MeSH and metadata in a single file.
Some of the PDF and citation files (eg. which comoe from the publishers' web site) seem to have some common string in thefile name, but that isn't remotely close to consistent (and Google is super hepful and calls every exported citation "scholar.ris") and more than a few publishers label their papers "download.pdf" so really just a mess w/o processing the actual content fo the files.
If this is described someone where, I am find w/ point in the right direction. But all the other times I found that someone wanted to import citations from a directory they seemed to do it one citation at a time, which is an aful lot of work w/ hundreds of citations and hundreds of PDFs.
THis isn't going to be a one-off use case either. I am probably going to end up with a couple dozen searchers with hundreds of papers in each--just the nature of the beast here.
Citation updating will be possible in a future version, but no ETA. I believe there are plugins that do some version of it now.
(But as adamsmith pointed out in the other thread, this isn't really a normal usage pattern for Zotero. You generally want to save using the Zotero Connector. And even if you did import metadata files for huge searches, you would generally want to let Zotero find available PDFs for those from a connection that had direct download access, not try to pair up files you downloaded separately.)
Having the abstract, and MeSH is the requirement. If there is a publication I need that I cannot get the PDF online, it gets pasted into an Envernote/Todoist "To do" for the next trip to the med school library or begged from a friend who has an Athens/Shibboleth sign-on.
I suspect my approach is, or would be, the default for a lot of others.
Is there a way to trigger an auto-download/reload of a citation?
Where does Zotero fetch the citation. I was really impressed to see it calling Firefox to presumably download citation data--not quite sure why it doesn't use curl or get, but I am having some trouble viewing citations so i cannot comment on whether or not it pulls in the abstract and MeSH terms.
As an aside, I have recently migrated to Zotero from Endnote--where I was able to do this as well as search for merge citations as well as search MedLine from directly in the app as if all of MedLine was local.
I wonder if the same couldn't be done in Zotero and will pick up the pace of learning Python! Being able to query PubMed, Google Scholar (in spite of its limits on citations), Science Direct, Wikipedia (very helpful for a quick synopsis and some articles have very useful bibliographies), Semantic Scholar etc.Maybe w/ a one click 'add this' option?
The hard part would mapping queries to MeSH headings/sub headings. If you query "South American viral hemorrhagic fever" you should end up in the MeSH 'Arenavirus disease' location. (or whatever it is called--point is, you should be able to query with synonyms but that isn't always the easiest thing to implement. I think MeSH comes with a pretty decent set of related terms, so in its particular case, no big. But for other cases (e.g. when using Library of Congress's descriptors to find something) you might have to DIY (WordNet?). It also takes either a little magic in your database (why I am also switching to PostgreSQL and ArangoDB from MariaDB--both support lexical closure searches). You want to pull in all the articles that are identified by a subclass of whatever your search matched. In Ovid-it gives you the option to "explode" (so far, no damage done!) but think it is an important behavior. E.g. if I searched on New World Arenavirus disease I would expect to get papers about Sabiavirus, Juninvirus, Machupovirus, etc.
THanks!
You wrote:
It also takes either a little magic in your database (why I am also switching to PostgreSQL and ArangoDB from MariaDB--both support lexical closure searches). You want to pull in all the articles that are identified by a subclass of whatever your search matched. In Ovid-it gives you the option to "explode" (so far, no damage done!) but think it is an important behavior.
https://www.zotero.org/support/dev/translators
Zotero recognizes the database or the publisher website and "finds" the best way to obtain the correct and appropriate metadata from that specific source. I fear that you are over-thinking or misunderstanding how Zotero works by asking about "curl or get".
Medline uses a controlled vocabulary (MeSH) with terms assigned to its records by human indexers. Many MeSH terms have synonyms so that if one queries the database using a known "non-preferred" search word the search will find all articles that have been assigned the "preferred" MeSH term. PubMed includes article records from Medline but also contains many more articles from journals that are not subjected to the indexing process. Both Medline and PubMed also allow text-word searches. The National Library of Medicine has a complex system of "term mapping " that allows for synonyms in searches when the concept is a commonly queried topic. The text-word mapping is far from comprehensive.
Other literature databases have their own thesauri with different index terms for the same concept and different treatment of professional jargon terms. Different databases will have different hierarchical structures and index term relationships.
Services like Ovid or EBSCO have systems intended to enhance a searcher's query capacity by attempting to do term mapping across databases. I have much better results from a query of each database itself than from a unified search using EBSCO, Ovid, or ProQuest -- even when I'm not especially an expert with the database.
The thesauri for databases are far from compatible. While there is often overlap in the scope of journals each database contains, the treatment of the contents of the journal articles will differ. The same article might be indexed in Compendex (an engineering database [because it concerns ergonomics]), ERIC (an education database [because it concerns athletics]), Medline, PsycInfo, and included but not indexed in other databases such as Scopus or Web of Science. Expecting Zotero to be able to do what EBSCO, Ovid, and ProQuest try but fail to do well is a wish I share but is, alas, unrealistic at best.
https://researchguides.uvm.edu/c.php?g=945157&p=6813923
"Exploding" an index term refers to searching on that term AND all (and any) of the more narrow terms below it in the thesaurus hierarchical structure.
Exploding a term is relevant only in a specific database with its own thesaurus. Even with a tool such as Ovid, one cannot explode a search term across multiple databases. There is a good example of this in the link I provided above.
Zotero doesn't have its own controlled vocabulary--a requirement when asking to explode a query term.
As the curator of an online literature database and a user of Zotero to help with populating my database, I've read every forum post here about Zotero and databases since 2007. I also read forum posts in tech-support venues for other software. While I only have a vague idea of what you hope to accomplish, I can say with certainty that the way you collect articles and accompanying PDFs is uncommon if not unique.
If you use Zotero as intended (mentioned by adamsmith and dstillman here and in your other thread) you would be more likely to be more successful with collecting references into your Zotero database.
What is puzzling is your interest in using thesaurus-based search techniques in your own Zotero database.
Zotero has a currently powerful (and improving) search system. Please take a few moments to articulate exactly what you want to accomplish with your project and someone might be more helpful. While I am not a Zotero expert I am a career expert in information-seeking behavior and database query techniques. You have had attempts to help you from the lead developer of Zotero and one of the most experienced volunteer tech support experts. If you take time to describe your project and how you hope Zotero can help meet your objectives, someone here can tell you if Zotero can be of use or if you need other tools. I am interested in but bewildered by your posts.