PubMed import partially broken

Since today or so the import from PubMed has problems: Zotero misses the URLs of references from PubMed and the date, when the reference was added (Firefox 15.0.1, Win XP, Zotero 3.0.8). Can anybody confirm?
Thanks, Michael
«13
  • I can confirm that has changed, but it was done on purpose:
    Since PubMed is a catalog of articles and doesn't actually store any articles, the URL and accessed date shouldn't be part of the citation. The URL is still attached to the item as a link, though.
  • Sorry, but this change has been a very bad idea. PubMed often links to the full articles on other websites and for these websites the import filters of Zotero may work or may not work, usually not. So it`s easier and much more consistent to import the reference from Pubmed and attach the file of the full text to it. And in my scientific communications with colleagues I want to have the URLs attached, so that they easily can jump to the abstract in PubMed, from there to full text. There has already been a switch in Zotero to stop the URL being a part of the output, that should still suffice. And of course I want to know when I first met an article in PubMed, that`s part of my research history. Who has had the idea for this change? Has he/she ever done medical research or prepared a medical presentation? "The URL is still attached to the item as a link, though." I cannot confirm this, the URL-field is empty now after an import from PubMed.
    From a software usability point of view I think it is also almost always a bad idea to take away functionality which has been years in place and many users are accustomed to.
    I hope this change is reversed, Zotero otherwise will have lost much of its value for me.
  • edited October 5, 2012
    First, I'd ask you to remember that much of the work on Zotero - including almost everything translator related - is done by unpaid volunteers. So I suggest keeping your posts polite and non-hyperbolic.

    The person who made the change is, in fact, a published biomedical researcher, I assume he uses PubMed regularly.

    To clear up some things:
    And of course I want to know when I first met an article in PubMed, that`s part of my research history.
    that's why Zotero has a "Date Added" field, which is and should be distinct from a date accessed field.
    "The URL is still attached to the item as a link, though." I cannot confirm this, the URL-field is empty now after an import from PubMed.
    as I say - it's "attached as a link" - not in the URL field but as an attachment that says "PubMed Link".
    If you export a PubMed imported item - say to RIS or to Zotero RDF to pass it on to a colleague the link will be included. You yourself can get to the pubmed site directly by double-clicking on the attachment.

    Given that, I still don't really see what you're losing.
  • "as I say - it's "attached as a link" - not in the URL field but as an attachment that says "PubMed Link"." That means, that on export there will be more information than I see in the "Infos" field (german UI)? That doesn`t make sense to me. The URL of a reference from PubMed is important and should be always easily accessible.

    So you confirm that one user had enough influence to change the perhaps most used import filter of Zotero for hundreds or thousands of users? Sorry, but that`s bad change management. I can understand the original argument of your user if he only looks at reference lists as an output for papers.

    But I use reference lists from Zotero much more often for quick email communications with colleagues (individually or hundreds of them on dental mailing lists), and then I need more information as I get now. That means: Choosing some references, push the right mouse button, choose citation style, copy the reference list to the clipboard, from there to email and I`m done.

    I still can`t see why the URL should not remain part of the reference as it did till yesterday.

    For me "date added" doesn`t make much difference from "date accessed". But I understand the difference. Is there a still a citation style which gives me URL and "date added" or "date accessed" for references imported from PubMed from today on?

    "Given that, I still don't really see what you're losing." You don`t have to see that. You might simply accept my view, because you would loose nothing doing so.

    Is there a way for to get the old import filter back? Is somewhere a repository of old import filters? It seems it has been a mistake to trust automatic updates by Zotero. I will switch off these automatisms if possible and go back to my automatic backup of my Firefox profile from last night.
  • There isn't a way for citation styles to access the attached link, whereas they could access the contents of the URL field. This is correct citation practice and ultimately the main reason the link had to move to a link attachment-- it would be incorrect for citations to articles retrieved from PubMub to include the PubMed URL, as they are not published there, but merely indexed. The switch, which was signed off on by the development lead (and neuroscientist) who handles site translation, is to provide correct data and citation for the larger user base in accordance with data curation standards.

    If your writing requires that those URLs be included in the citations, then we can help you make changes to the translator to populate the URL field-- I don't recommend you turn off translator updates, since many sites change frequently and you will no longer seamlessly maintain access to the thousands of sites parsed by the translator collection.
  • (for quick access, note also the option of using the "National Library of Medicine (NLM) - Grant with PMID" which will include PMIDs for articles retrieved from PubMed. Another alternative would be using a citation style like APA that includes DOIs, which also allow for quick access)
  • Okay, I went back to my old Mozilla/Zotero profile of last night (5 GB, 160,000 files, 3,750 directories) but it didn`t help. I would have to go back to an older backup (when where these awful changes introduced?), but not now. I`m in the middle of preparing a 90 minutes presentation for a dental faculty ("RCTs, evidence, guidelines"). And basically my most used tool for that (besides Scrivener), Zotero, has been broken by people not knowing library science or EBM/EBD very well, IMHO:

    > Post by: *ajlyon*
    > There isn't a way for citation styles to access the
    > attached link, whereas they could access the contents of the URL
    > field. This is correct citation practice

    No, it depends. If you are doing EBM/EBD or statistics about medical literature on different topics, you most often will never touch the original article. You will wade through hundreds of references and their abstracts, delivered by PubMed, and PubMed will be the only source. PubMed sometimes doesn`t deliver correct references, especially foreign author names may be wrong, pages missing, so PubMed is a source of it`s own and should be handled as that by Zotero. Based on this original source (PubMed) a preselection of references will be done, which will be documented in EBM/EBD (URLs remain handy for this), and only a small percentage of original papers will be ordered.

    The same is true of course of amazon.com (or any other library catalog). If I get a reference from Amazon, I want to have the URL from Amazon with my reference. Amazon frequently makes mistakes (shortened titles, wrong author names).

    Your mental frame, how you see library catalogs, is wrong. If you only use a library catalog or anothers author`s work for citing the reference, that has to be shown in the reference. If you get the original article, you may correct the import from PubMed, remove the URL or replace it by a better, direct URL, if you have one.

    But you don`t have to remove the URL manually in Zotero for a usual reference list, if you don`t want, because there is already an option for this: If you go to Zotero/Preferences/Styles you find "Include URLs of paper articles. When this option is disabled, Zotero includes URLs when citing journal, magazine, and newspaper articles only if the article does not have a page range specified".
    If you have a paper article, you will have a page range, problem solved. If no page range, URL will show. By not importing URLs from library catalogs you have destroyed the reminder function of URLs on reference list ("get a page range!"). You have taken away much value.

    > and ultimately the main reason the link had to move to a link
    > attachment-- it would be incorrect for citations to articles
    > retrieved from PubMub to include the PubMed URL, as they are not
    > published there, but merely indexed.

    Everybody knows that original articles are not in PubMed directly accessible, where is the beef? And as I wrote above, there is no such thing as "merely indexing". It`s a librarian`s work, who adds value (MESHes, translating of foreign titles), and people and systems make errors. Zotero already ignores an important part of this original, valuable work, done by librarians, by no longer importing MESHes form PubMed into tags.

    > The switch, which was signed off on by the development lead (and
    > neuroscientist) who handles site translation,

    You should better have asked modern librarians for confirmation, e. g.:
    Rethlefsen M, Rothman D, Mojon D. Internet Cool Tools for Physicians. 1st ed. Springer; 2008.
    (That I got this reference from Amazon.com is missing now, and that`s bad.)

    > is to provide correct data and citation for the larger user
    > base in accordance with data curation standards.

    I don`t buy that. I`m using Medline/PubMed for 25 years and scientific libraries for 30 years. I think I know what I`m talking about.

    > If your writing requires that those URLs be included in the
    > citations, then we can help you make changes to the translator to
    > populate the URL field

    That would be a nice, first step. I still would need a citation style to get this URLs out automatically.

    > adamsmith 2 hours ago
    > (for quick access, note also the option of using the "National
    > Library of Medicine (NLM) - Grant with PMID" which will include PMIDs
    > for articles retrieved from PubMed. Another alternative would be
    > using a citation style like APA that includes DOIs, which also allow
    > for quick access)

    PMID or DOIs are not part of an URL in these styles. That means extra work for readers of electronic output (ebooks, presentations etc.). My mentioned presentation will be offered for download as a PDF. I want that the readers can open my PDF, read it, click the URL in the reference and arrive in my source. My whole workflow from PubMed to my PDF worked perfectly automatically this way and now it is broken for no good reason at all.
  • the translator repository is here:
    https://github.com/zotero/translators
    if you want any more detailed help I at least will either need a hefty consulting fee or an even heftier change in attitude.
  • > adamsmith
    > if you want any more detailed help I at least will either need a
    > hefty consulting fee or an even heftier change in attitude.

    You want a fee for bringing back a function which was taken away after being in Zotero for years, dragging users into this software? That`s a nice business model. I would never have started using Zotero without the functions I`m mssing now. Eliminating the import of Meshes already slowed me down sometimes, but now Zotero has become much less useful for serious work with PubMed, at least for me.

    Whether Zotero is free or not doesn`t matter (at least not for me, I chose Zotero because of features, not of price). Zotero ist part of the service industry (like me), anyhow. I have written already, how software developers should act. And the companies I`m dealing with (and paying for) are acting this way.

    I wanted to recommend Zotero in my presentation. I cannot do this without reservations which I had not a few days ago.

    But thanks for the hint to \zotero\translators (no repository with old binaries? weak). I went back to my backup of 2012-09-29, grabbed "NCBI PubMed.js" (inside it reads: "lastUpdated": "2012-03-12 01:14:39") and replaced the broken one in my active Zotero (anybody interested can get it here: http://www.logies.de/NCBI%20PubMed.js or http://www.logies.de/NCBI PubMed.js). It took about 3 hours for me (us) to solve this problem, sorry, that`s more than I can stand. Now at least PubMed is working again as it should, other library catalogs like Amazon are still broken.

    It seems, that the integration of Zotero with Scrivener does not become better. After finishing this project I will test Endnote and perhaps Bookends. Thanks for the years I could use Zotero, but it seems that time has come to move on:
    1. Anon. Literature and Latte • View topic - Zotero integration. Available at: http://www.literatureandlatte.com/forum/viewtopic.php?f=4&t=18847&p=133722&hilit=bibliographic+software#p133722. Accessed October 6, 2012.
    2. Anon. Literature and Latte • View topic - Best bibliography tool to use with Scrivener? Available at: http://www.literatureandlatte.com/forum/viewtopic.php?f=2&t=18942&hilit=bibliography+software. Accessed October 6, 2012.
  • Sorry for not chiming in earlier. I'm the one who actually made the change to the PubMed translator, but the idea that URLs to items in catalogs should be stored as attachments instead of the URL field has been around at least since 2006 (http://forums.zotero.org/discussion/105/scraper-add-url-from-which-page-was-scraped/) though I cannot find the original discussion at the moment. The PubMed translator has been incorrect this whole time unfortunately. (ajlyon's post above summarizes it quite well)

    I'm also not part of the core Zotero development team, so what I'm about to say is my personal opinion and not that of Zotero developers.

    Zotero, first and foremost, is a bibliographic management software to help you (A) organize your research and (B) produce correct citations. Everything else that you can do with Zotero is a huge bonus. Having said that, I think everyone here is excited to see Zotero being used for more than its intended purpose. In your case this means generating citations that are, while useful, not technically correct.

    I think typically the developers here are quite accepting of new ideas and suggestions for improvements to help make Zotero more useful. Not all of these requests can be accommodated for one reason or another though (e.g. incompatibility with other features or lack of developers working on Zotero). It is unfortunate that your "my way or the highway" tone and language has turned this potentially fruitful discussion into a somewhat sour situation. Nevertheless, I do use PubMed myself and I see how including a catalog link in the citation for personal communications could be useful, so I would like to turn this discussion around and try to come up with some sort of solution.

    Here are some ways that I can see this being resolved:

    1. Additional, possibly hidden, option to substitute catalog links for URLs when the URL is missing (either upon import or when generating citations. I prefer the latter). [technical note: In order for this to work correctly, I think we would also want to properly distinguish catalog links from other links (e.g. file links) through an additional attachmentType property (this could replace snapshot:true with attachmentType:snapshot).]

    2. Same as above but implemented as a Zotero addon. I can see this option having much fewer objections, but I believe some changes would still need to be made to the Zotero client.

    3. If someone really wanted this right away, I think this can be accomplished via a custom export translator. Of course you wouldn't be able to flip through different citation styles without hard-coding them, so it's not a solution for the general public.

    4. Not exactly a complete solution (since sometimes links are not available), and I'm not sure this would even be a correct way to handle it, but when (if?) we do turn PubMed (and other catalog translators like Google Scholar) into a recursive translator (i.e. have it navigate to the linked site and use that sites translator to retrieve additional metadata/attachments), we could store the URL of the site hosting the resource. This doesn't even have to wait for recursive translators. We could find the direct URLs right now. My concern is whether it would be correct to say that the article was retrieved from the linked website when the user never visited it. I feel like the answer is no, but I'm interested in other opinions.
  • Indeed, the arrogance of the OP should not prevent us to imagine a solution.
    It's not orthodox but what about adding a dedicated field for these URLs? Under "Library Catalog" and "Call Number".
  • I actually thought that you noticed this issue thinking about 4.)
    I think that would be my preferred solution, though I understand the concern about not actually visiting the site.
    It's not orthodox but what about adding a dedicated field for these URLs? Under "Library Catalog" and "Call Number".
    too hacky for my taste. Also, there would still need to be a CSL variable to map that too and I doubt that's going to happen.
  • that said, though - if the main purpose is to just have an option to include a link to the original article, that can just be done with the DOI, which is stored with pretty much all PubMed articles by adding
    http://dx.doi.org/ + DOI to the end of the citation style - that would likely be more stable than the URL link to the original resource.
  • Talking about arrogance... When I updated my profile on this website some days ago, I could read, that real names should be the norm. It seems, that nobody involved in the development process is using a real name.
    @aurimas: Thanks for trying to find a solution. I think if a software problem needs 6 years or more to be tackled, it is no problem worth being tackled because all users annoyed by this problem will already have left the software. Then a change can only annoy more users, here it has been me.
    "Zotero, first and foremost, is a bibliographic management software" No, I can`t agree. For years it had been advertised as a tool to store all kind of information. And usually it is much better to have some information at all instead of having it not because it doesn`t fit into a predefined "bibliographic style". Somehow I feel that this is a cultural problem. Here in germany, at least in the medical field, I can`t imagine that bibliographic style is taken too seriously, and of course it is not a part of the medical curriculum (as it is in the humanities). In medicine, the reference should allow to identify the used source and, at first look, should look more or less, as the journal wants, and if a URL is added as a bonus, that`s fine, regardless whether there is a style for that or not.
    I have frozen my Zotero now and will see how long it will work till some important catalog change will break it. PubMed seems to be quite stable, though. I think, that they know their customers well... Any busy clinician will hate any change in the software he/she uses if it is not absolutely neccessary or brings a definite benefit. We heal patients, not reference notes. The change to "NCBI PubMed.js" was superfluous and damaging from this point of view.
    So nice that you try to solve the problem. If this should happen, I hope that I notice it somehow and may try.
  • It's not orthodox but what about adding a dedicated field for these URLs? Under "Library Catalog" and "Call Number".
    I suppose you mean that we should use the additional field along with an option that allows users to either include this field in the citation or not. This could work. On the other hand, I think we may still have to support the way current links are stored (i.e. the attached links) so that users don't have to manipulate old entries. In that case, we might as well just stick with the attachments.
    if the main purpose is to just have an option to include a link to the original article, that can just be done with the DOI, which is stored with pretty much all PubMed articles by adding
    http://dx.doi.org/ + DOI to the end of the citation style - that would likely be more stable than the URL link to the original resource.
    I think this is a great idea. I'm not sure why it never crossed my mind. It would still have the same concern as (4). I'm not sure how errata are handled in terms of DOIs. Accoring to doi.org, it's not frowned upon to redirect the DOI to the corrected material. Looking at the Journal of Biological Chemistry, for example, it seems that they do assign a new DOI to the corrections (e.g. http://www.jbc.org/content/285/11/8054 => http://www.jbc.org/content/287/41/34494) If most journals follow this rule, then I think this solution would be the best approach for now.
    I think if a software problem needs 6 years or more to be tackled, it is no problem worth being tackled because all users annoyed by this problem will already have left the software.
    The thread I link to above and several other questions regarding this "feature" did not raise concerns about including the URLs in citations. I think the issues were mostly resolved with the understanding that the links are stored as attachments. AFAIK, this is the first time this particular issue has come up.
    "Zotero, first and foremost, is a bibliographic management software" No, I can`t agree. For years it had been advertised as a tool to store all kind of information. And usually it is much better to have some information at all instead of having it not because it doesn`t fit into a predefined "bibliographic style". Somehow I feel that this is a cultural problem. Here in germany, at least in the medical field, I can`t imagine that bibliographic style is taken too seriously, and of course it is not a part of the medical curriculum (as it is in the humanities). In medicine, the reference should allow to identify the used source and, at first look, should look more or less, as the journal wants, and if a URL is added as a bonus, that`s fine, regardless whether there is a style for that or not.
    While you may disagree that correctness of citations is not that important, there is clearly a good reason why over 1600 styles are available for different journals (including german ones) and more than half the posts on this forum are about tweaking a style one way or another (down to including or not including periods at the end of citations). You might not care about it, but plenty of editors and academics do pick on the tinyest things. The best Zotero can do is follow the guidelines.
  • over 1600 styles
    over 2800. The citationstyle people - me included - take some pride in that number ;-)
  • Including an URL pointing to DOI and from there to the publisher would be no solution for me. If I have got the reference from PubMed and want colleagues to point to it, I want to send them to an interface they know, which would be PubMed. PubMed offers Meshes and this great feature "related articles". Both are not offered by publishers. If the colleague is forwarded to the publisher and wants to import it to his own bibliographic software, this would rise the risk of inconsistencies in references because of so many publishers. I tried one (!) example and the result is horrible:
    1. Maserejian NN, Trachtenberg FL, Hauser R, u. a. Dental composite restorations and neuropsychological development in children: Treatment level analysis from a randomized clinical trial. NeuroToxicology. (0). Available at: http://www.sciencedirect.com/science/article/pii/S0161813X12001970. Zugegriffen Oktober 6, 2012.
    2. Maserejian NN, Trachtenberg FL, Hauser R, u. a. Dental composite restorations and neuropsychological development in children: Treatment level analysis from a randomized clinical trial. Neurotoxicology. 2012. Available at: http://www.ncbi.nlm.nih.gov/pubmed/22906860. Zugegriffen Oktober 6, 2012.

    Both should be the same, but they are not. ISSN and Date are different, the name of the journal is written different. That`s a bad joke.
  • edited October 6, 2012
    At most, half of the new PubMed records have DOIs. All have PMIDs and URLs can be formed using them (as immediately above).

    The PubMed record will have the original add date and a last-revised date. For the vast majority of early PubMed records,much of the information is incomplete or not yet in standard PubMed format. I'm not yet writing about the records with missing MeSH terms and those listed as ePub. -- author names, article titles, and other things will change as edits move the article to maturity. The number and nature of MeSH terms will also change from none, to several, to more, or to fewer. In other words, the typical PubMed record undergoes several changes throughout the first 4 to 6 months after being added. Much later, when ePub articles are updated with volume, issue, and page range info, there are often other changes made.
  • Thanks DWL, that's helpful as always.
    That suggest to me that the best medium-run way to make this work is the PMID - unfortunately it's currently not being saved in a way that can be turned into a URL (it has PMID: in front of it in the extra field) - I think with the current Zotero fields that's the only reasonable way to do this, but once we (finally) get a PMID field in Zotero, a citation style that does with the PMID what I suggest for the DOI above would be easy enough.
  • It`s nice that I can agree on URLs generated from PMIDs. If you should fix import from PubMed, please again import MESHes. If you want to do professional research in PubMed, you work with Meshes, collecting and arranging them to complex search strategies (ending up e. g. in RSS-Feeds from PubMed based on these search strategies). Earlier I could use Zotero for skimming through MESHes, now I need PubMed again for this, which is not as fast. For other catalogs import from publishers to tags seems still to be available, so I don`t know why this was eliminated for PubMed.
  • edited October 7, 2012
    MESHes should still be imported where they're present, as long as you have the "automatically tag items" option checked.
    As DWL notes above, they're not present for all items in PubMed, especially newer ones, but where they are they're imported (I don't know if PubMed updates these at different times - seems unlikely, but just in case, what matters is the XML, not the display on the page.)
  • @adamsmith: Thanks! In recent months I was alerted of new references usually via RSS-feeds (only some of them then imported to Zotero), and they are obviously too young to have Meshes included. On older references import of Meshes to tags still works as it should. I think it was a good idea to switch off automatic updates of import filters. Recognizing the reason for a problem becomes more difficult when things might have (or not have) happened in the background.
  • A long discussion. I really need to save the Pubmed URL in the Zotero entry. Do I have to do this by hand or will it be readded?

    When was this change introduced, i.e. when was the Pubmed URL removed? (I have to go through old entries and fix them.)
  • As noted above, the URL is saved as a link attachment. Since this is not a URL for the actual article, it is not supposed to be included in citations.

    If you have another use case for this, please let us know.
  • @aurimas: I use to save the article from Pubmed for four reasons:

    1) The tags they add there. It is more uniform.

    2) The abstract is often more readable there. You can see it from a mobile too.

    3) You have a link to the original article. And if the whole article is available for free in Pubmed Central there is a link to it in Pubmed, but I have never seen a link from the original article to Pubmed Central. (Maybe I just have not looked for it in all cases... ;-)

    4) In some cases not even an abstract is available in the original place without a login.

    So I really want the link to Pubmed. Everyone involved in reading the references should know what it means, I guess.
  • I think I know what the problem is now. Currently Zotero will not attach links to items if you have disabled PDF attachments in Zotero preferences.

    This will be fixed in the next Zotero version that is due by the end of the week. If you want to have a sneak peek you can install the 3.0.9 "beta" from http://www.zotero.org/support/dev_builds

    Or you can enable PDF attachments in Preferences -> General -> Automatically attach associated PDFs and other files when saving items"
  • @aurimas: Will that add the link to Pubmed?? ;-)
  • Unfortunately that did not work for me now in the standalone Zotero (with Google Chrome).
  • Which suggestion did you follow?

    Are you sure you are looking in the right place? http://imgur.com/qLNLW
Sign In or Register to comment.