Upgrade StorageRePEc translator
The RePEc translator has some problems.
Firstly, looking at the 271.js translator file it appears to be specific to http://ideas.repec.org and excludes http://econpapers.repec.org (the other main RePEc service which happens to be run by me).
Secondly, it appears to be using a very round about way of getting the bibliographic data by requesting it in RIS-format when it is already available in Google meta tags in the original html page.
Both EconPapers and IDEAS use the Google meta tags so creating a translator that works for both sites should not be to difficult. Ideally one that works better than the default embedded metadata translator which fails to pick up the abstract.
Firstly, looking at the 271.js translator file it appears to be specific to http://ideas.repec.org and excludes http://econpapers.repec.org (the other main RePEc service which happens to be run by me).
Secondly, it appears to be using a very round about way of getting the bibliographic data by requesting it in RIS-format when it is already available in Google meta tags in the original html page.
Both EconPapers and IDEAS use the Google meta tags so creating a translator that works for both sites should not be to difficult. Ideally one that works better than the default embedded metadata translator which fails to pick up the abstract.
Thanks for the quick response.
think along the lines of if (url.match(/econpapers\.repec/))
It works for search results as well as is possible (they're a bit weird on econpapers). It also tries to capture working papers as reports and published journal articles as journal articles - as far as I have tested correctly.
edit: doesn't look like this is able to get pdf attachments in most (any?) of the relevant cases.
Search results are a bit complicated on EconPapers (it is possible to get something which is easer to parse by switching to the non-frames view). Here is how it can be figured out (I think) with javascript: 1) find the <span> with class=activetab and note its id (tab1, tab2, etc). 2) find the <div> of class list with the corresponding id (list1, list2, etc), this will hold the currently visible search results. 3) There is one child <p> for each result that should be fairly easy to parse.
Full text links are not in the meta tags. This must be parsed from the html and is different on IDEAS and EconPapers. A form on IDEAS and a <href> on EconPapers. Note that these do a roundtrip to our servers for easy recording of the click and are then redirected to the actual full text somewhere else.
For working papers, Zotero is looking for the citation_publisher metatag - do you have an example where that's not working?
are search results not working satisfactorily from EconPapers? I tried to get that right.
It's currently looking at '//div[@class="abstractframe"]//li/a' in the results frame.
I don't think auto-downloading of pdfs will work bc of the redirect.
The '//div[@class="abstractframe"]//li/a' does not seem to work, I don't think there is a li to be found and I just get a dialog box with one check box and 'false' next to it. In any case this would one get one search hit at a time.
I think it would be better to use something like
detect : FW.Url().match(/scripts\/search/\search.asp),
choices : {
titles : FW.Xpath('//div[@class="activelist"]//a').text().trim(),
urls : FW.Xpath('//div[@class="activelist"]//a').key("href")
}
although I've probably got the syntax wrong.
I'll look at the searches again - thanks!
For me search results work just fine - the string you suggested gives me the authors, not the articles.
I've pushed an updated version that should get reports for econpapers, will generally get journal titles, and will put the series into report type.
1) keywords/tags are neither imported from IDEAS (example) nor Econpapers (example). citation_keywords is not empty on both sites.
2) Report Number is not imported on IDEAS. It is not in the meta tags, so that may make it a bit tricky. It could be scraped from the body text, e.g. from the Handle (the last item of the handle is the paper number (which must be unique) - see the tutorial or the ReDIF documentation). I would prefer to use only the number, not the preceding letters. Of course we could ask Christian Zimmermann (maintainer of IDEAS) to add the report number to the meta tags.
3) It would be better to scrape the file URL as the main URL for the Zotero record IMHO, because it is quite uncommon to use links to RePEc in citations or references. I.e. submit e.g. http://ftp.iza.org/dp6212.pdf to the URL field instead of http://EconPapers.repec.org/RePEc:iza:izadps:dp6212. Since Sune and Christian would probably oppose (right?), why not add the IDEAS link as an additional link to the Zotero record? I've seen that with other translators.
Right now, the file URL would have to be scraped from the body text. One could also use the citation_pdf_url or DC.identifier tags if they were provided. (These are also suggested by Google, but not of any use for IDEAS/Econpapers in combination with Google Scholar, since the URL "must refer to a file in the same subdirectory as the HTML abstract")
4) The Series Title field would be a better fit for citation_journal_title (IDEAS) or series (Econpapers) IMHO (though that may cause problems with e.g. Bibtex export later on?). Report type should just be working paper, discussion paper, or report, then.
Finally, @SuneK: I know particles are a pain, but it's 'van Ours, Jan' not 'Ours, Jan van' (at least in Dutch, and I think he even prefers his middle initial, so 'van Ours, Jan C'). I know from a conversation with Christian that this is mainly the responsibility of the publisher, but IDEAS got it right this time.
Nevertheless, thanks again for the update!
BTW: I am using Zotero 2.1.10, Firefox 9.0.1 and RePEc.js is "lastUpdated": "2012-01-08 21:25:00".
2. Yeah, not going to do that. If you can get Christian to implement it in the metadata I'll use it in the translator. (Generally, Working papers are handled much better in the metadata in Econpapers. On IDEAS I have to engage in a lot of guesswork to even distinguish WPs from journal articles).
3. I see the issues, but it's not easy. If you look at the site structure, getting the original URL is hard. That might have to involve more hackery than I want to employ. Again, though - if it were in the metadata I'd happily take it.
4. I actually disagree here. It's much easier to get citations right with the whole title (like "IZA Discussion Paper") under report type than combining series and report type. So not going to do that.
Anyway, I will point Christian at this discussion and hope that Sune is still onboard.
2) Actually the ReDIF specification distinguishes between five different types of resources: paper, article, book, chapter, software (see Section 2). So that info is available and would just have to be published to the metadata, too. No guess work necessary then and support for more item types possible. :)
3) I understand. Though 'u=' (EP) and 'File URL:' (IDEAS), followed by the respective File URLs seem to be more or less unique on each page. So possibly not THAT hard.
4) That depends on the citation style, I suppose. The APA style does not look too bad with my suggestions:
Ohinata, A., & van Ours, J. C. (2011). How Immigrant Children Affect the Academic Achievement of Native Dutch Children (Discussion Paper No. 6212). IZA Discussion Papers. Bonn: Institute for the Study of Labor (IZA). Retrieved from http://ftp.iza.org/dp6212.pdf
One more thing:
5) The publishing place is also not imported. It is not provided by IDEAS nor EP though, except within the address field. It could be provided however, since it also is part of the ReDIF spec (section 3.1 'location').
EDIT:
2) I just saw that the resource type is implemented in the metadata of Econpapers already. It's called 'redif-type' and AFAIK you don't make use of it in the translator yet. If you need an example for the software type: ESTOUT on Econpapers and on IDEAS.
Ohinata, A., & van Ours, J. C. (2011). How Immigrant Children Affect the Academic Achievement of Native Dutch Children (IZA Discussion Paper No. 6212). Bonn: Institute for the Study of Labor (IZA). Retrieved from http://ftp.iza.org/dp6212.pdf
A better APA citation than yours ;-).
I'll take a look at 2 and 3, but it'll take some time. You're more than welcome to take a stab yourself. This is a framework translator, so the code is super easy to follow and all you really need are xpaths and regex.
Second, and more general, I think what bothers me is that the mapping does not properly resemble the naming of the fields in Zotero. I think e.g. IZA Discussion Papers is indeed a series title (or collection-title in CSL) and not a report type or genre. So getting your citation with the series title mapped to report title in Zotero and subsequently to genre in CSL actually does not seem very intuitive to me. Then again, resolving this properly would lead to a whole lot of work (BibTex export for one, which also only uses 'type' for technical reports and has no series field). One solution could be to map the series title to both the Series Title and the Report Type fields, but that would be both redundant and incompatible with the current APA style. So I guess will have to let it go. :)
BTW, strangely enough I could not find a Zotero RDF Specification. Where is it?
Regarding giving it a shot myself: I would love to, but my knowledge of xpath and regex is very minimal and time is also an issue here. ;) I'll contact Christian first and see what I can do.
Is there actually a specification for the High Press meta-tags available?
For papers, RePEc also has information on the length (ReDIF spec, 5.1. Only IDEAS publishes it though). If that were in the meta-tags (citation_pages or citation_length?) it could be scraped as well to # of pages, while we are at it. :)
1. keywords should already work (as tags). If they don't you may have the option to automatically associated tags turned off in the Z. preferences.
2. I'd have to check - there may well be a reason I went with wp-number and not the google/highwire field.
I think the google meta specs are here:
http://scholar.google.ca/intl/en/scholar/inclusion.html
3. URL (and in many cases: pdf download) looks like it might be possible. I have it working for IDEA (locally), but Econpapers is harder.
4. Series title - what you say, it's complicated. I'm not going to mess with that. There is no data standard for working papers in the first place.
5. number of pages is a current limitation/bug in Zotero: Reports don't have a number of pages field, just a pages field. Else I'd put the length there, yes.
2. is now using the technical_report_number in econ papers. It's parsing the number or number hyphen combination at the end of the URL for IDEA. Any reports on how that works welcome.
3. URLs and pdf download should now work (pdf download only where a pdf is linked, else Zotero will throw out the result.
4. Not done yet - but thinking about this I could just remove the plural s at the end of papers (I'd so something like - search for "apers" at the end of the string and replace it by "aper". That way it won't catch other names (like "Working Paper Series") a). Makes sense?
I've tried to test these, but more testing would be welcome.
There is an issue with authors in idea - they're not consistent (sometimes "Firstname Lastname" and sometiems "Lastname, Firstname") and the simple translator can't distinguish between the two.
I might be able to fix that at some point, but right now Zotero will assume "Firstname Lastname".
1) Keywords don't work for me. Neither with the old one nor with the two new translators (checked with the Ohinata & van Ours Paper above, amoung others). Option to import tags is on in Zotero prefs and works with the Sciencedirect translator for example.
2) Report number seems to work. Needless to say that it would be better to use citation_technical_report_number on IDEAS as well once it's provided (e.g. the URLs sometimes do not contain a hyphen when the report number does), but in the meantime it's a nice work-around.
3) File URLs are scraped nicely. It would be nice to attach a link to the abstract on EP/IDEAS as well (citation_abstract_html_url). Like when you add a RIS record via clipboard - the attached link is the same as in the URL field in that case though. I have not checked automatic PDF download since I am not using that.
4) Sounds reasonable. Some will still require manual tweaking but we will just have to live with that.
Regarding "Firstname Lastname": I discussed that with Christian a while back and apparently it's a mess. The ReDIF spec is not strict enough and the publishers provide RePEc with all sorts of combinations (though EP and IDEAS seem to handle things differently as well). Anyways, I would very much opt for "Lastname, Firstname", though. That's what Highwire uses and if it is provided that way, chances are that it also gets e.g. particles right (your current implementation on IDEAS really scrambles up e.g. Jan C. van Ours). I.e. LN,FN contains slightly more information than FN LN. If you want to make it fancy, you could search for a comma, if yes assume LN,FN, else assume FN LN. ;) (Well, in a way Zotero already does that, right? If FN LN is provided but LN, FN assumed, the full name is maped to the LN field. Two clicks on the 'switch to single field' button and you have the proper mapping - provided it does not contain particles. That does not work with assuming FN LN, so again LN, FN would be preferred.)
Regarding the meta-tag specs: Google did not draft the citation_* meta-tags but merely recommends their usage for inclusion in Google Scholar. "Google Scholar supports Highwire Press tags (e.g., citation_title), [among others]" Highwire (sic!) Press seems to have drafted them and uses them for the journals they host. (NB: The two Highwire translators do not make use of them, but instead scrape the RIS citations from their site (if I understand correctly). There is some data in the tags that's not in the RIS citations, e.g. ISSN number and journal abbreviation, so it might be worth a look whether the Highwire translators could be improved?! Should I write a new post on that?). The Google Scholar guidelines provide nice examples, but I would not consider them a proper spec. Some tags Highwire definitely uses are not included (e.g. citation_doi, citation_abstract_html_url, citation_public_url) and others would be helpful as well (e.g. publisher place or resource type (article, paper, software, etc.)). I guess Highwire Press may/should have a spec.
I asked Christian and Sune whether they could provide file URL and publication place/location (both) as well as resource type, report number and (full) date of publication (IDEAS) in the meta tags. Christian replied shortly that he will look into it as part of the current redesign job. If they do, the translators could be merged again, I suppose, because the information should be equivalent then.
Remaining issues:
- other resource types (software, chapter, book): On EP the ESTOUT example above is scraped with the embedded metadata translator as a website, from IDEAS it shows up as a report, so importing e.g. the RIS data via clipboard still gives better results. On EP it would already be possible to destinguish between the types via the redif-type tag.
- number of pages, once in the tags (still need to ask) and fixed in Zotero (or map it to the pages field in the meantime?)
- publishing place once in the tags.
Report number: Using the last part of the handle is a bad idea. It often corresponds to the actual report number but can also be completely nonsensical.
URL: Apart from the fact that I like it when people link to EconPapers I would argue that the EP/IDEAS abstract URL is better than the scraped full text URL in a citation. The reason is that EP/IDEAS URLs are stable while the link to a departmental working paper easily changes when the web is redone, server software is upgraded or the phase of the moon changes.
Pages for working papers: It is pretty rare that we have that information but it certainly can be put in the metadata but which field name, citation_pages? Will Zotero use it?
Publishing place: Once again pretty rare information, less than 50 out of 4600 series, and hardly worth the effort. It is more common that we have the address, but do you want something like "The Economic Research Institute, Stockholm School of Economics, P.O. Box 6501, 113 83 Stockholm, Sweden" cluttering up your references?
Author names: This is tricky since the raw data is inconsistent. Some use last_name, first_name, some first_name last_name and some last_name first_name (without the comma). EconPapers tries to guess and put it in last_name, first_name form but obviously gets it wrong sometimes. I will see what we can do for the vans and vons of this world (How would you treat Bruno van Pottelsberghe de la Potterie?)
I'm not happy using the last bit of the URL for report number - and I'm not doing it for Econpapers, where you have the number in the metadata - but I do think it's better than nothing and it seems to work more often than not. As soon as report numbers show up in some form in IDEA metadata I'll replace that.
I'll defer to you for the URL and change that back. Parsing the URL will still be useful for direct pdf download.
I'll eventually improve the author scraping - I think (hope!) I can use a hook in the translator to plug in a function along the lines suggested above that will test for a comma and then decide what to do.
Ideally "Bruno van Pottelsberghe de la Potterie" would be "van Pottelsberghe de la Potterie, Bruno" - Zotero is quite sophisticated dealing with these types of names for citations.
I disagree with you on the URL, though. One of the main usage scenarios of the URL field in the main record of a Zotero item is to use it for citations. It is common practice to cite the source directly and I have yet to come across a citation that refers to a RePEc record. That's why I think the full text URL would be highly preferable for the URL field. I agree that the respective RePEC abstract URL should be added to the Zotero item (as an attachment) to make papers traceable in case a direct link changes or one simply wants to consult the RePEC record again.
The full text URL could also be provided in a meta-tag by the way, so it would not have to be scraped (though that seems to work quite well too).
# of pages & publishing place: Even though that information may not be very common in the RePEC records, I would in general utilize all data that is available with a reasonable amount of effort. Zotero cannot handle # of pages in the current version, but will as of version 3.1. I was not able to find an 'official' tag for this, but my suggestion would be citation_number_of_pages, as the citation tags seem to be quite descriptive (cf. citation_technical_report_number).
I certainly would not want the whole address in the records, but 50 something is more than nothing and it's probably just one line of code? It's not uncommon to use the place information in citations as well.
Maybe people don't cite repec URLs, but, given my experience with the lack of stable URLs of many econ working papers - especially those published by universities/departments - maybe they should. The source is still going to be in the citation. There is no rule against it that I'm aware of. If anything, citation styles would encourage the use of a URL that's as stable as possible.
(On a side note, though: It seems to me that just because RePEc URLs may be rather stable it does not imply that the source is always available (example) So unless the providers feed the new URLs to RePEc either solution is as unstable as the other as far as access to the source is concerned. But, case closed.)
I saw yesterday's update of the translator with the RePEc Snapshot attachment. Very nice, exactly what I meant. It's kinda sad to see that go again. ;)
Maybe you could implement it the other way around, then? RePEc URLs to the URL field and file URLs as an attachment?
Could you also switch the author name back to assuming lastname, firstname (or the way it was in the single translator)? The current one messes up the name if it is in lastname, firstname form and then it's not very useful. The former was much more robust IMHO.
I tested the keywords/tags issue again on another computer and also with a clean profile and it just does not work for me. Let me know if you need more info on that.
It includes a bunch of fixes:
-most notably a hook that will deal with authors regardless of order.
-It reverts the URL to the repec one. I fix the Snapshot issue hopefully - this should now always attach a snapshot of the repec page.
- It now imports software - that works better on Econpapers than on idea, where there's not much metadata.
- I implement the ...apers ---> ...aper fix for working paper series discussed above.
- some clean-up.
I have been able to replicate the tag issue now - that should work, I don't know why it doesn't - problem goes deeper than this translator and won't get fixed any time soon.
I get a "could not save data" error on IDEAS though, at least for papers. I first thought that it was due to the changes to the meta-data Christian is implementing right now. So I e-mailed him, but he said he did not remove any meta-tags. I tried an older IDEAS translater then and that works fine, so something must be wrong with the new version.
Christian has been/is adding citation_publication_date, redif-type, citation_number_of_pages, citation_technical_report_number. citation_publication_place may be added at a later date, since that is not trivial and not a priority. Right now, citation_publication_date may still be empty due to a bug. The redif-type content will be article, paper, software, book or chapter, as in EP, i.e. without 'redif-' in the content field as it is now. Once the scripts are changed it can take up to a month for all pages to be refreshed.
- Snapshots: I don't know whether that was a misunderstanding, but my suggestion was to attach the URL to the PDF instead of the Snapshot of the RePEc page. The snapshot is not necessary IMO, since URL already goes to the URL field now. I would like to have the URL to the PDF/source in the Zotero record somewhere, without having to attach the PDF automatically. (Or is that not possible? Right now, I only get the Snapshot when 'Automatically attach PDFs and other files' is turned on in the Zotero preferences. A snapshot is probably an 'other file' but a link should be possible? If I import a RIS record from the clipboard, I do get a link attachment with 'Automatically attach PDFs' turned off.).
- Software: Import works fine. On IDEAS it apparently puts the 'report number' in the version field, which does not make sense IMO. We could use citation_journal_title for Series Title. It may be better to give priority to citation_publication_date over citation_year for the date, because that seems to be the date of the latest version. It may also be better to turn the attachments off or only use the first URL found, since most Software records provide multiple URLs (example - SSC is the by far largest software series on RePEc). On EP we could use dc.publisher for the Company field and "series" for Series Title.
Any ideas on why the keywords work on your setup and not on mine (updated to Zotero 3.0.1 and Firefox 10 in the meantime)?
I tracked down the "could not save data" error on IDEAS and it apparently appears only when names contain a middle initial and are given in the "LN, FN" order. My main testing example still is the Asako Ohinata & Jan C. van Ours paper above, so that's why I got the error right away. It seems to be caused by the hook for the author names combined with the lack of the 'true' statement in the cleanAuthor function (if that makes any sense). That is to say, cleanAuthor("author") translates to {"firstName": "van Ours, Jan", "lastName": "C", "creatorType": "author"}. The hook then gives a "TypeError: item.creators[i].firstName.match(/,.+/) is null". cleanAuthor("author", "true"), however, imports the name correctly, i.e. {"firstName": "Jan C", "lastName": "van Ours", "creatorType": "author"}.
I don't know whether the hook can be fixed to deal with such names (I cannot). I think though that the LN, FN form is the cleanest we get from RePEc and should just take it as it is. That is use cleanAuthor("author", "true") and adapt the hook so as to deal with names in the FN LN order. Is that an option?
I will add a few more changes together with some additional test cases and post them on gist soon.
- test cases: I replaced/added some test cases, mostly to cover different naming schemes for author names (I even came across Bruno van Pottelsberghe de la Potterie by accident). According to Christian the Philippine Journal of Development is not a very good one, since they seem to change handles fairly often.
- author names: I changed it to cleanAuthor("author", "true") and disabled the hook for the time being.
All test cases work now, but full names go to the last name field when provided in the FN LN form. But it takes only two clicks on the 'toggle single field' button to get most names right (well, all that have one first name and one last name only, including hyphenated ones.).
I also added/changed some other stuff, e.g. citation_publication_date, redif-type, citation_number_of_pages, citation_technical_report_number, citation_publication_place, citation_software_version and the other things I said about software in post 28. Not all pages have the new meta-data already, so I left in what soon will be 'legacy-code', which can be cleaned-up in a while. Some meta-data may not even see the light of day, but if it does, it's already in the translator. I did not touch the snapshot/link attachment issue, so that is still open as far as I am concerned. I also did not touch the EP translator yet.
Two more questions:
1. I mapped citation_publication_date to the date field (in addition to citation_year). If publication_date is empty (and citation_year is not), it returns a [object XrayWrapper [object Attr]]. Can it be changed so that it takes the value from citation_year if citation_publication_date is empty? (Note that this may only be relevant for a short time, until citation_publication_date is updated on all pages. I would be interested in how to set priorities in the framework translator though).
2. Is there a way to add an en-dash to a page range instead of a hyphen when importing (a hyphen is grammatically and typographically incorrect, see e.g. Wikipedia)? If I add the dash to the translator, it will show up in Scaffold but not in the Zotero library upon import.
https://github.com/adam3smith/translators/blob/repec/RePEc%20-%20IDEAS.js
(your link above is wrong, though, so I can't tell in detail - I may want to copy some of the additional data lines).
For your questions
1. If the tag isn't there it should work - if it's empty we currently can't do anything, we'd need a fancier translator for that (do you have an example?)
2. Zotero corrects that when creating citations. I'm not sure about bibtex export, but it should. If it doesn't that would be a different topic.
The attachment issue will not change - I don't think what you want can be done - if it can, it involves more of a hassle than I'm willing to invest.