> You can't easily define another search engine, no.
> And even if you could - what would you use?""

Well, sadly, like most researchers, I too have a narrow field of research so I would use IEEE Xplore which has most of the papers for which I already have a PDF that I want to get into Zotero properly. I don't need Google Scholar for this, which quite often has poor results when automated (and poor bibtex entries in general).

> FWIW, PDFs for which Zotero finds a DOI on the first
> couple of pages will do quite a bit better - the data
> comes from CrossRef, but you still won't get abstracts
> and keywords.""

Wait, wait. So Zotero scans the first two pages looking for a DOI then looks online in the CrossRef database to fetch info?

Here are some suggestions based on the above -- if/when I have some time, I might actually give this a go myself if the JS API is flexible (I haven't checked it yet). Zotero already supports importing from lots of online publishers and databases, e.g. IEEE Xplore. It would be great if Zotero had either or both of the following features:

- a search box on the menu bar that would search a chosen publisher/database (e.g. IEEE Xplore) and get the first match as a Zotero entry - rather than me having to manually browse to the publishers entry, search there and then click the Zotero import icon. The search box would have a small dropdown menu to choose the publisher/database to search in, and also remember it afterwards, like the current search box does. This feature could then be exposed in the API so that people can script it and perform batch imports like the one I'm trying to do now.

- a button on the menu bar (not a menu > submenu > entry, too many clicks) to do the above search using the currently highlighted entry when it wasn't yet fully imported, e.g. it's a drag and dropped pdf. The search should try first extracting metadata and if that fails then it should use the actual file name (without extension, after beautifying it a bit replacing [-_.] with spaces etc) since most of the time, the file name is very close to the actual full title of the paper. The user should also be able to specify which publisher or online database to search. Zotero can (should?) even then perform all this when a PDF is drag-and-dropped. You can have an additional option in the Preferences so that a multiple entries are presented to the user corresponding to the search results, just like Zotero is currently doing with search results from Google Scholar for instance.

Should I make these suggestions in some other dedicated area (email)?

Do you think you'd implement these anytime in the future? I think they would be really helpful. Managers like Jabref already have specific publisher/database search features. Zotero could be a lot better as you already support importing from a lot more of them.

Reagrds.

But you'd only have to send the search string to the online publisher/database search engine, exactly in the same way you'd make it with a browser, i.e. Zotero would send same HTTP GET query as if it was from the browser on that webpage (*). The returned html would be the same as if the search was performed with the browser directly, and it would be fetched and parsed in the same way Zotero already does now.

I don't see the huge amount of work at all to be honest. Surely, adding a search box or button on the GUI is not that much of a monstrous task. The translators would not need much alteration either, just a template HTTP GET query with a placeholder for the actual search string.

If you really don't want to do it (why? it'd be so useful) then I could actually do this myself as I have some experience with FireFox extensions and JS. Where can I find the necessary sources to implement this?

(*) there is an associated issue in that the publisher may change the HTTP GET query format so Zotero will have to be kept up to date. However, that happens rarely and you must already be doing this anyway when parsing html pages, since publishers may change the html layout too.

Regards.

p.s. If I actually do it, would you consider bumping up my space on Zotero? :)

> something like - you right-click on the title and select
> "complete using IEEEXplore" - that might even allow users
> to get the PDFs.

That's pretty much what I was suggesting above (the 2nd suggestion) although I was talking about a button to save users from right-clicking. Right-clicking would still be fine since you allow doing this for multiple selected/highlighted entries. Having this would be of real value since Google Scholar results are quite poor.

Since this would involve minimal GUI changes, then it could also be augmented with filename -> cleaned filename -> title -> metadata retrieval.

Is Zotero currently using the "Title" metadata field in the PDF when retrieving metadata of drag and dropped PDFs?

This might be one of those "leave the choice to the user, don't make it for them" issues. I think the title should be used and then Zotero would present the user with a list of results and checkboxes, just as it does when clicking the icon on a search results page on Google Scholar.

Would be extremely helpful since most of the time (virtually always in my case) the first result is the correct one when searching using the full title, not just a few words from the title. Granted, for mass-import you can just disable this if multiple entries are selected.

In my case, I could script this. i.e. update the PDFs whose "title" field is empty or not matching the article title (I have a lot of these) using the file name. Then drag-drop into Zotero and right click to have Zotero fetch the first result of a title search for each of them.

I'd actually like to try this. If it proves to work well then I'll post the patch. Could you please give me a head start by pointing me to the API zone and source files I need to implement a "title" based search using an alternate search engine, e.g. IEEE Xplore?

> One issue to consider is that currently Zotero is
> preventing false positives by searching for chunks
> of the full text rather than just the title.

Are you sure about that? That would require fetching the PDF from the remote site as well and then compare (unless you're searching a full-text indexed database) ... for all PDFs I tried it returns much to quickly.

Thanks for the code entry point. How about the source code entry point for the "Retrieve Metadata for PDF" feature (from the right-click menu of a drag-and-dropped PDF).

I know you pointed me to the dev list, but would same me some time to reexplain the issue there if you knew the above too. I'm happy to code beyond the API.

This might interest you: https://github.com/zotero/zotero/issues/99
(edit: http://forums.zotero.org/discussion/22882/)

Maybe I'm wrong but lookup engines could help you here:
If you are able to import very basic metadata (from filenames), you can try to write a lookup engine which would search for your articles in your favourite database (three clicks at the moment: select the item, click on the "localize" green arrow, select your lookup engine). Then you'll just have to add the result in zotero (one click). Far from being perfect but with more coding you can improve/automate this workflow.
Just my two cents.

This reminds me this thread too.

Personally I hate endnote's integrated search. It always retrieved a lot of articles that I didn't want. I would probably still prefer using the publisher websites for retrieving articles, but if people want to use search from Zotero, I think we can actually integrate it pretty easily. Most translators support search result pages, so we can simply send queries from zotero to whichever supported website the user chooses and display the "multiple" dialog so they can select what to import. Most of the functionality is there. Just need to build urls for search queries. As I said though, native websites are probably still preferred, since you can browse abstracts, etc.

@aurimas: that's exactly what I suggested in a post above yours when describing the HTTP GET request. Virtually all functionality is already there in Zotero so when adding a PDF and then clicking on "import metadata for pdf", the user could select which search engine to send the query to using a HTTP GET query that uses the title extracted from the pdf if nothing better is detected.

This way people (like me) can also automate batch import jobs which currently is quite a pain for all pdfs that do not have proper metadata fields already which Zotero can use to search ... and even if it does, the returned result is of quite poor quality because of the search engine used. Don't get me wrong, Google Scholar is great, but its bibtex entries suck.

If there are no metadata fields found in the PDF then Zotero can attempt to turn the filename into the search string to be sent to the search engine. Most of my pdfs filenames actually contain the full paper title.

I could have a go at this if you guys are not going to anytime soon. I don't have immediate free time either but if I manage to code it before you begin then you can decide what to do with it.

Well, I have a ton of pdfs I'd like to import and start using. You guys can take your time.

> - do we take people to the search result
> page (aurimas' solution), or auto-import
> the first item?

I think both myself and aurimas were saying Zotero should present the multiple choice list with checkboxes representing the search results. I would further enhance this to automatically take the first result in case multiple entries (pdfs) are selected in Zotero when clicking on "retrieve metadata for pdf"

> - where do we look in the pdfs for the title?

I think I described this several times now. First, in the pdf metadata itself. Every pdf has embedded metadata (many times they are empty). Check your pdf viewer > document properties > metadata/description. E.g. Acrobat: http://bit.ly/M3jef7 . If that fails, then my suggestion was to use the filename, after cleaning it up with s/[^a-z0-9]/ /g or similar so that search engines have a cleaner search string. Zotero may even allow the user to edit the search string before being sent to the search engine.

> - how do we prevent false positives?

Let's not. I'd be interested in receiving the full list of results in the form of the multiple checkboxes dialog that Zotero extracts from the html search results page.

> - how/where do people select the database they want to query?

Preferences for starters, but a dropdown list on the menu bar would be much better as a second implementation stage.

> - for completing items: How do we deal with conflicts?

In what sense conflicts? You mean duplicates? The user would deal with that afterwards, it wouldn't be Zotero's fault.

> - how/where is the search triggered?

I'm not sure I follow. A HTTP GET request straight to the selected search engine, exactly the same way the browser would do it if I was to search for the same string on the search engine's webpage (E.g. "multi-carrier burst contention" would result in http://ieeexplore.ieee.org/search/searchresult.jsp?newsearch=true&queryText=multi-carrier+burst+contention ). Zotero would then parse it exactly in the same way it already does if I was to press its "save to Zotero" icon

> and I'm sure there are more.
>
> Sending the search queries via translators is
> really the easiest part here. Implementing it
> in a way that works and is intuitive is the
> main chunk of the work.

I don't see the complexity to be honest but it's not my software. I will probably have a go at it for myself just to be able to mass-import all my pdfs. It's too much of a pain to do it manually using the conventional method.

I mean: How does the user tell Zotero s/he wants to search for something? I have absolutely no sense of the GUI vision behind all this yet.

I made several suggestions in this thread about that:

a) a search box (you said it's too much work),
b) a button to trigger a search using the highlighted entry that a drag-and-dropped pdf produced (again, apparently too much work),
c) via the "retrieve metadata for pdf" entry which would automatically search using the specified search engine, using metadata fields from the pdf or the tital via the filename -- which is what we've been discussing for the last several posts

If you don't want to take the time to discuss & think through these issues with people who have used Zotero for much longer as well as the core devs who have written it that's obviously fine.

I'm too tired to debate this awkward comment or understand why we're discussing willingness issues when I clearly spent time in this thread and am willing to contribute. That said, I think I was clear in several posts above that I'd like to import a large number of pdfs faster than the snail pace Zotero currently allows, and I'd like to do it soon. For that purpose, I can get to work rightaway and hack it to my liking; why should I wait weeks/months for people to decide on how a GUI that I don't care about should look like? It's primarily for my own use, with potential of being of use to others too.

I however made several suggestions re: the GUI for the features I proposed in case devs are interested to implement them in Zotero but you're not obliged to use any code I produce if you think it was either rushed or not properly discussed beforehand. It also doesn't mean I'm not willing to discuss implementation aspects for a Zotero adoption and adjust my code accordingly afterwards. But who knows how tired I'll be then ...

It does, unfortunately, mean that what you code will likely only be marginally useful for inclusion in Zotero.

I intend to implement c) above with IEEE Xplore. My code might already be working before you even discuss GUI placement. If I was part of the dev team then I'd use something like that or parts of it. But I'm not, and you're not obliged to use any of it. I'm open to discussion regarding a global Zotero implementation, but for the immediate purpose of importing my pdfs I'd rather get to work.

What I could see is a way to complete existing but incomplete or low quality entries (e.g. retrieved from google scholar) using this: something like - you right-click on the title and select "complete using IEEEXplore" - that might even allow users to get the PDFs.

This could even be automated, right? If the user could specify a preferred database (e.g. PubMed for the life science folks) for metadata retrieval, Zotero could still use Google Scholar for the initial full-text search, but instead of saving the metadata of the first match, it could use that to query PubMed, and save the best match from there (if there is one).

just as one example of why this is complex:

> - where do we look in the pdfs for the title?
I think I described this several times now. First, in the pdf metadata itself. Every pdf has embedded metadata (many times they are empty).

If you check PDFs from IEEExplore, you'll find that many times the XMP title tag isn't blank. But it doesn't contain the paper title, it has:
"Paper Title (use style: paper title)"

This one for example - but this was the case for three out of four papers I randomly downloaded from IEEExplore.

[1] J. You, X. Jiang, N. Wang, Z. Shen, Q. Ma, and W. Peng, “Study of without blankholder drawing for individual titanium implant forming,” in Mechanic Automation and Control Engineering (MACE), 2010 International Conference on, 2010, pp. 5766 –5769.

The problem is, of course, that searching for this in IEEExplore does produce results --> as a result, depending on the setting, if you're trying to get metadata for ten of these papers either you get wrong metadata for eight papers or you get eight completely useless search results that you have to click away. Neither is acceptable. So no, just using the XMP title tag is not a viable option.

I think using the filename is more promising, for IEEE the default is just a number, which at least doesn't produce false positives - but I'm nots sure that's the case for other databases, too.
This has to work in a way that it doesn't just work for users who have saved files in a specific way.

@Rintze
I like that thought in general. One downside is that it doesn't circumvent google scholar, which I'd really like to be able to do, because GS locks out people batch-importing papers.

I mailed this to zotero-dev group before: http://labs.crossref.org/styled-6/pdf_extract.html. We can use similar logic for abstract. For instance: first paragraph with more than 3 lines of text and the same font size as the main text (in case there is some sort of copyright block). Or we can try identifying title and authors. It does get quote complicated though.

@aurimas: I also pointed to CrossRef's PDF-EXTRACT tools for content extraction.

Here's a technique that I just tried and worked surprisingly well. It's borrowed from a former colleague, Stephen Kell, who wrote a set of quite nifty scripts a long while ago: http://www.inf.usi.ch/postdoc/kells/goodies/research/bibtex/

- get the full text from the pdf starting with page 2; skip first page as it might be boilerplate

- crossref's pdf-extract can be used with "pdf-extract extract --sections" and then concatenating the <line> nodes (pdf-extract is very slow though). I advise against using ps2ascii from ghostscript as it still has new line issues. I also advise against pdftotext as it doesn't work on binary encoded pdfs. Ghostscript can be used to skip the first page and then pass it to pdf-extract.

- extract the first 10 or so consecutive "sane" words, i.e. that do not have any weird chars; regex is your friend. As an optional method, these can also be passed through aspell to detect any non-spellcheck'able words -- this should take care of words inside figures which end up concatenated and not making sense. Continue until 10 really "sane" consecutive words are found.

- pass these as the search string to google scholar. This actually worked for me surprisingly well. At this point, Zotero may prompt the user with a multiple choice list to select the best entry -- by default, the first one. if none are found, then maybe those 10 words were actually bogus ones (from figures, not properly extracted, etc) in which case, the above step could be repeated starting with where those 10 words left off or from the 3rd page. repeat this a maximum number of times before giving up.

- take the google scholar first entry info and then use that to search consecutively in a ranked list of search engines (e.g. 1-ieee xplorer, 2-acm.org, 2-springer, 3-citeseerx etc) until a match is found and then fetch all metadata, including the pdf.

I tried this approach manually on a few papers and it worked really well. I'm building now a shell script so that you guys can try it on some of your pdf samples. You'll need ghostscript and crossref's pdf-extract (which in turn requires ruby).

@adamsmith: I'm trying to spend time constructively, i.e. to find a method that works on a good set of samples, rather than counter-examples.

Please point me to where Rintze described that full algorithm that I just did. I can't see it.

You first said it should work in general and not for specific cases like mine. I provided an idea and an algorithm exactly for that. Now you seem to be saying pretty much the opposite. Let's stick to a normal usage case that covers most users rather than finding exceptions. Most users are not doing mass-imports (*).

For that purpose, GS can and I think should be used. It's pretty much the only full-text search engine that is that vast and has some reliability. Zotero uses it right now any so the lockout argument doesn't really hold -- my algorithm above wouldn't be different from that perspective to what Zotero already does.

Regardless of the method, there will always be examples for which it doesn't work. The idea is to find something that covers the majority of usage cases if we are to continue this discussion. I'm no longer discussing my mass-import (which I pretty much solved already anyway).

(*) when doing mass imports, Zotero can just introduce a delay of N seconds between importing each entry to avoid the lockout. Users will probably happily wait to get their database fully imported with high quality metadata.