Import Annotations for multiple PDFs

jandavid

When importing a large PDF library into Zotero is there a way to have Zotero automatically convert all embedded annotations of the PDF files to Zotero annotations?

I know that in Zotero's "Export Library ..." dialogue, when choosing to "Export files" I have the option to "Include annotations", which according to the documentation should embed Zotero annotations in the PDF (to be read by external PDF readers).

I want to achieve the opposite. I want to have Zotero move all embedded PDF annotations into Zotero's database (deleting the embedded annotations in the PDF files).
So essentially I'd like to do in bulk what I can do file by file by
- opening a PDF in the Zotero PDF viewer and then
- choosing "Import Annotations ..." from the File menu

Is this possible? Or possible to implement?

emilianoeheyns

We've been discussing this in the context of importing entries using BBT (or any other import translator). Import translators don't deal with the contents of attachments, so this would be a job for the import framework itself.

kdb_research

+1 for this

jandavid

Just following up, is this is something it could be considered to implement in a future release?

jandavid

Just drawing attention to this again.
Any chance this could be considered in a future release?

jandavid

For anyone else with this issue, with the help of Claude I've come up with a script that imports annotations for any PDF in the library that has the tag annotationsImport. Can be run from Tools -> Developer -> Run JavaScript

Explanations are in the script. It has a dry-run option for testing first. Also it has an option to turn all highlights to yellow, to avoid the issue discussed here where the imported color is too bright:
https://forums.zotero.org/discussion/126747/color-of-imported-highlights#latest
The option is set to "false" by default.

IMPORTANT: It only works on highlights and underlines, not on freehand/ink or other elements. This is because the annotation engine of Zotero can only import freehand etc. that was previously created in Zotero. See here for discussion:
https://forums.zotero.org/discussion/126434/import-annotations-doesnt-work-for-freehand-markup-and-text-notes#latest

Here's the code. Paste it into the window in Tools -> Developer -> Run JavaScript and run as async function.


// Import the annotations embedded in PDFs into the Zotero database, for all items tagged "annotationsImport".
// Equivalent to File -> "Import Annotations..." in the reader, run for every tagged PDF.
//
// Run in Zotero: Tools -> Developer -> Run JavaScript
// (tick "Run as async function" if the window offers that checkbox).
// Start with DRY_RUN = true and LIMIT = 3, read the output, then widen.
//
// Based on Zotero.PDFWorker.import(itemID, isPriority, password, transfer) in
// chrome/content/zotero/xpcom/pdfWorker/manager.js (main branch). That function does the
// whole job itself (reads the PDF, saves the annotations, and with transfer = true removes
// them from the PDF file); it returns only a Boolean, not the annotations.

const TAG = "annotationsImport";
const DRY_RUN = true; // true: only report which PDFs have embedded annotations
const LIMIT = 3; // 0 = all attachments; small number = test run
const TAG_DONE = "annotations_imported"; // replaces TAG on items whose PDFs were all imported cleanly
const SWAP_TAGS = true;
const RECOLOR_HIGHLIGHTS = false; // set all editable highlights to Zotero's yellow
const YELLOW = "#ffd400"; // Zotero's default yellow (underlines are left alone)

const log = [];
let nAtt = 0, nDone = 0, nNone = 0, nErrors = 0;

// 1. all items (parents or standalone attachments) carrying the tag
const s = new Zotero.Search();
s.libraryID = Zotero.Libraries.userLibraryID;
s.addCondition("tag", "is", TAG);
const ids = await s.search();
const items = await Zotero.Items.getAsync(ids);

// 2. collect their PDF attachments
const attachments = new Map();
function addAtt(att) {
if (att && att.isPDFAttachment()) attachments.set(att.id, att);
}
for (const item of items) {
if (item.isAttachment()) addAtt(item);
else if (item.isRegularItem()) for (const id of item.getAttachments()) addAtt(Zotero.Items.get(id));
}
log.push(`Items tagged "${TAG}": ${items.length}; PDF attachments: ${attachments.size}`);

// 3. import
const counts = (att) => {
const a = att.getAnnotations();
return { editable: a.filter(x => !x.annotationIsExternal).length,
external: a.filter(x => x.annotationIsExternal).length };
};

// Highlights (editable ones only) whose color is not yet yellow
const wrongColor = (att) => att.getAnnotations().filter(x =>
!x.annotationIsExternal && x.annotationType === "highlight" &&
(x.annotationColor || "").toLowerCase() !== YELLOW);

let nRecolored = 0, nRetagged = 0;
const okAtt = new Set(); // attachments imported without errors and without leftover external annotations

for (const att of attachments.values()) {
if (LIMIT && nAtt >= LIMIT) { log.push(`(stopped at LIMIT = ${LIMIT})`); break; }
nAtt++;
const parent = att.parentItem;
const label = `${parent ? parent.getField("title") : att.getField("title")} [${att.attachmentFilename}]`;
try {
if (!(await att.fileExists())) { log.push(`MISSING FILE: ${label}`); nErrors++; continue; }

if (DRY_RUN) {
const has = await Zotero.PDFWorker.hasAnnotations(att.id, true, "");
const c = counts(att);
log.push(`${has ? "WOULD IMPORT (PDF has embedded annotations)" : "no embedded annotations"}` +
` | now in DB: ${c.editable} editable, ${c.external} external` +
`${RECOLOR_HIGHLIGHTS ? `, ${wrongColor(att).length} highlight(s) to recolor` : ""}: ${label}`);
if (has) nDone++; else nNone++;
continue;
}

// transfer = true: annotations become normal, editable Zotero annotations and are
// removed from the PDF file in Zotero storage (exactly like the menu item).
const any = await Zotero.PDFWorker.import(att.id, true, "", true);
let recolored = 0;
if (RECOLOR_HIGHLIGHTS) {
for (const a of wrongColor(att)) {
a.annotationColor = YELLOW;
await a.saveTx();
recolored++;
}
nRecolored += recolored;
}
const c = counts(att);
if (any) nDone++; else nNone++;
if (!c.external) okAtt.add(att.id);
log.push(`${any ? "imported" : "nothing to import"} | now in DB: ${c.editable} editable, ` +
`${c.external} still external${c.external ? " <-- CHECK" : ""}` +
`${RECOLOR_HIGHLIGHTS ? `, ${recolored} highlight(s) recolored` : ""}: ${label}`);
} catch (e) {
nErrors++;
log.push(`ERROR: ${label}: ${e}`);
}
}

// 4. swap the tag on every tagged item whose PDF attachments were ALL handled cleanly
if (SWAP_TAGS && !DRY_RUN) {
for (const item of items) {
const pdfs = item.isAttachment() ? [item]
: item.isRegularItem() ? item.getAttachments().map(id => Zotero.Items.get(id)).filter(a => a && a.isPDFAttachment())
: [];
const title = item.getField("title") || item.key;
if (!pdfs.length) { log.push(`TAG KEPT (no PDF attachment): ${title}`); continue; }
if (!pdfs.every(a => okAtt.has(a.id))) { log.push(`TAG KEPT (not all PDFs processed cleanly): ${title}`); continue; }
item.removeTag(TAG);
item.addTag(TAG_DONE);
await item.saveTx();
nRetagged++;
}
}

log.push("");
log.push(`${DRY_RUN ? "DRY RUN. " : ""}Attachments looked at: ${nAtt}, ` +
`${DRY_RUN ? "with embedded annotations" : "imported"}: ${nDone}, ` +
`without: ${nNone}, highlights recolored: ${nRecolored}, ` +
`items retagged "${TAG}" -> "${TAG_DONE}": ${nRetagged}, errors: ${nErrors}`);
return log.join("\n");