Thanks adam. I'm still having no joy. Trying to bootstrap a basic display of collections:
function doExport() {
var collections = []; while (collection = Zotero.nextCollection()) { // First grab all the collections collections.push(collection); } Z.debug(collections); Z.debug(JSON.stringify(collections)); Z.write(JSON.stringify(collections)); }
but only getting this in the debug window:
(3)(+0000000): []
(3)(+0000001): Translate: []
(i.e. the empty array `collections`)
What am I doing wrong?
Edit: I do have `getCollections` set to true in `configOptions`
Looks right to me. The things I'd do to troubleshoot is 1) put a debug statement into the while loop and 2) test with the Zotero RDF translator, which has nextCollection() implemented
The issue seems to be that Zotero.nextCollection() simply isn't iterating. Nothing I put within the while() loop is executed. The result of Z.debug(Z.nextCollection()) is (3)(+0000001): ===>false<=== (boolean).
I noticed this error in the javascript error console:
I've been playing with the Zotero RDF translator, and it seems Zotero.nextCollection() only has output when exporting (i) a collection, that (ii) has child collections. In all other cases I see the same behaviour as above.
Is it possible, when exporting just 1 or more _items_, to access the collection(s) they reside in? All I want to do is include the names of those collections in the export.
It seems I forgot that I already asked about this on zotero-dev last year -- apologies for repeating myself >.<
It seems it's not possible, because Zotero.nextCollection() treats the exported collection as the root, which is thus inaccessible. And getting any information about collections when exporting item(s) is completely impossible (that's what I was trying above).
Is there ever likely to be greater access within the translator sandbox to collection information? It would be extremely useful for any export translator that seeks to include arbitrary collection info (which is surely a common consideration).
Yes and yes; the first one is simply already available to extensions, the second one can be achieved but involves more effort and a lot of care. To do the second you're going to have to apply run time patches, and those can very easily destabilize Zotero. Given how nearly everything is async now, race conditions are something to look out for, too.
Do you think that's the easiest approach to 'break out' of the translator sandbox? Given that all I'm trying to achieve is to include the names of collections in an export, maybe an addon is overkill?
What you describe cannot be done without an extension. And breaking out of the sandbox is (as far as I know, and I've experimented with a lot of things in this domain) the only way to do this.
That said, can we first talk about what you want to achieve? Breaking out of the sandbox opens an avenue for hard-to-debug problems that will land in the lap of the Zotero devs first -- speaking from experience here.
I'm creating a JSON translator that formats items according to the schema for Roam (a note-taking application). It's pretty simple, except that I need the collection names to make it really powerful for categorisation by topic.
I think I can work around the sandbox by doing an API call and getting the collection from the library online.
I've been experimenting with Zotero.Utilities.doGet(), which works to communicate with the Zotero servers, but I can't seem to access the response outside of the call. For example, this doesn't work:
function doExport() { // Just an experiment var response, url = "https://api.zotero.org/users/[user id]/collections/[collection key]?v=3&key=[API key]"; Zotero.Utilities.doGet(url, function(text){ response = text; Z.debug(text); // this works }); var collectionTitle = response.data.name; // this is empty }
I'm sure it's some noob issue with scoping and/or async... my JS is pretty rusty. But if I can get this to work it will be the last piece of the my translator puzzle :)
You can't do async calls from an export translator, but that code wouldn't have worked in any case -- var collectionTitle = response.data.name; will always execute before response = text; even in places where you can do async. The reasons for that would take us fairly deep into the JS language design. Anyhow, to get this kind of thing to work, you need to either nest callbacks, chain promises, or (preferably IMO) use async/await, you can't do it with the pattern you have in that code. But none of that will work in export translators.
But if this is what you want, why bother with a translator? If you're going to write a plugin anyhow, you can just generate the JSON file from the plugin itself. No restrictions there.
@emilianoeheyns thanks, yes, I had a feeling it might ultimately not be possible -- I was playing around with Promises etc but couldn't make anything work :(
I was hoping I could solve that and avoid having to build a full addon, since the learning curve is so much steeper even to begin.
Can you suggest where I might start with that? I found this thread on zotero-dev (which you contributed to): https://groups.google.com/d/topic/zotero-dev/wLZdrPiaKeA/discussion. It seems there are some important differences between 'bootstrapped' extensions and XUL-based extensions, plus the documentation is out of date.
I've tried* your generator-zotero-plugin node package, but my impression is we should be using bootstrapped extensions instead of XUL?
Apologies for the noob questions, but can you give me any initial pointers on where to begin? It'd be very much appreciated :)
*but couldn't get it to work -- got `Couldn't find username` errors :(
If you just export the entire library, you'll get all collection info, I don't know if that would work for you. It'd certainly be simpler for you.
WRT getting started with a plugin, there's also https://www.zotero.org/support/dev/sample_plugin. That is probably a simpler base if your plugin doesn't do much. My generator has the advantage that you can include libraries from npmjs, and that it uses typescript, a superset of javascript with typechecking (well I consider that a benefit anyway) at the cost of more complexity in the toolchain. And that it has plugin releases/updates integrated. To mimick what a translator does, you'll want to use Zotero.getActiveZoteroPane().getSelectedItems() or Zotero.getActiveZoteroPane().getSelectedCollection() to get your selection, await Zotero.Items.getAsync() to fetch the items, and call Zotero.Utilities.Internal.itemToExportFormat(item) to transform them to the json objects you get in the translators. await Zotero.File.putContentsAsync will let you write text to a file. But you're going to have to get comfortable reading through the Zotero codebase to figure out how to do stuff.
generator-zotero-plugin will try to pick up your github username but if it can't find your username that way it should complain but should leave it set to TOCHANGE in package.json -- there was an error in the generator that is fixed in 0.0.31.
WRT bootstrapped vs XUL, bootstrapped is a better experience for the end-user as you don't need a restart for install/upgrade/uninstall. I find XUL to be less effort for me to build UIs with, and I really don't like UI work, so anything that has me doing less of it gets my vote. Both bootstrapped and XUL extensions are deprecated technology so I don't see any benefit of one over the other, other than the UI/restart issue. When Zotero moves to Electron, absolutely all extensions will have to be partially rebuilt towards the plugin architecture that will emerge.
Re the translator, exporting the whole library is indeed an option but the usability tradeoff is pretty large, and given the central importance of collection data to its usefulness in Roam I think it's better to go the addon route (unfortunately!).
That's amazing info regarding addon development, thank you -- you've saved me many hours, even just by specifying the right methods :) I'll start with the hello world addon and go from there. I looks like I might be able to re-use much of what I've written :)
Both bootstrapped and XUL extensions are deprecated technology so I don't see any benefit of one over the other, other than the UI/restart issue.
Bootstrapped extensions, with HTML UIs, will certainly more closely resemble whatever plugin system emerges later. Even as long as we're on Mozilla, I would strongly recommend trying to avoid writing any new UI in XUL, since XUL may go away for us before the rest of the Mozilla architecture goes away.
Thanks @dstillman -- is there a good, basic hello world for a bootstrapped extension? You mentioned on zotero-dev that Zutilo is bootstrapped but I wonder if there's something simpler to work from?
@emilianoeheyns that's one way of using it, but it's got a lot of computational stuff up the hood that is immensely powerful for "building a second brain". For more see #roamcult on twitter, or e.g. roambrain.com. I think it has the potential to revolutionise academic workflows (well mine, anyway) which is why I'm so excited about it - and hence the connection to Zotero.
If you're interested I can send you an invite link :)
It's browser-only for now, although the offline support there is pretty good. They are working on an API and apps will likely come, but if you browse the #roamcult tag you'll see it doesn't have the same ethos as most startups...
Can you point me to an email address? I've been asked not to make the invite link public.
emiliano.heyns@iris-advies.com. APIs might be interesting -- always looking to connect Zotero to more plaintext-editing environments, but offline is less of a concern to me than data ownership; offline doesn't mean data portability. Data portability is a must.
Not sure what I should think of people self-labeling as cult members. It seems to me that cult members are ill placed to offer balanced advice about their object of obsession.
Hehe that's part of its unique 'charm'. People are excited about it and for whatever reason that hashtag took hold -- I think it was intended ironically in the first instance. Given how bad the interface is, and the lack of polished website, beautiful on-boarding copy etc etc, the fact it has funding etc is testament to the value of its fundamental product (all of those things are coming, but the founders are focusing on core functionality at the moment).
In terms of portability, everything is exportable as JSON and markdown, and the creators have vowed (FWIW) never to lock-in or anything like that. As far as I can tell, and I've been watching for a while, there's no intention of baiting a buy-out from GAFAM or similar; the founders seem genuinely invested in the philosophy of the system and aren't out just to make money. But time will of course tell. The privacy policy is here, and explicitly states that note data won't be used for advertising. Indeed, I believe the plan is to charge quite a high monthly amount (~$15), with the intention of using subscriptions as the revenue source rather than data (a refreshing idea in 2020).
Open source alternatives are available that implement the core feature (bidirectional linking), but there's a lot more under-the-hood in Roam than just that, as you'll see if you explore it a bit.
Anyway, I'm sounding like a cult member myself, so I'll shut up and let you decide for yourself. Email with link is on its way!
Upgrade Storage
function doExport() {
var collections = [];
while (collection = Zotero.nextCollection()) { // First grab all the collections
collections.push(collection);
}
Z.debug(collections);
Z.debug(JSON.stringify(collections));
Z.write(JSON.stringify(collections));
}
but only getting this in the debug window:
(3)(+0000000): []
(3)(+0000001): Translate: []
(i.e. the empty array `collections`)
What am I doing wrong?
Edit: I do have `getCollections` set to true in `configOptions`
Does your translator have the collections option in the header?1) put a debug statement into the while loop and
2) test with the Zotero RDF translator, which has nextCollection() implemented
{"translatorID": "284f527b-c47d-4e65-b6ff-1af31d5f9fb1",
"label": "JSON testing",
"creator": "Laurence D",
"target": "json",
"minVersion": "5.0",
"maxVersion": "",
"priority": 25,
"configOptions": {
"getCollections": "true"
},
"inRepository": false,
"translatorType": 2,
"lastUpdated": "2020-05-21 00:08:00"
}
function doExport() {
var collection, collections = [];
while (collection = Zotero.nextCollection()) {
Zotero.debug(collection);
collection.push(collection);
}
Zotero.write(JSON.stringify(collections, null, "\t"));
}
The issue seems to be that Zotero.nextCollection() simply isn't iterating. Nothing I put within the while() loop is executed. The result of Z.debug(Z.nextCollection()) is
(3)(+0000001): ===>false<=== (boolean).I noticed this error in the javascript error console:
[JavaScript Warning: "unreachable code after return statement" {file: "resource://zotero/loader.jsm -> resource://zotero/bluebird/utils.js" line: 201 column 4 source: " eval(obj);"}]but it appears on loading Zotero, not on running the translator, so I assumed it's just a quasi-bug within the main application.
Is it possible, when exporting just 1 or more _items_, to access the collection(s) they reside in? All I want to do is include the names of those collections in the export.
It seems it's not possible, because Zotero.nextCollection() treats the exported collection as the root, which is thus inaccessible. And getting any information about collections when exporting item(s) is completely impossible (that's what I was trying above).
Is there ever likely to be greater access within the translator sandbox to collection information? It would be extremely useful for any export translator that seeks to include arbitrary collection info (which is surely a common consideration).
A couple of questions:
Do you think that's the easiest approach to 'break out' of the translator sandbox? Given that all I'm trying to achieve is to include the names of collections in an export, maybe an addon is overkill?
That said, can we first talk about what you want to achieve? Breaking out of the sandbox opens an avenue for hard-to-debug problems that will land in the lap of the Zotero devs first -- speaking from experience here.
I'm creating a JSON translator that formats items according to the schema for Roam (a note-taking application). It's pretty simple, except that I need the collection names to make it really powerful for categorisation by topic.
I think I can work around the sandbox by doing an API call and getting the collection from the library online.
I've been experimenting with
Zotero.Utilities.doGet(), which works to communicate with the Zotero servers, but I can't seem to access the response outside of the call. For example, this doesn't work:function doExport() { // Just an experiment
var response, url = "https://api.zotero.org/users/[user id]/collections/[collection key]?v=3&key=[API key]";
Zotero.Utilities.doGet(url, function(text){
response = text;
Z.debug(text); // this works
});
var collectionTitle = response.data.name; // this is empty
}
I'm sure it's some noob issue with scoping and/or async... my JS is pretty rusty. But if I can get this to work it will be the last piece of the my translator puzzle :)
var collectionTitle = response.data.name;will always execute beforeresponse = text;even in places where you can do async. The reasons for that would take us fairly deep into the JS language design. Anyhow, to get this kind of thing to work, you need to either nest callbacks, chain promises, or (preferably IMO) use async/await, you can't do it with the pattern you have in that code. But none of that will work in export translators.But if this is what you want, why bother with a translator? If you're going to write a plugin anyhow, you can just generate the JSON file from the plugin itself. No restrictions there.
That's quite disappointing since getting collection names is the last piece of the puzzle -- you can see the rest of my code here: https://github.com/melat0nin/translators/blob/master/Roam JSON.js
I was hoping I could solve that and avoid having to build a full addon, since the learning curve is so much steeper even to begin.
Can you suggest where I might start with that? I found this thread on zotero-dev (which you contributed to): https://groups.google.com/d/topic/zotero-dev/wLZdrPiaKeA/discussion. It seems there are some important differences between 'bootstrapped' extensions and XUL-based extensions, plus the documentation is out of date.
I've tried* your
generator-zotero-pluginnode package, but my impression is we should be using bootstrapped extensions instead of XUL?Apologies for the noob questions, but can you give me any initial pointers on where to begin? It'd be very much appreciated :)
*but couldn't get it to work -- got `Couldn't find username` errors :(
WRT getting started with a plugin, there's also https://www.zotero.org/support/dev/sample_plugin. That is probably a simpler base if your plugin doesn't do much. My generator has the advantage that you can include libraries from npmjs, and that it uses typescript, a superset of javascript with typechecking (well I consider that a benefit anyway) at the cost of more complexity in the toolchain. And that it has plugin releases/updates integrated. To mimick what a translator does, you'll want to use
Zotero.getActiveZoteroPane().getSelectedItems()orZotero.getActiveZoteroPane().getSelectedCollection()to get your selection,await Zotero.Items.getAsync()to fetch the items, and callZotero.Utilities.Internal.itemToExportFormat(item)to transform them to the json objects you get in the translators.await Zotero.File.putContentsAsyncwill let you write text to a file. But you're going to have to get comfortable reading through the Zotero codebase to figure out how to do stuff.generator-zotero-plugin will try to pick up your github username but if it can't find your username that way it should complain but should leave it set to TOCHANGE in package.json -- there was an error in the generator that is fixed in 0.0.31.
WRT bootstrapped vs XUL, bootstrapped is a better experience for the end-user as you don't need a restart for install/upgrade/uninstall. I find XUL to be less effort for me to build UIs with, and I really don't like UI work, so anything that has me doing less of it gets my vote. Both bootstrapped and XUL extensions are deprecated technology so I don't see any benefit of one over the other, other than the UI/restart issue. When Zotero moves to Electron, absolutely all extensions will have to be partially rebuilt towards the plugin architecture that will emerge.
Re the translator, exporting the whole library is indeed an option but the usability tradeoff is pretty large, and given the central importance of collection data to its usefulness in Roam I think it's better to go the addon route (unfortunately!).
That's amazing info regarding addon development, thank you -- you've saved me many hours, even just by specifying the right methods :) I'll start with the hello world addon and go from there. I looks like I might be able to re-use much of what I've written :)
If you're interested I can send you an invite link :)
Can you point me to an email address? I've been asked not to make the invite link public.
Not sure what I should think of people self-labeling as cult members. It seems to me that cult members are ill placed to offer balanced advice about their object of obsession.
In terms of portability, everything is exportable as JSON and markdown, and the creators have vowed (FWIW) never to lock-in or anything like that. As far as I can tell, and I've been watching for a while, there's no intention of baiting a buy-out from GAFAM or similar; the founders seem genuinely invested in the philosophy of the system and aren't out just to make money. But time will of course tell. The privacy policy is here, and explicitly states that note data won't be used for advertising. Indeed, I believe the plan is to charge quite a high monthly amount (~$15), with the intention of using subscriptions as the revenue source rather than data (a refreshing idea in 2020).
Open source alternatives are available that implement the core feature (bidirectional linking), but there's a lot more under-the-hood in Roam than just that, as you'll see if you explore it a bit.
Anyway, I'm sounding like a cult member myself, so I'll shut up and let you decide for yourself. Email with link is on its way!