Tuesday, May 25, 2010

Introducing jsctags

On the Developer Tools team, we're always interested in making development of web apps easier, no matter what editor or browser you're using. To that end, I'm happy to introduce a new code indexing solution for JavaScript designed to make development of large JavaScript applications a little simpler and faster. It'll be the foundation of the code completion features in Bespin, but since it's based on the venerable ctags format, you can use it now in many editors. The project's known as jsctags, and it's located here on GitHub. It requires only the node.js framework, which was selected because it's well-known and easy to set up.

ctags has become the de facto standard for simple code indexing tasks. Editors that support ctags, such as Vim and TextMate (with the TmCodeBrowser plugin), can read ctags-compatible tags files to allow you to quickly find information about symbols. You can jump to the location where a symbol is defined, perform autocompletion against names defined in the tags file, and get module and type information for any listed symbol. For example, here's a screenshot of Vim using the output of jsctags to inspect the attr function from jQuery:



The obvious question is: Hasn't this been done in JavaScript before? After all, languages like C++, Java, and Python have featured good support for ctags for years now. It's true, in fact, that the Exuberant Ctags tool features a JavaScript language module already. The difficulty, however, is that JavaScript, especially modern JavaScript, is hard. The traditional pure parser-based methods of tag extraction simply don't work very well for the kind of JavaScript people write today. Here's a comparison of the number of tags (roughly, the number of exported functions and variables) found in some popular JavaScript libraries and plugins by jsctags and Exuberant Ctags 5.8:







Library/PluginExuberant Ctagsjsctags
jQuery 1.4.211219
jsTree 0.9.9a5272
Prototype 1.6.118389
script.aculo.us 1.8.33234
MooTools 1.2.4170192


Moreover, jsctags has more useful output for developers: it knows about CommonJS modules and the CommonJS packaging standard, and it can automatically detect the module system in use.

How does all this work? Briefly, what's going on behind the scenes in jsctags is a very simple form of what's known as abstract interpretation. jsctags actually interprets the scripts sent to it using an interpreter loosely based on Narcissus. Its interpretation is very simple and restricted: no loops or conditionals are executed, no external functions are callable by the script, unreferenced variables spring into existence instead of throwing ReferenceErrors, and so on. jsctags isn't a full-blown JavaScript interpreter by any means: all it's concerned about is how definitions make it to the window and/or exports object.

There will be more in store for jsctags in the future! Dimitris Vardoulakis is beefing up the parser to support Mozilla-specific extensions and is investigating the possibility of adding type inference. In the near future, we'll be using jsctags in Bespin to provide code completion.

Feel free to leave suggestions in the comments and fork the project on GitHub. Feedback is much appreciated!

6 comments:

Andy L said...

Support for CoffeeScript would be great.

jason said...

Thanks for this!

Are there plans to add output options like ctags has? I hooked jsctags to TmCodeBrowser and it doesn't seem to like it. I checked and the output formats are a little different, and I don't think TmCodeBrowser can actually read jsctags' output.

Patrick Walton said...

Hi Jason,

Thanks for your report! I've created an issue on GitHub.

Anonymous said...

I'm getting the following error, does anyone know the problem?

/usr/local/lib/jsctags/traits.js:191
var required = freeze({ toString: function() { return ''; }
^
TypeError: Cannot call method 'isConfigurable' of undefined
at freeze (native)
at /usr/local/lib/jsctags/traits.js:191:18
at Object. (/usr/local/lib/jsctags/traits.js:596:2)
at Module._compile (node.js:458:23)
at Module._loadScriptSync (node.js:465:10)
at Module.loadSync (node.js:334:12)
at loadModule (node.js:279:14)
at require (node.js:407:14)
at Object. (/usr/local/lib/jsctags/ctags/reader.js:39:13)
at Module._compile (node.js:458:23)

Nathan L Smith said...

@anonymous: I just opened an issue for this before I scrolled down to see your comment: http://github.com/pcwalton/jsctags/issues#issue/9

Anonymous said...

How long should it take to run jsctags on jquery?
I've been waiting for over 13 minutes and it's not done yet. (new macbook pro, node.js as of Sat Mar 19, 2011).

It seems to work with jquerymobile within a few seconds, but I only get a tags file with about 10 lines in it...