Introducing jsctags

On the Developer Tools team, we're always interested in making development of web apps easier, no matter what editor or browser you're using. To that end, I'm happy to introduce a new code indexing solution for JavaScript designed to make development of large JavaScript applications a little simpler and faster. It'll be the foundation of the code completion features in Bespin, but since it's based on the venerable ctags format, you can use it now in many editors. The project's known as jsctags, and it's located here on GitHub. It requires only the node.js framework, which was selected because it's well-known and easy to set up.

ctags has become the de facto standard for simple code indexing tasks. Editors that support ctags, such as Vim and TextMate (with the TmCodeBrowser plugin), can read ctags-compatible tags files to allow you to quickly find information about symbols. You can jump to the location where a symbol is defined, perform autocompletion against names defined in the tags file, and get module and type information for any listed symbol. For example, here's a screenshot of Vim using the output of jsctags to inspect the attr function from jQuery:



The obvious question is: Hasn't this been done in JavaScript before? After all, languages like C++, Java, and Python have featured good support for ctags for years now. It's true, in fact, that the Exuberant Ctags tool features a JavaScript language module already. The difficulty, however, is that JavaScript, especially modern JavaScript, is hard. The traditional pure parser-based methods of tag extraction simply don't work very well for the kind of JavaScript people write today. Here's a comparison of the number of tags (roughly, the number of exported functions and variables) found in some popular JavaScript libraries and plugins by jsctags and Exuberant Ctags 5.8:

Library/Plugin	Exuberant Ctags	jsctags
jQuery 1.4.2	1	1219
jsTree 0.9.9a	52	72
Prototype 1.6.1	18	389
script.aculo.us 1.8.3	3	234
MooTools 1.2.4	170	192

Moreover, jsctags has more useful output for developers: it knows about CommonJS modules and the CommonJS packaging standard, and it can automatically detect the module system in use.

How does all this work? Briefly, what's going on behind the scenes in jsctags is a very simple form of what's known as abstract interpretation. jsctags actually interprets the scripts sent to it using an interpreter loosely based on Narcissus. Its interpretation is very simple and restricted: no loops or conditionals are executed, no external functions are callable by the script, unreferenced variables spring into existence instead of throwing ReferenceErrors, and so on. jsctags isn't a full-blown JavaScript interpreter by any means: all it's concerned about is how definitions make it to the window and/or exports object.

There will be more in store for jsctags in the future! Dimitris Vardoulakis is beefing up the parser to support Mozilla-specific extensions and is investigating the possibility of adding type inference. In the near future, we'll be using jsctags in Bespin to provide code completion.

Feel free to leave suggestions in the comments and fork the project on GitHub. Feedback is much appreciated!