Shell Lexing for Node.js

June 19
2018

I published node-shlex, a Node.js package for tokenizing UNIX-like shell commands and performing shell escaping.

var split = require("shlex").split

// returns: [ 'ls', '-al', '/' ]
split('ls -al /')

// returns [ 'rm', '-Rf', '/Volumes/Macintosh HD' ]
split('rm -Rf "/Volumes"/Macintosh\ HD') 

I began by porting the shlex module from the Python Standard Library, originally contributed by Eric S. Raymond. But I found the code to be incomprehensible, with a bewildering state machine and a complex matrix of modes of operation. My rewrite of the main tokenization routine is 45% fewer lines of code, including whitespace and a few comments, but still matches the Python module’s output on over 60 tests.

As an alternative, there is shell-quote, which gets 1.1 million weekly downloads and is used by over 350 other packages. But it seems to be abandoned with several known bugs, and has a half-baked implementation of environment variable interpolation that might lead to unexpected behavior.