Antlr is a good tool but it’s missing a critical component. People want to parse input, modify the parse tree, then output the new tree. In other words, they want a refactoring system. But, Antlr is a parser generator, not a transformation system.

This is why I wrote Trash. It has tools to insert, delete, replace, and move nodes of a parse tree. But, these primitives are not enough.

I just added an important tool that was missing from the tool kit: trpiggy. “Piggy” is the next step in tree rewriting for Trash. Although it is only a beginning, the tool implements an engine of sorts for tree rewriting. If you are familiar with XSLT, this should look familiar. Input to Piggy is a simplified XSLT template and a parse tree. It outputs a modified tree, which can then be printed. A Piggy spec is a list of xpath -> {{ ... }} ; rules that rewrite parse tree nodes. Piggy rewrites not only the tree, but the simulated token and char streams that were used while creating the tree.

Example

There was a recent question in StackOverflow to read declarations in Lua and convert some into JSON. This can be done via an Antlr parser and a tree walker, but it is very combersome. A much better approach is with Piggy, where you can think of each “visitor method” as a template for a replacement tree node. To illustrate what I mean, consider this Piggy solution for the StackOverflow question.

Lua Input:

local   tbl = {
   SomeObject = {
      Key = "Value",
      AnotherKey = {
         Key1 = "Value 1",
         Key2 = "Value 2",
         Key3 = "Value 3",
      }
   },
   AnotherObject = {
      Key = "Value",
      AnotherKey = {
         Key1 = "Value 1",
         Key2 = "Value 2",
         Key3 = "Value 3",
      }
   }
}

Piggy template input “templates.txt”:

//chunk -> {{<block>}} ;
//block -> {{<stat>}} ;
//stat[attnamelist and explist] -> {{<explist>}} ;
//explist -> {{ {<exp>} }} ;
//exp[position()=1 and tableconstructor] -> {{<tableconstructor>}} ;
//exp[position()>1 and tableconstructor] -> {{, <tableconstructor>}} ;
//fieldlist -> {{<field>}} ;
//field[position()>1] -> {{, "<NAME>" : <exp> }} ;
//field[position()=1] -> {{ "<NAME>" : <exp> }} ;

Command to parse, modify tree, and output:

trparse input.txt | trpiggy templates.txt | trprint

Output:

 {{ "SomeObject " : { "Key " : "Value"  , "AnotherKey " : { "Key1 " : "Value 1"  , "Key2 " : "Value 2" , "Key3 " : "Value 3"  } } , "AnotherObject " : { "Key " : "Value"  , "AnotherKey " : { "Key1 " : "Value 1"  , "Key2 " : "Value 2" , "Key3 " : "Value 3"  } } }}

Piggy template spec is an order list of pattern. Each pattern contains an XPath expression to note what nodes are to be rewritten and the rewrite expression. Currently, the rewrite expression contains off-channel text and nodes to insert as children in the replaced tree node.

The “engine” for tree rewriting is very primitive. And, the templates are really not that sophisticated. But, I plan to insert a real template engine to handle creation of tokens, tree nodes, and token stream text, as well as some meta to perform the usual template operations like control flow, function calls, etc. The idea is to pick up one of the existing rewrite engines and adapt it here.

Possible template engines to use:

–Ken