Script Crusher

Heres the problem.. as soon as you start building large applications usings ExtJs your users say two things

• Its slow to load
• Can you translale it into their language

The speed issues are mainly due to

• Loading of many javascript files (With our currently quite small applications, there are still more than 30 single javascript files - one class per file, which can take over  10 seconds to load.)
• The javascript contains comments and whitespace (As I like to make my code readable and maintainable)

To address these issues, there are a few javascript compression tools out there, however, most have rather painful setup requirements, like getting java working, and are not very hacker friendly.

Now that all my interface is embedded into javascript code, including all the button labels etc. l have been wondering for quite a while how I was going to translate the application and keep it up-to-date (in the Flexy template engine we extract the strings from the HTML, and have a tool that manages PO files based on them). Since this did not seem feasible for Javascript, I was begining to dread the thought of finding all the strings and wrapping them with something like a gettext type call.

So it turned out my little dmdscript hacking may have some use...
It took a small amount of hacking to add tokenizing support to dmdscript, by adding a new class for the javascript interface and few minor changes to dmdscript's internal tokenizer I got this little script to work.

   var data = File.read(fname);
   var x = Script.tokenize(data);
	
   for(var i =0; i < x.length; i++) {
       if (x[i].tok == "whitespace") {
           if (x[i].val.indexOf("\n") > -1) {
		print("\n");
		continue;
           }
           print(" ");
           continue;
       }
       print(x[i].val);
    }
    print(";\n");

Basically it reads the file, then creates an array of objects, each one representing a token. (I should perhaps change the token to an integer and use  Script.Token.whitespace as a constant... ).

The script will basically strip all the whitespace and comments, and convert whitespace into either a line break or a space.. (as linebreaks are actually relivant in Javascript).

The next issue was how to deal with language conversion. The code above can be used to test for strings by doing:

for(var i =0; i < x.length; i++) {
    if (x[i].tok == "string") {
       println("got string: " + x[i].val);
    }
}

So I came up with the idea of using the quote style to determine if a string needs translating.
single quotes = does not need translating
double quotes = a translable string.

It was amazingly easy to change my existing code to do this, I could then generate a list of strings for each file, and output a JSON structured file which can be used to store the translations mapping.

#djs ScriptCrusher.js -- --strings /tmp/Hello.js > /tmp/zh_HK.js

Would generate a file:

"Hello.js" : {
    "Hello":       "Hello",
    "World":      "World",
}

Which can be edited and translated.

Then when Crunching the script up, it uses the map to convert strings from one language into another.
eg.

#djs ScriptCrusher.js -- --langfile /tmp/zh_HK.js /tmp/Hello.js \
	> /tmp/Hello.zh_HK.js
#djs ScriptCrusher.js --/tmp/Hello.js > /tmp/Hello.en.js

(I've not done the merging of old/new yet, but it should be pretty trivial).

djs should be pretty simple to build: (requires gdc - available in Debian and Ubuntu + probably others)

#svn co http://www.akbkhome.com/svn/gtkDS
#cd gtkDS
#sh djs.gdc.sh

The full ScriptCrusher is here..
http://www.akbkhome.com/svn/gtkDS/test/ScriptCrusher.js

Happy Hacking..