Showing posts with label javascript. Show all posts
Showing posts with label javascript. Show all posts

Monday, April 16, 2012

Demystifying JSCrush

Some of you may have seen Philip Buchanan’s award winning. Autumn Evening entry for JS1K 2012 Love. While skimming the source I saw that he had used Aivo Paas’s JSCrush to compress the code. The JSCrush website is intriguing with the source code minified and then passed through JSCrush itself (so it can be submitted to JS1K). The JSCrush version used in the page, in <script> tags is the JSCrushed version. As a weekend project I tried to ‘reverse engineer’ JSCrush and understand it. It took me about 4 hours. What follows is a walk-through of the process.
JSCrush is a very interesting JavaScript program, liberally abusing eval(), global variables and insane levels of nesting to achieve a sort of compression.
Remember that your browser’s web development tools are indispensible for activities like this. I made extensive use of Firebug.

The de-obfuscation

I chose to start with the compressed version in the <script> tag rather than the plain text in the upper text field.
The syntax is clearly wrong for JavaScript, and it is all stuck in a string assigned to _. The part at the end is interesting though (properly formatted).
For every character in $ it is splitting _ on the character, using with to make the resulting array the scope. Then joining the pieces using the last piece and reassigning to _. For example:
_ = "HelloRWorldRCrushedRAB"
$='R'

var temp = _.split($) // => temp = ['Hello', 'World', 'Crushed', 'AB']
var last = temp.pop() // => last = 'AB',
                      //    temp = ['Hello', 'World', 'Crushed']
_ = temp.join(last)   // => _ = 'HelloABWorldABCrushed'
Remember this step, it is key to how JSCrush works. These steps are repeated for every character in $, after which the ‘decompressed’ output is the minified source code:
You can see this for yourself by putting a console.log(_) just before the eval(_).
Now we’ve a fair idea of how JSCrush is doing decompression. Compressed scripts are stored in _, decompressed using the loop and then executed using eval().
The next thing I did was to un-minify the source (manually):
One change I’ve made is the call to setTimeout(). I’ve converted it to a function to make it easier to read, and directly used the script tags innerHTML, since I had the decompressed source in the tag. The JSCrush code generates the textareas and button as part of it’s run and assumes body.children[9] to be the <script> tag with the JSCrush compressed source. Hence it replaces the eval call with the program source itself so that the inner eval() call in setTimeout() extracts the decompressed source and puts it as the value of the first textarea. It then calls L(), the JSCrush crushing function to compress the original code back, so that you get the compressed version of JSCrush in the lower text field. Mind-boggling.
The setTimeout() without a time simply causes the code to be executed after the script has finished evaluating completely.

Understanding

Now that we’ve decompressed code, it still has the scars of minification – single letter variable names and no comments. Time to start reading the code. Line 1 is just setting up the HTML for the user. Lines 2-4 is the first interesting piece. The array Q is being populated with all the ASCII characters, in reverse order! The characters \n, \r, \\, ' and " are excluded, as are \0 and DEL, so that Q has 121 characters. Rather than using a readable if statement, Aivo is using the fact that && is ‘short-circuiting’ in JavaScript. Much space saving here.
Next we come to the definition of L. Line 12 just removes blank lines, whitespace and single line comments (except those following code). It also escapes backslashes so that the code is ready to be put into a string later. This is assigned to the letters i and s. Be warned from here, in the goal of smaller size, variables frequently change their meaning to promote reuse. s is always going to point to the code, but i is used as a counter all over the place.
Next, B is half the length of the program, m is the empty string. Line 15 is where it starts getting interesting. The pattern:
encodeURI(string).replace(/%../g, 'i')
occurs thrice in the code. Its task is to get the byte length of the string rather than the number of characters that string.length gives. In ASCII there is no difference, but if there are Unicode characters, they may occupy 2-4 bytes. encodeURI will replace each byte with a ‘%xx’ code with xx being the hexadecimal byte value. Replacing this with the single letter ‘i’ will get us one ‘i’ for every byte, so that the length of the resulting string is the byte length. This was one of the many clever tricks present in the JSCrush code. They might be well known, but this is the first time I came across it.
The initialization in the for loop is only to save a byte, it does not affect the loop itself in any way. Similarly the m = c + m call can be moved to the end of the loop body. This construct will generate the decompression sequence contained in $. This for loop is then actually an infinite while loop.
Line 43 is again a trade-off of readability for size. Here it is in a cleaner form:
c = 0
i = 121
while (!c && i) {
 if (! (~s.indexOf(Q[i])))
  c = Q[i]
 --i;
}
~ is binary NOT. If Q[i] is not found in the source, then indexOf will return -1, NOT -1 = 0 and !0 === true, so that this code is actually saying:
For every character Qi in Q in reverse:
    If the source does NOT have the character in it:
        c = Qi
Or, c is set to an ASCII character that is not present in the program. Initially it will be ASCII 1, then perhaps 2 and so on. This ‘c’ is now the character that will be used to join the pieces obtained in Lines 20-32. This is one round. When all the characters have been used up, compression stops (Line 18).
Lines 12-32 basically try to find long, repetitive strings that can be replaced with a single character, to get the best compression. JSCrush follows a brute-force approach to find these segments. With single variable names, the code is a mess, so here is a cleaned up version which makes things much clearer:
Lines 9-28 try to find segments which repeat atleast twice in the code. Longer segments will give better compression, so we try all of them. For segments of length 1, we try every character in the string, for segments of length 2, we try every pair and so on. If it repeats we keep track of the segment count.
The segmentLengthBound = longestSegmentLength (B=Z) bit is interesting, and it took me some thinking to figure it out. It relies on the following facts:
  • The longest segment in the current source is longestSegmentLength.
  • Splitting by something, and then joining by a character not in the source will not lead to creation of longer segments.
So we can restrict segmentLength of the next round to segmentLengthBound.
Lines 32-41 choose the best segment to substitute in this round. The expression (R=o[i])*j-R-j-1 may seem cryptic, until you look a little later in the code where the split and join is done, and you remember how JSCrush works. R * j is the number of bytes we will remove by replacing this segment. But to join the split, we’ll need one character for every repetition, followed by one character to separate the segment suffix itself. The conditional asks if this leads to actual, and better, compression than what we already know of. If no such segment was found, we are done compressing. Otherwise we split by the segment, join the pieces by the join character and tack on the segment at the end. One round done!
Once multiple rounds have been done, the script is compressed, and only some trivial things remain. The value of B is now changed to store the quotes (double or single), based on which are fewer. Since the compressed program is stored in a string, using the quotes that appear less times means less ’\’ to substitute, each of which costs a byte. We then prepare the boilerplate, setting _ to the now compressed source, setting $ to the decompression sequence m and adding the evaluation code. The savings accomplished are announced too.
One trick I picked up in the code is forcing a certain digit view precision.
i/S * 100
would give a float percentage with many digits after the floating point. Instead multiplying by 1000, gets us two digits in the integral positions, bitwise OR-ing with 0 casts to an integer, losing the floating point digits, then dividing by 100 gets us the two digits we want.

Summary

JSCrush works by:
  1. Finding the first unused ASCII character to act as the join
  2. Finding the substring of the program text that gives the best space savings if its repetitions are all replaced by the ASCII character from 1.
  3. Splitting the source on 2 and joining the pieces using 1, tacking on 2 to the end. This string replaces the original source.
  4. Repeating 1, 2 and 3 until no more savings are possible or we’re all out of ASCII.
  5. Wrapping the compressed source into a string, then using the list of join ASCII characters to unroll the string.
  6. Unrolling is performed by splitting on every ASCII character used in 3, extracting the original repeated substring 2 from the split and joining the parts.
I hope this (long) post was interesting and educational. If you have feedback, a comment would be great.

Thursday, April 22, 2010

Quick and dirty Instant Messaging with Redis

(Aside: Yes this is a post after a zillion years, but I have a few more lined up, and waiting for some important stuff in about a week)

Redis is a wicked cool NoSQL database, in that storing stuff is not the only thing that it does. Mathias Meyer already has a collection of Redis use cases, but this great idea is mine. Like all good ideas it emerged in the shower :) ( I was not aware of Pieter Noordhuis' MUC when I did this, in either case mine deals with one-to-one IM )

Using the new publish-subscribe commands and a bit of node.js code, here is a tiny instant messaging server. An explanation follows, refer to the code while reading it, Blogger sucks for publishing code.

Here is how it works. For every client, the server maintains two connections to redis. This is because a subscriber is not allowed to invoke other commands. So we have a subscriber connection and publisher connection. Consider two clients are now connected, Alice and Bob.

When Alice connects ( NICK Alice ), we insert her nickname into 'mclarens:inside', a Redis set. This allows us to have a WHO command to list online members. Bob to does the same. To start chatting with Alice, Bob initiates a conversation ( TALKTO Alice ). The node server does none of its own client management. Instead each client just maintains subscriptions to two channels/classes. Each client is always subscribe to '[nick]:info', where it is notified of talk initiations and exits etc. When Bob wants to talk to Alice, he sends a 'start Bob' to Alice's info channel, 'Alice:info'. Then both Alice and Bob subscribe to the class 'Alice:Bob', using alphabetical order to decide the name. When either of them wants to talk to the other, they do MSG [nick] [message]. The connection uses the publish redis connection to send a message to the channel, resulting in both sides being notified of the incoming message.

To terminate the chat, one side just has to send a 'STOP [nick]'. That unsubscribes the user from the class so that he no longer receives messages. It also sends a 'stop [nick]' to the other side, so that he/she can also unsubscribe.

On QUIT, we simply remove ourselves from the mclarens:inside set, unsubscription is handled automatically by Redis!

That's it, simple Instant Messaging! Now this lacks any kind of security and ignore lists and status but proves the point. In fact I'm thinking of using this as the backend for the IM part of the XMPP server I'm hacking on. At this point it is in no shape to have this feature just yet, but it should some day.

(Thanks to roidrage and tnm on #redis for pointing out that I had to use two Redis connections for PubSub.)

Sunday, December 20, 2009

Trending


Twitter usually syncs with the (developed) world pretty fast in hashtags. But I'm surprised Copenhagen is not on it.

To add some semblance of code: I've been mucking with node.js and some of the effect is simple TagLib bindings with that being used for something more important soon. Much improved ( or rather simplified ) kwin-tiling too with a few regressions like moving and resizing.

Friday, April 13, 2007

Rounded Corners

Rounded corners have become a staple of web 2.0. So I decided to write my own version. So whats special in this among all the others. Well mine is relatively customisable pertaining to the elements to round. Further it allows elements to have variable width and scales on resizing. The images used use the mountain top technique so that the same image is required, whatever the background color of the elements. Of course if you have a non-white background you will have to edit the images. To check them out see Rounded Corners on 22bits.

Sunday, March 18, 2007

Introducing Pixelframe

In Trac(k)ing I mentioned the "secret project" I was working on. Well I have decided its time to let the world know what it is, even though its not finished yet.

Its called Pixelframe.

Pixelframe is meant to be a lightweight image gallery. It uses PHP for the backend and my custom javascript toolkit ( called Juice ) for frontend effects and ajax requests. Juice will also be released seperately once it is done. Pixelframe is easy to setup and use and doesn't have ( and won't have ) a lot of features. It is purposely made to be featureless. For description and development status check the Pixelframe page on Google code .

For now those interested can check out the latest bug free build which has the backend almost implemented. --> Demo

Friday, March 02, 2007

Learning the hard way

I have come acrooss some rare occasions in programming in which the language/environment/common sense really wants you to learn after literally breaking you head. Thats what happened today in a javascript app I am doing (Shh! Its still a secret). For 2 days I have been trying to use everything I know to clean up a problem I was having with every node firing the event when only one should've been. That was until I realised it was due to this small error hiding in the corner.

window.addEventListener instead of this.addEventListener

So note to self, PAY ATTENTION

Sunday, November 05, 2006

Learning

I have decided to take a break from creating software to learning some new things for a few days. So I have started learning Ruby, advanced JavaScript which I didn't know before, especially DOM scripting and Ajax. And I am also reading Beginning Algorithms which is a really good book for an introduction to algorithms in a easy to understand concrete way. That said I am starting to like Ruby and feel the same enjoyment that is felt when coding in Python. I have not entirely abandoned projects though. They are on the drawing board and in the design process.
School begins from tomorrow after three weeks of Diwali vacations and we have lots of exams coming up which will severely curtail my computer time. I will be freed only in the second week of December when preparations for the school Sports Day begins and studies take a backseat.
This is the last day of enjoyment so I go of to enjoy...