Apache Cassandra: Iterate over all columns in a row

Recently I have been using Cassandra for one of my projects, and one of the needs is to iterate over all columns of a row. Each column represents an individual data, of type identified by row id, and keeps changing. So I can’t simply use a set of known column names. Using the setRange call on a SliceQuery and setting a large count is also not an option, since Cassandra will try to load the entire set of columns into memory. Instead I’ve written this iterator which takes a query on which row key and column family has been set, and will load columns as they are requested. By default it loads a 100 columns at a time. You could make it take the count as a parameter and all, but this works for me for now.

The one ‘problem’ with this is the removal of the last column to ensure that there are no duplicates, but still having a start point for the next query. This is because each column is independent, so you cannot ask a column who it’s next neighbour is and start the next query from there. If anybody has a tip to make it more elegant, I’d love to hear it.

foss.in 2010

The 10th and sadly last foss.in took place this week in Bangalore. I was there for all three days of the conference and it was great. This time around we were 6 students from my college. In addition, at the last minute a lot of KDE developers did turn up, including Vishesh Handa, Pradeepto, Abhishek and Sujith.

Danese Cooper’s opening talk about the architecture of Wikipedia was inspiring. That one of the top 5 websites in the world, is powered by just 450 servers and 60 people is an inspiring tale of open source skill and passion. Danese was here to consider a possible server location in India to serve Asia. Wikimedia was also kind enough to give t-shirts as part of the foss.in delegate kit. The calendar in the delegate kit also has stunning wildlife photography.

foss.in lunches are always delicious so that came before any talk. With Nokia being one of the sponsors, Meego was being pushed heavily with 2 days of talks and workshops. Nokia is already pushing QML as a… but I’m getting ahead of myself.

So yes, on day 1, Aditya Bhatt had a talk on libface which by the way, will identify anything as a face. Rahul Sundaram’s talk about Fedora’s early mistakes was good learning for other FOSS projects. But I was distracted by this time by the announcement that Nokia was giving away two or three N900s!

To those who don’t know, the N900 is a hand held computer powered by Maemo, with complete root access and lots of developer friendly features. At the same time it gives any smart-phone the run for its money. Nokia has been agressively pushing the N900 to developers so that the Meego platform can start getting application support. Since Qt applications run directly on Meego, KDE developers are a particularly strong N900 market. Now it was my chance to get one! The catch was to make a QML/Qt based program and submit it on the third day of the conference.

I’ve had QML in my sights for quite some time, but never took out the time to actually try it out. Here was an opportunity. So I made Solarwolf in pure QML and JavaScript. QML’s state concept is very interesting and makes sense if you’ve read Computation and State-machines. Of course there are certain areas where it was much easier to write imperative code in JS. The code itself was written sprint style and is not the neatest or most refactored. What QML really needs is something similar to inheritance so that certain common properties (for example all Sprites have movement deltas) can be factored out. In addition it is odd that all Qt properties and classes have not been exposed in QML’s scripting environment. Things like QRect::intersects() would have been very useful.

Day 2 had Abhishek Patil’s talk about scripting Kate which was insightful and shows what an awesome piece of software Kate is. Anand Mitra’s talk about ZFS was the best technical talk in my two years at foss.in. He clearly had a good understanding of ZFS (seeing that I had nothing) and made everyone aware of the challenges of porting such core programs like filesystems. Leveraging the Git Object Store by Ramkumar Ramachandra was a good way to know how dumb git is. After that I skipped all the talks to hack on Solarwolf, so I’m not aware of the others. Fahrenheit did do a good show however.

By that night my game was pretty much done, it was working on Dinesh’s N900 and so with a bit of polish the next morning I was ready for submission. But with that, and talking to a few people about various things I missed all the talks, including CouchDB and QtWebkit’s graphics layer.

Aanjhan Ranganathan’s “A Hacker’s Apology” was a brilliant end to the final foss.in. The point about preaching Hacker Culture, and not politics or philosophy is the most important thing today. Encouraging everyone in the world to be curious about their surroundings and always asking why should be encouraged, rather than fighting about whether its Linux or GNU/Linux :). The slides aren’t up yet, but everyone should see them when they are available.

Following some moments of nostalgia by Atul Chitnis, The Raghu Dixit Project brought the house down with their superb music!

I won!

That’s right. I have a brand new N900. With a high-quality Bluetooth headset too. This chronologically goes in on the last day at around 5:00 pm, but I thought it deserved a seperate section. It’s a great phone except for the lack of STK support. Some of the little UI touches and shortcuts are especially heartening. They lend true geekiness to the phone. The camera is more than good enough for me, it plays virtually everything I throw at it, and it has inbuilt UPnP support! Now its time to play Angry Birds.

---

Me, Aditya, Dinesh, Mohit, Viranch and Nirmal.

Posted via email from nikhil's posterous

Custom hash objects in Python

It's quite common to use strings, integers and other 'native' Python data types as hash keys. But sometimes it is much easier to be able to use your own class instances as keys. Python's magic methods allow you to do this.

Note: This is not a tip on implementing hash functions, this is how you can remove a certain layer of peeking around into objects

__hash and __cmp__

Consider a useless HTML parser with a simple node design where you want to associate the node name with its attributes.

You want to use the absolute node name as a unique hash.

The solution is to define custom implementations for __hash__ and __cmp__, two magic methods. For more information and constraints about them take a look at the Python docs.

The builtin functions hash(obj) and cmp(obj1, obj2) will attempt to call there __underscored__ counterparts on objs.

import UserDict # allow NodeAttrs to behave like a dictionary, not significant for this example

class NodeName(object):
  def __init__(self, name, parent=None):
    self.name = name
    self.parent = parent

  def __str__(self):
    return (parent and str(parent) or '') + self.name

  def __hash__(self):
    # the hash of our string is our unique hash
    return hash(str(self))

  def __cmp__(self, other):
    # similarly the strings are good for comparisons
    return cmp(str(self), str(other))

class NodeAttrs(UserDict.UserDict):
  def __init__(self, attrs={}):
    self.update(attrs)

Where we assume that the parser is doing the heavy lifting of parsing the name, and putting the attributes in a dictionary. Now to use this in a dictionary you would do the following:

>>> d = {}
>>> d[node_name] = attrs # node_name is an instance of NodeName and attrs

... # do anything which can be done to a dictionary and its keys

Thats it! For more magic methods see Python __Underscore__ Methods

Programming: The invisible wall

In the past few weeks, as I've tried and failed at various projects, there has been a realisation that you can only go so far with random reading and coding.

The first few years of programming are spent coding projects pretty minor on the algorithms/data structures side.

But when you finally start growing out of it and when 'cool' projects for you start to become more complex, a seemingly invisible wall smacks you. The wall seperates amatuer and scientist. And unless you are/I am prepared to study, that wall isn't gonna give way.

Crossing the wall is going to require some serious studying, where I actually try to solve the exercises at the back of Algorithms rather than just read the book. At college I'll probably be able to get my hands on a hard copy.

And once this wall is broken, there will be more walls to break.

If Programming Languages were Lego...

• Assembly would be a huge set of 1x1 blocks. You could do everything other blocks could do, and more, but it took a lot of time and planning to put them all together the right way.
• C would be 1x1 with a few 2x1 and 4x1 thrown in for convenience.
• C++ would be just like C, but there would be all these new highly specific pieces which would only be used by a few people. Of course, there would be a large collection of prebuilt objects like roads, and doors and windows, and roofs that you could put together.
• Java would be a set of Duplo. Easy to learn, but bulky, crappy, and basically needing about 2 layers of blocks below your own to be able to do anything useful. And you would need all kinds of wrapping objects to reach to a particular height, because Duplos are twice the height
• Python would be a mix of Legos and Technic, easy to read, yet able to do all kinds of complex cool stuff you wanted too. But each part would be made out of silicone, to be moulded to almost any kind of arrangement.
• Ruby would be like Python, but with a added Framework and a community with their own Religion.
• PHP would be what happens when different Legos end up mixing on the floor. You would have thousands of pieces, but gluing them together would be difficult, because you wouldn't know what they were called.
• Factor and other stack based languages, would be the dark horses. Lego bricks with connectors on only one side. Fun to use, but a bit challenging as well.

This is a work in progress, I haven't got all the right comparisons yet.

Why factor is cool

Before beginning, I must admit I haven't messed with Factor for a couple of weeks now, nor do I contribute to it, I'm a Factor n00b. Please contradict peacefully.

Of course there are the usual reasons that it is open source, most of the standard libraries are in Factor itself, and that it has a really cool development environment. But almost every popular language these days has all that stuff.

The real reason I think factor is cool is because it still isn't near complete!

It's small and so the sense of community is much stronger, you can be the first to do things in Factor, when they've already been done elsewhere, and anyone with a decent education and experience could easily grok the theory too.

Small syntax

I would say Factor's basic constructs are even smaller than Python's. Just knowing a few shuffle words and USING:, defining words and quotations, gives you almost everything you need to know.

Everything else is just built on top of this.

Small population

At present Factor users are probably just in 2 digits. And there are no products (that I know of). Which means you could probably restructure the whole language without affecting anyone. This allows Factor to improve by rewrite, rather than by add-on.

Educational

Writing libraries for a new language means doing stuff that most have probably never done. So you have to learn about Unicode, or just traverse strings recursively (?!?) like I had to do in brainfacktor. ( Could it be better? )

Basically by keeping the core small, keeping a single stack and enforcing recursion by not having loops, Factor really forces iterative programmers to go stack based/functional. And that itself is a lot of fun, even though it is hard.

Reading about all the developers coding the standard libraries is also very informative, as is looking through the Factor sources.

And it would be fun to look at the interpreter, though I've never done it.



So that was my attempt at some evangelism :D

Most programmers are atheists

If this survey represents a signifact portion of them. Of course these are the programmers who are involved on the web, not much of all the zombies coding for the sake of it.

Programming: What they don't teach in school

This is my honest opinion, feel happy to flame or contradict me, but be polite.

Much has already been written about how much High School computer science teaching sucks ( High school computer science education ). As a student who has just left high school I just wanted to present my 2 cents on how bad the seperation between RealWorld programming and ClassRoom programming is. A common rebuttal is that there isn't enough time. I agree there isn't, but half-baked knowledge is dangerous. If you teach something, teach it qualitatively, not quantitatively.

The Problems

Tool Power

A programmer is only as good as his tools, and how well he can use them. But schools generally lock down students to a specific ( often crappy ) tool. For example, we've Turbo C++ 3.0 ( 1992 ) on all computers. That is a 16 year old environment, supporting 16 year old standards. That's bad! In the days of Linux and Wikipedia do you really want kids raised on old proprietary standards. The ICSE board is very good atleast in their terms. They say that any Operating System ( why are we still on Windows ) with a latest C++ compiler should be used. It seems our school isn't listening.

If you are on Java, you probably have BlueJ. Now I've nothing against the BlueJ folks, but hiding all the compilation details from the student is denying knowledge. Students ( especially the ones who got into CS just for fun ) are of course happy that they don't muck around with "hard" stuff. But thats not how things work in the RealWorld. You should know how to use the command line, invoke javac and java. And, fricking, you should know that Java files have a .java extension and .class is the compiled byte code.

Another issue is shortcuts, quick search and replace, commenting. Kids don't know any of these things. It's type, click, type. Click Save. Click Compile. And to change something there is always manual replace.

File Management

Typical high school level programs, if well written, are never longer than 2 pages. They are often crappy, in the fact that the logic is repetitive, their is no purpose to the program and that students still don't get it!

So when it's project time, what the instructor gets is a dump of humongous shit, often with all the code in main, bad indentation and no comments. That's because the instructor hasn't taught the kids anything about decoupling programs, and seperating classes in files, and using header files. Often the instructor himself has no idea about build systems.

Bad Core Concepts

One really bad thing I see among my fellow students, is the complete inability to realise that each statement is an individual component. It evaluates to something, and other statements can use that value to do their job. This causes repetitive coding. To illustrate see this code sample.

function isSpam(email) {
 if(email.contains('$$$'))
     return true;
 else
     return false;
}

Students aren't taught that email.contains is returning true or false, and all we care about is true and false. They are never taught to see function documentation, just use it the way the instructor tells them too. This code can be simplified to the following, because email.contains is already doing what we want it to do, why decorate it?


function isSpam(email) {
return email.contains('$$$');
}


Beautiful Code: Of Readability

Bad indenting. Student code is full of bad indenting. Instructor code is full of bad indenting.

Learn to appreciate code people. It's just as much art as painting or poetry. Encourage right indentation, make liberal use of whitespace, clean up your logic, don't nest too much. You should be able to go back to it after a few months and not have to spend time just cleaning it up.

Follow standard conventions for variable names, make sure your method names and parameter names make sense. Use camelCase or _underscores_. Make booleans start with 'is' or simply the condition you're representing. If any variable has more purpose than a loop counter or a position marker, it should have a meaningful name.

Name your files meaningfully, keep it lowercase, and keep all your fake 31337 ness to yourself.

Type for today, Think for tomorrow

It's a matter of thinking for 3 hours, writing 3 minutes worth of elegant code, versus thinking for 3 minutes, spending 3 hours typing up code.

-- Compsci.ca

The post above also mentions some of the crap I've mentioned here, like the 4600 line Sudoku.
Student code is littered with conditionals highly specific. The very reason you're using a high level language is to avoid this. I've seen quiz programs where each question and its answer is in an if clause, embedded in the program! Cardinal sin. Use a file and parse the file, if you can't do that, atleast put all the questions and answers in two arrays ( don't tell me about maps ). Then you can just use a loop to pick out the question and its right answer.

Make your code extensible. Just because this is some dumb program which you're writing in school, doesn't mean it should be a waste of muscle energy, disk space and time.

If you have to choose day's of the week, put them in an array. You want to find out the number of days in a month, use a 12 element array. Or better yet, use your brain a bit more and write this:


int daysInMonth(int month) {
if(month==2)
return 28;

else if((month%2 == 1 && month <= 7) || (month%2 == 0 && month >= 8))
return 31;

else return 30;
}


It might seem like more code, but you can write this code once and use it anywhere. And you are using your brain.

Solutions

Use better languages

Far too many schools have jumped on the Java bandwagon. If you are teaching kids who don't have much programming experience start with a higher level, small language. Use Python. You should be able to finish explaining most of the language within a week. Besides the interactive interpreter is great for messing around to know exactly how something works, how to make it go wrong and where it can fail. You will also spare them from the compile cycle ( I know, it kind of contradicts my first problem :p ), allowing them instant gratification. Spare them the brackets and let them develop the logic. Logic is far more important than syntax. Syntax changes, but logic is immortal.

Real World Knowledge

Once in a while just stop the programming, and discuss technology related issues, latest news, ethical considerations. Anything which expands their minds and gets them more interested in computers. There are more than enough anecdotes and quotes about programming, use them. Instructors should stay on top of news stories. You don't have to know what monads are, but at least know what Linux is. One of my instructors had never heard of Opera and Mozilla ?!? Whatever language you are teaching, make sure you know it well. The same instructor above didn't know that the Java GUI framework is called Swing. Make sure you are aware of atleast major upgrades and integrate them.

The Open Source ecosystem is set to flourish. Make sure your students are aware of what it is. Encourage them to try out FLOSS. If you can't convert to Linux, atleast use open source compilers, tools and languages on your Windows machines. Set aside a computer where you can mess with distros.

Encourage frequent group projects. Let the kids make plans. Tell them how to go about design. Make sure its extensible and adaptable. If the project will need something more than what is taught, tell them to read about it, while you do too.

Good Reading

I don't know about the rest of the world, but in India, Computer Science textbooks for high school are crap. They are full of errors, bad practices, incorrect language, and often old software. There are hundreds of better quality books available online, often royalty free. There is Wikipedia and Wikibooks. While using them, remember to tell your students how cool the Internet is, and teach them about sharing.

Mark for the big picture

There is this huge emphasis on theory at the high school level, mostly due to lack of time for practical assessment. Which means students have to often write programs and submit them without testing. In such situations, don't cut marks for syntax errors, unless they mess up the logic ( missing braces ). The compiler/interpreter is going to catch them anyway. As long as the program works correctly logically, missing semicolons are OK.

Rank readability above extensibility, extensibility above just-getting-it-done. If a student has used some cool logic, or exploited the specifications, give him some appreciation. After all programming is all about getting the best out of constraints, and pushing the limits. If he/she has looked to optimize, that's good too.

Don't hand hold

Far too many instructors help students fix their errors. Don't help them fix the errors, help them understand what the error is. Any decent language is quite verbose at error reporting, and in case of languages which support exceptions, the class name is often a very good indicator. Any decent compiler also reports line numbers. Let them fix the problem for themselves. Time well used now, is time saved later. Tell them it's called debugging.

Create some passion

Kate Masukomi sums it up nicely. Most programmers today are doing it to pay the bills. Don't let your students be one of them. If someone just doesn't seem to have any aptitude, maybe he should look into some other field. Encourage the good ones to expand out of the syllabus, mess with different languages and read sites like Proggit. Tell them to blog and learn HTML/CSS and create their own site. Don't create hundreds of typists, the world would prefer fifty real programmers ( thanks to lankythoughts for the link! ).

Conclusion

If something isn't done quickly, the IT world will be in a mess. Already the lack of real talent is leading to a shortage of employees. According to me, the only real coders are the people who are up breast with new developments and technologies. They don't have to know how everything works, but they should know that it exists. That's passion. They are the ones who read blogs, write them, comment on articles, are always hacking on side projects and have commits on some open source project. Make your students a part of that group.

P.S. Forgive my immodesty in certain areas :)

Factor -ize

Its been a week since i discovered Factor. So first, it's stack based, it's small, it's almost tiny. I've never programmed in a stack based language before. But I liked it, really makes you think. Infact I haven't got any program written yet, just playing around in the interactive interpreter. It's got really good documentation by way of the API, what it could use is a quick start guide, ie. a list of some basic operations of the stack, using conditionals and looping and arrays and hashmap, because right now you've to look through the documentation for each concept, which means the unity is broken.

But great language, small community, even smaller existing code. What is bad is that most of the 'responders' give 404s. :(

Tetrablocks and Login Win32 versions

I forgot to post this but Windows executables of both Tetrablocks and Login have been uploaded to 22bits. Compiling them on Windows was a bit of a nightmare. Tetrablocks has got all of its text drawing code removed since I couldn't get SDL_ttf to link. So you won't see the score or the game over message when playing it on Windows.

ColourCode: 1.0 Released

I'm quite pleased to announce the release of version 1.0 of ColourCode. You can download and know more about it from the ColourCode homepage. Please leave your feedback and bugs in the comments

ColourCode: Some thoughts about the future

Before I write about the possible future of ColourCode, let me get a few things out of the way. First today I fixed an extensionless file bug, and you can now force languages in ColourCode.

Now that thats out of the way, I was just thinking about how much interest I still have in ColourCode. I mean writing it was a lot of fun and gave me some new insights into design but I don't feel the same interest I had in parsing and analysing language files, that I had when I began ColourCode. Then it was a new challenge, now its really piling up and intruding in other plans I have. Also I am now officially in the 12th standard, which in India means a really tough exam at the end of the year, and a hundred other entrance tests. My computer time has dropped to just 1.5 hours a day, not enough to do everything. So this version of ColourCode will be the last for a long time ( atleast till the end of March ). I will continue fixing any bugs found, but there will be no new features. So please forgive me.

Thanks a lot for being a user of ColourCode, and check out my other projects.

ColourCode: 1.0 almost done

ColourCode is done. It now has support for PHP and HTML which it didn't have before. Perl support will probably get in before the stable release. This version has been bumped to 1.0. I think I've made a really good project and that it deserves the 1.0 tag. PDF support will not enter the 1.0 release but is on track for 1.1.

All that is left now is testing some highly tweaked files. All the standard test files I've been using are passing well. Also I need to write some documentation about implementing custom language handlers and formatters.

You can download today's development build which fixed all known bugs - colourcode-20070807.tar.gz

Please leave your comments. For usage see the usage.html file under the docs folder.

ColourCode:Updates

Language support

ColourCode now has support for C# and Javascript, completing support for the original language set. I'l be adding Perl and HTML soon enough.

Bugs

On the C based languages front there is a small logic error in the multiline comment system. I am currently trying to iron it out.

Snippet feature dropped

I've decided to drop the snippet feature since its not universal (doesn't make sense in PDF) and since CSS is now dynamically generated, giving the CSS data to the user and having him embed it will be quite annoying. So you can ofcourse embed the file in your HTML since its valid. You'll just need to remove the DOCTYPE declaration and the head section, move the style to your head and remove the end body and html tags. But its best if you just use the pages by themselves, perhaps linking to them.

Note: Nightly builds are now updated everyday at the homepage. So you can always check out the developments yourself.

ColourCode: Almost done

In the last two days ColourCode has reached a stage where I can say that it is finally ready for release. Everything apart from implementing language handlers is done. PDF support might not be added. It all depends on how patient I am in releasing it. Meanwhile you can download a nightly build and view demos for Ruby and Python from the link above.

ColourCode: Making progress

ColourCode has been getting along quite well for the last week. I now firmly have the basic design for my plans in working code. The language parser and handlers are coordinating well and this might just be one of the best designs I've ever envisioned.
Once the main program is in place writing language handlers and formatters for ColourCode will be really easy.
At this stage only the Ruby language handler is complete, and until I can make it work perfectly for a large amount of code, no other handlers are expected.
ColourCode 0.x will also have PDF support due to the Ruby-PDF project.

You can download a development snapshot made at the time of writing this. Most code which needs to be commented is commented.

TetraBlocks: on freeze

TetraBlocks has been put of freeze for a few days. I've got pieces rotating and stuff, but thats it for now. Right now its time to concentrate on more generic issues like writing a small highscore and menu framework, because I know that these two things are necessary in almost any game you will ever create. So if I can get those two things done, it will save me a lot of time.

Desertion...

...or why I never maintain my old programs much.

The reason I create some software, fix a few bugs, do a few minor version upgrades and then let it be, until someone really needs something is because, well, I like to do a lot of different projects in different fields. Now it so happens that I am still in school, now going to standard 12, which in India means bad. So I get just about 2 hours a day on the computer. Which means that any project takes atleast two months to complete. If I ever want to complete my Todo list then I have no choice but to abandon older ones, or return to them after quite a few months. That is why ColourCode 0.3 is on hold. So sorry users of my software. But if you really want upgrades, you may send me a million emails, and then I will do it, or just hack the code yourself, its open source.

I will try my best to take up maintenance of all of it as soon as this year is over.

Pixelframe Preview

As Pixelframe is nearing completion I have uploaded a pre-alpha not as a download but as a demo. Most of the features are complete. The few remaining are sorting out bugs and writing some documentation. To see the demo with two galleries try these links

• http://22bits.exofire.net/demos/pixelframe/client/?album=Wallpapers
• http://22bits.exofire.net/demos/pixelframe/client/?album=Inkscape

If you have comments about improvements, please post them below.

Dynamic delegation in PHP

While working on Pixelframe I devised 2 methods to dynamically delegate tasks to appropriate functions. These methods may already be known and used but these are mine and I thought it would be good to post them. These are particularly useful for handling requests where you call a certain function/script based on one parameter ( say 'action' ) and pass the rest of the parameters to the function /script. So here goes.

Delegation to functions in the same script

//the key for the action name in _POST
define("KEY_PARAM", "action");

//set up value=>funcToCall delegation array
//add new entries for each action=>function pair
$DELEGATES = array( "next"=>"nextImage",
                   "previous"=>"previousImage");


//check if the action we received from the request is defined
//check if the function is defined
if(array_key_exists($_GET[KEY_PARAM], $DELEGATES) &&
  function_exists($DELEGATES[$_GET[KEY_PARAM]]))
{
   //get the function string
   $func = $DELEGATES[$_GET[KEY_PARAM]];
   //remove this so it doesn't get passed to function, not really necessary
   unset($_GET[KEY_PARAM]);

   //call the function
   call_user_func($func, $_GET);
}
else {
   die("No such action {$_GET[KEY_PARAM]}");
}

Delegation to other scripts

When using this kind of delegation you should either decide a common function which IS implemented by all scripts or add your code to dynamically call the appropriate function ( say each function has the name the same as the script ).

//now replace funcToCall with script to include
$DELEGATES = array( "savechanges" => "save.php",
                   "changepassword" => "change_password.php",
                   "editpref" => "editpref.php" );

if(array_key_exists($_GET[KEY_PARAM], $DELEGATES)) {
   //include the script
   //assuming they are in the same directory
   include_once($DELEGATES[$_GET[KEY_PARAM]]);
   unset($_GET[KEY_PARAM]);
   //here handler is the function
   //now that the script is included, its available to us
   handler($_GET);
}
else {
   die("No such action {$_GET[KEY_PARAM]}.");
}

Well that covers it up. I hope it helps someone out there