A poor man's Notational Velocity on Linux

I use Notational Velocity on my Mac all the time. It holds all my notes, lists and any other snippet of text. I love the interface and simplicity, and most of all I love the simple use of text files in Dropbox as a store. This way I can access my notes anywhere, without needing NV to be installed. I also love the global key binding feature so that I can quickly raise it with Cmd+Shift+N.

At work (more on this soon) I started using a Thinkpad x230 running Archlinux. But I sorely missed NV. I experimented with nvpy, but it didn't cut it for me at all. The tkinter UI looks bad in a Qt/GTK desktop, notes are saved in JSON by default, and the text file option is a sort of hack which stores the title in the first line, messing up the notes in NV. So rather than write my own version, I got an almost as nice, and definitely more powerful NV equivalent in Linux.

I am going to assume you use a standard desktop environment like KDE or that your window manager is EWMH compatible. You'll need:

• To know how to define custom global shortcuts to run a command. For KDE this is System Settings -> Shortcuts and Gestures -> Edit -> New -> Global Shortcut -> Command/URL
• gvim
• wmctrl (available in Arch community repo).

Create a new shortcut which should launch the following command string

gvim --remote-silent +':lcd %:p:h | :au FocusLost * :wa' \
'/home/nikhil/Dropbox/Notational Data' && wmctrl -a 'GVIM'

You should edit the path to point to your Dropbox/NV directory. Now whenever you press the global shortcut combination you should see gvim with a list of all files (notes). Press Enter on a file to open it.

We use remote-silent to make sure that gvim uses an existing window if it is already open. The :lcd %:p:h option sets vim's current working directory to the NV directory. This will be useful later. We use the autocommand FocusLost to save the file whenever the gvim window loses focus (simulating NV's autosave feature). Finally wmctrl raises the window to the top by matching the string to the title. If you use gvim on a regular basis (I use terminal vim) and have other windows open, you'll have to tweak this.

So this setup is completely like NV, except for one divergence. Whereas NV searchs the note title and content together, our system will treat it as two flows. To search note titles/file names use / when in the main view. As part of my standard vim plugin set I have ctrlp and ack.vim^* which will serve us well here. To always have access to note titles use ctrlp. I map it to sf so that I get quick fuzzy find. Similarly to search note contents I map sd to trigger ack.vim. This is where setting vim's current directory is important. Both plugins will use it as the base search directory.

This NV approximation is fast and works almost as well as the original, although without a slick interface. But nice fonts and a good vim colour scheme come pretty close.

---

* You'll need ack installed to use ack.vim. ack does not include text files by default. Put --text in ~/.ackrc to do so.

QHttpServer: Web Apps in Qt

Qt is a great GUI toolkit, but it is also an entire C++ standard library waiting to be used for other tasks. In addition the network module is really powerful. I’ve been playing around with node.js for a while now, and realized that Qt’s default asynchronous nature maps over perfectly to create a event-based web server. To make things better, Ryan Dahl’s small and fast http-parser is freely available. So I just combined the two, and here is QHttpServer.

Here is a ‘web-application’ written completely in C++/Qt. You can even try it out.

#!cpp
#include "greeting.h"

#include <QCoreApplication>
#include <QRegExp>
#include <QStringList>

#include <qhttpserver.h>
#include <qhttprequest.h>
#include <qhttpresponse.h>

Greeting::Greeting()
{
    QHttpServer *server = new QHttpServer;
    server->listen(QHostAddress::Any, 5000);
    connect(server, SIGNAL(newRequest(QHttpRequest*, QHttpResponse*)),
            this, SLOT(handle(QHttpRequest*, QHttpResponse*)));
}

void Greeting::handle(QHttpRequest *req, QHttpResponse *resp)
{
    QRegExp exp("^/user/([a-z]+)$");
    if( exp.indexIn(req->path()) != -1 )
    {
        resp->setHeader("Content-Type", "text/html");
        resp->writeHead(200);
        QString name = exp.capturedTexts()[1];

        QString reply = tr("<html><head><title>Greeting App</title></head><body><h1>Hello  %1!</h1></body></html>");
        resp->end(reply.arg(name).toAscii());
    }
    else
    {
        resp->writeHead(403);
        resp->end("You aren't allowed here!");
    }
}

int main(int argc, char **argv)
{
    QCoreApplication app(argc, argv);

    Greeting hello;
    
    app.exec();
}

Awesome isn’t it? You launch an instance of QHttpServer, it emits a signal whenever a new request comes in, you can handle and respond to it. The code is fully documented, so you can do a git clone and run doxygen in the docs/ folder to get API documentation. For now it doesn’t deal with everything that can happen in HTTP, but it does know about keep-alives (no chunked encoding though). QHttpServer is streaming, see the body data example.

Here is a simple ApacheBench run comparing qhttpserver and node.

QHttpServer Greeting app:

> ab -n 1000 -c 100 http://localhost:5000/user/nikhil
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        
Server Hostname:        localhost
Server Port:            5000

Document Path:          /user/nikhil
Document Length:        88 bytes

Concurrency Level:      100
Time taken for tests:   0.467 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      151000 bytes
HTML transferred:       88000 bytes
Requests per second:    2142.47 [#/sec] (mean)
Time per request:       46.675 [ms] (mean)
Time per request:       0.467 [ms] (mean, across all concurrent requests)
Transfer rate:          315.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   4.7      0      23
Processing:    14   43  18.9     40     123
Waiting:       13   43  18.9     40     123
Total:         14   45  18.9     41     130

Percentage of the requests served within a certain time (ms)
  50%     41
  66%     48
  75%     53
  80%     58
  90%     69
  95%     80
  98%     98
  99%    117
 100%    130 (longest request)

node.js greeting app:

> ab -n 1000 -c 100 http://localhost:5000/user/nikhil
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)


Server Software:        
Server Hostname:        localhost
Server Port:            5000

Document Path:          /user/nikhil
Document Length:        88 bytes

Concurrency Level:      100
Time taken for tests:   0.441 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      151000 bytes
HTML transferred:       88000 bytes
Requests per second:    2267.94 [#/sec] (mean)
Time per request:       44.093 [ms] (mean)
Time per request:       0.441 [ms] (mean, across all concurrent requests)
Transfer rate:          334.43 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   4.4      0      18
Processing:     4   40  22.7     37     101
Waiting:        4   40  22.9     37     101
Total:          4   42  21.7     39     101

Percentage of the requests served within a certain time (ms)
  50%     39
  66%     50
  75%     57
  80%     63
  90%     73
  95%     83
  98%     91
  99%     95
 100%    101 (longest request)

While not a very scientific benchmark, this does show them pretty close. C++ is compiled, but I believe node has less ‘layers’ and a particularly efficient I/O infrastructure which allows it to be better even with an interpreted language.

I would really like to see this being used in Qt apps that need a web interface for remote control, or for simple delivery of files.

In addition, if you will be attending conf.kde.in in Bangalore, India on March 9th-13th, my talk on Qt Scripting will involve writing a thin JavaScript wrapper over this and doing some other cool stuff! So be there.

Quick and dirty Instant Messaging with Redis

(Aside: Yes this is a post after a zillion years, but I have a few more lined up, and waiting for some important stuff in about a week)

Redis is a wicked cool NoSQL database, in that storing stuff is not the only thing that it does. Mathias Meyer already has a collection of Redis use cases, but this great idea is mine. Like all good ideas it emerged in the shower :) ( I was not aware of Pieter Noordhuis' MUC when I did this, in either case mine deals with one-to-one IM )

Using the new publish-subscribe commands and a bit of node.js code, here is a tiny instant messaging server. An explanation follows, refer to the code while reading it, Blogger sucks for publishing code.

Here is how it works. For every client, the server maintains two connections to redis. This is because a subscriber is not allowed to invoke other commands. So we have a subscriber connection and publisher connection. Consider two clients are now connected, Alice and Bob.

When Alice connects ( NICK Alice ), we insert her nickname into 'mclarens:inside', a Redis set. This allows us to have a WHO command to list online members. Bob to does the same. To start chatting with Alice, Bob initiates a conversation ( TALKTO Alice ). The node server does none of its own client management. Instead each client just maintains subscriptions to two channels/classes. Each client is always subscribe to '[nick]:info', where it is notified of talk initiations and exits etc. When Bob wants to talk to Alice, he sends a 'start Bob' to Alice's info channel, 'Alice:info'. Then both Alice and Bob subscribe to the class 'Alice:Bob', using alphabetical order to decide the name. When either of them wants to talk to the other, they do MSG [nick] [message]. The connection uses the publish redis connection to send a message to the channel, resulting in both sides being notified of the incoming message.

To terminate the chat, one side just has to send a 'STOP [nick]'. That unsubscribes the user from the class so that he no longer receives messages. It also sends a 'stop [nick]' to the other side, so that he/she can also unsubscribe.

On QUIT, we simply remove ourselves from the mclarens:inside set, unsubscription is handled automatically by Redis!

That's it, simple Instant Messaging! Now this lacks any kind of security and ignore lists and status but proves the point. In fact I'm thinking of using this as the backend for the IM part of the XMPP server I'm hacking on. At this point it is in no shape to have this feature just yet, but it should some day.

(Thanks to roidrage and tnm on #redis for pointing out that I had to use two Redis connections for PubSub.)

GMail Notifier on an Arduino

The exams are over and I've been hacking a bit on the Arduino today. So I came up with a simple hack which blinks an LED on the Arduino if you've got unread mails in your Gmail inbox. I assume that you're familiar with the basics of Arduino.

Equipment

• A computer with an internet connection


• Python


• pySerial


• Arduino ( I'm using Duemilanove )


• Red LED


• Pushbutton (optional)


• Wires
  

On the computer

A python script will run continuously on the computer, and fetch the Gmail RSS feed every few minutes. pySerial will be used to notify the Arduino of new mails.

Here are our imports and constants

# ~ Gmail Notifier for Arduino
# ~ This file is released under the public domain

import httplib
import getpass
import base64
import re
import time
import serial

INTERVAL = 5 # check every INTERVAL minutes

serv = 'mail.google.com'
path = '/mail/feed/atom'
# ask user name and password and encode them for authentication
auth = base64.encodestring(
                 '%s:%s'%(raw_input('Username: '),
                   getpass.getpass()))

So first, fetching the feed. We'll use httplib. Here is the code:

def getfeed():
  print 'Checking...'
  conn = httplib.HTTPSConnection(serv)
  conn.putrequest('GET', path)
  conn.putheader('Authorization', 'Basic %s'%auth)
  conn.endheaders()
  return conn.getresponse().read()

Next lets get the count. Gmail replies in the following format. In the case of new mails there is more information, but we don't care about that. We're mainly interested in fullcount.

<?xml version="1.0" encoding="UTF-8"?>
<feed version="0.3" xmlns="http://purl.org/atom/ns#">
<title>Gmail - Inbox for nsm.nikhil@gmail.com</title>
<tagline>New messages in your Gmail Inbox</tagline>
<fullcount>0</fullcount>
<link rel="alternate" href="http://mail.google.com/mail" type="text/html" />
<modified>2009-02-22T11:00:33Z</modified>
</feed>

So we'll use regular expressions to get the count.

def count(data):
 matches = re.findall('<fullcount>([0-9]+)</fullcount>', data)
 if len(matches) == 0:
     print 'Error in parsing feed, check user name and password are correct'
     return 0
 return int(matches[0])

We'll need to write this to the serial port.

def writeSer(data):
 try:
     # the best way to find this out is to launch the Arduino environment
     # and see what it says under Tools -> Serial Port
     ser = serial.Serial('/dev/ttyUSB0')
     ser.write(data)
 except serial.serialutil.SerialException:
     print 'Error writing to serial device'
     raise

Now that we're done with the functions, it's time to make them work together

# subtract so that we check first time
last_check = time.time() - INTERVAL*60

while True:
    if time.time() - last_check < INTERVAL*60:
        continue
    last_check = time.time()
    msgs = count(getfeed())
    print msgs,'mails'
    writeSer(str(msgs))

Thats the computer part.

On the Arduino

The circuit :



The LED is on pin 13 and goes to ground. The button takes 5V through the power pins on the analog side, via a 220 ohm resistor. The other leg is grounded. Pin 4 can be used to read the state of the button. The wire from pin 4 connects to the 5V leg of the button.

The code is dead simple and so is presented together.

int ledPin = 13; // connect led to digital pin 13, or use default small one
int bPin = 4; // connect button to digital pin 4
boolean blink = false; // holds our current state

int INTERVAL = 200; // led blink rate in milliseconds

void setup() {
  Serial.begin(9600);
  pinMode(ledPin, OUTPUT);
}

void loop() {
  if(Serial.available() > 0)
    blink = Serial.read() > 48;// 48 is 0 is ASCII

  if(digitalRead(bPin) == LOW) // button pressed
    blink = false;

  // blink is true if we got serial input
  // or we had got serial input and button hasn't been pressed yet.
  if(blink) {
    digitalWrite(ledPin, HIGH);
    delay(INTERVAL);
    digitalWrite(ledPin, LOW);
    delay(INTERVAL);
  }
  else
    digitalWrite(ledPin, LOW);
}

The button is used to switch off the blinking once you've noticed that you've got mail and don't want it to keep blinking till the next check.

Running

Verify and Upload the code to the Arduino. You can try entering numbers in the Serial Monitor to check that the circuit works. Remember to switch off the Serial Monitor.

Now start your script, python mailarduino.py. Enter authentication details. Now sit back and relax... oh wait, you've got mail.

Full python script ( mailarduino.py )

import httplib
import getpass
import base64
import re
import time
import serial

INTERVAL = 5 # check every INTERVAL minutes

serv = 'mail.google.com'
path = '/mail/feed/atom'

auth = base64.encodestring(
                '%s:%s'%(raw_input('Username: '),
                getpass.getpass()))

def count(data):
 matches = re.findall('<fullcount>([0-9]+)</fullcount>', data)
 if len(matches) == 0:
     print 'Error in parsing feed, check user name and password are correct'
     return 0
 return int(matches[0])

def getfeed():
 print 'Checking...'
 conn = httplib.HTTPSConnection(serv)
 conn.putrequest('GET', path)
 conn.putheader('Authorization', 'Basic %s'%auth)
 conn.endheaders()
 return conn.getresponse().read()

def writeSer(data):
 try:
     ser = serial.Serial('/dev/ttyUSB0')
     ser.write(data)
 except serial.serialutil.SerialException:
     print 'Error writing to serial device'
     raise

last_check = time.time() - INTERVAL*60 # subtract so that we check first time

while True:
    if time.time() - last_check < INTERVAL*60:
        continue
    last_check = time.time()
    msgs = count(getfeed())
    print msgs,'mails'
    writeSer(str(msgs))

opensocial Last.fm recent tracks

After two days of OpenSocial hacking, here is my app, which fetches the users Recent tracks and displays it on the profile. Now since there is no way to directly fetch the last.fm username, I had to resort to asking the user. Also 1.0 although usable has a few errors which aren't handled smoothly. But they do not pose security risks. For now its in the sandbox, if you have access you can view it here Last.fm Recent Track and give me some feedback. It won't be on Orkut for a few days since there is a pretty large queue for applications waiting to be moderated.

Oh yeah, here's the project page on Google Code.

Glowpad

Glowpad is a small Python/Pygame program to read in a file and generate a word/pattern with randomly coloured squares which morph to white. I basically wrote it after seeing the cool effect that appears at the end of one of the Sony ads(the robot one) where the same effect happens with the word feel. So I hacked this program in just half an hour.

To run the program, extract it to any place on your computer. you NEED to have Python and Pygame installed before you can run it. After you install them just run the program by executing

python glowpad.py 

. Two patterns FEEL and LINUX are included. So to apply the effect on Linux use

python glowpad.py linux

.
To create your own patterns just create a new file in any editor. Make the pattern you want using an uppercase B. The colours are chosen randomly. You can add padding or whatever you want around the characters. The only thing Glowpad makes sense of is an uppercase B, any other character is ignored. So this is an example. Just save the file and run it as shown above

##############################
-B----BBBBB-BBBBB-B---B-B---B-
-B------B---B---B-B---B--BBB--
-B------B---B---B-B---B---B---
-B------B---B---B-B---B--BBB--
-BBBB-BBBBB-B---B-BBBBB-B---B-
##############################

DOWNLOAD:http://22bits.exofire.net/downloads/glowpad.tar.gz