What’s Up Next?

I have been planning to revamp my old website and only had time to do it during christmas break.  I had this idea of making it a portal of sorts for me. The rational for me is that I now rely on several “community” accounts like Multiply, Flickr and WordPress for my needs instead of creating everything on my own. The approach I decided on was to display a timeline out of the combined RSS feeds from my different accounts – and I wanted to do this without any backend coding.

To do this, I had to do the following steps:

1. Get RSS feeds from my accounts (Multiply, Flickr, WordPress and Plurk)
2. Combine the RSS feeds into one single feed using Yahoo Pipes
   1. Do some search and replace on the feed titles (like remove the date from my Multiply entries)
   2. Sort entries by date in descending order
   3. Truncate the entries to the first 30 entries only
   4. Return the feed in JSON format
3. Use jQuery (javascript library that does AJAX and JSONP) to retrieve my feed from Yahoo Pipes
4. Write a javascript routine to layout the entries timeline in correct order and spacing

RSS Feeds
Majority of today’s website support RSS feeds, although RSS feeds have differences in format, Yahoo Pipes is able to handle the more common differences. What are the feed URLs?

• http://xxxxx.multiply.com/feed.rss
• http://www.plurk.com/xxxxx.xml
• http://xxxxx.wordpress.com/feed

The “xxxxx” is your account name. You just need to know what the URLs are as input to Yahoo Pipes.

Yahoo Pipes

Yahoo Pipes is a website that allows you to post process RSS feeds (or organized HTML). It has a  drag and drop interface where you can specify the various operations your feed can go through until the final output is achieved. Some used it to cross reference Google Map and an RSS feed with location information (longtitude, latitude) to generate result like finding  apartments near your location. One of its more common use is for combining multiple feeds into a single one and there are a lot of tutorial out there already. You can specify the format used for the output feed (RSS or JSON).

One caveat here is to use the “y:published” date in your pipe operations (like sorting)  instead of the “pubDate” field of the RSS feed as different RSS feeds would have different date formats while “y:published” returns the date as processed by Yahoo Pipes.

jQuery

“jQuery is a fast and concise JavaScript Library that simplifies HTML document traversing, event handling, animating, and Ajax interactions for rapid web development.” as quoted from its website. One ability of this library is to use JSONP for calling cross-domain APIs. Normal AJAX requires that you only call your own server for security reasons. JSONP is a convention between the client and the server to bypass this restriction using script tag injection. More explanation here. It is this convention that allows us to call Yahoo Pipes directly from javascript without the use of any backend server code (at least, in your own server). Note that JSONP requires that the server supports this convention for it to work.

JSONP requires passing along a callback function in your query parameter and for accessing Yahoo Pipes, this parameter must be named as “_callback” (another name will not work, as I had to find out the hard way).

Calling Yahoo Pipes from jQuery

// call yahoo pipes where XXXXX is your pipe application id, asking yahoo to return it as json and passing the callback function
$.getJSON("http://pipes.yahoo.com/pipes/pipe.run?_id=XXXXX&_render=json&_callback=?", function(data){
// data here is the JSON object with count, value (feed entries are inside the value property)</code>

$.each(data.value.items, function( i, item) {
// this function will be called for each entry
// i is the count starting from zero, item is the feed entry
// e.g. item.title will return the feed title and so on
});

//  do some cleanup and exit
});

I also did some clean up like removing image tags from the item.description before using them since I choose to display the description on my website as tooltip popups. To remove the image tags, I used javascript regular expressions to match the tags and replace them with a space.

var re = /<img(\s+)(.+?)>/ig;
var desc = item.description.replace(re, " ");

In Yahoo Pipes, I choose to prefix my RSS feeds (with “Multiply: “, “Flickr: “, etc) in order to identify its source (of course, you can also check the item.link) and these prefixes are replaced with HTML code for displaying the mini icons for each account type.

The Timeline Effect

I used CSS absolute position in order to plot the timeline. This basically translates into a graph ( I used 20 pixels to display for one day horizontally, for example).  Since the RSS is already sorted from latest to oldest, I just loop through the entries, extract the date and multiply it by my spacing (factor  in the year and month in the same manner) and that’s the horizontal position.

var position = ((((firstYear-entryYear)*12) + (firstMonth-entryMonth)) * 20 * 31) + ((31 - entryDay) * 20) - ((31-firstDay) * 20);
// where 31 days (in a month) and 20 pixels spacing (for a day)
// firstYear, firstMonth and firstDay is the date from the first entry
// entryYear, entryMonth and entryDay is the date of the current entry

The issue I had here was the vertical positioning:

• there can be multiple entries on the same day and overlap each over
• there might be two entries from two different days but overlapping each other side by side ( consider also different lengths of title text)

The solution I finally decided on was quite simple: Store the horizontal tail position (end of title text, not the start position) of the titles that are already displayed in an array with the row as index. For each new title entry,  loop through the array to check that the tail position is not overlapping with the current starting position (in other words, check  for the first available non-overlapping row). If available, then use that row, if not, add a new row at the bottom.

(Side note: A friend told me that horizontal scrolling is annoying – so I added an option to view the information as a list.)

The Finished Product

Presenting my own web 2.0/mashup/portal website (just the first page; the other pages… to follow when I have time ).

Non-rectangular forms in SWT are created using Regions, which are areas that you can add and subtract from. Now if you know the shape or path of your non-rectangular form beforehand, it is pretty simple to code.

        Display display = new Display();
	Shell shell = new Shell(display, SWT.NO_TRIM | SWT.ON_TOP);
	//define a region
	Region region = new Region();
	region.add(circle(67, 67, 67));
	region.subtract(circle(20, 67, 50));
	//define the shape of the shell using setRegion
	shell.setRegion(region);

What happens if I want to construct a non-rectangular form based on a bitmap image with transparency (alpha channel)? Something like this:

One quick approach is to do a pixel by pixel comparison and build the region one pixel at a time, however, this has a performance penalty (almost 3 seconds for a 400×250 pixel image). A closer examination points to the multiple calls to add the region as the problem. For an 400×250 image, there would be 100000 calls if done pixel by pixel. What was needed was a way to group contiguous areas together into a larger rectangle to cut down on the number of calls.

The approach I took is to split the image into four quadrants and check if each quadrant is a solid rectangle or not. If a quadrant is not solid (meaning it contains transparent pixels) then that quadrant is further broken down into smaller quadrants until one of the quadrant’s dimension is one pixel and could not be broken down further or the quadrant is totally transparent. Another further optimization is to look for adjacent rectangles that can be further combined into a single rectangle. With this, I was able to cut down the number of calls to add region to 747 for the above image and performance is now down to less than one sec from previous 3 seconds. I guess there are other ways to do this also and further optimizations can be done but this works for me.

The following code returns a region given an image or image data.

package sift.util;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.eclipse.swt.graphics.Image;
import org.eclipse.swt.graphics.ImageData;
import org.eclipse.swt.graphics.Rectangle;
import org.eclipse.swt.graphics.Region;

/**
 * ImageDataHelper.java
 *
 * Copyright (c) 2008 August King Li
 *
 * @author  August Li
 * @version  1.0
 */
public class ImageDataHelper {
    public List<Rectangle> rectangleList;

    private boolean isWholeAlpha(ImageData data, Rectangle rect, boolean alpha) {
        if (alpha) {
            for (int y = 0; y < rect.height; y++) {
                for (int x = 0; x < rect.width; x++) {
                    if (! (data.getAlpha(rect.x + x, rect.y + y) == 255)) {
                        return false;
                    }
                }
            }
        }
        else {
            for (int y = 0; y < rect.height; y++) {
                for (int x = 0; x < rect.width; x++) {
                    if (!(data.getPixel(rect.x + x, rect.y + y) != 0)) {
                        return false;
                    }
                }
            }
        }
        return true;
    }

    private void quadCheck(List<Rectangle> rectList, ImageData data, Rectangle rect, boolean alpha) {
        if (rect.width == 1 && rect.height == 1) {
            if (isWholeAlpha(data, rect, alpha)) {
                addList(rectList, rect);
            }
        }
        else if (rect.width == 1 && rect.height > 1) {
            int h1 = (rect.height / 2);
            int h2 = rect.height - h1;
            Rectangle r1 = new Rectangle(rect.x, rect.y, 1, h1);
            checkRectangle(rectList, data, alpha, r1);
            Rectangle r3 = new Rectangle(rect.x, rect.y + h1, 1, h2);
            checkRectangle(rectList, data, alpha, r3);
        }
        else if (rect.width > 1 && rect.height == 1) {
            int w1 = (rect.width / 2);
            int w2 = rect.width - w1;
            Rectangle r2 = new Rectangle(rect.x, rect.y, w1, 1);
            checkRectangle(rectList, data, alpha, r2);
            Rectangle r4 = new Rectangle(rect.x + w1, rect.y, w2, 1);
            checkRectangle(rectList, data, alpha, r4);
        }
        else {
            int w1 = (rect.width / 2);
            int h1 = (rect.height / 2);
            int w2 = rect.width - w1;
            int h2 = rect.height - h1;
            Rectangle r1 = new Rectangle(rect.x, rect.y, w1, h1);
            checkRectangle(rectList, data, alpha, r1);
            Rectangle r2 = new Rectangle(rect.x + w1, rect.y, w2, h1);
            checkRectangle(rectList, data, alpha, r2);
            Rectangle r3 = new Rectangle(rect.x, rect.y + h1, w1, h2);
            checkRectangle(rectList, data, alpha, r3);
            Rectangle r4 = new Rectangle(rect.x + w1, rect.y + h1, w2, h2);
            checkRectangle(rectList, data, alpha, r4);
        }
    }

    private void addList(List<Rectangle> rectList, Rectangle rect) {
        boolean adj = false;
        for (Rectangle r : rectList) {
            if (r.width == rect.width && r.x == rect.x && r.y + r.height == rect.y) {
                //on top
                adj = true;
                r.height = r.height + rect.height;
            }
            else if (r.height == rect.height && r.y == rect.y && r.x + r.width == rect.x) {
                //on side
                adj = true;
                r.width = r.width + rect.width;
            }
        }
        if (!adj) {
            rectList.add(rect);
        }
    }

    private void checkRectangle(List<Rectangle> rectList, ImageData data, boolean alpha, Rectangle rect) {
        if (isWholeAlpha(data, rect, alpha)) {
            addList(rectList, rect);
        }
        else {
            quadCheck(rectList, data, rect, alpha);
        }
    }

    private void generateRectangleList(ImageData imageData) throws IOException {
        rectangleList = new ArrayList<Rectangle>(100);
        boolean alpha = false;
        if (imageData.alphaData != null) {
            alpha = true;
            Rectangle rect = new Rectangle(0, 0, imageData.width, imageData.height);
            quadCheck(rectangleList, imageData, rect, alpha);
        }
        else {
            ImageData mask = imageData.getTransparencyMask();
            Rectangle rect = new Rectangle(0, 0, mask.width, mask.height);
            quadCheck(rectangleList, mask, rect, alpha);
        }
    }

    public Region computeRegionFromImageData(ImageData imageData) throws Exception {
        generateRectangleList(imageData);
        //calls to region.add are slow, so the above code optimizes the no. of rectangles before calling add
        Region r = new Region();
        for (Rectangle rect : rectangleList) {
            r.add(rect);
        }
        return r;
    }

    public Region computeRegionFromImage(Image image) throws Exception {
        return computeRegionFromImageData(image.getImageData());
    }

}

Java applications with multiple JAR dependencies are sometimes messy to mass deploy due to multiple library files and the need to setup the classpath (although the latter can be done using the Manifest file). Eclipse 3.4M4 and up now has the Runnable JAR Export wizard which creates a single JAR with all your external libraries (such as log4j, swt, xstream JARs) repackaged nicely within. The idea itself is not new but what is new is that it is now integrated with Eclipse, however, there are reported problems with Spring. You can also use “Fat JAR Plugin” (on which the Eclipse wizard is based upon) or One Jar (which allows you to have JARs within a JAR using a custom class loader – around since 2004).

Lately, I have been using xstream for serializing POJOs to and from XML, so far I have been using it for application configuration files and really liked its simple approach. I even had it generate an RSS feed for me with less than 30 minutes of coding. Read their two minute tutorial here.