ImageScape FAQ
This project is an investigation into true image query interfaces. Toward this goal, we implemented (1) a general sketch query interface and (2) object oriented(thru the image icons) image queries.
Frequently Asked Questions

In what areas is research occurring?
We are looking at (1) extensive testing of the different sketch to image matching algorithms; (2) better color edge detection; (3) fast sketch matching methods; and (4) object recognition.

Who was involved with this project?
The principal investigator was Michael Lew. The lead for the initial version of the java sketch interface was Kim Lempinen. Significant enhancements were made regarding image coding/decoding (a 100:1 decrease in transmitted bytes) the sketch algorithms and debugging by Kim Lempinen and Michael Lew. Improvements to the Java client interface, an automatic testing program, and combining neural networks with the sketch engine are being worked on currently by Kim Lempinen. The web search engine for the thumbnail database, the object recognition programs, and the integration with the server side of the image query engine are being improved by Michael Lew.

How does the sketch matching work?
Sketch matching works on the assumption that the sketches made by people are similar to the edge maps found by computers. The user sketch is sent from the user's browser to our server at Leiden University. Then the sketch is compared to the edge maps from our database based on similarity in shape. Thus the database images which have edge maps which are most similar to the user sketch are selected.

In order to achieve scalable results for very large image databases, we focussed on multi-scale approaches. This means that the original image is reduced in size to smaller or low resolution copies. The actual matching is first performed on the smaller copies and then the set of best results is refined by using the higher resolution copies. The final set is given to the user.

How does the object detection/recognition work?
Our approach stems from our work on human face detection in complex backgrounds. We use information theory to determine which features are the best for classification and then apply principal component analysis for the detection engine. In addition we perform color and texture analysis for color images. For instance, we also use skin detection by color for detecting faces. See the link for Leiden University below for more details. Note that detection should not be interchanged with recognition. In the case of human faces, face detection refers to locating where the human faces are within an image. Face recognition would refer to identifying who the person is. Face recognition algorithms would be of minimal usefulness for detecting faces and vice-versa.

What research institutions have similar ongoing research?
The most relevant research is on detecting human faces in complex backgrounds. Three representative research institutions which have been publishing in the scientific world are: Leiden University (M. Lew and N. Huijsmans), Carnegie Mellon University (H. Rowley and T. Kanade), and MIT (K. Sung and T. Poggio). In particular you can find working WWW pages, demos or executables at

Leiden University: Information Theory (M. Lew and N. Huijsmans)

Carnegie Mellon University: Neural Networks (H. Rowley and T. Kanade)

We aren't aware of a web page at MIT for this topic, but numerous technical reports describe their methods.

A special purpose skin detector was created at Berkeley (D. Forsyth). However, its not clear if their method can be easily generalized to detect other objects.

How effective are the sketch and object recognition methods?
We are currently performing benchmarking all of the methods. The sketch methods are particularly difficult to evaluate since the ground truth is not always clear. Suppose the user draws a circle which he intends to be a basketball. Then suppose the algorithm shows the user a picture of the planet Jupiter. Is the algorithm wrong? Thus, the problem is that one sketch may have multiple correct answers.

The object recognition methods are designed to have a low "false alarm" rate. This means that we would prefer to have the algorithm err on the side of missed objects as opposed to objects which are misclassified. In other words, the user is being shown objects which the algorithm is "confident" about. Fewer hits, but less noise!

ImageScape Visual Query Interface

Try our prototype Java sketch interface and indexing engine. You must be using Netscape 3.0 or 4.0

ImageScape Visual WebSearching Demo

Media Lab Overview
LIACS Homepage
MM Conf
ACM Multimedia
Science Direct
IEEE Library
LIACS Publications
ACM Digital Library