Complete Benchmarks
The IAPR TC-12 Benchmark
The image collection of the IAPR TC-12 Benchmark consists of 20,000 still natural images taken from locations around the world and comprising an assorted cross-section of still natural images. This includes pictures of different sports and actions, photographs of people, animals, cities, landscapes and many other aspects of contemporary life. read more>>
ImageCLEF Wikipedia Track
ImageCLEF's Wikipedia Retrieval task provides a testbed for the system-oriented evaluation of visual information retrieval from a collection of Wikipedia images. The aim is to investigate retrieval approaches in the context of a large and heterogeneous collection of images (similar to those encountered on the Web) that are searched for by users with diverse information needs. read more >>
TRECVid benchmark
TREC conference series is sponsored by the National Institute of Standards and Technology (NIST) with additional support from other U.S. government agencies. The goal of the conference series is to encourage research in information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. In 2001 and 2002 the TREC series sponsored a video “track” devoted to research in automatic segmentation, indexing, and content-based retrieval of digital video. Beginning in 2003, this track became an independent evaluation (TRECVID) with a 2-day workshop taking place just before TREC. read more>>
The MediaMill challenge
The MediaMill challenge builds upon the TRECVID collection defined above. It gives annotations and intermediate results for 101 semantic concepts. It provides an easy entry point to perform semantic video indexing on a large collection. read more>>
Benchmarking Suite Retrieval Analyzer of the University of Arizona
The benchmarking suite Retrieval Analyzer includes a collection of three mapping methods described in this paper. The inputs to Retrieval Analyzer are a vector of computer scores (your image retrieval scores for the image pairs we have provided in our ground truth data) and corresponding human scores (our ground truth data). The outputs consist of a correlation score and an estimated precision-recall curve. We provide an option for chosing from any of the three mapping methods, but we recommend using the method that maximizes the correlation score. read more>>
