New photographs from dead photographers with convolutional neural networks
Thu 12 Nov 2015
Researchers out of the University of Pittsburgh have used Convolutional Neural Networks (CNNs) to identify the strange obsessions and unique styles of 41 well-known photographers and even to generate new photographs which accord with their unique perspectives on the world around us.
Seeing Behind the Camera: Identifying the Authorship of a Photograph [PDF], by Christopher Thomas, a Ph.D. Candidate in Computer Science focusing on Machine Vision, and Assistant Professor in Computer Science Adriana Kovashka, outlines the creation of a new sampled database of 180,000 photographs from some of the greatest names in photography from the 20th century, including Ansel Adams, Lewis Hine, Dorothea Lange and Marion Wolcott.
The authors trained a high-level convolutional neural network from scratch in order to evaluate the hypothesis, employing the two existing CNNs CaffeNet and Hybrid-CNN research from Dengyong Zhou, and creating their own new evaluator called PhotographerNET. Previous academic efforts have been made to recognise style, but mainly in the field of paintings. The authors refer to previous work from Sergey Karayev in this regard.
It was important to divide the work into ‘low level’ and ‘high level’ classification, since there are descriptors and other clues in historical photographs which, though useful to develop training sets, can constitute ‘cheating’ in terms of image evaluation. Therefore low-level descriptors used included the L*a*b* Color Histogram, the Speed-Up Robust Features (SURF) descriptor and the GIST descriptor [PDF]. These systems evaluate qualities in media which constitute unintentional metadata, such as the unique features in colour emulsions used in photographic film in the last 100 years, which in themselves can narrow down likely matches without ‘intelligently’ making decisions based on pure style.
By contrast, Stanford University’s Object Bank Descriptor system was used to identify ‘pure’ matches, along with the other CNN-based techniques described in the paper.
The authors envision the possibility of a ‘search by author’ facility for image databases wherein a sample photograph could constitute the initial query, and contend that such a capability would permit identification of unlicensed use of copyrighted works. Additionally the classification and quantification of photographic styles, previously an abstruse object or research, could make new associations possible between the works of photographers who have (or had) certain stylistic traits or particular obsessions in common.
Though it’s not explicitly mentioned within the paper, the work also infers the possibility of attributing an associative style with existing photo-posters who contribute their work via online image databases, such as Facebook and Flickr, which permit academic or commercial analysis, an interesting route to a ‘recommender’ system with various cultural and commercial possibilities.
As a novel conclusion to the research, the authors attempted to create new photographs which replicate the style of the 41 major photographers featured in their new database (see image), and were able to do so by identifying the strange quirks and visual fetishes that each artist inevitably, perhaps subconsciously, brought to their output. For example the photographer Carol M. Highsmith’s obsession with signage allowed high-level descriptors to associate a new image with her output. By contrast the late Arthur Rothstein (1915-1985), was drawn to assemblies and crowds to document the human condition, whilst Ukraine-born Jack Delano’s work for the Farm Security Administration is hallmarked by a fascination for uniforms and the ‘common people’.