Co-author Carl Doersch discusses "What Makes Paris Look Like Paris?" (cacm.acm.org/magazines/2015/12/194622), a Research Highlights article in the December 2015 CACM.
00:00 Ah, Paris. The restaurants, the architecture, the street life. That's not Paris, you say? You're right! But how did you know that?
00:15 We feel a city's distinctive qualities, often without knowing why. Somehow, the way that buildings, vehicles, signs, and other objects interact tells us that we're in Paris. Or London. Or Prague.
00:35 Join us as a computer vision researcher tells us how he found out What Makes Paris Look Like Paris.
00:45 [Intro graphics/music]
00:54 Carl Doersch is a doctoral student in the Machine Learning Department at Pittsburgh's Carnegie Mellon University. A trip to Paris with his advisor opened his eyes to architectural features he'd never noticed before.
01:12 CARL DOERSCH: And it just struck me a lot that you can go to different boulevards in Paris and see the same kinds of balconies and railings on the windows, and all of these features that are just repeated everywhere.
01:27 So Doersch and advisor Alexei Efros joined three other researchers to teach computers how to unlock the city's visual secrets.
01:36 They started with thousands of Google Street View images from Paris and eleven other cities for comparison. Next, they let their computers divide each image into about 25,000 patches.
01:49 DOERSCH: So if you start with a patch that's too small, you're going to find lots of matches because you have very few features in the patch. You'll be able to find lots of things that are similar to that. But you're probably not going to have anything interesting in that patch.
02:04 After dividing the images into patches, the algorithm selects some at random and looks for elements that occur frequently, but more often in Paris than in not-Paris.
02:15 DOERSCH: So for example: The Eiffel Tower is very informative. You see that, you know immediately that you're looking at Paris. But the problem is there's only one of them. … On the other hand, if you think about something like ordinary streets, like just the pavement, that's very frequent, but it's not discriminative.
02:33 After selecting patches, the real magic begins, as the algorithm looks for similar ones, with the knowledge of whether the source image is from Paris or another location.
02:44 CARL DOERSCH [4867/0127] If the street sign is actually distinctive, then it's only going to match to other street signs in Paris. And the pavement is going to match to pavement that occurs everywhere.
02:53 The algorithm then uses the best-matching patches as training data, and the winnowing process repeats. The result is a system that can go far beyond street scenes.
03:05 DOERSCH: You could also apply this to images of products and, for example, try to figure out what differentiates Apple products from everybody else's products.
03:14 But even if the algorithm is used solely on cities, what the paper calls "computational geocultural modeling" could reveal knowledge about who we are, and where we came from.
03:27 DOERSCH: We can get a map which shows us where the architectural influence crossed borders, and it can tell us how cultures interacted in the past.
03:37 Find out more in in the Research Highlights article, "What Makes Paris Look like Paris?", in the December 2015 issue of Communications of the ACM, .
03:49 [Outro and credits]