Sketch2Photo: Internet Image Montage on Vimeo

See all Show me

1. Sketch2Photo: Internet Image Montage

2 months ago

Due to a name conflict with a Google SketchUp Plugin, we now change title to Sketch2Photo.
We present a system that composes a realistic picture from a simple
freehand sketch annotated with text labels. The composed picture
is generated by seamlessly stitching several photographs in agreement
with the sketch and text labels; these are found by searching
the Internet. Although online image search generates many inappropriate
results, our system is able to automatically select suitable
photographs to generate a high quality composition, using a filtering
scheme to exclude undesirable images. We also provide a novel
image blending algorithm to allow seamless image composition.
Each blending result is given a numeric score, allowing us to find
an optimal combination of discovered images. Experimental results
show the method is very successful; we also evaluate our system using
the results from two user studies.

Credits

1,158 Likes

eric peltier
34 seconds ago
Sebastian Wolf
10 hours ago
Jinho Jung
2 days ago
Michelle Li
2 days ago
CIGSTU
2 days ago
jotaeme1979
3 days ago
sinC
5 days ago
Ellen Chisa
6 days ago
alex milde
6 days ago
pomi lee
7 days ago
See all 1158 likes

Tags

Andrew Mager 2 months ago

What year are you guys from?
Andy Smith 2 months ago

That's the best comment I've ever seen!
Bastian Allenstein 2 months ago

lol
Alexander Tesfamichael 2 months ago

definitely past 2020
Oscar Tong 1 month ago

Which planet are you guys from ....
richard cave 1 month ago

wow, a lot of matte painters are going to be so interested in this.
Andrew Mager 2 months ago

My blost post about it: blogs.zdnet.com/weblife/?p=965
dlew 2 months ago

a lot of agency people just lost their jobs
Zoltan Csaki 2 months ago

ummm, no. a lot of clients just got the ability to add more 'clip-art' to their crappy presentations.
dlew 2 months ago

hahahaha yes!
XSportSeeker 1 month ago

Fortunately, only the ones who deserves it...
MsTeeCee 1 month ago

Maybe it'll be really expensive or too complicated and requires design management.
M .Barron 2 months ago

I'm speechless... I should draw a stick figure bowing down to this project. Incredible.
Chris Mallgogny 2 months ago

Fake.
Tim Mrozek 2 months ago

Nah, not fake, I saw them showing this off at Siggraph
Nilesh Babu 2 months ago

unbelievable!
simon scott 2 months ago

moving images are still safe !
ReFreezed 1 month ago

Not for much longer, I fear...
Ivan George 2 months ago

And I thought Photoshop was amazing...
Sharath Patali 2 months ago

Brilliant
roger 2 months ago

Awesome. Their site must have been flooded, the high interest has taken their site down.
happyjimmy 2 months ago

My Brother has been working in this field for a number of years.. This is highly dooable.
dan heaver 2 months ago

this is brilliant.
but you guys could use a writer.
I'm available ;)
Craig Ellis 1 month ago

Now if only they could make a version which made promos...

Get off my internetz Heaver! ;-)
adam 2 months ago

please be fake im graduating soon haha
Knut Sverre 2 months ago

And the use for this is..?
Darko Stanimirović 2 months ago

Agencies, art directors, designers, clients themselves... Though, I feel like some clients would make an end product out of this!

Perfect for this "crisis". Do an ugly composite, steal a couple of images off the web, don't hire a professional!
WooHoo!
Darko Stanimirović 2 months ago

I like how this video got 72 plays at Oct 4th and then 26K the next day! :)
StudioThought, LLC 2 months ago

Superpixel to the rescue!!
Jassu 2 months ago

Great Job!!!
Quba Michalski 2 months ago

So I am to believe that while we still don't have the technology to mask an object out of a photo automatically, this system will not only do that, but find the images for me?

Good one.
David C. 2 months ago

Go read:
doi.acm.org/10.1145/1597990.1598050
Felipe Bachian 2 months ago

Awesome!
Sysop 2 months ago

Looks too good to be true?
ugur3d 2 months ago

Superpixel ???? => A Pixel is pixel.
Its a fake
Rhythmic Circus 2 months ago

They just mean groups of similar pixels.
Marck Al 2 months ago

I don't think it's real, looks like a bogus to me.
Jens Roland 2 months ago

The examples lead me to believe that it's a fake. It's a good fake, but it's fake. The lights&shadows match too well, the hand-drawn outlines are nowhere close to the results (e.g. the sitting dog, which is drawn from the side but inserted from the front, which is a completely different figure), and the "superpixel masking/blending" looks basically flawless. The state of the art Photoshop masking plugins can't get close to those examples in any automated setting. With some user involvement in picking the source images and adjusting the masking, this is doable, but fully automated? I call fake.
Pak-Kei Mak 2 months ago

Even if it's fake, it's a great concept.
Christina Lee 2 months ago

Yea.
jonwadsworth 2 months ago

I was ready to call fake... but they posted (alleged) executables:

cg.cs.tsinghua.edu.cn/montage/files/Binary.zip

And their main project site:

cg.cs.tsinghua.edu.cn/montage/main.htm

Anybody with technical skeelz wanna take a look and see if it's for real?
Juancito Caminador 2 months ago

It's a Trap!
lol
Erick Magnus 2 months ago

Awesomeeee!
Tony xu 2 months ago

it will have a good future!
Justin Phillips 2 months ago

Inconceivable!
Michael Holroyd 2 months ago

If you think it's fake then you don't have much faith in peer-reviewed journals, considering this was published at SIGGRAPH Asia this year...
Go read the paper, the solution is pretty obvious.
Jamtron 2 months ago

This was on SNL last Saturday.
Robb M. 2 months ago

amazing. we live in the future.
Christina Lee 2 months ago

Criticism is a human attempt for excusing our own lack of creativity...I say, if you don't think it's all that...think of something more original. Realism is not exactly what they are going for here, we're hopefully all having fun and harnessing technology in new ways that work for us and others. Some people just don't get it.
Matt White 2 months ago

It kind of reminds me of the series "Look Around You."

If you haven't seen that do yourself a favor and watch it!
Wesley Kandel 2 months ago

Haha! Yes!
Stuart Paciej 1 month ago

Haha I remember seeing that for the first time not knowing it wasn't a genuine science program though when they used a revolver to destroy the test tubes the penny dropped...
Vinod P 2 months ago

Incredible!!. Imagine a PhotoSketch enabled Electronic Color Story book for Kids!
Nik Skavinsky 2 months ago

So where's the link to this tool? or is it still a project.
Andreas Beer 2 months ago

where can i download the freeware tool? :D ;)
Voetmann 2 months ago

The end of photoshop and all AD´s ;o)
Silvio Vailante 2 months ago

OK…! I love it!
But, where i take this for me????
Youssef Sarhan 2 months ago

this pwns... ha i'd hate to be a graphic designer in the future..
James McWilliams 2 months ago

Yeh cos a graphic designer's job relies entirely on doing bad photo montages? (sigh, some people ;) )
David C. 2 months ago

So, I did a Google Scholar search and found an identically named program that was presented at SIGGRAPH 2009 (Aug 2009), but by a completely different set of authors:

doi.acm.org/10.1145/1597990.1598050

This paper lists authors from Berlin and Paris. And there is no overlap in authorship between this paper and this video.

This video has the accompanying conference paper hosted at the Nat. Uni. of Singapore:

ece.nus.edu.sg.nyud.net/stfpage/eletp/Papers/sigasia09_photosketch.pdf

UPDATE: The paper by Eitz et al. is cited by this Tsinghua Uni group.
Gabriel Shalom 1 month ago

Whether its a fake or its legitimate its awesome!
tr.im/B4xJ
judy tillinger 2 months ago

but wait - these images are just grabbed from ANYWHERE online? Is there no protection for photographers?
Will Marshall 2 months ago

This is how the internet works. There'd be no other way to do it: and there's no meaningful way to evaluate the copyright status of a bunch of pixels.
Perre DiCarlo 2 months ago

Genius. So glad to see masking and assembly pushed so far forward... Makes me imagine that Adobe-Macromedia is just sitting on piles of money doing little innovative R&D.
Gabriel Shalom 1 month ago

Well they're doing this at adobe... vimeo.com/2345579
Perre DiCarlo 1 month ago

Hey Gabriel, thanks for reminding me of that demo. You're right. I likely understated Adobe's 'little' innovation. My point is more about what we see in their products, which hasn't made masking radically easier despite the regular updates and new masking features. If they're rethinking it in R&D, we designers aren't seeing it show up in their products.

When I saw the video you pointed to (I think it was on digg several months ago), I thought of it like I think of car companies showing us fantastic cool 'concept cars' that'll never make it into consumer's hands. Instead I see another taurus or camry. That metaphor is apt, I think. When are we going to see these features in Premier or After Effects? Hard to justify spending so much bill on their updates when they don't deliver leaps in thinking like we just witnessed in any of these video demos...
Dan Goldman 13 days ago

These things take time to get into commercial products. Have patience! And chew on this in the meantime: vimeo.com/5024379

;-)
Culture Now 2 months ago

Hey Tao and Co, We are completely in awe of this project. We posted it over on our site (culturenow.com/site/item.cfm?item=39521) and would be thrilled with any updates in the future. Keep up the fantastic and inspiring work.
Adrian Diaconescu 2 months ago

You had me at "We present a system..." :) Awesome!
Henk Dawson 2 months ago

I've been doing this kind of work for 20 years "by hand" using the computer and it's still a challenge for me to get it right after all these years of practice. I'd be really surprised if this can be automated as well as this demo suggests. Really surprised.
Kevin Bjorke 2 months ago

Alternative Title: "Final Nails in the Still-Creaking Coffin of Image Copyright"
David C. 2 months ago

Ariel Shamir, one of the authors listed here, is also co-author on seam carving:

youtube.com/watch?v=vIFCV2spKtg
idtdi 2 months ago

不错哦
Ivan 2 months ago

Excellent tool for Stalinist regimes, dictators etc.
Timothy Peter Robb 2 months ago

I call shop! This is so fake, you can see the pixels!
nacider 2 months ago

could be a useful tool for creating basic storyboard type composited images , but light years away from replacing a skilled compositor or mac op. The image search does appear to function and fufill the basic task of finding and blending images but in a very crude way , there is no allowance for thing such as lighting direction colour cast , shadows reflections , atomoshere, depth haze image grain matching black levels, chromatic lens effects and the field of view of each element used. there is a nearly endless list of thing that a good artist needs to take into consideration to complete a believable composite image. I'm not knocking it as it is very interesting product just explaning why it wont replace an exerperienced artist, Yet ;)
James McWilliams 2 months ago

First sensible comment on here :)... lets just go crazy and assume that you COULD automate masking/compositing entirely... why would that suddenly make artists redundant? These are technical things - do people really believe that automatically comping together a bunch of photos is the be all and end all of an compositor/artists job?? Are some people that stupid? I think most compositors would love to not sit for hours keying out elements of an image and would prefer to focus on the creative aspects that no automation can solve.
It never ceases to amaze me how people get so caught up in techno-hype. Rant over ;)
Oli 2 months ago

AWESOME!
great interface,
super thinking, great development, this can act as a plugin for google imagebank, tineye, flickr like online app. and any photobank services.
you are millionaire:)
i had to reblog it on abracadabran.posterous.com/photosketch-how-come-this-was-not-done-earlie

hope you do not mind!
Julian Zoller 2 months ago

Pretty shure its fake.

Reminds me of this video:
chilloutzone.to/video/bildbearbeitung-2-0.html
Thomas Mathai 2 months ago

When you say fake, what do you mean? That it's not professional grade compositing?

This is an academic presentation. They aren't showing you how to create a professional quality composition automatically.

They are presenting new ways of gathering and isolating images based on user defined criteria.

This technology still needs a person with skills to make it look good, it just speeds up the process.
James McWilliams 2 months ago

Great post
Tobiasz Usewicz 2 months ago

What will come up if I look for bacon + apollo13 + dancing reindeer + hungry teapot?
Mark–Daniel Legrand 2 months ago

Great interface, where can one get it?
Janet F 2 months ago

AWESOME fun and useful, too. Please inform us as to where to obtain this!
Alex Beston 1 month ago

i think its just a scholars project
Tao Chen 2 months ago

I am so excited that so many people like our work. I want to respond to some concerns about our work: It can automatically choose some images for composition, but it is not fully automatic, it still needs some simple, intuitive user interactions in composition step. You can check cg.cs.tsinghua.edu.cn/montage/ to see our paper, user studies and supplemental materials.

As this is a research project, the online software/product
is not ready for publish, we are still working on it.
Lorenzo Araneo 2 months ago

eventually we will have a world where no talent is needed, just software, here is to the downfall of the creative world! blah
Thomas Mathai 2 months ago

Wow, this sounds familiar. Guess you don't see this as a tool to help with the creative process instead of subverting it.
Lorenzo Araneo 2 months ago

this not a tool, this is a substitute for a designer, there is no "process". you scribble a few lines and it does everything for you, the process is done by this concept not the designer then again if you use this you aren't really a designer anyways. Someone who uses a Ghost Writer for a book is not a writer, someone who would use this concept is not a designer and hardly a "creative".
Thorsten Miess 2 months ago

dude? what the? seriously...speechless .
Seems to be impressive.
Philip Riley 2 months ago

Research is fun and all but then so is making art. While technically impressive you are not doing anything genuinely useful. Sure this will jolly up some powerpoint presentations for the aesthetically challenged but in real terms it's a novelty. People like making pictures they are not going to stop just because you automated the interesting bits.

Please use this technology to improve as in make actually useful the myriad masking programs out there that cannot distinguish properly between the foreground and the background. Then you might make something people can use and yourselves some money.
Thomas Mathai 2 months ago

What don't you find useful? If anything it would be like the Variation tool in Photoshop. If you have a lot of elements that you would like to composite together, but would like to see the possiblities quickly, I can see this tool to help with the process.

I'm sure Photoshop was very novel at version 1.0. Is it still novel?
Philip Riley 2 months ago

Photoshop took an arduous task and made it simple. This takes a simple task and makes it pointless
Starman 1 month ago

Wrong. It took a difficult task and made it SEARCHABLE.
Philip Riley 1 month ago

?
Talitha Shipman 2 months ago

Probably fake and illegal. Hardly anyone mentioned that second little issue with this system.
Thomas Mathai 2 months ago

Illegal in what way and where?

Most photoshop work is taking different elements and combining them together.

If you mean using other people's images from the internet, that would be considered copyright infringement. Though I think they are only using those as examples of what it can do.
Jason Drakeford 2 months ago

I can only see this used as a storyboard/comp tool. There is no way you can use these stitched images in a final product or ad since the random internet images it uses are copyrighted and opens you up to a rabbit hole of legal issues.
Thomas Mathai 2 months ago

I don't think it's designed just to grab images from the internet.
occubus jive 2 months ago

wow, can't believe all the super paranoid and judgemental comments! this is what it must have been like when scientists told people the earth was round! haha!

this is awesome on so many levels! tao chen thank you and don't let small minded people stand in your way!

for the trolls and naysayers:
do not fear what you may not understand...it makes no sense...educate yourself instead