Enhancing and Experiencing Spacetime Resolution with Videos and Stills on Vimeo

00:00

534

See all Show me

6. Enhancing and Experiencing Spacetime Resolution wi…

by pro

3 years ago

5. Appearance stabilization in PhotoTourism

by pro

3 years ago

4. Interactive Video Cutout

by pro

3 years ago

3. Using Photographs to Enhance Videos of a Static Scene

by pro

3 years ago

2. GradientShop: A novel approach to image and video …

by pro

3 years ago

1. Painterly rendering using GradientShop

by pro

3 years ago

Followup work to the "Using Photographs to Enhance Videos of a Static Scene" project:
vimeo.com/1513129.

In this new work we focus on using photographs to increase the spatial resolution and temporal resolution (i.e., frame rate) of videos containing moving objects (i.e., dynamic scenes).

Project website:
grail.cs.washington.edu/projects/enhancing-spacetime/

Credits

78 Likes

daralthus
2 months ago
Orhan Nasufovski
2 months ago
Stelios Hadjiperis
3 months ago
Travis Newton
6 months ago
Boris Yakubchik
9 months ago
patrick
11 months ago
Juho Vainio
11 months ago
Alex Fatkulin
11 months ago
Paul Sucharski
1 year ago
Gabriel Netto
1 year ago

See all 78 likes

Tags

Michael Crumpton 3 years ago

Quite Amazing. One big advantage I can see is that shooting low rez video with high rez key frames would take a lot less disk space and the equipment needed is less expensive than for a HD or Imax camera. The heavy lifting required to do the interpolation can be done after the video is shot on a regular computer.
dalas verdugo 3 years ago

Wow, I would actually want to reproduce some of those morphing artifacts. That's a very cool stylistic effect, even though to you it is undesired.
Louis du Mont 3 years ago

Makes me wonder what sort of computing power would be able to calculate the footage and stills in real-time. It would amazing to make a hybrid camera with this sort of processing built in, allowing image clarity and frame rate way beyond the native capability of the cameras sensor system.
Jared Lyon 2 years ago

This video as well as the early piece both completely amaze me. Keep up the great work!
Ian Johnson 2 years ago

Are the high resolution images necessary for increasing the temporal resolution, if you already have high quality SD or HD video? As a video editor, I run into those artifacts all the time using optical flow plugins for slow motion. Your method would have application now, even without a hybrid camera if it can improve on current methods of interpolation between video frames.
pro 2 years ago

No. High resolution images are not necessary for increasing the temporal resolution of the video.
The cricket(@6:50) and soccer(@7:51) results in the video did not involve any photographs.

Recently there was also a paper at Siggraph 2009 which specifically dealt with the problem of increasing video frame-rate:
"Moving Gradients: A Path-Based Method for Plausible Image Interpolation"
.cs.columbia.edu/~dhruv/
David Coiffier 2 years ago

What makes me skeptical about all of this demonstration, is that lores material is just derived from hires material, providing perfect match between hi & lo res. But that's a case that won't ever exist in real life. What about taking stills apart from video shooting, that won't perfectly match in space or colors ? That would be more interesting, and more related to real cases...
Or is there something I didn't get ?
pro 2 years ago

The paper showed a couple of examples captured by putting a video camera next to a point and shoot. These examples were meant to be an evaluation of our system on real-world data. The one of these examples can be seen at the very end of the video - @ 8:04.

Also, the experiments involving low-res material derived from high-res material is a realistic test case for hybrid cameras. These cameras can simultaneously capture low-res video with a few photographs every so often. Now you have the problem of converting the lowres-video into high-res using the intermittent photos that do not match perfectly in spacetime when the camera and the scene objects are moving. This is exactly the problem we address in this paper. To test our system we take a high-res video, down-sample and add noise to simulate low-res video and keep a few of the high-resolution frames from the original video to simulate the photographs. Then we compare our reconstruction to the original video and see what the error rate is.
Diamond Jef 2 years ago

keep on going, i am waiting for commercial program able to use it in my home, what You are doing is amaizing !!!!!!!!!!!!!!!
Jim Smith 2 years ago

Total BS. A decent video editor such as Sony Vegas or Final Cut Express can do all of this with far less trouble and effort.

If you are that concerned about your video, why not get a decent camera and learn something about making videos to start with? None of that is terribly expensive and works far better than this nonsense.

Finally, the narrator has an annoying voice and delivery. She also obviously is working from a script and doesn't know her subject or even the script very well.
Vladimir Tomin 1 year ago

"A decent video editor such as Sony Vegas or Final Cut Express can do all of this with far less trouble and effort."

So Final Cut can take data from photos and increase resolution of video using it? I'm not saying you're wrong, I'm just asking you to teach me how to do this in Final Cut Pro. Step-by-step would be awesome.
Gabriel Netto 1 year ago

"A decent video editor such as Sony Vegas or Final Cut Express can do all of this with far less trouble and effort."

I work with postproduction. No program does this yet. Not Final Cut (express!!!) not (decent?) vegas, not After effects, not Nuke, not Combustion, Flame, or any software at all...
Try something simple on your vegas: Take 24p footage and convert it to 29,97 for example. Then make a slowmotion with the result footage. Even with Final Cut Pro 7 you'll get issues. Now enhance quality with some photos taken by another camera... take a look at Pro's "Using Photographs to Enhance Videos of a Static Scene" and tell me if there's ANY software capable of this today!
Gabriel Netto 1 year ago

By the way, incredible, Pro! I'm following your research long time ago.
I would like to know if your system is capable of consistent Interlaced to Progressive conversion.
Keep it on!
Jim Smith 1 year ago

I disagree. I no longer use Vegas but I have excellent results with Final Cut and there are many third-party addition that do all of these things very will.

So you maybe shilling for a product but that doesn't make any of what you post true.
OutlandishMediaCorp 2 months ago

Have there been any updates to this? Your research area is very exciting.
Orhan Nasufovski 2 months ago

Who's still sitting on this technology? Get it out there!