...As a separate tool tho. No need to try to bung it into puppet tool.

Look at Quartz Composer...

Hmm, this basically looks like a bunch of stuff that I and others have been doing with Quartz Composer (a tool that is included without extra charge with Apple OS X).

Take a look at my motion tracking and Augmented Reality videos, and the augmented reality group here at vimeo for more ideas along these lines...

I think it is really cool that Adobe and the University of Washington have gotten together to try to achieve this, I just want to point out that a handful of people are already doing this with fairly easy to use developer tools, that aren't quite as pricey as AE, and that are more expandable by the user.

Lol, I think this is way impressive... it is just that I keep seeing these clips like this from a major corporation in conjunction with major universities, and I am thinking "hello, this stuff is already being done by artists out there" and if anything they are just packaging up ideas that are already out there and making it into something to sell for big $$$.

I inherently apologize for this post, because I don't like writing "downer" stuff, but I can't help myself!

Love the ideas though!

Hey, thanks for your comments! My main research goal was to streamline the user interface and workflow, which I think is still fairly complex in most tools, including Quartz Composer. I've sent you more detailed thoughts directly, but didn't want to seem too defensive in the comments here. ;-)

Oh, I see what you are doing with the interface and it is AMAZING! I didn't mean to imply that any of the tools can't use major improvement in the interfaces.

It is a thing of having to use a lot of Adobe products and having come to loathe them (totally unjustified... I CAN be objective, lol), in a silly Coke vs. Pepsi kinda way, and also from having been bitten by Adobe licensing schemes in the workplace.

I know from our private discussion that we are coming from exactly the same place, and it is cool to meet your acquaintance. I'm really impressed with what you are doing.

I blinked quite a few times, but I think that IS what's in the video, isn't it? Anyway, thanks for reprinting it in the comments so people can just click on it!

Yeah, video manipulation seems a little wordy but video editing means something else specific. Gotta be a better word...

Anyway, in answer to your comment, no, the current system doesn't track across cuts.

There is totally a better word: Tesseracting. Because what you are developing at Adobe is a prototypical system for frameless, hypercubist cinema. I write a blog on the future of cinema and your tool fits nicely into my forecasting of the end of the celluloid-influenced paradigm of flat video frames, transforming them into hypercubes.

Of course, the really exciting development would be for Adobe to get a partner (like Google or Sony) to work with data cloud INPUT on the front end so that the video is already laden with the metadata you need to particlize the image into discrete objects.

Your interface is brilliant because it directly visualizes the objects in the frame as discrete elements of cinematic reality, creating a highly intuitive moment where the user "reaches into" the scene.

If there is any way I can get involved in your research or beta test please let me know! I am totally impressed that Adobe has decided to go in this direction and that you have had the opportunity to publish this out here on Vimeo where the most enthusiastic creative videomakers can give it some feedback!

I will be blogging about this video soon, but in the meantime I think you would enjoy reading my theory and learning more about why I use the words "hypercubist" and "tesseracting" -- they are part of my larger theory of continuity for hypercubist cinema which I call Quantinuity (quantum + continuity).

Thanks so much!

quantumcinema.blogspot.com/2008/11/quantinuity-towards-theory-of.html

Yes, the tracking and grouping are described in my UIST paper, which is linked from the Adobe URL at the end of the video. (Here it is again, for easy clickability.)

adobe.com/technology/graphics/video_visualization_and_interaction.html

The tracking is done using the Particle Video method of Sand and Teller (cited in the paper), and the grouping is done using my own algorithm.

However, it's done in an offline process, whereas you are looking for something real-time, right? You might want to look into a KLT tracker. I've used this one, which runs on the GPU (but it's a little finicky):

cs.unc.edu/~ssinha/Research/GPU_KLT/

Lol

I also see a cottage industry that could develop in evidence tampering; with video files manipulated to match evidence needed in cases.

"Sure, I can place your suspect at the place and time on this surveillance tape. It doesn't matter what time he was actually there."

Natch, I always see the worst in what could happen, but it looks like a fun tool to use once you remove my cynical observations. :D

Check out the video on the Adobe website, which includes an example of this kind of interaction! In that video I showed the ability to get to a product website by clicking on objects in the video. I left this example out of the Vimeo post because it's SOOOO commercial it will surely be controversial. (Though I should point out that in my implementation, it's completely user-driven, the annotations fade in only when the mouse is in the video player window.)

That sounds like exactly what I was talking about. Being able to interact and find more information.

I couldn't get the video to load on the adobe site... it just spins. Firefox and IE. Is this the correct link: adobe.com/technology/graphics/video_visualization_and_interaction.html ?

Would also be cool to play a role in how the video ends... by clicking certain things you could potentially change the storyline, have a fork in the video but not something as cheesy or obvious as a choose your own adventure.

Thanks! I think the puppet video is a little too dry and technical to post here as-is, so I would like to re-cut and re-narrate it like this one. But it'll be a month or more before my schedule opens up enough to do that. Patience!

Well, what if you replace the word "absorbed" with "funded?" Still feel the same way? I couldn't have done this work without support from Adobe.

BTW, since you mentioned Photosynth, co-developed by colleagues at UW and MS, "absorbed" is an even less apt adjective, since they've actually published source code for the back end: phototour.cs.washington.edu/bundler/

I haven't gotten around to publishing my source code yet, but if I find the time to clean it up I might do something like that in the future... Probably the only hurdle will be replacing the code that ISN'T Adobe's!

I don't have the legal chops to comment much, but I hope that your ability to do things like release the source code is protected by your funding arrangement.

So, Adobe had some of their people working on this project too, and that portion of the code belongs to Adobe?

I agree that researchers have to eat, but I hate it when private funds turn research discoveries into "private property."

nice, the "bundler" link even has a Linux binary! so no evil to see here..

The technical details are in the UIST paper at the following URL:

adobe.com/technology/graphics/video_visualization_and_interaction.html

The tracking is done using the Particle Video method of Sand and Teller (cited in the paper). It's a variant of optical flow.