Their technique uses data from still photos to improve the quality of video footage, such as detail, dynamic range, exposure and lighting. Furthermore, it can perform retouching, remove unwanted objects and change surface textures by automatically interpolating changes made to a single frame.

Their video demonstrates it in glorious, trippy detail:

The research was done by GRAIL, the Graphics and Imaging Laboratory
of the University of Washington‘s Department of Computer Science and Engineering.

The technique seems to work in these stages:

1. Video and photos are captured of a scene, near-simultaneously.
2. A 3D point-cloud model of the scene is built by analysing the motion of the video footage from multiple angles – more than just stereo.
3. The high quality photo data is mapped onto this 3D model.
4. The motion of the video is recreated within the 3D model resulting in a completely algorithm-generated video.
5. Glitches and artefacts due to mapping errors or incomplete data are eliminated with their “Spacetime Fusion” technique.

If the photos are edited prior to the start of processing – for example, if an object is painted out from one of them – these changes will be mapped into the enhanced video automatically.

Some digital cameras can shoot stills and video at the same time. Therefore it should be possible to generate full HDR video at tens of megapixels entirely automatically, non-destructively and in-camera.

It’s not clear how many photos it requires to get a good result, or what kind of real-world situations would cause problems for the process.

The researchers also make it very clear that the technique only works when the subject is static – the camera can move, but the subject it is filming cannot.

But regardless, I’m pretty excited to see what happens next and I hope that some camera manufacturers and software developers are taking notice. It seems like research is fast, but commerce is slow.

The Arriflex 435 Xtreme, the last iteration of the world's most popular 35mm film camera.

Back in the 1980s, video was supposed to be the nail in the coffin for film (and look what happened to video). But thirty years later, the sound of hammering is hard to ignore…

• The three biggest camera manufacturers (Arri, PanaVision and Aaton) stopped making film cameras this year.
• This year’s threatened SAG strikes resulted in 90% of TV pilots going digital, owing to a clause in rival union AFTRA’s contracts permitting “videotaped” (i.e. digital) filming.
• The world’s main HDCAM SR factory in Sendai was destroyed in the earthquake resulting in many producers switching to “tapeless”* formats, making a bold step forward after 100 years of using physical media.
• The penetration of digital projectors in US cinemas passed the 50% mark, encouraged by the Virtual Print Fee paid to theatre owners who convert to digital projection by 2013.

This is old news for those stills photographers who still shoot 35mm. Soon filmmakers will have the same experience. Film cameras will be repaired and serviced only by specialists and enthusiasts. Lab processing options will be limited, while the costs are already sky-rocketing. Even basic post-production will rely on a dwindling population of experienced handlers, archivists and restoration technicians.

But those future filmmakers who choose to shoot on film will come to enjoy a level of respect which celluloid photographers already enjoy. It will become a speciality format and an expensive hobby, but a viable medium nonetheless for directors who want film’s unique aesthetic. As long as Fuji and Kodak keep making the stock, people will use it.

Benjamin Button was a watershed in "beautiful" digital cinematography.

Digital is past its infancy now. Unlike colour film which took more than thirty years to become the dominant format, digital has taken about twenty. The pioneer spirit has attracted many forward-thinking filmmakers who were already well-versed in the digitisation of post-production.

James Cameron and David Fincher – both directors with background in Visual Effects – have pushed digital acquisition to hitherto unimagined possibilities. I won’t dwell on Cameron’s revolutionary camera tech for Avatar, or Fincher’s championing of both Viper and RED cameras**.

What we’re really seeing is a broadening of the media palette, when film and digital are two options available to any filmmakers. Typewriters didn’t kill writing, and cassettes didn’t kill the live scene.

We’re also seeing a highly technical fractionation of the acquisition workflow: digital is not just about exposing and processing. Now the final image quality depends on many subtle distinctions in highly abstract concepts like sensor design, codecs, colour curves, data wrangling and conforming strategies. The connection between any of those stages and the final result is very hard to trace.

Of course, while digital continues to be an exciting and often surprising way of shooting, no one believes today’s digital standards will bear any comparison to whatever will be king in another 30 years.

But whatever the next thing is, it remains to be seen.

———————

This article was inspired by Debra Kaufman’s article on Creative Cow who also wrote the Viper interview with DoP Claudio Miranda.

* It’s an oddly anachronistic tendency to name new technologies after whatever it is they lack from their antecedents, like calling radios “wireless”.

** RED’s status as the technology front runner looks assured – unexpectedly for a startup that released its first product, the RED One, less than 5 years ago. In September, Soderbergh’s Contagion became the first commercial feature shot and released on the RED’s Epic “Tattoo” prototypes. The now commercially released Epics are currently lensing The Hobbit, The Girl With The Dragon Tattoo and The Amazing Spider-man. RED lovers are also expecting an announcement on November 2nd about the very long awaited Scarlet.

(I’m not forgetting the importance of Art Direction here. This is the major factor in any film’s colour palette. In fact, in recognition of the challenges that colour represented, separate Best Art Direction Academy Awards were given for black and white and colour films between 1940-1966.)

But these days we shoot digitally and grade digitally. Consequently nearly all decisions about colour made during production can be changed in post-production. Digital grading makes it possible to retrospectively alter everything.

We have greater control over colour than ever before and this has been as much of a revolution as surround sound, 3D, HD and photo-real computer-generated effects. But it is very much overlooked and this is not without certain qualifications.

The colour grade of a film is critical in selling it. The grade is obvious from the first frame of the trailer and at the first glimpse of the poster. It imparts a sense of the style, production values and tone quicker than any other aspect of the marketing arsenal. I love it when a film has a complex interesting grade – although it must be said that the most stylised grades are often the most banal films.

But everything digital has its drawbacks. The resolution of compressed video images is relatively poor compared to film. Not just in terms of how sharp the image is, but also in terms of its “colour space” which essentially means how well the colour spectrum is represented in the digital image.

In nature, colour is a continuous spectrum, but representing it digitally means the spectrum is broken up. The richness we expect of the smooth colour gradients we see in skies and sunsets is lost. We know this implicitly – photos of beautiful sunsets never match the real thing. The image below is an exaggerated representation of these kind of discontinuites.

The problems inherent in digitisation have more subtle and compounding effects on the recorded image too. For example, many digital cameras tend to favour capturing the detail in greens and blues (rather than reds) because our eyes are also more attuned to those colour ranges. Consequently garish bold colours, which are relatively less common in nature, are afforded less of the “colour space” and consequently appear more jagged and pixellated in digital images.

Finally, many digital displays can’t accurately depict certain colours. The optical illusion below called “The Eclipse of Mars” shows you the colour cyan – never seen before on a PC screen! (Click for larger version.)

(This post was inspired by a recent post on Pete Emslie’s blog which made me think about how we think about colour in movies.)

For further reading, check out Graeme Nattress’ excellent and comprehensive analysis of colour in digital video.

However the new trailer for GoW3 has attempted to do the same and is receiving something of a backlash. Certainly Ars Technica’s Ben Kuchera isn’t impressed.

Of course it wasn’t made by Fincher. This new trailer bears none of the visual fluency of the first: the broken doll face, the hesitant marine, the long slow buildup, the rain like tears…

Also Mad World’s lyrics seem so pointedly about computer games – “the dreams in which I’m dying / are the best I’ve ever had” – that it’s surprising no one did it before.

The new one has none of this. It’s explosive throughout, counterpointed by music that bears little relation to the picture, and the ending is somehow both cheesy and an anticlimax.

There is an underlying truism of filmmaking here: you can’t create emotion with music, but music can bolster what emotion is already there.

The shallow, hollow, clones that most first-person shooters are today clearly don’t justify this. But is this a sign of a yearning that they should move us? I think so.

It’s hard to imagine how monolithic AAA game series like GoW can accomodate any emotional undertow, but there are many games out there which have managed it successfully – Craig Keeble’s blog has a good list.

I still believe it’s possible for a game to be dramatic and narrative and it’s up to game makers to learn the best way to do this, not just by copying how movies do it, but by finding the medium’s own unique lexicon.

He talks about some software called Epagogix which can predict the box office success of a movie based solely upon its screenplay.

Regardless of the director or stars attached or the marketing spend, its predictions are accurate to within 6%.

Furthermore, the makers of the software can suggest changes to the screenplay and give the exact monetary value of those changes.

Give the hero a sidekick – add $15 million.

Give the hero a full-face mask costume – lose $30 million.

In particular he mentions the Nicole Kidman vehicle, “The Interpreter” which was a notable turkey after going through several screenwriters and directors before ending up in theatres as a mediocre and forgettable blockbuster.

Epagogix made several suggestions to change the script including relocating the opening scene from an imaginary African state, and changing the entire twist at the end. Their advice was ignored, and the movie vanished after a couple of weeks.

However they do concede that such a formula is really only effective at this upper echelon of $30m+ budgeted Hollywood flicks. The more arthouse, auteurish fare below that threshold seem to carry an expectation of an eccentric lack of formula.

The presentation was based on Gladwell’s New Yorker article “The Formula” which you can read in full here.

Many thanks to @teh_aimee and @chelfyn for arranging another awesome NN!

UPDATE: the fullscreen version of the Keynote presentation is here, with commentary recorded during a rehearsal: http://vimeo.com/26697596

Here’s the blurb:

Ever since we started recording people on film, another bunch of people have wanted to change that recording, to reframe and reinterpret its indelible truth into something that better serves the purpose of the filmmaker.

This process is film editing. Usually it’s a force for good – making complex arguments accessible, making protracted events succinct, or simply for making a dull story more compelling.

But the reversioning of the truth which is so intrinsic to editing, can be deceptive in the wrong hands. It started as a tool for propaganda and continues to manipulate audiences today.

In this talk, I demonstrate and explain some common tricks of the editing room – and make examples of some of their most nefarious abusers…

The fourth Pirates movie sold more tickets in 2D than 3D. This clearly worried Michael Bay and Paramount whose Transformers: Dark of the Moon is about to hit the same markets.

The film’s $195m budget has reportedly been inflated by Bay’s demands on the technology, involving “over 4000 crew” and the most “technically demanding sequences ever shot. And shot in 3D“. Bay says they’ve optimised the grade on 3D prints to improve their brightness, sharpness and contrast.

Now they’re on marketing overdrive to get people to see the 3D version or, better still, IMAX.

Part of this campaign has included a revealing letter to projectionists decrying the “dark dingy” projection that has become a widespread problem in cinemas:

The main reason* is the polarising filters found in 3D projector lenses which cut out a significant amount of the light projected.

How much, though, depends on who you ask…

• 20% according to Sony, the manufacturers of the projectors.
• 50% according to Roger Ebert’s poignant blog entry on this subject
• 83% according to the Boston Globe’s source at rival specialist projection company Boston Light & Sound

What’s worse is cinemas are not bothering to swap out these lenses when showing a 2D movie. Consequently every film, whether 2D or 3D, could now be subject to a loss of light.

The same Boston Globe article showed that changing lenses was not straightforward:

Opening the projector alone involves security clearances and Internet passwords, “and if you don’t do it right, the machine will shut down on you.’’

Sony responded that it’s a 20 minute job to change the lens for someone with the “correct expertise”. Although it’s unclear if such experts include the average projectionist, and it’s also unclear why this kind of hardware DRM is even necessary.**

So it seems there are several factors at play that will be setting the quality of our movie-going experience for the foreseeable future.

• Hollywood and the multiplexes are still betting heavily on 3D, despite diminishing returns.
• Multiplexes are operating a rapid turnaround policy to accomodate more screenings and parallel programming of 2D and 3D versions, which means they don’t swap out the lenses.
• The benefits of digital projection are now revealing their oh-so-predictable downside: DRM paranoia.

It’s a complex issue and it will be interesting to see where it goes.

———————————————————

* Another reason for the darker images is that theatre owners are keeping their expensive projector bulbs underpowered to extend their working life. Any benefit of this is questionable – or at best negligible – but this tactic certainly isn’t new.

** Having been a projectionist myself, I can tell you it is no easy job. In the multiplexes, typically one person will be operating several projectors simultaneously with the schedule staggered so they can physically get between screens on time. In some megaplexes, projectors are arranged in a single long corridor so you can get there by bicycle. In these situations, a 20 minute changeover is impossible.

AllThingsD has a decent explanation of how this works:

Lytro’s camera works by positioning an array of tiny lenses between the main lens and the image sensor, with the microlenses measuring both the total amount of light coming in as well as its direction.

This suggests a major compromise in image quality, so perhaps it’s the advent of ultra-high resolution CCDs that is making this possible now. The simulations show immense promise.

For narrative media, this creates a fascinating opportunity…

Traditional photography imposes a single point of focus for an audience. As movie-goers we are used to that restriction and it is even an aesthetically pleasing effect.

For editors this adds yet another dimension to our concerns.

It used to be that we just worried about the two spatial dimensions (x, y) to tweak framing.

Then there’s the third dimension, time (t), which is the main preoccupation of traditional editors, further complicated by time-stretching tools like Twixtor.

Then there’s digital colour manipulation – another three dimensions (R, G, B).

And now we can set focus in the edit – our third spatial dimension (z), and sixth overall.*

For interactive media, we as viewers don’t want to be guided like that. In fact we want to tell the camera where to look. By combining Lytro-like image capture with retinal tracking, interactive films will be able to adjust the focal point as we glance around the screen.

This also means more flexible 3D and an evolution that will take us beyond the current 150-year old paradigms.

Regardless of Lytro’s fortunes, the theory behind it is unquestionably revolutionary.

——————————

* Editors who do compositing will tell you we’ve been working in the z-dimension for a long time.

A mere two months after it was previewed at NAB, Final Cut Pro X has been released to some excoriating reviews. I can’t remember ever seeing such a poorly received product release.

The reviews speak for themselves (as of 23 June 2011). Many of the five star reviews seem to be from new users, or from people expecting that “features will be added” in forthcoming updates. Opinions are polarised, but the consensus is that this is an incomplete and deeply flawed product.

What’s strange is there seemed to be no great demand for a new version of FCP. Why rush out a product that is “not ready for professional use”? (Larry Jordan’s view, which he subsequently adjusted.)

Admittedly it was clearly time for a 64-bit version, but to rebuild it from scratch, and mess with the basic paradigm of FCP’s editing when it is already in a battle for credibility against Avid, seems reckless.

Walter Biscardi has written a comprehensive and accurate summary of the main omissions, which I won’t repeat here, but he concluded saying that’s he “done with this game” – meaning he’s done with Final Cut.

Premature? Perhaps. Future updates will certainly fix the widely reported bugs and glitches, but it seems unlikely they will restore these missing features.

It is clear, however, that Apple’s ill-judged release has alienated their FCP user base and done considerable damage to their Pro credentials. The $299 price point should have been warning enough.

At NAB, people joked that it looked like iMovie Pro. The joke, it seems, was on us.

I’m a film and video editor, and there are many editors with fantastic and insightful blogs online already.

Many of these focus on new techniques and technology which are constantly reshaping the landscape of post-production. Other blogs are more “editorial” with a combination of opinion, criticism and inspection.

My goal is a little of both.

For me, the magic of cinema is the magic of editing. Post-production is where it all comes together. It’s an exciting place, and I’d like to share a little of that excitement with you.