Last week, Steve Lohr of the New York Times explored the evolving landscape of computer vision. Steve notes that computer vision – which broadly includes robotics, object detection and recognition, sequencing, as well as modeling and object interaction - is moving into the mainstream; what would have required a super computer just a decade ago is now possible on common computers of all types. At Microsoft, we believe in the judicious use of these technologies as one way Windows helps you be more creative in how you find and organize your media.
In addition to leading facial recognition technology, the latest release of Windows Live Photo Gallery includes two new technologies called Photo Fuse and Retouch. To bring these technologies to our customers, we turned to one of Microsoft strongest assets - Microsoft Research. Windows Live has a history of partnering with Microsoft Research as an efficient way to move cutting-edge innovations into mainstream products.
Photo Editing Innovation in Windows Live Essentials 2011
Photo Fuse & composite imaging
Photo Fuse started as a Microsoft Research (MSR) project, in conjunction with the University of Washington, as an attempt to solve the common problem of taking flawed group shots. Iterating over the course of several years, Photo Fuse is based on the same high-quality image matching technology found in Photosynth, Bing image search, and panoramic stitching.
Photo Fuse makes it possible to take the best parts of similar photos and fuse them together into one composite shot. For example, it’s common in a group picture for someone to have closed their eyes or looked away from the camera in a series of attempted pictures. In Photo Gallery, you can select all the pictures and Photo Fuse will guide you through picking the best slices from each photo and then merge them into one.
You can also use it for other scenarios like taking unwanted people or objects out of a scene. Or, you can get creative like Jason Moore recently did in exploring Seattle.
Here’s an example from the folks who worked hard to make Photo Fuse a reality. You can see how we combined the two photos on the left to make the one on the right.
Another related problem that had to be solved was how to effectively composite while taking into account the minor shifts in camera position between shots. Between any two given shots, people move their cameras ever so slightly. Here, MSR helped combine existing technology with panoramic stitch to remove motion gaps between the images. This image stabilization step is tricky because it needs to “see past” all of the moving people and stabilize on the background.
Retouch made easy
Retouching photos and removing unwanted artifacts is a feature usually reserved for expensive professional tools and has required tedious work to do. The traditional approach is to select a patch to clone and then blend that patch on top of the blemish you want to remove. Instead, Photo Gallery will analyze the surrounding pixels of the blemish to find out which path or patch will be the best fit. Afterwards, the patch is isolated, rotated, scaled in place, and then blended. The result is often better than what one could achieve by manually manipulating a photo – and it’s done in one click.
For example, Arwa didn’t care for the bruise on her arm in the photo below.
So she used Retouch to remove it.
(Click for a full resolution image, where you can see how imperceptible the retouching really is.)
Photo Fuse and Retouch are just two examples of how Windows is using computer vision technologies in surprising and fun ways. We encourage you to try Photo Gallery yourself and let us know what you think. (Photo Gallery is free and available to customers who use Windows Vista & Windows 7.)
Brad Weed
Group Program Manager
Windows Live Photo Gallery, Movie Maker, and Writer
More...