Hello,

We noticed you're browsing in private or incognito mode.

To continue reading this article, please exit incognito mode or log in.

Not an Insider? Subscribe now for unlimited access to online articles.

  • xiaojie2020
  • Business Impact

    How to mod a smartphone camera so it shoots a million frames per second

    Researchers have developed an image-processing trick that allows any electronic camera to take ultra-high-speed images–but only in black and white.

    Back in the 1930s, a young electrical engineer named Harold Edgerton began experimenting with electronic flash photography at MIT. He famously used it to freeze the action of everyday objects such as a stream of water, a bursting balloon, and a bullet hitting an apple.

    Edgerton went on to pioneer a wide range of single and multiflash techniques (as well as many other imaging techniques). Many of these are now available as standard on popular cameras and smartphones.

    Since then, shutter speeds have edged higher. Some iPhones can snap single images with a shutter speed of 1/10,000th of a second. Specialist cameras can record images at the rate of 40,000 per second. But this kind of gear is complex and expensive, so these techniques are inaccessible for most people.

    Today, that looks set to change thanks to the work of Sam Dillavou at Harvard University and a couple of colleagues, who have developed a technique that allows ordinary electronic cameras and smartphones to record images at the breathtaking rate of millions of frames per second.

    First some background. Modern cameras use electronic pixels to record the light that hits them. These pixels register the intensity, usually on a scale of 16 levels. This is determined by the bit level of pixels. So during an exposure, each pixel can record 14 shades of gray between black and white.

    But for many types of imaging this level of detail is unnecessary—all that’s needed is a simple black or white image. In that case, the extra gray detail is redundant.

    That gave Dillavou and colleagues an idea for teasing single images apart into multiple shorter exposures.

    Here’s how. Imagine a white wall covered by a black curtain. Now open the curtain from left to right to reveal the wall, and record the movement in a single exposure.

    arxiv.org

    During the exposure, the pixels on the left-hand side of the image first record the black curtain but then the white wall as the curtain moves across the frame. This averages out to a light gray.

    By contrast, pixels recording the right-hand side of the image see the black curtain for most of the exposure. Only at the end of the exposure is the white wall revealed. In that case, the average is a dark gray.

    Similarly, the pixels in the middle of the image will record a mid-gray. And the final image appears as a graduated gray, which represents the movement of the curtain.

    The key idea of Dillavou and colleagues is that this single image can be thought of as many black-and-white snapshots of the moving curtain superimposed on each other. Indeed, the number of images that can be teased out of a single image depends on the number of bit levels the pixels record.

    Armed with this insight, they developed a mathematical technique to extract these images from a single image, an idea they call the virtual frame technique. In this case, it can create 16 images of the curtain moving across the wall. 

    This technique has the potential to turn almost any electronic camera into a high-speed marvel. The team has tested it by comparing the virtual frames with those taken by a high-speed camera recording at 40,000 frames per second.

    Image of group of woman taking photo with smartphone

    They placed a stretched rubber sheet in front of a beam splitter that sent the light to both cameras, so both had the same view of the sheet. The team set up the lighting so that the rubber sheet appeared black while the background appeared white. They used a scalpel to nick the rubber sheet and recorded the way it split with both cameras as the fracture raced across the sheet.

    Finally, the team used the virtual frame technique to extract high-frame-rate images and compare them with those from the high-speed camera shooting at 40,000 frames per second. A video of this comparison is downloadable here.

    The results show that the virtual frame technique has huge potential. First, both sets of images show a similar fracturing process, which confirms the merit of the approach. But it also has a number of significant advantages. It records 1.3 megapixels in each image, compared with the standard camera’s 60 kilopixels. This dramatically increases the field of view and therefore the distance over which the fracture can be recorded. 

    The virtual frame rate is much faster, too. “The effective frame rate achieved using [the virtual framing technique] is 1MHz,” say Dillavou and co.

    They go on to study how applying the new technique to a variety of different cameras increases their frame rates. The numbers are eye-opening. “As an example, the Nikon D850 records virtual frames at 16MHz while maintaining a resolution over 50Mpx,” they say. Even an iPhone X is capable of recording at up to a million frames per second using this technique.

    There’s little reason to think this can’t be managed easily, since all the processing is done later. “This enhanced virtual frame rate does not require any change in the operation of the camera,” say the team.

    However, there are some important limits to the types of phenomena that can be recorded in this way. First, the object of interest must be black against a white background (or vice versa). Any grays in the scene will ruin the results.

    And the phenomenon itself must be what Dillavou and co call monotonic. In other words, any change during an exposure must proceed from black to white (or vice versa) but not back again. In the example with curtain, the virtual frame technique works if the curtain moves in one direction but not if it moves back and forth.

    Nevertheless, the virtual frame technique is a significant advance for photography. It has the potential to turn the smartphone camera in your back pocket into a high-speed recorder, the likes of which would have impressed even Doc Edgerton.

    Ref: arxiv.org/abs/1811.02936 : Virtual Frame Technique: Ultrafast Imaging with Any Camera

    Be the leader your company needs. Implement ethical AI.
    Join us at EmTech Digital 2019.

    Register now
    Image of group of woman taking photo with smartphone
    More from Business Impact

    How technology advances are changing the economy and providing new opportunities in many industries.

    Want more award-winning journalism? Subscribe to Insider Plus.
    • Insider Plus {! insider.prices.plus !}*

      {! insider.display.menuOptionsLabel !}

      Everything included in Insider Basic, plus the digital magazine, extensive archive, ad-free web experience, and discounts to partner offerings and MIT Technology Review events.

      See details+

      Print + Digital Magazine (6 bi-monthly issues)

      Unlimited online access including all articles, multimedia, and more

      The Download newsletter with top tech stories delivered daily to your inbox

      Technology Review PDF magazine archive, including articles, images, and covers dating back to 1899

      10% Discount to MIT Technology Review events and MIT Press

      Ad-free website experience

    /3
    You've read of three free articles this month. for unlimited online access. You've read of three free articles this month. for unlimited online access. This is your last free article this month. for unlimited online access. You've read all your free articles this month. for unlimited online access. You've read of three free articles this month. for more, or for unlimited online access. for two more free articles, or for unlimited online access.