Snapping a Picture to Connect to a Printer or Other Device

What if you could connect your smartphone to all kinds of “smart” appliances around you just by taking photos of them?

That’s the plan behind a research project called Snap-To-It, created by researchers at Carnegie Mellon University and two other schools. A Snap-To-It Android smartphone app is used to take a picture of a device like a printer or digital projector, and the system figures out which appliance you’re trying to connect to and sends your phone the right interface to use it—to, say, print out a document or upload and control a slide presentation, as the researchers demonstrate in a video.

The aim is to make the growing numbers of Internet-connected devices, many of which use different standards to communicate and may each require their own smartphone app to use, easier to interact with, especially when you’re in a new place. A paper on the project will be presented in May at the ACM CHI 2016 computer-human interaction conference in San Jose, California.

“The ability to walk into a space that I haven’t been into before and just start using equipment that I’m authorized to is really nice,” says Anind Dey, an associate professor at Carnegie Mellon and a coauthor of the paper.

Adrian de Freitas, a graduate student and the paper’s lead author, says that Snap-To-It works behind the scenes by broadcasting the photo you take to all the devices on the network in the area; a computer-vision algorithm runs for each device, comparing your photo to existing ones to figure out what you’re trying to connect to. Information about your location and the direction your phone is pointed is considered, too, in figuring out which device is the one you want. If Snap-To-It isn’t quite sure what device you’re trying to use, it will send you an ordered list of its top five guesses. The system needs to be set up with the Internet-connected appliances in advance, which requires taking pictures of a gadget and setting up which of its functions a Snap-To-It user can use.

Researchers determined that without filtering out blurry images or images snapped from odd angles, the app connected the user to the right device 82 percent of the time when using just one reference image. Now the researchers are toying with how to use smartphone sensors—a barometer, for instance—to better determine your position within a building, which would make it easier for Snap-To-It to figure out what device you’re trying to connect to.

They’re also considering other ways the system could be useful beyond enabling you to control devices—perhaps you could take a photo of a device and get more information about what it can do, or snap an image to control several devices simultaneously.