PixelTone: A Multimodal Interface for Image Editing

G. Laput, M. Dontcheva, G. Wilensky, W. Chang, A. Agarwala, J. Linder, and E. Adar (CHI 2013) [Pre-PhD]

Photo editing can be a challenging task, and it becomes even more difficult on the small, portable screens such as camera phones that are now frequently used to edit images. To address this problem we present PixelTone, a multimodal photo editing interface that combines speech and direct manipulation. We observe existing image editing practices and derive a set of principles that guide our design. In particular, we use natural language for expressing desired changes to an image, and sketching to localize these changes to specific regions. To support the language commonly used in photo-editing we develop a customized natural language interpreter that maps user phrases to specific image processing operations. Finally, we perform a user study that evaluates and demonstrates the effectiveness of our interface.



This work was completed during an internship at Adobe Research in San Francisco. It was demoed live at the Adobe Tech Summit 2013, and was published+presented at the CHI Conference (CHI 2013) in Paris, France.


  Download the paper (PDF)

Pixeltone: A Multimodal Interface for Image Editing
Gierad Laput, Mira Dontcheva, Gregg Wilensky, Walter Chang, Aseem Agarwala, Jason Linder, and Eytan Adar
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13)



Gizmodo, "Adobe’s Developing a Brilliant Photo Editing App You Can Just Talk To"
NBC, "Voice-controlled photo app PixelTone: Shades of 'Blade Runner'"
Discovery Channel, Featured on the Daily Planet's digit@l Segment
PetaPixel, "PixelTone: A Futuristic Image Editor That Lets You ‘Shop Photos Using Your Voice"
Pop Photo, "PixelTone: A Voice Controlled Photo Editing App for iPad,"
Xataka, "Sácame más delgado en esta foto" así es el retoque de imágenes de PixelTone,"
Tec Mundo, Brazil, "Adobe desenvolve editor de fotos que pode ser comandado por voz,"
Tech Genius, Italy, "Adobe sta sviluppando un Photoshop a comandi vocali?,"
Zive, Czech Republic, "PixelTone: takto budeme ovládat Photoshop a další za pár let Více na,"
Sina News, Hong Kong, "PixelTone,"
Adobe TV, via Featured Videos on Peek
Adobe Leaders: Breakthrough Innovation, “Tell an Image What to Do,”