What do you picture when you hear… “WILMAAA!!?”
This is Sandra Tsing Loh with the Loh Down on Science.
When describing a scene, humans create a visual image of it in their mind’s eye. Could computers do the same?
Meet Tanmay Gupta and researchers from the University of Illinois Urbana-Champaign. They want to teach computers to visualize written words. Their test? Have a computer draw Flintstones cartoons — from a description!
First they described a three second clip from the Flintstones in a short sentence. Then, they had a computer read the text and make a totally random short video. After comparing the generated clip to the original, the computer tweaked its work and checked to see if it got better. Then it tried again… millions of times!
After feeding the computer descriptions from the entire series, the researchers tried text that WASN’T in the originals. For example, the computer read, “Fred is wearing a red hat walking into the living room.” What did the computer do?
It created a new clip of Fred, wearing a red cowboy hat, walking into the living room!
This work helps computers better understand human language by giving them a VISUAL understanding of sentences.
And that makes them say…”Yabba Dabba Doooo!”