"Images are the best way to learn visual properties," said Abhinav Gupta, assistant research professor in Carnegie Mellon's Robotics Institute.
"[They] also include a lot of common sense information about the world. People learn this by themselves and, with NEIL, we hope that computers will do so as well."
Examples of the links that NEIL has made include the facts that cars are found on roads and that ducks can resemble geese.
The program can also make mistakes, say the research team. It may think that the search term "pink" relates to the pop star rather than the colour because an image search would be more likely to return this result.
To prevent errors like this, humans will still need to be part of the program's learning process, according to Abhinav Shrivastava, a PhD student working on the project.
"People don't always know how or what to teach computers," he said. "But humans are good at telling computers when they are wrong."
Another reason for NEIL to run is to create the world's largest visual knowledge database where objects, scenes, actions, attributes and contextual relationships can be labelled and catalogued.
"What we have learned in the last five to 10 years of computer vision research is that the more data you have, the better computer vision becomes," Mr Gupta said.
The program requires a vast amount of computer power to operate and is being run on two clusters of computers that include 200 processing cores.
The team plans to let NEIL run indefinitely.