The historical record of the Civil War is patchy at best, but Kurt Luther, an assistant professor of computer science at Virginia Tech, is doing his best to fill in a few of the gaps. In August 2018, he and a team of collaborators at the Virginia Center for Civil War Studies and Military Images magazine created a website — Photo Sleuth¬†— that allows users to upload and tag photos of Civil War-era soldiers and, with the help of a facial recognition algorithm, connect them to profiles with detailed military records.

The initial database contained about 15,000 portraits from public domain sources, and in the months since its launch, more than 4,000 volunteers have contributed over 8,000 photos.

“Typically, crowdsourced research such as this is challenging for novices if users don’t have specific knowledge of the subject area,” said Luther, who was inspired to create Photo Sleuth in 2013 after spotting a Civil War-era portrait of Oliver Croxton, his great-great-great uncle, in a Heinz History Center exhibit.

Photo Sleuth adopts a “haystack” architecture of sorts, whereby newly uploaded photos contribute to a library that’s successively narrowed down. Users scan in the fronts and backs of photos and add metadata like inscriptions, format, and visual clues such as coat color, shoulder straps, and insignias — tags that link to search filters which prioritize the most likely matches. Then, the aforementioned facial recognition algorithm maps up to 27 distinct “facial landmarks” to identify possible matches.

After Photo Sleuth’s backend IDs the likeliest matches, users inspect them with a comparison tool and link them with biographical details from public sources, including the Sailors Database and National Park Service Soldiers. According to Luther, more than 600 users contributed over 2,000 Civil War photos to the website in the first month, about half of which¬†were identified. Of the roughly 100 unknown photos linked to specific soldiers, an expert analysis found that over 85 percent were “probably” or “definitely” correct.

That’s barely a dent in the estimated four million Civil War-era images that may exist in the historical record, but Luther — who’s scheduled to present his research surrounding Photo Sleuth at the Association for Computing Machinery’s Intelligent User Interfaces conference in Los Angeles this month — is undeterred. “The step-by-step process of tagging visual clues and applying search filters linked to military service records makes this detective work more accessible,” he said, “even for those that may not have a deeper knowledge of Civil War military history.”

The Photo Sleuth project has the support of the National Science Foundation, and won Microsoft’s $25,000 Cloud AI Research Challenge and the Best Demo Award at the Human Computation and Crowdsourcing 2018 conference in Zurich, Switzerland.