Facebook open-sources data set for code search AI benchmark

Facebook AI researchers created code search data sets that utilize information from GitHub and Stack Overflow. The release contains an evaluation data set of 287 Stack Overflow question-and-answer pairs including code snippets, as well as a search corpus of code snippets from nearly 25,000 Android repositories on GitHub.

The Neural Code Search Evaluation Data Set was published on arXiv in August and revised Wednesday. The Stack Overflow data comes from the Stack Overflow Data Dump, while the GitHub Rest API supplied the rest of the data.

“We intend for this data set to serve as a benchmark for evaluating search quality across a variety of code search models,” Facebook AI said in a blog post.

The paper also shares results of two AI models created by Facebook as a test run of the corpus and data set.

Code search is meant to give developers a way to surface chunks of programming language code using natural language. A number of code search initiatives are underway such as GitHub’s Semantic Code Project and machine learning initiative and startups like recent Y Combinator graduate Metacode.

In other developments in AI for software developers, this spring Google Brain introduced AI that predicts code based on previous edits.