All the sessions from Transform 2021 are available on-demand now. Watch now.
Researchers from Google AI (formerly Google Research) and Toyota Technological Institute of Chicago have created ALBERT, an AI model that achieves state-of-the-art results that exceed human performance. ALBERT now claims first place on major NLP performance leaderboards for benchmarks like GLUE and SQuAD 2.0, and high RACE performance score.
On the Stanford Question Answering Dataset benchmark (SQUAD), ALBERT achieves a score of 92.2, on General Language Understanding Evaluation (GLUE) benchmark, ALBERT achieves a score of 89.4, and on ReAding Comprehension from English Examinations (RACE) benchmark, ALBERT gets a score of 89.4%.
ALBERT is a version of Transformer-based BERT that “uses parameter reduction techniques to lower memory consumption and increase the training speed of BERT,” according to a paper published on OpenReview.net Wednesday and on arXiv on Thursday. The paper was published alongside other papers being considered for publication as part of the International Conference of Learning Representations, which will take place in April 2020 in Addis Ababa, Ethiopia. ICLR will be the first international AI community conference held in Africa.
“Our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs,” the paper reads.
ALBERT is the latest derivative of BERT to claim a top spot in major benchmark tests. In late July, Facebook AI Research introduced RoBERTa, a model that achieved state-of-the-art results, and in May, Microsoft AI researchers introduced Multi-Task Deep Neural Network (MT-DNN), a model that achieved top marks in 7 of 9 GLUE benchmarks.
Each of the models achieves performance that outpaces average human performance.
In other Transformer-related news, Hugging Face, a startup whose PyTorch library for easy use of major Transformer models like BERT, Open AI’s GPT-2 and Google’s XLNet today made that library available for TensorFlow. PyTorch-Transformers has seen more than 500,000 Pip installs since the start of year, Hugging Face CEO Clément Delangue told VentureBeat.
VentureBeatVentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact. Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:
- up-to-date information on the subjects of interest to you
- our newsletters
- gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
- networking features, and more