Neural Attention Archives - Deep Learning Reviews

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Stan Kriventsov
February 10, 2021
Natural Language Processing / Neural Attention
0 Comments

Review of paper by William Fedus, Barret Zoph, and Noam Shazeer, Google Brain, 2021.

Modern deep learning models, especially in natural language processing, usually strive to achieve better accuracy by increasing the parameter size of the model (often combined with training on larger datasets), which comes at a huge computational cost. In this paper, in order to achieve better computational efficiency, the authors divide the fully connected layers in their Transformer model’s blocks into sets of many alternatives (experts), whereby only one expert is chosen for each given input in each layer. This provides the opportunity to increase the size of the model as desired (within the available memory constraints) by increasing the count of experts while maintaining a constant computational complexity per input token.

Point Transformer

Review of paper by Hengshuang Zhao¹, Li Jiang², Jiaya Jia², et al, ¹University of Oxford, ²The Chinese University of Hong Kong, 2020

The authors develop a neural attention layer for working with 3D point data and implement it in a Point Transformer network that shows new state-of-the-art results (in some cases by a significant margin) on a number of standard 3D benchmarks.

Attention Augmented Differentiable Forest for Tabular Data

Review of paper by Yingshi Chen, Xiamen University, 2020

The author has developed a new “differentiable forest”-type neural network framework for predictions on tabular data that has some similarity to the recently suggested NODE architecture and employs squeeze-and-excitation “tree attention blocks” (TABs) to show performance superior to gradient boosted decision trees (e.g. XGBoost, LightGBM, Catboost) on a number of benchmarks.

An Image is Worth 16×16 Words: Transformers For Image Recognition At Scale

Review of paper by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al, Google Research, 2020

This paper develops a novel way of using Transformer neural attention models for visual recognition tasks.

Big Bird: Transformers for Longer Sequences

Stan Kriventsov
August 2, 2020
Neural Attention / Natural Language Processing
0 Comments

Review of paper by Manzil Zaheer, Guru Guruganesh, Avinava Dubey et al, Google Research, 2020

In this paper, the authors present a Transformer attention model with linear complexity that is mathematically proven to be Turing complete (and thus as powerful as the original quadratic attention model) and achieves new state-of-the-art results on many NLP tasks involving long sequences (e.g. question answering and summarization), as well as genomics data.

Linformer: Self-Attention with Linear Complexity

Stan Kriventsov
June 28, 2020
Neural Attention / Natural Language Processing
0 Comments

Review of paper by Sinong Wang, Belinda Z. Li, Madian Khabsa et al, Facebook AI Research, 2020

This paper suggests an approximate way of calculating self-attention in Transformer architectures that has linear space and time complexity in terms of the sequence length, with the resulting performance on benchmark datasets similar to that of the RoBERTa model based on the original Transformers with much less efficient quadratic attention complexity.

No more pages to load

Older Posts →