Transformers without Normalization: A Quiz on Dynamic Tanh (DyT) Practice Questions - Machine Learning, Transformers, Normalization Layers

Question 1

What is the primary function of Dynamic Tanh (DyT) in Transformers?

Accepted Answer

A) To replace normalization layers by emulating their behavior with a tanh-like function

Answer

B) To introduce a new type of activation function for better gradient flow

Answer

C) To compute batch statistics for normalization

Answer

D) To reduce the number of parameters in the model

Question 2

Which observation inspired the development of DyT?

Accepted Answer

B) Layer normalization often produces tanh-like, S-shaped input-output mappings

Answer

A) Layer normalization produces linear mappings in all cases

Answer

C) Batch normalization is more effective than layer normalization in Transformers

Answer

D) Normalization layers are unnecessary in shallow networks

Question 3

What is the mathematical formulation of DyT?

Accepted Answer

B) DyT(x) = γ * tanh(αx) + β

Answer

A) DyT(x) = tanh(αx) + β

Answer

C) DyT(x) = tanh(x) / α

Answer

D) DyT(x) = α * tanh(x) + γ

Question 4

In which tasks did DyT demonstrate comparable or superior performance to traditional normalization layers?

Accepted Answer

C) Across diverse settings including recognition, generation, and self-supervised learning

Answer

A) Only in supervised image classification

Answer

B) Only in language modeling

Answer

D) Only in small-scale models

Question 5

What is the role of the learnable parameter α in DyT?

Accepted Answer

B) It scales the input to adjust the range of activations

Answer

A) It acts as a bias term

Answer

C) It replaces the need for gradient descent

Answer

D) It computes the mean and variance of the input

Question 6

Which of the following is NOT a benefit of DyT over traditional normalization layers?

Accepted Answer

C) Better performance in all cases without hyperparameter tuning

Answer

A) Reduced computational overhead

Answer

B) No need to compute activation statistics

Answer

D) Improved training and inference speed

Question 7

How does DyT handle extreme values in the input activations?

Accepted Answer

B) By squashing them non-linearly via the tanh function

Answer

A) By linearly scaling them down

Answer

C) By ignoring them during training

Answer

D) By clipping them to a fixed range

Question 8

What was the default initialization value for α in non-LLM models?

Accepted Answer

B) 0.5

Answer

A) 0.1

Answer

C) 1.0

Answer

D) 2.0

Question 9

Which of the following models was NOT tested with DyT in the paper?

Accepted Answer

C) ResNet

Answer

A) Vision Transformer (ViT)

Answer

B) ConvNeXt

Answer

D) LLaMA

Question 10

What key insight does DyT provide about normalization layers in deep networks?

Accepted Answer

B) Their primary role can be emulated by simpler, more efficient operations

Answer

A) They are indispensable for all deep networks

Answer

C) They should always be combined with batch normalization

Answer

D) They are only useful in convolutional networks