Jeff LiuExperiment Reports
Back to all reports
Deep learning

Normalization and regularization in Transformers

· Pang Liu

Original report

PDF · 14 pages · 2.7 MB

Read PDFDownload

Loading PDF…

An IWSLT2017 French-to-English study comparing normalization placement and type, dropout, warmup and weight decay.

Report date
Original authors
Pang Liu
Report language
English · 14 pages

Main observations

Training stability, BLEU and translation examples are examined together. The report notes repetitive outputs in some Pre-LN examples that aggregate metrics do not fully reflect.

Scope and limitations

BLEU and validation curves do not replace inspection of actual translations. Conclusions are bounded by the reported models and training conditions.

Source

View the original source and its recorded version

Preserved as a historical record. The report describes the software, conditions and observations of its original experiment.