This paragraph clarifies what I mentioned in Part 1!

Daniel Bourke

Nov 28, 2018·1 min read

I see now, in Part 1 it’s specifically the Encoder module for the attention paper (as clearly stated by your article).

My fault for getting it wrong.

Daniel Bourke

I play at the crossroads of technology, health and art. Broadcasting from: www.mrdbourke.com

BERT stands for Bidirectional Encoder Representations from Transformers. It is basically a bunch of Transformer encoders stacked together (not the whole Transformer architecture but just the encoder). The concept of bidirectionality is the key differentiator between BERT and its predecessor, OpenAI GPT. BERT is bidirectional because its self-attention layer performs self-attention on both directions. Let’s see an example.

Dissecting BERT Part 2: BERT Specifics

1.3K

Francisco Ingham