neural_tangents.stax.PositionalEmbedding
- class neural_tangents.stax.PositionalEmbedding(value)[source]
Type of positional embeddings to use in a
GlobalSelfAttentionlayer.- NONE
no additional positional embeddings.
- SUM
positional embeddings are added to activations.
- CONCAT
positional embeddings are concatenated with activations.
- __init__()
Attributes