A step-by-step tutorial for implementing Transformer architectures from scratch in PyTorch.

🚧 This code is under rapid development. 方法及模型正在快速补充中。 🚧

1. 阅读代码，理解模型结构的理论实现
2. 根据代码中的提示，尝试实现模型结构
3. 对比 solutions 目录下的代码，检查代码是否正确

1. Read the code for each architecture, understand its theoretical basis.
2. Try to implement the code with the hint in the code.
3. Compare your implementation with the provided solution.

This repository provides detailed, well-documented implementations of Transformer architectures. Each implementation includes:

• 📝 Step-by-step explanations
• 💡 Detailed comments for every key component
• 🔍 Mathematical derivations where necessary
• ⚡ Working examples and demonstrations

transformer_from_scratch/
├── vanilla_transformer/          # 经典 Transformer 结构
│   ├── attention.py             # 注意力机制实现
│   ├── encoder.py              # 编码器实现
│   ├── decoder.py              # 解码器实现
│   └── transformer.py          # 完整模型
│
├── attention_variants/          # 注意力机制变体
│   └── linear_attention       # 线性注意力
│
├── efficient_transformer/      # 轻量级/高效架构
│   └── performer              # Performer
│
├── vision_transformer/        # 视觉 Transformer
│   ├── vit                   # Vision Transformer
│   └── swin                  # Swin Transformer
│
└── SOLUTIONS                # 答案

Contributions are welcome! Please feel free to:

• Add new implementations
• Improve existing explanations
• Suggest new architectures

This project is licensed under the MIT License - see the LICENSE file for details.