AWS and i14 Deploy Laplace 3B Linear Attention Model on Trainium2
i14 and AWS ported the Laplace 3B linear attention model to AWS Trainium2 using native PyTorch support. Its recurrent state stays at 29.5 MB per sequence, keeping inference memory fixed regardless of context length.
i14 collaborated with AWS to run the Laplace 3B linear attention model on AWS Trainium2 using TorchNeuron Native. By replacing softmax attention with Gated Linear Attention, the recurrent state remains at 29.5 MB per sequence. On a single trn2.48xlarge instance, the model achieved about 120,000 training tokens per second and served 11,544 output tokens per second across 1,280 concurrent sequences.
Key takeaways
- Laplace 3B uses Gated Linear Attention to hold recurrent state at 29.5 MB per sequence, changing training attention scaling to O(T) and per-token inference to O(1).
- On one AWS trn2.48xlarge instance, the model trained at approximately 120,000 tokens per second and served 11,544 output tokens per second across 1,280 concurrent sequences.
Source：AWS Machine Learning Read the original ↗
Prepared, checked and automatically published by AI; not individually reviewed by a human.