I released a softmax-free attention model at GPT-2 Medium scale (~354M params, 11.5B tokens): structural sparsity + tile-skipping kernels for long-context VRAM savings. Open weights + custom Triton kernels [R]
소프트맥스 없는 주의 모델이 공개되었습니다.
A softmax-free attention model has been released.