Home / AI Research / On-Policy Morphing into Hybrid Attention Models
ONAI Research
On-Policy Morphing into Hybrid Attention Models
by ByteDance-Seed · Disen Lan, Xuanda Wang, Jianbin Zheng, Huafeng Kuang, Wei Tang, Yu Cheng
Hybrid attention models combine full and linear attention to achieve a better quality--efficiency trade-off. Converting pretrained Transformers into such models requires identifying which layers should retain full attention under a prescribed budget. We identify two off-policy mismatches in FlashMorph's layer selection: (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery. We introduce FlashMorph2, an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end to end using their own hidden states, and…
Upvotes13
Comments2
Published17 Nov 2026
More from ByteDance-Seed
| SESeed 2.1 TurboAI Model APIs · #115 | $0.50 / $2.50 |
| UIUI-TARS-1.5-7BAI Models · #1358 | 228.5K |
| SESeededit AppAI Apps · #392 | 451 |
| SESeed-2.0-LiteAI Model APIs · #208 | $0.25 / $2.00 |
| SESeed-2.0-MiniAI Model APIs · #213 | $0.10 / $0.40 |
| BABagelAI Apps · #850 | 222 |