2025DeepSeek V2混合專家語言模型

:]405.04434v5DeepSeekV2DeepSeekV2236Btoken21B128KtokenDeepSeek?V2DeepSeek?V2DeepSeek?V2點可在/deepseek?ai/DeepSeek?V2上找到。（二（a（bDeepSeek

(LLM（Anthropic2023Google2023OpenAI，2023LLM2022DeepSeekV221B（1MLA（2(FFN)架構,2024)DeepSeekMoEDeepSeek?V圖圖（DeepSeekAI2024（SFT2024（GRPODeepSeek?V2（RL）。DeepSeek?V2DeepSeek?V2DeepSeekV2DeepSeek?V2節(jié)省了42.5%的訓練成本93.3%的KV緩存5.76DeepSeek?V2Chat(SFT)和 {[，；{[，；{{{{ DeepSeek?V2Chat(RLDeepSeek?V2Chat(RL202438.92023AlignBenc（Liu2023英文開放式對話測試表明DeepSeek?V2Chat(RL)在開源聊天模型中具有頂級性能AlignBenchDeepSeekV2MLADeepSeekMoEDeepSeekV2Lite于開源社區(qū)157億個參數(shù)24億個DeepSeekV22017）（FFNFFNMLAFFN2024））DeepSeekV2DeepSeek67（DeepSeekAI2024年）。2017)(MQA(Shazeer2019(GQA(Ainslieetal.2023)DeepSeekV2注意（MLA）MLA準MHA首先產生q q=k=v=MLAq

，]=q

q[k,1;k,2;...;，]=k[v,1;v,2;...;v，]=

問鉀

)五

，=

，

五， ckv

∈ MLAMLA點的鍵和值3直觀地

c= q c

′×

′al.,2024)RoPE與低秩不兼容RoPE

RoPEMLA[q,1;q,2;...;q，]=q=RoPE(c),kRoPE(h),

問o，=

問

u

[o,1;o,2;...;o，

分別表示；RoPE(·)表示應用RoPE矩陣的運算；[·;·MLA ≈2.25DeepSeekMoEFFNal.2024）DeepSeekMoE如GShard（Lepikhin等人，

=u+

FFN(

，F(xiàn)FN(

（u

，∈Topk({, },

0，=Softmax

和FFN

DeepSeekMoEDeepSeekV2token top?K路由大致相同的良好性能。2017）（LDevBal2021Lepikhi2021LExpBal=1

= 1（Token

衡因子；V2的訓練過程中{E1,E2,...,E}并將每組部署在單個設備上?衡損失計算如下：LDevBal=2

′ |E|

′=

tokenLCommBal=3

′′=′′=

tokentoken67（DeepSeekAI20248.1Ttokentokentoken1260512128DeepSeekV2總參數(shù)token激活21B。訓練超參數(shù)AdamW（LoshchilovHutter，2017）10.95，weight_decay0.10.9，2使用預熱和步（DeepSeek?AI20246練了大約90%的token0.3162.4×10－41.0225B個token2304逐漸增加到9216，然后在剩余的訓練中保持92164K，并在8.1Ttoken上訓練=token泡流水線并行(Qietal.,2023)8路專家并行(Lepikhinetal.,2021)和ZeRO?1數(shù)據并行(Rajbhandarietal.,2020)（NIAHMLA（Dao2023）GPUNVSwitchInfiniBandYaRN32132K(NIAH)DeepSeekV2在高達128K的所有上下文窗?長度上都表現(xiàn)良好。20202023202320192020201820222017等人2019年2019)2019)(Cui2019)。2020al.2020）20192021）202120212023代碼數(shù)據集包括HumanEval(Chenetal.,2021)MBPP(Austinetal.,2021)和CRUXEval(Guetal.,2024)2023)AGIEval（DeepSeek?AI2024NaturalQuestionsDROPMATHGSM8KHumanEvalMBPP、CRUXEvalBBHAGIEvalCLUEWSCCMRC和CMath同標記器的模型進行公?在表2DeepSeek?V2與幾個具有代表性的開源模型進行了比較DeepSeek67B(DeepSeek?AI,2024)（我們之前的版本）Qwen1.572B(Baietal.,2023)、LLaMA370B(AI@Meta,2024)和Mixtral8x22B(Mistral2024)DeepSeekV267B（1Qwen1.5MoE BBH(EM3MMLU(Acc.5DROP(F13HellaSwag(Acc.10PIQA(Acc.0WinoGrande(Acc.5RACE?Middle(Acc.5NaturalQuestions(EM)5次AGIEval(Acc.)0032245C?Eval（認證5C3兒童（賬戶0CCPM0基準（公制DeepSeekQwen1.5Mixtral基準（公制DeepSeekQwen1.5MixtralLLaMA Qwen1.5（28x22B2DeepSeekV2基準測試中DeepSeek?V2Mixtral8x22BlBV2（3與3B2n22DeepSeekV22024在預訓練期間從未接觸過SFT數(shù)據。訓練成本DeepSeek?V2tokenFLOPDeepSeek67BDeepSeekDeepSeek67B可以節(jié)省42.5%的訓練成本。2024Zhao2023）67BDeepSeek?V2的即時輸入吞吐量超過每秒100K個token。（DeepSeekAI2024V22epoch510－6DeepSeek?V2Chat(SFT（MMLUARC）DeepSeek?V2Chat(SFT(IFEval)（Zhouetal.，2023）202391202441LiveCodeBench(Jainetal.2024MT?Bench(Zhengetal.2023)AlpacaEval2.0(Duboisetal.2024AlignBench(Liuetal.2023)Qwen1.572BChatLLaMA?3?70BInstructMistral?8x22BInstructChat（RL（GRPOo2024）GRPO J()= (),{

—

（）

=1

（+

（+

和123和RAM壓力（1（22023GPU標準基準測試評估DeepSeek?V2Chat(SFTDeepSeek?V2ChatDeepSeekV2DeepSeek?在與其他模型的比較中DeepSeek?V2Chat(SFTQwen1.572BChatDeepSeekV2ChatDeepSeek?DeepSeekV2DeepSeekV2DeepSeekV2DeepSeek?V2Chat(RLMistral8x22BInstructQwen1.572BChatLLaMA370BInstructDeepSeek?V2Chat(RLMT?BenchDeepSeekV2DeepSeekV2Chat(SFTQwen1.55NaturalQuestionsMMLU5ARC253IFEval0HumanEval0MBPP3CRUXEval?I?COT28CLUEWSC5C?Eval5CMMLU58x22B指示v0.1Qwen1.572B聊天LLaMA370B表3|2(SFT)2)InstructMT?BenchAlpacaEvalDeepSeekV24.0我們的模型在支持中文的頂級法學碩士中名列前茅DeepSeek?V2ChatChat（RL （DeepSeek?V2Chat7.737.807.668.297.997.338.618.678.477.148.368.108.288.378.538.337.617.817.418.177.568.538.137.307.347.268.178.048.137.477.567.377.597.816.937.426.847.006.677.887.886.585.756.438.027.827.588.006.317.937.386.627.607.264.864.974.747.386.727.287.765.355.686.826.715.817.937.517.206.917.777.767.566.837.037.28Chat4.654.544.714.814.757.016.516.767.477.077.344.746.765.846.977.297.186.924.716.636.906.366.746.646.593.923.764.076.596.226.053.755.296.713.756.285.817.133.926.265.965.503.393.163.615.584.914.526.666.256.082.682.293.074.464.314.264.504.634.916.976.756.316.195|GPT?4?0613AlignBench2024Zhou20242022)BBHDeepSeekV2DeepSeekV2DeepSeekV2模型中頂級的性能MoEDeepSeekV2AI@MetaLlama3模型卡2024年/meta?llama/llama3/blob/main/MODEL_CARD.mdJ.AinslieJ.Lee?ThorpM.deJongY.ZemlyanskiyF.Lebrón和S.SanghaiGqa從多頭檢查點訓練廣義多查詢變壓器模型arXiv:2305.1324520232023ucing?claudeJAustinAOdenaMNyeMBosmaHMichalewskiDDohanEJiangCCaiMTerryQLearXivarXiv:2108.077322021。WSWuBXuJXuAYangHYangJYangSYangYYaoBYuHYuanZYuan、JXYZCZhouJZhouXZhouTZhuQwenarXivarXiv:2309.166092023Y.BiskR.ZellersRLBrasJ.Gao和Y.ChoiPIQA用自然語言推理物理常識AAAI人工智能會2020Press2020M.Chen,J.Tworek,H.Jun,Q.Yuan,HPdeOliveiraPinto,J.Kaplan,H.Edwards,Y.Burda,N.G.Brockman,A.Ray,R.Puri,G.Krueger,M.Petrov,H.Khlaaf,G.Sastry,P.Mishkin,B.Chan,S.N.Ryder,M.Pavlov,A.Power,L.Kaiser,M.Bavarian,C.Winter,P.Tillet,FPSuch,D.Cummings,M.Plappert,F.Chantzis,E.Barnes,A.Herbert?Voss,WHGuss,A.Nichol,A.Paino,N.Tezak,J.Tang,I.BabuschkinSBalajiSJainWSaundersCHesseANCarrJLeikeJAchiamVMisraE.MorikawaA.RadfordM.KnightM.Brundage、M.MuratiK.MayerP.WelinderB.McGrewD.AmodeiS.McCandlishI.Sutskever和W.ZarembaCoRR，abs/2107.033742021年。AI2推理挑戰(zhàn)arcCoRR，abs/1803.05457201/abs/1803.05457K.CobbeV.KosarajuM.BavarianM.ChenH.JunL.KaiserM.PlappertJ.TworekJ.HiltonR.Nakano等人arXiv:2110.141682021《2019年自然語言處理實證方法會議和第9屆國際自然語言處理聯(lián)合會議(EMNLP?IJCNLP5883–58892019年11言學協(xié)會doi：10.18653/v1/D19?1600URL/D19?1600DeepseekmoeCoRRabs/2401.060662024URL/10.48550/arXiv.2401.06066。.FlashAttention?22023年。LLMDROPNAACLHLT20192019）201910.18653/V1/N19?1246URL/10.18653/v1/n19?1246。和alpacaevaltransformers2021URL/abs/2101.03961。LGaoSBidermanSBlackLGoldingTHoppeCFosterJPhangHHeAThiteNNabeshimaThePile用于800GBarXivarXiv:2101.000272020年。gemini2023google?gemini?aiCruxeval2024DHendrycksCBurnsSBasartAZouMMazeikaDSongJSteinhardtarXivarXiv:2009.033002020D.HendrycksC.BurnsS.KadavathA.AroraS.BasartE.TangD.Song和J.SteinhardtarXiv預印本Hai?llmhai?llmC.HooperS.KimH.MohammadzadehMWMahoneyYSShaoK.Keutzer和A.GholamiKvquantabs2401.180792024S.胡涂韓何崔龍鄭方黃MinicpmarXiv:2404.063952024C?EvalLivecodebencharXiv:2403.079742024M.JoshiE.ChoiD.Weld和L.ZettlemoyerTriviaQA用于閱讀理解的大規(guī)模遠程監(jiān)督挑戰(zhàn)數(shù)據集R.Barzilay和M.?Y.Kan編輯的《計算語言學協(xié)會第55屆年會論文集（第1卷）1601–16112017年7doi10.18653/v1/P17?1147/T.KwiatkowskiJ.PalomakiO.RedfieldM.CollinsAPParikhC.AlbertiD.EpsteinI.PolosukhinJ.DevlinK.LeeK.ToutanovaL.JonesM.KelceyM.ChangAMDaiJ.UszkoreitQ.Le和S.Petrov。7452–4662019doi10.1162/網址/10.1162/tacl_a_00276W.KwonZ.LiS.ZhuangY.ShenL.ChengCHYuJEGonzalezH.Zhang和I.Stoicapagedattention2023ACMSIGOPS29G.LaiQ.XieH.LiuY.Yang和EHHovyRACE：來自考試的大規(guī)模閱讀理解數(shù)據集M.PalmerR.Hwa和S.Riedel編輯的《2017年自然語言處理實（EMNLP2017），丹?哥本哈根，2017年99?11785?7942017doi：10.18653/V1/D17?1082URL/10.18653/v1/d17?1082。DLepikhinHLeeYXuDChenOFiratYHuangMKrikunNShazeerZChenGshard2021H.LiY.ZhangF.KotoY.YangH.ZhaoY.GongN.Duan和T.Baldwin。CMMLU測量中文的大規(guī)模多任務語言理解。arXiv預印本arXiv:2306.09212,2023CcpmX.劉X.雷S.王Y.黃Z.馮B.文J.程P.KeY.徐WLTamX.張、L.孫H.王J.張M.黃Y.東和J.唐。Alignbench大型語言模型的中文對齊基準測試。CoRR，abs/2311.187432023doi10.48550/AI.Loshchilov和F.HutterarXivarXiv:1711.05101201年ChatGPT2022arXiv:2303.087742023L.OuyangJ.WuX.JiangD.AlmeidaC.WainwrightP.MishkinC.ZhangS.Agarwal、K.Slama352773027744202B.PengJ.QuesnelleH.Fan和E.ShippoleYarn大型高效上下文窗?預印本arXiv:2309.000712023S.RajbhandariJ.RasleyO.Ruwase和Y.HeZero針對訓練萬億參數(shù)模型的內存優(yōu)化SC20國際高性能IEEE2020C.RiquelmeJ.PuigcerverB.MustafaM.NeumannR.JenattonASPintoD.Keysers和N.Houlsby2021NeurIPS20212021proceedings.neurips.cc/paper2021/hash/48237d9f2dea8c74c2a72126cf63d933?Abstract.html。KSakaguchiRLBrasCBhagavatulaYChoi2019DeepseekmatharXiv:2402.033002024.，N.ShazeerA.MirhoseiniK.MaziarzA.DavisQVLeGEHinton和J.Dean合專家層ICLR2017OpenR2017/Roformer568:1270632024年。2019M.SuzgunN.ScalesN.Sch?rliS.GehrmannY.TayHWChungA.ChowdheryQVLe、EHChiD.Zhou等人arXiv預印本arXiv:2210.092612022年。VaswaniNShazeerNParmarJUszkoreitLJonesANGomezI.波羅蘇欣統(tǒng)的進展，2017年30月。JWeiYTayRBommasaniCRaffelBZophSBorgeaudDYogatamaMBosmaDZhouDMetzlerarXivarXiv:2206.076822022。Cmath202L.H.X.LC.YY.K.D.YuC.YuY.TianQ.DongW.LiuB.ShiY.Cui，R.WangW.XieY.LiY.PattersonZ.TianY.ZhangH.Zhou，S.Liu,Z.Zhao,Q.Zhao,C.Yue,X.Zhang,Z.Yang,K.Richardson和Z.LanCLUE中文語言理解評估COLING2020）20202020doi10.18653/V1/2020.COLING?MAIN.419URL/10.18653/v1/2020.coling?main.419。A.YoungB.ChenC.LiC.HuangG.ZhangG.ZhangH.LiJ.ZhuJ.ChenJ.ChangYi01doi：10.18653/v1/p19?1472URL/10.18653/v1/p19?1472。Y.ZhaoC.LinK.ZhuZ.YeL.ChenS.ZhengL.CezeA.KrishnamurthyT.Chen和B.KasikciAtom：abs/2310.191022023CZhengMHuangASunChid用于完形填空測試的大型中文成語數(shù)據集AKorhonenDR2019）2019728821778–7872019doi10.18653/V1/P19?1075URL/10.18653/v1/p19?1075。W.?LY.S.Z.Y.Z.LinZ.LiD.LiEP2023AGIEvalabs/2304.06364202310.48550/arXiv.2304.06364C.Zhou,P.Liu,P.Xu,S.Iyer,J.Sun,Y.Mao,X.Ma,A.Efrat,P.Yu,L.Yu36，2024J.Zhou,T.Lu,S.Mishra,S.Brahma,S.Basu,Y.Luan,D.Zhou,和L.HouarXiv:2311.079112023利陳二航李芳云林富麗羅光博郝冠輝丁華建新麗月張明傳張

峰梁文軍高文濤張小碧小涵王軒譚一元劉ZZ卓書李子涵王

耀輝李毅鄭雨辰數(shù)據注釋北風輝李JL蔡佳RLSS

坤查玉婷嚴震張DeepSeekV2Lite512DeepSeekMoEMoE264DeepSeekV2LiteDeepSeek7BDeepSeek7BDeepSeekMoE16B6.9B16.4B2T C6|DeepSeek?V2?LiteDeepSeekMoE16BDeepSeek7B超參數(shù)設置為1=0.92=0.95、weight_decay=0.1在前2K步中從0升至最大值0.31690token4.2設置為

4K SFTDeepSeekDeepSeekMoEDeepSeekDeepSeekMoE7B聊 16B聊6.9B16.4B2T C7|DeepSeek?V2?LiteDeepSeekMoE16BDeepSeek7BDeepSeek?V2?Litec ，]=q ，]=q=RoPE(c問q，[k,1;k,2;...;

c，]=k

，

[v,1;v,2;...;

，=v

問鉀

o，=

)五

u [o,1;o,2;...;

，

87BMHAGQAMQA1.33TMHAMLA14%基準（公制#組BBH（歐洲515.8110.615715.6BBH（歐洲5GPT?4?1106Qwen?110B?聊天2?13B12U12U2RCompReasOvrAccReas.ScoreLiveCodeBench202391DeepSeekV2Chat（RLLiveCodeBench年4月1日期間。(12|AGIEvalB3°C Q）A(i)AAAZnot((A))=not((True))=notTrue=FalseFalse。True）=非（True）=False。BZQ不不（不（假）是A(i)（False）=（A）”AA（False=AZ14|BBH C答案 答案 案 16|C3OPTIONS17|CCPM35A，351200A=100900:A10(x+y)=300+8(x+y)x+y=150750肺胸膜C.胸乳頭肌D19|CMMLU20|CMRC201821|DROPOPTIONSMax個售價為2*100=200美分225?200=25美分25。Rommel5x315Jenny15217TimJenny17=1212717171*151530154545Angela和包裹加起來為27IndrasA24|GSM8K25|HellaSwag的定義域。$x?2$5x>0$\det\mathbf{A2$$\det\mathbf{B12,$$\det(\mathbf{A$\det(\mathbf{A\mathbf{B}(\det\mathbf{A})(\detmathbf{B}(2)(12\boxed{24}$$24$。我希望它是正確的。Terrell2012$2\cdot12\cdot20=480$n=30n$磅的重物。n&=480/30=\box