基于樣本重要性的分布式深度學(xué)習(xí)通信優(yōu)化策略
打開(kāi)文本圖片集
Distributed deep learning communication optimization strategy basedonsampleimportance
MENG Yugong （GuangxiCodemakerInformation TechnologyCo.，Ltd.，Nanning53ooo3，China）
Abstract：Thecomputing nodes indistributeddeep learning（DDL）need to frequentlyexchangegradientdatawith the server，whichresultsinlargecommunicationoverhead.Inviewof this，aDDLcommunicationoptimizationstrategybasedon sampleimportance isproposed.Itmainlyincludesthreecontents.Theimportancedistributionofdatasamples isexploredby confirmatoryexperiments.Theimportanceofdatasamplesisevaluatedbycross-entropylossIncombinationwiththenetwork statusawarenessmechanismandbytakingtheend-to-end network delayasthenetwork status feedback indicator，thecomputing nodesareusedtoadjustthecompressionratiosofthetransmissongradientdynamicall，whichreducesnetworktraficwhile ensuringmodelconvergence，therebyimproving thetraining eficiencyofDDL.Experimentalresultsshowthattheproposed methodanimprovecommunicationeffciencyefectivelyindistributed trainingsenariosofdiferentscalesIncomparisonwith the existing gradient compression strategies，the proposed method can reduce distributed training time by up to 40% ：
Keywords：DDL;stochasticgradient descent;sampleimportance;cross -entropy；network statusawareness;dynamic compression
0 引言
深度神經(jīng)網(wǎng)絡(luò)（DNN）被廣泛用于支持越來(lái)越多的人工智能應(yīng)用，如計(jì)算機(jī)視覺(jué)、自然語(yǔ)言處理2和網(wǎng)絡(luò)優(yōu)化等。（剩余7952字）