反向聚焦細(xì)粒度多模態(tài)語義對齊的視頻字幕模型
打開文本圖片集
中圖分類號：TP391 文獻標(biāo)志碼：A 文章編號：1001-3695（2025）07-009-1986-08
doi：10.19734/j. issn.1001-3695.2024.11.0492
Abstract：Existingvideocaptioningoftenintroducemultimodal informationtoassistmodelsinextractingcriticalandfinegrained details fromcomplex anddynamic visual content.However，these methods tendtooverlook thesemantic gapscaused by representationaldiferencesamong modalities.Tobridgethesegaps，facilitateefectivecross-modalalignmentandeficientfusion，andenancetheextractionoffine-grainedsmanticinformatio，thispperproposedareverse-focusfingranedultio dal semanticalignmentforvideocaptioning（RM4Cap）.Thismodelcombinedanimage-textpaircorpusand facilitatedsemanticalignmentbetweenvideoandimage，indirectlyaligningvideorepresentationswithtextintheimage-textpairs.Anditdesignedareverse attention focusing algorithm to suppress redundant scene informationwhile highlighting inconspicuous objects and their interactions.Experimentsconductedonthe MSVDand MSRVTTdatasetsshow thatthe model significantlyoutperforms existing methods in metricssuch as CIDErand BLEU-4.It efectivelyresolves thealignmentchallenges andredundancy issues in multimodal fusion，further demonstrating its ability to narrow the cross-modal semantic gap.
Key words：video captioning；multimodal； reverse attention；semantic alignment； semantic gap
0 引言
視頻字幕是一個連接視覺和語言并將視覺內(nèi)容以自然語言描述的跨模態(tài)任務(wù)。（剩余21688字）