Full handwritten notes: https://web.goodnotes.com/s/TL37bZae7UNxYWtBRCxmLR#page-1 (for some reason the note is coming blank 😞)

Motivation:

1. Most videos don’t have the description or metadata
2. Youtube has:
   1. title
   2. description
   3. likes / dislikes
   4. hashtags

Methods:

Video

1. video → caption → embedding
2. video → Key frame → embedding
3. video → Key frame → caption -> embedding

Audio

1. video → ASR → embedding
2. video → ASR → translation → embedding

OCR

1. video → key frame → OCR → text → embedding
2. video → key frame → OCR → text → summary → embedding

Description

1. Description → embedding
2. Description → summary → embedding

Team Members:

• Video, Audio, OCR, Text, Information Retrieval

Data:

Multivent:

• ~2400 data
• Professional → Edited → Raw (Easy to Hard to retrieve)

InternVid:

• ~10M data
• Large array of categories
• Multilingual

Scale / Multivent 2.0:

Train

• 108k InternVid video as distractor
• 1400 train queries
• No multivent video here

Sub-Train / Train-2k

• 2000 videos, including all relevant videos
  • Used for quick prototyping

Test

• 110k from Intervid as needle & distractor + 2400 multivent data
• 2546 Test queries → 2400 multivent + ~146 InternVid

Query Creation:

Multivent

• MultiVENT has description or title or tweet as query, which is really long
• MultiVENT has a doc. associated with each video
  • 255 base event queries
  • 884 specific event queries

InternVID:

• InternVid has title
  • From title or desc. base query
  • specific event query from transcribed speech and OCR

Evaluation:

• Primary metric: NDCG@10
  • Score 0.3 - 0.4 - decent
  • Score 06 - 0.7 - good

Pre-Scale Baseline

1. Video → PySceneDetect → 10 frames → Clip
2. Video → PySceneDetect → 10 frames → OCR → PaddleOCR → Clip
3. ASR → Whisper → Clip
4. Desc → Clip

After-Scale:

1. Video → 16 uniform frames → OCR → ICDAR → original + google translate → original + LLAMA  summarized google translate → ColBERT
2. Video → 16 uniform frames → Siglip → maxSim
3. Desc → original + google translate → ColBERT
4. ASR → whisper → original + google translate → original + LLAMA summarized google translate →  ColBERT
5. Weighted fusion based on if the video is professional, edited or raw (using trained classifier)
6. query → query decomposition
7. Rerank with MonoT5

Takeaways:

1. Late Fusion better
2. Summarize OCR & STT text representation
3. SigLIP best as a vision encoder rather than CLIP
4. Combination of Translation & Original help
5. No single modality is sufficient
6. The more text we can get is better
7. LLM summarization works better than raw text

Papers to Read:

1. https://aclanthology.org/2022.lrec-1.739/
2. https://arxiv.org/abs/2104.00650
3. https://arxiv.org/abs/2106.11097
4. https://arxiv.org/abs/2212.03191
5. https://arxiv.org/abs/2305.18500