LanguageBind commited on
Commit
84bed6c
1 Parent(s): 499d15b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +345 -0
README.md CHANGED
@@ -1,3 +1,348 @@
1
  ---
2
  license: mit
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
  ---
4
+ <p align="center">
5
+ <img src="https://s11.ax1x.com/2024/02/01/pFMDAm9.png" width="250" style="margin-bottom: 0.2;"/>
6
+ <p>
7
+ <h2 align="center"> <a href="https://arxiv.org/pdf/2310.01852.pdf">【ICLR 2024 🔥】LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment</a></h2>
8
+ <h5 align="center"> If you like our project, please give us a star ⭐ on GitHub for latest update. </h2>
9
+
10
+
11
+
12
+ ## 📰 News
13
+ * **[2024.01.27]** 👀👀👀 Our [MoE-LLaVA](https://github.com/PKU-YuanGroup/MoE-LLaVA) is released! A sparse model with 3B parameters outperformed the dense model with 7B parameters.
14
+ * **[2024.01.16]** 🔥🔥🔥 Our LanguageBind has been accepted at ICLR 2024! We earn the score of 6(3)8(6)6(6)6(6) [here](https://openreview.net/forum?id=QmZKc7UZCy&noteId=OgsxQxAleA).
15
+ * **[2023.12.15]** 💪💪💪 We expand the 💥💥💥 VIDAL dataset and now have **10M video-text data**. We launch **LanguageBind_Video 1.5**, checking our [model zoo](#-model-zoo).
16
+ * **[2023.12.10]** We expand the 💥💥💥 VIDAL dataset and now have **10M depth and 10M thermal data**. We are in the process of uploading thermal and depth data on [Hugging Face](https://huggingface.co/datasets/LanguageBind/VIDAL-Depth-Thermal) and expect the whole process to last 1-2 months.
17
+ * **[2023.11.27]** 🔥🔥🔥 We have updated our [paper](https://arxiv.org/abs/2310.01852) with emergency zero-shot results., checking our ✨ [results](#emergency-results).
18
+ * **[2023.11.26]** 💥💥💥 We have open-sourced all textual sources and corresponding YouTube IDs [here](DATASETS.md).
19
+ * **[2023.11.26]** 📣📣📣 We have open-sourced fully fine-tuned **Video & Audio**, achieving improved performance once again, checking our [model zoo](#-model-zoo).
20
+ * **[2023.11.22]** We are about to release a fully fine-tuned version, and the **HUGE** version is currently undergoing training.
21
+ * **[2023.11.21]** 💥 We are releasing sample data in [DATASETS.md](DATASETS.md) so that individuals who are interested can further modify the code to train it on their own data.
22
+ * **[2023.11.20]** 🚀🚀🚀 [Video-LLaVA](https://github.com/PKU-YuanGroup/Video-LLaVA) builds a large visual-language model to achieve 🎉SOTA performances based on LanguageBind encoders.
23
+ * **[2023.10.23]** 🎶 LanguageBind-Audio achieves 🎉🎉🎉**state-of-the-art (SOTA) performance on 5 datasets**, checking our ✨ [results](#multiple-modalities)!
24
+ * **[2023.10.14]** 😱 Released a stronger LanguageBind-Video, checking our ✨ [results](#video-language)! The video checkpoint **have updated** on Huggingface Model Hub!
25
+ * **[2023.10.10]** We provide sample data, which can be found in [assets](assets), and [emergency zero-shot usage](#emergency-zero-shot) is described.
26
+ * **[2023.10.07]** The checkpoints are available on 🤗 [Huggingface Model](https://huggingface.co/LanguageBind).
27
+ * **[2023.10.04]** Code and [demo](https://huggingface.co/spaces/LanguageBind/LanguageBind) are available now! Welcome to **watch** 👀 this repository for the latest updates.
28
+
29
+ ## 😮 Highlights
30
+
31
+ ### 💡 High performance, but NO intermediate modality required
32
+ LanguageBind is a **language-centric** multimodal pretraining approach, **taking the language as the bind across different modalities** because the language modality is well-explored and contains rich semantics.
33
+ * The following first figure shows the architecture of LanguageBind. LanguageBind can be easily extended to segmentation, detection tasks, and potentially to unlimited modalities.
34
+
35
+ ### ⚡️ A multimodal, fully aligned and voluminous dataset
36
+ We propose **VIDAL-10M**, **10 Million data** with **V**ideo, **I**nfrared, **D**epth, **A**udio and their corresponding **L**anguage, which greatly expands the data beyond visual modalities.
37
+ * The second figure shows our proposed VIDAL-10M dataset, which includes five modalities: video, infrared, depth, audio, and language.
38
+
39
+ ### 🔥 Multi-view enhanced description for training
40
+ We make multi-view enhancements to language. We produce multi-view description that combines **meta-data**, **spatial**, and **temporal** to greatly enhance the semantic information of the language. In addition we further **enhance the language with ChatGPT** to create a good semantic space for each modality aligned language.
41
+
42
+
43
+
44
+ ## 🤗 Demo
45
+
46
+ * **Local demo.** Highly recommend trying out our web demo, which incorporates all features currently supported by LanguageBind.
47
+ ```bash
48
+ python gradio_app.py
49
+ ```
50
+
51
+ * **Online demo.** We provide the [online demo](https://huggingface.co/spaces/LanguageBind/LanguageBind) in Huggingface Spaces. In this demo, you can calculate the similarity of modalities to language, such as audio-to-language, video-to-language, and depth-to-image.
52
+
53
+
54
+
55
+ ## 🛠️ Requirements and Installation
56
+ * Python >= 3.8
57
+ * Pytorch >= 1.13.1
58
+ * CUDA Version >= 11.6
59
+ * Install required packages:
60
+ ```bash
61
+ git clone https://github.com/PKU-YuanGroup/LanguageBind
62
+ cd LanguageBind
63
+ pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
64
+ pip install -r requirements.txt
65
+ ```
66
+
67
+ ## 🐳 Model Zoo
68
+
69
+ The names in the table represent different encoder models. For example, `LanguageBind/LanguageBind_Video_FT` represents the fully fine-tuned version, while `LanguageBind/LanguageBind_Video` represents the LoRA-tuned version.
70
+
71
+ You can freely replace them in the recommended [API usage](#-api). We recommend using the fully fine-tuned version, as it offers stronger performance.
72
+
73
+ <div align="center">
74
+ <table border="1" width="100%">
75
+ <tr align="center">
76
+ <th>Modality</th><th>LoRA tuning</th><th>Fine-tuning</th>
77
+ </tr>
78
+ <tr align="center">
79
+ <td>Video</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Video">LanguageBind_Video</a></td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Video_FT">LanguageBind_Video_FT</a></td>
80
+ </tr>
81
+ <tr align="center">
82
+ <td>Audio</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Audio">LanguageBind_Audio</a></td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Audio_FT">LanguageBind_Audio_FT</a></td>
83
+ </tr>
84
+ <tr align="center">
85
+ <td>Depth</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Depth">LanguageBind_Depth</a></td><td>-</td>
86
+ </tr>
87
+ <tr align="center">
88
+ <td>Thermal</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Thermal">LanguageBind_Thermal</a></td><td>-</td>
89
+ </tr>
90
+ </table>
91
+ </div>
92
+
93
+
94
+ <div align="center">
95
+ <table border="1" width="100%">
96
+ <tr align="center">
97
+ <th>Version</th><th>Tuning</th><th>Model size</th><th>Num_frames</th><th>HF Link</th><th>MSR-VTT</th><th>DiDeMo</th><th>ActivityNet</th><th>MSVD</th>
98
+ </tr>
99
+ <tr align="center">
100
+ <td>LanguageBind_Video</td><td>LoRA</td><td>Large</td><td>8</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Video">Link</a></td><td>42.6</td><td>37.8</td><td>35.1</td><td>52.2</td>
101
+ </tr>
102
+ <tr align="center">
103
+ <td>LanguageBind_Video_FT</td><td>Full-tuning</td><td>Large</td><td>8</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Video_FT">Link</a></td><td>42.7</td><td>38.1</td><td>36.9</td><td>53.5</td>
104
+ </tr>
105
+ <tr align="center">
106
+ <td>LanguageBind_Video_V1.5_FT</td><td>Full-tuning</td><td>Large</td><td>8</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Video_V1.5_FT">Link</a></td><td>42.8</td><td>39.7</td><td>38.4</td><td>54.1</td>
107
+ </tr>
108
+ <tr align="center">
109
+ <td>LanguageBind_Video_V1.5_FT</td><td>Full-tuning</td><td>Large</td><td>12</td><td>Coming soon</td>
110
+ </tr>
111
+ <tr align="center">
112
+ <td>LanguageBind_Video_Huge_V1.5_FT</td><td>Full-tuning</td><td>Huge</td><td>8</td><td><a href="https://huggingface.co/LanguageBind/LanguageBind_Video_Huge_V1.5_FT">Link</a></td><td>44.8</td><td>39.9</td><td>41.0</td><td>53.7</td>
113
+ </tr>
114
+ <tr align="center">
115
+ <td>LanguageBind_Video_Huge_V1.5_FT</td><td>Full-tuning</td><td>Huge</td><td>12</td><td>Coming soon</td>
116
+ </tr>
117
+ </table>
118
+ </div>
119
+
120
+ ## 🤖 API
121
+ **We open source all modalities preprocessing code.** If you want to load the model (e.g. ```LanguageBind/LanguageBind_Thermal```) from the model hub on Huggingface or on local, you can use the following code snippets!
122
+ ### Inference for Multi-modal Binding
123
+ We have provided some sample datasets in [assets](assets) to quickly see how languagebind works.
124
+ ```python
125
+ import torch
126
+ from languagebind import LanguageBind, to_device, transform_dict, LanguageBindImageTokenizer
127
+
128
+ if __name__ == '__main__':
129
+ device = 'cuda:0'
130
+ device = torch.device(device)
131
+ clip_type = {
132
+ 'video': 'LanguageBind_Video_FT', # also LanguageBind_Video
133
+ 'audio': 'LanguageBind_Audio_FT', # also LanguageBind_Audio
134
+ 'thermal': 'LanguageBind_Thermal',
135
+ 'image': 'LanguageBind_Image',
136
+ 'depth': 'LanguageBind_Depth',
137
+ }
138
+
139
+ model = LanguageBind(clip_type=clip_type, cache_dir='./cache_dir')
140
+ model = model.to(device)
141
+ model.eval()
142
+ pretrained_ckpt = f'lb203/LanguageBind_Image'
143
+ tokenizer = LanguageBindImageTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir/tokenizer_cache_dir')
144
+ modality_transform = {c: transform_dict[c](model.modality_config[c]) for c in clip_type.keys()}
145
+
146
+ image = ['assets/image/0.jpg', 'assets/image/1.jpg']
147
+ audio = ['assets/audio/0.wav', 'assets/audio/1.wav']
148
+ video = ['assets/video/0.mp4', 'assets/video/1.mp4']
149
+ depth = ['assets/depth/0.png', 'assets/depth/1.png']
150
+ thermal = ['assets/thermal/0.jpg', 'assets/thermal/1.jpg']
151
+ language = ["Training a parakeet to climb up a ladder.", 'A lion climbing a tree to catch a monkey.']
152
+
153
+ inputs = {
154
+ 'image': to_device(modality_transform['image'](image), device),
155
+ 'video': to_device(modality_transform['video'](video), device),
156
+ 'audio': to_device(modality_transform['audio'](audio), device),
157
+ 'depth': to_device(modality_transform['depth'](depth), device),
158
+ 'thermal': to_device(modality_transform['thermal'](thermal), device),
159
+ }
160
+ inputs['language'] = to_device(tokenizer(language, max_length=77, padding='max_length',
161
+ truncation=True, return_tensors='pt'), device)
162
+
163
+ with torch.no_grad():
164
+ embeddings = model(inputs)
165
+
166
+ print("Video x Text: \n",
167
+ torch.softmax(embeddings['video'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
168
+ print("Image x Text: \n",
169
+ torch.softmax(embeddings['image'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
170
+ print("Depth x Text: \n",
171
+ torch.softmax(embeddings['depth'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
172
+ print("Audio x Text: \n",
173
+ torch.softmax(embeddings['audio'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
174
+ print("Thermal x Text: \n",
175
+ torch.softmax(embeddings['thermal'] @ embeddings['language'].T, dim=-1).detach().cpu().numpy())
176
+ ```
177
+ Then returns the following result.
178
+ ```bash
179
+ Video x Text:
180
+ [[9.9989331e-01 1.0667283e-04]
181
+ [1.3255903e-03 9.9867439e-01]]
182
+ Image x Text:
183
+ [[9.9990666e-01 9.3292067e-05]
184
+ [4.6132666e-08 1.0000000e+00]]
185
+ Depth x Text:
186
+ [[0.9954276 0.00457235]
187
+ [0.12042473 0.8795753 ]]
188
+ Audio x Text:
189
+ [[0.97634876 0.02365119]
190
+ [0.02917843 0.97082156]]
191
+ Thermal x Text:
192
+ [[0.9482511 0.0517489 ]
193
+ [0.48746133 0.5125386 ]]
194
+ ```
195
+ ### Emergency zero-shot
196
+ Since languagebind binds each modality together, we also found the **emergency zero-shot**. It's very simple to use.
197
+ ```python
198
+ print("Video x Audio: \n", torch.softmax(embeddings['video'] @ embeddings['audio'].T, dim=-1).detach().cpu().numpy())
199
+ print("Image x Depth: \n", torch.softmax(embeddings['image'] @ embeddings['depth'].T, dim=-1).detach().cpu().numpy())
200
+ print("Image x Thermal: \n", torch.softmax(embeddings['image'] @ embeddings['thermal'].T, dim=-1).detach().cpu().numpy())
201
+ ```
202
+ Then, you will get:
203
+ ```
204
+ Video x Audio:
205
+ [[1.0000000e+00 0.0000000e+00]
206
+ [3.1150486e-32 1.0000000e+00]]
207
+ Image x Depth:
208
+ [[1. 0.]
209
+ [0. 1.]]
210
+ Image x Thermal:
211
+ [[1. 0.]
212
+ [0. 1.]]
213
+ ```
214
+
215
+ ### Different branches for X-Language task
216
+ Additionally, LanguageBind can be **disassembled into different branches** to handle different tasks. Note that we do not train Image, which just initialize from OpenCLIP.
217
+ #### Thermal
218
+ ```python
219
+ import torch
220
+ from languagebind import LanguageBindThermal, LanguageBindThermalTokenizer, LanguageBindThermalProcessor
221
+
222
+ pretrained_ckpt = 'LanguageBind/LanguageBind_Thermal'
223
+ model = LanguageBindThermal.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
224
+ tokenizer = LanguageBindThermalTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
225
+ thermal_process = LanguageBindThermalProcessor(model.config, tokenizer)
226
+
227
+ model.eval()
228
+ data = thermal_process([r"your/thermal.jpg"], ['your text'], return_tensors='pt')
229
+ with torch.no_grad():
230
+ out = model(**data)
231
+
232
+ print(out.text_embeds @ out.image_embeds.T)
233
+ ```
234
+
235
+ #### Depth
236
+ ```python
237
+ import torch
238
+ from languagebind import LanguageBindDepth, LanguageBindDepthTokenizer, LanguageBindDepthProcessor
239
+
240
+ pretrained_ckpt = 'LanguageBind/LanguageBind_Depth'
241
+ model = LanguageBindDepth.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
242
+ tokenizer = LanguageBindDepthTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
243
+ depth_process = LanguageBindDepthProcessor(model.config, tokenizer)
244
+
245
+ model.eval()
246
+ data = depth_process([r"your/depth.png"], ['your text.'], return_tensors='pt')
247
+ with torch.no_grad():
248
+ out = model(**data)
249
+
250
+ print(out.text_embeds @ out.image_embeds.T)
251
+ ```
252
+
253
+ #### Video
254
+ ```python
255
+ import torch
256
+ from languagebind import LanguageBindVideo, LanguageBindVideoTokenizer, LanguageBindVideoProcessor
257
+
258
+ pretrained_ckpt = 'LanguageBind/LanguageBind_Video_FT' # also 'LanguageBind/LanguageBind_Video'
259
+ model = LanguageBindVideo.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
260
+ tokenizer = LanguageBindVideoTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
261
+ video_process = LanguageBindVideoProcessor(model.config, tokenizer)
262
+
263
+ model.eval()
264
+ data = video_process(["your/video.mp4"], ['your text.'], return_tensors='pt')
265
+ with torch.no_grad():
266
+ out = model(**data)
267
+
268
+ print(out.text_embeds @ out.image_embeds.T)
269
+ ```
270
+
271
+ #### Audio
272
+ ```python
273
+ import torch
274
+ from languagebind import LanguageBindAudio, LanguageBindAudioTokenizer, LanguageBindAudioProcessor
275
+
276
+ pretrained_ckpt = 'LanguageBind/LanguageBind_Audio_FT' # also 'LanguageBind/LanguageBind_Audio'
277
+ model = LanguageBindAudio.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
278
+ tokenizer = LanguageBindAudioTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
279
+ audio_process = LanguageBindAudioProcessor(model.config, tokenizer)
280
+
281
+ model.eval()
282
+ data = audio_process([r"your/audio.wav"], ['your audio.'], return_tensors='pt')
283
+ with torch.no_grad():
284
+ out = model(**data)
285
+
286
+ print(out.text_embeds @ out.image_embeds.T)
287
+ ```
288
+
289
+ #### Image
290
+ Note that our image encoder is the same as OpenCLIP. **Not** as fine-tuned as other modalities.
291
+ ```python
292
+ import torch
293
+ from languagebind import LanguageBindImage, LanguageBindImageTokenizer, LanguageBindImageProcessor
294
+
295
+ pretrained_ckpt = 'LanguageBind/LanguageBind_Image'
296
+ model = LanguageBindImage.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
297
+ tokenizer = LanguageBindImageTokenizer.from_pretrained(pretrained_ckpt, cache_dir='./cache_dir')
298
+ image_process = LanguageBindImageProcessor(model.config, tokenizer)
299
+
300
+ model.eval()
301
+ data = image_process([r"your/image.jpg"], ['your text.'], return_tensors='pt')
302
+ with torch.no_grad():
303
+ out = model(**data)
304
+
305
+ print(out.text_embeds @ out.image_embeds.T)
306
+ ```
307
+
308
+ ## 💥 VIDAL-10M
309
+ The datasets is in [DATASETS.md](DATASETS.md).
310
+
311
+ ## 🗝️ Training & Validating
312
+ The training & validating instruction is in [TRAIN_AND_VALIDATE.md](TRAIN_AND_VALIDATE.md).
313
+
314
+ ## 👍 Acknowledgement
315
+ * [OpenCLIP](https://github.com/mlfoundations/open_clip) An open source pretraining framework.
316
+ * [CLIP4Clip](https://github.com/ArrowLuo/CLIP4Clip) An open source Video-Text retrieval framework.
317
+ * [sRGB-TIR](https://github.com/rpmsnu/sRGB-TIR) An open source framework to generate infrared (thermal) images.
318
+ * [GLPN](https://github.com/vinvino02/GLPDepth) An open source framework to generate depth images.
319
+
320
+ ## 🔒 License
321
+ * The majority of this project is released under the MIT license as found in the [LICENSE](https://github.com/PKU-YuanGroup/LanguageBind/blob/main/LICENSE) file.
322
+ * The dataset of this project is released under the CC-BY-NC 4.0 license as found in the [DATASET_LICENSE](https://github.com/PKU-YuanGroup/LanguageBind/blob/main/DATASET_LICENSE) file.
323
+
324
+ ## ✏️ Citation
325
+ If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.
326
+
327
+ ```BibTeX
328
+ @misc{zhu2023languagebind,
329
+ title={LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment},
330
+ author={Bin Zhu and Bin Lin and Munan Ning and Yang Yan and Jiaxi Cui and Wang HongFa and Yatian Pang and Wenhao Jiang and Junwu Zhang and Zongwei Li and Cai Wan Zhang and Zhifeng Li and Wei Liu and Li Yuan},
331
+ year={2023},
332
+ eprint={2310.01852},
333
+ archivePrefix={arXiv},
334
+ primaryClass={cs.CV}
335
+ }
336
+ ```
337
+
338
+
339
+ ## ✨ Star History
340
+
341
+ [![Star History](https://api.star-history.com/svg?repos=PKU-YuanGroup/LanguageBind&type=Date)](https://star-history.com/#PKU-YuanGroup/LanguageBind&Date)
342
+
343
+
344
+ ## 🤝 Contributors
345
+
346
+ <a href="https://github.com/PKU-YuanGroup/LanguageBind/graphs/contributors">
347
+ <img src="https://contrib.rocks/image?repo=PKU-YuanGroup/LanguageBind" />
348
+ </a>