Giovanni Beltrame
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Giovanni Beltrame has a total of 5 co-authors. Here are the top co-authors by number of shared publications:
Esteban Carlin
Indah Permata NTUST Taiwan NTUST Taiwan NTUST Taiwan Polytechnique Montréal Sari NTUST Taiwan NTUST Taiwan
2 shared publicationsEuhid Aman
Indah Permata NTUST Taiwan NTUST Taiwan NTUST Taiwan Polytechnique Montréal Sari NTUST Taiwan NTUST Taiwan
2 shared publicationsGhaluh Indah Permata Sari
2 shared publications- Hsing-Kuo Pao
National Taiwan University of Science and Technology
2 shared publications - Yie-Tarng Chen
Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
2 shared publications
Latest Publications
- Open Access
BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment.
Open AccessBitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 147-154
BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Euhid Aman, Esteban Carlin, Hsing-Kuo Kenneth Pao, Giovanni Beltrame, Ghaluh Indah Permata Sari, Yie-Tarng Chen. Proceedings of the First BabyLM Workshop. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 147-154
Open AccessBitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Euhid Aman, Esteban Carlin, Hsing-Kuo Kenneth Pao, Giovanni Beltrame, Ghaluh Indah Permata Sari, Yie-Tarng Chen. Proceedings of the First BabyLM Workshop. 2025.