Zheyu Zhang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zheyu Zhang has a total of 8 co-authors. Here are the top co-authors by number of shared publications:
Bolei Ma
Department of Statistics LMU Munich ♡ Munich
2 shared publicationsDavid Rügamer
Department of Statistics LMU Munich ♡ Munich
2 shared publications- Ercong Nie
LMU Munich
2 shared publications - Han Yang
GESIS -Leibniz-Institute for the Social Sciences Cologne Germany
2 shared publications André F. T. Martins
Instituto Superior Técnico Universidade de Lisboa (Lisbon ELLIS Unit
1 shared publication- Chengzhi Hu
Institute of Informatics LMU Munich
1 shared publication
Latest Publications
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 276-310
mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
Peiqin Lin, Chengzhi Hu, Zheyu Zhang, Andre Martins, Hinrich Schuetze. Findings of the Association for Computational Linguistics: EACL 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 276-310
Open AccessmPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
Peiqin Lin, Chengzhi Hu, Zheyu Zhang, Andre Martins, Hinrich Schuetze. Findings of the Association for Computational Linguistics: EACL 2024. 2024.
- Open Access
Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
Large Language Models (LLMs) demonstrate remarkable performance on a variety of natural language understanding (NLU) tasks, primarily due to their in-context learning ability. This ability could be applied to building babylike models, i.e. models at small scales, improving training efficiency. In this paper, we propose a "CoThought" pipeline, which efficiently trains smaller "baby" language models (BabyLMs) by leveraging the Chain of Thought prompting of LLMs. Our pipeline restructures a dataset of less than 100M in size using GPT-3.5-turbo, transforming it into task-oriented, human-readable texts that are comparable to the school texts for language learners. The BabyLM is then pretrained on this restructured dataset in a RoBERTa fashion. In evaluations across 4 benchmarks, our BabyLM outperforms the vanilla RoBERTa in 10 linguistic, NLU, and question-answering tasks by more than 3 points, showing a superior ability to extract contextual information. These results suggest that compact LMs pretrained on small, LLM-restructured data can better understand tasks and achieve improved performance.
Open AccessBaby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
Large Language Models (LLMs) demonstrate remarkable performance on a variety of natural language understanding (NLU) tasks, primarily due to their in-context learning ability. This ability could be applied to building babylike models, i.e. models at small scales, improving training efficiency. In this paper, we propose a "CoThought" pipeline, which efficiently trains smaller "baby" language models (BabyLMs) by leveraging the Chain of Thought prompting of LLMs. Our pipeline restructures a dataset of less than 100M in size using GPT-3.5-turbo, transforming it into task-oriented, human-readable texts that are comparable to the school texts for language learners. The BabyLM is then pretrained on this restructured dataset in a RoBERTa fashion. In evaluations across 4 benchmarks, our BabyLM outperforms the vanilla RoBERTa in 10 linguistic, NLU, and question-answering tasks by more than 3 points, showing a superior ability to extract contextual information. These results suggest that compact LMs pretrained on small, LLM-restructured data can better understand tasks and achieve improved performance.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 130-142
Baby’s CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
Zheyu Zhang, Han Yang, Bolei Ma, David Rügamer, Ercong Nie. Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 130-142
Open AccessBaby’s CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
Zheyu Zhang, Han Yang, Bolei Ma, David Rügamer, Ercong Nie. Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning. 2023.