Jingtao Zhou
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jingtao Zhou has a total of 25 co-authors. Here are the top co-authors by number of shared publications:
- Alex Olwal
Google, United States
3 shared publications - Jason Mayes
Google Research, United States
3 shared publications - Jing Jin
Google Research, United States
3 shared publications - Jun Jiang
Google Research, United States
3 shared publications - Kristen Wright
Google Research, United States
3 shared publications - Mark Sherwood
Google Research, United States
3 shared publications
Latest Publications
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-22
InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-22
Open AccessInstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
- Open Access
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Experiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual Programming
Foundational multi-modal models have democratized AI access, yet the construction of complex, customizable machine learning pipelines by novice users remains a grand challenge. This paper demonstrates a visual programming system that allows novices to rapidly prototype multimodal AI pipelines. We first conducted a formative study with 58 contributors and collected 236 proposals of multimodal AI pipelines that served various practical needs. We then distilled our findings into a design matrix of primitive nodes for prototyping multimodal AI visual programming pipelines, and implemented a system with 65 nodes. To support users’ rapid prototyping experience, we built InstructPipe, an AI assistant based on large language models (LLMs) that allows users to generate a pipeline by writing text-based instructions. We believe InstructPipe enhances novice users onboarding experience of visual programming and the controllability of LLMs by offering non-experts a platform to easily update the generation.
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Open AccessExperiencing InstructPipe: Building Multi-modal AI Pipelines via Prompting LLMs and Visual Programming
Foundational multi-modal models have democratized AI access, yet the construction of complex, customizable machine learning pipelines by novice users remains a grand challenge. This paper demonstrates a visual programming system that allows novices to rapidly prototype multimodal AI pipelines. We first conducted a formative study with 58 contributors and collected 236 proposals of multimodal AI pipelines that served various practical needs. We then distilled our findings into a design matrix of primitive nodes for prototyping multimodal AI visual programming pipelines, and implemented a system with 65 nodes. To support users’ rapid prototyping experience, we built InstructPipe, an AI assistant based on large language models (LLMs) that allows users to generate a pipeline by writing text-based instructions. We believe InstructPipe enhances novice users onboarding experience of visual programming and the controllability of LLMs by offering non-experts a platform to easily update the generation.
- Open Access
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Experiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines
We demonstrate Visual Blocks for ML, a visual programming platform that facilitates rapid prototyping of ML-based multimedia applications. As the public version of Rapsai [3], we further integrated large language models and custom APIs into the platform. In this demonstration, we will showcase how to build interactive AI pipelines in a few drag-and-drops, how to perform interactive data augmentation, and how to integrate pipelines into Colabs. In addition, we demonstrate a wide range of community-contributed pipelines in Visual Blocks for ML, covering various aspects including interactive graphics, chains of large language models, computer vision, and multi-modal applications. Finally, we encourage students, designers, and ML practitioners to contribute ML pipelines through https://github.com/google/visualblocks/tree/main/pipelines to inspire creative use cases. Visual Blocks for ML is available at http://visualblocks.withgoogle.com.
- 29 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessExperiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines
We demonstrate Visual Blocks for ML, a visual programming platform that facilitates rapid prototyping of ML-based multimedia applications. As the public version of Rapsai [3], we further integrated large language models and custom APIs into the platform. In this demonstration, we will showcase how to build interactive AI pipelines in a few drag-and-drops, how to perform interactive data augmentation, and how to integrate pipelines into Colabs. In addition, we demonstrate a wide range of community-contributed pipelines in Visual Blocks for ML, covering various aspects including interactive graphics, chains of large language models, computer vision, and multi-modal applications. Finally, we encourage students, designers, and ML practitioners to contribute ML pipelines through https://github.com/google/visualblocks/tree/main/pipelines to inspire creative use cases. Visual Blocks for ML is available at http://visualblocks.withgoogle.com.
- 9 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 486-492
Solving Tracking Challenges: Lightweight Algorithm for Single Pedestrian Tracking on Nano-UAVs
With its pocket size and flexible flight characteristics, the nano-UAV can achieve more stealthy flight. If the full on-board deployment of the single pedestrian tracking algorithm can be realized on the nano-UAV, It will completely unmanned tasks such as personnel monitoring and security inspections, and has a broad application prospects. However, it is very challenging to design single pedestrian tracking algorithms for nano-UAV platforms. On the one hand, nano-UAVs are not able to deploy the algorithms fully on-board due to their strictly limited computing and storage resources; on the other hand, there are various tracking challenges (multi-scale changing target tracking, small-scale target tracking) for single pedestrian tracking tasks. In this paper, we design a lightweight single pedestrian tracking algorithm that ensures its resource footprint meets the limitations of nano-UAVs. The algorithm proposed in this paper consists of a multi-task learning front-end and a fused ReID tracking back-end, and we design a complete algorithm deployment process, including dataset construction, model training quantification and model deployment, which can effectively solve the challenges of current tracking tasks. A high efficiency (computational efficiency reaches 4.8 MACs/Cycle, and energy efficiency reaches 4.7 mJ/frame) has been achieved on the nano-UAV, achieving an accuracy score of 0.988 and a success score of 0.578 with good robustness in flight. The full-board deployment of nano-UAVs can achieve real-time tracking efficiency (28fps), which has application value and greatly promotes the research work of nano-UAVs.
- 9 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 486-492
Solving Tracking Challenges: Lightweight Algorithm for Single Pedestrian Tracking on Nano-UAVs
With its pocket size and flexible flight characteristics, the nano-UAV can achieve more stealthy flight. If the full on-board deployment of the single pedestrian tracking algorithm can be realized on the nano-UAV, It will completely unmanned tasks such as personnel monitoring and security inspections, and has a broad application prospects. However, it is very challenging to design single pedestrian tracking algorithms for nano-UAV platforms. On the one hand, nano-UAVs are not able to deploy the algorithms fully on-board due to their strictly limited computing and storage resources; on the other hand, there are various tracking challenges (multi-scale changing target tracking, small-scale target tracking) for single pedestrian tracking tasks. In this paper, we design a lightweight single pedestrian tracking algorithm that ensures its resource footprint meets the limitations of nano-UAVs. The algorithm proposed in this paper consists of a multi-task learning front-end and a fused ReID tracking back-end, and we design a complete algorithm deployment process, including dataset construction, model training quantification and model deployment, which can effectively solve the challenges of current tracking tasks. A high efficiency (computational efficiency reaches 4.8 MACs/Cycle, and energy efficiency reaches 4.7 mJ/frame) has been achieved on the nano-UAV, achieving an accuracy score of 0.988 and a success score of 0.578 with good robustness in flight. The full-board deployment of nano-UAVs can achieve real-time tracking efficiency (28fps), which has application value and greatly promotes the research work of nano-UAVs.