Scott Cohen

Science, technology, engineering, arts, and mathematics (STEAM) education represents an array of fields that have significant promise for the future careers of students. However, in deaf education, little research has been conducted to understand how best to provide access to STEAM learning opportunities for deaf students. This manuscript explores STEAM learning and Deaf Education through the lens of translanguaging. Translanguaging is the use of multiple linguistic resources by multilingual individuals. The authors recorded deaf teens attending a STEAM camp as they engaged in a collaborative problem-solving activity to explore the language resources they used to make sense of and communicate their understanding of the problem during various stages of the activity (gathering information, generating ideas, and evaluating ideas). We viewed their interactions through a translanguaging lens. We found that the campers used an array of both language-based (ASL, spoken English, gesture, and fingerspelling) and tool-based (writing on a whiteboard, engaging with informational papers, using computers or phones) translanguaging activities to gather information and communicate with one another. While selection of language resources did not differ by activity stage, they did differ by group, suggesting that campers were sensitive to the communication needs of their group mates.

Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets. Furthermore, our trained model is flexible and can support multiple editing use cases, such as object insertion, replacement, removal and standard inpainting. In particular, our trained model combined with a novel automatic image completion pipeline achieves state-of-the-art results on the standard inpainting task.

Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets. Furthermore, our trained model is flexible and can support multiple editing use cases, such as object insertion, replacement, removal and standard inpainting. In particular, our trained model combined with a novel automatic image completion pipeline achieves state-of-the-art results on the standard inpainting task.

Translating complex scientific words with abstract concepts from spoken/written language to American Sign Language (ASL) is challenging in the United States. Specifically, sign language users experience barriers to communicating these complex and abstract science ideas. Language development is also hindered by the small percentage of deaf and hard-of-hearing (DHH) Science, Technology, Engineering, and Math (STEM) professionals in advanced STEM fields who have limited opportunities to use ASL to communicate their specialized knowledge with one another. A recent trend to address this issue is developing digital content on translating scientific terms into ASL vocabulary and storytelling (i.e., a vocabulary of a scientific term or person and a narrative on how science works), such as ASL Core, ASL Clear, and Atomic Hands. Still, not much is known about how the community adopts these new and unfamiliar signs that would help them better articulate scientific concepts through signed language. Moreover, academia and educated professionals act as gatekeepers in delivering new language production to the community without regard to their input on contextualizing the sophisticated and complex science concepts to their home and cultural experiences.

Cameras and image-editing software often process images in the wide-gamut ProPhoto color space, encompassing 90% of all visible colors. However, when images are encoded for sharing, this color-rich representation is transformed and clipped to fit within the small-gamut standard RGB (sRGB) color space, representing only 30% of visible colors. Recovering the lost color information is challenging due to the clipping procedure. Inspired by neural implicit representations for 2D images, we propose a method that optimizes a lightweight multi-layer-perceptron (MLP) model during the gamut reduction step to predict the clipped values. GamutMLP takes approximately 2 seconds to optimize and requires only 23 KB of storage. The small memory footprint allows our GamutMLP model to be saved as metadata in the sRGB image -- the model can be extracted when needed to restore wide-gamut color values. We demonstrate the effectiveness of our approach for color recovery and compare it with alternative strategies, including pre-trained DNN-based gamut expansion networks and other implicit neural representation methods. As part of this effort, we introduce a new color gamut dataset of 2200 wide-gamut/small-gamut images for training and testing. Our code and dataset can be found on the project website: https://gamut-mlp.github.io.

Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results. Furthermore, annotating training data pairs for compositing requires substantial manual effort from professionals, and is hardly scalable. Thus, with the recent advances in generative models, in this work, we propose a self-supervised framework for object compositing by leveraging the power of conditional diffusion models. Our framework can hollistically address the object compositing task in a unified model, transforming the viewpoint, geometry, color and shadow of the generated object while requiring no manual labeling. To preserve the input object's characteristics, we introduce a content adaptor that helps to maintain categorical semantics and object appearance. A data augmentation method is further adopted to improve the fidelity of the generator. Our method outperforms relevant baselines in both realism and faithfulness of the synthesized result images in a user study on various real-world images.