Ning Wang

Modeling long sequences is crucial for various large-scale models; however, extending existing architectures to handle longer sequences presents significant technical and resource challenges. In this paper, we propose an efficient and flexible attention architecture that enables the extension of context lengths in large language models with reduced computational resources and fine-tuning time compared to other excellent methods. Specifically, we introduce correlation-aware selection and merging mechanisms to facilitate efficient sparse attention. In addition, we also propose a novel data augmentation technique involving positional encodings to enhance generalization to unseen positions. The results are as follows: First, using a single A100, we achieve fine-tuning on Llama2-7B with a sequence length of 32K, which is more efficient than other methods that rely on subsets for regression. Second, we present a comprehensive method for extending context lengths across the pre-training, fine-tuning, and inference phases. During pre-training, our attention mechanism partially breaks translation invariance during token selection, so we apply positional encodings only to the selected tokens. This approach achieves relatively high performance and significant extrapolation capabilities. For fine-tuning, we introduce Cyclic, Randomly Truncated, and Dynamically Growing NTK Positional Embedding (CRD NTK). This design allows fine-tuning with a sequence length of only 16K, enabling models such as Llama2-7B and Mistral-7B to perform inference with context lengths of up to 1M or even arbitrary lengths. Our method achieves 100\% accuracy on the passkey task with a context length of 4M and maintains stable perplexity at a 1M context length. This represents at least a 64-fold reduction in resource requirements compared to traditional full-attention mechanisms, while still achieving competitive performance.

Using DNA as the next-generation medium for data storage offers unparalleled advantages in terms of data density, storage duration, and power consumption as compared to existing data storage technologies. To meet the high-speed data writing requirements in DNA data storage, this paper proposes a novel design for an ultra-high-density and high-throughput DNA synthesis platform. The presented design mainly leverages two functional modules: a dynamic random-access memory (DRAM)-like integrated circuit (IC) responsible for electrode addressing and voltage supply, and the static droplet array (SDA)-based microfluidic structure to eliminate any reaction species diffusion concern in electrochemical DNA synthesis. Through theoretical analysis and simulation studies, we validate the effective addressing of 10 million electrodes and stable, adjustable voltage supply by the integrated circuit. We also demonstrate a reaction unit size down to 3.16 × 3.16 μm², equivalent to 10 million/cm², that can rapidly and stably generate static droplets at each site, effectively constraining proton diffusion. Finally, we conducted a synthesis cycle experiment by incorporating fluorescent beacons on a microfabricated electrode array to examine the feasibility of our design.

Exosomes, with diameters ranging from 30 to 150 nm, are saucer-shaped extracellular vesicles (EVs) secreted by various type of human cells. They are present in virtually all bodily fluids. Owing to their abundant nucleic acid and protein content, exosomes have emerged as promising biomarkers for noninvasive molecular diagnostics. However, the need for exosome separation purification presents tremendous technical challenges due to their minuscule size. In recent years, microfluidic technology has garnered substantial interest as a promising alternative capable of excellent separation performance, reduced reagent consumption, and lower overall device and operation costs. In this context, we hereby propose a novel microfluidic strategy based on thermally oxidized deterministic lateral displacement (DLD) arrays with tapered shapes to enhance separation performance. We have achieved more than 90% purity in both polystyrene nanoparticle and exosome experiments. The use of thermal oxidation also significantly reduces fabrication complexity by avoiding the use of high-precision lithography. Furthermore, in a simulation model, we attempt to integrate the use of dielectrophoresis (DEP) to overcome the size-based nature of DLD and distinguish particles that are close in size but differ in biochemical compositions (e.g., lipoproteins, exomeres, retroviruses). We believe the proposed strategy heralds a versatile and innovative platform poised to enhance exosome analysis across a spectrum of biochemical applications.