Ankit Singh Rawat

The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling such large models to compact models for efficient deployment also necessitates a large amount of (labeled or unlabeled) training data. In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data. TGT exploits the fact that the teacher has acquired a good representation of the underlying data domain, which typically corresponds to a much lower dimensional manifold than the input space. Furthermore, we can use the teacher to explore input space more efficiently through sampling or gradient-based methods; thus, making TGT especially attractive for limited data or long-tail settings. We formally capture this benefit of proposed data-domain exploration in our generalization bounds. We find that TGT can improve accuracy on several image classification benchmarks as well as a range of text classification and retrieval tasks.

In the problem of structured signal recovery from high-dimensional linear observations, it is commonly assumed that full-precision measurements are available. Under this assumption, the recovery performance of the popular Generalized Lasso (G-Lasso) is by now well-established. In this paper, we extend these types of results to the practically relevant settings with quantized measurements. We study two extremes of the quantization schemes, namely, uniform and one-bit quantization; the former imposes no limit on the number of quantization bits, while the second only allows for one bit. In the presence of a uniform dithering signal and when measurement vectors are sub-gaussian, we show that the same algorithm (i.e., the G-Lasso) has favorable recovery guarantees for both uniform and one-bit quantization schemes. Our theoretical results, shed light on the appropriate choice of the range of values of the dithering signal and accurately capture the error dependence on the problem parameters. For example, our error analysis shows that the G-Lasso with one-bit uniformly dithered measurements leads to only a logarithmic rate loss compared to the full-precision measurements.

This paper considers a distributed storage system, where multiple storage nodes can be reconstructed simultaneously at a centralized location. This centralized multi-node repair (CMR) model is a generalization of regenerating codes that allow for bandwidth efficient repair of a single failed node. This work focuses on the trade-off between the amount of data stored and repair bandwidth in the CMR model. In particular, repair bandwidth bounds are derived for the minimum storage multi-node repair (MSMR) and the minimum bandwidth multinode repair (MBMR) operating points. The tightness of these bounds is analyzed via code constructions. The MSMR point is characterized by codes achieving this point under functional repair for the general set of CMR parameters, as well as with codes enabling exact repair for certain CMR parameters. The MBMR point, on the other hand, is characterized with exact repair codes for all CMR parameters for systems that satisfy a certain entropy accumulation property. Finally, the model proposed here is utilized for the secret sharing problem, where the codes for the multi-node repair problem are used to construct communication efficient secret sharing schemes with the property of bandwidth efficient share repair.

Hybrid cloud is a widely adopted framework where on-premise storage and/or compute resources are combined with public cloud system. This paper explores the storage aspect of this framework, which requires designing coding schemes that are aware of both local and global components of the available storage space. The coding schemes should provide efficient repair mechanisms for the data stored on the public cloud (global storage space) and utilize the local storage space to facilitate seamless access to the overall information stored on the hybrid cloud storage. This paper presents a mathematical model for hybrid cloud storage which takes all these requirements into account. The paper then extends the information flow graph approach to characterize the fundamental limits on access bandwidth of the system, i.e., the amount of data downloaded from the public cloud during the data reconstruction process. This paper also presents several explicit coding schemes that utilize the available local storage space to attain the fundamental limit on the access bandwidth. The setup where multiple clients with varying local storage spaces are supported by a single global storage space is also addressed.

Minimum storage regenerating (MSR) codes form a special class of maximum distance separable (MDS) codes by providing mechanisms for exact regeneration of a single code block in their codewords by downloading the minimum amount of information from the remaining code blocks. As a result, the MSR codes find application to distributed storage systems to enable node repairs with the optimal repair band-width. However, the construction of exact-repairable MSR codes requires working with a large sub-packetization level, which restricts the employment of these codes in practice. This paper explores exact-repairable MDS codes that significantly reduce the required sub-packetization level by achieving slightly suboptimal repair bandwidth as compared to the MSR codes. This paper presents a general approach to combine an MSR code with large sub-packetization level with a code with large enough minimum distance to construct exact-repairable MDS codes with small sub-packetization level and near-optimal repair bandwidth. For a given number of parity blocks, the codes constructed using this approach have their sub-packetization level scaling logarithmically with the code length. In addition, the obtained codes require field size linear in the code length and ensure load balancing among the intact code blocks in terms of the information downloaded from these blocks during a node repair.

This paper revisits the problem of designing secure minimum storage regenerating (MSR) codes for distributed storage systems (DSS). A secure MSR code ensures that a DSS does not reveal the stored information to a passive eavesdropper. The eavesdropper is assumed to have access to the content stored on ℓ ₁ number of storage nodes in the system and the data downloaded during the bandwidth efficient repair of an additional ℓ ₂ number of storage nodes. This paper combines the Gabidulin codes based precoding [1] and a new construction of MSR codes (without security requirements) by Ye and Barg [2] in order to obtain secure MSR codes. Such optimal secure MSR codes were previously known only in the setting where the eavesdropper was allowed to observe the repair of ℓ ₂ nodes among a specific subset of k nodes [1], [3]. The secure coding scheme presented in this paper allows the eavesdropper to observe repair of any ℓ ₂ out of ii nodes in the system and characterizes the secrecy capacity of linear repairable MSR codes.

This paper presents a construction for high-rate MDS codes that enable bandwidth-efficient repair of a single node. Such MDS codes are also referred to as the minimum storage regenerating (MSR) codes in the distributed storage literature. The construction presented in this paper generates MSR codes for all possible number of helper nodes d as d is a design parameter in the construction. Furthermore, the obtained MSR codes have polynomial sub-packetization (a.k.a. node size) α. The construction is built on the recent code proposed by Sasidharan et al. [1], which works only for d = n — 1, i.e., where all the remaining nodes serve as the helper nodes for the bandwidth-efficient repair of a single node. The results of this paper broaden the set of parameters where the constructions of MSR codes were known earlier.

In this paper we present several results regarding distance preserving maps between nonbinary Hamming spaces and combinatorial (adversarial) joint source-channel coding. In an (α, β)-map from one Hamming space to another, any two sequences that are at least α relative distance apart, are mapped to sequences that are relative distance at least β apart. The motivation to study such maps come from (D, δ)-joint source-channel coding (JSCC) schemes, where any encoded sequence must be recovered within a relative distortion D, even in the presence of δ proportion of adversarial errors. We provide bounds on the parameters of both (α, β)-maps and (D,δ)-JSCC for nonbinary alphabets. We also provide constructive schemes for both, that are optimal for many cases.

This paper considers a distributed storage system, where multiple storage nodes can be reconstructed simultaneously at a centralized location. This centralized multi-node repair (CMR) model is a generalization of regenerating codes that allow for bandwidth-efficient repair of a single failed node. This work focuses on the trade-off between the amount of data stored and repair bandwidth in this CMR model. In particular, repair bandwidth bounds are derived for the minimum storage multi-node repair (MSMR) and the minimum bandwidth multi-node repair (MBMR) operating points. The tightness of these bounds are analyzed via code constructions. The MSMR point is characterized through codes achieving this point under functional repair for general set of CMR parameters, as well as with codes enabling exact repair for certain CMR parameters. The MBMR point, on the other hand, is characterized with exact repair codes for all CMR parameters for systems that satisfy a certain entropy accumulation property.

This paper studies the problem of information symbol availability in codes: we refer to a systematic code as code with (r, t)-availability if every information (systematic) symbol can be reconstructed from t disjoint groups of other code symbols, each of the sizes at most r. This paper shows that it is possible to construct codes that can support a scaling number of parallel reads while keeping the rate to be an arbitrarily high constant. It further shows that this is possible with the minimum Hamming distance arbitrarily close to the Singleton bound. This paper also presents a bound demonstrating a tradeoff between rate, minimum Hamming distance, and availability parameters. Our codes match the aforementioned bound, and their constructions rely on certain combinatorial structures. Resolvable designs provide one way to realize these required combinatorial structures. The two constructions presented in this paper require field sizes, which are linear and exponential in the code length, respectively. From a practical standpoint, our codes are relevant for distributed storage applications involving hot data, i.e., the information, which is frequently accessed by multiple processes in parallel.