FDFred Douglis

Fred Douglis

*This is an algorithmically generated author record based on publicly available information.

Fred Douglis has a total of 142 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Secure NDN Packet Encapsulation

    by Daniel Townley, Young Kim, Fred Douglis, Jesse Elwell, Constantin Serban, Lan Wang, Alex Afanasyev, Lixia Zhang

    Packet encapsulation is a general network technique that provides an essential building block for constructing secure networks. While extensively used in IP networks over the last few decades, secure packet encapsulation remains largely unexplored in the context of Named Data Networking (NDN) networks. NDN represents a radical departure from traditional endpoint-oriented networking by making secured data the centerpiece of communication. This new data-centric design brings both advantages and new challenges for the development of secure packet encapsulation that can preserve essential properties of an NDN network, including in-network data caching and builtin multicast data delivery. In this paper, we first identify the major differences between encapsulation solution designs in IP and NDN, highlighting the ensuing challenges, both inherent and practical. We then present a novel design to achieve secure NDN data packet encapsulation, and showcase an implementation suite that enables efficient fetching of securely encapsulated data. 1 1 The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)


  • Open Access

    FLEET—Fast Lanes for Expedited Execution at 10 Terabits: Program Overview

    by Fred Douglis, Seth Robertson, Eric van Den Berg, Josephine Micallef, Marc Pucci, Alex Aiken, Keren Bergman, Maarten Hattink, Mingoo Seok

    The DARPA FastNICs program targets orders of magnitude improvement in applications such as deep learning training by making radical improvements to network performance: While raw bandwidth has grown dramatically, the fundamental roadblock to application performance has been in delivering that data to the application. FLEET provides a primarily off-the-shelf solution with high-end servers and shared computational and storage resources connected via PCIe over a reconfigurable MEMS optical switch; it uses custom Optical NICs to allow arbitrary topologies that can be configured before or even during execution to take advantage of shared resources and to flow data between components. FLEET 's software is derived from Stanford Legion, which we are modifying to use the FLEET hardware and to plan application execution for these dynamic network topologies.


  • Open Access

    Vision Paper: Grand Challenges in Resilience: Autonomous System Resilience through Design and Runtime Measures

    by Saurabh Bagchi, Vaneet Aggarwal, Somali Chaterji, Fred Douglis, Aly El Gamal, Jiawei Han, Brian J. Henz, Henry Hoffmann, Suman Jana, Milind Kulkarni, et al.

    In this article, we put forward the substantial challenges in cyber resilience in the domain of autonomous systems and outline foundational solutions to address these challenges. These solutions fall into two broad themes: resilience-by-design and resilience-by-reaction. We use several application drivers from autonomous systems to motivate the challenges in cyber resilience and to demonstrate the benefit of the solutions. We focus on some autonomous systems in the near horizon (autonomous ground and aerial vehicles) and also a little more distant (autonomous rescue and relief). For resilience-by-design, we focus on design methods in software that are needed for our cyber systems to be resilient. In contrast, for resilience-by-reaction, we discuss how to make systems resilient by responding, reconfiguring, or recovering at runtime when failures happen. We also discuss the notion of adaptive execution to improve resilience, execution transparently and adaptively among available execution platforms (mobile/embedded, edge, and cloud). For each of the two themes, we survey the current state, and the desired state and ways to get there. We conclude the paper by looking at the research challenges we will have to solve in the short and the mid-term to make the vision of resilient autonomous systems a reality. This article came out of discussions that started at the NSF-sponsored Grand Challenges in Resilience Workshop held at Purdue in 2019 with the co-authors contributing to going into the depth of the issues and then this article.


  • Customizable Scale-Out Key-Value Stores

    by Ali Anwar, Yue Cheng, Hai Huang, Jingoo Han, Hyogi Sim, Dongyoon Lee, Fred Douglis, Ali R. Butt

    Enterprise KV stores are often not well suited for HPC applications, and thus cumbersome end-to-end KV design customization is required to meet the needs of modern HPC applications. To this end, in this article we present bespoKV, an adaptive, extensible, and scale-out KV store framework. bespoKV decouples the KV store design into the control plane for distributed management and the data plane for local data store. For the control plane, bespoKVprovides pre-built modules, called controlets, supporting common distributed functionalities (e.g., replication, consistency, and topology) and their various combinations. This decoupling allows bespoKV to take a user-provided single-server KV store, called a datalet, and transparently enables a scalable and fault-tolerant distributed KV store service. The resulting distributed stores are also adaptive to consistency or topology requirement changes and can be easily extended for new types of services. Such specializations enable innovative uses of KV stores in HPC applications, especially for emerging applications that utilize KV-friendly workloads. We evaluate bespoKV in a local testbed as well as in a public cloud settings. Experiments show that bespoKV-enabled distributed KV stores scale horizontally to a large number of nodes, and performs comparably and sometimes 1.2× to 2.6× better than the state-of-the-art systems.


  • Backup to the Future: How Workload and Hardware Changes Continually Redefine Data Domain File Systems

    by Yamini Allu, Fred Douglis, Mahesh Kamat, Philip Shilane, Hugo Patterson, Ben Zhu

    Disk-based backup systems use data deduplication to replace redundant data chunks with references. These systems initially supported workloads from their tape-based predecessors, but changes to hardware and applications have forced them to adapt in interesting and challenging ways.


  • A Case for End-to-End Deduplication

    by Fred Douglis

    Deduplication is nearly ubiquitous in backup environments, common for data distribution, and increasingly important for wide-area networking. Each of these three domains handles deduplication in a separate manner, but integrating them into an end-to-end deduplication paradigm would enable efficiencies and simplifications that will improve performance, scalability, and functionality.


  • Top Cited Papers
    Open Access

    A Comprehensive Study of the Past, Present, and Future of Data Deduplication

    by Wen Xia, Hong Jiang, Dan Feng, Fred Douglis, Philip Shilane, Yu Hua, Min Fu, Yucheng Zhang, Yukun Zhou

    Data deduplication, an efficient approach to data reduction, has gained increasing attention and popularity in large-scale storage systems due to the explosive growth of digital data. It eliminates redundant data at the file or subfile level and identifies duplicate content by its cryptographically secure hash signature (i.e., collision-resistant fingerprint), which is shown to be much more computationally efficient than the traditional compression approaches in large-scale storage systems. In this paper, we first review the background and key features of data deduplication, then summarize and classify the state-of-the-art research in data deduplication according to the key workflow of the data deduplication process. The summary and taxonomy of the state of the art on deduplication help identify and understand the most important design considerations for data deduplication systems. In addition, we discuss the main applications and industry trend of data deduplication, and provide a list of the publicly available sources for deduplication research and studies. Finally, we outline the open problems and future research directions facing deduplication-based storage systems.


  • Cloud Storage [Guest editors' introduction]

    by Robin Chen, Fred Douglis

    This special issue of Internet Computing surveys topics and challenges surrounding cloud storage. These include software-defined object storage in OpenStack Swift, shared services for file synchronization, convergent dispersal for secure cloud storage with high storage efficiency, quality of service for tiered storage in the cloud, and an application of cloud storage for the Internet of Things.


  • Open Access

    Pannier

    by Cheng Li, Philip Shilane, Fred Douglis, Grant Wallace

    Classic caching algorithms leverage recency, access count, and/or other properties of cached blocks at per-block granularity. However, for media such as flash which have performance and wear penalties for small overwrites, implementing cache policies at a larger granularity is beneficial. Recent research has focused on buffering small blocks and writing in large granularities, called containers, but it has not explored the ramifications and best strategies for caching compound blocks consisting of logically distinct, but physically co-located, blocks. Containers may have highly diverse blocks, with mixtures of frequently accessed, infrequently accessed, and invalidated blocks. We propose and evaluate Pannier, a flash cache middleware that provides high performance while extending flash lifespan. Pannier uses three main techniques: (1) leveraging block access counts to manage cache containers, (2) incorporating block liveness as a property to improve flash cache space efficiency, and (3) designing a multi-step feedback controller to ensure a flash cache does not wear out in its lifespan while maintaining performance. Our evaluation shows that Pannier improves flash cache performance and extends lifespan beyond previous per-block and container-aware caching policies. More fundamentally, our investigation highlights the importance of creating new policies for caching compound blocks in flash.


  • Open Access

    RAIDShield

    by Ao Ma, Rachel Traylor, Fred Douglis, Mark Chamness, Guanlin Lu, Darren Sawyer, Surendar Chandra, Windsor Hsu

    Modern storage systems orchestrate a group of disks to achieve their performance and reliability goals. Even though such systems are designed to withstand the failure of individual disks, failure of multiple disks poses a unique set of challenges. We empirically investigate disk failure data from a large number of production systems, specifically focusing on the impact of disk failures on RAID storage systems. Our data covers about one million SATA disks from six disk models for periods up to 5 years. We show how observed disk failures weaken the protection provided by RAID. The count ofreallocated sectorscorrelates strongly with impending failures.With these findings we designed RAIDShield, which consists of two components. First, we have built and evaluated an active defense mechanism that monitors the health of each disk and replaces those that are predicted to fail imminently. This proactive protection has been incorporated into our product and is observed to eliminate 88% of triple disk errors, which are 80% of all RAID failures. Second, we have designed and simulated a method of using the joint failure probability to quantify and predict how likely a RAID group is to face multiple simultaneous disk failures, which can identify disks that collectively represent a risk of failure even when no individual disk is flagged in isolation. We find in simulation that RAID-level analysis can effectively identify most vulnerable RAID-6 systems, improving the coverage to 98% of triple errors.We conclude with discussions of operational considerations in deploying RAIDShieldmore broadly and new directions in the analysis of disk errors. One interesting approach is to combine multiple metrics, allowing the values of different indicators to be used for predictions. Using newer field data that reports an additional metric,medium errors, we find that the relative efficacy of reallocated sectors and medium errors varies across disk models, offering an additional way to predict failures.