Sebastian Friston

3D models in the architecture, engineering and construction (AEC) sector are becoming bigger by the day, yet AEC professionals still need to be able to federate and collaborate on such massive 3D scenes over the Internet. In this case study, we present the 3D Repo Infinite Geometry Streaming solution on the 40 Leadenhall Street project, the largest construction to ever receive a planning permission in the City of London. Over 100 GB of raw geometry has been processed and stored in the cloud while all the visibility calculations were performed client-side for seamless real-time exploration of the continuous 3D space.

Social VR (SVR) systems are VR systems with a common subset of features facilitating unstructured social interaction. In the real world, social situations have many purposes, each with a different set of requirements, and roles its participants take - creator, moderator, performer, visitor, etc. Yet, common SVR systems typically offer only a single client to users. Even if there are versions for different platforms, there is a one-size-fits-all approach to the user experience. Consequently users need to employ workarounds or build their own functionality to support specific roles, where this is possible at all. We argue that platforms need to develop more open frameworks that support different processes and user interactions. One way to do this is through using appropriate web standards and an open messaging system in order to allow distributed clients that can leverage the strongest features of heterogeneous computing platforms. Supporting asymmetrical capabilities greatly increases the scope of supported virtual social interactions and potential use cases of SVR. We take a qualitative experimental approach to exploring cross platform support in this way, from a designers perspective. We use the open-source SDK Ubiq, and create a library that allows building Ubiq Peers using web standards and thus clients that can operate solely in a web browser or certain Javascript environments. We validate our approach by demonstrating six proof of concept demonstrators that would be difficult or impossible to achieve in most other SVR systems, and report on what we encountered for the benefit of other SVR designers.

We share our experiences of teaching virtual reality with Ubiq, an open-source system for building social virtual reality (VR). VR as a subject touches on many areas, including perception, human–computer interaction, and psychology. In our VE module, we consider all aspects of VR. In recent years, networked VR, and in particular social VR, has become increasingly relevant, at the same time as demand for online and hybrid teaching has increased. Commercial social virtual reality systems have proliferated, but for a number of reasons, this has not resulted in systems any more suitable for research and teaching. As a result we created Ubiq, a system for building social VR applications designed first for research and teaching. In this article, we describe how Ubiq came to be, and our experiences of using it in our virtual environments module over the last two years.

Image-warping , a per-pixel deformation of one image into another, is an essential component in immersive visual experiences such as virtual reality or augmented reality. The primary issue with image warping is disocclusions, where occluded (and hence unknown) parts of the input image would be required to compose the output image. We introduce a new image warping method, Metameric image inpainting - an approach for hole-filling in real-time with foundations in human visual perception. Our method estimates image feature statistics of disoccluded regions from their neighbours. These statistics are inpainted and used to synthesise visuals in real-time that are less noticeable to study participants, particularly in peripheral vision. Our method offers speed improvements over the standard structured image inpainting methods while improving realism over colour-based inpainting such as push-pull. Hence, our work paves the way towards future applications such as depth image-based rendering, 6-DoF 360 rendering, and remote render-streaming.

Developing mixed-reality (MR) experiments is a challenge as there is a wide variety of functionality to support. This challenge is exacerbated if the MR experiment is multi-user or if the experiment needs to be run out of the lab. We present Ubiq-Exp - a set of tools that provide a variety of functionality to facilitate distributed and remote MR experiments. We motivate our design and tools from recent practice in the field and a desire to build experiments that are easier to reproduce. Key features are the ability to support supervised and unsupervised experiments, and a variety of tools for the experimenter to facilitate operation and documentation of the experimental sessions. We illustrate the potential of the tools through three small-scale pilot experiments. Our tools and pilot experiments are released under a permissive open-source license to enable developers to appropriate and develop them further for their own needs.

Distributed virtual environments (DVEs) are challenging to create as the goals of consistency and responsiveness become contradictory under increasing latency. DVEs have been considered as both distributed transactional databases and force-reflection systems. Both are good approaches, but they do have drawbacks. Transactional systems do not support Level 3 (L3) collaboration: manipulating the same degree-of-freedom at the same time. Force-reflection requires a client-server architecture and stabilisation techniques. With Consensus Based Networking (CBN), we suggest DVEs be considered as a distributed data-fusion problem. Many simulations run in parallel and exchange their states, with remote states integrated with continous authority. Over time the exchanges average out local differences, performing a distribued-average of a consistent, shared state. CBN aims to build simulations that are highly responsive, but consistent enough for use cases such as the piano-movers problem. CBN’s support for heterogeneous nodes can transparently couple different input methods, avoid the requirement of determinism, and provide more options for personal control over the shared experience. Our work is early, however we demonstrate many successes, including L3 collaboration in room-scale VR, 1000’s of interacting objects, complex configurations such as stacking, and transparent coupling of haptic devices. These have been shown before, but each with a different technique; CBN supports them all within a single, unified system.

Recent years have seen a resurgence of virtual reality (VR), sparked by the repurposing of low-cost COTS components. VR aims to generate stimuli that appear to come from a source other than the interface through which they are delivered. The synthetic stimuli replace real-world stimuli, and transport the user to another, perhaps imaginary, “place.” To do this, we must overcome many challenges, often related to matching the synthetic stimuli to the expectations and behavior of the real world. One way in which the stimuli can fail is its latency–– the time between a user's action and the computer's response. We constructed a novel VR renderer, that optimized latency above all else. Our prototype allowed us to explore how latency affects human–computer interaction. We had to completely reconsider the interaction between time, space, and synchronization on displays and in the traditional graphics pipeline. Using a specialized architecture––dataflow computing––we combined consumer, industrial, and prototype components to create an integrated 1:1 room-scale VR system with a latency of under 3 ms. While this was prototype hardware, the considerations in achieving this performance inform the design of future VR pipelines, and our human factors studies have provided new and sometimes surprising contributions to the body of knowledge on latency in HCI.

We suggest a rasterization pipeline tailored towards the needs of HMDs, where latency and field-of-view requirements pose new challenges beyond those of traditional desktop displays. Instead of image warping for low latency, or using multiple passes for foveation, we show how both can be produced directly in a single perceptual rasterization pass. We do this with per-fragment ray-casting. This is enabled by derivations of tight space-time-fovea pixel bounds, introducing just enough flexibility for the requisite geometric tests, but retaining most of the simplicity and efficiency of the traditional rasterizaton pipeline. To produce foveated images, we rasterize to an image with spatially varying pixel density. To compensate for latency, we extend the image formation model to directly produce "rolling" images where the time at each pixel depends on its display location. Our approach overcomes limitations of warping with respect to disocclusions, object motion and view-dependent shading, as well as geometric aliasing artifacts in other foveated rendering techniques. A set of perceptual user studies demonstrates the efficacy of our approach.

The Dexmo glove is a haptic exoskeleton that provides kinesthetic feedback in virtual reality. Unlike many other gloves based on string-pulleys, the Dexmo uses a free-hinged link-bar to transfer forces from a crank to the fingertips. It also uses an admittance-based controller parameterized by position, as opposed to an impedance-based controller parameterized by force. When setting the controller's target position, developers must use its native angular coordinate system. The Dexmo has a number of uninstrumented degrees of freedom. Mature forward models can reliably predict the hand pose, even with these unknowns. When it comes to computing angular controller parameters from a target pose in Cartesian space however, things become more difficult. Complex models that provide attractive visuals from a small number of sensors can be non-trivial or even impossible to invert. In this letter, we suggest side-stepping this issue. We sample the forward model in order to build a lookup table. This is embedded in three-dimensional space as a curve, on which traditional queries against world geometry can be performed. Controller parameters are stored as attributes of the sample points. To compute the driver parameters for a target position, the application constrains the position to the geometry, and interpolates them. This technique is generalizable, stable, simple, and fast. We validate our approach by implementing it in Unity 2017.3 and integrating it with a Dexmo glove.