An open-source multi-person virtual reality framework for interactive molecular dynamics: from quantum chemistry to drug binding

Conceptually, the mechanics of nanoscale molecular objects arise through electrostatic forces acting on particles in non-uniform fields, and are relatively well characterized owing to decades of study. Nevertheless, because dynamics at this scale differ from the familiar mechanics of everyday objects, they are often non-intuitive, even for highly trained researchers. Moreover, because molecular systems have many of degrees of freedom, their motion involves a complicated, highly correlated, and 3D many-body dynamical choreography with few analogues in day-to-day experience. We recently described how advances in virtual reality (VR) enable researchers to manipulate real-time dynamics simulations of molecular structures in 3D. In this article, we discuss VR's design affordances, outline cognitive and perceptual principles for understanding how people experience VR, and provide an overview of efforts to use immersive technologies for the molecular sciences. We also introduce 'Narupa', a flexible, open-source, multi-person VR software framework designed to enable groups of researchers to simultaneously cohabit real-time simulation environments and interactively inspect, visualize, and manipulate the dynamics of complex molecular structures with atomic-level precision. We highlight the potential of VR to furnish insight into microscopic 3D dynamical concepts. We outline a range of application domains where VR is proving useful in enabling research and communication, including biomolecular conformational sampling, transport dynamics in materials, reaction discovery using 'on-the-fly' quantum chemistry, protein-ligand binding, and machine learning potential energy surfaces. We describe ongoing experiments using sound and proprioception to enable new forms of integrated multisensory molecular perception, and outline future applications for immersive technologies like VR in molecular science.


Introduction
Owing to the fact that nanoscale molecular objects exist on very small lengthscales and move on very fast timescales which are different from those of our day-to-day phenomenological experience, our 'ways of knowing' the nanoscale world are indirect -i.e., our conceptions of this world often rely on abstracted and idealized models, and our perceptions of this world tend not to arise from our direct sensory perception, but are mediated by data-feeds from instruments. 2 Over the last few decades, there have been significant advances in the experimental and computational techniques that can be used to study, understand, and design nanoscale systems, helping to refine how we imagine structure and dynamics at the nanoscale. For example, methods like super-resolution fluorescence microscopy and cryo-electron microscopy offer a rich toolset for gaining insight into the dynamical choreography of nature at the molecular scale.
While instruments vary enormously in the kinds of data they provide, and the methods required to set up an experiment, they all more or less share the same high-level blueprint: each is designed to subject a molecular system to some external perturbation and then monitor its response, providing data for numerical analysis, which can be used to construct graphical representations such as plots, images and movies. It has been proposed that advances in nano-engineering may one day allow us to design and construct nanoscale structures and machines with the sort of surgical precision that is possible in the design and engineering of macroscopic objects. For example, in his oft-quoted 'plenty of room at the bottom' lecture, 3 Richard Feynman speculated on whether scientists would one day have the ability to carry out routine atomic level manipulation at the scale of individual atoms 4-5 -a kind of atomically resolved surgery -which remains a holy grail for scientists working at the nanoscale. As we make progress in our ability to engineer and design nano-scale structure, we can glimpse a new fundamental challenge emerging: namely, our ability to understand and engineer molecular motion, dynamics, and flexibility.
The human organism is a sophisticated sensory machine capable of integrating complex and simultaneous data across multiple channels, including the visual, auditory, olfactory, and somatosensory cortexes. Recent research in psychology and neuroscience has shown clear links between multi-sensory processing and attention. 6 Nevertheless, the sensory mechanisms we use to obtain insight and navigate the complex and dynamic terra incognita of nanoscale structures are limited. The large-scale tangible 3d models which were once commonplace within molecular research (discussed in section 3) have mostly been replaced by 2d plots, images, movies, and articles, primarily designed for parsing by our visual cortex. Lacking the ability to obtain direct sensory perception of the nanoscale objects on which we one day dream of carrying out atomic surgery, we rely almost exclusively on our sense of vision and our ability to undertake cognitive abstraction. In some sense, we underutilize the extremely rich sensory machinery which we have evolved as thinking and feeling beings, which enables us to process the intertwined matrix of sensations, perceptions, and information from the natural world around us. Sticking for a moment with the surgical metaphor, we can carry out a quick thought experiment contrasting the indirect (predominantly visual and abstract) methods which nanoscale scientists (let's call them nano-technicians) use to perceive their objects of study, versus the array of techniques that trained surgeons use to perceive their objects of study. In contrast to the nano-technician, the surgeon's practice relies on a more direct set of perceptual methods, which actively integrate a wider range of sensory input across the visual, auditory, olfactory, and the somatosensory systems, all of which are crucial to the surgeon as they explore the terra incognita of a living body, its tissues, and its recesses. For example, the 'feel' of a tissue (e.g., its texture and its response to pressure), mediated by mechanoreceptors and thermoreceptors in the surgeon's somatosensory system, provides indispensable information for augmenting the surgeon's visual inputs as they navigate bodies, and for informing idealized mental models of anatomy, tissues, physiology, and histology. Recent research even suggests that the surgeon's instruments may form part of their extended somatosensory system. 7 Compared to the nano-technician, the surgeon's practice requires sensitivity to a broader spectrum of integrated sensations and perceptions, owing to the fact that their knowledge practice requires 'embodied' forms of cognitioni.e., processes of thinking through doing, sensing, and moving in order to obtain insight about the natural world. The knowledge practice of the nano-technician depends far less on the body's own evolved sensory mechanisms, and instead relies primarily on cognizing abstracted mathematical models and engaging with instruments which produce graphs, images, and movies. Because the instrument does the sensing, thinking through doing, sensing, and moving is less important here. In this respect, the nano-technician has a knowledge practice which is always 'one step removed' from the objects of study. In his visionary essay The Ultimate Display, Ivan Sutherland hinted at this sense of being 'one step removed', highlighting our lack of intuition for those realms to which we do not have direct perceptual access: We live in a physical world whose properties we have come to know well through long familiarity. We sense an involvement with this physical world which gives us the ability to predict its properties well. For example, we can predict where objects will fall, how well-known shapes look from other angles, and how much force is required to push objects against friction... We lack corresponding familiarity with the forces on charged particles, forces in non-uniform fields, the effects of non-projective geometric transformations, and high-inertia, low friction motion… 8 Figure 1: schematic of Narupa, our open-source multi-person iMD-VR system, showing two users manipulating a simulated C60 molecule. Each user's position is determined using a real-time optical tracking system composed of synchronized IR light sources. Each user's HMD is rendered locally on a computer fitted with a suitable GPU; molecular dynamics calculations and maintenance of global user position data take place on a separate server, which can be cloud-mounted. As long as the network connecting client and server enables sufficiently fast data transfer, system latency is imperceptible to the human senses.
Over the past several years, our laboratory has been carrying out an interdisciplinary research program developing technologies which enable direct multisensory perception of molecular simulations. 1,[9][10][11][12][13] The recent emergence of robust virtual reality (VR) technologies has been a key enabler in facilitating these efforts. In this perspective, we will outline our recent experiments combining affordable new forms of high-end virtual reality technologies with real-time interactive molecular dynamics (MD), which enable nano-technicians to manipulate rigorous real-time simulations of molecular systems, as shown in Fig 1 and Video 1 (vimeo.com/244670465). Application of VR in order to provide molecular insight remains a research area which is still in its infancy. Nevertheless, it is fertile territory for exploration, owing to the fact that the architecture of molecular and nanoscale objects is characterized by considerable complexity which is intrinsically (a) three-dimensional, (b) dynamic, and (c) difficult to perceive directly using experimental tools.
New tools like VR have the potential to change the kind of science that people undertake, 14 but there remains a great deal of work to be done in developing the technology and understanding the kinds of applications to which it is best suited. The ideal scenario is one in which technology development and scientific applications are closely coupled, so that each can inform the other -i.e., so that technological developments enabling new applications, and new applications informing directions for focusing subsequent technological developments.
Psychologists and designers often refer to the 'affordances' of a particular environment or technology. 'Affordances' are the features of a particular environment or technology that elicit a particular kind of behavior or interaction. The concept of an 'affordance' was first introduced by the psychologist James Gibson, [15][16][17] and popularized by Don Norman as a concept for thinking about human-computer interaction design. 18 For example, a computer screen-mouse-keyboard combination clearly has a distinct set of design affordances compared to a virtual reality interface. Both technologies enable the rendering of computer generated images; however, a virtual reality interface allows one to walk around in space to inspect the image from various angles and quickly intuit depth, while a screen requires that the user observe the image from a particular perspective and carry out a sequence of 2d manipulations to understand depth. As another example, the two different technologies enable rather distinct forms of human-computer interaction. A keyboard primarily emphasizes the transmission of text-based information via button presses and a mouse affords one-handed manipulations in two dimensions in order to navigate the screen. Neither a set of wireless tracked VR controllers nor a pair of VR gloves is well suited to rapid text input like that which is afforded by a keyboard, but they afford precise and intuitive two-handed spatial manipulation. From a research perspective, a key question for the molecular sciences involves understanding those particular areas where the affordances of new VR environments (compared to 2d screenmouse-keyboard environments) enable deeper insight, better intuition for nanoscale design and engineering, more effective scientific communication and collaboration, and accelerated research progress in understanding important molecular systems and concepts. 19 In the remainder of this article, we describe people who use VR as 'participants' rather than 'users', recognizing that VR is different from other forms of human-computer interface because the human can actively participate in the virtual world. 20 In a recent paper, Goddard et al. have outlined a number of the software tools which have emerged for use in headmounted virtual reality environments, 21 many of which have their conceptual origins in software frameworks originally designed for use in stereoscopic, multi-projector CAVE-like environments. [22][23][24][25][26][27][28] In the last few years, software frameworks which have emerged for head mounted VR displays can be broadly schematized according to the extent of active participation which they enable. These include applications: • Enabling a participant to inspect either a static molecular structure or a pre-recorded molecular trajectory in three dimensions. In such applications, the role of the participant is primarily observational; the head mounted display essentially operates as a mechanism for enabling a 360-degree video where the participant can look around; [29][30][31][32] • Where a participant has a more active role, and can navigate a simulated space to inspect a structural rendering from various angles and quickly intuit depth. In many cases, participants are able to manipulate aspects of the structural model or trajectory -e.g., changing its representation and rendering options, pausing and resuming the trajectory, showing or hiding certain parts of the structure, rotating/translating the model, and perhaps querying structural aspects of the model (e.g., bond distances, angles, residue names, etc.); 21, 33-37 • Enabling a participant to carry out modifications on a molecular structure, e.g., to build or modify molecules by connecting together atoms or amino acids, replacing one functional group with another functional group, etc.; 33 To date, our research has specifically explored interactive molecular dynamics in virtual reality (iMD-VR) -i.e., applications that emphasize real-time simulation, in which the affordances of two-handed interaction within the threedimensional VR space enable a participant to 'reach out and manipulate' rigorous MD simulations, and carry out detailed three-dimensional structural manipulations in real-time. 1 Understanding the perceptual mechanisms enabling cognitive abstractions to be transformed into tangible dynamic realities which participants report they can 'feel' is a fascinating area spanning computing, human-computer interaction, and cognitive science (discussed further in section 4). The iMD-VR prototype which we described in ref 1 was designed to investigate whether real-time iMD-VR using cloud-mounted supercomputing could be used to accelerate molecular simulation tasks in preset simulation setups (discussed in section 5.1). The proof-of-principle framework we described in ref 1 was limited in a few key respects: (1) it was designed for individual iMD-VR participants, and did not enable access to the multi-person functionality shown in Fig 1; (2) the simulations available to users were predefined in advance, and could not be modified; and (3) it required access to cloud computing over fast networks. To coincide with the publication of this article, we have made our iMD-VR software framework publicly available as an open-source project which we have provisionally called 'Narupa'. Source is available at gitlab.com/intangiblerealities along with a stable executable build at irl.itch.io/narupaxr. The name 'Narupa' combines the prefix 'nano' and suffix 'arūpa' (a Sanskrit word describing non-physical and non-material objects), which represents our attempt to capture what it is like to interact with simulated nanoscale objects in VR. Narupa builds on the capabilities of the proof-of-principle framework outlined in ref 1: (1) it enables multiple participants to inhabit the same iMD-VR environment; (2) it enables participants to set up their own simulations using a flexible force API (discussed further in section 2.5); and (3) it can be set up to run on local networks (i.e., does not require access to cloud computing over fast networks). Narupa captures much of our iMD-VR research work to date, including a variety of research applications (described in section 5). It has three key design emphases: (1) the integration of real-time simulation methodologies into our interaction framework, which enables participants to 'feel' the dynamical responses of molecular systems; (2) the ability to make the VR experience one which is social, so that multiple participants are able to cohabit the same virtual world together, either together in the same room, or distributed remotely; and (3) active engagement with designers, artists, and human-computer interaction (HCI) experts, in order to create a framework which not only has scientific utility, but which represents best HCI practice, and which is aesthetically compelling. 10,12,38 This latter point has been crucial to our development process and is particularly important given the level of immersion which can be achieved in VR environments.
Faced with traditional scientific publication formats, one of the most well-known difficulties for workers in VR concerns exactly how to write about it. 20 For example, VR pioneer Mel Slater has suggested that the difficulty in writing about VR arises from the fact that it offers to participants experiences which have a perceptual analogue in 'physical reality', but also things which go beyond the bounds of physical reality. This a particularly important point for the purposes of this article, given that the 'direct experience of molecular realities' falls into a class of perceptual experience which does not have a very good analogue in our day-to-day phenomenological experience, and therefore makes for a challenging piece of writing! Sections 1 and 4 of this article specifically reflect the difficulty in trying to explain in text the more qualitative perceptual aspects of what it is like to 'feel' simulated molecular objects in VR, and utilize a style which is likely to be unfamiliar to workers in the physical sciences. Specifically, these sections are written as firstperson and third-person accounts from the perspective of D. R. Glowacki, using a style that is becoming increasingly prevalent within the human and social sciences 39 and also within the field of human-computer interaction. 40 Throughout, this article refers to a number of videos (listed in Table 1), each with a hyperlinked URL, which we encourage the reader to watch, because we have found that they go a long way toward overcoming the difficulties inherent in writing about the more qualitative, experiential, and perceptual aspects of multi-person iMD-VR.  Table 1: outline of the videos discussed in this article, along with their respective URLs, a brief description, and the force engine utilized to make the video

Touching Molecular Abstractions
I specifically remember (in Sept 2016) the first time that I 'went into VR' and had an experience of reaching out to manipulate a real-time simulation of molecular system. The system was OH + CH4, which we were simulating using a multi-state reactive EVB model. [49][50] This system, probably the best studied oxidation reaction in all of atmospheric chemistry, is one which I had only ever written algorithms to simulate, following the same standard protocol as every other computational chemist: input file preparation… batch submission… waiting… waiting… view & analyze output files… identify mistakes in input files…. Modify input files… batch submission… waiting… I went into VR, reached out, and used my wireless force 'tweezers' to 'lock onto' and manipulate the OH using one hand and CH4 using the other. As I manipulated these molecules, I could sense the nuances of the damped numerical integration algorithm which we had written to simulate their dynamics -the slight lag as they overcame their inertia and accelerated toward my handheld force 'tweezers'; the vibrational wobbliness of the Hydrogens as I slightly shook the methane Carbon, the non-local electrostatic repulsion which arose as I brought them in close proximity to one another, and the translational and vibrational damping they experienced when I released them, as a consequence of the thermostat we had implemented. After inspecting their otherworldly dynamics, I attempted a reaction to make CH3 + H2O, which I knew our MS-EVB model should be able to capture. I accelerated the OH toward the CH4 and then released its, mimicking a sort of crossed molecular beam experiment at a high collision energy. The first few times I tried, I didn't put enough kinetic energy into the relative translational motion of the reactants, and the system couldn't quite get over the barrier.
A few other times, there appeared to be enough energy in the translations; but I couldn't get over the entropic barrieri.e., the orientation of the OH wasn't quite right to abstract a hydrogen. But I finally I got it: I sent the OH and CH4 at one another with enough translational energy, and a good enough orientation to overcome entropy's randomizing influence. Video 2 (vimeo.com/315218999) illustrates some of the results from these early experiments.
In those first ten minutes of playing with a reaction that I had read about and studied ad nauseum, I still believe that I gained more intuitive insight into entropy that I had in years of studying statistical mechanics. Having subsequently used our VR system as tool for explaining entropy (a concept many find difficult to understand, from undergraduates to senior professors), I have witnessed postgraduate students, post-docs, and faculty members gain similar insight, to the extent that I think that I've convinced myself this is a real effect. The exhilaration that I felt after that first experience arose in part because I had reached out and touched objects whose dynamics -until that point -had primarily been accessible to me as abstracted numerical and algorithmic concepts. In the weeks following my initial enthusiasm, I Seeing different forms of abstraction play out as different forms of embodied expression raises an interesting question: might it be possible for things to also work the other way around -i.e., can different ways of knowing (relying on a broader spectrum of sensory inputs beyond vision and abstract cognitive models) usefully inform the shape of our abstractions, and our ability to undertake effective scientific communication? Tools that enable us to develop an intuitive feel for how nanoscale objects behave could ultimately furnish insight which augments our abstract models, and help to make progress in understanding these systems. At the moment, computational fields tend to privilege those who are able to deftly process a particular flavor of mathematical cognitive abstraction. I remember one particular conversation following a lecture I gave at IBM research in Zurich in the summer of 2018. Over lunch, a respected professor expressed to me his concern that the intuition offered by our VR framework made things 'too easy'. He seemed concerned that, should students and researchers be presented with such visceral experiences of nanoscale reality, they would feel they no longer needed to toil away in the details of abstraction, and that this might be a bad thing for their education. My response to him was that cognitive abstraction was unlikely to cease becoming an important part of computational science anytime soon; however, it was also important to recognize that intelligence takes many different forms. I also argued that, if the technology exists for turning these cognitive abstractions into more embodied forms of knowing, accessible to a broader range of our sensory modalities, then there was no reason to privilege one particular way of knowing. By developing tools which engage a broader cross section of our collective sensory capabilities, the potential exists to make complicated problems more accessible to a wider spectrum of intelligence, perhaps illuminating aspects of our cognitive abstractions which have been heretofore inaccessible.

Defining what constitutes 'Virtual Reality'
VR technologies offering sophisticated forms of embodied digital experience have made a recent resurgence, and a detailed review of the history of VR is available in a number of other places. 20,51 For historical context, it's worth pointing out that VR technologies have been available for much longer than the latest hype cycle. In the medical field, for example, VR has been used to enable detailed surgical simulations, and has an established track record for more than a decade. A number of studies have quantitively shown that VR-trained surgeons complete surgical procedures faster, with significantly lower error rates (for example, a 2002 paper reported 7x fewer errors). 52 Animation firms like Dreamworks have reported cost reductions of 3x following adoption of VR technologies which allow their digital animators to reach into scenes and carry out direct manipulations (e.g., to animated characters) in 3d. 53 The distinguishing feature of the current VR resurgence is the fact that technology which was previously only available in specialist research labs or medical school facilities, is now available at considerably lower prices. Driven mostly by the consumer gaming market, recent advances in VR hardware provide commodity-priced solutions to the longstanding problem of co-located interaction in three dimensions. Human computer interaction technologies are considered to be co-located when there is a perfect alignment between the interaction sites in physical space and the interaction sites in virtual space. 54 Touchscreens, for example, solve the problem of 2D co-location because the interaction site in physical space is identical to the interaction site in virtual space. This is a significant reason why children at a very young age find it straightforward to navigate a touchscreen. Combining infrared optical tracking, inertial movement units (IMUs), and application specific integrated circuits (ASICS), commodity VR technology such as the HTC Vive offers fully colocated interaction in three dimensions, tracking a participant's real-time 3D position with errors less than a centimeter, and allowing participants to reach out and touch simulated objects in the virtual world, as shown in Fig 1 & Video 1.
A wide array of relatively distinct technologies are currently available which are often referred to as 'virtual reality'.
However, it is important to address a widespread misconception: strapping a screen to one's head implies nothing about the level of immersion the participant experiences. VR pioneers like Jaron Lanier have emphasized this point, highlighting the fact that a number of frameworks which are often referred to as 'virtual reality' enable participants to do little more than 'just looking around in a spherical video'. 51 Lanier, along with other HCI researchers, has made a point to distinguish those technologies which do afford reaching out to touch the virtual world: If you can't reach out and touch the virtual world and do something to it, you are a second class citizen within it... a subordinate ghost that cannot even haunt. 51 From this point on in this article, we use the term 'virtual reality' specifically in reference to technologies like the Oculus Rift and the HTC Vive, whose design affordances enable one to 'reach out and touch' simulated realities. In an excellent recent review of virtual reality principles and applications, 20  augmented reality (AR), and mixed reality (MR). While a detailed discussion of the available emerging technologies is beyond the scope of this paper, and complicated owing to the fact that the technology is evolving rapidly, we note that the various forms of embodied digital interaction (whether they are forms of virtual, augmented, or mixed reality) are sometimes referred to on aggregate as 'XR', or 'extended' reality. Having experimented with a wide range of available technologies, we have found the aforementioned HTC Vive to be generally robust for the purposes of molecular simulation and visualization. Moreover, it also allows us to design experiences which enable groups of people within the same space to simultaneously co-habit the same simulated virtual world. However, the technology is steadily advancing, and many of the ideas in this paper are not limited to VR. They could easily be extended to any of a range of XR technologies, so long as their affordances enable one to 'reach out and touch' simulated realities, and then carry out spatial manipulations with a sufficient degree of precision so as to enable workers to carry out detailed atomic adjustments and rearrangements.

Narupa: an open-source VR framework
Building on our previous work using optical tracking technologies to interactively steer real-time molecular dynamics simulations, 9  To date, our available resources and space constraints have allowed us to simultaneously co-locate six participants in the same room within the same simulation. The interaction shown in Video 1, where multiple participants in the same room are able to easily pass a simulated molecule between themselves (or e.g., collaboratively tie a knot in a protein) as if it were a tangible object, represents a class of simulated virtual experience which is simply not possible within the large-scale immersive stereoscopic CAVE environments that have become popular within academic and industrial research institutions around the world. 22 Fig 1 schematically illustrates how the server can be cloud-mounted, making it straightforward for remotely located workers to occupy the same virtual space. While a real-time molecular dynamics simulation of C60 is relatively cheap, the client-server architecture shown in Fig 1 enables

Force biasing
The VR-enabled interactive MD shown in Video 1 effectively amounts to a real-time classical dynamics simulation which responds to real-time biasing forces, building on our previous work using optical tracking technologies to interactively steer real-time molecular simulations. 9 In classical mechanics, the time-dependent dynamics of molecular systems are solved by numerically integrating Newton's equations of motion. The vector of forces acting on a set of atoms F(t) can be written in terms of the system's potential energy V, i.e.: Eq (1) where q is a vector containing the position of each atom in the ensemble. Our system effectively allows participants to interactively chaperone a real time MD simulation by splitting V into two different components Eq (2) where Vint corresponds to the system's internal potential energy, and Vext corresponds to the additional potential energy added when a participant exerts a force on a specific atom (or group of atoms) when they grab it using the handheld wireless controller shown in Fig 1. Substituting Eq (2) into Eq (3) then gives Eq (3) The external forces can be implemented in a number of ways, including by projecting a spherical Gaussian field into the system at the point specified by the participant, and applying the field to 'lock onto' the nearest atom $ as follows: (4) where , ' is the atomic mass of the nearest atom,is a scale factor that tunes the strength of the interaction, qj is the position of atom j, gi is the position of the interaction site, and . controls the width of the interactive fields.
is variable parameter that the participant can set, so as to achieve responsive interaction while preserving dynamical stability, and . is typically set to the default value of 1nm. While an interaction is active, it is always applied to the same atom (or group of atoms), which means a participant can dynamically adjust the course and strength of the interaction simply by repositioning their field with respect to the atoms with which they are interacting, until they decide to 'let go'. As an alternative to the Gaussian potential outlined above, we also use spring potentials, a technique used by previous iMD implementations that predate modern virtual reality, 55 which take the following form: (5) The Gaussian field has the advantage that the maximum force is limited by the Gaussian height, while the spring has no limit. To prevent instability in the molecular system, the maximum force a participant can apply is limited so as not exceed a maximum value.
The Gaussian potential has more flexibility for tuning the strength of the potential, and the fact that it decays to zero at long distances reduces the chance of accidentally exerting a large force on an atom. On the other hand, the spring potential may be more intuitive in some cases, because it allows one to increase the strength of the force by simply increasing the distance. Determining which interactive potential is better for particular applications remains a question for further study in participatory tests. Much of the 'art' of iMD-VR involves understanding how to set the interaction parameters in Eq (4) and Eq (5) so as to enable smooth, stable, and intuitive dynamics for a given dynamics simulation setup, which does not excessively perturb the system. Narupa enables participants to easily modify the value of the scaling parameter c from within VR, tuning the interaction on the fly as they experiment with a given system.

Interaction Selection and Force Damping
The applications discussed below in section 4 led us to design new interaction algorithms beyond those described in section 2.3, in order to facilitate molecular manipulation in more complex systems like biomolecules. In particular, we realized that there were many cases in which it was advantageous to be able to apply a force to an entire subunit of a given molecular system, for example if one wishes to manipulate a portion of a protein's secondary structure and ensure that it remains intact. To address this, we have implemented a selection interface, shown in Video 3 (vimeo.com/305459472), which allows a participant to identify a group of atoms which they would like to manipulate (a similar selection interface also enables a participant to choose different renderings for different parts of the molecule).
Having specified a particular selection, the participant can then exert an interactive force on the center of mass of the entire subunit, in a fashion that keeps secondary structures intact. Such a method is also extremely useful studying systems linked to protein-ligand binding, enabling a researcher to for example exert an interactive force on an entire ligand. If we let 0 ) be the center of mass of the atoms included within a particular selection, and assume that an interactive potential is applied to this group, then the overall force to apply to the atoms, 1 * , is calculated as in the single atom case described by Eq (4) and Eq (5), except instead of a single atomic position, the center of mass is used as the center of interaction, effectively substituting 0 ) for ' ' , and setting , ' to 1, which gives: Eq (6) This total force is divided amongst the atoms and applied in a mass-weighted fashion as follows: Eq (7) The resulting interaction allows complex manipulations which for example, can preserve protein secondary structure.
We also realized that, similar to medical surgery, the ability to carry out manipulations on a real-time MD simulation depends critically on the nano-technician's ability to make gentle movements which do not irreversibly perturb too many parts of the system. Interacting with the atomic system by applying bias potentials enables the motion of the system to be integrated as usual. However, in some cases the accumulation of biasing forces on the system can have unintended consequences, as the forces are integrated into the velocities of the atoms of the system. This can make a system challenging to control, because the only way for an atom (or selection thereof) to lose the momentum added by participant manipulation either by: (1) velocity-damping energy transfer through collisions with other parts of the system, (2) velocity dampening and friction from the thermostat, or (3) the participant applying a force in the opposite direction. One strategy which avoids excess momentum build-up during interactive molecular simulations involves performing continuous energy minimization 56 rather than continuously integrating the system dynamics. This strategy works well for small molecular systems and reactions, in which manipulating a single atom and having the system constantly minimize its energy is tractable. Inspired by this strategy, we have developed a hybrid method which uses velocity reinitialization as a way to mitigate the effects of accumulated momentum. Upon interacting with a single atom or group of atoms, the molecular dynamics continues to be integrated as usual, except now the interactive biasing potentials are also being applied. Once the participant stops interacting with the atoms, the atoms involved in the interaction have their velocities randomly drawn from a Maxwell-Boltzmann distribution at a target temperature of 67, where 7 is the target equilibrium temperature of the thermostat, and 6 ∈ (0,1] is a scale factor chosen by the participant, which by default is set to a value of 0.5. To maintain stability, velocities are typically reinitialised to a temperature lower than the target equilibrium temperature. This is similar to the Andersen thermostat, except rather than being applied to atoms at random, the velocity re-initialization is specifically targeted at those atoms involved in an interaction. By reinitializing the velocities, any overall momentum in the atoms in a particular direction is removed. Of course, there is a timescale associated with re-equilibration, but applying interactive forces already takes the system out of equilibrium, and the benefit of being able to accurately manipulate groups of atoms which are in an approximately correct ensemble, outweighs this effect.
Scheme 1: Outline of the Narupa simulation server. A simulation consists of an integrator and an 'atomic system', which in turn consist of topology, force field and thermostat modules. All modules conform to APIs that enable them to be substituted for existing implementations.

The Narupa Force API
Simulating the dynamics of a particular molecular system requires an engine to calculate the internal forces. Here we benefit from the fact that our framework has been designed to flexibly communicate with a wide range of force engines via a defined application programming interface (API). As illustrated in Scheme 1, the API functions in a straightforward manner, sending coordinates to a force engine, and receiving forces in return. The idea here is that the force engines which communicate to Narupa can effectively operate as 'black boxes', which simply plugin to Narupa.
For example, we have connected our API to the following force engines: an implementation of the MM3 forcefield 41,57 ; the OpenMM molecular dynamics package, which allows access to a range of GPU-accelerated force engines 42 ; PLUMED, using the VMD IMD API, 55 which is capable of communicating with a wide range of programs, e.g., GROMACS 44 and LAAMPS; 58 the tight binding density functional theory package DFTB+ 43 ; and the semi-empirical quantum chemistry package SCINE. 46 The flexibility of our API enables us to undertake VR-enabled interactive simulations on a wide range of systems, and optionally include either implicit (e.g., continuum) or explicit (e.g., TIP3P water) solvent models. In cases where we model explicit solvent, we do not typically visualize the solvent molecules, in order to maintain clarity and high-quality rendering. Force integration is typically undertaken using a Velocity Verlet integrator, with an Andersen thermostat 59 set to a predefined target temperature. A time step of 1 fs is typical, although we recently implemented the SETTLE and CCMA constrained dynamics algorithms, 60-61 which enables us to achieve stable dynamics utilizing greater timesteps of up to 2 fs for biomolecular systems. The scientific applications outlined in section 5 benefit from the flexibility of this force plugin architecture. Narupa includes options which enable participants to store trajectories which they generate whilst in VR, for subsequent analysis and post-processing.

Narupa renderers
The flexibility of the Narupa force API enables the simulation of a wide range of molecular systems, and we are consequently working to implement a number of aesthetics and rendering schemes. Familiar styles such as ball-andstick, liquorice and VDW representations are available, as well as a ribbon renderer for protein structures, some of which are shown in Video 3. These styles can be applied to any selection layer created in VR, enabling intuitive customization. The visualization settings can then be stored for repeat use or transmitted to other participants to synchronize visualization for shared experiences. High performance rendering of molecular structures in VR is a challenge, requiring a target frame rate of 90 frames per second for each eye, which is further complicated by the requirement for rendering of simulations that are continuously updating from data being received over the network. We are currently working to improve rendering performance, and build additional renderers, such as a secondary structure renderer which can dynamically indicate biomolecular features such as alpha helices and beta sheets. For example, Video 4 (vimeo.com/315239519) shows a first person perspective of a real-time MD simulation of neuraminidase (PDB 3TI6) displayed using a prototype secondary structure renderer which we will soon add to the main Narupa source distribution. This renderer uses the DSSP algorithm 62 to calculate the hydrogen bonds and secondary structure present in the molecule. This is combined with a cubic Hermite spline passing through the alpha carbon chain of the enzyme to render a continuous 3D chain. The secondary structure assignment is used to color the chain appropriately and to stretch the chain to highlight arrows and helices. The video shows how bits of the secondary structure flicker in and out over the duration of the MD simulation.

Narupa examples
Narupa comes packaged with a number of stable examples, which participants can inspect in order to guide them in setting up their own interactive simulations. At present, the following examples are packaged with Narupa: • Two C60 buckyballs at 300K simulated with a timestep of 1fs. This is the usual introductory simulation for familiarizing users with the iMD-VR environment. • A carbon nanotube and methane molecule simulated at 200K with a 0.5fs timestep. The 'task' here is to pass the methane molecule through the nanotube.
• A short helicene fragment at 300K and with a 1fs timestep, which users can manipulate to switch between conformations characterized by either a left or right-handed screw sense.
• A 17-ALA helical peptide chain at 300K and with a 2fs timestep simulated with the Amber99SB forcefield, used to demonstrate the ability to tie a molecular knot. This simulation requires the OpenMM package.
• The enzyme H7N9 Neuraminidase and the drug Oseltamivir, to demonstrate drug unbinding and rebinding, with the protein using the Amber03 force field, and the drug force field parameterized using GAFF. This simulation is run using an Andersen thermostat at 300K, with a Verlet integrator with timestep 0.5fs. This simulation also requires the OpenMM package.
• The smallest known knotted protein MJ0366 in its native state, to illustrate the utility of 3D visualization, simulated with the Amber03 forcefield using an Andersen thermostat at 300K with a Verlet integrator with timestep 0.5fs. This simulation requires the OpenMM package.

Designing new forms of molecular interaction: a brief history
The use of VR in surgical contexts -where it is intended to simulate a surgeon's experience of manipulating and cutting human tissues -is rather distinct from the use of VR to manipulate molecular structure and dynamics. Perhaps the most important difference pertains to the design reference. Surgical simulator applications have a well-defined and measurable design reference, with a well-defined design question: how does the simulation 'feel' compared to an experience involving human tissue? Molecular applications, on the other hand, have no similarly well-defined design reference -i.e., there is neither a clear answer to the question "What does a molecular system 'feel' like?' nor to the question "what should a molecular system 'feel' like?". The lack of reference is a central part of what makes developing a real-time molecular simulation and manipulation framework such a fascinating and creative challenge, which must necessarily consider aesthetics, design, and participant psychology in order to be effective.
Historical efforts to use computing for designing new forms of molecular interaction have been strongly influenced by the kinds of tangible (e.g., plastic, metal, wood, etc.) molecular models that have been historically important in chemistry and biochemistry -e.g., tangible three-dimensional (3D) molecular models like Dorothy Hodgkin's crystallographic model of penicillin's structure, 63 Pauling's models to identify the structure of alpha-helices, 64 Watson and Crick's famous DNA model, and the 65 large room-sized models, made from e.g., wire, plastic, brass, balsawood, and plasticene which were used to refine and represent protein crystal structures by pioneers such as Kendrew, Perutz [66][67] , and Levitt. 68 Physical models like these provide structural insight, but cannot represent the often non-intuitive mechanics that determine how molecules move and flex. The first researchers to pursue the idea that computers could be used to construct tangible molecular models whose motion was based on rigorous physical laws included Fred Brooks 69 and Kent Wilson 70 , pioneers whose interests spanned both scientific simulation and human-computerinteraction (HCI). Brooks and Wilson were amongst the first to imagine how -were such a thing possible -such technology would offer better insight, and also have the potential to accelerate research workflows. Following on from the ideas outlined by Sutherland, they speculated that interactive molecular simulation (iMS) frameworks would lead to models which would be as intuitive to manipulate as the old tangible models, but which followed rigorous physical laws, and which could be used to tackle hard rare event sampling problems. Brooks designed an immersive six-degreeof-freedom force-feedback haptic system which participants could manipulate to carry out molecular docking tasks. 69,71 Inspired by this work, Klaus Schulten and co-workers subsequently miniaturized Brooks' setup: by manipulating a desktop-mounted haptic pointer, participants could steer the real-time dynamics of molecules rendered on a stereographic screen. 72 This has remained the dominant setup which has since been extended by others, including Marc Baaden, Markus Reiher, Todd Martinez, and co-workers to interactively manipulate molecular mechanics 73 and quantum chemistry simulations. 56,74 To date, the vast range of published iMS approaches have utilized the approach which Schulten et al. adapted from Brooks 69 -i.e., the participant manipulates what is essentially a small pen-shaped mouse that can move in three translational dimensions (x,y,z), and three rotational dimensions (rx, ry, rz). This pen-shaped mouse is attached to a robotic arm which can be programmed to 'resist', a phenomenon which workers in HCI often refer to as 'forcefeedback'. Brooks' and co-workers original system was mounted at the UNC Dept of Computer Science from 1965 -2000 with NIH support. 69,71,[75][76] In 1989, Brooks described a six-degree-of-freedom (DOF) force-feedback haptic system (built from an enormous robotic arm called the Argonne Remote Manipulator, or 'ARM') which participants could manipulate in order to carry out molecular docking tasks. 76 Building on evidence that force feedback tools allowed participants to efficiently carry out remote manipulation tasks relevant to space research, underwater operations, and nuclear/radiation laboratories, 77 Brooks sought to investigate whether the same was true for manipulation of molecular models. He designed a study in which seven participants were instructed to carry out a simple force minimization task emulating a ligand-receptor molecular docking-type problem -namely a rigid diatomic molecule in which each atom is acted upon by three unique harmonic forces, and initialized in a non-optimal configuration. In a first set of experiments, participants manipulated a six DOF force feedback arm "blind" -i.e., guided by nothing but haptic feedback. In a second set of experiments, Brooks turned off the haptic motors. Rather than forcefeedback, participants relied upon visual feedback from a stereoscopic display showing changes in the force vectors as they carried out the optimization. Subsequent analysis of this simple task showed that participants were able to minimize the interaction potential energy a factor of two faster relying upon "blind" force-feedback compared to visual feedback.

'Feeling' molecules in virtual reality
Brooks did much to develop practical iMS strategies, and nearly everybody who has persisted in exploring iMS over the years has adopted his 6-dof haptic approach. The miniaturization of such haptic devices has also made them practical for use within surgical simulators, where they can operate (for example) as a surgical knife, or be programmed to accurately simulate the resistance of tissues. As a result of the work by both Brooks and Wilson, many workers in iMS have concluded that simulating the 'feeling' of a molecular structure requires the use of force feedback haptics connected to robotic arms. One problem with these sorts of haptic devices is that they face a well-known limitation in their ability to achieve what HCI experts call 3D 'co-location' (described in section 2). For interactive molecular simulations, 3D co-location is an important design consideration, owing to the fact that molecules are 3D objects which move in 3D. In principle, co-located solutions involving haptics are possible -e.g., by co-locating the haptic device within the VR environment. However, such strategies require compatibility between multiple layers of non-commodity technologies, whose technological cost and sophistication may outweigh their benefits. Moreover, haptic technologies face fundamental limitations, owing to the fact that while there are excellent solutions available for specific types of interaction (e.g., pushing a needle through tissue in a surgical simulation, or using an exoskeleton to apply force feedback to an arm), there are no generalized solutions in the form of a single device which enables participants in a VR environment to feel anything (e.g., in the same way that visual or auditory display can be programmed to display anything). For example, Slater has argued that a generalized haptic solution is likely only possible in the form of a direct brain interface, in which case VR then becomes a branch of applied neuroscience. 78 Haptic technologies like a robotic arm which I can pull, and which then pulls back, offer one particular form of 'felt' sensation; however, our own research experience to date strongly suggests that haptic pointers are not required to achieve a sense of feeling, and that felt sensation can be accomplished via proprioceptive mechanisms. Somewhat surprisingly, our experience of taking thousands of people into VR over the past few years, and enabling them to manipulate a range of different molecular structures, has shown that people do indeed 'feel' molecular responses as they interact with and manipulate them in VR. One particularly notable example which I remember occurred during a visit by Professor Keiron Burke to Bristol. When I first offered to take Keiron into VR with me, he made no efforts to withhold his skepticism -to the extent that I wondered whether I should simply abandon the idea and talk to him about our more conventional research activities. I persevered anyway, and by the end of a 30-minute experience, Keiron was positively buzzing with possibilities for extending what I had shown him. I suspect that he had a similar experience as I did when I first entered into VR, and found myself able to reach out and directly access dynamical structures that I had previously been able to access primarily as cognitive abstractions. I remember one comment Keiron made in particular. Having just instructed him to 'thread methane through a nanotube" (at which he was very proficient), I then showed him a simulation of a small peptide (17-Alanine), which I instructed him to perturb from its native structure and then tie into a knot (at which he was also very proficient). While he was manipulating the peptide, he said something along the lines of "this feels so much different than the nanotube and fullerene". In what follows, I will refer to this particular experience as the 'Burke Perception Experiment' (BPE).
I specifically remember the BPE because myself and my research colleagues often hear comments along these lines when we take people through our progression of VR simulation demos. In fact, it has become so commonplace that we have begun to design human-computer interaction experiments to try and unravel why people say this. If you watch people from outside of VR, as shown in Video 1, they appear to be grasping at air. They are not touching anything physical. And yet multiple people, from a wide range of backgrounds, consistently comment on the fact that different molecules simulations 'feel' differently. In a first attempt to unravel the mechanisms which might be at play here, we have been developing a concept of 'layered perceptions'. [79][80] At the moment, we believe that one's ability to 'feel' a molecular object in VR arises from a layering of visual perception on top of proprioception (the non-visual sense through which we perceive the position and movement of our body). So when Keiron Burke reaches out to 'touch' a nanotube in VR, he locks his force tweezers onto an atom (or selection atoms) in a nanotube, whose underlying physics are dominated by covalent interactions (simulated in real-time). The form of these forces requires Keiron to move in a particular way in order to make the system respond as he wishes. The protein, on the other hand, has dynamics which are largely governed by much weaker non-bonded interactions. And therefore, Keiron must move in a slightly different way in order to tie the protein into a knot. Our working hypothesis is that Keiron's proprioceptive sensations are working alongside his visual sense to project a sense of 'feeling' onto objects which are otherwise only virtual -i.e., his brain is integrating visual and proprioceptive details to 'fill in' the details of what such an object would feel like. This hypothesis is grounded in part from published work demonstrating that virtual reality can be used to heighten proprioceptive recovery in stroke patients, 81 along with research showing that well-constructed VR experiences operate so as to encourage the brain to 'fill in' the perceptual details of a given scenario. 20 We are currently working to design experiments to test these hypotheses in further detail.
Whatever the precise mechanism, it appears that people can 'feel' simulated objects which do not have a material essence. From that sense of 'feeling', they can derive a sort of embodied awareness as how nanoscale systems behave (at least within the approximation of classical dynamics on an approximate PES), and respond to perturbation. This is an important insight because it means that it is possible to 'feel' a molecule without expensive haptic technologies, which are non-commodity pieces of equipment and therefore tend to be rather expensive and cumbersome. Moreover, by heightening our proprioceptive sensitivities, it may be possible to enhance our ability to 'feel' simulated realities.
Because there is no design reference for what a molecule should 'feel' like, using subtle mechanisms like proprioception represent an approach which is equally reasonable compared to more obvious haptic mechanisms, and the extent to which we can effectively design for the proprioceptive sense of feeling remains to be seen. Further human-computer interaction tests will provide insight into each approach's respective strengths and weaknesses.

Measuring Task Completion Times
As we have come to demonstrate this framework more extensively, the same critiques have arisen again and again, cast either as 'This is a cute gimmick', or alternatively 'What real (if any) research benefit does this lead to?' For this reason, we published recent work aimed at quantitatively evaluating the extent to which our framework accelerated some typical molecular simulation tasks. To do so, we carried out a series of controlled HCI studies, in which participants were tasked with a range of molecular manipulation goals: (1) threading methane through a nanotube; (2) changing screw-sense of a helicene molecule from left to right handed; and (3) tying a protein knot. The results, shown in Fig 2, quantitatively demonstrate that participants within the interactive VR environment can complete molecular modelling tasks more quickly than they can using conventional interfaces like a mouse or a touchscreen, especially for molecular pathways and structural transitions whose conformational choreographies are intrinsically 3-dimensional.

For tasks A and C, Fig 2 indicates that VR provides a clear acceleration benefit compared to the other platforms,
and also that -the more inherently 3D the task, the greater the benefit. The knot-tying task results (Fig 2C) are the most dramatic. A task like knot-tying, which is so intrinsically 3D, is very difficult to accomplish outside of VR. For the nanotube task (Fig 2A), the accomplishment rates, mean time, and median time in VR are approximately a factor of two faster than on other platforms. At first glance, the helicene task ( Fig 2B) is a case in which VR appears to provide little significant rate enhancement compared to other platforms. Observation of the study participants show that this is because changes in helicene screw-sense are most efficiently accomplished using a simple 2D circular motion, as shown in Video 1. Essentially, the 2D limitations of the mouse and touchscreen constrain the participant to carrying out a motion which is well suited to inducing changes in molecular screw-sense, so that VR provides little additional benefit.
Closer inspection of the helicene time distributions shows that VR does afford some advantage: the median time required to change molecular screw-sense in VR is 30-40% less than the median time required on a touchscreen or using a keyboard/mouse.

Reassuringly, we found zero instances where users experienced VR-sickness during the experiments carried out to
gather the data in Fig 2. To date, thousands of people have volunteered to experience it, and very few (less than ten) instances have arisen where participants report any form of sickness -a very small probability. This is an important point, because there is a widespread misconception that a VR experience necessarily entails some form of motionrelated illness. This is not in fact the case. The causes of VR sickness are well understood by workers in human computer interaction and psychology. One of the most common causes of VR sickness arises from inconsistency between the visual information arriving to the brain and the information arriving for processing by the vestibular and proprioceptive system. For example, a sure-fire way to induce VR sickness is by simulating motion within the VR headset whilst a participant is stationary. In such a case, the brain's visual system is presented cues suggesting motion, at odds with the cues to the vestibular and proprioceptive systems, which are not experiencing motion. This perceptual disconnect leads to sickness in significant fractions of people (including several of the authors on this article!). For high-performance scientific applications like those being discussed herein, VR sickness can sometimes arise as a result of computational bottlenecks which cause the system to 'lag'. In such cases, it is often possible to improve system performance through detailed optimizations, or at least to define the operational performance limits of the system which avoid participants experiencing illness. The important point is this: high-end commodity VR enables designers to avoid experiences which lead to illness. In the vast majority of cases, the origins of VR sickness are well-understood, and neither designers nor participants should settle for VR experiences which induce illness.

Measuring Task Reversibility in Complex Systems
If VR is to evolve into a sophisticated tool for carrying out detailed atomic manipulations on systems which are larger and more complicated than those shown in Fig 2, then a critical question is the extent to which complex structural manipulations -e.g., in a biomolecule -are in fact reversible. The level of reversibility is an indicator of the level of control which a nano-technician has over the systems they are investigating. In a first attempt to try and evaluate this, we have been looking at loop motions in the well-studied protein cyclophilin A (CypA), where there is evidence that large-scale collective motions take place. [82][83] Here we highlight some preliminary results which we have obtained during studies of '100s' loop in CypA (formed from residues [100][101][102][103][104][105][106][107][108][109][110], and which undergoes a gating motion shown in

Protein-ligand binding
The preliminary results outlined above suggest that complex biomolecular manipulations using interactive molecular dynamics in VR are in fact reversible. With this knowledge in hand, we have been exploring additional biomolecular application domains where VR might be used to provide insight into biomolecular structure, function, and dynamics.
One specific domain where we have been concentrating our efforts involves the use of iMD-VR to undertake flexible docking of small molecule ligands to protein structures, as illustrated in Fig 5. Broadly speaking, the discovery of molecular binding poses using interactive molecular dynamics amounts to a four-dimensional puzzle in which correct solutions are found by moving, rotating, and fitting a ligand into a protein binding pocket. Whilst there are increasing efforts aimed at using molecular dynamics to examine protein-ligand binding, 84-87 an iMD-VR approach focuses on providing experts with a straightforward and intuitive means for expressing their molecular intuition and design insight to evaluate potential drug designs and corresponding binding hypotheses. Using Narupa, we have been exploring the extent to which human design intuition can be used to guide binding hypotheses, discover potential binding poses, and generate dynamical binding pathways for analyzing binding kinetics and mechanisms. Resolving the kinetic mechanisms of the ligand-protein association process has increasingly been recognized to provide additional insight into safe and differentiated responses of candidate therapeutics. 85 For example, Video 6 (vimeo.com/274862765) shows interactive binding experiments which we undertook to dock the benzamidine ligand to the trypsin protein using our OpenMM interface. Specifically, the figure was generated beginning from the benzamidine-trypsin complex (PDB:1S0R), which we parameterized using GAFF and the Amber14SB forcefield, treating solvent effects using an OBC2 implicit solvent model. With an implicit solvent model, the trypsin protein structure is prone to denaturation, and therefore we applied a restraining potential to the trypsin backbone atoms, in order to maintain the tertiary protein structure. The movie shows benzamidine being interactively guided out of the trypsin binding pocket, and then re-docked. Our preliminary results, established through tests carried out in collaboration with participants at a recent UK CCP-BioSim workshop, indicate that participants, starting from a state where benzamidine was undocked, were then able to identify the trypsin binding pocket and subsequently generated a dynamical pathway which established a bound pose. These preliminary results provide evidence that it is indeed possible to accelerate protein-ligand binding rare events, and also that the Narupa toolset furnishes sufficient control for this class of rare events to be reversible, consistent with the conclusions of section 5.2. Combined, these results suggest that the spatial cognition of a trained biochemist can furnish insight into protein-ligand binding events, in order to quickly explore a wide range of thermodynamic states and kinetic pathways. Using their intuition, participants were able to manipulate the benzamidine in a fashion that allowed the primarily electrostatic binding forces

Molecular Transport in Zeolites
We have also been applying the Narupa iMD-VR framework to understand the transport of small molecules through periodic solid-state materials like zeolites 88 and metal-organic frameworks (MOFs). 89 Compared to protein structures of the sort discussed above, nanoporous materials like these can we constructed from a number of different elements, and are often characterized by a similarly wide range of distinct bonding patterns. Whereas the important interactions governing protein-ligand type interactions tend to occur relatively near the surface of protein structure, the same is not true for small molecule transport in structures like zeolites. Small molecule transport in structures like these tends to occur in channels which are buried in the interior, and which have a complex branched structure, which can lead to transport which involves non-intuitive directionalities. Such structures are particularly important for industrial applications owing to the fact that they are able to accommodate small molecules like hydrocarbons, facilitating both transport 90-91 and catalysis. For example, within the petrochemical industry, these sorts of materials have essential functions as catalysts for processes like hydroxylation, alkylation, and epoxidation, 88 where they operate at much higher temperatures and pressures than typical biocatalysts. Figure 6 : ZMS-5 2ME-hexane structure. The left hand panel shows interaction with a methyl group using the NarupaXR controllers, in order to manipulate the hydrocarbon position. The right hand panel shows the van-der Waals radius representation for the same structure where the hydrocarbon has been partially extracted from the structure.
The fact that such materials typically find application under more extreme conditions means that studying them in iMD-VR requires a force regime which is quite distinct from those which are typically used in our biomolecular studies. It also means that these structures are more robust to the formation of local 'hotspots' of the sort that can sometimes arise in iMD-VR applications. Figure 6 and video 8 (vimeo.com/312957045) shows a ZSM-5 zeolite structure which we have recently begun to study using Narupa, in order to better understand the transport kinetics of 2methyl-hexane. In order to study this particular system, we connected the Narupa API to PLUMED, which enables communication with a wide variety of force engines including DL_POLY, 45  As the video shows, the VR enables one to perform detailed inspection of the zeolite microstructure, interact with substrates in order to navigate them within the channels, and test a range of pathways in order to understand the mechanism and kinetics for adsorption, desorption, and transport. In our preliminary studies on small-molecule transport through zeolite frameworks, we have found that the ability to manipulate and deform the channel has enabled us to better understand how the channel structure and its corresponding flexibility impacts on the hydrocarbon transport dynamics. 92

Reaction discovery using interactive ab initio dynamics
A particularly prevalent problem in the chemical sciences involves mapping complex networks of reactions in order to predict how a particular system (e.g., the gas mixture in a combustion engine, 93 [101][102][103] we have been using Narupa to investigate the extent to which human intuition might be harnessed to accelerate mechanism discovery and understand how human search differs from machine search. Video 9 (vimeo.com/312963823) shows a participant's first-person perspective as they manipulate a real-time simulation using a quantum mechanical force engine to 'discover' chemical reactions in the OH + propyne system. Fig   7 shows preliminary data obtained from a participant group of 21 University of Bristol undergraduate students, each of whom were given five minutes using iMD-VR in Narupa to discover reactions in this way. In our preliminary tests, the students were given a very simple instruction to 'discover' as many different reactions as they could. Forces in these simulations were obtained thorough an interface with the semi-empirical Scine code using the PM6 level of theory. 46 Fig 8 shows a comparison of these preliminary participant-generated results with those obtained from the ChemDyME automated reaction mechanism generator (github.com/RobinShannon/ChemDyME) using the same level of theory. 100 In Fig 6 each node in the network diagram represents a different molecular configuration, with all originating from the green OH + propyne node. Fig 6 shows that people in VR and ChemDyME initially found many of the same reactions, represented by the red nodes. The reactions sampled in VR (blue) and by ChemDyME (orange) then diverge, characterized by two very different search strategies. ChemDyME sampling covers a smaller number of reactions with lots of dead end nodes, whereas human guided VR-sampling identifies many more channels, with significant interconversion between nodes. Preliminary analysis indicates that human guided VR searches were particularly adept at finding association and dissociation processes -e.g., involving high energy association and dissociations of a single species into 2 or more fragments. In this instance, ChemDyME appears better at finding isomerization barriers.
We devised a preliminary 'scoring function' for comparing the performance of the respective search strategies. The scoring function awarded points for finding new pathways. In an attempt to incentivize iMD-VR users to discover lower energy pathways, less points were awarded for higher energy pathways. Fig 7 shows this scoring function applied to the VR and the ChemDyME results as a function of the number of timesteps. Humans in VR were extremely effective at finding a large number of high scoring bimolecular channels. Compared to ChemDyME, our preliminary scoring function implementation appears to incentivize human experts to find more channels overall, but to miss lower energy channels. Moving forward, we plan to investigate the extent to which different scoring mechanisms in conjunction with auditory feedback might influence search strategies. We are particularly interested in understanding the difference in human vs. computer search strategies, and understanding whether human search techniques might be used to devise new kinds of automated search algorithms. 103 Figure 7: Comparison of reactions (edges) and species (nodes) found by humans in VR vs. those from the automated ChemDyME software. The starting node (OH + propyne) is green, those chemical species found in both VR and ChemDyME are in red, those found in VR only are in blue and those found from ChemDyME only are in orange. The lower panel shows the time-dependent score as reactions are discovered, using both VR and ChemDyME

Exploring chemical space using force engines derived from machine learning
In general, quantum mechanical approaches are computationally expensive compared to molecular mechanics, and the size of simulation that can be performed is very limited. Parallelized semi-empirical methods such as those detailed above enable us to explore systems with 100 -150 atoms at interactive latencies. Using machine learning, it is possible to train models which are faster than quantum mechanical methods, and which reproduce quantum mechanical energy surfaces. To enable us to explore even larger systems, we been exploring using Narupa as an iMD-VR strategy for rapidly sampling chemical space and building up data which can then be used to train machine learning algorithms in order to learn potential energy functions.
We have recently shown how exploration of chemical space by human participants using real-time interactive ab initio molecular dynamics in virtual reality can be used to train neural GPU-accelerated neural networks (NN) to learn reactive potential energy surfaces (PESs). 47 Video 10 (vimeo.com/311438872) shows our first application using this strategy, focussed on hydrogen abstraction reactions of CN radical + isopentane using real-time semi-empirical quantum chemistry through a plugin to the SCINE Sparrow package developed by Reiher and co-workers 104-106 (scine.ethz.ch), which includes implementations of tight-binding engines like DFTB alongside a suite of other semi-empirical methods. 46 To obtain the results described herein, we have utilized the SCINE Sparrow implementation of PM6, with the default set of parameters. Using real-time PM6 in VR, we were able to sample a wide range of H-abstraction pathways at the primary, secondary, and tertiary sites on isopentane. Using as an illustrative example abstraction of a primary Hydrogen, Figure 8 compares the PESs predicted by NNs trained using data obtained from iMD-VR versus NNs trained using a more traditional method, namely molecular dynamics (MD) constrained to sample a predefined grid of points along those coordinates which define hydrogen abstraction reactions (shown as D1 and D2 in Fig 8A).  More broadly, we developing an API plugin enabling communication between Narupa and the QML quantum machine learning package initiated by von Lilienfield and co-workers, 107 enabling its use as a force engine for iMD-VR. QML has available a wide range of machine learned models to provide forces and energies trained on high-level quantum data. Kernel-based QML models can be used to describe molecular potential energy surfaces with spectroscopic accuracy, using only a very limited amount of training data. QML models like these are inherently fast, with O(N) scaling if used with appropriate cut-offs.

Sound as a real-time data channel
In a typical real-time interactive MD simulation, a massive quantity of data is available. Making sense of this data often requires reducing its dimensionality by removing irrelevant features to produce an accurate but course-grained representation of the process under investigation. In the molecular sciences, the most familiar form of data visualization comes in the form of molecular structures and animations. However, for large molecules, there is typically a great deal more data than a participant can easily synthesize and understand on-the-fly. Moreover, research in human perception has shown that audio information can have an important impact on visual perception. 108 This has been exploited in previous auditory displays of MD simulation data by integrating sonification frameworks into commonly used Python APIs [109][110] in line with a recent recognition of the importance of interaction in sonification, Ballweg et al outlined a method for interactively exploring a sonification of biomolecular simulations via a keyboard and mouse. 111 In recent work, we have begun to explore sound as a mechanism for engaging the auditory channel to process information. 11 In particular, we have been exploring real-time data 'sonification' as a way to augment structural visual information without splitting attention (e.g., required when one has to simultaneously look at multiple visual displays). Compared to visual displays, sound is vastly underutilized means for data processing in the molecular sciences, in part owing to the fact that the representational mechanisms are less well defined. Depicting an atom as a sphere and a bond as a stick is an arbitrary decision, but intelligible owing to the fact that both an atom and a sphere are spatially delimited.
Attempting to define such a clearly delimited object in the audio realm is not as straightforward, neither spatially nor compositionally. It is difficult to imagine what constitutes an 'atomistic' object in a piece of audio design or music.
Our work to date suggests that sound is best utilized for representing non-local and dynamic properties of the sort which are important in molecular science (e.g. potential energy, free energy, electrostatic energy, local temperature, strain energy, etc.). Owing to their non-locality, properties such as these are extremely difficult to visualize using conventional rendering strategies (and even if there were effective strategies, would lead to significant visual congestion). Sound, however, offers an excellent means for representing these sorts of things, and in many cases, the ear is able to detect dynamical events at a finer temporal resolution that the eye. For example, Video 11 (vimeo.com/312994336) shows a real-time interactive simulation of 17-ALA peptide in which the potential energy is tracked in real-time, in order to interactively generate sound. The video shows how the sound dynamically changes as the participant manipulates the peptide, taking it from its native folded state to a high-energy knotted state which is kinetically trapped. Eventually, the participant unties the peptide knot, and the protein relaxes to a lower energy state, which is again reflected in the sound. As discussed above, rendering the real-time potential energy of a molecule in real-time is a challenge for visual display methods, owing to the fact that the potential energy of a molecule is a non-local descriptor which depends on the entire coordinate vector q; however, sound is particularly well suited to describing such a thing.

Beyond controller-based interaction
Several participants who have experienced Narupa to date have remarked that the controllers act as a barrier in their ability to feel the dynamics of the simulated molecular systems. 13,80 As a result of these comments, we have been pursuing another avenue of research which involves the ability to reach out and 'directly touch' real-time molecular simulations in VR -i.e., without being mediated by a wireless VR controller like that wielded by the participants shown in the Figure 1 schematic. Over the last year, we have been experimenting with a wide range of VR-compatible glove technologies. For example, technologies like the Noitom Hi5 glove and the Manus VR glove, which are equipped with 9 degree of freedom (DOF) inertial movement units (IMUs) and several finger mounted bend sensors, are able to perform real-time calculations of the relative position of the hand. To get positional tracking in a VR system, these gloves can be combined with a wrist-mounted HTC Vive Tracker. Both of these gloves are primarily designed for gestural tracking of hand poses, and distinguishing amongst a variety of potential hand poses (e.g., telling the difference between a 'thumbsup' or a Vulcan 'live long and prosper' gesture), for application in motion capture studios. The experience of seeing one's own hands tracked in VR is indeed thrilling; however, the glove models outlined above are relatively expensive, and their performance is not particularly well suited to the kinds of tasks that a nano-technician might want to carry out in VR. For example, in a real-time VR-enabled simulation, the ability to accurately distinguish between hand poses is far less important than the ability to accurately detect when a nano-technician is reaching out to 'grasp' a particular atom (or selection of atoms) between their thumb and their forefinger. Figure 9: images on the left show the pinch sensing data glove which we designed to enable robust molecular manipulation. The sequence of images on the right show a glove-wearing VR user's first person perspective within Narupa as they tie a knot in a real-time simulation of a 17-ALA peptide.
We have made progress in designing our own data gloves, 48 a prototype of which is shown in Fig 9. By sewing modern conductive fabrics into the glove, we can detect when a participant closes one of two circuits, either by making a pinching motion between (a) their thumb and index finger, or (b) their thumb and forefinger. The absolute position of the hand is obtained from mounting an HTC Vive Tracker on the back of the glove. Our preliminary results, obtained from preliminary user studies carried out in our own laboratory, suggest that participants find the molecular and atomic interaction afforded by this glove extremely intuitive. Compared to the standard HTC Vive handheld controllers (shown in Fig 1), participants have indicated that they prefer the direct sense of 'touching' virtual simulations afforded by this glove. Moreover, for accomplishing a range of molecular manipulation tasks, we have found that the Fig 9 glove design results in more stable iMD-VR experiences than the much more expensive Manus and Noitom gloves. Video 12 (vimeo.com/305823646) shows the perspective of participant who is wearing these gloves as they undertake a protein knot-tying task. To 'touch' an atom and exert a force on it, the participant simply reaches out to the atom they wish to touch, and pinches together their thumb and forefinger. We are currently working to undertake more thorough HCI experiments aimed at evaluating the Fig 9 glove design compared to the standard wireless controllers shown in Fig 1.

Conclusions & Future directions
In this article, we have attempted to provide an overview of the potential which the current generation of immersive technologies (spanning both VR and XR) hold for advancing the molecular sciences. We have introduced our opensource multi-person iMD-VR framework Narupa, and described some of our initial applications across different areas of molecular science, including small molecules, materials, and biochemistry. VR technologies are at an early stage right now, along with our understanding of how to used them, and how to design for them. A number of workers have suggested that we are a stage with VR which is similar to that which characterizes any new media technology -e.g., film, radio, television, and computers. 20,112 For example, the maturation of motion pictures was associated with the development of a film-making lexicon (close ups, cross cuts, flash backs, etc.) along with corresponding design principles and standardized methods, which enabled communication between designers in order to effectively harness the power of the new medium. The same is true for immersive technologies like VR: our understanding of the medium is such that we lack a comprehensive design lexicon for harnessing its power. For example, at this stage most VRenabled scientific visualization frameworks are essentially traditional scientific visualization frameworks which are displayed in a different medium.
Like many domains of scientific computing, the basic workflow for molecular simulation has remained largely unchanged for the last 30 -40 years: i.e., iterative cycles of job submission to HPC resources, followed by visualization on a 2D display. 1 At some point, this paradigm will change, and it may be that immersive technologies like VR, combined with the power of modern HPC and fast networks, drive this change. The extent to which a new technology ends up being adopted within a particular domain is difficult to foresee; nevertheless, we believe that the range of research applications outlined herein provide a glimpse into what might be possible should next-generation immersive interaction technologies like VR find more widespread use within the molecular sciences. Adoption is only likely to arise by demonstrations (e.g., controlled user studies) which show that XR technologies are better than existing technologies in some measurement space, and also by good applications which generalize to other areas of nanoscience.
We are working closely to support a number of international colleagues who are building their own multi-person VR laboratories, and working together to build a development community around Narupa so that we can extend its capabilities in a number of application domains and better understand the ways in which immersive technologies like VR can be productively used within the molecular sciences. For example, we are looking to extend Narupa beyond atomistic modelling to coarse-grained approaches, and we are also looking to establish protocols whereby user generated iMD-VR pathways can then be used as input to automated free energy sampling methods like Markov State Models, 86, 113 path metadynamics, 114 or adaptive BXD 115 . The aim is to enable participants to use their intuition and expertise to quickly generate a dynamical hypothesis from within iMD-VR (e.g., a potential protein-ligand dynamical binding pathway), and then send that data for automated processing to generate a free energy curve using a supercomputing workflow. We are also carrying out work to benchmark the performance of public cloud computing networks, in order to assess their adequacy for supporting real-time research simulation workflows which utilize both molecular mechanics and quantum mechanics force engines.
Immersive technologies like VR represent an interesting research domain precisely because they cannot be disentangled from issues linked to human perception. As a result, their usage and development represent an inherently cross-disciplinary pursuit, which -if it is to be successful -must connect scientists, technologists, interaction designers, artists, and psychologists. With recent advances in high-performance computing, data science, robotics, and machine learning, many have begun to speculate about the future of scientific practice, asking important questions as to the sort of scientific future we should be consciously working to design over the next few decades. 19,51 In an increasingly automated future which is reliant on machines, it is important to think carefully about and discuss the role which human creative expression and human perception will play. Narratives of our emerging technological future sometimes default to a philosophical sentiment which casts automation as the ultimate end, leaving one to wonder how exactly the human fits in. So long as human creativity continues to play an important role in the process of scientific understanding, discovery, and design, then we believe that immersive frameworks like VR may have a crucial role to play in our emerging scientific future. Precisely because VR is a technology which is ultimately designed for the human perceptual system, it represents a technology where the human cannot be automated away. In our view, advanced visualization and interaction frameworks are complementary to research activities aimed at increasing the automation of research tasks and scientific discovery, because they provide an efficient means for humans to undertake communication and collaboration, and express high-level creative scientific and design insight, leaving automated frameworks to subsequently sort out the computational and mechanistic details.