Details of a Researcher - Sugimoto, Maki

Papers - Sugimoto, Maki

Division display >> ／ All the affair displays 1 - 44 of about 44

Effects of the Number of Bodies on Ownership of Multiple Bodies

Kondo R., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 314 - 316 2023.03

Virtually reality can induce body ownership (Sense of being one's own body) of a not innate body or multiple bodies. However, the relationship between the number of bodies and body ownership has not been clarified. In a study that investigated body ownership for one, two, and four virtual bodies, the greater the number of bodies, the closer the body ownership tended to approach the strength of ownership for a single body. Therefore, in the present study, we investigated whether increasing the number of bodies strengthens body ownership for multiple bodies. In the experiment, multiple virtual bodies that moved in synchronization with the participant's movement were lined up in a single file line, and the participant observed the multiple virtual bodies through a head-mounted display from the position of the hindmost virtual body. We measured body ownership and drifts in self-location. Our results showed that contrary to expectations, the greater the number of bodies, the weaker body ownership.

Access to Document (DOI)

Investigating Effects of Facial Self-Similarity Levels on the Impression of Virtual Agents in Serious/Non-Serious Contexts

Niwa M., Masai K., Yoshida S., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 221 - 230 2023.03

Recent technological advances have enabled the use of AI agents to assist with human tasks and augment human cognitive abilities in a variety of contexts, including decision making. It is critical that users trust these AI agents in order to use them effectively. Given that people tend to trust other people who are similar to themselves, incorporating features of one's own face into the AI agent's face may improve one's trust in the AI agent. However, it is still unclear how impressions differ when comparing agents with the same appearance as one's own and some similarities under the same conditions. Recognizing the appropriate level of similarity when using a self-similar agent is important for establishing a trustworthy agent relationship between people and the AI agent. Therefore, we investigated the effect of the degree of self-similarity of the face of the AI agent on the user's trust in the agent. We examined users' impressions of four AI agents with different degrees of face self-similarity in different scenarios. The results showed that the AI agent, whose similarity to the user's facial feature was slightly recognizable but not obvious, received higher ratings on the feeling of closeness, attractiveness, and facial preferences. These self-similar AI agents were also more trustworthy in everyday non-serious decisions and were more likely to improve people's trustworthiness in such situations. Finally, we discuss the potential applications of our findings to design real-world AI agents.

Access to Document (DOI)

Exploring the Effect of Transfer Learning on Facial Expression Recognition using Photo-Reflective Sensors embedded into a Head-Mounted Display

Nakamura F., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 317 - 319 2023.03

As one of the techniques to recognize head-mounted display (HMD) user's facial expressions, the photo-reflective sensor (PRS) has been employed. Since the classification performance of PRS-based method is affected by rewearing an HMD and difference in facial geometry for each user, the user have to perform dataset collection for each wearing of an HMD to build a facial expression classifier. To tackle this issue, we investigate how transfer learning improve within-user and cross-user accuracy and reduce training data in the PRS-based facial expression recognition. We collected a dataset of five facial expressions (Neutral, Smile, Angry, Surprised, Sad) when participants wore the PRS-embedded HMD five times. Using the dataset, we evaluated facial expression classification accuracy using a neural network with/without fine tuning. Our result showed fine tuning improved the within-user and cross-user facial expression classification accuracy compared with non-fine-tuned classifier. Also, applying fine tuning to the classifier trained with the other participant dataset achieved higher classification accuracy than the non-fine-tuned classifier.

Access to Document (DOI)

Sensory Attenuation with a Virtual Robotic Arm Controlled Using Facial Movements

Fukuoka M., Nakamura F., Verhulst A., Inami M., Kitazaki M., Sugimoto M.

IEEE Transactions on Visualization and Computer Graphics (IEEE Transactions on Visualization and Computer Graphics) 2023

ISSN 10772626

When humans generate stimuli voluntarily, they perceive the stimuli more weakly than those produced by others, which is called sensory attenuation (SA). SA has been investigated in various body parts, but it is unclear whether an extended body induces SA. This study investigated the SA of audio stimuli generated by an extended body. SA was assessed using a sound comparison task in a virtual environment. We prepared robotic arms as extended bodies, and the robotic arms were controlled by facial movements. To evaluate the SA of robotic arms, we conducted two experiments. Experiment 1 investigated the SA of the robotic arms under four conditions. The results showed that robotic arms manipulated by voluntary actions attenuated audio stimuli. Experiment 2 investigated the SA of the robotic arm and innate body under five conditions. The results indicated that the innate body and robotic arm induced SA, while there were differences in the sense of agency between the innate body and robotic arm. Analysis of the results indicated three findings regarding the SA of the extended body. First, controlling the robotic arm with voluntary actions in a virtual environment attenuates the audio stimuli. Second, there were differences in the sense of agency related to SA between extended and innate bodies. Third, the SA of the robotic arm was correlated with the sense of body ownership.

Access to Document (DOI)

Investigating the Minimal Condition of the Dynamic Invisible Body Illusion

Kondo R., Sugimoto M.

Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023) 601 - 602 2023

Visual-tactile synchronization or visual-motor synchronization makes a non-innate body feel like one's own body (illusory body ownership). A recent study has shown that body ownership is induced to an empty space between the hands and feet from the motion of only hands and feet (dynamic invisible body illusion). However, it is unclear whether both hands and feet are necessary to induce the illusion. In this study, we investigated the minimal condition for the dynamic invisible body illusion by manipulating the presentation of hands and feet. Our results suggest that both hands and feet are necessary for the illusion.

Access to Document (DOI)

Exploring Enhancements towards Gaze Oriented Parallel Views in Immersive Tasks

Teo T., Sakurada K., Sugimoto M.

Proceedings - 2023 IEEE Conference Virtual Reality and 3D User Interfaces, VR 2023 (Proceedings - 2023 IEEE Conference Virtual Reality and 3D User Interfaces, VR 2023) 620 - 630 2023

Parallel view is a technique that allows a VR user to see multiple locations at a time. It enables the user to control several remote or virtual body parts while seeing parallel views to solve synchronous tasks. However, these techniques only explored the benefits and drawbacks of a user performing different tasks. In this paper, we explored enhancements on a singular or asynchronous task by utilizing information obtained in parallel views. We developed three prototypes where parallel views are fixed, moving in symmetric order, or following the user's eye gaze. We conducted a user study to compare each prototype against traditional VR (without parallel views) in three types of tasks: object search and interaction tasks in a 1) simple environment and 2) complex environment, and 3) object distances estimation task. We found parallel views improved multi-embodiment while each technique helped different tasks. No parallel view provided a clean interface, thus improving spatial presence, mental effort, and user performance. However, participants' feedback highlighted potential usefulness and a lower physical effort by using parallel views to solve complicated tasks.

Access to Document (DOI)

Assessing Individual Decision-Making Skill by Manipulating Predictive and Unpredictive Cues in a Virtual Baseball Batting Environment

Tani Y., Kobayashi A., Masai K., Fukuda T., Sugimoto M., Kimura T.

We propose a virtual reality (VR) baseball batting system for assessing individual decision-making skill based on swing judgement of pitch types and the underlying prediction ability by manipulating combinations of pitching motion and ball trajectory cues. Our analy-sis of data from 10 elite baseball players revealed highly accurate swing motions in conditions during which the batter made precise swing decisions. Delays in swing motion were observed in conditions during which predictive cues were mismatched. Our findings indicated that decision-making based on pitch type influences the inherent stability of decision and accuracy of hitting, and that most batters made decisions based on pitching motion cues rather than on ball trajectory.

Access to Document (DOI)

Effects of virtual hands and feet on the onset time and duration of illusory body ownership

Kondo R., Sugimoto M.

Scientific Reports (Scientific Reports) 12 ( 1 ) 2022.12

In the illusory body ownership, humans feel as if a rubber hand or an avatar in a virtual environment is their own body through visual-tactile synchronization or visual-motor synchronization. Despite the onset time and duration of illusory body ownership has been investigated, it is not clear how the onset time and duration change when a part of the body is missing from the full-body. In this study, we investigated the completeness of the full-body for the illusion onset and duration by comparing the following conditions: complete avatar, avatar missing hands and feet, and avatar with hands and feet only. Our results suggest that avatar hands and feet only shorten the duration of the illusion, and missing body parts, such as only hands and feet or no hands and feet, reduce the sense of body ownership and of agency. However, the effects of avatar completeness on the onset time are unclear, and no conclusions can be made in either direction based on the current findings.

Access to Document (DOI)

Embodiment of supernumerary robotic limbs in virtual reality

Arai K., Saito H., Fukuoka M., Ueda S., Sugimoto M., Kitazaki M., Inami M.

Scientific Reports (Scientific Reports) 12 ( 1 ) 2022.12

The supernumerary robotic limb system expands the motor function of human users by adding extra artificially designed limbs. It is important for us to embody the system as if it is a part of one’s own body and to maintain cognitive transparency in which the cognitive load is suppressed. Embodiment studies have been conducted with an expansion of bodily functions through a “substitution” and “extension”. However, there have been few studies on the “addition” of supernumerary body parts. In this study, we developed a supernumerary robotic limb system that operates in a virtual environment, and then evaluated whether the extra limb can be regarded as a part of one’s own body using a questionnaire and whether the perception of peripersonal space changes with a visuotactile crossmodal congruency task. We found that the participants can embody the extra-limbs after using the supernumerary robotic limb system. We also found a positive correlation between the perceptual change in the crossmodal congruency task and the subjective feeling that the number of one’s arms had increased (supernumerary limb sensation). These results suggest that the addition of an extra body part may cause the participants to feel that they had acquired a new body part that differs from their original body part through a functional expansion.

Access to Document (DOI)

Split body: Extending self-location by splitting a body left and right

Kondo R., Sugimoto M.

Frontiers in Virtual Reality (Frontiers in Virtual Reality) 3 2022.10

Is it possible to feel part of one’s own body even when the body part is separated from the body? If so, we could exist in different locations by splitting the self-body and using our body in each location. In a study in which the illusion that two bodies are one’s own body was induced using virtual reality (VR), the participants felt as if they were at two locations through the two bodies. However, this illusion was weak and reported only subjectively. We hypothesized that this was because two presented bodies moved synchronously with one participant’s movement or the simultaneous stroking of one participant’s body and two presented bodies switched their attention and weakened body ownership. In this study, we investigated whether splitting one body into left and right in VR could induce body ownership and extend the self-location while maintaining a one-to-one correspondence between the participant’s body and the presented body. The results showed that weaker body ownership was induced in the split body than in the normal body and self-location was extended to the right side of the body. The participants did not report a sense of having more than one body but reported a sense of body spreading, suggesting that the split bodies were perceived as a single body extending to the right side.

Access to Document (DOI)

Human Latent Metrics: Perceptual and Cognitive Response Correlates to Distance in GAN Latent Space for Facial Images

Shimizu K., Ienaga N., Takada K., Sugimoto M., Kasahara S.

Proceedings - SAP 2022: ACM Symposium on Applied Perception (Proceedings - SAP 2022: ACM Symposium on Applied Perception) 2022.09

Generative adversarial networks (GANs) generate high-dimensional vector spaces (latent spaces) that can interchangeably represent vectors as images. Advancements have extended their ability to computationally generate images indistinguishable from real images such as faces, and more importantly, to manipulate images using their inherit vector values in the latent space. This interchangeability of latent vectors has the potential to calculate not only the distance in the latent space, but also the human perceptual and cognitive distance toward images, that is, how humans perceive and recognize images. However, it is still unclear how the distance in the latent space correlates with human perception and cognition. Our studies investigated the relationship between latent vectors and human perception or cognition through psycho-visual experiments that manipulates the latent vectors of face images. In the perception study, a change perception task was used to examine whether participants could perceive visual changes in face images before and after moving an arbitrary distance in the latent space. In the cognition study, a face recognition task was utilized to examine whether participants could recognize a face as the same, even after moving an arbitrary distance in the latent space. Our experiments show that the distance between face images in the latent space correlates with human perception and cognition for visual changes in face imagery, which can be modeled with a logistic function. By utilizing our methodology, it will be possible to interchangeably convert between the distance in the latent space and the metric of human perception and cognition, potentially leading to image processing that better reflects human perception and cognition.

Access to Document (DOI)

MultiSoma: Motor and Gaze Analysis on Distributed Embodiment With Synchronized Behavior and Perception

Miura R., Kasahara S., Kitazaki M., Verhulst A., Inami M., Sugimoto M.

Frontiers in Computer Science (Frontiers in Computer Science) 4 2022.05

Human behavior and perception are optimized for a single body. Yet, the human brain has plasticity, which allows us to extend our body schema. By utilizing technology like robotics or virtual reality (VR), we can modify our body parts or even add a new body to our own while retaining control over these parts. However, the update of body cognition when controlling multiple bodies has not been well examined. In this study, we explore the task performance and body cognition of humans when they have multiple full bodies as an extended embodiment. Our experimental system allows a participant to control up to four bodies at the same time and perceive sensory information from them. The participant experiences synchronizing behavior and vision perception in a virtual environment. We set up three tasks for multiple bodies and evaluated the cognition of these bodies with their gazing information, task performances, and subjective ratings. We found that humans can have the sense of body ownership and agency for each body when controlling multiple bodies simultaneously. Furthermore, it was observed that people manipulate multiple bodies by actively switching their attention in a static environment and passively switching their attention in a dynamic environment. Distributed embodiment has the potential to extend human behavior in cooperative work, parallel work, group behavior, and so on.

Access to Document (DOI)

Evaluation of Spatial Directional Guidance Using Cheek Haptic Stimulation in a Virtual Environment

Nakamura F., Verhulst A., Sakurada K., Fukuoka M., Sugimoto M.

Frontiers in Computer Science (Frontiers in Computer Science) 4 2022.05

Spatial cues play an important role in navigating people in both physical and virtual spaces. In spatial navigation, visual information with additional cues, such as haptic cues, enables effective guidance. Most haptic devices are applied to various body parts to make mechanical stimuli, while few devices stimulate a head despite the excellent sensitivity. This article presents Virtual Whiskers, a spatial directional guidance technique by cheek haptic stimulation using tiny robot arms attached to a Head-Mounted Display (HMD). The tip of the robotic arm has photo reflective sensors to detect the distance between the tip and the cheek surface. Using the robot arms, we stimulate a point on the cheek obtained by calculating an intersection between the cheek surface and the target direction. In the directional guidance experiment, we investigated how accurately participants identify the target direction provided by our guidance method. We evaluated an error between the actual target direction and the participant's pointed direction. The experimental result shows that our method achieves the average absolute directional error of 2.54° in the azimuthal plane and 6.54° in the elevation plane. We also conducted a spatial guidance experiment to evaluate task performance in a target search task. We compared the condition of visual information, visual and audio information, and visual information and cheek haptics for task completion time, System Usability Scale (SUS) score, NASA-TLX score. The averages of task completion time were M = 6.39 s, SD = 3.34 s, and M = 5.62 s, SD = 3.12 s, and M = 4.35 s, SD = 2.26 s, in visual-only condition, visual+audio condition, and visual+haptic condition, respectively. In terms of the SUS score, visual condition, visual+audio condition, and visual+haptic condition achieved M = 55.83, SD = 20.40, and M = 47.78, SD = 20.09, and M = 80.42, SD = 10.99, respectively. As for NASA-TLX score, visual condition, visual+audio condition, and visual+haptic condition resulted in M = 75.81, SD = 16.89, and M = 67.57, SD = 14.96, and M = 38.83, SD = 18.52, respectively. Statistical tests revealed significant differences in task completion time, SUS score, and NASA-TLX score between the visual and the visual+haptic condition and the visual+audio and the visual+haptic condition.

Access to Document (DOI)

Analysis and Observation of Behavioral Factors Contributing to Improvement of Embodiment to a Supernumerary Limb

Nishida N., Iwasaki Y., Teo T., Fukuoka M., Sugimoto M., Chen P.H., Kato F., Kitazaki M., Iwata H.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 116 - 120 2022.03

Research has been conducted on attaching a robotic arm to a human body and using it as a supernumerary limb. However, it remains unclear what type of behaviors contribute to the induction of embodiment in a supernumerary limb. In this study, we created a virtual environment where we could observe behaviors that contribute to the embodiment of avatars and tools, such as "looking at one's own body in a mirror,""moving the supernumerary limb quickly and accurately,"and "paying constant attention to the supernumerary limb,"and verified whether these behaviors are also effective in inducing the embodiment of the supernumerary limb. After spending 40 min in a behavior observation space, the left hand, the side with the supernumerary limb, was used significantly less frequently. In addition to these behaviors, "self-touching,""transferring objects between their own hand and the supernumerary limb,"and "whole-body movement"were also observed. The observations suggest that these behaviors may change the awareness of the supernumerary limb and contribute to the improvement of its embodiment.

Access to Document (DOI)

The Reference Frame of Robotic Limbs Contributes to the Sense of Embodiment and Motor Control Process

Sakurada K., Kondo R., Nakamura F., Fukuoka M., Kitazaki M., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 104 - 115 2022.03

The robotic limbs using a body remapping approach often follow either the user's reference frame 1) fixed outside the user's body (Space frame), 2) centered on a user's torso (Torso frame), or 3) centered on a user's head (Head frame). In this study, we investigate the effect of the reference frame on the sense of embodiment and the user's motor control process. We asked 12 participants to perform some point-to-point tasks with the virtual robotic limbs remapped to the participants' feet. The virtual robotic limbs tip followed the participants' feet and each reference frame condition (Space frame, Torso frame, and Head frame). As a result, the reaction time, the movement straightness, and the movement priority were significantly high rate in the Torso frame condition. The subjective score on the sense of embodiment showed that the reference frame condition contributed to the sense of embodiment.

Access to Document (DOI)

Synchronous and Asynchronous Manipulation Switching of Multiple Robotic Embodiment Using EMG and Eye Gaze

Nakanishi Y., Fukuoka M., Kasahara S., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 94 - 103 2022.03

Through the use of multiple avatars and robots, the construction of an alter ego whose senses and movements are synchronized with those of the operator has been explored. There are two possible states of operation when manipulating those bodies as synchronous operation: multiple bodies are synchronized simultaneously, and asynchronous operation: a specific body is selectively operated. We propose a system that allows intuitive switching between the active robot and the operation state by Eye gaze and EMG. A two-stage pick-and-place task was performed with the proposed system, and task performance and subjective evaluation of embodiment were analyzed. To consider the coordinates of the tracker in the asynchronous operation of the robot arm, we performed a similar study for a coordinate condition: the absolute coordinate condition, where the coordinates of the tracker are stored in space, and the relative coordinate condition, where only the motion of the tracker is added.

Access to Document (DOI)

Effects of Body Duplication and Split on Body Schema

Kondo R., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 320 - 322 2022.03

Visual-tactile synchronization or visual-motor synchronization makes a not innate body feel as if it was one's own body (illusory body ownership). Although recent studies have shown that body ownership can be induced in multiple bodies, it is unclear whether the body ownership is switched between bodies, or we feel that multiple bodies belong to us simultaneously. If the body ownership is induced in multiple bodies simultaneously, we expected the body schema to be extended to correspond to the number of bodies. In this study, we investigated whether body ownership is induced in multiple bodies simultaneously by measuring the body schema when the bodies are replicated and split. Experiment 1 investigated whether two side-by-side virtual bodies moved synchronously with the participant's movements to extend the body schema. In Experiment 2, virtual split bodies in which a virtual body was split into left and right sides were presented. As a result, the body schema was extended only in the split bodies. However, it changed even in the no body ownership condition, suggesting that the appearance of the split bodies simply affected the body schema, not that body ownership of the split bodies occurred at the same time. The results of the questionnaire indicated that the split bodies might be perceived as a single body.

Access to Document (DOI)

Smart Surgical Light: Identification of Surgical Field States Using Time of Flight Sensors

Itabashi Y., Nakamura F., Kajita H., Saito H., Sugimoto M.

Journal of Medical Robotics Research (Journal of Medical Robotics Research) 7 ( 1 ) 2022.03

ISSN 2424905X

This work presents a method for identifying surgical field states using time-of-flight (ToF) sensors equipped with a surgical light. It is important to understand the surgical field state in a smart surgical room. In this study, we aimed to identify surgical field states by using 28 ToF sensors with a surgical light installed on each. In the experimental condition, we obtained a sensor dataset by changing the number of people, posture, and movement state of a person under the surgical light. The identification accuracy of the proposed system was evaluated by applying machine learning techniques. This system can be realized simply by attaching ToF sensors to the surface of an existing surgical light.

Access to Document (DOI)

Spatiotemporal Video Highlight by Neural Network Considering Gaze and Hands of Surgeon in Egocentric Surgical Videos

Yoshida K., Hachiuma R., Tomita H., Pan J., Kitani K., Kajita H., Hayashida T., Sugimoto M.

Journal of Medical Robotics Research (Journal of Medical Robotics Research) 7 ( 1 ) 2022.03

ISSN 2424905X

In the medical field, surgical videos can be used to introduce surgical skills. Medical students and residents watch the videos to study the surgical skills and increase learning speed by compensating for the lack of experience in surgical rooms due to limited opportunity to join in surgery. To record egocentric surgical videos by a wearable camera is a solution to record surgical skills of a surgeon in detail. However, most egocentric surgical videos are of quite long duration. For example, in the case of tumor removal in breast surgery, a video recording time often reaches 2h. With that length, it is time consuming to see important scenes in the video, particularly because many surgical videos include nonessential scenes such as sterilization and preparation of tools. For extracting specific scenes from a long video, we can apply scene estimation by machine learning. Furthermore, it is important to know where the surgeon is looking to observe the area of the incision in detail. In particular, it is vital to be able to zoom in on key elements, allowing viewers to see the incision area and the fine details of the necessary surgical skills. In this study, we aimed to highlight incision scenes from egocentric surgical videos in the spatiotemporal domain by utilizing two neural networks for the temporal and spatial highlights. For the temporal highlights, we designed a neural network that estimates the incision scenes by learning gaze speed, hand movements, number of hands, and background movements in egocentric surgical videos. For the spatial highlights, in order to estimate the important area to zoom in, we designed a neural network that learns the surgeon's gaze on natural features of surgical scenes to form a probability map as a representation of the estimated gaze area. The estimated gaze area was also used to calculate the appropriate zoom-in position and zoom-in ratio. To control the highlighted parameters in accord with user preferences, we also made a user interface that allows for the selection of playback speed gain and zoom ratio gain. For the evaluation, we verified the performance of the networks by a quantitative assessment and conducted a user study with medical doctors by showing an actual surgical video to obtain a qualitative assessment on the proposed system.

Access to Document (DOI)

Analyzing the Effect of Diverse Gaze and Head Direction on Facial Expression Recognition with Photo-Reflective Sensors Embedded in a Head-Mounted Display

Nakamura F., Murakami M., Suzuki K., Fukuoka M., Masai K., Sugimoto M.

IEEE Transactions on Visualization and Computer Graphics (IEEE Transactions on Visualization and Computer Graphics) 2022

ISSN 10772626

As one of the facial expression recognition techniques for Head-Mounted Display (HMD) users, embedded photo-reflective sensors have been used. In this paper, we investigate how gaze and face directions affect facial expression recognition using the embedded photo-reflective sensors. First, we collected a dataset of five facial expressions (Neutral, Happy, Angry, Sad, Surprised) while looking in diverse directions by moving 1) the eyes and 2) the head. Using the dataset, we analyzed the effect of gaze and face directions by constructing facial expression classifiers in five ways and evaluating the classification accuracy of each classifier. The results revealed that the single classifier that learned the data for all gaze points achieved the highest classification performance. Then, we investigated which facial part was affected by the gaze and face direction. The results showed that the gaze directions affected the upper facial parts, while the face directions affected the lower facial parts. In addition, by removing the bias of facial expression reproducibility, we investigated the pure effect of gaze and face directions in three conditions. The results showed that, in terms of gaze direction, building classifiers for each direction significantly improved the classification accuracy. However, in terms of face directions, there were slight differences between the classifier conditions. Our experimental results implied that multiple classifiers corresponding to multiple gaze and face directions improved facial expression recognition accuracy, but collecting the data of the vertical movement of gaze and face is a practical solution to improving facial expression recognition accuracy.

Access to Document (DOI)

Virtual Reality Sonification Training System Can Improve a Novice's Forehand Return of Serve in Tennis

Masai K., Kajiyama T., Muramatsu T., Sugimoto M., Kimura T.

Proceedings - 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2022 (Proceedings - 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct, ISMAR-Adjunct 2022) 845 - 849 2022

Virtual reality (VR) is gaining interest as a platform for sports skills training. VR allows for information manipulation and feedback that would be difficult in reality. This is particularly useful in open skill sports where players must adjust their behavior in response to environmental factors. Auditory feedback (sonification) is constructive for sports training in VR. However, this has not been well studied in open skill-specific situations due to the difficulty of accounting for environmental factors in reality. This study focuses on a serve return, an important scene in tennis. It investigates the effects of sonification on the forehand return stroke in VR by comparing score displays and trajectory visualizations. We designed the sonification based on the difference between experienced and novice players' movements in VR. We then conducted a between-subjects experiment to investigate the effect of the sonification (N=20). The results showed that the system with sonification effectively improved the timing of hip movement for preparing a slow serve return compared to the system without sonification.

Access to Document (DOI)

Techniques for using VRChat to Replace On-site Experiments

Teo T., Sakurada K., Fukuoka M., Sugimoto M.

In this paper, we present a technique to conduct on-site laboratory experiments using VRChat, a social Virtual Reality platform. COVID-19 pandemic has created challenges and restrictions on re-searchers in running experiments for Human-Computer Interaction research. Hence, universities and institutions set guidelines for social distancing and encourage people to work from home. Therefore, participants are harder to recruit for experiments that run in a physical laboratory. Our technique uses VRChat as a substitution for running on-site experiments despite social restrictions. We exploit features in VRChat to enable researchers to run experiments with domestically recruited participants. Furthermore, we propose a hassle-free method to retrieve study data. This paper also discusses and offers design considerations to run an on-site experiment using VRChat over a homemade system.

Access to Document (DOI)

Embodiment of an Avatar with Unnatural Arm Movements

Hapuarachchi H., Ishimoto H., Sugimoto M., Inami M., Kitazaki M.

We developed an 'unnatural avatar' of which the lower arms could be bent in reversed/unnatural directions. Participants controlled the avatar in a virtual environment to perform reaching tasks. We measured their senses of body ownership and agency. Compared to a normal avatar, the embodiment towards the unnatural avatar was significantly lower. However, senses of body ownership and agency towards both avatars were higher when the avatar's movements were synchronized with the movements of the participants compared to when there was a delay of one second. These results suggest that visuo-motor synchrony induces embodiment even towards avatars with reversed/unnatural limb movements.

Access to Document (DOI)

Consistent Smile Intensity Estimation from Wearable Optical Sensors

Masai K., Perusquia-Hernandez M., Sugimoto M., Kumano S., Kimura T.

2022 10th International Conference on Affective Computing and Intelligent Interaction, ACII 2022 (2022 10th International Conference on Affective Computing and Intelligent Interaction, ACII 2022) 2022

Smiling plays a crucial role in human communication. It is the most frequent expression shown in daily life. Smile analysis usually employs computer vision-based methods that use data sets annotated by experts. However, cameras have space constraints in most realistic scenarios due to occlusions. Wearable electromyography is a promising alternative; however, issue of user comfort is a barrier to long-term use. Other wearable-based methods can detect smiles, but they lack consistency because they use subjective criteria without expert annotation. We investigate a wearable-based method that uses optical sensors for consistent smile intensity estimation while reducing manual annotation cost. First, we use a state-of-art computer vision method (OpenFace) to train a regression model to estimate smile intensity from sensor data. Then, we compare the estimation result to that of OpenFace. We also compared their results to human annotation. The results show that the wearable method has a higher matching coefficient (r=0.67) with human annotated smile intensity than OpenFace (r=0.56). Also, when the sensor data and OpenFace output were fused, the multimodal method produced estimates closer to human annotation (r=0.74). Finally, we investigate how the synchrony of smile dynamics among subjects and their average smile intensity are correlated to assess the potential of wearable smile intensity estimation.

Access to Document (DOI)

AnkleSens: Foot Posture Prediction Using Photo Reflective Sensors on Ankle

Kikui K., Masai K., Sasaki T., Inami M., Sugimoto M.

IEEE Access (IEEE Access) 10 33111 - 33122 2022

Recognizing foot gestures can be useful for subtle inputs to appliances and machines in everyday life, but for a system to be useful, it must allow users to assume various postures and work in different spaces. Camera-based and pressure-based systems have limitations in these areas. In this paper, we introduce AnkleSens, a novel ankle-worn foot sensing device that estimates a variety of foot postures using photo reflective sensors. Since our device is not placed between the foot and the floor, it can predict foot posture, even if we keep the foot floating in the air. We developed a band prototype with 16 sensors that can be wrapped around the leg above the ankle. To evaluate the performance of the proposed method, we used eight foot postures and four foot states as preliminary classes. After assessing a test dataset with the preliminary classes, we integrated the eight foot postures into five. Finally, we classified the dataset with five postures in four foot states. For the resulting 20 classes, the average classification accuracy with our proposed method was 79.57% with user-dependent training. This study showed the potential of foot posture sensing as a new subtle input method in daily life.

Access to Document (DOI)

Virtual Whiskers: Cheek Haptic-Based Spatial Directional Guidance in a Virtual Space

Nakamura F., Verhulst A., Sakurada K., Fukuoka M., Sugimoto M.

Proceedings - SIGGRAPH Asia 2021 XR, SA 2021 (Proceedings - SIGGRAPH Asia 2021 XR, SA 2021) 2021.12

In spatial navigation, adding haptic cues to visual information lets users understand the spatial information better. Most haptic devices stimulate various body parts, while few devices target our heads that are sensitive to mechanical stimuli. This paper presents Virtual Whiskers, a spatial directional guidance technique using cheek haptics in a virtual space. We created a cheek haptic stimulation device by attaching two tiny robot arms to a Head-Mounted Display. The robot arms trace the cheek with proximity sensors to estimate the cheek surface. Target azimuthal and elevational directions are translated into a point on the cheek surface. The robot arms touch the point to present target directional cues. We demonstrate our technique in two applications.

Access to Document (DOI)

X-Wing: Propeller-Based Force Feedback to Head in a Virtual Environment

Watanabe K., Nakamura F., Sakurada K., Teo T., Sugimoto M.

Proceedings - SIGGRAPH Asia 2021 XR, SA 2021 (Proceedings - SIGGRAPH Asia 2021 XR, SA 2021) 2021.12

In this demonstration, we present X-Wing, a force feedback device using ducted fans attached to a Virtual Reality (VR) Head Mounted Display (HMD). Applying haptic technologies to VR has become a standard solution to enhance the immersive experience. It is also essential for virtual interactions where user experience and performance are crucial. This research focuses on a head-based, wearable device that provides forces to the VR user using exertion force by electric thrusters with controllable strengths and directions. Our system allows the user to experience forces based on their virtual momentum and velocity. It enables a VR user to receive force feedback based on different thrust power converted into translational and rotational force for a unique VR experience such as virtual aerial simulation. We present the prototype to the attendees of SIGGRAPH Asia 2021 through a live demonstration.

Access to Document (DOI)

Peripersonal space in the front, rear, left and right directions for audio-tactile multisensory integration

Matsuda Y., Sugimoto M., Inami M., Kitazaki M.

Scientific Reports (Scientific Reports) 11 ( 1 ) 2021.12

Peripersonal space (PPS) is important for humans to perform body–environment interactions. However, many previous studies only focused on the specific direction of the PPS, such as the front space, despite suggesting that there were PPSs in all directions. We aimed to measure and compare the peri-trunk PPS in four directions (front, rear, left, and right). To measure the PPS, we used a tactile and an audio stimulus because auditory information is available at any time in all directions. We used the approaching and receding task-irrelevant sounds in the experiment. Observers were asked to respond as quickly as possible when a tactile stimulus was applied to a vibrator on their chest. We found that peri-trunk PPS representations exist with an approaching sound, irrespective of the direction.

Access to Document (DOI)

The effects of body direction and posture on taking the perspective of a humanoid avatar in a virtual environment

Ueda S., Nagamachi K., Nakamura J., Sugimoto M., Inami M., Kitazaki M.

PLoS ONE (PLoS ONE) 16 ( 12 December ) 2021.12

Visual perspective taking is inferring how the world looks to another person. To clarify this process, we investigated whether employing a humanoid avatar as the viewpoint would facilitate an imagined perspective shift in a virtual environment, and which factor of the avatar is effective for the facilitation effect. We used a task that involved reporting how an object looks by a simple direction judgment, either from the avatar's position or from the position of an empty chair. We found that the humanoid avatar's presence improved task performance. Furthermore, the avatar's facilitation effect was observed only when the avatar was facing the visual stimulus to be judged; performance was worse when it faced backwards than when there was only an empty chair facing forwards. This suggests that the avatar does not simply attract spatial attention, but the posture of the avatar is crucial for the facilitation effect. In addition, when the directions of the head and the torso were opposite (i.e., an impossible posture), the avatar's facilitation effect disappeared. Thus, visual perspective taking might not be facilitated by the avatar when its posture is biomechanically impossible because we cannot embody it. Finally, even when the avatar's head of the possible posture was covered with a bucket, the facilitation effect was found with the forward-facing avatar rather than the backward-facing avatar. That is, the head/gaze direction cue, or presumably the belief that the visual stimulus to be judged can be seen by the avatar, was not required. These results suggest that explicit perspective taking is facilitated by embodiment towards humanoid avatars.

Access to Document (DOI)

Communications in virtual environment improve interpersonal impression

Kato Y., Sugimoto M., Inami M., Kitazaki M.

Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021 (Proceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021) 613 - 614 2021.03

Pseudo physical touch is used for communications in virtual environments such as VRChat. We aimed to test if the pseudo-touch communication affects social impression in a virtual environment. Nineteen participants performed the controlled experiment with a partner who was an experimenter with three types of communications: no touch, pseudo touch, and actual touch. Subjective ratings of attractiveness and the communication easiness with the partner increased in all conditions, suggesting that the communication in virtual environments improves interpersonal attraction and communicability either with or without physical or pseudo touch.

Access to Document (DOI)

Dynamic Shared Limbs: An Adaptive Shared Body Control Method Using EMG Sensors

Takizawa R., Hagiwara T., Verhulst A., Fukuoka M., Kitazaki M., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 10 - 18 2021.02

An inter-personal shared body is an extended body that allows an avatar or robot to be controlled by multiple people. It is possible for operators to share control over the actions of the shared body. A shared body can be operated well when the same actions are performed by its operators. However, when their actions are not the same, we need to have an adaptive control method for the operators to maintain a sense of agency and ownership. In this study, we propose dynamic adjustment methods to control a shared body using EMG sensors and movements of the operators.

Access to Document (DOI)

Virtual Whiskers: Spatial Directional Guidance using Cheek Haptic Stimulation in a Virtual Environment

Nakamura F., Verhulst A., Sakurada K., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 141 - 151 2021.02

Spatial cues are an important element of navigating people in physical/virtual spaces. In terms of spatial navigation, integrating vision with other modalities, such as haptics, can guide users more effectively. Haptic cues are presented on the body parts that are sensitive to stimuli such as hands and a head. The head is reported to be superior to the body for spatial directional perception. In this paper, we propose Virtual Whiskers, a spatial directional guidance technique by haptic stimulation of the cheeks using tiny robot arms attached to a Head-Mounted Display (HMD). We deploy photo reflective sensors attached to the tip of 2 robotic arms to detect the distance between the tip and the cheek surface. Using the robot arms, we stimulate a point on the cheek obtained by calculating an intersection between the cheek surface and the target direction. We experimentally investigated how accurately participants identify the target direction provided by our guidance method. We evaluated an error between the actual target direction and the participant's pointed direction. The experimental result shows that our method achieves the average absolute directional error of 2.76 degrees in the azimuthal plane and 7.32 degrees in the elevation plane. We also conducted a spatial guidance experiment to evaluate task performance in a target search task. We compared the condition of only vision and vision with haptics for task completion time. The average of task completion time in visual-only condition was M=12.45 s, SD=14.51 s, and visual with haptic condition resulted in M=6.91 s, SD=5.48 s. Statistical test revealed a significant difference in task completion time between the visual condition and the visual+haptic condition.

Access to Document (DOI)

MultiSoma: Distributed Embodiment with Synchronized Behavior and Perception

Miura R., Kasahara S., Kitazaki M., Verhulst A., Inami M., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 1 - 9 2021.02

Human behavior and perception are optimized for a single body. Yet, the human brain has plasticity, which allows us to extend our body schema. By utilizing technology like robotics or virtual reality (VR), we can modify our body parts or even add a new body to our own while retaining control over these parts. However, the update of body cognition when controlling multiple bodies has not been well examined. In this study, we explore the task performance and body cognition of humans when they have multiple full bodies as an extended embodiment. Our experimental system allows a participant to control up to four bodies at the same time and perceive sensory information from them. The participant experiences synchronizing behavior and vision perception in a virtual environment. We set up two tasks for multiple bodies and evaluated the cognition of these bodies by their task performances and subjective ratings. We found that humans can have the sense of body ownership and agency for each body when controlling multiple bodies simultaneously. Distributed embodiment has the potential to extend human behavior in cooperative work, parallel work, group behavior, and so on.

Access to Document (DOI)

Exploring a Dynamic Change of Muscle Perception in VR, Based on Muscle Electrical Activity and/or Joint Angle

Ferrand E., Verhulst A., Inami M., Sugimoto M.

ACM International Conference Proceeding Series (ACM International Conference Proceeding Series) 298 - 300 2021.02

We display a virtual avatar changing its biceps appearance to fit the user's biceps contraction. The avatar's biceps changes its size and color based on either: (EMG) the biceps activity (recorded with a electromyography [EMG] sensor); or (ANG) the elbow angle; or (EMG + ANG) the biceps activity and elbow angle. The users also wear an ad hoc haptic device around their biceps to give the illusion of stronger biceps contractions. In the Virtual Environment, the participants (N=10) look at their right biceps and do elbow flexion. They rate EMG, ANG and EMG + ANG (with and without the haptic device) along Body Ownership and Agency. We found that ANG (without the haptic device) was rated noticeably lower on the Agency than EMG and EMG + ANG (both without the haptic device). It is likely better to change the appearance of a muscle using the user's muscle contraction rather than the user's movement when considering the user's muscle ownership and agency. The addition of the ad hoc haptic device appeared to not affect the Sense of Embodiment.

Access to Document (DOI)

Investigating Textual Visual Sound Effects in a Virtual Environment and their impacts on Object Perception and Sound Perception

Fabre T., Verhulst A., Balandra A., Sugimoto M., Inami M.

Proceedings - 2021 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021 (Proceedings - 2021 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2021) 320 - 328 2021

In comics, Textual Sound Effects (TE) can describe sounds, but also actions, events, etc. TE could be used in Virtual Environment to efficiently create an easily recognizable scene and add more information to objects at a relatively low design cost. We investigate the impact of TE in a Virtual Environment on objects’ material perception (on category and properties) and on sound perception (on volume [dB] and spatial position). Participants (N=13, repeated measures) categorized metallic and wooden spheres and significantly changed their reaction time depending on the TE congruence with the spheres’ material/sound. They then rated a sphere’s properties (i.e., wetness, warmness, softness, smoothness, and dullness) and significantly changed their rating depending on the TE. When comparing 2 sound volumes, they perceived a sound associated with a shrinking TE as less loud and a sound associated with a growing TE as louder. When locating an audio source location, they located it significantly closer to a TE.

Access to Document (DOI)

Facial Expression Recognition in Daily Life by Embedded Photo Reflective Sensors on Smart Eyewear

SUGIMOTO Maki

Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI2016) 317 - 326 2016

FuwaFuwa: 複数のフォトリフレクタモジュールを用いた柔軟物への接触検知手法

SUGIMOTO Maki

日本バーチャルリアリティ学会論文誌 20 ( 3 ) 209 - 217 2015.09

Joint Work

環境3Dモデルによるカメラトラッキングを用いた温度分布のAR表示システム

中川　航，松本一紀，フランソワドゥソルビエ，杉本麻樹，斎藤英雄，仙田修司，柴田剛志，池谷彰彦

映像情報メディア学会誌 69 ( 4 ) J160 - J168 2015.04

Research paper (scientific journal), Joint Work, Accepted

Virtual Slicer: Visualizer for Tomographic Medical Images Corresponding Handheld Device to Patient

SUGIMOTO Maki

The International Journal of Virtual Reality 15 ( 1 ) 10 - 17 2015

Joint Work

Move-it: a paperclip-shaped sensing and actuating system for sticky notes

Kentaro Yasu, Kathrin Probst, Maki Sugimoto, Michael Haller and Masahiko Inami

ROBOMECH Journal 2014 Vol. 1:12 2014.09

Research paper (scientific journal), Joint Work, Accepted

プカプカメラ：浮遊カメラを用いた三人称視点による旅の記録体験の拡張

山本翼，杉浦裕太，南澤孝太，杉本麻樹，稲見昌彦

日本バーチャルリアリティ学会論文誌 8 ( 3 ) 271 - 281 2013

Joint Work

RGB-Dカメラから得られる部分物体形状と影に基づく光源推定

池田拓也，小山田雄仁，杉本麻樹，斎藤英雄

映像情報メディア学会誌 67 ( 4 ) J124 - 133 2013

Joint Work

Feature-based Alignment Method for Projecting Virtual Content on a Movable Paper Map

Sandy Martedi, Maki Sugimoto, Hideo Saito, Bruce Thomas

IEEJ Trans. on Electronics, Information, and Systems 133 ( 3 ) 672 - 679 2013

Joint Work

Automatic Sub-pixel Projector Calibration; Supporting Improved Calibration for Projected Environments

Ross T. Smith, Guy Webber, Maki Sugimoto, Michael Marner and Bruce H. Thomas

ITE Transactions on Media Technology and Applications 1 ( 3 ) 204 - 213 2013

Joint Work

　 Previous page - Next page 　