論文 - 藤代　一成

L2H-NeRF: Low to high frequency guided NeRF for 3D reconstruction with a few input scenes

T. Bao, J. Ye, Z. Bao, C.S. Leow, H. Hu, J. Lu, I. Fujishiro, J. Xu

The Visual Computer （Springer-Verlag）  41 （ 9 ） 6757 - 6768 2025年07月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  0178-2789

　概要を見る

Nowadays, three-dimensional (3D) reconstruction techniques are becoming increasingly important in the fields of architecture, game development, movie production, and more. Due to common issues in the reconstruction process, such as perspective distortion and occlusion, traditional 3D reconstruction methods face significant challenges in achieving high-precision results, even when dense data are used as inputs. With the advent of neural radiance field (NeRF) technology, high-fidelity 3D reconstruction results are now possible. However, high computational resources are usually required for NeRF computations. Recently, a few data inputs are used to ensure the highest quality. In this paper, we propose an innovative low- to highfrequency-guided NeRF (L2H-NeRF) framework that decomposes scene reconstruction into coarse and fine stages. For the first stage, a low-frequency enhancement network based on a vision transformer is proposed, where the low-frequency-based globally coherent geometric structure is recovered, with the dense depth restored in a depth completion way. In the second stage, a high-frequency enhancement network is incorporated, where the high-frequency-related detail is compensated by robust feature alignment across adjacent views using a plug-and-play feature extraction and matching module. Experiments demonstrate that both the accuracy of the geometric structure and the feature detail of the proposed L2H-NeRF outperforms state-of-the-art methods.

• Access to Document (DOI)

Visual simulation of opal using bond percolation throughthe weighted Voronoi diagram and the Ewald construction

Yokota S., Fujishiro I.

Visual Computer （Springer-Verlag）  40 （ 8 ） 5005 - 5016 2024年07月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り,  ISSN  01782789

　概要を見る

Opal, a renowned gemstone, exhibits an optical phenomenon known as “play of color,” which is classified as a kind of structural color. This unique property makes opal a highly valuable material from the natural science, industry, and humanity perspectives. Opal has a complex internal structure called colloidal polycrystalline, and thus, representations of opal remain unestablished in computer graphics. In this study, to approximate the complex internal structure of opal, we imitate its formation process using the three-dimensional additively weighted Voronoi tessellation and percolation model. Further, we apply path tracing to compute the diffraction patterns of visible light inside opal by utilizing the Ewald construction employed in X-ray diffraction theory. Finally, we achieve a visually plausible output through spectral rendering.

• Access to Document (DOI)

VigNet: Semiautomatic generation of vignette illustrations from video

Mayu Namai, Issei Fujishiro

IIEEJ Transactions on Image Electronics and Visual Computing （IIEEJ）  12 （ 1 ） 15 - 22 2024年06月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り

　概要を見る

A variety of summarization techniques has recently been proposed to manage the growing volume of media data, but most are oriented toward homogeneous media conversion, resulting in a limited compression ratio. In this study, we focus on the creation of a vignette illustration that represents the story in an animation or game briefly and that allows the viewer to understand its world perspective at a glance. If video can be converted into vignette illustration, it is expected to provide a more highly compressed summary of the media information. This paper proposes VigNet, a system that semiautomatically converts an input video into vignette illustrations so they reflect users’ preferences.

Efficient particle-based fluid surface reconstruction using mesh shaders and bidirectional two-level grids

Yuki Nishidate, Issei Fujishiro

ACM Proceedings of Computer Graphics and Interactive Techniques （ACM）  7 （ 1 ） 1 - 14 2024年05月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り

　概要を見る

In this paper, we introduce a novel method for particle-based fluid surface reconstruction that incorporates mesh shaders for the first time. This approach eliminates the need to store triangle meshes in a GPU's global memory, resulting in a significant reduction in the amount of memory needed. Furthermore, our method employs a bidirectional two-level uniform grid, which not only accelerates the computationally expensive stage of surface cell detection but also effectively addresses the issue of vertex overflow among the mesh shaders. Experimental results prove that our method outperforms the state-of-the-art method, achieving both acceleration and memory reductions simultaneously, without sacrificing quality. The method is highly practical, with no major limitations other than requiring a GPU that supports the mesh shaders.

• Access to Document (DOI)

DiffMat: Latent diffusion models for image-guided material generation

Yuan L., Yan D., Saito S., Fujishiro I.

Visual Informatics （Elsevier）  8 （ 1 ） 6 - 14 2024年03月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り,  ISSN  25432656

　概要を見る

Creating realistic materials is essential in the construction of immersive virtual environments. While existing techniques for material capture and conditional generation rely on flash-lit photos, they often produce artifacts when the illumination mismatches the training data. In this study, we introduce DiffMat, a novel diffusion model that integrates the CLIP image encoder and a multi-layer, cross-attention denoising backbone to generate latent materials from images under various illuminations. Using a pre-trained StyleGAN-based material generator, our method converts these latent materials into high-resolution SVBRDF textures, a process that enables a seamless fit into the standard physically based rendering pipeline, reducing the requirements for vast computational resources and expansive datasets. DiffMat surpasses existing generative methods in terms of material quality and variety, and shows adaptability to a broader spectrum of lighting conditions in reference images.

• Access to Document (DOI)

HYDRO: Optimizing interactive hybrid images for digital signage content

Masanori Nakayama, Karin Uchino, Ken Nagao, Issei Fujishiro

Virtual Reality & Intelligent Hardware （Elsevier Science）  5 （ 6 ） 565 - 577 2023年12月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り,  ISSN  20965796

　概要を見る

In modern society, the digital signage installed in many large-scale facilities supports daily life. However, owing to their limited screen size, it is difficult to simultaneously provide different types of information to many viewers at varying distances from the screen. Therefore, in this study, we extend the existing research on the use of hybrid images for tiled displays. To facilitate smoother information selection, a new interactive display method is proposed that incorporates touch-activated widgets as high-frequency parts of hybrid images. These widgets are novel because they are more visible to viewers near the display. We developed an authoring tool called the hybrid image display resolution optimizer (HYDRO), which features two types of control functions to optimize the visibility of touch-activated widgets in terms of placement and resolution. The effectiveness of the proposed method is demonstrated empirically through a quantitative user study and an eyetracking-based qualitative evaluation.


• Access to Document (DOI)

Multiview SVBRDF capture from unified shape and illumination

Yuan L., Fujishiro I.

Visual Informatics （Elsevier）  7 （ 3 ） 11 - 21 2023年09月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り,  ISSN  25432656

　概要を見る

This paper proposes a stable method for reconstructing spatially varying appearances (SVBRDFs) from multiview images captured under casual lighting conditions. Unlike flat surface capture methods, ours can be applied to surfaces with complex silhouettes. The proposed method takes multiview images as inputs and outputs a unified SVBRDF estimation. We generated a large-scale dataset containing the multiview images, SVBRDFs, and lighting appearance of vast synthetic objects to train a two-stream hierarchical U-Net for SVBRDF estimation that is integrated into a differentiable rendering network for surface appearance reconstruction. In comparison with state-of-the-art approaches, our method produces SVBRDFs with lower biases for more casually captured images.

• Access to Document (DOI)

Visual simulation of crack and bend generation in deteriorated films coated on metal objects: Combination of static fracture and position-based deformation

Ishitobi A., Nakayama M., Fujishiro I.

Visual Computer （Springer-Verlag）  39 （ 8 ） 3403 - 3415 2023年08月

研究論文（学術雑誌）, 共著, 最終著者, 査読有り,  ISSN  0178-2789

　概要を見る

Weathering, an expression of degradation caused by rain and wind, is essential for photorealistic computer graphics. One of the commonest targets of weathering is metal, which is omnipresent in reality. However, for the realistic reproduction of scenes, many of which display degradation, the application of rust-proof paint to metal surfaces cannot be ignored. In our study, we propose a weathering method for coated films on metal objects, which are modeled using a three-dimensional (3D) triangular polygon mesh and deformed by combining two kinds of simulations: static simulation, for determining fractures based on the balance of the internal forces, and position-based bend simulation for moving vertices according to geometric constraints. Our method can digitally reproduce the deterioration of coated films using complex 3D deformation, which is difficult to achieve by material manipulation only.

• Access to Document (DOI)

Controllable automatic generation of non-player characters in 3D anime style

Li R., Oji R., Fujishiro I.

Computer Animation and Virtual Worlds （Wiley）  34 （ 2 ） e2047:1 - e2047:16 2023年03月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り,  ISSN  15464261

　概要を見る

Game creators always use prefabricated models as non-player characters (NPCs) to create an atmosphere for game scenes. However, players are likely to see different NPCs with the same 3D models because creators usually cannot prepare a high number of NPC models for big game scenes. In this article, we present an anime-like character customization system, where each customizing parameter can adjust the shape or color for the corresponding part of the character model. Based on this system, we propose an improved approach for generating a rich variety of 3D anime-like NPCs including body models and clothing items in different styles. We introduce a neural network to control the facial appearances, Gaussian mixture models to control the colors of hair and clothes, and a Bayesian network to control the outfits of clothing items. We demostrate that the proposed approach can maintain the variety and stability of the generated characters.

• Access to Document (DOI)

Multi-dimensional time-series subsequence clustering for visual feature analysis of blazar observation datasets

Sawada N., Uemura M., Fujishiro I.

Astronomy and Computing （Elsevier Science）  41 （ 100663 ） 1 - 12 2022年11月

研究論文（学術雑誌）, 共著, 最終著者, 査読有り,  ISSN  2213-1337

　概要を見る

Exploring hidden structures in subsequences extracted from long-term time-series data is one of the primary tasks of time-series data analysis. Clustering is one of the most commonly used techniques in that context; however, various existing issues must be addressed, such as the way to extract less-overlapping subsequences and the definition of the inter-subsequence similarity. Especially in multi-dimensional data analysis, correlations among variables should also be emphasized. To boost users’ exploration of the universalities of subsequences, we incorporate multi-dimensional time-series subsequence clustering methods and visual clustering analysis interface into TimeTubesX, which is an integrated visual analytics environment for multi-dimensional time-dependent observation datasets of blazars. TimeTubesX extracts and filters subsequences with various lengths according to the characteristics of the data and clusters them automatically in consideration of correlations among observed attributes. And then, it allows users to visually examine the clustering results in terms of the cluster features, intercluster transitions, and temporal distributions of clusters. Through the application to two practical case studies, we demonstrate how the enhanced TimeTubesX enables users to see not only instances but also universalities (i.e., time-series motifs or cluster prototypes) in time-series observations of blazars.

• Access to Document (DOI)

FORSETI: A visual analysis environment enabling provenance awareness for the accountability of e-autopsy reports

Wang B., Adachi N., Fujishiro I.

Visual Informatics （Elsevier Science）  6 （ 3 ） 69 - 80 2022年09月

研究論文（学術雑誌）, 共著, 最終著者, 責任著者, 査読有り,  ISSN  25432656

　概要を見る

Autopsy reports play a pivotal role in forensic science. Medical examiners (MEs) and diagnostic radiologists (DRs) cross-reference autopsy results in the form of autopsy reports, while judicial personnel derive legal documents from final autopsy reports. In our prior study, we presented a visual analysis system called the forensic autopsy system for e-court instruments (FORSETI) with an extended legal medicine markup language (x-LMML) that enables MEs and DRs to author and review e-autopsy reports. In this paper, we present our extended work to incorporate provenance infrastructure with authority management into FORSETI for forensic data accountability, which contains two features. The first is a novel provenance management mechanism that combines the forensic autopsy workflow management system (FAWfMS) and a version control system called lmmlgit for x-LMML files. This management mechanism allows much provenance data on e-autopsy reports and their documented autopsy processes to be individually parsed. The second is provenance-supported immersive analytics, which is intended to ensure that the DRs’ and MEs’ autopsy provenances can be viewed, listed, and analyzed so that a principal ME can author their own report through accountable autopsy referencing in an augmented reality setting. A fictitious case with a synthetic wounded body is used to demonstrate the effectiveness of the provenance-aware FORSETI system in terms of data accountability through the experience of experts in legal medicine.

• Access to Document (DOI)

3D distance field-based apparel modeling

Masanori Nakayama, Takami Yamamoto, Issei Fujishiro

IIEEJ Transactions on Image Electronics and Visual Computing （IIEEJ）  10 （ 1 ） 55 - 65 2022年06月

研究論文（学術雑誌）, 共著, 最終著者, 査読有り

　概要を見る

This study aims to design an apparel item by converting a polygon model of the human body obtained through a 3D scan into a volume model. From the model, we can quickly generate a curved surface that retains a constant distance from the surface of the body, thus being able to design an apparel item that ensures accurate allowance by defining the distance field for the surface of the body. The curved surface, which would be an ideal garment that can envelop the body, can be generated by tweaking the threshold value for an isosurface from the surface of the body. Furthermore, by drawing design lines on the curved surface, the pattern required for actual sewing can be designed in 3D space. The curved surface trimmed by the design lines is converted into a polygon mesh. A dynamics simulation is applied to smooth the curvatures so that the desired flat surface pattern can be obtained. As a result of actually sewing the virtually designed pattern and trying it on, it was shown that we can generate clothes with the appropriate allowance and proportions.

Illumination-aware group portrait compositor

Ohkawara M., Fujishiro I.

Visual Computer （Springer-Verlag）  38 （ 12 ） 4009 - 4018 2022年05月

研究論文（学術雑誌）, 共著, 最終著者, 査読有り,  ISSN  01782789

　概要を見る

We present a novel compositing framework for full-length human figures that maintains their surface details and appends the localized nature of light and shadow, thereby synthesizing composite results with high visual coherence. The framework is extended from the compositing pipeline proposed in our previous study so that it deploys five stages for photometric information estimation, as well as for 3D reconstruction, global illumination simulation, lighting transfer, and compositing. Based on the interpretation that a sense of coexistence can be achieved through visual coherence, we demonstrate that the proposed framework functions properly as a group portrait compositor. The composite results that the proposed framework composed the images separately rendered 3D human models compared favorably with the results which rendered multiple avatars together. Based on this empirical evaluation, the proposed framework is expected as a new means of fostering a sense of coexistence in remote societies and of efficiently generating highly photorealistic cyberworlds.

• Access to Document (DOI)

LineM: assessing metamorphopsia symptom using line manipulation task

Zhu Z., Toyoura M., Fujishiro I., Go K., Kashiwagi K., Mao X.

Visual Computer （Springer-Verlag）  38 （ 5 ） 1607 - 1617 2022年05月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  01782789

　概要を見る

Metamorphopsia patients experience distortions in their views. The number of metamorphopsia patients is arising with the increase in the elderly population. For early detection of metamorphopsia, examination approaches have been proposed. However, none of the existing methods can be employed to precisely assess the perceptions of patients. We propose LineM as a novel method to precisely measure the distorted views of metamorphopsia patients based on interactive line manipulation task involving the patients. To explore the potential of LineM, a case study which uses the measured results from LineM to visualize the distorted views of metamorphopsia patients is presented. Furthermore, a novel technique for compensating the distorted views of individual metamorphopsia patients using the results from LineM is also presented. Considering that elderly individuals account for the majority of metamorphopsia patients, an intuitive and user-friendly prototype system has been implemented. The effectiveness and usability of LineM and the compensation method were demonstrated via a qualitative and subjective experiment that involved metamorphopsia patients and elderly participants.

• Access to Document (DOI)

TimeTubesX: A query-driven visual exploration of observable, photometric, and polarimetric behaviors of blazars

Naoko Sawada, Makoto Uemura, Johanna Beyer, Hanspeter Pfister, Issei Fujishiro:

IEEE Transactions on Visualization and Computer Graphics （IEEE）  28 （ 4 ） 1917 - 1929 2022年04月

研究論文（学術雑誌）, 共著, 最終著者, 査読有り,  ISSN  1077-2626

　概要を見る

Blazars are celestial bodies of high interest to astronomers. In particular, through the analysis of photometric and polarimetric observations of blazars, astronomers aim to understand the physics of the blazar's relativistic jet. However, it is challenging to recognize correlations and time variations of the observed polarization, intensity, and color of the emitted light. In our prior study, we proposed TimeTubes to visualize a blazar dataset as a 3D volumetric tube. In this paper, we build primarily on the TimeTubes representation of blazar datasets to present a new visual analytics environment named TimeTubesX, into which we have integrated sophisticated feature and pattern detection techniques for effective location of observable and recurring time variation patterns in long-term, multi-dimensional datasets. Automatic feature extraction detects time intervals corresponding to well-known blazar behaviors. Dynamic visual querying allows users to search long-term observations for time intervals similar to a time interval of interest (query-by-example) or a sketch of temporal patterns (query-by-sketch). Users are also allowed to build up another visual query guided by the time interval of interest found in the previous process and refine the results. We demonstrate how TimeTubesX has been used successfully by domain experts for the detailed analysis of blazar datasets and report on the results.

• Access to Document (DOI)

Personalized image recoloring for color vision deficiency compensation

Zhenyang Zhu, Masahiro Toyoura, Kentaro Go, Kenji Kashiwagi, Issei Fujishiro, Tien-Tsin Wong, Xiaoyang Mao

IEEE Transactions on Multimedia （IEEE）  24   1721 - 1734 2022年04月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  1520-9210

　概要を見る

Several image recoloring methods have been proposed to compensate for the loss of contrast caused by color vision deficiency (CVD). However, these methods only work for dichromacy (a case in which one of the three types of cone cells loses its function completely), while the majority of CVD is anomalous trichromacy (another case in which one of the three types of cone cells partially loses its function). In this paper, a novel degree-adaptable recoloring algorithm is presented, which recolors images by minimizing an objective function constrained by contrast enhancement and naturalness preservation. To assess the effectiveness of the proposed method, a quantitative evaluation using common metrics and subjective studies involving 14 volunteers with varying degrees of CVD are conducted. The results of the evaluation experiment show that the proposed personalized recoloring method outperforms the state-of-the-art methods, achieving desirable contrast enhancement adapted to different degrees of CVD while preserving naturalness as much as possible.

• Access to Document (DOI)

Enhancing edge indicator for visual field loss compensation for homonymous hemianopia patients

Ichinose K., Zhao X., Fujishiro I., Toyoura M., Kashiwagi K., Go K., Mao X.

Visual Computer （Springer-Verlag）  38 （ 4 ） 1263 - 1273 2022年04月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  0178-2789

　概要を見る

Homonymous hemianopia (HH) is a visual field defect in which the same side of both eyes is blind. In this paper, we propose an out-of-visual-field information compensation system for HH patients using an optical see-through head-mounted display. In the proposed system, in order to reduce the obstruction in the remaining visual field, indicators are placed at the boundary between the remaining visual field and the defective part to notify the HH patient about the events occurring at the blinded side of the visual field. Two different kinds of indicators, namely brightness-based and flicker-based, are designed to visualize the spatial information of the events. We conducted experiments with simulated HH participants to verify the effectiveness of the proposed system and compare the two designed indicators, as well as an existing method that makes use of the overlay overview window. The experimental results proved that the proposed system can react more intuitively and is faster than the existing method in the part near the central visual field. Moreover, it can alleviate the obstruction problem of the existing method.

• Access to Document (DOI)

FORSETI: a visual analysis environment for authoring autopsy reports in extended legal medicine mark-up language

Wang B., Asayama Y., Boussejra M.O., Shojo H., Adachi N., Fujishiro I.

Visual Computer （Springer-Verlag）  37 （ 12 ） 2951 - 2863 2021年06月

研究論文（学術雑誌）, 査読有り,  ISSN  1432-2315

　概要を見る

In forensic autopsy, medical examiners (MEs) and diagnostic radiologists (DRs) cooperate with each other to perform an autopsy of the corpse. Effective computational assistance tools are imperative for facilitating the intricate collaborative work involved in the autopsy. In this paper, we present an integrated visual analysis environment named FORSETI (forensic autopsy system for e-court instruments), whose technical essence is twofold. The first is to be designed on the basis of an extended version of legal medicine mark-up language for authoring reports on physical autopsy (PA) as well as on virtual autopsy (VA). The second lies in autopsy juxtaposition, which seamlessly assists the MEs and DRs in referring to the VA and PA works, respectively. A fictitious case with the Visible Female Dataset is used to demonstrate the effectiveness of an initial prototype of the FORSETI system.

• Access to Document (DOI)

Experiencing GPU path tracing in online courses

Masaru Ohkawara, Hideo Saito, Issei Fujishiro

Graphics and Visual Computing （Elsevier Science）  4   200022:1 - 200022:10 2021年06月

研究論文（学術雑誌）, 共著, 査読有り

　概要を見る

In consideration of interdependency between image sensing/recognition (computer vision, CV) and 3D image synthesis (computer graphics, CG) in visual computing, Keio University, Department of Information and Computer Science, reorganized its first course on CV and CG in the undergraduate program into a series of three courses on visual computing in the 2019 academic year. One salient feature of these courses is its newly introduced programming assignment with two specific goals: to experience GPU computing and to understand the path tracing algorithm. The purpose is to help students easily understand the trend in visual computing and vividly envision the future CG area. Specifically, two types of tasks were given to students: material design and analysis of the relationship between sampling and noise. The educational material builds on Google Colaboratory, a cloud-based development environment, to be independent of the students’ hardware. Owing to the judicious design, students can unhesitatingly work on the programming assignment with relatively inexpensive hardware, such as laptop PCs, tablets, or even smartphones, whenever they have a standard off-campus network environment. In the second academic year (2020) of these courses, this type of exercise was especially valuable for students who had to take the courses online because of the COVID-19 pandemic. The effects of the new courses with the educational materials were empirically proven in terms of quantitative and qualitative perspectives.

• Access to Document (DOI)

Different Eye Movement Behaviors Related to Artificial Visual Field Defects-A Pilot Study of Video-Based Perimetry

Mao C., Go K., Kinoshita Y., Kashiwagi K., Toyoura M., Fujishiro I., Li J., Mao X.

IEEE Access （IEEE）  9   77649 - 77660 2021年05月

研究論文（学術雑誌）, 共著, 査読有り,  ISSN  2169-3536

　概要を見る

Visual field defects (VFDs) can be caused by numerous diseases, some of which (e.g., glaucoma) are the main causes of blindness in humans. The present perimetry, which is defined as the measurement of visual field function, has high requirements for patients such as long-term fixation and reliable interaction with the system. In this pilot study, we combined the videos and eye-tracking techniques in a free-watching task and explored the different eye movement behaviors of people with several types of artificial VFDs, such as hemianopia, altitudinal VFDs, and tunnel vision. We carried out the task in a gaze-contingent modality where 38 participants with normal vision were recruited to watch a group of videos monocularly. The eye gaze data were recorded by the tracker during the task. We hypothesized that people with simulated VFDs will produce a more active eye movement coping mechanism to compensate for their visual field disadvantage. A new measurement called eye movement amount (MA) was proposed to describe the amount of eye movement toward a specific direction. Statistically significant differences caused by artificial VFDs were observed by comparing the MA values between the VFD groups and the control group. In addition, we found artificial hemianopia and inferior VFDs lead participants to produced increased MA in the horizontal and vertical directions respectively. Artificial tunnel vision, on the contrary, induces a decreased MAs in both horizontal and vertical directions. The proposed metrics can be used as potential biomarkers for distinguishing VFDs in a free-watching task.

• Access to Document (DOI)