Details of a Researcher

Papers - Saito, Hideo

Division display 41 - 60 of about 356 ／ All the affair displays >>

3D PixMix: Image Inpainting in 3D Environments

Mori S., Herling J., Broll W., Kawai N., Saito H., Schmalstieg D., Kalkofen D.

Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018 (Adjunct Proceedings - 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR-Adjunct 2018) 1 - 2 2018.07

ISSN 9781538675922

© 2018 IEEE. State of the art methods for diminished reality (DR) propagate pixel information from keyframes to later frames to achieve real-time image inpainting in 3D spaces. However, this approach assumes a planar scene and produces artifacts, if the scene geometry is not sufficiently planar. In this paper, we present 3D PixMix, a new real-time inpainting method that addresses non-planar scenes by considering both color and depth information in the inpainting process. We define cost functions for both the color and the geometric appearance in the inpainting scheme and use an RGB-D sensor for depth fusion using a SLAM. Comparison results against the conventional PixMix show that 3D PixMix obtains the equivalent or even better quality in 3D scenes with additional depth information.

Access to Document (DOI)

Superimposing thermal-infrared data on 3D structure reconstructed by RGB visual odometry

Yamaguchi M., Truong T.P., Mori S., Nozick V., Saito H., Yachida S., Sato H.

IEICE Transactions on Information and Systems (IEICE Transactions on Information and Systems) E101D ( 5 ) 1296 - 1307 2018.05

ISSN 09168532

© 2018 The Institute of Electronics, Information and Communication Engineers. In this paper, we propose a method to generate a threedimensional (3D) thermal map and RGB + thermal (RGB-T) images of a scene from thermal-infrared and RGB images. The scene images are acquired by moving both a RGB camera and an thermal-infrared camera mounted on a stereo rig. Before capturing the scene with those cameras, we estimate their respective intrinsic parameters and their relative pose. Then, we reconstruct the 3D structures of the scene by using Direct Sparse Odometry (DSO) using the RGB images. In order to superimpose thermal information onto each point generated from DSO, we propose a method for estimating the scale of the point cloud corresponding to the extrinsic parameters between both cameras by matching depth images recovered from the RGB camera and the thermal-infrared camera based on mutual information. We also generate RGB-T images using the 3D structure of the scene and Delaunay triangulation. We do not rely on depth cameras and, therefore, our technique is not limited to scenes within the measurement range of the depth cameras. To demonstrate this technique, we generate 3D thermal maps and RGB-T images for both indoor and outdoor scenes.

Access to Document (DOI)

Line-based SLAM using non-overlapping cameras in an urban environment

Kawasaki A., Hara K., Saito H.

IEICE Transactions on Information and Systems (IEICE Transactions on Information and Systems) E101D ( 5 ) 1232 - 1242 2018.05

ISSN 09168532

© 2018 The Institute of Electronics, Information and Communication Engineers. We propose a method of line-based Simultaneous Localization and Mapping (SLAM) using non-overlapping multiple cameras for vehicles running in an urban environment. It uses corresponding line segments between images taken by different frames and different cameras. The contribution is a novel line segment matching algorithm by warping processing based on urban structures. This idea significantly improves the accuracy of line segment matching when viewing direction are very different, so that a number of correspondences between front-view and rear-view cameras can be found and the accuracy of SLAM can be improved. Additionally, to enhance the accuracy of SLAM we apply a geometrical constraint of urban area for initial estimation of 3D mapping of line segments and optimization by bundle adjustment. We can further improve the accuracy of SLAM by combining points and lines. The position error is stable within 1.5m for the entire image dataset evaluated in this paper. The estimation accuracy of our method is as high as that of ground truth captured by RTK-GPS. Our high accuracy SLAM algorithm can be apply for generating a road map represented by line segments. According to an evaluation of our generating map, true positive rate around the vehicle exceeding 70% is achieved.

Access to Document (DOI)

Simultaneous object segmentation and recognition by merging CNN outputs from uniformly distributed multiple viewpoints

Nakajima Y., Saito H.

IEICE Transactions on Information and Systems (IEICE Transactions on Information and Systems) E101D ( 5 ) 1308 - 1316 2018.05

ISSN 09168532

© 2018 The Institute of Electronics, Information and Communication Engineers. We propose a novel object recognition system that is able to (i) work in real-time while reconstructing segmented 3D maps and simultaneously recognize objects in a scene, (ii) manage various kinds of objects, including those with smooth surfaces and those with a large number of categories, utilizing a CNN for feature extraction, and (iii) maintain high accuracy no matter how the camera moves by distributing the viewpoints for each object uniformly and aggregating recognition results from each distributed viewpoint as the same weight. Through experiments, the advantages of our system with respect to current state-of-the-art object recognition approaches are demonstrated on the UW RGB-D Dataset and Scenes and on our own scenes prepared to verify the effectiveness of the Viewpoint-Class-based approach.

Access to Document (DOI)

Synchronizing 3D point cloud from 3D scene flow estimation with 3D Lidar and RGB camera

Hiroki Usami, Hideo Saito, Jun Kawai and Noriko Itani

IS&T International Symposium on Electronic Imaging 2018: 3D Image Processing, Measurement (3DIPM), and Applications 2018 proceedings. (IS and T International Symposium on Electronic Imaging Science and Technology) 1 - 6 2018.01

Research paper (international conference proceedings), Joint Work, Accepted

We present a method for synchronizing three-dimensional (3D)
point cloud from 3D scene with estimation using a 3D Lidar and an
RGB camera. These 3D points sensed by the 3D Lidar are not
captured at the same time, which makes it difficult to measure the
correct shape of the object in a dynamic scene. In our method, we
generate synchronized 3D points at arbitrary times using linear
interpolation in four-dimensional space, time and space. For
interpolating the 3D point, we obtain corresponding 3D point
matching with the pixel value captured by the RGB camera in a
continuous frame. The experimental results demonstrate the
effectiveness of the presented method by depicting a synchronized
3D point cloud that is correctly shaped.

Access to Document (DOI)

Vehicle pose estimation from drive recorder images by monocular SLAM and matching with rendered 3D point cloud of surrounding environment

Kurobe A., Kinoshita H., Saito H.

IS and T International Symposium on Electronic Imaging Science and Technology (IS and T International Symposium on Electronic Imaging Science and Technology) 2371 - 2376 2018

© 2018, Society for Imaging Science and Technology. Vehicle pose estimation is a vital technology for reconstructing the circumstaces of traffic accidents. We propose a novel method for reconstructing the trajectory of vehicles from drive recorder images and a point cloud around the road. First, we apply ORB-SLAM to image sequence of the drive recorder for obtaining the vehicle pose trajectory; however this is based on relative coordinates and a relative scale. For estimating the absolute coordinates and scale of the trajectory, which cannot be obtained from a monocular SLAM like ORB-SLAM, we match the feature points detected in the image sequence with the three-dimensional (3D) point cloud of surrounding environment. For finding 3D points matching the feature points, we generate candidate images by the rendering 3D point cloud of the surrounding environment using the position initially estimated by the Global Positioning System (GPS). Next, we match to obtain the 3D two-dimensional (2D) generated images and drive recorder image to get 3D-2D point correspondences between the 3D point cloud and the drive recorder images; thus, we can convert the relative estimation of the camera pose by ORB-SLAM to the coordinates of the 3D point cloud of the surrounding environment. In the evaluation experiments, we confirmed the effectiveness of our method by comparing the vehicle poses estimated by our method, with those of RTKGPS, which exhibits high measurement precision.

Access to Document (DOI)

Synthesis of wide-FOV RGB-D images by registration and upsampling of 3D Lidar with omnidirectional RGB camera

Usami H., Miyata S., Saito H.

23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016 (23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016) 4 2340 - 2342 2018

ISSN 9781510845510

Multi-view surface inspection using a rotating table

Kaichi T., Mori S., Saito H., Sugano J., Adachi H.

IS and T International Symposium on Electronic Imaging Science and Technology (IS and T International Symposium on Electronic Imaging Science and Technology) 2021 - 2026 2018

© 2018, Society for Imaging Science and Technology. In this paper, we introduce a method to visually inspect in- dustrial parts by using multi-view images of an industrial part on a rotating table. During the visual inspection of the parts' sur- faces, the relationship between the camera pose, the light vectors, and the normal vectors of a part's surface is the key factor in de- tecting abnormalities. We can change the relationship between the three factors by rotating the part; then, the abnormalities are visible in certain positional relationships. We, therefore, track the points on a part's surfaces in an image sequence and discrimi- nate abnormal points (such as points on scratches or dents) from normal points based on pixel value transitions while rotating the part. The experimental results, which were based on real data of industrial parts, showed that the proposed method could detect the abnormalities on the surfaces.

Access to Document (DOI)

Constant velocity 3d convolution

Sekikawa Y., Ishikawa K., Saito H.

IEEE Access (IEEE Access) 6 76490 - 76501 2018

© 2018 IEEE. We propose a novel 3-D convolution method, cv3dconv, for extracting spatiotemporal features from videos. It reduces the number of sum-of-products operations in 3-D convolution by thousands of times by assuming the constant moving velocity of the features. We observed that a specific class of video sequences, such as video captured by an in-vehicle camera, can be well approximated with piece-wise linear movements of 2-D features in a temporal dimension. Our principal finding is that a 3-D kernel, represented by constant velocity, can be decomposed into a convolution of a 2-D-shaped kernel and a 3-D-velocity kernel, which is parameterized using only two parameters. We derived an efficient recursive algorithm for this class of 3-D convolution, which is exceptionally suited for sparse spatiotemporal data, and this parameterized decomposed representation imposes a structured regularization along a temporal direction. We experimentally verified the validity of our approximation using a controlled dataset, and we also showed the effectiveness of the cv3dconv by adopting it for deep neural networks (DNNs) in visual odometry estimation task using publicly available event-based camera dataset captured in urban road scene. Our DNN architecture improves the estimation accuracy for about 30% compared with the existing states-of-the-arts architecture designed for event data.

Access to Document (DOI)

An overview of augmented visualization: Observing the real world as desired

Mori S., Saito H.

APSIPA Transactions on Signal and Information Processing (APSIPA Transactions on Signal and Information Processing) 7 2018

Copyright © The Authors, 2018 This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.. Over 20 years have passed since a free-viewpoint video technology has been proposed with which a user's viewpoint can be freely set up in a reconstructed three-dimensional space of a target scene photographed by multi-view cameras. This technology allows us to capture and reproduce the real world as recorded. Once we capture the world in a digital form, we can modify it as augmented reality (i.e., placing virtual objects in the digitized real world). Unlike this concept, the augmented world allows us to see through real objects by synthesizing the backgrounds that cannot be observed in our raw perspective directly. The key idea is to generate the background image using multi-view cameras, observing the backgrounds at different positions and seamlessly overlaying the recovered image in our digitized perspective. In this paper, we review such desired view-generation techniques from the perspective of free-view point image generation and discuss challenges and open problems through a case study of our implementations.

Access to Document (DOI)

A simple 3D scanning system of human foot using a smartphone with a depth camera

Kobayashi T., Ienaga N., Sugiura Y., Saito H., Miyata N., Tada M.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 84 ( 12 ) 996 - 1002 2018

ISSN 09120289

© 2018 Japan Society for Precision Engineering. All rights reserved. This paper presents a method to measure foot shape from a 3D point cloud as input captured from multiple directions using a smartphone depth camera. Such a 3D point cloud could potentially include noise or omit parts of the foot due to occlusion. To deal with this occlusion problem, we propose to use a dataset of 3D foot shapes collected by a precise 3D shape scanner of foot shapes. According to the dataset of 3D foot shapes, we can generate a deformable model by performing a principal component analysis (PCA) on the dataset. Then we minimize the error of the shape represented by the deformable model and the 3D point cloud acquired by the smartphone camera, to recover a complete 3D shape of the entire foot with high accuracy. We test this method by comparing the 3D shape produced by our proposed method to the 3D shape precisely measured by the 3D scanner. Our proposed method can scan the foot shape with an error of about 1,13mm.

Access to Document (DOI)

A refocus-interface for diminished reality work area visualization

Maezawa M., Mori S., Saito H.

IS and T International Symposium on Electronic Imaging Science and Technology (IS and T International Symposium on Electronic Imaging Science and Technology) 1121 - 1124 2018

© 2018, Society for Imaging Science and Technology. In this paper, we present a refocus interface to set the parameters used for diminished reality (DR)-based work area visualization and a multiview camera-based rendering scheme. The refocus interface allows the user to determine two planes - one for setting a virtual window, through which the user can observe the background occluded by an object, and the other for a background plane, which is used for the subsequent background rendering. The background is rendered considering the geometric and appearance relationships of the multiview cameras observing the scene. Our preliminary results demonstrate that our DR system can visualize the hidden background.

Access to Document (DOI)

Synthesis of top view image and the detection of obstacles using multiple cameras for monitoring rea around a truck

Uehara K., Saito H., Yamamoto K., Sato H.

23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016 (23rd International Display Workshops in conjunction with Asia Display, IDW/AD 2016) 4 2304 - 2307 2018

ISSN 9781510845510

Augmented Visualization: Observing as Desired

Shohei Mori and Hideo Saito

Proceedings of 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) (Proceedings - 9th Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2017) 2018-February 351 - 356 2017.12

Research paper (international conference proceedings), Accepted, ISSN 9781538615423

Over 20 years have passed since a free-viewpoint video technology has been proposed in which a user's viewpoint can be freely set up reconstructed 3D space of a target scene photographed by multi-view cameras. This technology allows us to capture and reproduce the real world as it recorded. Once we capture the world in a digital form, we can modify the world as augmented reality does (i.e., placing virtual objects in the digitized real world). As oppose to this concept, the augmented world also allows us to see through real objects by synthesizing the backgrounds that cannot be observed in our raw perspective directly. The key idea is to generate the background image using multi-view cameras observing the backgrounds at different positions, and seamlessly overlaying the recovered image in our digitized perspective. In this article, our studies regarding such desired view generation techniques are reviewed, and its future directions and open problems are discussed.

Access to Document (DOI)

Visual Inspection by Capturing a Rotating Industrial Part

Tomoya KAICHI, Shohei MORI, Hideo SAITO, Junichi SUGANO, Hideyuki ADACHI

Journal of the Japan Society for Precision Engineering 83 ( 12 ) 1184 - 1191 2017.12

Research paper (scientific journal), Accepted

Visual inspection is a vital step for maintaining the quality of industrial parts. In visual inspection, the relation between the three factors, camera pose, lights direction and normal vector of a part is very important to detect abnormalities on industrial part. We propose a visual inspection method for industrial parts using an image sequence captured while rotating an industrial part using a rotating table. By rotating a part, we can easily change the relation between three factors. We track the points on part's surfaces in an image sequence. We discriminate abnormal surface points such as scratches from the normal ones based on pixel value transitions in an image sequence. For accurately tracking the points on the part surface, we propose a novel camera calibration method of the rotating table and the camera with a telecentric lens and a pose estimation method for the rotating part. We use angular invariant feature vectors for improving robustness against types and shapes of industrial parts to be inspected. We presented experimental results using real data of industrial parts and verified that the proposed method could detect scratches on surfaces.

Access to Document (DOI)

Line-Based SLAM Considering Directional Distribution of Line Features in an Urban Environment

Kei Uehara, Hideo Saito, Kosuke Hara

The journal of the Institute of Image Electronics Engineers of Japan : visual computing, devices & communications 46 ( 4 ) 522 - 532 2017.12

Research paper (scientific journal), Joint Work, Accepted

本論文では，線分特徴の事前分布を考慮した自動運転車のためのSLAM (Simultaneous Localization and Mapping) を提案する．提案手法は，道路標示や建物など，線分を検出しやすい物体が存在する都市環境を走行する車両に搭載されたカメラにより撮影される動画像を入力とすることを前提とし，この動画像から検出される線分特徴の方向に関する制約を加えることによりSLAM の精度を向上させるものである．線分特徴の方向に関する制約を加えるため，本手法では道路標示中の線分が車両進行方向に対して平行，もしくは垂直であるものが多いことに着目し，それら2 つの線分間の角度分布が混合ガウス分布に則ると仮定し，バンドル調整による最適化に導入する．これにより，カメラの位置姿勢と線分特徴の位置推定の精度を高めることが期待できる．本手法の有効性を確かめるため，カメラを1 つ用いたシングルカメラシステムと，2 つ用いたマルチカメラシステムの両方に提案手法を適用し精度検証を行った．その結果マルチカメラによる従来手法の精度と，シングルカメラによる本手法の精度が同等となった．また提案手法により生成した路面地図は，線分の方向に関する制約を導入したことにより補正され，精度向上を確認した．

Real-time scene reconstruction and triangle mesh generation using multiple RGB-D cameras

Meerits Siim, Nozick Vincent, Saito Hideo

Journal of Real-Time Image Processing (Springer) 16 ( 6 ) 1 - 13 2017.11

Research paper (scientific journal), ISSN 1861-8200

We present a novel 3D reconstruction system that can generate a stable triangle mesh using data from multiple RGB-D sensors in real time for dynamic scenes. The first part of the system uses moving least squares (MLS) point set surfaces to smooth and filter point clouds acquired from RGB-D sensors. The second part of the system generates triangle meshes from point clouds. The whole pipeline is executed on the GPU and is tailored to scale linearly with the size of the input data. Our contributions include changes to the MLS method for improving meshing, a fast triangle mesh generation method and GPU implementations of all parts of the pipeline.

Access to Document (DOI)

Semantic Object Selection and Detection for Diminished Reality Based on SLAM with Viewpoint Class

Nakajima Yoshikatsu, Mori Shohei, Saito Hideo

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 338 - 343 2017.10

Research paper (international conference proceedings), Accepted

We propose a novel diminished reality method which is able to (i) automatically recognize the region to be diminished, (ii) work with a single RGB-D sensor, and (iii) work without pre-processing to generate a 3D model of the target scene by utilizing SLAM, segmentation, and recognition framework. Especially, regarding the recognition of the area to be diminished, our method is able to maintain high accuracy no matter how the camera moves by distributing the viewpoints for each object uniformly and aggregating recognition results from each distributed viewpoint as the same weight. These advantages are demonstrated on the UW RGB-D Dataset and Scenes.

Access to Document (DOI)

An Instant See-Through Vision System Using a Wide Field-of-View Camera and a 3D-Lidar

Oishi Kei, Mori Shohei, Saito Hideo

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 344 - 347 2017.10

Research paper (international conference proceedings)

Diminished reality (DR) enables us to see through real objects occluding some areas in our field of view. This interactive display has various applications, such as see-through vision to visualize invisible areas, work area visualization in surgery and landscape simulation. In this paper, we propose two underlying problems in see-through vision, in which hidden areas are observed in real time. First, see-through vision methods require a common area to calibrate every camera in the environment. However, the field of view is limited and many approaches rely on a time-consuming calibration, sensors, or fiducial markers. Second, see-through vision applications assume that the background is planar to ease image alignment. We therefore present a place-and-play see-through vision system using a wide field-of-view RGB-D camera. We validated the accuracy and the robustness of our system and showed results in various environments to show the applicability.

Access to Document (DOI)

Diminished Reality for Privacy Protection by Hiding Pedestrians in Motion Image Sequences Using Structure from Motion

Yagi Kentaro, Hasegawa Kunihiro, Saito Hideo

2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) 334 - 337 2017.10

Research paper (international conference proceedings), Accepted

We present a method for generating images in which people are hidden from image sequences taken with a hand-held camera. Our method is basically used for privacy protection of people whose images are unintentionally captured in image sequences. We hide people from images by reconstructing a 3D model of background and projecting it to 2D images. By detecting the area in which people are present beforehand, we can reconstruct a 3D model of the background without people. In the experiment, We compare our method with some conventional approaches for diminished reality.

Access to Document (DOI)

　 Previous page - Next page 　