Details of a Researcher - Aoki, Yoshimitsu

Papers - Aoki, Yoshimitsu

Division display 41 - 60 of about 192 ／ All the affair displays >>

DOCUMENT SHADOW REMOVAL WITH FOREGROUND DETECTION LEARNING FROM FULLY SYNTHETIC IMAGES

Matsuo Y., Akimoto N., Aoki Y.

Proceedings - International Conference on Image Processing, ICIP (Proceedings - International Conference on Image Processing, ICIP) 1656 - 1660 2022

ISSN 15224880

Shadow removal for document images is a major task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large-scale and diverse dataset is laborious and remains a great challenge. Thus, only small real datasets are available. To create relatively large datasets, a graphic renderer has been used to synthesize shadows, nonetheless, it is still necessary to capture real documents. Thus, the number of unique documents is limited, which negatively affects a network's performance. In this paper, we present a large-scale and diverse dataset called fully synthetic document shadow removal dataset (FSDSRD) that does not require capturing documents. The experiments showed that the networks (pre-)trained on FSDSRD provided better results than networks trained only on real datasets. Additionally, because foreground maps are available in our dataset, we leveraged them during training for multitask learning, which provided noticeable improvements. The code is available at: https://github.com/IsHYuhi/DSRFGD.

Access to Document (DOI)

Detecting Spinal Alignment Through Depth Image

Ishibashi R., Kameshima H., Nishio Y., Sato Y., Watanabe K., Aoki Y.

Proceedings - 16th International Conference on Signal-Image Technology and Internet-Based Systems, SITIS 2022 (Proceedings - 16th International Conference on Signal-Image Technology and Internet-Based Systems, SITIS 2022) 330 - 334 2022

Adolescent Idiopathic Scoliosis (AIS) is a common form of scoliosis found among adolescents. The abnormal spinal curvature along with twisting of the vertebrae requires early detection before the vertebrae alignment worsens. Furthermore, surgical treatment is crucial for those with severe conditions. In Japan, annual scoliosis screening is conducted during school health checkups. These screening procedures often involve an initial screening without the use of ionizing radiation (such as X-rays) and an X-ray screening later for those suspected with scoliosis. Currently, X-ray examinations produce the most accurate results for detecting the spinal position. However, noninvasive alternative methods, such as Moire imaging, provide satisfactory initial results. Unfortunately, the production of Moire cameras have halted in recent years. This paper proposes an alternative, non-invasive and non-ionizing radioactive method to detect spinal alignment. Depth images provide an extra dimension of information compared to RGB images. The proposed method utilizes this extra dimension of depth values to create different types of images that are then trained using Convolutional Neural Networks to predict the spinal alignment. Results indicate that Moire images reproduced from depth images produce the best spinal alignment. However, other types of images derived from the depth image also have higher accuracy when compared with the depth image itself.

Access to Document (DOI)

Temporal Cross-Modal Attention for Audio-Visual Event Localization

Nagasaki Y., Hayashi M., Kaneko N., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 88 ( 3 ) 263 - 268 2022

ISSN 09120289

In this paper, we propose a new method for audio-visual event localization 1) to find the corresponding segment between audio and visual event. While previous methods use Long Short-Term Memory (LSTM) networks to extract temporal features, recurrent neural networks like LSTM are not able to precisely learn long-term features. Thus, we propose a Temporal Cross-Modal Attention (TCMA) module, which extract temporal features more precisely from the two modalities. Inspired by the success of attention modules in capturing long-term features, we introduce TCMA, which incorporates self-attention. Finally, we were able to localize audio-visual event precisely and achieved a higher accuracy than the previous works.

Access to Document (DOI)

Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation

Akimoto N., Matsuo Y., Aoki Y.

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition) 2022-June 11431 - 11440 2022

ISSN 10636919

We address the problem of generating a 360-degree image from a single image with a narrow field of view by estimating its surroundings. Previous methods suffered from overfitting to the training resolution and deterministic generation. This paper proposes a completion method using a transformer for scene modeling and novel methods to improve the properties of a 360-degree image on the output image. Specifically, we use CompletionNets with a transformer to perform diverse completions and Adjust-mentNet to match color, stitching, and resolution with an input image, enabling inference at any resolution. To improve the properties of a 360-degree image on an output image, we also propose WS-perceptual loss and circular inference. Thorough experiments show that our method out-performs state-of-the-art (SOTA) methods both qualitatively and quantitatively. For example, compared to SOTA methods, our method completes images 16 times larger in resolution and achieves 1.7 times lower Fréchet inception distance (FID). Furthermore, we propose a pipeline that uses the completion results for lighting and background of 3DCG scenes. Our plausible background completion enables perceptually natural results in the application of inserting virtual objects with specular surfaces.

Access to Document (DOI)

Determinant analysis and developing evaluation indicators of grade of execution score of double axel jump in figure skating

Hirosawa S., Watanabe M., Aoki Y.

Journal of Sports Sciences (Journal of Sports Sciences) 40 ( 4 ) 470 - 481 2022

ISSN 02640414

A figure skating jump score is determined by the sum of the base value based on the difficulty and grade of execution (GOE) that indicates the performance quality. Therefore, performing a high-quality jump to obtain a high GOE is essential to win a competition. However, the relationship between the GOE and kinematic parameters remains unclear. We analysed the horizontal distance, vertical height, and landing speed of double axel jumps in the Ladies’ Short Program at the 2019 World Championships. The highest GOE group had significantly larger horizontal distances than the middle and lower groups, while the landing speed and vertical height were not significantly different. A principal component regression analysis was conducted to clarify the contrast between the three variables affecting the GOE. The results showed that greater horizontal distance and landing speed compared to vertical height (component 1) and greater horizontal distance compared to landing speed (component 3) contributed to higher GOE. We divided skaters into four clusters using these two components and provided general GOE acquisition strategies for each cluster. Finally, to apply our results to the industry, we proposed two new evaluation indicators which are highly correlated with the two components and easy to interpret.

Access to Document (DOI)

Improving Annotation Efficiency through Uncertain Sample Selection in Active Learning

Kawano Y., Nota Y., Mochizuki R., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 88 ( 2 ) 211 - 216 2022

ISSN 09120289

Active learning refers to label-efficient algorithms that use the most representative samples for labeling when creating training data. In this paper, we propose a model that derives the most informative unlabeled samples from the output of a task model. The tasks arc a classification problem, multi-label classification and a semantic segmentation problem. The model consists of an uncertainty indicator generator and a task model. After training the task model with labeled samples, the model predicts unlabeled samples, and based on the prediction results, the uncertainty indicator generator outputs an uncertainty indicator for each unlabeled sample. Samples with a higher uncertainty indicator are considered to be more informative and selected. As a result of experiments using multiple datasets, our model achieved better accuracy than conventional active learning methods and reduced execution time by a factor of 10.

Access to Document (DOI)

Secrets of Event-Based Optical Flow

Shiba S., Aoki Y., Gallego G.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)) 13678 LNCS 628 - 645 2022

ISSN 03029743

Event cameras respond to scene dynamics and offer advantages to estimate motion. Following recent image-based deep-learning achievements, optical flow estimation methods for event cameras have rushed to combine those image-based methods with event data. However, it requires several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. We investigate key elements: how to design the objective function to prevent overfitting, how to warp events to deal better with occlusions, and how to improve convergence with multi-scale raw events. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark. Moreover, our method allows us to expose the issues of the ground truth flow in those benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Our code is available at https://github.com/tub-rip/event_based_optical_flow.

Access to Document (DOI)

Self-Supervised Learning of Inlier Events for Event-based Optical Flow

Nagata J., Aoki Y.

BMVC 2022 - 33rd British Machine Vision Conference Proceedings (BMVC 2022 - 33rd British Machine Vision Conference Proceedings) 2022

Event cameras asynchronously report per-pixel intensity changes with high temporal resolution. Their sparse and temporally precise nature is well suited for fast visual flow estimation. The normal optical flow can be estimated by fitting a plane to spatio-temporal events. However, least-squares plane fitting suffers from outliers due to the significant noise of events. Existing methods involve 1) iterative outlier rejection or 2) goodness-of-fit rejection after single-shot planar fitting from greedily selected spatially neighboring events. In contrast to these methods, we propose a method of selecting the events supporting a plane before performing a fitting, using the inlier probability from a lightweight neural network that captures both global and local structures. During inference, single-shot planar fitting is performed from only events with a higher inlier probability. We model each event selection by a Bernoulli distribution with the inlier probability and train the network to maximize the inlier count while sampling in a self-supervised manner. We verify that our event selection improves the accuracy of optical flow estimation with publicly available real data.

Optical flow estimation by matching time surface with event-based cameras

Nagata J., Sekikawa Y., Aoki Y.

Sensors (Switzerland) (Sensors (Switzerland)) 21 ( 4 ) 1 - 14 2021.02

ISSN 14248220

In this work, we propose a novel method of estimating optical flow from event-based cameras by matching the time surface of events. The proposed loss function measures the timestamp consistency between the time surface formed by the latest timestamp of each pixel and the one that is slightly shifted in time. This makes it possible to estimate dense optical flows with high accuracy without restoring luminance or additional sensor information. In the experiment, we show that the gradient was more correct and the loss landscape was more stable than the variance loss in the motion compensation approach. In addition, we show that the optical flow can be estimated with high accuracy by optimization with L1 smoothness regularization using publicly available datasets.

Access to Document (DOI)

Alleviating over-segmentation errors by detecting action boundaries

Ishikawa Y., Kasai S., Aoki Y., Kataoka H.

Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021 (Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021) 2321 - 2330 2021.01

We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our contributions are three-fold: (i) We propose a framework for temporal action segmentation, the ASRF, which divides temporal action segmentation into frame-wise action classification and action boundary regression. Our framework refines frame-level hypotheses of action classes using predicted action boundaries. (ii) We propose a loss function for smoothing the transition of action probabilities, and analyze combinations of various loss functions for temporal action segmentation. (iii) Our framework outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in terms of segmental edit distance and up to 16.1% in terms of segmental F1 score. Our code is publicly available1.

Access to Document (DOI)

Analysis of recent re-identification architectures for tracking-by-detection paradigm in multi-object tracking

Ishikawa H., Hayashi M., Phan T.H., Yamamoto K., Masuda M., Aoki Y.

VISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications) 5 234 - 244 2021

ISSN 9789897584886

Person re-identification is a vital module of the tracking-by-detection framework for online multi-object tracking. Despite recent advances in multi-object tracking and person re-identification, inadequate attention was given to integrating these technologies to provide a robust multi-object tracker. In this work, we combine modern state-of-the-art re-identification models and modeling techniques on the basic tracking-by-detection framework and benchmark them on heavily occluded scenes to understand their effect. We hypothesize that temporal modeling for re-identification is crucial for training robust re-identification models for they are conditioned on sequences containing occlusions. Along with traditional image-based re-identification methods, we analyze temporal modeling methods used in video-based re-identification tasks. We also train re-identification models with different embedding methods, including triplet loss, and analyze their effect. We benchmark the re-identification models on the challenging MOT20 dataset containing crowded scenes with various occlusions. We provide a thorough assessment and investigation of the usage of modern re-identification modeling methods and prove that these methods are, in fact, effective for multi-object tracking. Compared to baseline methods, results show that these models can provide robust re-identification proved by improvements in the number of identity switching, MOTA, IDF1, and other metrics.

Scenery Image Extension via Inpainting with a Mirrored Input

Akimoto N., Ito D., Aoki Y.

IEEE Access (IEEE Access) 9 59286 - 59300 2021

Generative image extension has the advantage of extending the overall image size while preserving the target image because, unlike other image extensions using interpolation, it completes the surroundings of the target image. However, existing generative image extension methods tend to have poor quality in the generation of outer pixels. One method deals only with a limited number of scene classes because the extension repeats the same semantics. We propose a mirrored input, which sandwiches the extended region by mirroring a part of the target image. This replaces generative image extension with an image inpainting problem and thus helps to achieve higher quality pixel generation and can extend semantics with more complex shapes than horizontal repetition. Experimental results show that our proposed method achieves a scenery image extension that exceeds the state-of-the-art generative image extension methods in both visual quality and FID score for datasets containing diverse scenes.

Access to Document (DOI)

Out-of-distribution detection for fungi images with similar features

Kawashima Y., Higo M., Tokiwa T., Asami Y., Nonaka K., Aoki Y.

Proceedings of SPIE - The International Society for Optical Engineering (Proceedings of SPIE - The International Society for Optical Engineering) 11794 2021

ISSN 0277786X

In order to create a classification model for fungi, it is necessary to have robustness against out-of-distribution data from the viewpoint of practicality. Therefore, in this paper, we perform out-of-distribution detection on a fungi. Unlike the case of conventional out-of-distribution detection, the characteristics of in-distribution data and out-of-distribution data in this paper are very similar. Therefore, the problem in which conventional methods using out-of-distribution data for validation are not effective is mentioned. We also verify whether the accuracy of out-of-distribution detection can be improved using the attention branch network.

Access to Document (DOI)

Multi-modal action segmentation in the kitchen with a feature fusion approach

Kogure S., Aoki Y.

Proceedings of SPIE - The International Society for Optical Engineering (Proceedings of SPIE - The International Society for Optical Engineering) 11794 2021

ISSN 0277786X

In this paper, we propose a "Multi-modal Action Segmentation approach"that uses three modalities: (i) video, (ii) audio, (iii) thermal to classify cooking behavior in the kitchen. These 3 modalities are assumed to be features related to cooking. However, there is no public dataset containing these three modalities. Therefore, we built the original dataset and frame-level annotation. We then examined the usefulness of Action Segmentation using multi-modal features. We analyzed the effects of each modality using three evaluation metrics. As a result, the accuracy, edit distance, and F1 value were improved by up to about 1%, 2%, and 8%, respectively, compared to the case when only images were used.

Access to Document (DOI)

Few-Shot Domain Adaptation For Many Class Classification Using Commercial Products

Takahashi R., Sato Y., Furuyama J., Yamaoka M., Tanabiki M., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering 87 ( 1 ) 78 - 82 2021

ISSN 09120289

In order to train a classifier for self checkout system in convenience store at low cost, a few-shot domain adaptation problem has to be solved. Since the system treats a classifier for large number of classes, conventional method of few-shot domain adaptation should be extended for many classes. This paper propose to exploit meta-class information by executing the adaptation on the normal-class level and the meta-class level simultaneously. The proposed method are shown to be effective for improving adaptation accuracy of a classifier for many classes. The results of our ablation study implies that i) the meta-class should be decided by using k-means clustering method rather than clustering manually, and that ii) the ratio between the number of normal-class and the number of meta-class should be fixed.

Access to Document (DOI)

Retrieving and Highlighting Action with Spatiotemporal Reference

Kasai S., Ishikawa Y., Hayashi M., Aoki Y., Hara K., Kataoka H.

Proceedings - International Conference on Image Processing, ICIP (Proceedings - International Conference on Image Processing, ICIP) 2020-October 1401 - 1405 2020.10

ISSN 9781728163956

In this paper, we present a framework thatjointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which relate to the nouns and verbs in captions. Through experiments, we show that our model generates various maps conditioned on different actions, in which conventional visual reasoning methods only go as far as to show a single deterministic saliency map. Also, our model improves retrieval recall over our baseline without alignment by 2-3% on the MSR-VTT dataset.

Access to Document (DOI)

Joint Pedestrian Detection and Risk-level Prediction with Motion-Representation-by-Detection

Kataoka H., Suzuki T., Nakashima K., Satoh Y., Aoki Y.

Proceedings - IEEE International Conference on Robotics and Automation (Proceedings - IEEE International Conference on Robotics and Automation) 1021 - 1027 2020.05

ISSN 9781728173955

The paper presents a pedestrian near-miss detector with temporal analysis that provides both pedestrian detection and risk-level predictions which are demonstrated on a self-collected database. Our work makes three primary contributions: (i) The framework of pedestrian near-miss detection is proposed by providing both a pedestrian detection and risk-level assignment. Specifically, we have created a Pedestrian Near-Miss (PNM) dataset that categorizes traffic near-miss incidents based on their risk levels (high-, low-, and no-risk). Unlike existing databases, our dataset also includes manually localized pedestrian labels as well as a large number of incident-related videos. (ii) Single-Shot MultiBox Detector with Motion Representation (SSD-MR) is implemented to effectively extract motion-based features in a detected pedestrian. (iii) Using the self-collected PNM dataset and SSD-MR, our proposed method achieved +19.38% (on risk-level prediction) and +13.00% (on joint pedestrian detection and risk-level prediction) higher scores than that of the baseline SSD and LSTM. Additionally, the running time of our system is over 50 fps on a graphics processing unit (GPU).

Access to Document (DOI)

FOE-based regularization for optical flow estimation from an in-vehicle event camera

Nagata J., Sekikawa Y., Hara K., Aoki Y.

Electronics and Communications in Japan (Electronics and Communications in Japan) 103 ( 1-4 ) 19 - 25 2020.04

ISSN 19429533

Optical flow estimation from an in-vehicle camera is an important task in automatic driving and advanced driver-assistance systems. However, there is a problem that optical flow estimation is mistakable with high contrast and high speed. Event camera can overcome these situations because it reports only the per-pixel intensity change with high dynamic range and low latency. However, the L1 smoothness regularization in the conventional optical flow estimation method is not suitable for radial optical flow in the driving scene. Therefore, we propose to use the focus of expansion (FOE) for regularization of optical flow estimation in event camera. The FOE is defined as the intersection of the translation vector of the camera and the image plane. The optical flow becomes radial from the FOE excluding the rotational component. Using the property, the optical flow can be regularized in the correct direction in the optimization process. We demonstrated that the optical flow was improved by introducing our regularization using the public dataset.

Access to Document (DOI)

QR-code reconstruction from event data via optimization in code subspace

Nagata J., Sekikawa Y., Hara K., Suzuki T., Yoshimitsu A.

Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020 (Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020) 2113 - 2121 2020.03

ISSN 9781728165530

We propose an image reconstruction method from event data, assuming the target images belong to a prespecified class like QR codes. Instead of solving the reconstruction problem in the image space, we introduce a code space that covers all the noiseless target class images and solves the reconstruction problem on it. This restriction enormously reduces the number of optimizing parameters and makes the reconstruction problem well posed and robust to noise. We demonstrate fast and robust QR-code scanning in difficult, high-speed scenes with industrial high-speed cameras and other reconstruction methods.

Access to Document (DOI)

Action Recognition in Sports Video Considering Location Information

Ichige R., Aoki Y.

Communications in Computer and Information Science (Communications in Computer and Information Science) 1212 CCIS 150 - 164 2020

ISSN 9789811548178

The purpose of this study is to develop a tactics analysis system using image recognition for rugby. With the Rugby World Cup in 2019 and the Tokyo Olympics in 2020, demand for sports video analysis is increasing. Rugby has more complicated play such as dense play than other sports, and the ball is hidden between players, making it difficult to track. By developing a high-precision analysis technology for rugby with few research cases, we thought that it could be used for other sports and industrial fields other than sports. In this research, we propose a method that adds spatial information to time-series information as a new feature. Using the coordinates obtained by projectively transforming the match video onto the bird’s-eye view image, play classification was performed using the player position, the ball position, and the dense area position as feature amounts. Also, in order to further improve the detection accuracy of the boundaries between plays, attention was paid to the positional relationship of each player on the field.

Access to Document (DOI)

　 Previous page - Next page 　