Details of a Researcher - Aoki, Yoshimitsu

Papers - Aoki, Yoshimitsu

Division display 21 - 40 of about 181 ／ All the affair displays >>

A Fast Geometric Regularizer to Mitigate Event Collapse in the Contrast Maximization Framework

Shiba S., Aoki Y., Gallego G.

Advanced Intelligent Systems (Advanced Intelligent Systems) 5 ( 3 ) 2023.03

Event cameras are emerging vision sensors and their advantages are suitable for various applications such as autonomous robots. Contrast maximization (CMax), which provides state-of-the-art accuracy on motion estimation using events, may suffer from an overfitting problem called event collapse. Prior works are computationally expensive or cannot alleviate the overfitting, which undermines the benefits of the CMax framework. A novel, computationally efficient regularizer based on geometric principles to mitigate event collapse is proposed. The experiments show that the proposed regularizer achieves state-of-the-art accuracy results, while its reduced computational complexity makes it two to four times faster than previous approaches. To the best of our knowledge, this regularizer is the only effective solution for event collapse without trading off the runtime. It is hoped that this work opens the door for future applications that unlocks the advantages of event cameras. Project page: https://github.com/tub-rip/event_collapse.

Access to Document (DOI)

Efficient Transformer-Based Compressed Video Modeling via Informative Patch Selection

Suzuki T., Aoki Y.

Sensors (Sensors) 23 ( 1 ) 2023.01

ISSN 14248220

Recently, Transformer-based video recognition models have achieved state-of-the-art results on major video recognition benchmarks. However, their high inference cost significantly limits research speed and practical use. In video compression, methods considering small motions and residuals that are less informative and assigning short code lengths to them (e.g., MPEG4) have successfully reduced the redundancy of videos. Inspired by this idea, we propose Informative Patch Selection (IPS), which efficiently reduces the inference cost by excluding redundant patches from the input of the Transformer-based video model. The redundancy of each patch is calculated from motions and residuals obtained while decoding a compressed video. The proposed method is simple and effective in that it can dynamically reduce the inference cost depending on the input without any policy model or additional loss term. Extensive experiments on action recognition demonstrated that our method could significantly improve the trade-off between the accuracy and inference cost of the Transformer-based video model. Although the method does not require any policy model or additional loss term, its performance approaches that of existing methods that do require them.

Access to Document (DOI)

Listening Human Behavior: 3D Human Pose Estimation with Acoustic Signals

Shibata Y., Kawashima Y., Isogawa M., Irie G., Kimura A., Aoki Y.

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition) 2023-June 13323 - 13332 2023

ISSN 10636919

Given only acoustic signals without any high-level information, such as voices or sounds of scenes/actions, how much can we infer about the behavior of humans? Unlike existing methods, which suffer from privacy issues because they use signals that include human speech or the sounds of specific actions, we explore how low-level acoustic signals can provide enough clues to estimate 3D human poses by active acoustic sensing with a single pair of microphones and loudspeakers (see Fig. 1). This is a challenging task since sound is much more diffractive than other signals and therefore covers up the shape of objects in a scene. Accordingly, we introduce a framework that encodes multichannel audio features into 3D human poses. Aiming to capture subtle sound changes to reveal detailed pose information, we explicitly extract phase features from the acoustic signals together with typical spectrum features and feed them into our human pose estimation network. Also, we show that reflected or diffracted sounds are easily influenced by subjects' physique differences e.g., height and muscularity, which deteriorates prediction accuracy. We reduce these gaps by using a subject discriminator to improve accuracy. Our experiments suggest that with the use of only low-dimensional acoustic information, our method outperforms baseline methods. The datasets and codes used in this project will be publicly available.

Access to Document (DOI)

Unsupervised Domain-Adaptive Semantic Segmentation with Uncertainty Loss

Kawano Y., Nota Y., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 89 ( 12 ) 921 - 925 2023

ISSN 09120289

In recent years, deep learning models have achieved great success in many fields, but their training relies on large labeled data. Annotation to obtain labeled samples is challenging, time-consuming, and expensive, and has been a major challenge in deep learning. In this study, we propose a method for unsupervised domain adaptation that can reduce annotation cost in terms of accuracy and time compared to conventional methods by using the uncertainty of the model in semantic segmentation. Unsupervised domain adaptation aims to adapt models trained on synthetic data to real-world data without the need for costly annotation of real images. Since the training data are images and correct labels created by the game engine, there is no need to manually annotate them. However, if the source (synthetic) data is trained using conventional supervised learning, the performance is significantly degraded because the domain is different from the target (real image) data.This study closes the domain gap between source and target by calculating the uncertainty in the target data from the model output on a pixel-by-pixel basis and minimizing this uncertainty as loss. Adding this uncertainty loss to conventional unsupervised domain adaptation methods results in a model that is more robust to the target domain and achieves state-of-the-art results.

Access to Document (DOI)

Preliminary Action Recognition

Kawano Y., Nagasaki Y., Hara K., Aoki Y., Kataoka H.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 89 ( 12 ) 915 - 920 2023

ISSN 09120289

In this paper, we propose preliminary action recognition which is an action anticipation method that can be used in real-time. Preliminary action recognition as a solution to action anticipation, which is the task of detecting the action of a person before the person performs the action. We define preliminary action class before action class that is anticipated by recognizing the preliminary actions specific to the action before it occurs, just as humans do. Conventional action anticipation methods have two problems for real-time use. First, it requires 10 to 20 seconds of video to anticipate 1 second ahead, and it can only anticipate every 10 seconds. Second, the video input in training always switches actions 1 second after the end of the video. This problem setting can be a leakage because there is no action break in real-time operation. Proposed method solves this problem and makes it possible to operate action prediction in real-time. In addition, since action classes and preliminary action classes tend to share a lot of similarities, it is difficult to distinguish between them. We propose a model that can discriminate the subtle differences in motion by increasing the latent distance between adjacent classes in terms of time.

Access to Document (DOI)

Event Collapse in Contrast Maximization Frameworks

Shiba S., Aoki Y., Gallego G.

Sensors (Sensors) 22 ( 14 ) 2022.07

ISSN 14248220

Contrast maximization (CMax) is a framework that provides state-of-the-art results on several event-based computer vision tasks, such as ego-motion or optical flow estimation. However, it may suffer from a problem called event collapse, which is an undesired solution where events are warped into too few pixels. As prior works have largely ignored the issue or proposed workarounds, it is imperative to analyze this phenomenon in detail. Our work demonstrates event collapse in its simplest form and proposes collapse metrics by using first principles of space–time deformation based on differential geometry and physics. We experimentally show on publicly available datasets that the proposed metrics mitigate event collapse and do not harm well-posed warps. To the best of our knowledge, regularizers based on the proposed metrics are the only effective solution against event collapse in the experimental settings considered, compared with other methods. We hope that this work inspires further research to tackle more complex warp models.

Access to Document (DOI)

Non-Deep Active Learning for Deep Neural Networks

Kawano Y., Nota Y., Mochizuki R., Aoki Y.

Sensors (Sensors) 22 ( 14 ) 2022.07

ISSN 14248220

One way to improve annotation efficiency is active learning. The goal of active learning is to select images from many unlabeled images, where labeling will improve the accuracy of the machine learning model the most. To select the most informative unlabeled images, conventional methods use deep neural networks with a large number of computation nodes and long computation time, but we propose a non-deep neural network method that does not require any additional training for unlabeled image selection. The proposed method trains a task model on labeled images, and then the model predicts unlabeled images. Based on this prediction, an uncertainty indicator is generated for each unlabeled image. Images with a high uncertainty index are considered to have a high information content, and are selected for annotation. Our proposed method is based on a very simple and powerful idea: select samples near the decision boundary of the model. Experimental results on multiple datasets show that the proposed method achieves higher accuracy than conventional active learning methods on multiple tasks and up to 14 times faster execution time from (Formula presented.) s to (Formula presented.) s. The proposed method outperforms the current SoTA method by 1% accuracy on CIFAR-10.

Access to Document (DOI)

Supporting collective physical activities by interactive floor projection in a special-needs school setting

Oki M., Akizuki S., Bourreau B., Takahashi I., Aoki Y., Yamamoto J., Suzuki K.

International Journal of Child-Computer Interaction (International Journal of Child-Computer Interaction) 32 2022.06

ISSN 22128689

This paper presents an algorithm to provide floor projection feedback according to the local distance and density of individuals. It is realized by a large-space floor projection system with a feedback function based on human tracking with laser ranging image sensors. The purpose is to support the cognition of spatial–temporal structures of groups of adolescents with neurodevelopmental disorders (NDs) that are conducting organized physical activity (PA). Observation and evaluation of behavioral changes in adolescents with NDs, when they were active with or without the floor projection based on the proposed algorithm, were conducted to validate its effectiveness. We observed that the proposed algorithm can be implemented in different organized PAs. It had the effect to help individuals in a behavior to keep a close distance to each other as a group rather than to keep the same distance apart from each other while walking.

Access to Document (DOI)

Fast Event-Based Optical Flow Estimation by Triplet Matching

Shiba S., Aoki Y., Gallego G.

IEEE Signal Processing Letters (IEEE Signal Processing Letters) 29 2712 - 2716 2022

ISSN 10709908

Event cameras are novel bio-inspired sensors that offer advantages over traditional cameras (low latency, high dynamic range, low power, etc.). Optical flow estimation methods that work on packets of events trade off speed for accuracy, while event-by-event (incremental) methods have strong assumptions and have not been tested on common benchmarks that quantify progress in the field. Towards applications on resource-constrained devices, it is important to develop optical flow algorithms that are fast, light-weight and accurate. This work leverages insights from neuroscience, and proposes a novel optical flow estimation scheme based on triplet matching. The experiments on publicly available benchmarks demonstrate its capability to handle complex scenes with comparable results as prior packet-based algorithms. In addition, the proposed method achieves the fastest execution time (>10 kHz) on standard CPUs as it requires only three events in estimation. We hope that our research opens the door to real-time, incremental motion estimation methods and applications in real-world scenarios.

Access to Document (DOI)

Detection of Osteoarthritis from Multimodal Hand Data

Andrade Guerreiro J.J., Aoki Y., Saito S., Suzuki K.

Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS (Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS) 2022-July 3607 - 3610 2022

ISSN 1557170X

Osteoarthritis (OA) describes a degenerative joint disorder that is prevalent among older people and typically results in swollen and inflamed joints. The aim of this paper is to develop a method using images, videos and thermal data of 100 patients taken at Keio University Hospital to detect OA in hands. By using hand pose estimation on the video data, joint angles can be calculated and subsequently transformed into feature vectors. For the thermal and RGB images, hand keypoint detectors were trained to identify and crop the appropriate joints within the images. The resulting extracted features are combined and further trained on Support Vector Machines and Convolutional Neural Networks to obtain the final binary classification for each joint. While the proposed method generally shows favorable accuracy and F1-scores on the Proximal (PIP) and Distal Interphalangeal (DIP) joints, the performance on the Metacarpophalangeal (MCP) joints is limited by the low occurrence of affected joints in the dataset. We further compare the different modalities and found that, apart from the combined approach, using video data provides the best results. Clinical Relevance - The proposed method shows promising first results for the usage of visual and thermal data in combination with machine learning in order to detect OA

Access to Document (DOI)

DOCUMENT SHADOW REMOVAL WITH FOREGROUND DETECTION LEARNING FROM FULLY SYNTHETIC IMAGES

Matsuo Y., Akimoto N., Aoki Y.

Proceedings - International Conference on Image Processing, ICIP (Proceedings - International Conference on Image Processing, ICIP) 1656 - 1660 2022

ISSN 15224880

Shadow removal for document images is a major task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large-scale and diverse dataset is laborious and remains a great challenge. Thus, only small real datasets are available. To create relatively large datasets, a graphic renderer has been used to synthesize shadows, nonetheless, it is still necessary to capture real documents. Thus, the number of unique documents is limited, which negatively affects a network's performance. In this paper, we present a large-scale and diverse dataset called fully synthetic document shadow removal dataset (FSDSRD) that does not require capturing documents. The experiments showed that the networks (pre-)trained on FSDSRD provided better results than networks trained only on real datasets. Additionally, because foreground maps are available in our dataset, we leveraged them during training for multitask learning, which provided noticeable improvements. The code is available at: https://github.com/IsHYuhi/DSRFGD.

Access to Document (DOI)

Detecting Spinal Alignment Through Depth Image

Ishibashi R., Kameshima H., Nishio Y., Sato Y., Watanabe K., Aoki Y.

Proceedings - 16th International Conference on Signal-Image Technology and Internet-Based Systems, SITIS 2022 (Proceedings - 16th International Conference on Signal-Image Technology and Internet-Based Systems, SITIS 2022) 330 - 334 2022

Adolescent Idiopathic Scoliosis (AIS) is a common form of scoliosis found among adolescents. The abnormal spinal curvature along with twisting of the vertebrae requires early detection before the vertebrae alignment worsens. Furthermore, surgical treatment is crucial for those with severe conditions. In Japan, annual scoliosis screening is conducted during school health checkups. These screening procedures often involve an initial screening without the use of ionizing radiation (such as X-rays) and an X-ray screening later for those suspected with scoliosis. Currently, X-ray examinations produce the most accurate results for detecting the spinal position. However, noninvasive alternative methods, such as Moire imaging, provide satisfactory initial results. Unfortunately, the production of Moire cameras have halted in recent years. This paper proposes an alternative, non-invasive and non-ionizing radioactive method to detect spinal alignment. Depth images provide an extra dimension of information compared to RGB images. The proposed method utilizes this extra dimension of depth values to create different types of images that are then trained using Convolutional Neural Networks to predict the spinal alignment. Results indicate that Moire images reproduced from depth images produce the best spinal alignment. However, other types of images derived from the depth image also have higher accuracy when compared with the depth image itself.

Access to Document (DOI)

Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation

Akimoto N., Matsuo Y., Aoki Y.

ISSN 10636919

We address the problem of generating a 360-degree image from a single image with a narrow field of view by estimating its surroundings. Previous methods suffered from overfitting to the training resolution and deterministic generation. This paper proposes a completion method using a transformer for scene modeling and novel methods to improve the properties of a 360-degree image on the output image. Specifically, we use CompletionNets with a transformer to perform diverse completions and Adjust-mentNet to match color, stitching, and resolution with an input image, enabling inference at any resolution. To improve the properties of a 360-degree image on an output image, we also propose WS-perceptual loss and circular inference. Thorough experiments show that our method out-performs state-of-the-art (SOTA) methods both qualitatively and quantitatively. For example, compared to SOTA methods, our method completes images 16 times larger in resolution and achieves 1.7 times lower Fréchet inception distance (FID). Furthermore, we propose a pipeline that uses the completion results for lighting and background of 3DCG scenes. Our plausible background completion enables perceptually natural results in the application of inserting virtual objects with specular surfaces.

Access to Document (DOI)

Determinant analysis and developing evaluation indicators of grade of execution score of double axel jump in figure skating

Hirosawa S., Watanabe M., Aoki Y.

Journal of Sports Sciences (Journal of Sports Sciences) 40 ( 4 ) 470 - 481 2022

ISSN 02640414

A figure skating jump score is determined by the sum of the base value based on the difficulty and grade of execution (GOE) that indicates the performance quality. Therefore, performing a high-quality jump to obtain a high GOE is essential to win a competition. However, the relationship between the GOE and kinematic parameters remains unclear. We analysed the horizontal distance, vertical height, and landing speed of double axel jumps in the Ladies’ Short Program at the 2019 World Championships. The highest GOE group had significantly larger horizontal distances than the middle and lower groups, while the landing speed and vertical height were not significantly different. A principal component regression analysis was conducted to clarify the contrast between the three variables affecting the GOE. The results showed that greater horizontal distance and landing speed compared to vertical height (component 1) and greater horizontal distance compared to landing speed (component 3) contributed to higher GOE. We divided skaters into four clusters using these two components and provided general GOE acquisition strategies for each cluster. Finally, to apply our results to the industry, we proposed two new evaluation indicators which are highly correlated with the two components and easy to interpret.

Access to Document (DOI)

Improving Annotation Efficiency through Uncertain Sample Selection in Active Learning

Kawano Y., Nota Y., Mochizuki R., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 88 ( 2 ) 211 - 216 2022

ISSN 09120289

Active learning refers to label-efficient algorithms that use the most representative samples for labeling when creating training data. In this paper, we propose a model that derives the most informative unlabeled samples from the output of a task model. The tasks arc a classification problem, multi-label classification and a semantic segmentation problem. The model consists of an uncertainty indicator generator and a task model. After training the task model with labeled samples, the model predicts unlabeled samples, and based on the prediction results, the uncertainty indicator generator outputs an uncertainty indicator for each unlabeled sample. Samples with a higher uncertainty indicator are considered to be more informative and selected. As a result of experiments using multiple datasets, our model achieved better accuracy than conventional active learning methods and reduced execution time by a factor of 10.

Access to Document (DOI)

Secrets of Event-Based Optical Flow

Shiba S., Aoki Y., Gallego G.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)) 13678 LNCS 628 - 645 2022

ISSN 03029743

Event cameras respond to scene dynamics and offer advantages to estimate motion. Following recent image-based deep-learning achievements, optical flow estimation methods for event cameras have rushed to combine those image-based methods with event data. However, it requires several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. We investigate key elements: how to design the objective function to prevent overfitting, how to warp events to deal better with occlusions, and how to improve convergence with multi-scale raw events. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark. Moreover, our method allows us to expose the issues of the ground truth flow in those benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Our code is available at https://github.com/tub-rip/event_based_optical_flow.

Access to Document (DOI)

Temporal Cross-Modal Attention for Audio-Visual Event Localization

Nagasaki Y., Hayashi M., Kaneko N., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 88 ( 3 ) 263 - 268 2022

ISSN 09120289

In this paper, we propose a new method for audio-visual event localization 1) to find the corresponding segment between audio and visual event. While previous methods use Long Short-Term Memory (LSTM) networks to extract temporal features, recurrent neural networks like LSTM are not able to precisely learn long-term features. Thus, we propose a Temporal Cross-Modal Attention (TCMA) module, which extract temporal features more precisely from the two modalities. Inspired by the success of attention modules in capturing long-term features, we introduce TCMA, which incorporates self-attention. Finally, we were able to localize audio-visual event precisely and achieved a higher accuracy than the previous works.

Access to Document (DOI)

Self-Supervised Learning of Inlier Events for Event-based Optical Flow

Nagata J., Aoki Y.

BMVC 2022 - 33rd British Machine Vision Conference Proceedings (BMVC 2022 - 33rd British Machine Vision Conference Proceedings) 2022

Event cameras asynchronously report per-pixel intensity changes with high temporal resolution. Their sparse and temporally precise nature is well suited for fast visual flow estimation. The normal optical flow can be estimated by fitting a plane to spatio-temporal events. However, least-squares plane fitting suffers from outliers due to the significant noise of events. Existing methods involve 1) iterative outlier rejection or 2) goodness-of-fit rejection after single-shot planar fitting from greedily selected spatially neighboring events. In contrast to these methods, we propose a method of selecting the events supporting a plane before performing a fitting, using the inlier probability from a lightweight neural network that captures both global and local structures. During inference, single-shot planar fitting is performed from only events with a higher inlier probability. We model each event selection by a Bernoulli distribution with the inlier probability and train the network to maximize the inlier count while sampling in a self-supervised manner. We verify that our event selection improves the accuracy of optical flow estimation with publicly available real data.

Optical flow estimation by matching time surface with event-based cameras

Nagata J., Sekikawa Y., Aoki Y.

Sensors (Switzerland) (Sensors (Switzerland)) 21 ( 4 ) 1 - 14 2021.02

ISSN 14248220

In this work, we propose a novel method of estimating optical flow from event-based cameras by matching the time surface of events. The proposed loss function measures the timestamp consistency between the time surface formed by the latest timestamp of each pixel and the one that is slightly shifted in time. This makes it possible to estimate dense optical flows with high accuracy without restoring luminance or additional sensor information. In the experiment, we show that the gradient was more correct and the loss landscape was more stable than the variance loss in the motion compensation approach. In addition, we show that the optical flow can be estimated with high accuracy by optimization with L1 smoothness regularization using publicly available datasets.

Access to Document (DOI)

Alleviating over-segmentation errors by detecting action boundaries

Ishikawa Y., Kasai S., Aoki Y., Kataoka H.

Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021 (Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021) 2321 - 2330 2021.01

We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our contributions are three-fold: (i) We propose a framework for temporal action segmentation, the ASRF, which divides temporal action segmentation into frame-wise action classification and action boundary regression. Our framework refines frame-level hypotheses of action classes using predicted action boundaries. (ii) We propose a loss function for smoothing the transition of action probabilities, and analyze combinations of various loss functions for temporal action segmentation. (iii) Our framework outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in terms of segmental edit distance and up to 16.1% in terms of segmental F1 score. Our code is publicly available1.

Access to Document (DOI)

　 Previous page - Next page 　