Details of a Researcher - Aoki, Yoshimitsu

Papers - Aoki, Yoshimitsu

Division display >> ／ All the affair displays 1 - 181 of about 181

DynamicVLN: Incorporating Dynamics into Vision-and-Language Navigation Scenarios

Sun Y., Qiu Y., Aoki Y.

Sensors 25 ( 2 ) 2025.01

Traditional Vision-and-Language Navigation (VLN) tasks require an agent to navigate static environments using natural language instructions. However, real-world road conditions such as vehicle movements, traffic signal fluctuations, pedestrian activity, and weather variations are dynamic and continually changing. These factors significantly impact an agent’s decision-making ability, underscoring the limitations of current VLN models, which do not accurately reflect the complexities of real-world navigation. To bridge this gap, we propose a novel task called Dynamic Vision-and-Language Navigation (DynamicVLN), incorporating various dynamic scenarios to enhance the agent’s decision-making abilities and adaptability. By redefining the VLN task, we emphasize that a robust and generalizable agent should not rely solely on predefined instructions but must also demonstrate reasoning skills and adaptability to unforeseen events. Specifically, we have designed ten scenarios that simulate the challenges of dynamic navigation and developed a dedicated dataset of 11,261 instances using the CARLA simulator (ver.0.9.13) and large language model to provide realistic training conditions. Additionally, we introduce a baseline model that integrates advanced perception and decision-making modules, enabling effective navigation and interpretation of the complexities of dynamic road conditions. This model showcases the ability to follow natural language instructions while dynamically adapting to environmental cues. Our approach establishes a benchmark for developing agents capable of functioning in real-world, dynamic environments and extending beyond the limitations of static VLN tasks to more practical and versatile applications.

Access to Document (DOI)

A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans

Mori E., Qiu Y., Kataoka H., Aoki Y.

Sensors 25 ( 2 ) 2025.01

In recent years, advancements in the interaction and collaboration between humans and have garnered significant attention. Social intelligence plays a crucial role in facilitating natural interactions and seamless communication between humans and Artificial Intelligence (AI). To assess AI’s ability to understand human interactions and the components necessary for such comprehension, datasets like Social-IQ have been developed. However, these datasets often rely on a simplistic question-and-answer format and lack justifications for the provided answers. Furthermore, existing methods typically produce direct answers by selecting from predefined choices without generating intermediate outputs, which hampers interpretability and reliability. To address these limitations, we conducted a comprehensive evaluation of AI methods on a video-based Question Answering (QA) benchmark focused on human interactions, leveraging additional annotations related to human responses. Our analysis highlights significant differences between human and AI response patterns and underscores critical shortcomings in current benchmarks. We anticipate that these findings will guide the creation of more advanced datasets and represent an important step toward achieving natural communication between humans and AI.

Access to Document (DOI)

Data Collection-Free Masked Video Modeling

Ishikawa Y., Kondo M., Aoki Y.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 15069 LNCS 37 - 56 2025

ISSN 03029743

Pre-training video transformers generally requires a large amount of data, presenting significant challenges in terms of data collection costs and concerns related to privacy, licensing, and inherent biases. Synthesizing data is one of the promising ways to solve these issues, yet pre-training solely on synthetic data has its own challenges. In this paper, we introduce an effective self-supervised learning framework for videos that leverages readily available and less costly static images. Specifically, we define the Pseudo Motion Generator (PMG) module that recursively applies image transformations to generate pseudo-motion videos from images. These pseudo-motion videos are then leveraged in masked video modeling. Our approach is applicable to synthetic images as well, thus entirely freeing video pre-training from data collection costs and other concerns in real data. Through experiments in action recognition tasks, we demonstrate that this framework allows effective learning of spatio-temporal features through pseudo-motion videos, significantly improving over existing methods which also use static images and partially outperforming those using both real and synthetic videos. These results uncover fragments of what video transformers learn through masked video modeling.

Access to Document (DOI)

Rethinking Image Super-Resolution from Training Data Perspectives

Ohtani G., Tadokoro R., Yamada R., Asano Y.M., Laina I., Rupprecht C., Inoue N., Yokota R., Kataoka H., Aoki Y.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 15075 LNCS 19 - 36 2025

ISSN 03029743

In this work, we investigate the understudied effect of the training data used for image super-resolution (SR). Most commonly, novel SR methods are developed and benchmarked on common training datasets such as DIV2K and DF2K. However, we investigate and rethink the training data from the perspectives of diversity and quality, thereby addressing the question of “How important is SR training for SR models?”. To this end, we propose an automated image evaluation pipeline. With this, we stratify existing high-resolution image datasets and larger-scale image datasets such as ImageNet and PASS to compare their performances. We find that datasets with (i) low compression artifacts, (ii) high within-image diversity as judged by the number of different objects, and (iii) a large number of images from ImageNet or PASS all positively affect SR performance. We hope that the proposed simple-yet-effective dataset curation pipeline will inform the construction of SR datasets in the future and yield overall better models. Code is available at: https://github.com/gohtanii/DiverSeg-dataset.

Access to Document (DOI)

Unsupervised Metric Learning for Expressing Color and Shape Information to Uncover Abstract Connections within Image Datasets

Obikane S., Tagawa H., Aoki Y.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 15321 LNCS 15 - 30 2025

ISSN 03029743

In this research, we propose a novel approach using unsupervised metric learning tailored to datasets characterized by complex similarities and connections, such as those found in paintings and makeup, which are challenging to express linguistically. These datasets often present the difficulty of adequately analyzing data points due to the intricate interplay of defining elements, a limitation of traditional labeling methods. Additionally, the high degree of specialization required makes annotation significantly costly. Unsupervised metric learning emerges as a powerful method for extracting more cost-effective features and for the comprehensive analysis of these datasets. Expanding upon previous research that utilized style transfer models, our study further explores feature design, specifically focusing on extracting detailed information about critical aspects of similarity assessment, such as color and shape. Our model adeptly incorporates visual information, unveiling the hidden abstract connections within datasets. We validated our approach using a dataset of Ukiyo-e, a genre of Japanese painting, and achieved accuracy comparable to supervised learning models. This research opens up new possibilities for the analysis of complex image datasets with abstract relational depth, fostering a deeper understanding of the data.

Access to Document (DOI)

RetinaViT: Efficient Visual Backbone for Online Video Streams

Suzuki T., Aoki Y.

Sensors 24 ( 17 ) 2024.09

In online video understanding, which has a wide range of real-world applications, inference speed is crucial. Many approaches involve frame-level visual feature extraction, which often represents the biggest bottleneck. We propose RetinaViT, an efficient method for extracting frame-level visual features in an online video stream, aiming to fundamentally enhance the efficiency of online video understanding tasks. RetinaViT is composed of efficiently approximated Transformer blocks that only take changed tokens (event tokens) as queries and reuse the already processed tokens from the previous timestep for the others. Furthermore, we restrict keys and values to the spatial neighborhoods of event tokens to further improve efficiency. RetinaViT involves tuning multiple parameters, which we determine through a multi-step process. During model training, we randomly vary these parameters and then perform black-box optimization to maximize accuracy and efficiency on the pre-trained model. We conducted extensive experiments on various online video recognition tasks, including action recognition, pose estimation, and object segmentation, validating the effectiveness of each component in RetinaViT and demonstrating improvements in the speed/accuracy trade-off compared to baselines. In particular, for action recognition, RetinaViT built on ViT-B16 reduces inference time by approximately 61.9% on the CPU and 50.8% on the GPU, while achieving slight accuracy improvements rather than degradation.

Access to Document (DOI)

Proto-Adapter: Efficient Training-Free CLIP-Adapter for Few-Shot Image Classification

Kato N., Nota Y., Aoki Y.

Sensors (Sensors) 24 ( 11 ) 2024.06

Large vision-language models, such as Contrastive Vision-Language Pre-training (CLIP), pre-trained on large-scale image–text datasets, have demonstrated robust zero-shot transfer capabilities across various downstream tasks. To further enhance the few-shot recognition performance of CLIP, Tip-Adapter augments the CLIP model with an adapter that incorporates a key-value cache model constructed from the few-shot training set. This approach enables training-free adaptation and has shown significant improvements in few-shot recognition, especially with additional fine-tuning. However, the size of the adapter increases in proportion to the number of training samples, making it difficult to deploy in practical applications. In this paper, we propose a novel CLIP adaptation method, named Proto-Adapter, which employs a single-layer adapter of constant size regardless of the amount of training data and even outperforms Tip-Adapter. Proto-Adapter constructs the adapter’s weights based on prototype representations for each class. By aggregating the features of the training samples, it successfully reduces the size of the adapter without compromising performance. Moreover, the performance of the model can be further enhanced by fine-tuning the adapter’s weights using a distance margin penalty, which imposes additional inter-class discrepancy to the output logits. We posit that this training scheme allows us to obtain a model with a discriminative decision boundary even when trained with a limited amount of data. We demonstrate the effectiveness of the proposed method through extensive experiments of few-shot classification on diverse datasets.

Access to Document (DOI)

Event-Based Background-Oriented Schlieren

Shiba S., Hamann F., Aoki Y., Gallego G.

IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE Transactions on Pattern Analysis and Machine Intelligence) 46 ( 4 ) 2011 - 2026 2024.04

Joint Work, ISSN 01628828

Schlieren imaging is an optical technique to observe the flow of transparent media, such as air or water, without any particle seeding. However, conventional frame-based techniques require both high spatial and temporal resolution cameras, which impose bright illumination and expensive computation limitations. Event cameras offer potential advantages (high dynamic range, high temporal resolution, and data efficiency) to overcome such limitations due to their bio-inspired sensing principle. This article presents a novel technique for perceiving air convection using events and frames by providing the first theoretical analysis that connects event data and schlieren. We formulate the problem as a variational optimization one combining the linearized event generation model with a physically-motivated parameterization that estimates the temporal derivative of the air density. The experiments with accurately aligned frame- and event camera data reveal that the proposed method enables event cameras to obtain on par results with existing frame-based optical flow techniques. Moreover, the proposed method works under dark conditions where frame-based schlieren fails, and also enables slow-motion analysis by leveraging the event camera's advantages. Our work pioneers and opens a new stack of event camera applications, as we publish the source code as well as the first schlieren dataset with high-quality frame and event data.

Access to Document (DOI)

Synthetic Document Images with Diverse Shadows for Deep Shadow Removal Networks

Matsuo Y., Aoki Y.

Sensors (Sensors) 24 ( 2 ) 2024.01

ISSN 14248220

Shadow removal for document images is an essential task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large, diverse dataset for document shadow removal takes time and effort. Thus, only small real datasets are available. Graphic renderers have been used to synthesize shadows to create relatively large datasets. However, the limited number of unique documents and the limited lighting environments adversely affect the network performance. This paper presents a large-scale, diverse dataset called the Synthetic Document with Diverse Shadows (SynDocDS) dataset. The SynDocDS comprises rendered images with diverse shadows augmented by a physics-based illumination model, which can be utilized to obtain a more robust and high-performance deep shadow removal network. In this paper, we further propose a Dual Shadow Fusion Network (DSFN). Unlike natural images, document images often have constant background colors requiring a high understanding of global color features for training a deep shadow removal network. The DSFN has a high global color comprehension and understanding of shadow regions and merges shadow attentions and features efficiently. We conduct experiments on three publicly available datasets, the OSR, Kligler’s, and Jung’s datasets, to validate our proposed method’s effectiveness. In comparison to training on existing synthetic datasets, our model training on the SynDocDS dataset achieves an enhancement in the PSNR and SSIM, increasing them from 23.00 dB to 25.70 dB and 0.959 to 0.971 on average. In addition, the experiments demonstrated that our DSFN clearly outperformed other networks across multiple metrics, including the PSNR, the SSIM, and its impact on OCR performance.

Access to Document (DOI)

Secrets of Event-based Optical Flow, Depth and Ego-motion Estimation by Contrast Maximization

Shiba S., Klose Y., Aoki Y., Gallego G.

IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE Transactions on Pattern Analysis and Machine Intelligence) 46 ( 12 ) 7742 - 7759 2024

ISSN 01628828

Event cameras respond to scene dynamics and provide signals naturally suitable for motion estimation with advantages, such as high dynamic range. The emerging field of event-based vision motivates a revisit of fundamental computer vision tasks related to motion, such as optical flow and depth estimation. However, state-of-the-art event-based optical flow methods tend to originate in frame-based deep-learning methods, which require several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate dense optical flow, depth, and ego-motion from events alone. The proposed method sensibly models the space-time properties of event data and tackles the event alignment problem. It designs the objective function to prevent overfitting, deals better with occlusions, and improves convergence using a multi-scale approach. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark and is competitive on the DSEC benchmark. Moreover, it allows us to simultaneously estimate dense depth and ego-motion, exposes the limitations of current flow benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Along with various downstream applications shown, we hope the proposed method becomes a cornerstone on event-based motion-related tasks. Code is available at <uri>https://github.com/tub-rip/event_based_optical_flow</uri>

Access to Document (DOI)

Improving Perceptual Loss with CLIP for Super-Resolution

Ohtani G., Kataoka H., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 90 ( 2 ) 217 - 223 2024

ISSN 09120289

Perceptual loss, calculated by VGG network pre-trained on ImageNet, has been widely employed in the past for super-resolution tasks, enabling the generation of photo-realistic images. However, it has been reported that grid-like artifacts frequently appear in the generated images. To address this problem, we consider that large-scale pre-trained models can make significant contributions to super-resolution across different scenes. In particular, by combining language, those models can exhibit a strong capability to comprehend complex scenes, potentially enhancing super-resolution performance. Therefore, this paper proposes new perceptual loss with Contrastive Language-Image Pre-training (CLIP) based on Vision Transformer (ViT) instead of VGG network. The results demonstrate our proposed perceptual loss can generate photorealistic images without grid-like artifacts.

Access to Document (DOI)

TAG: Guidance-Free Open-Vocabulary Semantic Segmentation

Kawano Y., Aoki Y.

IEEE Access (IEEE Access) 12 88322 - 88331 2024

Semantic segmentation is a crucial task in computer vision, where each pixel in an image is classified into a category. However, traditional methods face significant challenges, including the need for pixel-level annotations and extensive training. Furthermore, because supervised learning uses a limited set of predefined categories, models typically struggle with rare classes and cannot recognize new ones. Unsupervised and open-vocabulary segmentation, proposed to tackle these issues, faces challenges, including the inability to assign specific class labels to clusters and the necessity of user-provided text queries for guidance. In this context, we propose a novel approach, TAG which achieves Training, Annotation, and Guidance-free open-vocabulary semantic segmentation. TAG utilizes pre-trained models such as CLIP and DINO to segment images into meaningful categories without additional training or dense annotations. It retrieves class labels from an external database, providing flexibility to adapt to new scenarios. Our TAG achieves state-of-the-art results on PascalVOC, PascalContext and ADE20K for open-vocabulary segmentation without given class names, i.e. improvement of +15.3 mIoU on PascalVOC.

Access to Document (DOI)

Guided by the Way: The Role of On-the-route Objects and Scene Text in Enhancing Outdoor Navigation

Sun Y., Qiu Y., Aoki Y., Kataoka H.

Proceedings - IEEE International Conference on Robotics and Automation 5198 - 5204 2024

ISSN 10504729

In outdoor environments, Vision-and-Language Navigation (VLN) requires an agent to rely on multi-modal cues from real-world urban environments and natural language instructions. While existing outdoor VLN models predict actions using a combination of panorama and instruction features, this approach ignores objects in the environment and learns data bias to fail navigation. According to our preliminary findings, most instances of navigation failure in previous models were due to turning or stopping at the wrong place. In contrast, humans intuitively frequently use identifiable objects or store names as reference landmarks, ensuring accurate turns and stops, especially in unfamiliar places. To address this insight gap, we propose an Object-Attention VLN (OAVLN) model that helps the agent focus on relevant objects during training and understand the environment better. Our model outperforms previous methods in all evaluation metrics under both seen and unseen scenarios on two existing benchmark datasets, Touchdown and map2seq.

Access to Document (DOI)

MaskDiffusion: Exploiting Pre-trained Diffusion Models for Semantic Segmentation

Kawano Y., Aoki Y.

IEEE Access 12 127283 - 127293 2024

Semantic segmentation is essential in computer vision for various applications, yet traditional approaches face significant challenges, including the high cost of annotation and extensive training for supervised learning. Additionally, due to the limited predefined categories in supervised learning, models typically struggle with infrequent classes and are unable to predict novel classes. To address these limitations, we propose MaskDiffusion, an innovative approach that leverages pretrained frozen Stable Diffusion to achieve open-vocabulary semantic segmentation without the need for additional training or annotation, leading to improved performance compared to similar methods. We also demonstrate the superior performance of MaskDiffusion in handling open vocabularies, including fine-grained and proper noun-based categories, thus expanding the scope of segmentation applications. Overall, our MaskDiffusion shows significant qualitative and quantitative improvements in contrast to other comparable unsupervised segmentation methods, i.e. on the Potsdam dataset (+10.5 mIoU compared to GEM) and COCO-Stuff (+14.8 mIoU compared to DiffSeg). All code and data are released at https://github.com/Valkyrja3607/MaskDiffusion.

Access to Document (DOI)

PCT: Perspective Cue Training Framework for Multi-Camera BEV Segmentation

Ishikawa H., Iida T., Konishi Y., Aoki Y.

IEEE International Conference on Intelligent Robots and Systems 13253 - 13260 2024

ISSN 21530858

Generating annotations for bird's-eye-view (BEV) segmentation presents significant challenges due to the scenes' complexity and the high manual annotation cost. In this work, we address these challenges by leveraging the abundance of unlabeled data available. We propose the Perspective Cue Training (PCT) framework, a novel training framework that utilizes pseudo-labels generated from unlabeled perspective images using publicly available semantic segmentation models trained on large street-view datasets. PCT applies a perspective view task head to the image encoder shared with the BEV segmentation head, effectively utilizing the unlabeled data to be trained with the generated pseudo-labels. Since image encoders are present in nearly all camera-based BEV segmentation architectures, PCT is flexible and applicable to various existing BEV architectures. In this paper, we applied PCT for semi-supervised learning (SSL) and unsupervised domain adaptation (UDA). Additionally, we introduce strong input perturbation through Camera Dropout (CamDrop) and feature perturbation via BEV Feature Dropout (BFD), which are crucial for enhancing SSL capabilities using our teacher-student framework. Our comprehensive approach is simple and flexible but yields significant improvements over various baselines for SSL and UDA, achieving competitive performances even against the current state-of-the-art.

Access to Document (DOI)

3D Human Scan with A Moving Event Camera

Kohyama K., Shiba S., Aoki Y.

IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 5586 - 5596 2024

ISSN 21607508

Capturing the 3D human body is one of the important tasks in computer vision with a wide range of applications such as virtual reality and sports analysis. However, conventional frame cameras are limited by their temporal resolution and dynamic range, which imposes constraints in real-world application setups. Event cameras have the advantages of high temporal resolution and high dynamic range (HDR), but the development of event-based methods is necessary to handle data with different characteristics. This paper proposes a novel event-based method for 3D pose estimation and human mesh recovery. Prior work on event-based human mesh recovery require frames (images) as well as event data. The proposed method solely relies on events; it carves 3D voxels by moving the event camera around a stationary body, reconstructs the human pose and mesh by attenuated rays, and fit statistical body models, preserving high-frequency details. The experimental results show that the proposed method outperforms conventional frame-based methods in the estimation accuracy of both pose and body mesh. We also demonstrate results in challenging situations where other frame-based methods suffer from motion blur. This is the first-of-its-kind to demonstrate event-only human mesh recovery, and we hope that it is the first step toward achieving robust and accurate 3D human body scanning from vision sensors.

Access to Document (DOI)

Time-weighted motion history image for human activity classification in sports

Komori H., Isogawa M., Mikami D., Nagai T., Aoki Y.

Sports Engineering (Sports Engineering) 26 ( 1 ) 2023.12

ISSN 13697072

Vision-based human activity classification has remarkable potential for various applications in the sports context (e.g., motion analysis for performance enhancement, active sensing for athletes, etc.). Recently, learning-based human activity classifications have been widely researched. However, in sports scenes in which more detailed and player-specific classifications are required, this is a quite challenging task; in many cases, only a limited number of datasets are available, unlike daily movements such as walking or climbing stairs. Therefore, this paper proposes a time-weighted motion history image, an effective image sequence representation for learning-based human activity classification. Unlike conventional MHI based on the assumption that “the newer frame is more important,” our method generates importance-aware representation so that the predictor can “see” the frames that contribute to analyzing the specific human activity. Experimental results have shown the superiority of our method.

Access to Document (DOI)

Action Quality Assessment Model Using Specialists’ Gaze Location and Kinematics Data—Focusing on Evaluating Figure Skating Jumps

Hirosawa S., Kato T., Yamashita T., Aoki Y.

Sensors (Sensors) 23 ( 22 ) 2023.11

ISSN 14248220

Action quality assessment (AQA) tasks in computer vision evaluate action quality in videos, and they can be applied to sports for performance evaluation. A typical example of AQA is predicting the final score from a video that captures an entire figure skating program. However, no previous studies have predicted individual jump scores, which are of great interest to competitors because of the high weight of competition. Despite the presence of unnecessary information in figure skating videos, human specialists can focus and reduce information when they evaluate jumps. In this study, we clarified the eye movements of figure skating judges and skaters while evaluating jumps and proposed a prediction model for jump performance that utilized specialists’ gaze location to reduce information. Kinematic features obtained from the tracking system were input into the model in addition to videos to improve accuracy. The results showed that skaters focused more on the face, whereas judges focused on the lower extremities. These gaze locations were applied to the model, which demonstrated the highest accuracy when utilizing both specialists’ gaze locations. The model outperformed human predictions and the baseline model (RMSE:0.775), suggesting a combination of human specialist knowledge and machine capabilities could yield higher accuracy.

Access to Document (DOI)

Boosting Semantic Segmentation by Conditioning the Backbone with Semantic Boundaries

Ishikawa H., Aoki Y.

Sensors (Sensors) 23 ( 15 ) 2023.08

ISSN 14248220

In this paper, we propose the Semantic-Boundary-Conditioned Backbone (SBCB) framework, an effective approach to enhancing semantic segmentation performance, particularly around mask boundaries, while maintaining compatibility with various segmentation architectures. Our objective is to improve existing models by leveraging semantic boundary information as an auxiliary task. The SBCB framework incorporates a complementary semantic boundary detection (SBD) task with a multi-task learning approach. It enhances the segmentation backbone without introducing additional parameters during inference or relying on independent post-processing modules. The SBD head utilizes multi-scale features from the backbone, learning low-level features in early stages and understanding high-level semantics in later stages. This complements common semantic segmentation architectures, where features from later stages are used for classification. Extensive evaluations using popular segmentation heads and backbones demonstrate the effectiveness of the SBCB. It leads to an average improvement of (Formula presented.) in IoU and a (Formula presented.) gain in the boundary F-score on the Cityscapes dataset. The SBCB framework also improves over- and under-segmentation characteristics. Furthermore, the SBCB adapts well to customized backbones and emerging vision transformer models, consistently achieving superior performance. In summary, the SBCB framework significantly boosts segmentation performance, especially around boundaries, without introducing complexity to the models. Leveraging the SBD task as an auxiliary objective, our approach demonstrates consistent improvements on various benchmarks, confirming its potential for advancing the field of semantic segmentation.

Access to Document (DOI)

Outdoor Vision-and-Language Navigation Needs Object-Level Alignment

Sun Y., Qiu Y., Aoki Y., Kataoka H.

Sensors (Sensors) 23 ( 13 ) 2023.07

ISSN 14248220

In the field of embodied AI, vision-and-language navigation (VLN) is a crucial and challenging multi-modal task. Specifically, outdoor VLN involves an agent navigating within a graph-based environment, while simultaneously interpreting information from real-world urban environments and natural language instructions. Existing outdoor VLN models predict actions using a combination of panorama and instruction features. However, these methods may cause the agent to struggle to understand complicated outdoor environments and ignore the details in the environments to fail to navigate. Human navigation often involves the use of specific objects as reference landmarks when navigating to unfamiliar places, providing a more rational and efficient approach to navigation. Inspired by this natural human behavior, we propose an object-level alignment module (OAlM), which guides the agent to focus more on object tokens mentioned in the instructions and recognize these landmarks during navigation. By treating these landmarks as sub-goals, our method effectively decomposes a long-range path into a series of shorter paths, ultimately improving the agent’s overall performance. In addition to enabling better object recognition and alignment, our proposed OAlM also fosters a more robust and adaptable agent capable of navigating complex environments. This adaptability is particularly crucial for real-world applications where environmental conditions can be unpredictable and varied. Experimental results show our OAlM is a more object-focused model, and our approach outperforms all metrics on a challenging outdoor VLN Touchdown dataset, exceeding the baseline by 3.19% on task completion (TC). These results highlight the potential of leveraging object-level information in the form of sub-goals to improve navigation performance in embodied AI systems, paving the way for more advanced and efficient outdoor navigation.

Access to Document (DOI)

A Fast Geometric Regularizer to Mitigate Event Collapse in the Contrast Maximization Framework

Shiba S., Aoki Y., Gallego G.

Advanced Intelligent Systems (Advanced Intelligent Systems) 5 ( 3 ) 2023.03

Event cameras are emerging vision sensors and their advantages are suitable for various applications such as autonomous robots. Contrast maximization (CMax), which provides state-of-the-art accuracy on motion estimation using events, may suffer from an overfitting problem called event collapse. Prior works are computationally expensive or cannot alleviate the overfitting, which undermines the benefits of the CMax framework. A novel, computationally efficient regularizer based on geometric principles to mitigate event collapse is proposed. The experiments show that the proposed regularizer achieves state-of-the-art accuracy results, while its reduced computational complexity makes it two to four times faster than previous approaches. To the best of our knowledge, this regularizer is the only effective solution for event collapse without trading off the runtime. It is hoped that this work opens the door for future applications that unlocks the advantages of event cameras. Project page: https://github.com/tub-rip/event_collapse.

Access to Document (DOI)

Efficient Transformer-Based Compressed Video Modeling via Informative Patch Selection

Suzuki T., Aoki Y.

Sensors (Sensors) 23 ( 1 ) 2023.01

ISSN 14248220

Recently, Transformer-based video recognition models have achieved state-of-the-art results on major video recognition benchmarks. However, their high inference cost significantly limits research speed and practical use. In video compression, methods considering small motions and residuals that are less informative and assigning short code lengths to them (e.g., MPEG4) have successfully reduced the redundancy of videos. Inspired by this idea, we propose Informative Patch Selection (IPS), which efficiently reduces the inference cost by excluding redundant patches from the input of the Transformer-based video model. The redundancy of each patch is calculated from motions and residuals obtained while decoding a compressed video. The proposed method is simple and effective in that it can dynamically reduce the inference cost depending on the input without any policy model or additional loss term. Extensive experiments on action recognition demonstrated that our method could significantly improve the trade-off between the accuracy and inference cost of the Transformer-based video model. Although the method does not require any policy model or additional loss term, its performance approaches that of existing methods that do require them.

Access to Document (DOI)

Listening Human Behavior: 3D Human Pose Estimation with Acoustic Signals

Shibata Y., Kawashima Y., Isogawa M., Irie G., Kimura A., Aoki Y.

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition) 2023-June 13323 - 13332 2023

ISSN 10636919

Given only acoustic signals without any high-level information, such as voices or sounds of scenes/actions, how much can we infer about the behavior of humans? Unlike existing methods, which suffer from privacy issues because they use signals that include human speech or the sounds of specific actions, we explore how low-level acoustic signals can provide enough clues to estimate 3D human poses by active acoustic sensing with a single pair of microphones and loudspeakers (see Fig. 1). This is a challenging task since sound is much more diffractive than other signals and therefore covers up the shape of objects in a scene. Accordingly, we introduce a framework that encodes multichannel audio features into 3D human poses. Aiming to capture subtle sound changes to reveal detailed pose information, we explicitly extract phase features from the acoustic signals together with typical spectrum features and feed them into our human pose estimation network. Also, we show that reflected or diffracted sounds are easily influenced by subjects' physique differences e.g., height and muscularity, which deteriorates prediction accuracy. We reduce these gaps by using a subject discriminator to improve accuracy. Our experiments suggest that with the use of only low-dimensional acoustic information, our method outperforms baseline methods. The datasets and codes used in this project will be publicly available.

Access to Document (DOI)

Unsupervised Domain-Adaptive Semantic Segmentation with Uncertainty Loss

Kawano Y., Nota Y., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 89 ( 12 ) 921 - 925 2023

ISSN 09120289

In recent years, deep learning models have achieved great success in many fields, but their training relies on large labeled data. Annotation to obtain labeled samples is challenging, time-consuming, and expensive, and has been a major challenge in deep learning. In this study, we propose a method for unsupervised domain adaptation that can reduce annotation cost in terms of accuracy and time compared to conventional methods by using the uncertainty of the model in semantic segmentation. Unsupervised domain adaptation aims to adapt models trained on synthetic data to real-world data without the need for costly annotation of real images. Since the training data are images and correct labels created by the game engine, there is no need to manually annotate them. However, if the source (synthetic) data is trained using conventional supervised learning, the performance is significantly degraded because the domain is different from the target (real image) data.This study closes the domain gap between source and target by calculating the uncertainty in the target data from the model output on a pixel-by-pixel basis and minimizing this uncertainty as loss. Adding this uncertainty loss to conventional unsupervised domain adaptation methods results in a model that is more robust to the target domain and achieves state-of-the-art results.

Access to Document (DOI)

Preliminary Action Recognition

Kawano Y., Nagasaki Y., Hara K., Aoki Y., Kataoka H.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 89 ( 12 ) 915 - 920 2023

ISSN 09120289

In this paper, we propose preliminary action recognition which is an action anticipation method that can be used in real-time. Preliminary action recognition as a solution to action anticipation, which is the task of detecting the action of a person before the person performs the action. We define preliminary action class before action class that is anticipated by recognizing the preliminary actions specific to the action before it occurs, just as humans do. Conventional action anticipation methods have two problems for real-time use. First, it requires 10 to 20 seconds of video to anticipate 1 second ahead, and it can only anticipate every 10 seconds. Second, the video input in training always switches actions 1 second after the end of the video. This problem setting can be a leakage because there is no action break in real-time operation. Proposed method solves this problem and makes it possible to operate action prediction in real-time. In addition, since action classes and preliminary action classes tend to share a lot of similarities, it is difficult to distinguish between them. We propose a model that can discriminate the subtle differences in motion by increasing the latent distance between adjacent classes in terms of time.

Access to Document (DOI)

Event Collapse in Contrast Maximization Frameworks

Shiba S., Aoki Y., Gallego G.

Sensors (Sensors) 22 ( 14 ) 2022.07

ISSN 14248220

Contrast maximization (CMax) is a framework that provides state-of-the-art results on several event-based computer vision tasks, such as ego-motion or optical flow estimation. However, it may suffer from a problem called event collapse, which is an undesired solution where events are warped into too few pixels. As prior works have largely ignored the issue or proposed workarounds, it is imperative to analyze this phenomenon in detail. Our work demonstrates event collapse in its simplest form and proposes collapse metrics by using first principles of space–time deformation based on differential geometry and physics. We experimentally show on publicly available datasets that the proposed metrics mitigate event collapse and do not harm well-posed warps. To the best of our knowledge, regularizers based on the proposed metrics are the only effective solution against event collapse in the experimental settings considered, compared with other methods. We hope that this work inspires further research to tackle more complex warp models.

Access to Document (DOI)

Non-Deep Active Learning for Deep Neural Networks

Kawano Y., Nota Y., Mochizuki R., Aoki Y.

Sensors (Sensors) 22 ( 14 ) 2022.07

ISSN 14248220

One way to improve annotation efficiency is active learning. The goal of active learning is to select images from many unlabeled images, where labeling will improve the accuracy of the machine learning model the most. To select the most informative unlabeled images, conventional methods use deep neural networks with a large number of computation nodes and long computation time, but we propose a non-deep neural network method that does not require any additional training for unlabeled image selection. The proposed method trains a task model on labeled images, and then the model predicts unlabeled images. Based on this prediction, an uncertainty indicator is generated for each unlabeled image. Images with a high uncertainty index are considered to have a high information content, and are selected for annotation. Our proposed method is based on a very simple and powerful idea: select samples near the decision boundary of the model. Experimental results on multiple datasets show that the proposed method achieves higher accuracy than conventional active learning methods on multiple tasks and up to 14 times faster execution time from (Formula presented.) s to (Formula presented.) s. The proposed method outperforms the current SoTA method by 1% accuracy on CIFAR-10.

Access to Document (DOI)

Supporting collective physical activities by interactive floor projection in a special-needs school setting

Oki M., Akizuki S., Bourreau B., Takahashi I., Aoki Y., Yamamoto J., Suzuki K.

International Journal of Child-Computer Interaction (International Journal of Child-Computer Interaction) 32 2022.06

ISSN 22128689

This paper presents an algorithm to provide floor projection feedback according to the local distance and density of individuals. It is realized by a large-space floor projection system with a feedback function based on human tracking with laser ranging image sensors. The purpose is to support the cognition of spatial–temporal structures of groups of adolescents with neurodevelopmental disorders (NDs) that are conducting organized physical activity (PA). Observation and evaluation of behavioral changes in adolescents with NDs, when they were active with or without the floor projection based on the proposed algorithm, were conducted to validate its effectiveness. We observed that the proposed algorithm can be implemented in different organized PAs. It had the effect to help individuals in a behavior to keep a close distance to each other as a group rather than to keep the same distance apart from each other while walking.

Access to Document (DOI)

Fast Event-Based Optical Flow Estimation by Triplet Matching

Shiba S., Aoki Y., Gallego G.

IEEE Signal Processing Letters (IEEE Signal Processing Letters) 29 2712 - 2716 2022

ISSN 10709908

Event cameras are novel bio-inspired sensors that offer advantages over traditional cameras (low latency, high dynamic range, low power, etc.). Optical flow estimation methods that work on packets of events trade off speed for accuracy, while event-by-event (incremental) methods have strong assumptions and have not been tested on common benchmarks that quantify progress in the field. Towards applications on resource-constrained devices, it is important to develop optical flow algorithms that are fast, light-weight and accurate. This work leverages insights from neuroscience, and proposes a novel optical flow estimation scheme based on triplet matching. The experiments on publicly available benchmarks demonstrate its capability to handle complex scenes with comparable results as prior packet-based algorithms. In addition, the proposed method achieves the fastest execution time (>10 kHz) on standard CPUs as it requires only three events in estimation. We hope that our research opens the door to real-time, incremental motion estimation methods and applications in real-world scenarios.

Access to Document (DOI)

Detection of Osteoarthritis from Multimodal Hand Data

Andrade Guerreiro J.J., Aoki Y., Saito S., Suzuki K.

Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS (Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS) 2022-July 3607 - 3610 2022

ISSN 1557170X

Osteoarthritis (OA) describes a degenerative joint disorder that is prevalent among older people and typically results in swollen and inflamed joints. The aim of this paper is to develop a method using images, videos and thermal data of 100 patients taken at Keio University Hospital to detect OA in hands. By using hand pose estimation on the video data, joint angles can be calculated and subsequently transformed into feature vectors. For the thermal and RGB images, hand keypoint detectors were trained to identify and crop the appropriate joints within the images. The resulting extracted features are combined and further trained on Support Vector Machines and Convolutional Neural Networks to obtain the final binary classification for each joint. While the proposed method generally shows favorable accuracy and F1-scores on the Proximal (PIP) and Distal Interphalangeal (DIP) joints, the performance on the Metacarpophalangeal (MCP) joints is limited by the low occurrence of affected joints in the dataset. We further compare the different modalities and found that, apart from the combined approach, using video data provides the best results. Clinical Relevance - The proposed method shows promising first results for the usage of visual and thermal data in combination with machine learning in order to detect OA

Access to Document (DOI)

DOCUMENT SHADOW REMOVAL WITH FOREGROUND DETECTION LEARNING FROM FULLY SYNTHETIC IMAGES

Matsuo Y., Akimoto N., Aoki Y.

Proceedings - International Conference on Image Processing, ICIP (Proceedings - International Conference on Image Processing, ICIP) 1656 - 1660 2022

ISSN 15224880

Shadow removal for document images is a major task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large-scale and diverse dataset is laborious and remains a great challenge. Thus, only small real datasets are available. To create relatively large datasets, a graphic renderer has been used to synthesize shadows, nonetheless, it is still necessary to capture real documents. Thus, the number of unique documents is limited, which negatively affects a network's performance. In this paper, we present a large-scale and diverse dataset called fully synthetic document shadow removal dataset (FSDSRD) that does not require capturing documents. The experiments showed that the networks (pre-)trained on FSDSRD provided better results than networks trained only on real datasets. Additionally, because foreground maps are available in our dataset, we leveraged them during training for multitask learning, which provided noticeable improvements. The code is available at: https://github.com/IsHYuhi/DSRFGD.

Access to Document (DOI)

Detecting Spinal Alignment Through Depth Image

Ishibashi R., Kameshima H., Nishio Y., Sato Y., Watanabe K., Aoki Y.

Proceedings - 16th International Conference on Signal-Image Technology and Internet-Based Systems, SITIS 2022 (Proceedings - 16th International Conference on Signal-Image Technology and Internet-Based Systems, SITIS 2022) 330 - 334 2022

Adolescent Idiopathic Scoliosis (AIS) is a common form of scoliosis found among adolescents. The abnormal spinal curvature along with twisting of the vertebrae requires early detection before the vertebrae alignment worsens. Furthermore, surgical treatment is crucial for those with severe conditions. In Japan, annual scoliosis screening is conducted during school health checkups. These screening procedures often involve an initial screening without the use of ionizing radiation (such as X-rays) and an X-ray screening later for those suspected with scoliosis. Currently, X-ray examinations produce the most accurate results for detecting the spinal position. However, noninvasive alternative methods, such as Moire imaging, provide satisfactory initial results. Unfortunately, the production of Moire cameras have halted in recent years. This paper proposes an alternative, non-invasive and non-ionizing radioactive method to detect spinal alignment. Depth images provide an extra dimension of information compared to RGB images. The proposed method utilizes this extra dimension of depth values to create different types of images that are then trained using Convolutional Neural Networks to predict the spinal alignment. Results indicate that Moire images reproduced from depth images produce the best spinal alignment. However, other types of images derived from the depth image also have higher accuracy when compared with the depth image itself.

Access to Document (DOI)

Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation

Akimoto N., Matsuo Y., Aoki Y.

ISSN 10636919

We address the problem of generating a 360-degree image from a single image with a narrow field of view by estimating its surroundings. Previous methods suffered from overfitting to the training resolution and deterministic generation. This paper proposes a completion method using a transformer for scene modeling and novel methods to improve the properties of a 360-degree image on the output image. Specifically, we use CompletionNets with a transformer to perform diverse completions and Adjust-mentNet to match color, stitching, and resolution with an input image, enabling inference at any resolution. To improve the properties of a 360-degree image on an output image, we also propose WS-perceptual loss and circular inference. Thorough experiments show that our method out-performs state-of-the-art (SOTA) methods both qualitatively and quantitatively. For example, compared to SOTA methods, our method completes images 16 times larger in resolution and achieves 1.7 times lower Fréchet inception distance (FID). Furthermore, we propose a pipeline that uses the completion results for lighting and background of 3DCG scenes. Our plausible background completion enables perceptually natural results in the application of inserting virtual objects with specular surfaces.

Access to Document (DOI)

Determinant analysis and developing evaluation indicators of grade of execution score of double axel jump in figure skating

Hirosawa S., Watanabe M., Aoki Y.

Journal of Sports Sciences (Journal of Sports Sciences) 40 ( 4 ) 470 - 481 2022

ISSN 02640414

A figure skating jump score is determined by the sum of the base value based on the difficulty and grade of execution (GOE) that indicates the performance quality. Therefore, performing a high-quality jump to obtain a high GOE is essential to win a competition. However, the relationship between the GOE and kinematic parameters remains unclear. We analysed the horizontal distance, vertical height, and landing speed of double axel jumps in the Ladies’ Short Program at the 2019 World Championships. The highest GOE group had significantly larger horizontal distances than the middle and lower groups, while the landing speed and vertical height were not significantly different. A principal component regression analysis was conducted to clarify the contrast between the three variables affecting the GOE. The results showed that greater horizontal distance and landing speed compared to vertical height (component 1) and greater horizontal distance compared to landing speed (component 3) contributed to higher GOE. We divided skaters into four clusters using these two components and provided general GOE acquisition strategies for each cluster. Finally, to apply our results to the industry, we proposed two new evaluation indicators which are highly correlated with the two components and easy to interpret.

Access to Document (DOI)

Improving Annotation Efficiency through Uncertain Sample Selection in Active Learning

Kawano Y., Nota Y., Mochizuki R., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 88 ( 2 ) 211 - 216 2022

ISSN 09120289

Active learning refers to label-efficient algorithms that use the most representative samples for labeling when creating training data. In this paper, we propose a model that derives the most informative unlabeled samples from the output of a task model. The tasks arc a classification problem, multi-label classification and a semantic segmentation problem. The model consists of an uncertainty indicator generator and a task model. After training the task model with labeled samples, the model predicts unlabeled samples, and based on the prediction results, the uncertainty indicator generator outputs an uncertainty indicator for each unlabeled sample. Samples with a higher uncertainty indicator are considered to be more informative and selected. As a result of experiments using multiple datasets, our model achieved better accuracy than conventional active learning methods and reduced execution time by a factor of 10.

Access to Document (DOI)

Secrets of Event-Based Optical Flow

Shiba S., Aoki Y., Gallego G.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)) 13678 LNCS 628 - 645 2022

ISSN 03029743

Event cameras respond to scene dynamics and offer advantages to estimate motion. Following recent image-based deep-learning achievements, optical flow estimation methods for event cameras have rushed to combine those image-based methods with event data. However, it requires several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. We investigate key elements: how to design the objective function to prevent overfitting, how to warp events to deal better with occlusions, and how to improve convergence with multi-scale raw events. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark. Moreover, our method allows us to expose the issues of the ground truth flow in those benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Our code is available at https://github.com/tub-rip/event_based_optical_flow.

Access to Document (DOI)

Temporal Cross-Modal Attention for Audio-Visual Event Localization

Nagasaki Y., Hayashi M., Kaneko N., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 88 ( 3 ) 263 - 268 2022

ISSN 09120289

In this paper, we propose a new method for audio-visual event localization 1) to find the corresponding segment between audio and visual event. While previous methods use Long Short-Term Memory (LSTM) networks to extract temporal features, recurrent neural networks like LSTM are not able to precisely learn long-term features. Thus, we propose a Temporal Cross-Modal Attention (TCMA) module, which extract temporal features more precisely from the two modalities. Inspired by the success of attention modules in capturing long-term features, we introduce TCMA, which incorporates self-attention. Finally, we were able to localize audio-visual event precisely and achieved a higher accuracy than the previous works.

Access to Document (DOI)

Self-Supervised Learning of Inlier Events for Event-based Optical Flow

Nagata J., Aoki Y.

BMVC 2022 - 33rd British Machine Vision Conference Proceedings (BMVC 2022 - 33rd British Machine Vision Conference Proceedings) 2022

Event cameras asynchronously report per-pixel intensity changes with high temporal resolution. Their sparse and temporally precise nature is well suited for fast visual flow estimation. The normal optical flow can be estimated by fitting a plane to spatio-temporal events. However, least-squares plane fitting suffers from outliers due to the significant noise of events. Existing methods involve 1) iterative outlier rejection or 2) goodness-of-fit rejection after single-shot planar fitting from greedily selected spatially neighboring events. In contrast to these methods, we propose a method of selecting the events supporting a plane before performing a fitting, using the inlier probability from a lightweight neural network that captures both global and local structures. During inference, single-shot planar fitting is performed from only events with a higher inlier probability. We model each event selection by a Bernoulli distribution with the inlier probability and train the network to maximize the inlier count while sampling in a self-supervised manner. We verify that our event selection improves the accuracy of optical flow estimation with publicly available real data.

Optical flow estimation by matching time surface with event-based cameras

Nagata J., Sekikawa Y., Aoki Y.

Sensors (Switzerland) (Sensors (Switzerland)) 21 ( 4 ) 1 - 14 2021.02

ISSN 14248220

In this work, we propose a novel method of estimating optical flow from event-based cameras by matching the time surface of events. The proposed loss function measures the timestamp consistency between the time surface formed by the latest timestamp of each pixel and the one that is slightly shifted in time. This makes it possible to estimate dense optical flows with high accuracy without restoring luminance or additional sensor information. In the experiment, we show that the gradient was more correct and the loss landscape was more stable than the variance loss in the motion compensation approach. In addition, we show that the optical flow can be estimated with high accuracy by optimization with L1 smoothness regularization using publicly available datasets.

Access to Document (DOI)

Alleviating over-segmentation errors by detecting action boundaries

Ishikawa Y., Kasai S., Aoki Y., Kataoka H.

Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021 (Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021) 2321 - 2330 2021.01

We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our contributions are three-fold: (i) We propose a framework for temporal action segmentation, the ASRF, which divides temporal action segmentation into frame-wise action classification and action boundary regression. Our framework refines frame-level hypotheses of action classes using predicted action boundaries. (ii) We propose a loss function for smoothing the transition of action probabilities, and analyze combinations of various loss functions for temporal action segmentation. (iii) Our framework outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in terms of segmental edit distance and up to 16.1% in terms of segmental F1 score. Our code is publicly available1.

Access to Document (DOI)

Scenery Image Extension via Inpainting with a Mirrored Input

Akimoto N., Ito D., Aoki Y.

IEEE Access (IEEE Access) 9 59286 - 59300 2021

Generative image extension has the advantage of extending the overall image size while preserving the target image because, unlike other image extensions using interpolation, it completes the surroundings of the target image. However, existing generative image extension methods tend to have poor quality in the generation of outer pixels. One method deals only with a limited number of scene classes because the extension repeats the same semantics. We propose a mirrored input, which sandwiches the extended region by mirroring a part of the target image. This replaces generative image extension with an image inpainting problem and thus helps to achieve higher quality pixel generation and can extend semantics with more complex shapes than horizontal repetition. Experimental results show that our proposed method achieves a scenery image extension that exceeds the state-of-the-art generative image extension methods in both visual quality and FID score for datasets containing diverse scenes.

Access to Document (DOI)

Analysis of recent re-identification architectures for tracking-by-detection paradigm in multi-object tracking

Ishikawa H., Hayashi M., Phan T.H., Yamamoto K., Masuda M., Aoki Y.

VISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021 - Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications) 5 234 - 244 2021

ISSN 9789897584886

Person re-identification is a vital module of the tracking-by-detection framework for online multi-object tracking. Despite recent advances in multi-object tracking and person re-identification, inadequate attention was given to integrating these technologies to provide a robust multi-object tracker. In this work, we combine modern state-of-the-art re-identification models and modeling techniques on the basic tracking-by-detection framework and benchmark them on heavily occluded scenes to understand their effect. We hypothesize that temporal modeling for re-identification is crucial for training robust re-identification models for they are conditioned on sequences containing occlusions. Along with traditional image-based re-identification methods, we analyze temporal modeling methods used in video-based re-identification tasks. We also train re-identification models with different embedding methods, including triplet loss, and analyze their effect. We benchmark the re-identification models on the challenging MOT20 dataset containing crowded scenes with various occlusions. We provide a thorough assessment and investigation of the usage of modern re-identification modeling methods and prove that these methods are, in fact, effective for multi-object tracking. Compared to baseline methods, results show that these models can provide robust re-identification proved by improvements in the number of identity switching, MOTA, IDF1, and other metrics.

Multi-modal action segmentation in the kitchen with a feature fusion approach

Kogure S., Aoki Y.

Proceedings of SPIE - The International Society for Optical Engineering (Proceedings of SPIE - The International Society for Optical Engineering) 11794 2021

ISSN 0277786X

In this paper, we propose a "Multi-modal Action Segmentation approach"that uses three modalities: (i) video, (ii) audio, (iii) thermal to classify cooking behavior in the kitchen. These 3 modalities are assumed to be features related to cooking. However, there is no public dataset containing these three modalities. Therefore, we built the original dataset and frame-level annotation. We then examined the usefulness of Action Segmentation using multi-modal features. We analyzed the effects of each modality using three evaluation metrics. As a result, the accuracy, edit distance, and F1 value were improved by up to about 1%, 2%, and 8%, respectively, compared to the case when only images were used.

Access to Document (DOI)

Out-of-distribution detection for fungi images with similar features

Kawashima Y., Higo M., Tokiwa T., Asami Y., Nonaka K., Aoki Y.

Proceedings of SPIE - The International Society for Optical Engineering (Proceedings of SPIE - The International Society for Optical Engineering) 11794 2021

ISSN 0277786X

In order to create a classification model for fungi, it is necessary to have robustness against out-of-distribution data from the viewpoint of practicality. Therefore, in this paper, we perform out-of-distribution detection on a fungi. Unlike the case of conventional out-of-distribution detection, the characteristics of in-distribution data and out-of-distribution data in this paper are very similar. Therefore, the problem in which conventional methods using out-of-distribution data for validation are not effective is mentioned. We also verify whether the accuracy of out-of-distribution detection can be improved using the attention branch network.

Access to Document (DOI)

Retrieving and Highlighting Action with Spatiotemporal Reference

Kasai S., Ishikawa Y., Hayashi M., Aoki Y., Hara K., Kataoka H.

Proceedings - International Conference on Image Processing, ICIP (Proceedings - International Conference on Image Processing, ICIP) 2020-October 1401 - 1405 2020.10

ISSN 9781728163956

In this paper, we present a framework thatjointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which relate to the nouns and verbs in captions. Through experiments, we show that our model generates various maps conditioned on different actions, in which conventional visual reasoning methods only go as far as to show a single deterministic saliency map. Also, our model improves retrieval recall over our baseline without alignment by 2-3% on the MSR-VTT dataset.

Access to Document (DOI)

Joint Pedestrian Detection and Risk-level Prediction with Motion-Representation-by-Detection

Kataoka H., Suzuki T., Nakashima K., Satoh Y., Aoki Y.

Proceedings - IEEE International Conference on Robotics and Automation (Proceedings - IEEE International Conference on Robotics and Automation) 1021 - 1027 2020.05

ISSN 9781728173955

The paper presents a pedestrian near-miss detector with temporal analysis that provides both pedestrian detection and risk-level predictions which are demonstrated on a self-collected database. Our work makes three primary contributions: (i) The framework of pedestrian near-miss detection is proposed by providing both a pedestrian detection and risk-level assignment. Specifically, we have created a Pedestrian Near-Miss (PNM) dataset that categorizes traffic near-miss incidents based on their risk levels (high-, low-, and no-risk). Unlike existing databases, our dataset also includes manually localized pedestrian labels as well as a large number of incident-related videos. (ii) Single-Shot MultiBox Detector with Motion Representation (SSD-MR) is implemented to effectively extract motion-based features in a detected pedestrian. (iii) Using the self-collected PNM dataset and SSD-MR, our proposed method achieved +19.38% (on risk-level prediction) and +13.00% (on joint pedestrian detection and risk-level prediction) higher scores than that of the baseline SSD and LSTM. Additionally, the running time of our system is over 50 fps on a graphics processing unit (GPU).

Access to Document (DOI)

FOE-based regularization for optical flow estimation from an in-vehicle event camera

Nagata J., Sekikawa Y., Hara K., Aoki Y.

Electronics and Communications in Japan (Electronics and Communications in Japan) 103 ( 1-4 ) 19 - 25 2020.04

ISSN 19429533

Optical flow estimation from an in-vehicle camera is an important task in automatic driving and advanced driver-assistance systems. However, there is a problem that optical flow estimation is mistakable with high contrast and high speed. Event camera can overcome these situations because it reports only the per-pixel intensity change with high dynamic range and low latency. However, the L1 smoothness regularization in the conventional optical flow estimation method is not suitable for radial optical flow in the driving scene. Therefore, we propose to use the focus of expansion (FOE) for regularization of optical flow estimation in event camera. The FOE is defined as the intersection of the translation vector of the camera and the image plane. The optical flow becomes radial from the FOE excluding the rotational component. Using the property, the optical flow can be regularized in the correct direction in the optimization process. We demonstrated that the optical flow was improved by introducing our regularization using the public dataset.

Access to Document (DOI)

QR-code reconstruction from event data via optimization in code subspace

Nagata J., Sekikawa Y., Hara K., Suzuki T., Yoshimitsu A.

Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020 (Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020) 2113 - 2121 2020.03

ISSN 9781728165530

We propose an image reconstruction method from event data, assuming the target images belong to a prespecified class like QR codes. Instead of solving the reconstruction problem in the image space, we introduce a code space that covers all the noiseless target class images and solves the reconstruction problem on it. This restriction enormously reduces the number of optimizing parameters and makes the reconstruction problem well posed and robust to noise. We demonstrate fast and robust QR-code scanning in difficult, high-speed scenes with industrial high-speed cameras and other reconstruction methods.

Access to Document (DOI)

Fast soft color segmentation

Akimoto N., Zhu H., Jin Y., Aoki Y.

ISSN 10636919

We address the problem of soft color segmentation, defined as decomposing a given image into several RGBA layers, each containing only homogeneous color regions. The resulting layers from decomposition pave the way for applications that benefit from layer-based editing, such as recoloring and compositing of images and videos. The current state-of-the-art approach for this problem is hindered by slow processing time due to its iterative nature, and consequently does not scale to certain real-world scenarios. To address this issue, we propose a neural network based method for this task that decomposes a given image into multiple layers in a single forward pass. Furthermore, our method separately decomposes the color layers and the alpha channel layers. By leveraging a novel training objective, our method achieves proper assignment of colors amongst layers. As a consequence, our method achieve promising quality without existing issue of inference speed for iterative approaches. Our thorough experimental analysis shows that our method produces qualitative and quantitative results comparable to previous methods while achieving a 300,000x speed improvement. Finally, we utilize our proposed method on several applications, and demonstrate its speed advantage, especially in video editing.

Access to Document (DOI)

Action Recognition in Sports Video Considering Location Information

Ichige R., Aoki Y.

Communications in Computer and Information Science (Communications in Computer and Information Science) 1212 CCIS 150 - 164 2020

ISSN 9789811548178

The purpose of this study is to develop a tactics analysis system using image recognition for rugby. With the Rugby World Cup in 2019 and the Tokyo Olympics in 2020, demand for sports video analysis is increasing. Rugby has more complicated play such as dense play than other sports, and the ball is hidden between players, making it difficult to track. By developing a high-precision analysis technology for rugby with few research cases, we thought that it could be used for other sports and industrial fields other than sports. In this research, we propose a method that adds spatial information to time-series information as a new feature. Using the coordinates obtained by projectively transforming the match video onto the bird’s-eye view image, play classification was performed using the player position, the ball position, and the dense area position as feature amounts. Also, in order to further improve the detection accuracy of the boundaries between plays, attention was paid to the positional relationship of each player on the field.

Access to Document (DOI)

Weakly Supervised Domain Adaptation with Point Supervision in Histopathological Image Segmentation

Obikane S., Aoki Y.

Communications in Computer and Information Science (Communications in Computer and Information Science) 1180 CCIS 127 - 140 2020

ISSN 9789811536502

When a model learned in a domain is applied to a different domain, even if in the same task, there is no guarantee of accuracy. This is a very important issue when deep learning and machine learning are applied in the field. In medical applications, there is a wide variety of domain bias, making it very difficult to create a model appropriate for all domains. Furthermore, semantic segmentation needs fine annotation and its high labor cost makes its application difficult. Histopathological image segmentation enables drug discovery and medical image analysis, but it is expensive due to its annotation cost and the need for the skills of histopathological experts. In this paper, we focus on a weakly supervised method using point annotation unique to histopathological image segmentation, and tackled on weakly supervised domain adaptation to suppress domain gaps. Providing point level annotation instead of fine annotation decreases the high cost of labor normally required.

Access to Document (DOI)

Flood Susceptibility Prediction via Data-Mining Based Bell-Curve Analogical-Hydrographs Analysis: A Case Study of Langat River Basin, Selangor, Malaysia

Khuzaimah S.N., Kiyoki Y., Aoki Y.

Frontiers in Artificial Intelligence and Applications (Frontiers in Artificial Intelligence and Applications) 321 442 - 457 2019.12

ISSN 9781643680446

In this study, we proposed data-mining based bell-curve analogical hydrographs analysis with lag time vertical axes and bankfull discharge horizontal axes to make flood susceptibility prediction. We utilized flood data reports, hourly/daily rainfall data and daily water discharge of Hulu Langat district, Selangor Malaysia from the year 2013-2016 to do flood susceptibility. We implement data mining concept by sorting the database, followed by plotting hydrograph to identify flood patterns and establish relationships to predict flood trends. This method is an intersection between the knowledge field of hydrology and mathematical modeling. When an outlier from the graph is detected, the knowledge from hydrology can be applied to understand the reason behind the appearance of outliers. Besides, the knowledge of mathematical modeling is necessary to assist us in predicting flood susceptibility. The purpose of this study is to predict the flood susceptibility which is vital to prepare the users/public well prepared for smooth and efficient evacuation. In 4 years context, our flood depth predictions are nearly 100% accurate. Factor influencing the lag time and steepness of rising limb are related to land use and topographical features. Implications of the results and future research directions are also presented.

Access to Document (DOI)

An application of artificial intelligence to diagnostic imaging of spine disease: Estimating spinal alignment from moiré images

Watanabe K., Aoki Y., Matsumoto M.

Neurospine (Neurospine) 16 ( 4 ) 697 - 702 2019.12

ISSN 25866583

The use of artificial intelligence (AI) as a tool supporting the diagnosis and treatment of spinal diseases is eagerly anticipated. In the field of diagnostic imaging, the possible application of AI includes diagnostic support for diseases requiring highly specialized expertise, such as trauma in children, scoliosis, symptomatic diseases, and spinal cord tumors. Moiré topography, which describes the 3-dimensional surface of the trunk with band patterns, has been used to screen students for scoliosis, but the interpretation of the band patterns can be ambiguous. Thus, we created a scoliosis screening system that estimates spinal alignment, the Cobb angle, and vertebral rotation from moiré images. In our system, a convolutional neural network (CNN) estimates the positions of 12 thoracic and 5 lumbar vertebrae, 17 spinous processes, and the vertebral rotation angle of each vertebra. We used this information to estimate the Cobb angle. The mean absolute error (MAE) of the estimated vertebral positions was 3.6 pixels (~5.4 mm) per person. T1 and L5 had smaller MAEs than the other levels. The MAE per person between the Cobb angle measured by doctors and the estimated Cobb angle was 3.42°. The MAE was 4.38° in normal spines, 3.13° in spines with a slight deformity, and 2.74° in spines with a mild to severe deformity. The MAE of the angle of vertebral rotation was 2.9° ± 1.4°, and was smaller when the deformity was milder. The proposed method of estimating the Cobb angle and AVR from moiré images using a CNN is expected to enhance the accuracy of scoliosis screening.

Access to Document (DOI)

Task-oriented function detection based on operational tasks

Ishikawa Y., Ishikawa H., Akizuki S., Yamazaki M., Taniguchi Y., Aoki Y.

2019 19th International Conference on Advanced Robotics, ICAR 2019 (2019 19th International Conference on Advanced Robotics, ICAR 2019) 635 - 640 2019.12

ISSN 9781728124674

We propose novel representations for functions of an object, namely Task-oriented Function, which is improved upon the idea of Afforadance in the field of Robotics Vision. We also propose a convolutional neural network to detect task-oriented functions. This network takes as input an operational task as well as an RGB image and assign each pixel an appropriate label for every task. Task-oriented funciton makes it possible to descibe various ways to use an object because the outputs from the network differ depending on operational tasks. We introduce a new dataset for task-oriented function detection, which contains about 1200 RGB images and 6000 pixel-level annotations assuming five tasks. Our proposed method reached 0.80 mean IOU in our dataset.

Access to Document (DOI)

Shot detection in racket sport video at the frame level using a recurrent neural network

Horie S., Sato Y., Furuyama J., Tanabiki M., Aoki Y.

Proceedings - 15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019 (Proceedings - 15th International Conference on Signal Image Technology and Internet Based Systems, SISITS 2019) 447 - 453 2019.11

ISSN 9781728156866

In recent years, there has been a demand in the sports industry to reduce the burden of data collection and video editing for tactical analysis. To achieve these, a system that can recognize the game context is needed. In this study, we proposed a method to identify the player's shot timing at the frame level during a ball-striking sport. In this study, players' shots were detected in video of a tennis match. It was shown that shots could be detected with an F-score value of 87% or more within an error range of 1 frame (0.033 sec) by considering time-series information using a recurrent neural network. This technology is expected to be applied not only to tennis, but also to other sports that involve ball shots, such as table tennis, baseball, and volleyball. At the same time, it can be used to detect moments of a specific action (for example, touching or hitting an object).

Access to Document (DOI)

Fine-grained action recognition in assembly work scenes by drawing attention to the hands

Kobayashi T., Aoki Y., Shimizu S., Kusano K., Okumura S.

ISSN 9781728156866

The analysis of assembly working scenes is an important tool for improving work efficiency and detecting worker error. Many factories choose to analyze videos of employees working by human observation. It is more efficient, however, to approach this as an action segmentation task, but this is difficult due to the fine-grained actions. In this paper, we focus on featuring the workers hands in action segmentation to describe the detailed moves in assembly work scenes and utilize the tool or product information that the worker is using. We propose two methods to draw attention to the hand from each image in the video: the first is by cutting out the workers hand image and extracting features, and the second is by using an attention module specialized to extract hand features by the training of a regression network. For both methods we combine different types of features - image features and pose features - which are both important information in fine-grained actions. Due to the lack of action datasets that focus on assembly work scenes, we create a new assembly work dataset for our experiments, and the results of those experiments show that our method is effective for analyzing fine-grained action segmentation tasks.

Access to Document (DOI)

360-Degree Image Completion by Two-Stage Conditional Gans

Akimoto N., Kasai S., Hayashi M., Aoki Y.

Proceedings - International Conference on Image Processing, ICIP (Proceedings - International Conference on Image Processing, ICIP) 2019-September 4704 - 4708 2019.09

ISSN 9781538662496

The latest generative adversarial networks (GANs) can generate realistic high resolution images. However, to the best of our knowledge, there are no GANs for generating 360-degree images. Therefore, this paper proposes the novel problem setting that by using a known area from the 360-degree image as an input, the remainder of the image can be completed with the GANs. To do so, we propose the approach of two-stage generation using network architecture with series-parallel dilated convolution layers. Moreover, we present how to rearrange images for data augmentation, simplify the problem, and make inputs for training the 2 stage generator. Our experiments show that these methods generate the distortion seen in 360-degree images in the outlines of buildings and roads, and their boundaries are clearer than those of baseline methods. Furthermore, we discuss and clarify the difficulty of our proposed problem. Our work is the first step towards GANs predicting an unseen area within a 360-degree space. nd

Access to Document (DOI)

Human-object maps for daily activity recognition

Ishikawa H., Ishikawa Y., Akizuki S., Aoki Y.

Proceedings of the 16th International Conference on Machine Vision Applications, MVA 2019 (Proceedings of the 16th International Conference on Machine Vision Applications, MVA 2019) 2019.05

ISSN 9784901122184

In the field of action recognition, when and where an interaction between a human and an object happens has the potential to be valid information in enhancing action recognition accuracy. Especially, in daily life where each activities are performed in longer time frame, conventional short term action recognition may fail to generalize do to the variety of shorter actions that could take place during the activity. In this paper, we propose a novel representation of human object interaction called Human-Object Maps (HOMs) for recognition of long term daily activities. HOMs are 2D probability maps that represents spatio-temporal information of human object interaction in a given scene. We analyzed the effectiveness of HOMs as well as features relating to the time of the day in daily activity recognition. Since there are no publicly available daily activity dataset that depicts daily routines needed for our task, we have created a new dataset that contains long term activities. Using this dataset, we confirm that our method enhances the prediction accuracy of the conventional 3D ResNeXt action recognition method from 86.31% to 97.89%.

Access to Document (DOI)

Tactile logging for understanding plausible tool use based on human demonstration

Akizuki S., Aoki Y.

British Machine Vision Conference 2018, BMVC 2018 (British Machine Vision Conference 2018, BMVC 2018) 2019

In this paper, we propose a novel human-object interaction (HOI) representation for understanding plausible tool use; this is done by using an object-centric method. Everyday tools (such as kitchen or DIY utensils) have the ideal region and direction for being touched by hands and surrounding objects. By analyzing the human demonstrations, tactile histories that occurred on the tools are accumulated to the tool's surface as the "Tactile Log", which can help to achieve the plausible tool use by robot arms. We also proposed a dataset for the evaluation of generated tactile logs, which is called the Object-Centric Interaction (OCI) dataset consists of RGB-D videos of plausible tool use by human and 3D models of the tools used in the videos. We confirm that the precision of our tactile logging is 0.77.

FOE-based regularization for optical flow estimation from an in-vehicle event camera

Nagata J., Sekikawa Y., Hara K., Aoki Y.

IEEJ Transactions on Electronics, Information and Systems (IEEJ Transactions on Electronics, Information and Systems) 139 ( 10 ) 1113 - 1118 2019

ISSN 03854221

Optical flow estimation from an in-vehicle camera is an important task in automatic driving and advanced driver- A ssistance systems. However, there is a problem that optical flow estimation is mistakable with high contrast and high speed. Event camera can overcome these situations because it reports only the per-pixel intensity change with high dynamic range and low latency. However, the LI smoothness regularization in the conventional optical flow estimation method is not suitable for radial optical flow in the driving scene. Therefore, we propose to use the focus of expansion (FOE) for regularization of optical flow estimation in event camera. The FOE is defined as the intersection of the translation vector of the camera and the image plane. The optical flow becomes radial from the FOE excluding the rotational component. Using the property, the optical flow can be regularized in the correct direction in the optimization process. We demonstrated that the optical flow was improved by introducing our regularization using the public dataset.

Access to Document (DOI)

FOE-based regularization for optical flow estimation from an in-vehicle event camera

Nagata J., Sekikawa Y., Hara K., Aoki Y.

Proceedings of SPIE - The International Society for Optical Engineering (Proceedings of SPIE - The International Society for Optical Engineering) 11049 2019

ISSN 9781510627734

Optical flow estimation in onboard cameras is an important task in automatic driving and advanced driver- assistance systems. However, there is a problem that calculation is mistakable with high contrast and high speed. Event cameras have great features such as high dynamic range and low latency that can overcome these problems. Event cameras report only the change in the logarithmic intensity per pixel rather than the absolute brightness. There is a method of estimating the optical ow simultaneously with the luminance restoration from the event data. The regularization using the L1 norm of differentiation is insufficient for spatially sparse event data. Therefore, we propose to use the focus of expansion (FOE) for regularization of optical ow estimation in event camera. The FOE is defined as the intersection of the translation vector of the camera and the image plane. The optical ow becomes radial from the FOE excluding the rotational component. Using the property, the optical ow can be regularized in the correct direction in the optimization process. We demonstrated that the optical ow was improved by introducing our regularization using the public dataset.

Access to Document (DOI)

Detect functions of objects based on operational task input

Ishikawa Y., Ishikawa H., Akizuki S., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 85 ( 12 ) 1136 - 1142 2019

ISSN 09120289

Recently, the development of deep learning has enabled robots to grasp objects more reliably than ever. Given this fact, there is an increasing demand for helper robots or home robots. To make these robots real, robots need to understand not only how to grasp objects but also their functions. We propose a new representation for the functions of objects, task-oriented function, which is based on operational task input. This representation makes it possible to describe a variety of ways to use an object. We also propose a new dataset for task-oriented function and a network to detect it. This model reached 79.7% mean IOU in our dataset.

Access to Document (DOI)

6D0F pose estimation for similar objects using spatial relationship of part-affordance

Akizuki S., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 85 ( 1 ) 107 - 112 2019

ISSN 09120289

In this paper, we propose a method to estimate 6D pose (3D position and pose) of the everyday objects, especially Kitchen or DIY tools, even if there are not the same 3D model of the target object. We assume that the tools that belong to same category have common spatial relation in part-affordances, the role of each part. The proposed method uses this assumption for pose estimation. By minimizing two types of consistencies, the spatial relationship of part-affordances and the shapes, we have confirmed that the proposed method achieved higher performance compared with traditional pose estimation method.

Access to Document (DOI)

Visual attribute manipulation using natural language commands

Kasai S., Akimoto N., Hayashi M., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 85 ( 12 ) 1102 - 1109 2019

ISSN 09120289

In this paper, a novel setting is tackled in which a neural network generates object images with transferred attributes, by conditioning on natural language commands. Conventional methods for object image transformation have used visual attributes, which are components that describe the object's color, posture, etc. This paper builds on this approach and finds an algorithm to precisely extract information from natural language commands, which transfers the attributes of an image and completes this image translation model. The effectiveness of our information extraction model is experimented, with additional tests to see if the change in visual attributes is correctly seen in the image.

Access to Document (DOI)

Measurement of Vertebral Rotation from Moire Image for Screening of Adolescent Idiopathic Scoliosis

Ran CHOI, Kota WATANABE, Yoshimitsu AOKI

IIEEJ Transactions on Image Electronics and Visual Computing (IIEEJ) 6 ( 2 ) 56 - 64 2018.12

Research paper (scientific journal), Joint Work, Accepted

Anticipating Traffic Accidents with Adaptive Loss and Large-Scale Incident DB

Suzuki T., Kataoka H., Aoki Y., Satoh Y.

ISSN 9781538664209

© 2018 IEEE. In this paper, we propose a novel approach for traffic accident anticipation through (i) Adaptive Loss for Early Anticipation (AdaLEA) and (ii) a large-scale self-annotated incident database for anticipation. The proposed AdaLEA allows a model to gradually learn an earlier anticipation as training progresses. The loss function adaptively assigns penalty weights depending on how early the model can anticipate a traffic accident at each epoch. Additionally, we construct a Near-miss Incident DataBase for anticipation. This database contains an enormous number of traffic near-miss incident videos and annotations for detail evaluation of two tasks, risk anticipation and risk-factor anticipation. In our experimental results, we found our proposal achieved the highest scores for risk anticipation (+6.6% better on mean average precision (mAP) and 2.36 sec earlier than previous work on the average time-to-collision (ATTC)) and risk-factor anticipation (+4.3% better on mAP and 0.70 sec earlier than previous work on ATTC).

Access to Document (DOI)

Generating effect animation with conditional GANs

Akimoto N., Kataoka H., Aoki Y.

SIGGRAPH Asia 2018 Posters, SA 2018 (SIGGRAPH Asia 2018 Posters, SA 2018) 2018.12

ISSN 9781450360630

Access to Document (DOI)

機能属性の空間配置に着目した類似形状物体の6自由度姿勢推定

秋月秀一, 青木義満

精密工学会誌 (精密工学会) 2018.12

Research paper (scientific journal), Joint Work, Accepted

GAN によるカラー調整と画像補完の同時実行

秋本直郁，秋月秀一，林昌希，青木義満

精密工学会誌 (精密工学会) 84 ( 12 ) 1033 - 1040 2018.12

Research paper (scientific journal), Joint Work, Accepted

追跡軌跡の再同定を用いたオンライン複数物体追跡

加藤直樹，田靡雅基，古山純子，里雄二，青木義満

画像電子学会論文誌 (画像電子学会) 47 ( 4 ) 391 - 400 2018.11

Research paper (scientific journal), Joint Work, Accepted

Superpixel convolution for segmentation

Suzuki T., Akizuki S., Kato N., Aoki Y.

Proceedings - International Conference on Image Processing, ICIP (Proceedings - International Conference on Image Processing, ICIP) 3249 - 3253 2018.08

ISSN 9781479970612

© 2018 IEEE. In this paper, we propose a novel segmentation algorithm based on convolutional neural networks (CNNs) on superpix-else CNNs are powerful methods for several computer vision tasks, but spatial information disappears through the pooling process. Moreover, since pooling compresses different types of pixels into a single value, pooling sometimes negatively affects the results of inference in segmentation task. We use superpixel pooling instead of general pooling to resolve this problem. However, general CNNs can't use superpixel images in which the adjacency relationships between pixels are broken. Therefore, we define CNNs and Dilated Convolution on superpixels. Finally, we show the effectiveness of proposed method on an HKU-IS dataset.

Access to Document (DOI)

Segmentationのためのスーパーピクセル上でのGraph Convolutional Neural Networksの提案

鈴木哲平，青木義満

電子情報通信学会和文論文誌D (電子情報通信学会) J101-D ( 8 ) 1120 - 1129 2018.08

Research paper (scientific journal), Joint Work, Accepted

Estimation of Scenes Contributing to Score in Tennis Video Using Attention

Kurose R., Hayashi M., Aoki Y.

Proceedings - 14th International Conference on Signal Image Technology and Internet Based Systems, SITIS 2018 (Proceedings - 14th International Conference on Signal Image Technology and Internet Based Systems, SITIS 2018) 672 - 675 2018.07

ISSN 9781538693858

The use of image processing technology for sports is increasing. By analyzing athletes and teams based on video analysis, scientific and objective analysis can be conducted separately from subjective analysis of experts, and individuals or teams can be evaluated with the same index. As can be seen from such a trend, recently, technological progress has made it possible to detect and analyze what can be visually confirmed in the sports video. It is certain that these technologies are helping coaches in the analysis of movement. However, these cannot play the role of coaches. The role is to judge what is important from what can be visually confirmed. There are also empirical and qualitative parts in coach's judgment, and no reproducibility of understanding the important parts unless it is a specialist. Based on this background, we thought that it would be useful to extract more important scenes during the game using quantitative information. Such technology can be applied to various sports, but this paper, we focused on tennis. In this thesis, the aim is to estimate which play was largely contributed to the tennis game result (score, failure). In addition, we don't use supervised information that can be obtained from an empirical point of view. This is to eliminate dataset dependency caused by using data created from qualitative information. Specifically, based on quantitative information such as athletes' movement and score result, we attempted to estimate the attention from unsupervised method.

Access to Document (DOI)

Single Image, Context Aware Action Estimation in Sports

Lanius C., Kobayashi D., Ouchi K., Aoki Y.

ISSN 9781538693858

Sports video analysis enables professional teams to prepare and practice more efficiently than ever before. It allows the team to work on their weak points and analyze upcoming opponents' strategies. As segmentation is a well researched topic, we assume a sports video where each frame is segmented into actors. We show that the location of the actors is correlated to their action. We develop a model to predict each actor's action in a context consistent and location aware manner. The set of context actors is an unordered set, thus we propose a permutation invariant structure for our classifier. We evaluate a location and a context aware network for single image sports action recognition on two datasets. One is an existing dataset of volleyball games. We present a new dataset for actor recognition in rugby. We compare the results to using only the image of the actor.

Access to Document (DOI)

Swimmer's stroke estimation using CNN and MultiLSTM

Kohei Hakozaki, Noki Kato, Masamoto Tanabiki, Junko Furuyama, Yuji Sato, Yoshimitsu Aoki

Journal of Signal Processing (RISP) 22 ( 4 ) 219 - 222 2018.07

Research paper (scientific journal), Joint Work, Accepted

Player pose analysis in tennis video based on pose estimation

Kurose R., Hayashi M., Ishii T., Aoki Y.

2018 International Workshop on Advanced Image Technology, IWAIT 2018 (2018 International Workshop on Advanced Image Technology, IWAIT 2018) 1 - 4 2018.05

ISSN 9781538626153

© 2018 IEEE. The demand for sports video analysis is expanding. Sports video analysis is used to analyze their own play and play of opponent players, and visualize movement and ability of players. Scientific and objective analysis become possible by incorporating video analysis. It is desired to improve the competition level by feeding back the analysis results to the players. Therefore, in this research, in the case of feeding back the analysis result to the athlete, research purpose is to realize a method which can understand and evaluate the details of the form hitting the ball in detail. First, joint position coordinates are estimated from input RGB tennis images by a posture estimation method. The joint position coordinates in each frame at the time of shot are classified using unsupervised method and represented by BoW. The feature vector is designed by combining this with the shot position. The probability of shot success is predicted using this feature vector. By visualizing BoW of the shot with the high probability of success and the high failure probability, it is possible to extract and compare poses that are likely to appear in each case without giving correct labels.

Access to Document (DOI)

Pose alignment for different objects using affordance cues

Akizuki S., Aoki Y.

2018 International Workshop on Advanced Image Technology, IWAIT 2018 (2018 International Workshop on Advanced Image Technology, IWAIT 2018) 1 - 3 2018.05

ISSN 9781538626153

© 2018 IEEE. In this paper, we propose a method to estimate 6D pose (3D position and pose) of the everyday objects, especially tools, even if there are not the same 3D model of the target object. We assume that the tools that belong to same category have common spatial relation in part-affordances, the role of each part. The proposed method uses this assumption for pose estimation. By minimizing two types of consistencies, the spatial relationship of part-affordances and the shapes, we have confirmed that the proposed method achieved higher performance compared with traditional pose estimation method.

Access to Document (DOI)

Temporal and Fine-grained Pedestrian Action Recognition on Driving Recorder Database

Hirokatsu Kataoka, Yutaka Satoh, Yoshimitsu Aoki, Shoko Oikawa, Yasuhiro Matsui

Sensors (MDPI) 18 ( 2 ) 2018.02

Research paper (scientific journal), Joint Work, Accepted, ISSN 14248220

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. The paper presents an emerging issue of fine-grained pedestrian action recognition that induces an advanced pre-crush safety to estimate a pedestrian intention in advance. The fine-grained pedestrian actions include visually slight differences (e.g., walking straight and crossing), which are difficult to distinguish from each other. It is believed that the fine-grained action recognition induces a pedestrian intention estimation for a helpful advanced driver-assistance systems (ADAS). The following difficulties have been studied to achieve a fine-grained and accurate pedestrian action recognition: (i) In order to analyze the fine-grained motion of a pedestrian appearance in the vehicle-mounted drive recorder, a method to describe subtle change of motion characteristics occurring in a short time is necessary; (ii) even when the background moves greatly due to the driving of the vehicle, it is necessary to detect changes in subtle motion of the pedestrian; (iii) the collection of large-scale fine-grained actions is very difficult, and therefore a relatively small database should be focused. We find out how to learn an effective recognition model with only a small-scale database. Here, we have thoroughly evaluated several types of configurations to explore an effective approach in fine-grained pedestrian action recognition without a large-scale database. Moreover, two different datasets have been collected in order to raise the issue. Finally, our proposal attained 91.01% on National Traffic Science and Environment Laboratory database (NTSEL) and 53.23% on the near-miss driving recorder database (NDRDB). The paper has improved +8.28% and +6.53% from baseline two-stream fusion convnets.

Access to Document (DOI)

Video-Based Person Re-identification by 3D Convolutional Neural Networks and Improved Parameter Learning

Kato N., Hakozaki K., Tanabiki M., Furuyama J., Sato Y., Aoki Y.

ISSN 9783319929996

© 2018, Springer International Publishing AG, part of Springer Nature. In this paper we propose a novel approach for video-based person re-identification that exploits convolutional neural networks to learn the similarity of persons observed from video camera. We take 3-dimensional convolutional neural networks (3D CNN) to extract fine-grained spatiotemporal features from the video sequence of a person. Unlike recurrent neural networks, 3D CNN preserves the spatial patterns of the input, which works well on re-identification problem. The network maps each video sequence of a person to a Euclidean space where distances between feature embeddings directly correspond to measures of person similarity. By our improved parameter learning method called entire triplet loss, all possible triplets in the mini-batch are taken into account to update network parameters. This parameter updating method significantly improves training, enabling the embeddings to be more discriminative. Experimental results show that our model achieves new state of the art identification rate on iLIDS-VID dataset and PRID-2011 dataset with 82.0%, 83.3% at rank 1, respectively.

Access to Document (DOI)

Simultaneous execution of color adjustment and image completion by generative adversarial network

Akimoto N., Hayashi M., Akizuki S., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 84 ( 12 ) 1033 - 1040 2018

ISSN 09120289

In this paper, we address the problem of performing natural paste synthesis by color adjustment and image completion, in order to solve the completion problem that can specify an object appearing in a completion area. We propose a synthesis network that can extract the context features of the input image and reconstruct an image with the feature, making the inserted object appear in the completion region. In addition, we propose a ingenious method to make input images and learning method using Generative Adversarial Network (GAN) that do not require collection of high cost learning data. We show that color adjustment and image completion based on context features are executed at the same time, and natural pasting synthesis can be performed by using these proposal methods.

Access to Document (DOI)

Online multi-object tracking with tracklet re-identification

Kato N., Tanabiki M., Furuyama J., Sato Y., Aoki Y.

Journal of the Institute of Image Electronics Engineers of Japan (Journal of the Institute of Image Electronics Engineers of Japan) 47 ( 4 ) 391 - 400 2018

ISSN 02859831

One of the major factors that makes the task of tracking multiple objects difficult is frequent occlusions in congestion situations. Many existing methods in multi-object tracking by online processing have adopted a tracking-by-detection approach to perform object detection in every frame of a video and associate the object bounding box obtained by it temporally. However, with the existing method, it was not possible to track objects that could not be acquired by detectors due to occlusion. Therefore, we propose a method to shift the state of the lost target to tracked state once again by tracklet re-identification. Embeddings expressing the high dimensional appearance features of a object are acquired by using a convolutional neural network and re-identification of the tracklet is made based on the distance of the embedding vector among the tracklets. At this time, by using the masked image of the object obtained by instance segmentation as the input of the network, it is possible to perform re-identification determination robust to the background change. Further, since the re-identification determination of the tracklet pair is performed based on the distance between the low-dimensional vectors, the increase in the calculation cost due to the re-identification processing is small. Experiments were carried out using MOT16 dataset, which is a public data set, and the effectiveness of this method is shown.

Access to Document (DOI)

行動遷移映像における姿勢特徴を中心とする学習を用いた時系列行動認識

鈴木智之，青木義満

精密工学会誌 (精密工学会) 83 ( 12 ) 1156 - 1165 2017.12

Research paper (scientific journal), Joint Work, Accepted

畳み込みニューラルネットワークによる距離学習を用いた動画像人物再同定

加藤直樹，箱崎浩平，田靡雅基，里雄二，古山純子，青木義満

精密工学会誌 (精密工学会) 83 ( 12 ) 1117 - 1124 2017.12

Research paper (scientific journal), Joint Work, Accepted

CNN-based Spine and Cobb Angle Estimator Using Moire Images

Ran CHOI, Kota WATANABE, Hiroaki JINGUJI, Nobuyuki FUJITA, Yoji OGURA, Satoru DEMURA, Toshiaki KOTANI, Kanichiro WADA, Masashi MIYAZAKI, Hideki SHIGEMATSU, Yoshimitsu AOKI

IIEEJ Transactions on Image Electronics and Visual Computing (IIEEJ) 5 ( 2 ) 135 - 144 2017.12

Research paper (scientific journal), Joint Work, Accepted

ラグビー映像解析システムの開発

大内一成、小林大祐、中洲俊信、青木義満

電子情報通信学会通信ソサイエティ和文論文誌 (電子情報通信学会) J100-B ( 12 ) 941 - 951 2017.12

Research paper (scientific journal), Joint Work, Accepted

Unconstrained and Calibration-free Gaze Estimation in a Room-scale Area using a Monocular Camera

Kimimasa Tamura, Ran Choi, Yoshimitsu Aoki

IEEE Access (IEEE) PP ( 99 ) 1 - 12 2017.08

Research paper (scientific journal), Joint Work, Accepted

CNNを用いた姿勢特徴抽出と行動遷移を考慮した人物行動認識

工藤雄太，指田岳彦，青木義満

電子情報通信学会和文論文誌D (電子情報通信学会) J100-D ( 7 ) 681 - 691 2017.07

Research paper (scientific journal), Joint Work, Accepted

Detecting Conversational Group Associates from Static Images

Shoichi Inaba, Yoshimitsu Aoki

Journal of Signal Processing (Research Institute of Signal Processing) 21 ( 4 ) 199 - 202 2017.07

Research paper (scientific journal), Joint Work, Accepted

Behavior Monitoring System by using Vision Sensors

AOKI Yoshimitsu

IEICE Communication Society Magazine (IEICE) 11 ( 1 ) 30 - 38 2017.06

Research paper (scientific journal), Single Work, Accepted

よりよい動作を素早く提案するシステム- 家庭用デジタルビデオカメラを利用して

岡村麻人，石井壮郎，林昌希，青木義満，黒瀬龍之介，窪田辰政，三橋大輔

スポーツパフォーマンス研究 9 146 - 156 2017.03

Research paper (scientific journal), Joint Work, Accepted

Moving object proposal by grouping with motion feature

鈴木哲平, 高山将太, 磯部翔, 増田誠, 青木義満

精密工学会誌 (精密工学会) 83 ( 2 ) 151 - 157 2017.02

Research paper (scientific journal), Joint Work, Accepted

Feasibility of a Respiratory Movement Evaluation Tool to quantify　thoracoabdominal movement for neuromuscular diseases

Fumio Liu, Michiyuki Kawakami, Kimimasa Tamura, Yoshihito Taki, Katsumi Shimizu, Tomoyoshi Otsuka, Tetsuya Tsuji, Chieko Miyata, Syoichi Tashiro, , Ayako Wada, Katsuhiro Mizuno, Yoshimitsu Aoki, Meigen Liu

Respiratory Care (Thomson Reuters) 62 ( 4 ) 421 - 431 2016.12

Research paper (scientific journal), Joint Work, Accepted

Parsing Human Skeletons in an Operating Room

Vasileios Belagiannis; Xinchao Wang; Horesh Beny Ben Shitrit; Kiyoshi Hashimoto; Ralf Stauder, Yoshimitsu Aoki, Michael Kranzfelder, Armin Schneider, Pascal Fua. Slobodan Ilic, Hubertus Feussner, Nassir Navab

Machine Vision and Applications (Springer) 27 ( 7 ) 1035 - 1046 2016.07

Research paper (scientific journal), Joint Work, Accepted

Multiple Objects Extraction from Aerial Imagery with Convolutional Neural Networks

Shunta Saito, Takayoshi Yamashita, Yoshimitsu Aoki

Journal of Imaging Science and Technology 60 ( 1 ) 10402-1 - 10402-9 2016.01

Research paper (scientific journal), Joint Work, Accepted

Access to Document (DOI)

Gaze Control and Dexterous Manipulation of Infants' Goal-Directed Actions - A Comparison between 11 month olds and 18 month olds -

Kazuhiro Abe, Kyohei Otsuka, Yoshimitsu Aoki and Yasuyo Minagawa

人間と社会の探究 - 慶應義塾大学大学院社会学研究科紀要 - ( 82 ) 17 - 35 2016

Research paper (bulletin of university, research institution), Joint Work

Seamline Determination Based on Semantic Segmentation for Aerial Image Mosaicking

Shunta Saito, Ryota Arai, Yoshimitsu Aoki

IEEE Access (IEEE) 3 2847 - 2856 2015.12

Research paper (scientific journal), Joint Work, Accepted, ISSN 2169-3536

Access to Document (DOI)

People Tracking in Dense Crowds using Optical Flow Clustering based Superpixel

高山将太，片岡裕雄，磯部翔，栗田尚樹，増田誠，青木義満

精密工学会論文誌 (精密工学会) 82 ( 3 ) 259 - 265 2015.12

Research paper (scientific journal), Joint Work, Accepted

Upper Body Pose Estimation for Team Sports Videos Using Poselets-Regressor of Spine Pose and Body Orientation Classifiers Conditioned by the Spine Angle Prior

Masaki Hayashi, Kyoko Ohshima, Masamoto Tanabiki, Yoshimitsu Aoki

IPSJ Transactions on Computer Vision and Applications (IPSJ) 7 121 - 137 2015.10

Research paper (scientific journal), Joint Work, Accepted

Human Tracking with Statistical Shape Model and Rough Pose Estimation by Random Forest

橋本潔，片岡裕雄，里雄二，田靡雅基，青木義満

精密工学会誌 (精密工学会) 81 ( 12 ) 1162 - 1167 2015.09

Research paper (scientific journal), Joint Work, Accepted

Ball Trajectory Extraction in Team Sports Videos by Focusing on Ball Holder Candidates for the Play Search & 3D Virtual Display System

Junji Kurano, Masaki Hayashi, Taiki, Yamamoto, Hirokatsu Kataoka, Masatomo Tanabiki, Junko Furuyama, Yoshimitsu Aoki

Journal of Signal Processing (Research Institute of Signal Processing) 19 ( 4 ) 2015.07

Research paper (scientific journal), Joint Work, Accepted

Relationship of Bitterness and Laryngeal Movement Revealed Using a Non-contact and Non-invasive Swallowing Function Evaluation Instrument

Hiroshi Furuuchi, Keita Tahata, Mai Nakajima, Toshio Shoji, Hitoshi Kawahara, Yoshimitsu Aoki

日本摂食嚥下リハビリテーション学会誌 (日本摂食嚥下リハビリテーション学会) 19 ( 1 ) 33 - 40 2015.04

Research paper (scientific journal), Accepted

Lower Body Pose Estimation in Team Sports Videos using Label-Grid Classifier integrated with Tracking-by-Detection

Masaki Hayashi, Kyoko Ohshima, Masamoto Tanabiki, Yoshimitsu Aoki

IPSJ Transactions on Computer Vision and Applications (IPSJ) 7 18 - 30 2015.02

Research paper (scientific journal), Joint Work, Accepted

アメリカンフットボール映像におけるプレー開始・終了位置と選手動作情報を用いたボール軌跡抽出

藏野隼二，林昌希，山本大樹，大島京子，田靡雅基，青木義満

精密工学会誌 (精密工学会) 81 ( 1 ) 91 - 98 2015.01

Research paper (scientific journal), Joint Work, Accepted

Robust Pose Estimation for Human Body Hidden with a Cover using Body Shape Feature obtained by Normal Vector

工藤雄太，指田岳彦，青木義満

精密工学会誌 (精密工学会) 80 ( 12 ) 1166 - 1175 2014.12

Research paper (scientific journal), Joint Work, Accepted

スーパーピクセル上の接平面に対する光線追跡を用いた距離画像の補間

Kiyoshi Matsuo, Yoshimitsu Aoki

精密工学会誌 (精密工学会) 80 ( 12 ) 2014.06

Research paper (scientific journal), Joint Work, Accepted

頭部姿勢変動に頑健なまぶた及び黒目の追跡手法

Kimimasa Tamura, Kiyoshi Hashimoto, Yoshimitsu Aoki

電気学会論文誌C (電気学会) 134 ( 5 ) 694 - 701 2014.05

Research paper (scientific journal), Joint Work, Accepted

These days, there is more demand of camera based gaze estimation method for a new interface and a new marketing measurement tool. Considering these applications, the system should track a new user without any operation like calibrations. It also admits user's natural head pose changes. Previous methods, however, need calibration procedure before execution and have less accuracy under head moving situation. In this paper, we propose the method which tracks user's eyelid and iris automatically and accurately. Our method is the pretreatment of gaze estimation without any calibration and head pose restraint. First of all we track the facial feature points from an input face image and estimate its head pose, extracting eye region image. On the eye region image, we track eyelid shape based on the eyelid shape model generated beforehand from PCA. Finally we track iris inside the eyelid based on the eye ball model. These eyelid and iris tracking are processed by Particle Filter. From the evaluation of database including head pose changes, we confirmed that accuracy of the eyelid and iris tracking is improved compared with previous methods.

スーパーピクセル上の接平面を利用した奥行き画像の補間

Kiyoshi Matsuo, Yoshimitsu Aoki

電気学会論文誌C (電気学会) 134 ( 4 ) 2014.04

Research paper (scientific journal), Joint Work, Accepted

日常生活での応用に適した視線推定に向けた高精度なまぶた及び黒目の追跡手法

田村仁優，橋本潔，青木義満

画像ラボ (日本工業出版) 25 ( 3 ) 16-23 2014.03

Research paper (scientific journal), Joint Work

Extended Feature Descriptor and Vehicle Motion Model with Tracking-by-detection for Pedestrian Active Safety

Hirokatsu Kataoka, Kimimasa Tamura, Kenji Iwata, Yutaka Satoh, Yasuhiro Matsui, Yoshimitsu Aoki

IEICE TRANSACTIONS on Information and Systems (IEICE) E-97D ( 2 ) 296-304 2014.02

Research paper (scientific journal), Joint Work, Accepted

Emerging Trend on Image Recognition Technologies

Yoshimitsu Aoki, Hironobu Fujiyoshi

非破壊検査 (非破壊検査協会) 63 ( 1 ) 18-23 2014.01

Research paper (scientific journal), Joint Work

Extended Feature and Motion with Tracking-by-detection for Pedestrian Active Safety

Hirokatsu Kataoka, Yasuhiro Matsui, Yoshimitsu Aoki

Journal of Society of Automotive Engineers of Japan (自動車技術会) 67 ( 12 ) 79-83 2013.12

Research paper (scientific journal), Joint Work

統計的人体形状モデルを用いた遮蔽や姿勢変動に頑健な人物追跡

橋本潔、加賀屋智之、片岡裕雄、里雄二、青木義満

画像電子学会誌 (画像電子学会) 42 ( 5 ) 681-689 2013.09

Research paper (scientific journal), Joint Work, Accepted

Apnea Detection Method for Cheyne - Stokes Respiration Analysis on Newborn

Taiga Niimi, Yushi Itoh, Michiya Natori, Yoshimitsu Aoki

International Journal of Optomechatronics (Taylor & Francis production department) 7 ( 2 ) 67-82 2013.05

Research paper (scientific journal), Joint Work, Accepted

Cavitation during high-temperature deformation in Al–Mg alloys

Hiroyuki Toda,Zul Azri Bin Shamsudina, Kazuyuki Shimizu, Kentaro Uesugi, Akihisa Takeuchi, Yoshio Suzuki, Mitsuru Nakazawa, Yoshimitsu Aoki, Masakazu Kobayashi

Acta Materialia (Science Direct) 61 ( 7 ) 2403-2413 2013.04

Research paper (scientific journal), Joint Work, Accepted

Compression and recovery micro-mechanisms in flexible graphite

Hiroyuki Toda, Kodai Tsubone, Kazuyuki Shimizu, Kentaro Uesugi, Akihisa Takeuchi, Yoshio Suzuki, Mitsuru Nakazawa, Yoshimitsu Aoki, Masakazu Kobayashi

Carbon (Elsevier) 59 184-191 2013.03

Research paper (scientific journal), Joint Work, Accepted

知覚融合センシングの現状と展望

AOKI Yoshimitsu

電気学会論文誌Ｃ (電気学会) 133 ( 3 ) 630-634 2013.03

Research paper (scientific journal), Single Work

高温容体化したAl-6Si-4Cu合金鋳物のポアと疲労特性の関係

稲森隆晃，戸田裕之，小林正和，中澤満，青木義満，高橋明宏

日本鋳造工学 (日本鋳造工学会) 85 ( 2 ) 69-75 2013.02

Research paper (scientific journal), Joint Work, Accepted

航空画像を用いた建物検出及び変更検出

中村智志，青木義満

画像電子学会論文誌 (画像電子学会) 42 ( 1 ) 25-29 2013.01

Research paper (scientific journal), Joint Work, Accepted

Cheyne-Stokes Respiration Detection Method for Newborns with Apnea

Taiga Niimi, Yushi Itoh, Michiya Natori, Yoshimitsu Aoki

ITE Transactions on Media Technology and Applications (ITE) 1 ( 4 ) 278 - 291 2013

Research paper (scientific journal), Joint Work, Accepted

画像の計測と認識，その産業応用の実用化動向

AOKI Yoshimitsu

OPlusE (アドコムメディア) 35 ( 1 ) 21-22 2012.12

Research paper (scientific journal), Single Work

Improvement CoHOG Detection and Texture Matching Tracking for Pedestrian's Active Safety

片岡裕雄，田村仁優，松井靖浩，青木義満

電気学会論文誌C (電気学会) 132 ( 7 ) 1108-1115 2012.07

Research paper (scientific journal), Joint Work, Accepted

車載カメラとセンシング技術の進化　総論

AOKI Yoshimitsu

OplusE (アドコムメディア) ( 6 ) 2012.06

Research paper (scientific journal), Single Work

Human Behavior Understanding by Integrating Statistical Modeling and Image Recognition

AOKI Yoshimitsu

電子情報通信学会誌 (電子情報通信学会) 95 ( 5 ) 410-415 2012.05

Research paper (scientific journal), Single Work

シルエット画像を用いたモデルベース人体体幹部形状推定

齋藤俊太，青木義満

画像ラボ (日本工業出版) 23 ( 3 ) 19-24 2012.03

Research paper (scientific journal), Joint Work

姿勢変動に伴う身体的特徴変化の統計的モデリングによる遮蔽に頑健な人物追跡

橋本潔，加賀屋智之，片岡裕雄，藤田光子，田靡雅基，野上建士，青木義満

映像情報インダストリアル (産業開発機構) 44 ( 3 ) 41-50 2012.03

Research paper (scientific journal), Joint Work

単眼カメラを用いたサッカー映像解析のための複数選手とボールの追跡

片岡裕雄，青木義満

画像電子学会誌 (画像電子学会) 40 ( 2 ) 152-159 2012.03

Research paper (scientific journal), Joint Work

2カメラFiber Grating呼吸モニタを用いた慢性閉塞性肺疾患症状のスクリーニング

佐藤勲，青木義満

日本医用画像工学会論文誌 (日本医用画像工学会) 30 ( 2 ) 73-82 2012.03

Research paper (scientific journal), Joint Work, Accepted

安全運転カウンセリングシステムの構築

春日伸予，青木義満

タカタ財団助成研究論文集　平成22年度版 (タカタ財団) 1-122 2012.01

Research paper (conference, symposium, etc.), Joint Work, ISSN 2185-5188

Symmetrical Judgment Area Reduction and ECoHOG Feature Descriptor for Pedestrian Detection

Hirokatsu Kataoka, Yasuhiro Matsui, Yoshimitsu Aoki

International Journal of VEHICLE SAFETY (INDERSCIENCE) 6 ( 1 ) 48-60 2012

Research paper (scientific journal), Joint Work, Accepted

感性計測・技能計測が拓く産業の画像技術

青木義満，加藤邦人，橋本洋志，舟橋琢磨，藤原孝幸

映像情報メディア学会誌 (映像情報メディア学会) 65 ( 11 ) 1524-1533 2011.11

Research paper (scientific journal), Joint Work

Three-dimensional strain mapping using in situ X-ray synchrotron microtomography

H Toda, E Maire, Y Aoki, M Kobayashi

Journal of Strain Analysis for Engineering Design (Sage Journals) 46 ( 7 ) 549-561 2011.10

Research paper (scientific journal), Joint Work, Accepted

人を計測,認識,理解するための画像センシング　~顔研究をきっかけとして~

AOKI Yoshimitsu

日本顔学会誌第11巻 (日本顔学会) 11 5-12 2011.09

Research paper (scientific journal), Single Work

非接触新生児呼吸モニタリングシステム

AOKI Yoshimitsu

OPlusE Vol.33 No.6 (アドコム・メディア株式会社) 575-578 2011.05

Research paper (scientific journal), Single Work

サブミクロンオーダ3D-CT画像を用いた材料内部の変形破壊特性解析の高精度化

中澤　満，小林正和，戸田裕之，青木義満

電気学会論文誌D (電気学会) 131 ( 4 ) 2011.04

Research paper (scientific journal), Joint Work, Accepted

炒め調理における関心領域の時系列温度計測

中澤満，赤坂梨絵，笠松千夏，青木義満

計測自動制御学会論文誌 (計測自動学会) 47 ( 3 ) 2011.03

Research paper (scientific journal), Joint Work, Accepted

サッカー映像解析のための遮蔽にロバストな複数選手追跡手法

片岡裕雄，青木義満

電気学会論文誌C (電気学会) 130 ( 11 ) 2058-2064 2010.11

Research paper (scientific journal), Joint Work, Accepted

一台のカメラで撮影した映像からグラウンド上の選手軌跡を取得する手法を提案する.追跡にはParticle Filter を用いているが,ここではさらにオクルージョンに対応するために選手の密集した領域を検出し,重心を配置しなおすことで複雑な環境の追跡を可能にしている.また,実際の試合映像中の複雑な場面に対して提案手法を適用することにより有効性を示した.

FG視覚センサを用いた非接触新生児呼吸モニタリングシステムの開発

倉見義幸，大関和夫，伊藤裕司，名取道也，青木義満

電気学会論文誌C (電気学会) 130 ( 9 ) 1581-1587 2010.09

Research paper (scientific journal), Joint Work, Accepted

FG視覚センサを用いた新生児用の非接触呼吸モニタリングシステムの開発を行った.FG視覚センサとは,赤外光の輝点群を照射する輝点投降機とCCDカメラにより構成される,アクティブステレオ法により物体表面の動きをリアルタイムで検出することができるセンサである. これにより呼吸の際に生じる胸郭や腹部の動きを非接触・無侵襲で検出することが可能である.臨床実験を行い,本システムにより得られる呼吸波形は極めて信頼性が高く,従来の呼吸モニタリング装置では得られない詳細な呼吸波形を取得できることが確認できた.

映像情報メディア年報〜2008年4月から2010年3月の進展〜

吉田俊之、入江豪、佐藤隆、芹沢昌宏、多田昌裕、田川憲男、広明敏彦、青木義満

映像情報メディア学会誌 (映像情報メディア学会) 64 ( 8 ) 1190-1197 2010.08

Research paper (scientific journal), Joint Work

Four-Dimensional Annihilation Behaviors of Micro Pores during Surface Cold Working

Hiroyuki Toda, Tomoyasu Yamaguchi, Mitsuru Nakazawa, Yoshimitsu Aoki, Kentaro Uesugi, Yoshio Suzuki and Masakazu Kobayashi

Materials Transactions (The Japan Institute of Metals) 51 ( 7 ) 1288-1295 2010.07

Research paper (scientific journal), Joint Work, Accepted

ミクロポアがアルミニウム合金ダイカストの強度・延性に及ぼす影響の統計的解析

戸田裕之、小林正和、伊藤真也、中澤満、青木義満、堀川宏、鈴木聡

鋳造工学 (日本鋳造工学会) 82 ( 7 ) 427-432 2010.07

Research paper (scientific journal), Joint Work, Accepted

解剖学的構造としわを手がかりとした手部寸法の画像計測

AOKI Yoshimitsu

電気学会論文誌C (電気学会) 130 ( 4 ) 686-695 2010.04

Research paper (scientific journal), Joint Work, Accepted

A hand version of the computer manikin, called “digital hand”, is considered as an effective assessment tool for hand-operated products at the design stage. In order to construct various sizes and proportions of hands in a computer, it is required to develop the system that can measure the accurate dimensions of hands automatically.
In this paper, an image-based measurement system of hand dimensions has been proposed. The system utilizes an image scanner for acquiring a hand image in 2D. First, the joint positions of fingers are estimated by extracting the edges of the wrinkles on a palm and fingers in this image. For this process, anatomical constraint is effectively considered. Then, the 2D dimensions of a hand are automatically calculated. In addition, based on the fact that the dimensions of a hand are highly correlated each other, the system estimates the 3D measures of a hand, such as the thickness and circumference of fingers and a palm by regression analysis.
The accuracy and reproducibility was validated using the proposed system by comparison with actual results measured by traditional method. The results prove our system has enough capability for actual hand measurement and its applications.

A Proposal of a Method to Analyze 3D Deformation / Fracture Characteristics inside Materials Based on a Stratified Matching Approach

NAKAZAWA MitsuruAOKI Yoshimitsu

Machine Vision and Applications (Springer Berlin) 21 ( 5 ) 687-694 2010.02

Research paper (scientific journal), Joint Work, Accepted

A 3-dimensional method for analyzing the morphology of patients with maxillofacial deformities

Masahiko Terajima, Akihiko Nakasima, Yoshimitsu Aoki, Tazuko K. Goto, Kenji Tokumori, Noriko Mori, Yoshihiro Hoshino

American Journal of Orthodontics and Dentofacial Orthopedics (Elsevier) 136 ( 6 ) 857-867 2009.12

Research paper (scientific journal), Joint Work, Accepted

中華炒め調理におけるシェフの「鍋のあおり」が具材と鍋温度変化に及ぼす影響

AOKI Yoshimitsu

日本調理科学会誌 (日本調理科学会) 42 ( 5 ) 334-341 2009.10

Research paper (scientific journal), Joint Work, Accepted

Development of a breathing monitoring system using fiber-grating 3D sensor and two cameras

AOKI Yoshimitsu

画像電子学会論文誌 (画像電子学会) 38 ( 4 ) 385-394 2009.07

Research paper (scientific journal), Single Work, Accepted

A 3-dimensional method for analyzing facial soft-tissue morphology of patients with jaw deformities

Masahiko Terajima, Yoshihide Furuichi, Yoshimitsu Aoki, Tazuko K. Goto, Kenji Tokumori，Akihiko Nakasima

American Journal of Orthodontics and Dentofacial Orthopedics 135 ( 6 ) 715-722 2009.06

Research paper (scientific journal), Joint Work, Accepted

Trends and Issues of Image Recognition Technology in Social Infrastructure

下村倫子 , 佐藤雄隆 , 青木義満

精密工学会誌 (精密工学会) 75 ( 2 ) 220-223 2009.02

Research paper (scientific journal), Joint Work, Accepted

レスキュー用アレイアンテナ搭載レーダシステム

AOKI Yoshimitsu

Transactions of the Society of Instrument and Control Engineers (計測自動制御学会) 44 ( 12 ) 962-969 2008.12

Research paper (scientific journal), Joint Work, Accepted

High accurate geometric correction for NOAA AVHRR data considering elevation effect and coastline feature

An Ngoc Van, Mitsuru Nakazawa, Yoshimitsu Aoki

IEICE Trans. On Communications (IEICE) E91-B ( 9 ) 2956-2963 2008.09

Research paper (scientific journal), Joint Work, Accepted

Four-dimensional analysis of stomatognathic function

Masahiko Terajima, Mizuki Endo, Yoshimitsu Aoki, Kyouko Yuuda, Haruaki Hayasaki, Tazuko K. Goto, Kenji Tokumori and Akihiko Nakasima

American Journal of Orthodontics and Dentofacial Orthopedics 134 ( 2 ) 276-287 2008.08

Research paper (scientific journal), Joint Work, Accepted

Three-dimensional computer-generated individual's head model reconstructed from cephalograms, facial photographs and dental cast models

Terajima M, Yanagita N, Ozeki K, Hoshino Y, Mori N, Goto TK, Tokumori K, Aoki Y, Nakasima A

American Journal of Orthodontics and Dentofacial Orthopedics 134 ( 1 ) 100-111 2008.07

Research paper (scientific journal), Joint Work, Accepted

High-density three-dimensional mapping of internal strain by tracking microstructural features

Masakazu Kobayashi, Hiroyuki Toda, Yuji Kawai, Tomomi Ohgaki, Kentaro Uesugi, David S. Wilkinson, Toshiro Kobayashi, Yoshimitsu Aoki, Mitsuru Nakazawa

Acta Materialia 56 ( 10 ) 2167-2181 2008.06

Research paper (scientific journal), Joint Work, Accepted

形成外科画像計測-顎外科矯正手術における手術支援システム

AOKI Yoshimitsu

計測と制御 (計測自動制御学会) 47 ( 1 ) 47-51 2008.01

Research paper (scientific journal), Single Work, Accepted

Four-dimensional visualization of mandibular movement using an individual's head model reconstructed from cephalograms

Kyouko Yuuda, Masahiko Terajima, Yoshimitsu Aoki, Haruaki Hayasaki, Akihiko Nakasima

Orthodontic Waves 66 ( 4 ) 113-121 2007.12

Research paper (scientific journal), Joint Work, Accepted

濃煙空間の計測・可視化による消防活動支援 -リアルタイムで煙の中を見透かすシステムの開発-

AOKI Yoshimitsu

計測と制御 (計測自動制御学会) 46 ( 8 ) 650-654 2007.08

Research paper (scientific journal), Single Work, Accepted

高鮮度ナビゲーションデータ作成支援のための道路案内標識の自動検出

AOKI Yoshimitsu

画像電子学会論文誌 36 ( 4 ) 456-464 2007.06

Research paper (scientific journal), Joint Work, Accepted

材料内部の変形・破壊特性評価のための３次元画像解析

中澤満, 青木義満, 小林正和, 戸田裕之

電気学会論文誌Ｃ (電気学会) 127 ( 6 ) 824-830 2007.06

Research paper (scientific journal), Joint Work, Accepted

高分解能X線CTイメージ解析と特徴点追跡法による三次元ひずみ計測

AOKI Yoshimitsu

日本金属学会誌 71 ( 2 ) 181-186 2007.02

Research paper (scientific journal), Joint Work, Accepted

Precise error correction method for NOAA AVHRR image using the same orbital images

An Ngoc Van, Yoshimitsu Aoki

Transactions on Electrical Eng., Electronics, and Communications 5 ( 2 ) 127-136 2007

Research paper (scientific journal), Joint Work, Accepted

Human and Object Detection in dense Smoke-filled Space using Millimeter-wave Radar based Measurement System

Masaki Sakai, Yoshimitsu Aoki

Journal of Robotics and Mechatronics 18 ( 6 ) 760-764 2006.12

Research paper (scientific journal), Joint Work, Accepted

実体模型操作と三次元画像表示の統合による顎矯正手術支援システム

下永吉達成 , 青木義満 , 不島健持 , 小林優

電子情報通信学会論文誌D 情報システム (電子情報通信学会) J89-D ( 10 ) 2164-2173 2006.10

Research paper (scientific journal), Joint Work, Accepted

顎口腔機能の四次元的解析システム～顎変形症患者への応用

寺嶋雅彦, 遠藤みずき, 青木義満, 祐田京子, 早崎治明, 後藤多津子, 徳森謙二, 中島昭彦

日本顎口腔機能学会雑誌 (日本顎口腔機能学会) 12 ( 2 ) 100-109 2006.02

Research paper (scientific journal), Joint Work, Accepted

Machine Vision Technologies to Connect with Humans

梅田和昇，青木義満

電気学会誌 (電気学会) 125 ( 11 ) 692-695 2005.11

Research paper (scientific journal), Joint Work

顎変形症患者における顎顔面形態の三次元統合システムの開発

寺嶋雅彦 , 大関加奈子 , 青木義満 , 祐田京子 , 後藤多津子 , 徳森謙二 , 樋口惣 , 中島昭彦

日本矯正歯科学会雑誌 (日本矯正歯科学会) 64 ( 2 ) 114-124 2005.06

Research paper (scientific journal), Joint Work, Accepted

Analysis of 109 Japanese childlen’s lip and nose shapes using 3-dimensional digitizer

A.Mori, T.Nakajima, T Kaneko, H.Sakuma, Y.Aoki

British Journal of Plastic Surgery 58 ( 3 ) 318-329 2005.04

Research paper (scientific journal), Joint Work, Accepted

Three-dimensional computer-generated individual's head model reconstructed from cephalograms, facial photographs and dental cast models

Akihiko NAKASIMA, Masahiko TERAJIMA,Noriko MORI,Yoshihiro HOSHINO, Kenji TOKUMORI,Yoshimitsu AOKI, Shuji HASHIMOTO

American Journal of Orthodontics and Dentofacial Orthopedics 127 ( 3 ) 282-292 2005.03

Research paper (scientific journal), Joint Work, Accepted

頭部Ｘ線規格画像を用いた顎顔面骨格三次元形状可視化システムの構築

青木義満，宮田宗樹，寺嶋雅彦，中島昭彦

日本医用画像工学会論文誌 (日本医用画像工学会) 22 ( 5 ) 250-258 2004.11

Research paper (scientific journal), Joint Work, Accepted

顎変形症患者における下顎運動四次表示システム

寺嶋雅彦，祐田京子，青木義満，早崎治明，徳森謙二，後藤多津子，中島昭彦

Orthodontic Waves Japanese Edition 63 ( 3 ) 193-205 2004.10

Research paper (scientific journal), Joint Work, Accepted

映像タグ付け支援を目的とした投球シーン自動アーカイブシステムの提案の自動検出

庄野雄紀，青木義満

画像電子学会論文誌 (画像電子学会) 33 ( 5 ) 729-736 2004.09

Research paper (scientific journal), Joint Work, Accepted

顎歯列形態・機能解析を目的とした四次元顎運動可視化システムの開発

青木義満，石川善章，寺嶋雅彦，中島昭彦

日本医用画像工学会論文誌 (日本医用画像工学会) 22 ( 4 ) 225-233 2004.09

Research paper (scientific journal), Joint Work, Accepted

三次元CT画像を用いた現代日本人成人男女の頭部生体計測と顔面標準三次元物理モデルの構築

森紀子 , 寺嶋雅彦 , 徳森謙治 , 中島昭彦 , 青木義満 , 橋本周司

人類學雜誌 (日本人類学会) 111 ( 1 ) 35-49 2003.06

Research paper (scientific journal), Joint Work, Accepted

健常日本人成人男性11名, 女性10名を対象にHelicalCTを撮影し, 三次元構築した3D-CTの顔表面画像上に26個の特徴点を設定して男女別に平均の三次元座標値を求め, 11項目の特徴点間距離を測定した。また骨および顔表面画像上に35個の特徴点を設定しその部位の軟組織の厚さを計測した。これらの生体計測値はこれまでに報告された値に比べてわずかに大きな傾向にあった。平均座標をもとに構築した顔面形態の物理モデルは, 現代日本人成人男女の標準モデルとしてパーソナルコンピューター上で自由に回転表示でき, 人類学分野をはじめとした多分野で使用できる可能性が示唆された。

ディジタル画像を用いた高精度画像計測のための画像補正

中村真吾 , 青木義満 , 橋本周司 , 畑浩二

画像電子学会誌 (画像電子学会) 31 ( 4 ) 534-541 2002.07

Research paper (scientific journal), Joint Work, Accepted

Physics-Based 3D Head model Reconstruction from Cephalograms for Medical Application

Yoshimitsu Aoki, M.Terajima, A.Nakasima, S.Hashimoto

Electronics and Communications in Japan(Part III: Fundamental Electronic Science) (WILEY) 85 ( 11 ) 45-55 2002.05

Research paper (scientific journal), Joint Work, Accepted

日本人成人男女の三次元骨格基準モデルの構築

星野慶弘, 寺嶋雅彦, 森紀子, 青木義満, 橋本周司, 中島昭彦

日本矯正歯科学会雑誌 (日本矯正歯科学会) 61 ( 5 ) 334-342 2002.05

Research paper (scientific journal), Joint Work, Accepted

下顎前突症患者の下顎枝矢状分割術による形態変化の三次元的シミュレーション

寺嶋雅彦 , 青木義満 , 徳森謙治 , 橋本周司 , 中島昭彦

日本顎変形症学会雑誌 (日本顎変形症学会) 11 ( 3 ) 194-204 2001.12

Research paper (scientific journal), Joint Work, Accepted

3次元顔画像を用いた特徴点抽出と個人認識

小高一慶 , 青木義満 , 橋本周司

画像電子学会誌 (画像電子学会) 30 ( 4 ) 414-422 2001.07

Research paper (scientific journal), Joint Work, Accepted

Simulation of Postoperative 3D Facial Morphology using Physics-based Head Model

Yoshimitsu Aoki, Masahiko Terajima, Akihiko Nakasima, Shuji Hashimoto

The Visual Computer Journal (Springer) 17 ( 2 ) 121-131 2001.03

Research paper (scientific journal), Joint Work, Accepted

頭部X線規格画像を用いた3次元頭部物理モデルの構築とその臨床応用

青木義満, 寺嶋雅彦, 中島昭彦, 橋本周司

電子情報通信学会論文誌D2 (電子情報通信学会) J84D-Ⅱ ( 2 ) 390-399 2001.02

Research paper (scientific journal), Joint Work, Accepted

利用可能なデータに応じた3次元頭部モデルの構築とその応用

青木義満 , 橋本周司

画像電子学会誌 (画像電子学会) 29 ( 5 ) 497-505 2000.09

Research paper (scientific journal), Joint Work, Accepted

分散協調処理を用いたロバストな能動的人物追跡システム

青木義満 , 久富健介 , 橋本周司

画像電子学会誌 (画像電子学会) 28 ( 5 ) 596-604 1999.10

Research paper (scientific journal), Joint Work, Accepted

解剖学的知見に基づく顔の物理モデリングによる表情生成

青木義満, 橋本周司

電子情報通信学会論文誌A (電子情報通信学会) J82A ( 4 ) 573-582 1999.04

Research paper (scientific journal), Joint Work, Accepted

　 Previous page - Next page 　