Details of a Researcher - Aoki, Yoshimitsu

Papers - Aoki, Yoshimitsu

Division display 1 - 20 of about 192 ／ All the affair displays >>

Exploring the influence of judging experience on individual differences in figure skating jump performance evaluations: a case study of novice judges

Hirosawa S., Kato T., Aoki Y.

Sport Sciences for Health 22 ( 1 ) 2026.03

ISSN 18247490

Background: Judges in figure skating are tasked with evaluating the technical quality of skaters’ performances using the grade of execution (GOE) score, a process that requires rapid integration of multiple technical and qualitative visual cues under time constraints, despite the inherently subjective nature of the evaluation criteria. Previous research suggests that judging accuracy is more impacted by previous experience as an athlete. However, the influence of judging experience on individual differences in evaluation remains unclear. Aims: This study investigated whether novice judging experience reduces individual differences in figure skating jump evaluations. Methods: Three certified judges and three skaters with similar athletic abilities evaluated 30 double Axel jumps from the 2019 World Championships using video-based judging. Participants assigned GOE scores and evaluated six GOE criteria while eye movements were recorded, and individual differences were analyzed using (1) judging accuracy, (2) gaze coordinates, and (3) intraclass correlations coefficients. Results and Conclusions: Results indicated that novice judging experience was not associated with improved judging accuracy or reduced individual differences in final GOE scores. However, judging experience was associated with lower dispersion in gaze coordinates and higher intraclass correlations coefficients in GOE criterion-based evaluations. Notably, substantial individual variation persisted when evaluating jumps matched to music, suggesting a challenge for judge development programs.

Access to Document (DOI)

Interaction Recognition Based on Relative Spatial Positions Between Individuals

Suzuki M., Torimi K., Aoki Y.

Seimitsu Kogaku Kaishi Journal of the Japan Society for Precision Engineering 92 ( 2 ) 193 - 198 2026

ISSN 09120289

Interaction recognition between multiple individuals is crucial in many practical scenarios, particularly in sports analysis, yet remains challenging due to complex dynamics and directional ambiguity. Most existing methods primarily focus on single-person actions, which are insufficient for analyzing multi-person interactions. The SportsHHI dataset, designed specifically for human-human interactions in sports videos, provides detailed annotations but faces limitations from strict subject-object direction definitions, class imbalance, and inadequate encoding of relative spatial information. To overcome these issues, this study proposes three enhancements: (1) redefining relative spatial encoding to include global positional context, (2) introducing a directionality-agnostic evaluation method suitable for bidirectional interactions, and (3) employing focal loss to address class imbalance. Experimental results on SportsHHI demonstrate the effectiveness of the proposed improvements, achieving up to an increase 2.67 at mAP compared to baseline methods.

Access to Document (DOI)

BasketLiDAR: The First LiDAR-Camera Multimodal Dataset for Professional Basketball MOT

Hayashi R., Torimi K., Nagata R., Ikeda K., Sako O., Nakamura T., Tani M., Aoki Y., Yoshioka K.

Mmsports 2025 8th International ACM Workshop on Multimedia Content Analysis in Sports 78 - 86 2025.10

Real-time 3D trajectory player tracking in sports plays a crucial role in tactical analysis, performance evaluation, and enhancing spectator experience. Traditional systems rely on multi-camera setups, but are constrained by the inherently two-dimensional nature of video data and the need for complex 3D reconstruction processing, making real-time analysis challenging. Basketball, in particular, represents one of the most difficult scenarios in the MOT field, as ten players move rapidly and complexly within a confined court space, with frequent occlusions caused by intense physical contact. To address these challenges, this paper constructs BasketLiDAR, the first multimodal dataset in the sports MOT field that combines LiDAR point clouds with synchronized multi-view camera footage in a professional basketball environment, and proposes a novel MOT framework that simultaneously achieves improved tracking accuracy and reduced computational cost. The BasketLiDAR dataset contains a total of 4,445 frames and 3,105 player IDs, with fully synchronized IDs between three LiDAR sensors and three multi-view cameras. We recorded 5-on-5 and 3-on-3 game data from actual professional basketball players, providing complete 3D positional information and ID annotations for each player. Based on this dataset, we developed a novel MOT algorithm that leverages LiDAR's high-precision 3D spatial information. The proposed method consists of a real-time tracking pipeline using LiDAR alone and a multimodal tracking pipeline that fuses LiDAR and camera data. Experimental results demonstrate that our approach achieves real-time operation, which was difficult with conventional camera-only methods, while achieving superior tracking performance even under occlusion conditions. The dataset is available upon request at: https://sites.google.com/keio.jp/keio-csg/projects/basket-lidar.

Access to Document (DOI)

A Comprehensive Analysis of a Social Intelligence Dataset and Response Tendencies Between Large Language Models (LLMs) and Humans

Mori E., Qiu Y., Kataoka H., Aoki Y.

Sensors 25 ( 2 ) 2025.01

In recent years, advancements in the interaction and collaboration between humans and have garnered significant attention. Social intelligence plays a crucial role in facilitating natural interactions and seamless communication between humans and Artificial Intelligence (AI). To assess AI’s ability to understand human interactions and the components necessary for such comprehension, datasets like Social-IQ have been developed. However, these datasets often rely on a simplistic question-and-answer format and lack justifications for the provided answers. Furthermore, existing methods typically produce direct answers by selecting from predefined choices without generating intermediate outputs, which hampers interpretability and reliability. To address these limitations, we conducted a comprehensive evaluation of AI methods on a video-based Question Answering (QA) benchmark focused on human interactions, leveraging additional annotations related to human responses. Our analysis highlights significant differences between human and AI response patterns and underscores critical shortcomings in current benchmarks. We anticipate that these findings will guide the creation of more advanced datasets and represent an important step toward achieving natural communication between humans and AI.

Access to Document (DOI)

DynamicVLN: Incorporating Dynamics into Vision-and-Language Navigation Scenarios

Sun Y., Qiu Y., Aoki Y.

Sensors 25 ( 2 ) 2025.01

Traditional Vision-and-Language Navigation (VLN) tasks require an agent to navigate static environments using natural language instructions. However, real-world road conditions such as vehicle movements, traffic signal fluctuations, pedestrian activity, and weather variations are dynamic and continually changing. These factors significantly impact an agent’s decision-making ability, underscoring the limitations of current VLN models, which do not accurately reflect the complexities of real-world navigation. To bridge this gap, we propose a novel task called Dynamic Vision-and-Language Navigation (DynamicVLN), incorporating various dynamic scenarios to enhance the agent’s decision-making abilities and adaptability. By redefining the VLN task, we emphasize that a robust and generalizable agent should not rely solely on predefined instructions but must also demonstrate reasoning skills and adaptability to unforeseen events. Specifically, we have designed ten scenarios that simulate the challenges of dynamic navigation and developed a dedicated dataset of 11,261 instances using the CARLA simulator (ver.0.9.13) and large language model to provide realistic training conditions. Additionally, we introduce a baseline model that integrates advanced perception and decision-making modules, enabling effective navigation and interpretation of the complexities of dynamic road conditions. This model showcases the ability to follow natural language instructions while dynamically adapting to environmental cues. Our approach establishes a benchmark for developing agents capable of functioning in real-world, dynamic environments and extending beyond the limitations of static VLN tasks to more practical and versatile applications.

Access to Document (DOI)

Data Collection-Free Masked Video Modeling

Ishikawa Y., Kondo M., Aoki Y.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 15069 LNCS 37 - 56 2025

ISSN 03029743

Pre-training video transformers generally requires a large amount of data, presenting significant challenges in terms of data collection costs and concerns related to privacy, licensing, and inherent biases. Synthesizing data is one of the promising ways to solve these issues, yet pre-training solely on synthetic data has its own challenges. In this paper, we introduce an effective self-supervised learning framework for videos that leverages readily available and less costly static images. Specifically, we define the Pseudo Motion Generator (PMG) module that recursively applies image transformations to generate pseudo-motion videos from images. These pseudo-motion videos are then leveraged in masked video modeling. Our approach is applicable to synthetic images as well, thus entirely freeing video pre-training from data collection costs and other concerns in real data. Through experiments in action recognition tasks, we demonstrate that this framework allows effective learning of spatio-temporal features through pseudo-motion videos, significantly improving over existing methods which also use static images and partially outperforming those using both real and synthetic videos. These results uncover fragments of what video transformers learn through masked video modeling.

Access to Document (DOI)

Unsupervised Metric Learning for Expressing Color and Shape Information to Uncover Abstract Connections within Image Datasets

Obikane S., Tagawa H., Aoki Y.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 15321 LNCS 15 - 30 2025

ISSN 03029743

In this research, we propose a novel approach using unsupervised metric learning tailored to datasets characterized by complex similarities and connections, such as those found in paintings and makeup, which are challenging to express linguistically. These datasets often present the difficulty of adequately analyzing data points due to the intricate interplay of defining elements, a limitation of traditional labeling methods. Additionally, the high degree of specialization required makes annotation significantly costly. Unsupervised metric learning emerges as a powerful method for extracting more cost-effective features and for the comprehensive analysis of these datasets. Expanding upon previous research that utilized style transfer models, our study further explores feature design, specifically focusing on extracting detailed information about critical aspects of similarity assessment, such as color and shape. Our model adeptly incorporates visual information, unveiling the hidden abstract connections within datasets. We validated our approach using a dataset of Ukiyo-e, a genre of Japanese painting, and achieved accuracy comparable to supervised learning models. This research opens up new possibilities for the analysis of complex image datasets with abstract relational depth, fostering a deeper understanding of the data.

Access to Document (DOI)

Rethinking Image Super-Resolution from Training Data Perspectives

Ohtani G., Tadokoro R., Yamada R., Asano Y.M., Laina I., Rupprecht C., Inoue N., Yokota R., Kataoka H., Aoki Y.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 15075 LNCS 19 - 36 2025

ISSN 03029743

In this work, we investigate the understudied effect of the training data used for image super-resolution (SR). Most commonly, novel SR methods are developed and benchmarked on common training datasets such as DIV2K and DF2K. However, we investigate and rethink the training data from the perspectives of diversity and quality, thereby addressing the question of “How important is SR training for SR models?”. To this end, we propose an automated image evaluation pipeline. With this, we stratify existing high-resolution image datasets and larger-scale image datasets such as ImageNet and PASS to compare their performances. We find that datasets with (i) low compression artifacts, (ii) high within-image diversity as judged by the number of different objects, and (iii) a large number of images from ImageNet or PASS all positively affect SR performance. We hope that the proposed simple-yet-effective dataset curation pipeline will inform the construction of SR datasets in the future and yield overall better models. Code is available at: https://github.com/gohtanii/DiverSeg-dataset.

Access to Document (DOI)

Is ImageNet Pre-Training Fair in Image Classification?

Yamada R., Takahashi R., Ohtani G., Mori E., Aoki Y., Kataoka H.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering 91 ( 4 ) 510 - 517 2025

ISSN 09120289

As part of efforts to address the issue of fairness in computer vision, image datasets are being reviewed, and inappropriate datasets are being temporarily suspended or withdrawn from public availability. What impact do pre-trained models using large-scale image datasets have on downstream tasks from the perspective of fairness? In this paper, we quantitatively evaluate how the pre-training methods, including manually supervised learning (MSL) and self-supervised learning (SSL) pre-training with ImageNet, affect downstream tasks in terms of fairness. We reached three findings from our experiments, summarized as follows: (i) SimCLRv2 and MoCov2 pre-trained models perform more fairly than MSL pre-trained models with ImageNet on IMDB-WIKI, FairFace, and CIFAR-10S. (ii) The MoCov2 pre-trained model achieves better performance in accuracy and fairness metrics than the SimCLRv2 pre-trained model. (iii) In MoCov2, color jitter tends to improve the fairness metrics for downstream tasks. By considering these findings, we demonstrate that SSL pre-trained models have the potential for fairer image recognition with similar accuracy compared to the MSL pre-trained model. By replacing human annotations with self-supervision, we can construct fair pre-trained models does not depend solely on human-annotated labels in the context of pre-training.

Access to Document (DOI)

RECA: A Pipeline for Refinement of Compressed Artifacts in Image Super-Resolution Training

Ohtani G., Kataoka H., Aoki Y.

IEEE Access 13 168361 - 168373 2025

Training datasets for image super-resolution (SR) are often constructed from web images. However, these images are typically stored in JPEG format, introducing compression artifacts that degrade SR performance. To ensure data quality, conventional approaches discard severely compressed images. While this preserves image quality, it reduces data diversity, which in turn limits model performance. To address this issue, we propose RECA (Refinement of Compressed Artifacts), a novel pipeline that enables the reuse of JPEG images that would otherwise be excluded. RECA first detects heavily compressed images using a blockiness measure, then restores visual quality using a real-world diffusion-based SR model, and finally applies bicubic downsampling to suppress artifacts introduced during enhancement. This process allows the effective inclusion of compressed images in training without sacrificing data quality. Experimental results demonstrate that RECA consistently outperforms filtering-based baselines in terms of PSNR and SSIM. These findings highlight the effectiveness of RECA in expanding usable training data and improving SR model performance.

Access to Document (DOI)

Pre-training with Synthetic Patterns for Audio

Ishikawa Y., Komatsu T., Aoki Y.

ICASSP IEEE International Conference on Acoustics Speech and Signal Processing Proceedings 2025

ISSN 15206149

In this paper, we propose to pre-train audio encoders using synthetic patterns instead of real audio data. Our proposed framework consists of two key elements. The first one is Masked Autoencoder (MAE), a self-supervised learning framework that learns from reconstructing data from randomly masked counterparts. MAEs tend to focus on low-level information such as visual patterns and regularities within data. Therefore, it is unimportant what is portrayed in the input, whether it be images, audio mel-spectrograms, or even synthetic patterns. This leads to the second key element, which is synthetic data. Synthetic data, unlike real audio, is free from privacy and licensing infringement issues. By combining MAEs and synthetic patterns, our framework enables the model to learn generalized feature representations without real data, while addressing the issues related to real audio. To evaluate the efficacy of our framework, we conduct extensive experiments across a total of 13 audio tasks and 17 synthetic datasets. The experiments provide insights into which types of synthetic patterns are effective for audio. Our results demonstrate that our framework achieves performance comparable to models pre-trained on AudioSet-2M and partially outperforms image-based pre-training methods.

Access to Document (DOI)

Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos

Ishikawa Y., Nakada S., Munakata H., Saito K., Komatsu T., Aoki Y.

Proceedings of the Annual Conference of the International Speech Communication Association Interspeech 2605 - 2609 2025

ISSN 2308457X

In this paper, we propose Language-Guided Contrastive AudioVisual Masked Autoencoders (LG-CAV-MAE) to improve audio-visual representation learning. LG-CAV-MAE integrates a pretrained text encoder into contrastive audio-visual masked autoencoders, enabling the model to learn across audio, visual and text modalities. To train LG-CAV-MAE, we introduce an automatic method to generate audio-visual-text triplets from unlabeled videos. We first generate frame-level captions using an image captioning model and then apply CLAP-based filtering to ensure strong alignment between audio and captions. This approach yields high-quality audio-visual-text triplets without requiring manual annotations. We evaluate LG-CAV-MAE on audio-visual retrieval tasks, as well as an audio-visual classification task. Our method significantly outperforms existing approaches, achieving up to a 5.6% improvement in recall@10 for retrieval tasks and a 3.2% improvement for the classification task.

Access to Document (DOI)

Iterative Event-Based Motion Segmentation by Variational Contrast Maximization

Yamaki R., Shiba S., Gallego G., Aoki Y.

IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 4918 - 4927 2025

ISSN 21607508

Event cameras provide rich signals that are suitable for motion estimation since they respond to changes in the scene. As any visual changes in the scene produce event data, it is paramount to classify the data into different motions (i.e., motion segmentation), which is useful for various tasks such as object detection and visual servoing. We pro-pose an iterative motion segmentation method, by classifying events into background (e.g., dominant motion hypoth-esis) and foreground (independent motion residuals), thus extending the Contrast Maximization framework. Experi-mental results demonstrate that the proposed method suc-cessfully classifies event clusters both for public and self-recorded datasets, producing sharp, motion-compensated edge-like images. The proposed method achieves state-of-the-art accuracy on moving object detection benchmarks with an improvement of over 30%, and demonstrates its possibility of applying to more complex and noisy real-world scenes. We hope this work broadens the sensitiv-ity of Contrast Maximization with respect to both motion parameters and input events, thus contributing to theoret-ical advancements in event-based motion segmentation es-timation. https://github.com/aoki-media-lab/event_based_segmentation.vcmax

Access to Document (DOI)

Formula-Supervised Sound Event Detection: Pre-Training Without Real Data

Shibata Y., Tanaka K., Bando Y., Imoto K., Kataoka H., Aoki Y.

ICASSP IEEE International Conference on Acoustics Speech and Signal Processing Proceedings 2025

ISSN 15206149

In this paper, we propose a novel formula-driven supervised learning (FDSL) framework for pre-training an environmental sound analysis model by leveraging acoustic signals parametrically synthesized through formula-driven methods. Specifically, we outline detailed procedures and evaluate their effectiveness for sound event detection (SED). The SED task, which involves estimating the types and timings of sound events, is particularly challenged by the difficulty of acquiring a sufficient quantity of accurately labeled training data. Moreover, it is well known that manually annotated labels often contain noises and are significantly influenced by the subjective judgment of annotators. To address these challenges, we propose a novel pre-training method that utilizes a synthetic dataset, Formula-SED, where acoustic data are generated solely based on mathematical formulas. The proposed method enables large-scale pre-training by using the synthesis parameters applied at each time step as ground truth labels, thereby eliminating label noise and bias. We demonstrate that large-scale pre-training with Formula-SED significantly enhances model accuracy and accelerates training, as evidenced by our results in the DESED dataset used for DCASE2023 Challenge Task 4. The project page is at https://yutoshibata07.github.io/Formula-SED/.

Access to Document (DOI)

BoundMatch: Boundary Detection Applied to Semi-Supervised Segmentation

Ishikawa H., Aoki Y.

IEEE Access 13 172776 - 172798 2025

Semi-supervised semantic segmentation (SS-SS) aims to mitigate the heavy annotation burden of dense pixel labeling by leveraging abundant unlabeled images alongside a small labeled set. While current consistency regularization methods achieve strong results, most do not explicitly model boundaries as a separate learning objective. In this paper, we propose BoundMatch, a novel multi-task SS-SS framework that explicitly integrates semantic boundary detection into a teacher-student consistency regularization pipeline. Our core mechanism, Boundary Consistency Regularized Multi-Task Learning (BCRM), enforces prediction agreement between teacher and student models on both segmentation masks and detailed semantic boundaries, providing complementary supervision from two independent tasks. To further enhance performance and encourage sharper boundaries, BoundMatch incorporates two lightweight fusion modules: Boundary-Semantic Fusion (BSF) injects learned boundary cues into the segmentation decoder, while Spatial Gradient Fusion (SGF) refines boundary predictions using mask gradients, yielding more reliable boundary pseudo-labels. This framework is built upon SAMTH, a strong teacher-student baseline featuring a Harmonious Batch Normalization (HBN) update strategy for improved stability. Extensive experiments on diverse datasets including Cityscapes and Pascal VOC show that BoundMatch achieves competitive performance against current state-of-the-art methods. Our approach achieves state-of-the-art results on the new Cityscapes benchmark with DINOv2 foundation model. Ablation studies highlight BoundMatch’s ability to improve boundary-specific evaluation metrics, its effectiveness in realistic large-scale unlabeled data scenario, and applicability to lightweight architectures for mobile deployment.

Access to Document (DOI)

RetinaViT: Efficient Visual Backbone for Online Video Streams

Suzuki T., Aoki Y.

Sensors 24 ( 17 ) 2024.09

In online video understanding, which has a wide range of real-world applications, inference speed is crucial. Many approaches involve frame-level visual feature extraction, which often represents the biggest bottleneck. We propose RetinaViT, an efficient method for extracting frame-level visual features in an online video stream, aiming to fundamentally enhance the efficiency of online video understanding tasks. RetinaViT is composed of efficiently approximated Transformer blocks that only take changed tokens (event tokens) as queries and reuse the already processed tokens from the previous timestep for the others. Furthermore, we restrict keys and values to the spatial neighborhoods of event tokens to further improve efficiency. RetinaViT involves tuning multiple parameters, which we determine through a multi-step process. During model training, we randomly vary these parameters and then perform black-box optimization to maximize accuracy and efficiency on the pre-trained model. We conducted extensive experiments on various online video recognition tasks, including action recognition, pose estimation, and object segmentation, validating the effectiveness of each component in RetinaViT and demonstrating improvements in the speed/accuracy trade-off compared to baselines. In particular, for action recognition, RetinaViT built on ViT-B16 reduces inference time by approximately 61.9% on the CPU and 50.8% on the GPU, while achieving slight accuracy improvements rather than degradation.

Access to Document (DOI)

Proto-Adapter: Efficient Training-Free CLIP-Adapter for Few-Shot Image Classification

Kato N., Nota Y., Aoki Y.

Sensors (Sensors) 24 ( 11 ) 2024.06

Large vision-language models, such as Contrastive Vision-Language Pre-training (CLIP), pre-trained on large-scale image–text datasets, have demonstrated robust zero-shot transfer capabilities across various downstream tasks. To further enhance the few-shot recognition performance of CLIP, Tip-Adapter augments the CLIP model with an adapter that incorporates a key-value cache model constructed from the few-shot training set. This approach enables training-free adaptation and has shown significant improvements in few-shot recognition, especially with additional fine-tuning. However, the size of the adapter increases in proportion to the number of training samples, making it difficult to deploy in practical applications. In this paper, we propose a novel CLIP adaptation method, named Proto-Adapter, which employs a single-layer adapter of constant size regardless of the amount of training data and even outperforms Tip-Adapter. Proto-Adapter constructs the adapter’s weights based on prototype representations for each class. By aggregating the features of the training samples, it successfully reduces the size of the adapter without compromising performance. Moreover, the performance of the model can be further enhanced by fine-tuning the adapter’s weights using a distance margin penalty, which imposes additional inter-class discrepancy to the output logits. We posit that this training scheme allows us to obtain a model with a discriminative decision boundary even when trained with a limited amount of data. We demonstrate the effectiveness of the proposed method through extensive experiments of few-shot classification on diverse datasets.

Access to Document (DOI)

Event-Based Background-Oriented Schlieren

Shiba S., Hamann F., Aoki Y., Gallego G.

IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE Transactions on Pattern Analysis and Machine Intelligence) 46 ( 4 ) 2011 - 2026 2024.04

Joint Work, ISSN 01628828

Schlieren imaging is an optical technique to observe the flow of transparent media, such as air or water, without any particle seeding. However, conventional frame-based techniques require both high spatial and temporal resolution cameras, which impose bright illumination and expensive computation limitations. Event cameras offer potential advantages (high dynamic range, high temporal resolution, and data efficiency) to overcome such limitations due to their bio-inspired sensing principle. This article presents a novel technique for perceiving air convection using events and frames by providing the first theoretical analysis that connects event data and schlieren. We formulate the problem as a variational optimization one combining the linearized event generation model with a physically-motivated parameterization that estimates the temporal derivative of the air density. The experiments with accurately aligned frame- and event camera data reveal that the proposed method enables event cameras to obtain on par results with existing frame-based optical flow techniques. Moreover, the proposed method works under dark conditions where frame-based schlieren fails, and also enables slow-motion analysis by leveraging the event camera's advantages. Our work pioneers and opens a new stack of event camera applications, as we publish the source code as well as the first schlieren dataset with high-quality frame and event data.

Access to Document (DOI)

Synthetic Document Images with Diverse Shadows for Deep Shadow Removal Networks

Matsuo Y., Aoki Y.

Sensors (Sensors) 24 ( 2 ) 2024.01

ISSN 14248220

Shadow removal for document images is an essential task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large, diverse dataset for document shadow removal takes time and effort. Thus, only small real datasets are available. Graphic renderers have been used to synthesize shadows to create relatively large datasets. However, the limited number of unique documents and the limited lighting environments adversely affect the network performance. This paper presents a large-scale, diverse dataset called the Synthetic Document with Diverse Shadows (SynDocDS) dataset. The SynDocDS comprises rendered images with diverse shadows augmented by a physics-based illumination model, which can be utilized to obtain a more robust and high-performance deep shadow removal network. In this paper, we further propose a Dual Shadow Fusion Network (DSFN). Unlike natural images, document images often have constant background colors requiring a high understanding of global color features for training a deep shadow removal network. The DSFN has a high global color comprehension and understanding of shadow regions and merges shadow attentions and features efficiently. We conduct experiments on three publicly available datasets, the OSR, Kligler’s, and Jung’s datasets, to validate our proposed method’s effectiveness. In comparison to training on existing synthetic datasets, our model training on the SynDocDS dataset achieves an enhancement in the PSNR and SSIM, increasing them from 23.00 dB to 25.70 dB and 0.959 to 0.971 on average. In addition, the experiments demonstrated that our DSFN clearly outperformed other networks across multiple metrics, including the PSNR, the SSIM, and its impact on OCR performance.

Access to Document (DOI)

Improving Perceptual Loss with CLIP for Super-Resolution

Ohtani G., Kataoka H., Aoki Y.

Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering (Seimitsu Kogaku Kaishi/Journal of the Japan Society for Precision Engineering) 90 ( 2 ) 217 - 223 2024

ISSN 09120289

Perceptual loss, calculated by VGG network pre-trained on ImageNet, has been widely employed in the past for super-resolution tasks, enabling the generation of photo-realistic images. However, it has been reported that grid-like artifacts frequently appear in the generated images. To address this problem, we consider that large-scale pre-trained models can make significant contributions to super-resolution across different scenes. In particular, by combining language, those models can exhibit a strong capability to comprehend complex scenes, potentially enhancing super-resolution performance. Therefore, this paper proposes new perceptual loss with Contrastive Language-Image Pre-training (CLIP) based on Vision Transformer (ViT) instead of VGG network. The results demonstrate our proposed perceptual loss can generate photorealistic images without grid-like artifacts.

Access to Document (DOI)

　 Previous page - Next page 　