論文 - 池原　雅章

Image Deraining with Frequency-Enhanced State Space Model

Yamashita S., Ikehara M.

Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics 15475 LNCS   318 - 334 2025年

ISSN  03029743

　概要を見る

Removing rain degradations in images is recognized as a significant issue. In this field, deep learning-based approaches, such as Convolutional Neural Networks (CNNs) and Transformers, have succeeded. Recently, State Space Models (SSMs) have exhibited superior performance across various tasks in both natural language processing and image processing due to their ability to model long-range dependencies. This study introduces SSM to image deraining with deraining-specific enhancements and proposes a Deraining Frequency-Enhanced State Space Model (DFSSM). To effectively remove rain streaks, which produce high-intensity frequency components in specific directions, we employ frequency domain processing concurrently with SSM. Additionally, we develop a novel mixed-scale gated-convolutional block, which uses convolutions with multiple kernel sizes to capture various scale degradations effectively and integrates a gating mechanism to manage the flow of information. Finally, experiments on synthetic and real-world rainy image datasets show that our method surpasses state-of-the-art methods. Code is available at https://github.com/ShugoYamashita/DFSSM.

• Access to Document (DOI)

Efficient and Effective Blind JPEG Image Improvement With Sequential Feature Processing

Ezumi S., Ikehara M.

IEEE Access 12   151975 - 151986 2024年10月

研究論文（学術雑誌）, 査読有り

　概要を見る

Image compression technologies such as JPEG enable efficient use of digital images. However, these technologies also degrade the quality of the image, resulting in a poor visual appearance and reduced usability. One way to solve the quality problem and achieve high efficiency in using high-quality JPEG images is to use quality improvement technologies based on artifact removal. In recent years, various methods using machine-learning-optimized neural networks have been devised to achieve high performance whereas maintaining versatility for JPEG images with various qualities. On the other hand, methods that achieve high-quality images need enormous computing costs, which is a barrier to easy use, especially when using images with large size. Given this situation, this paper proposes a method that achieves higher performance with smaller computational resources while maintaining the image quality in broader situations. The proposed method consists of a Quality Estimation Part (QE Part) that estimates the quality of the input image and Image Processing Parts (IP Parts) that process the image based on the estimated quality representation. Sequential processing is carried out by connecting multiple IP Parts in a cascade, which enables efficient and effective processing of different features. Each IP Part consists of multiple Processing Blocks, which enable effective quality improvement while maintaining efficiency. These measures enable the proposed method to achieve state-of-the-art quantitative results and better qualitative results that outperform conventional methods for images with various qualities. The code and the pre-trained models are released at https://github.com/ezumi-keio/Sequential_Processing-main.

• Access to Document (DOI)

Edge-Guided Low-Light Image Enhancement Based on GAN with Effective Modules

Matsui T., Ikehara M.

European Signal Processing Conference    456 - 460 2024年09月

研究論文（国際会議プロシーディングス）, 査読有り,  ISSN  22195491

　概要を見る

Under low-light conditions, the images taken may not be satisfactorily bright and may degrade in appearance. Low-light image enhancement (LLIE) is a process that converts such dim images into images taken under normal lighting conditions. The primary objectives of LLIE are to diminish noise and artifacts, maintain the integrity of edges and textures, and restore the image’s natural brightness and colors. Deep learning-based methods have shown remarkable success in this field recently but are hindered by their lengthy processing times due to intricate network architectures. To address the balance between performance and processing speed, we introduce a streamlined network equipped with efficient modules. Our approach incorporates a GAN (Generative Adversarial Network) framework enhanced with preprocessing for edge and texture extraction. We also integrate Channel Attention for color and illumination correction, Res FFT-ReLU for noise reduction, and Pixel Shuffler for high-frequency detail preservation. Our experiments show that our method surpasses traditional LLIE techniques in both quality and processing speed.

• Access to Document (DOI)

MSARNet: Efficient JPEG Artifact Removal Using Multi-Stage Style Network

Ezumi S., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2024年01月

研究論文（国際会議プロシーディングス）, 査読有り,  ISSN  0747668X

　概要を見る

With the rapid development of photography and information processing technologies, we use more and more digital images in our daily lives. JPEG is one of the most widely used digital image formats because of its high efficiency and widespread support. JPEG Artifact Removal is a task that removes artifacts in JPEG images, such as noise and color distortion in JPEG images. Existing methods for JPEG Artifact Removal require high computational costs, which means stricter performance requirements or longer processing time. We propose a novel method for JPEG Artifact Removal named Multi-Stage style Artifacts Removal Net (MSARNet), which meets high performance, high versatility, and low computational cost. MSARNet adopts multi-stage processing, and images are processed in processing stages step by step. This process enables our proposed method to deal with different types of images effectively and efficiently. Additionally, MSARNet estimates the quality value of the input image, which contributes to effective feature processing. Experimental results show that our proposed method handles various artifacts well and outperforms existing methods in artifact removal tasks with lower computational costs.

• Access to Document (DOI)

Multiple Adverse Weather Removal Using Masked-Based Pre-Training and Dual-Pooling Adaptive Convolution

Yamashita S., Ikehara M.

IEEE Access 12   58057 - 58066 2024年

研究論文（学術雑誌）

　概要を見る

Removing artifacts caused by multiple adverse weather, including rain, fog, and snow, is crucial for image processing in outdoor environments. Conventional high-performing methods face challenges, such as requiring pre-specification of weather types and slow processing times. In this study, we propose a novel convolutional neural network-based hierarchical encoder-decoder model that addresses these issues effectively. Our model utilizes knowledge of feature representations obtained from masked-based pre-training on a large-scale dataset. To remove diverse degradations efficiently, we employ a proposed dual-pooling adaptive convolution, which improves representational capability of weight generating network by using average pooling, max pooling, and filter-wise global response normalization. Experiments conducted on both synthetic and real image datasets show that our model achieves promising results. The performance on real images is also improved by a novel learning strategy, in which a model trained on the synthetic image dataset is fine-tuned to the real image dataset. The proposed method is notably cost-effective in terms of computational complexity and inference speed. Moreover, ablation studies show the effectiveness of various components in our method.

• Access to Document (DOI)

Faster Training of Large Kernel Convolutions on Smaller Spatial Scales

Fukuzaki S., Ikehara M.

IEEE Access 12   161312 - 161328 2024年

研究論文（学術雑誌）

　概要を見る

Computational requirements for training state-of-the-art neural network models are increasing on vision tasks because high computational factors have become known to be effective in improving quality. While research in the image-processing field requires a lot of trials, this trend makes proving hypotheses difficult for researchers in computationally restricted environments. Neural convolution with a wide receptive field is one of the high-computational factors with quality improvement. This study aims to accelerate the training of the large kernel convolutions by resizing both training images and convolution filters to a smaller scale. Applying this strategy requires careful training designs to replace conventional training on the target scale, and we propose four techniques to improve the quality of the trained models. In our experiment, we apply our proposals to train an image classifier model modified from RepLKNet-31B on the image classification task of the CIFAR-10, CIFAR-100, and STL-10 datasets. Our proposed framework trains almost the same models 4.62-4.91 times faster than the standard training on the target spatial scale, keeping its accuracy, and provides 2.61-2.79 times further training acceleration and stability in accuracy compared to Progressive Learning. In addition to the training acceleration, our framework can simultaneously train models for multiple scales without any scale-specific tuning, which provides scalable usage considering the computational costs.

• Access to Document (DOI)

More Direct and stage-wise network for Face Super Resolution

Horiguchi Y., Ikehara M., Shibasaki K.

Apsipa ASC 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference 2024 2024年

　概要を見る

In recent years, U-net-based models have demonstrated high performance in high-magnification face super resolution (FSR) tasks and are capable of outputting more vivid super resolution (SR) images. However, this model has some drawbacks, such as the possibility of unnecessary computations and uniform block selection at all stages of the decoder. Therefore, we propose a Direct and Stage-wise (DS) Net, which improves on CTCNet [1], a U-net based model with high qualitative results. This model outperforms previous networks by eliminating encoders to reduce computational waste, and by focusing on global feature extraction in the small tensor size stage.

• Access to Document (DOI)

Face Drawing GAN by Channel Attention and Matrix Product Attention

Ogura H., Ezumi S., Ikehara M.

Proceedings International Conference on Image Processing Icip    1588 - 1594 2024年

ISSN  15224880

　概要を見る

Face photo-sketch synthesis tasks have been developed with Generative Adversarial Networks (GANs) based on Convolutional Neural Network (CNN) and Vision Transformer (ViT). CNN is good at capturing local features, but its locality results in blurred images and contours. ViT is good at capturing global information, but is not as good as CNN in capturing local features, and while it can prevent blurring of contours and other lines, it does not reflect fine texture. Therefore, we propose a Face Drawing GAN, which generates high-quality face sketches by capturing both local and global features. Face Drawing GAN is a CNN-based model and it incorporates Channel Attention, which functionally adjusts the weights of channels, and Matrix Product Attention (MP Attention), which weights pixels based on the similarity between the vertical and horizontal sides of images obtained by matrix product. Through the experiments, we confirmed that our proposed MP Attention assists in capturing global features and Face Drawing GAN is capable of generating face sketches that outperform conventional methods.

• Access to Document (DOI)

Enhancing YOLOv7 with GLF-Trans for Precision in Small Object Detection

Yoshikawa N., Ikehara M.

Apsipa ASC 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference 2024 2024年

　概要を見る

Object detection techniques are essential for identifying objects and accurately determining their locations in images and videos. This paper proposes an approach to enhance the performance of YOLOv7, the YOLO (You Only Look Once) series, by improving detection accuracy, particularly for small objects. The accurate detection of small objects is crucial in various real-world applications such as autonomous driving, surveillance, and medical imaging, where the identification of small, often critical objects can have significant implications. The proposed method balances the current one-step and two-step algorithms used in object detection, with the goal of improving the accuracy of small object detection. We apply several improvements to the YOLOv7 framework, including the introduction of an attention module, the Global-Local Fusion Transformer (GLF-Trans), and the addition of a tiny-head. These enhancements are designed to further enhance the accuracy of small object detection. The experimental results show that the proposed method achieves higher accuracy than the previous model, with a specific improvement of 3.8% in mAP0.5 on the VisDrone2019 dataset.

• Access to Document (DOI)

UNet Based Multi-Scale Recurrent Network for Lightweight Video Deblurring

Yae S., Ikehara M.

IEEE Access 11   117520 - 117527 2023年

　概要を見る

With the recent widespread use of smartphones and digital video cameras, the opportunities to handle digital video have increased significantly. However, despite improvements in the performance of the hardware, the captured video often contains information that is not necessary for the purpose of the video. In particular, factors such as camera shake and object movement can cause blurring in video. So, we propose a method to deblur a video by software processing of the video after shooting. In conventional methods, video deblurring is often performed using a network whose main task is video super-resolution. However, in super-resolution, the size of the input and output images are different, while the input and output images are the same size in deblurring. In the case of deblurring, the input images are input to the network after a simple downsampling process, which is not optimized for the same size as the input and output images. Therefore, the proposed method constructs a multi-scale network based on UNet. The UNet-based network is a successful method for single image deblurring. Because a video is a sequence of multiple images, we use a method that has been successful in single image deblurring. Furthermore, we add improvements to the network based on the structure of MPRNet. The feature extraction modules of the bottom and the second stage of the network are replaced with a single-stage UNet. These improvements resulted in a 34.80dB in PSNR and 0.973 in SSIM on the GoPro dataset despite about 75% of the FLOPs of BasicVSR++ and 3% of the FLOPs of VRT. On the DVD dataset, the proposed model achieved 34.36dB in PSNR and 0.966 in SSIM. Further ablation studies show the effectiveness of various components in our proposed model.

• Access to Document (DOI)

Training Large Kernel Convolutions with Resized Filters and Smaller Images

Fukuzaki S., Ikehara M.

GCCE 2023 - 2023 IEEE 12th Global Conference on Consumer Electronics    32 - 33 2023年

　概要を見る

Convolution is an essential component in neural networks for vision tasks. Convolutions on large receptive fields are more suitable than small kernel convolutions when aggregating global features in convolutional neural networks. However, large kernel convolutions require much computation and memory usage, slowing training neural networks. Then, we propose to train convolution weights with small images, resizing the convolution filters. While this idea shortens the time for training filters, simply applying this causes profound degradation. In this paper, we introduce four techniques that suppress degradation; weight scaling, removing Batch Normalization, defining a minimum resolution, and training with various-size images. In our experiment, we apply our proposals to train an image classification model based on RepLKNet-B on the image classification task of the CIFAR-100 dataset. Training with our proposals is approximately eight times faster than conventional training on the target spatial scale, keeping its accuracy.

• Access to Document (DOI)

Toward High-Quality Real-Time Video Denoising With Pseudo Temporal Fusion Network

Shibasaki K., Ikehara M.

IEEE Access 11   81466 - 81476 2023年

　概要を見る

With the increasing availability of high-resolution video recording and streaming, there is a need for fast and high-quality video denoising methods that can handle high-resolution videos. However, many existing methods fail to achieve high-quality denoising performance and computationally effecient at the same time. This paper proposes a video denoising network, Pseudo Temporal Fusion Network (PTFN), that satisfies these requirements. PTFN adopts a new Pseudo Temporal Fusion (PTF) module that captures pseudo-temporal relationships between video frames in combination with the Temporal Shift Module. PTFN also adopts a modern ConvBlock paradigm that breaks away from the classical ConvBlock paradigm, contributing to denoising performance and computationally effecient. PTFN achieves better performance than existing video denoising methods in terms of both video quality and computational effeciency. Specifically, PTFN has only about 16.7% of the computational cost of existing lightweight methods, while it improves denoising performance. PTFN is also superior in terms of memory consumption. It can process 1080p videos with a GPU with 24 GB RAM. In addition, a lighter version (PTFN Half) can process 2K videos at high speed under the same conditions.

• Access to Document (DOI)

Stereo Matching with Supplementary Boundary Information

Hashimoto K., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2023-January 2023年

ISSN  0747668X

　概要を見る

Stereo matching, which estimates the disparity of corresponding points in paired images, has problems with accuracy degraded by occlusion, repeated texture, texture-free areas, and gloss. To address this problem, we focused on object boundary information, which is less susceptible to such problems. This is because occlusion is more likely to occur at the boundaries, and even if the corresponding points disappear due to occlusion, the boundaries still have the corresponding parts. The boundary information is also robust to such problems because it is not affected by repeated textures, untextured areas, or gloss. Therefore, we propose a method for generating the boundary ground truth from the depth ground truth and develop a network and loss function for estimating the boundary information. We also propose that the accuracy can be improved by incorporating the estimated boundary information into a state-of-the-art network as an auxiliary.

• Access to Document (DOI)

Single Image Fence Removal Using Fast Fourier Transform

Kume K., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2023-January 2023年

ISSN  0747668X

　概要を見る

Fences extensively cover the image in a variety of shapes, making fence removal from the image is a challenging task. Most fence removal in a single image is performed in two stages: fence area estimation and inpainting as shown in the Fig 1. Conventional fence detection has utilized information such as edge information, luminance information, and fence regularity to deal with various shapes. However, those methods have resulted in missing fences and false detections of structures similar to fences. In this paper, we propose a network that incorporates a module that includes a fast Fourier transform. This allows us to extract global features of fences that extensively cover the image in various shapes. Furthermore, we create a dataset of rectangular fences, which is not available in previous datasets. Experimental results on existing and proposed datasets show that our method achieves better results, both quantitatively and qualitatively, by suppressing missing fences and false positives of other structures.

• Access to Document (DOI)

RGB-D Salient Object Detection Using Saliency and Edge Reverse Attention

Ikeda T., Ikehara M.

IEEE Access 11   68818 - 68825 2023年

　概要を見る

RGB-D salient object detection is a task to detect visually significant objects in an image using RGB and depth images. Although many useful CNN-based methods have been proposed in the past, there are some problems such as blurring of object boundaries and inability to detect important parts of objects, leading to a decrease in detection accuracy. In this paper, we propose RGB-D salient object detection using Saliency and Edge Reverse Attention(SERA), which combines the fusion of saliency and edge features with reverse attention. The reverse attention process has the effect of making it easier to capture the boundaries of objects and undetected objects by inverting the up-sampled saliency features and weighting other saliency features, and the edge reverse attention process has the effect of making salient object regions stand out by inverting the edge features and weighting saliency features. The interaction between the salient object features and the edge features enhances each other's features and refines the information on the boundary of objects and salient object regions. In addition, to make it easier to refer to the global information of an image, we introduced the Multi-Scale Interactive Module(MSIM), which is capable of acquiring information at rich scales by converting feature maps to different resolutions and interacting with them. In addition to the salient object output, supervised learning is applied to multiple edge outputs of each resolution to improve the accuracy of both salient objects and boundary areas. Experimental results on five benchmarks show that the proposed method quantitatively performs better than the conventional method, and qualitatively improves the sharpness of object boundaries and the accuracy of detecting important parts of objects.

• Access to Document (DOI)

Pose-aware Disentangled Multiscale Transformer for Pose Guided Person Image Generation

Shibasaki K., Ikehara M.

European Signal Processing Conference    506 - 510 2023年

ISSN  22195491

　概要を見る

Pose Guided Person Image Generation (PGPIG) is the task that transforms the pose of a person's image from the source image, its pose information, and the target pose information. Most existing PGPIG methods require additional pose information or tasks, which limits their application. Moreover, they use CNNs, which can only extract features from neighboring pixels and cannot consider the consistency of the entire image. This paper proposes a PGPIG network solving these problems by using a module containing Axial Transformers with large receptive field. The proposed method disentangles the PGPIG task into two subtasks: “rough pose transformation” and “detailed texture generation”. In the former task, low-resolution feature maps are transformed by blocks containing Axial Transformer. The latter task uses a CNN network with Adaptive Instance Normalization. Experiments show that the proposed method has competitive performance with other state-of-the-art methods. Furthermore, despite achieving excellent performance, the proposed network has a significantly fewer parameters than existing methods.

• Access to Document (DOI)

Pose-Aware Disentangled Multiscale Transformer for Pose Guided Person Image Generation

Shibasaki K., Ikehara M.

IEEE Access 11   146054 - 146064 2023年

　概要を見る

Pose Guided Person Image Generation (PGPIG) is a task that involves generating an image of a person in a target pose, given an image in a source pose, source pose information and target pose information. Many of the existing PGPIG techniques need extra pose-related data or tasks, which cause the limitations on their applicability. In addition CNNs are used as the feature extractor which do not have long-range dependency. However, CNNs can only extract features from neighboring pixels and cannot consider image consistency. This paper introduces a PGPIG network that solves these challenges by incorporating modules that use Axial Transformers with wide receptive fields. The proposed approach disentangles the PGPIG task into two subtasks: 'rough pose transformation' and 'detailed texture generation.' In 'rough pose transformation,' lower-resolution feature maps is processed by Axial Transformer-based blocks. These blocks employ an Encoder-Decoder structure, which allows the network to use the pose information well and improves the stability and performance of the training. The latter subtask employs a CNN network with Adaptive Instance Normalization. Experimental results show the competitive performance of the proposed method compared to existing methods. The proposed method achieves lowest LPIPS in Deep Fashion dataset and FID in Market-1501 dataset. Remarkably, despite the great results obtained, the number of parameters of the proposed network is significantly less in contrast to existing methods.

• Access to Document (DOI)

Low-Light Image Enhancement Using a Simple Network Structure

Matsui T., Ikehara M.

IEEE Access 11   65507 - 65516 2023年

　概要を見る

Under low-light conditions, captured images can be affected by unsatisfactory lighting conditions. Low-light image enhancement called LLIE is a digital image processing to obtain natural normal-light images from the low-light image. LLIE includes three main tasks: reducing noise and artifacts, preserving edges and textures, and reproducing natural brightness and color. In recent years, many types of research have focused on deep-learning-based approaches that can achieve excellent performance. However, one primary problem with these methods is that inference time is long owing to complex network structures. To solve the trade-off between the performance and implementation time, we propose a simple network with effective modules. We utilize a U-Net structure and pre-processing is added to preserve edges and textures. Moreover, we embed Channel Attention to restore color and illumination, Res FFT-ReLU to reduce noise, and Pixel Shuffler to preserve the high-frequency components. According to our experimental results, the proposed method achieves better performance and faster inference time than conventional LLIE methods.

• Access to Document (DOI)

Inverted Residual Fourier Transformation for Lightweight Single Image Deblurring

Yae S., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2023-January 2023年

ISSN  0747668X

　概要を見る

In recent years, encoder-decoder structures are widely used for single image deblurring and successfully restore high quality images. How-ever, FLOPs and the number of parameters tend to increase to restore a high-quality image. Thus, we propose a new lightweight network (IRFTNet) based on DeepRFT. This network has two features to improve performance and lightweight. First, a new backbone called Inverted Residual Fourier Transformation block (IRFTblock) based on inverted residual block is introduced to decrease computational complexity. Second, a new module called Lower Feature Synthesis (LFS) was introduced to efficiently transfer encoder information from lower layers to upper layers. These improvements resulted in a 32.98dB in PSNR on the GoPro dataset, despite approximately half FLOPs and the number of parameters of DeepRFT.

• Access to Document (DOI)

Inverted Residual Fourier Transformation for Lightweight Single Image Deblurring

Yae S., Ikehara M.

IEEE Access 11   29175 - 29182 2023年

　概要を見る

Single image deblurring aims to restore a sharp image by removing blurred areas in the single image. Such blurred images are not only visually unpleasant, but cause various problems in many applications like image recognition. In recent years, with the development of deep learning, neural networks are used in many single image deblurring. Especially, encoder-decoder structures are widely used for single image deblurring and successfully restore high quality images. However, FLOPs and the number of parameters tend to increase to restore a high-quality image. Thus, this paper proposes a new lightweight network (IRFTNet) based on UNet, which is a widely used basic network of encoder-decoder structures. Our proposed network has three features to improve performance and lightweight. First, a new backbone called Inverted Residual Fourier Transformation block (IRFTblock) based on inverted residual block is introduced to decrease computational complexity. Second, a new module called Lower Feature Synthesis (LFS) is introduced to efficiently transfer encoder information from lower layers to upper layers. Finally, multiple outputs structure proposed in MIMO-UNet is introduced. These improvements resulted in a 32.98dB in PSNR on the GoPro dataset, despite approximately half FLOPs and the number of parameters of DeepRFT. Further ablation studies show the effectiveness of various components in our proposed model.

• Access to Document (DOI)