論文 - 池原　雅章

Efficient and Effective Blind JPEG Image Improvement With Sequential Feature Processing

Ezumi S., Ikehara M.

IEEE Access 12   151975 - 151986 2024年10月

研究論文（学術雑誌）, 査読有り

　概要を見る

Image compression technologies such as JPEG enable efficient use of digital images. However, these technologies also degrade the quality of the image, resulting in a poor visual appearance and reduced usability. One way to solve the quality problem and achieve high efficiency in using high-quality JPEG images is to use quality improvement technologies based on artifact removal. In recent years, various methods using machine-learning-optimized neural networks have been devised to achieve high performance whereas maintaining versatility for JPEG images with various qualities. On the other hand, methods that achieve high-quality images need enormous computing costs, which is a barrier to easy use, especially when using images with large size. Given this situation, this paper proposes a method that achieves higher performance with smaller computational resources while maintaining the image quality in broader situations. The proposed method consists of a Quality Estimation Part (QE Part) that estimates the quality of the input image and Image Processing Parts (IP Parts) that process the image based on the estimated quality representation. Sequential processing is carried out by connecting multiple IP Parts in a cascade, which enables efficient and effective processing of different features. Each IP Part consists of multiple Processing Blocks, which enable effective quality improvement while maintaining efficiency. These measures enable the proposed method to achieve state-of-the-art quantitative results and better qualitative results that outperform conventional methods for images with various qualities. The code and the pre-trained models are released at https://github.com/ezumi-keio/Sequential_Processing-main.

• Access to Document (DOI)

Edge-Guided Low-Light Image Enhancement Based on GAN with Effective Modules

Matsui T., Ikehara M.

European Signal Processing Conference    456 - 460 2024年09月

研究論文（国際会議プロシーディングス）, 査読有り,  ISSN  22195491

　概要を見る

Under low-light conditions, the images taken may not be satisfactorily bright and may degrade in appearance. Low-light image enhancement (LLIE) is a process that converts such dim images into images taken under normal lighting conditions. The primary objectives of LLIE are to diminish noise and artifacts, maintain the integrity of edges and textures, and restore the image’s natural brightness and colors. Deep learning-based methods have shown remarkable success in this field recently but are hindered by their lengthy processing times due to intricate network architectures. To address the balance between performance and processing speed, we introduce a streamlined network equipped with efficient modules. Our approach incorporates a GAN (Generative Adversarial Network) framework enhanced with preprocessing for edge and texture extraction. We also integrate Channel Attention for color and illumination correction, Res FFT-ReLU for noise reduction, and Pixel Shuffler for high-frequency detail preservation. Our experiments show that our method surpasses traditional LLIE techniques in both quality and processing speed.

• Access to Document (DOI)

MSARNet: Efficient JPEG Artifact Removal Using Multi-Stage Style Network

Ezumi S., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2024年01月

研究論文（国際会議プロシーディングス）, 査読有り,  ISSN  0747668X

　概要を見る

With the rapid development of photography and information processing technologies, we use more and more digital images in our daily lives. JPEG is one of the most widely used digital image formats because of its high efficiency and widespread support. JPEG Artifact Removal is a task that removes artifacts in JPEG images, such as noise and color distortion in JPEG images. Existing methods for JPEG Artifact Removal require high computational costs, which means stricter performance requirements or longer processing time. We propose a novel method for JPEG Artifact Removal named Multi-Stage style Artifacts Removal Net (MSARNet), which meets high performance, high versatility, and low computational cost. MSARNet adopts multi-stage processing, and images are processed in processing stages step by step. This process enables our proposed method to deal with different types of images effectively and efficiently. Additionally, MSARNet estimates the quality value of the input image, which contributes to effective feature processing. Experimental results show that our proposed method handles various artifacts well and outperforms existing methods in artifact removal tasks with lower computational costs.

• Access to Document (DOI)

Multiple Adverse Weather Removal Using Masked-Based Pre-Training and Dual-Pooling Adaptive Convolution

Yamashita S., Ikehara M.

IEEE Access 12   58057 - 58066 2024年

研究論文（学術雑誌）

　概要を見る

Removing artifacts caused by multiple adverse weather, including rain, fog, and snow, is crucial for image processing in outdoor environments. Conventional high-performing methods face challenges, such as requiring pre-specification of weather types and slow processing times. In this study, we propose a novel convolutional neural network-based hierarchical encoder-decoder model that addresses these issues effectively. Our model utilizes knowledge of feature representations obtained from masked-based pre-training on a large-scale dataset. To remove diverse degradations efficiently, we employ a proposed dual-pooling adaptive convolution, which improves representational capability of weight generating network by using average pooling, max pooling, and filter-wise global response normalization. Experiments conducted on both synthetic and real image datasets show that our model achieves promising results. The performance on real images is also improved by a novel learning strategy, in which a model trained on the synthetic image dataset is fine-tuned to the real image dataset. The proposed method is notably cost-effective in terms of computational complexity and inference speed. Moreover, ablation studies show the effectiveness of various components in our method.

• Access to Document (DOI)

Faster Training of Large Kernel Convolutions on Smaller Spatial Scales

Fukuzaki S., Ikehara M.

IEEE Access 12   161312 - 161328 2024年

研究論文（学術雑誌）

　概要を見る

Computational requirements for training state-of-the-art neural network models are increasing on vision tasks because high computational factors have become known to be effective in improving quality. While research in the image-processing field requires a lot of trials, this trend makes proving hypotheses difficult for researchers in computationally restricted environments. Neural convolution with a wide receptive field is one of the high-computational factors with quality improvement. This study aims to accelerate the training of the large kernel convolutions by resizing both training images and convolution filters to a smaller scale. Applying this strategy requires careful training designs to replace conventional training on the target scale, and we propose four techniques to improve the quality of the trained models. In our experiment, we apply our proposals to train an image classifier model modified from RepLKNet-31B on the image classification task of the CIFAR-10, CIFAR-100, and STL-10 datasets. Our proposed framework trains almost the same models 4.62-4.91 times faster than the standard training on the target spatial scale, keeping its accuracy, and provides 2.61-2.79 times further training acceleration and stability in accuracy compared to Progressive Learning. In addition to the training acceleration, our framework can simultaneously train models for multiple scales without any scale-specific tuning, which provides scalable usage considering the computational costs.

• Access to Document (DOI)

Pose-aware Disentangled Multiscale Transformer for Pose Guided Person Image Generation

Shibasaki K., Ikehara M.

European Signal Processing Conference    506 - 510 2023年

ISSN  22195491

　概要を見る

Pose Guided Person Image Generation (PGPIG) is the task that transforms the pose of a person's image from the source image, its pose information, and the target pose information. Most existing PGPIG methods require additional pose information or tasks, which limits their application. Moreover, they use CNNs, which can only extract features from neighboring pixels and cannot consider the consistency of the entire image. This paper proposes a PGPIG network solving these problems by using a module containing Axial Transformers with large receptive field. The proposed method disentangles the PGPIG task into two subtasks: “rough pose transformation” and “detailed texture generation”. In the former task, low-resolution feature maps are transformed by blocks containing Axial Transformer. The latter task uses a CNN network with Adaptive Instance Normalization. Experiments show that the proposed method has competitive performance with other state-of-the-art methods. Furthermore, despite achieving excellent performance, the proposed network has a significantly fewer parameters than existing methods.

• Access to Document (DOI)

Pose-Aware Disentangled Multiscale Transformer for Pose Guided Person Image Generation

Shibasaki K., Ikehara M.

IEEE Access 11   146054 - 146064 2023年

　概要を見る

Pose Guided Person Image Generation (PGPIG) is a task that involves generating an image of a person in a target pose, given an image in a source pose, source pose information and target pose information. Many of the existing PGPIG techniques need extra pose-related data or tasks, which cause the limitations on their applicability. In addition CNNs are used as the feature extractor which do not have long-range dependency. However, CNNs can only extract features from neighboring pixels and cannot consider image consistency. This paper introduces a PGPIG network that solves these challenges by incorporating modules that use Axial Transformers with wide receptive fields. The proposed approach disentangles the PGPIG task into two subtasks: 'rough pose transformation' and 'detailed texture generation.' In 'rough pose transformation,' lower-resolution feature maps is processed by Axial Transformer-based blocks. These blocks employ an Encoder-Decoder structure, which allows the network to use the pose information well and improves the stability and performance of the training. The latter subtask employs a CNN network with Adaptive Instance Normalization. Experimental results show the competitive performance of the proposed method compared to existing methods. The proposed method achieves lowest LPIPS in Deep Fashion dataset and FID in Market-1501 dataset. Remarkably, despite the great results obtained, the number of parameters of the proposed network is significantly less in contrast to existing methods.

• Access to Document (DOI)

Low-Light Image Enhancement Using a Simple Network Structure

Matsui T., Ikehara M.

IEEE Access 11   65507 - 65516 2023年

　概要を見る

Under low-light conditions, captured images can be affected by unsatisfactory lighting conditions. Low-light image enhancement called LLIE is a digital image processing to obtain natural normal-light images from the low-light image. LLIE includes three main tasks: reducing noise and artifacts, preserving edges and textures, and reproducing natural brightness and color. In recent years, many types of research have focused on deep-learning-based approaches that can achieve excellent performance. However, one primary problem with these methods is that inference time is long owing to complex network structures. To solve the trade-off between the performance and implementation time, we propose a simple network with effective modules. We utilize a U-Net structure and pre-processing is added to preserve edges and textures. Moreover, we embed Channel Attention to restore color and illumination, Res FFT-ReLU to reduce noise, and Pixel Shuffler to preserve the high-frequency components. According to our experimental results, the proposed method achieves better performance and faster inference time than conventional LLIE methods.

• Access to Document (DOI)

Inverted Residual Fourier Transformation for Lightweight Single Image Deblurring

Yae S., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2023-January 2023年

ISSN  0747668X

　概要を見る

In recent years, encoder-decoder structures are widely used for single image deblurring and successfully restore high quality images. How-ever, FLOPs and the number of parameters tend to increase to restore a high-quality image. Thus, we propose a new lightweight network (IRFTNet) based on DeepRFT. This network has two features to improve performance and lightweight. First, a new backbone called Inverted Residual Fourier Transformation block (IRFTblock) based on inverted residual block is introduced to decrease computational complexity. Second, a new module called Lower Feature Synthesis (LFS) was introduced to efficiently transfer encoder information from lower layers to upper layers. These improvements resulted in a 32.98dB in PSNR on the GoPro dataset, despite approximately half FLOPs and the number of parameters of DeepRFT.

• Access to Document (DOI)

Inverted Residual Fourier Transformation for Lightweight Single Image Deblurring

Yae S., Ikehara M.

IEEE Access 11   29175 - 29182 2023年

　概要を見る

Single image deblurring aims to restore a sharp image by removing blurred areas in the single image. Such blurred images are not only visually unpleasant, but cause various problems in many applications like image recognition. In recent years, with the development of deep learning, neural networks are used in many single image deblurring. Especially, encoder-decoder structures are widely used for single image deblurring and successfully restore high quality images. However, FLOPs and the number of parameters tend to increase to restore a high-quality image. Thus, this paper proposes a new lightweight network (IRFTNet) based on UNet, which is a widely used basic network of encoder-decoder structures. Our proposed network has three features to improve performance and lightweight. First, a new backbone called Inverted Residual Fourier Transformation block (IRFTblock) based on inverted residual block is introduced to decrease computational complexity. Second, a new module called Lower Feature Synthesis (LFS) is introduced to efficiently transfer encoder information from lower layers to upper layers. Finally, multiple outputs structure proposed in MIMO-UNet is introduced. These improvements resulted in a 32.98dB in PSNR on the GoPro dataset, despite approximately half FLOPs and the number of parameters of DeepRFT. Further ablation studies show the effectiveness of various components in our proposed model.

• Access to Document (DOI)

Enhanced Unpaired Image-to-Image Translation via Transformation in Saliency Domain

Shibasaki K., Ikehara M.

IEEE Access 11   137495 - 137505 2023年

　概要を見る

Unpaired image to image translation is the task of converting images in unpaired datasets. The primary goal of the task is to translate a source image into the image aligned with the target domain while keeping the fundamental content. Existing researches have introduced effective techniques to translate images with unpaired datasets, focusing on preserving the fundamental content. However, these techniques have limitations in dealing with significant shape changes and preserving backgrounds that should not be transformed. The proposed method attempts to address these problems by utilizing the saliency domain for translation and simultaneously learning the translation in the saliency domain as well as in the image domain. The saliency domain represents the shape and position of the main object. The explicit learning of transformations within the saliency domain improves network's ability to transform shapes while maintaining the background. Experimental results show that the proposed method successfully addresses the problems of unpaired image to image translation and achieves competitive metrics with existing methods.

• Access to Document (DOI)

Enhanced Residual Fourier Transformation Network for Lightweight Image Super-resolution

Yang Y., Ikehara M.

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023    826 - 832 2023年

　概要を見る

Residual blocks have been widely used in the single image super-resolution (SR), boosting the performance of SR results. The stacking of residual blocks helps to grab more information needed for the recovery of high-resolution (HR) images, but at the cost of larger model size and computational complexity. To address this problem, we propose a lightweight enhanced residual fourier transformation network (ERFTN). To ensure the network is lightweight enough while being capable of capturing both spatial and frequency information, we introduce a novel enhanced residual fourier transformation block (ERFTB) with an enhanced spatial attention (ESA) block. In addition, we adopt the contrastive loss for training acceleration without additional parameters. Extensive experiments demonstrate that our method can achieve SR performance superior to the state-of-the-art SR methods while reducing approximately half of the number of parameters (e.g., with 190K number of parameters, achieving 32.39dB in PSNR on Urban100 dataset).

• Access to Document (DOI)

RGB-D Salient Object Detection Using Saliency and Edge Reverse Attention

Ikeda T., Ikehara M.

IEEE Access 11   68818 - 68825 2023年

　概要を見る

RGB-D salient object detection is a task to detect visually significant objects in an image using RGB and depth images. Although many useful CNN-based methods have been proposed in the past, there are some problems such as blurring of object boundaries and inability to detect important parts of objects, leading to a decrease in detection accuracy. In this paper, we propose RGB-D salient object detection using Saliency and Edge Reverse Attention(SERA), which combines the fusion of saliency and edge features with reverse attention. The reverse attention process has the effect of making it easier to capture the boundaries of objects and undetected objects by inverting the up-sampled saliency features and weighting other saliency features, and the edge reverse attention process has the effect of making salient object regions stand out by inverting the edge features and weighting saliency features. The interaction between the salient object features and the edge features enhances each other's features and refines the information on the boundary of objects and salient object regions. In addition, to make it easier to refer to the global information of an image, we introduced the Multi-Scale Interactive Module(MSIM), which is capable of acquiring information at rich scales by converting feature maps to different resolutions and interacting with them. In addition to the salient object output, supervised learning is applied to multiple edge outputs of each resolution to improve the accuracy of both salient objects and boundary areas. Experimental results on five benchmarks show that the proposed method quantitatively performs better than the conventional method, and qualitatively improves the sharpness of object boundaries and the accuracy of detecting important parts of objects.

• Access to Document (DOI)

UNet Based Multi-Scale Recurrent Network for Lightweight Video Deblurring

Yae S., Ikehara M.

IEEE Access 11   117520 - 117527 2023年

　概要を見る

With the recent widespread use of smartphones and digital video cameras, the opportunities to handle digital video have increased significantly. However, despite improvements in the performance of the hardware, the captured video often contains information that is not necessary for the purpose of the video. In particular, factors such as camera shake and object movement can cause blurring in video. So, we propose a method to deblur a video by software processing of the video after shooting. In conventional methods, video deblurring is often performed using a network whose main task is video super-resolution. However, in super-resolution, the size of the input and output images are different, while the input and output images are the same size in deblurring. In the case of deblurring, the input images are input to the network after a simple downsampling process, which is not optimized for the same size as the input and output images. Therefore, the proposed method constructs a multi-scale network based on UNet. The UNet-based network is a successful method for single image deblurring. Because a video is a sequence of multiple images, we use a method that has been successful in single image deblurring. Furthermore, we add improvements to the network based on the structure of MPRNet. The feature extraction modules of the bottom and the second stage of the network are replaced with a single-stage UNet. These improvements resulted in a 34.80dB in PSNR and 0.973 in SSIM on the GoPro dataset despite about 75% of the FLOPs of BasicVSR++ and 3% of the FLOPs of VRT. On the DVD dataset, the proposed model achieved 34.36dB in PSNR and 0.966 in SSIM. Further ablation studies show the effectiveness of various components in our proposed model.

• Access to Document (DOI)

Training Large Kernel Convolutions with Resized Filters and Smaller Images

Fukuzaki S., Ikehara M.

GCCE 2023 - 2023 IEEE 12th Global Conference on Consumer Electronics    32 - 33 2023年

　概要を見る

Convolution is an essential component in neural networks for vision tasks. Convolutions on large receptive fields are more suitable than small kernel convolutions when aggregating global features in convolutional neural networks. However, large kernel convolutions require much computation and memory usage, slowing training neural networks. Then, we propose to train convolution weights with small images, resizing the convolution filters. While this idea shortens the time for training filters, simply applying this causes profound degradation. In this paper, we introduce four techniques that suppress degradation; weight scaling, removing Batch Normalization, defining a minimum resolution, and training with various-size images. In our experiment, we apply our proposals to train an image classification model based on RepLKNet-B on the image classification task of the CIFAR-100 dataset. Training with our proposals is approximately eight times faster than conventional training on the target spatial scale, keeping its accuracy.

• Access to Document (DOI)

Toward High-Quality Real-Time Video Denoising With Pseudo Temporal Fusion Network

Shibasaki K., Ikehara M.

IEEE Access 11   81466 - 81476 2023年

　概要を見る

With the increasing availability of high-resolution video recording and streaming, there is a need for fast and high-quality video denoising methods that can handle high-resolution videos. However, many existing methods fail to achieve high-quality denoising performance and computationally effecient at the same time. This paper proposes a video denoising network, Pseudo Temporal Fusion Network (PTFN), that satisfies these requirements. PTFN adopts a new Pseudo Temporal Fusion (PTF) module that captures pseudo-temporal relationships between video frames in combination with the Temporal Shift Module. PTFN also adopts a modern ConvBlock paradigm that breaks away from the classical ConvBlock paradigm, contributing to denoising performance and computationally effecient. PTFN achieves better performance than existing video denoising methods in terms of both video quality and computational effeciency. Specifically, PTFN has only about 16.7% of the computational cost of existing lightweight methods, while it improves denoising performance. PTFN is also superior in terms of memory consumption. It can process 1080p videos with a GPU with 24 GB RAM. In addition, a lighter version (PTFN Half) can process 2K videos at high speed under the same conditions.

• Access to Document (DOI)

Stereo Matching with Supplementary Boundary Information

Hashimoto K., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2023-January 2023年

ISSN  0747668X

　概要を見る

Stereo matching, which estimates the disparity of corresponding points in paired images, has problems with accuracy degraded by occlusion, repeated texture, texture-free areas, and gloss. To address this problem, we focused on object boundary information, which is less susceptible to such problems. This is because occlusion is more likely to occur at the boundaries, and even if the corresponding points disappear due to occlusion, the boundaries still have the corresponding parts. The boundary information is also robust to such problems because it is not affected by repeated textures, untextured areas, or gloss. Therefore, we propose a method for generating the boundary ground truth from the depth ground truth and develop a network and loss function for estimating the boundary information. We also propose that the accuracy can be improved by incorporating the estimated boundary information into a state-of-the-art network as an auxiliary.

• Access to Document (DOI)

Single Image Fence Removal Using Fast Fourier Transform

Kume K., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2023-January 2023年

ISSN  0747668X

　概要を見る

Fences extensively cover the image in a variety of shapes, making fence removal from the image is a challenging task. Most fence removal in a single image is performed in two stages: fence area estimation and inpainting as shown in the Fig 1. Conventional fence detection has utilized information such as edge information, luminance information, and fence regularity to deal with various shapes. However, those methods have resulted in missing fences and false detections of structures similar to fences. In this paper, we propose a network that incorporates a module that includes a fast Fourier transform. This allows us to extract global features of fences that extensively cover the image in various shapes. Furthermore, we create a dataset of rectangular fences, which is not available in previous datasets. Experimental results on existing and proposed datasets show that our method achieves better results, both quantitatively and qualitatively, by suppressing missing fences and false positives of other structures.

• Access to Document (DOI)

People Removal Using Edge and Depth Information

Yae S., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2022-January 2022年

ISSN  0747668X

　概要を見る

In this paper, we propose a people removal method from a single image for privacy and other reasons using a three-stage network of depth estimation, semantic segmentation, and inpainting, as shown in Fig. 1. In this three-stage network, we improve semantic segmentation for detecting people. We focus on a special situation of a person and construct a network. It is known that the accuracy of conventional methods can be improved by using edge information. The accuracy of segmentation can be further improved by increasing the accuracy of the edge map. In addition, edge detection does not work well when the person and the background are of the similar color, because edge detects the brightness change of the image. Therefore, in this paper, an adversarial loss function for edge maps is proposed. In addition, since an image with people is expected to have a depth difference from the background image, we use a trained depth estimation network to include the depth image in the input. In this way, it is possible to construct a network for people removal with a high accuracy both quantitatively and qualitatively.

• Access to Document (DOI)

MULTI-STAGE FEATURE ALIGNMENT NETWORK FOR VIDEO SUPER-RESOLUTION

Suzuki K., Ikehara M.

Proceedings - International Conference on Image Processing, ICIP    2001 - 2005 2022年

ISSN  15224880

　概要を見る

Video super-resolution aims at generating high-resolution video frames using multiple adjacent low-resolution frames. An important aspect of video super-resolution is the alignment of neighboring frames to the reference frame. Previous methods directly align the frames either using optical flow or deformable convolution. However, directly estimating the motion from low-resolution inputs is hard since they often contain blur and noise that hinder the image quality. To address this problem, we propose to conduct feature alignment across multiple stages to more accurately align the frames. Furthermore, to fuse the aligned features, we introduce a novel Attentional Feature Fusion Block that applies a spatial attention mechanism to avoid areas with occlusion or misalignment. Experimental results show that the proposed method achieves competitive performance to other state-of-the-art super-resolution methods while reducing the network parameters.

• Access to Document (DOI)