論文 - 池原　雅章

Enhanced Unpaired Image-to-Image Translation via Transformation in Saliency Domain

Shibasaki K., Ikehara M.

IEEE Access 11   137495 - 137505 2023年

　概要を見る

Unpaired image to image translation is the task of converting images in unpaired datasets. The primary goal of the task is to translate a source image into the image aligned with the target domain while keeping the fundamental content. Existing researches have introduced effective techniques to translate images with unpaired datasets, focusing on preserving the fundamental content. However, these techniques have limitations in dealing with significant shape changes and preserving backgrounds that should not be transformed. The proposed method attempts to address these problems by utilizing the saliency domain for translation and simultaneously learning the translation in the saliency domain as well as in the image domain. The saliency domain represents the shape and position of the main object. The explicit learning of transformations within the saliency domain improves network's ability to transform shapes while maintaining the background. Experimental results show that the proposed method successfully addresses the problems of unpaired image to image translation and achieves competitive metrics with existing methods.

• Access to Document (DOI)

Enhanced Residual Fourier Transformation Network for Lightweight Image Super-resolution

Yang Y., Ikehara M.

2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2023    826 - 832 2023年

　概要を見る

Residual blocks have been widely used in the single image super-resolution (SR), boosting the performance of SR results. The stacking of residual blocks helps to grab more information needed for the recovery of high-resolution (HR) images, but at the cost of larger model size and computational complexity. To address this problem, we propose a lightweight enhanced residual fourier transformation network (ERFTN). To ensure the network is lightweight enough while being capable of capturing both spatial and frequency information, we introduce a novel enhanced residual fourier transformation block (ERFTB) with an enhanced spatial attention (ESA) block. In addition, we adopt the contrastive loss for training acceleration without additional parameters. Extensive experiments demonstrate that our method can achieve SR performance superior to the state-of-the-art SR methods while reducing approximately half of the number of parameters (e.g., with 190K number of parameters, achieving 32.39dB in PSNR on Urban100 dataset).

• Access to Document (DOI)

Single Image Raindrop Removal Using a Non-Local Operator and Feature Maps in the Frequency Domain

Ezumi S., Ikehara M.

Proceedings of 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2022    1343 - 1350 2022年

　概要を見る

Taking a photo on a rainy day may result in a photo with raindrops. Images containing raindrops have a significant impact on the visual impression and accuracy when applied to image recognition systems. Thus, an automatic high-quality raindrop removal method is desired for outdoor image processing systems as well as for acquiring good-looking images. Several existing methods have been proposed to tackle this problem, but they often fail to keep global consistency and generate unnatural patterns. In this paper, we tackle this prob-lem by introducing a non-local operator. The non-local operator combines features in distant locations with matrix multiplication and enables consistency in distant locations. In addition, high-frequency components such as edges are more affected in images with raindrops. Inspired by the nature that high-frequency components can be separated from other components in the frequency domain, we also propose to process feature maps in the frequency domain, which are obtained by the fast Fourier transform operation and processed by several convolution layers. Experimental results show that our method effectively removes raindrops and achieves state-of-the-art performance.

• Access to Document (DOI)

Single Image Raindrop Removal Using a Non-Local Operator and Feature Maps in the Frequency Domain

Ezumi S., Ikehara M.

IEEE Access 10   91976 - 91983 2022年

　概要を見る

Taking a photo on a rainy day may result in a photo with raindrops. Images containing raindrops have a significant impact on the visual impression and accuracy when applied to image recognition systems. Thus, an automatic high-quality raindrop removal method is desired for outdoor image processing systems as well as for acquiring good-looking images. Several existing methods have been proposed to tackle this problem, but they often fail to keep global consistency and generate unnatural patterns. In this paper, we tackle this problem by introducing a non-local operator. The non-local operator combines features in distant locations with matrix multiplication and enables consistency in distant locations. In addition, high-frequency components such as edges are more affected in images with raindrops. Inspired by the nature that high-frequency components can be separated from other components in the frequency domain, we also propose to process feature maps in the frequency domain, which are obtained by the fast Fourier transform operation and processed by several convolution layers. Experimental results show that our method effectively removes raindrops and achieves state-of-the-art performance.

• Access to Document (DOI)

Scale Recurrent Network for Single Image Dehazing

Imai W., Ueki Y., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2022-January 2022年

ISSN  0747668X

　概要を見る

The quality of the images taken outside is directly affected by floating atmospheric particles. To keep the quality of the image, haze removal methods play a critical role. In this paper, we propose a new multi-scale network structure that is rarely used in haze removal and a new loss function to generate high-quality haze-free images. Despite a number of proposed dehazing methods, they are not able to capture the haze accurately without being confused by other objects that are similar in color to haze and completely remove the haze throughout the image. Our proposed network architecture takes the Scale Recurrent Network structure and we incorporate dilated convolution to catch the more global features like haze with the wide receptive field. Additionally, we train our network with a new loss function using dark channel prior to more effectively learn the haze features. Compared with state-of-the-art methods, our proposed method achieves better results on both synthetic and real-world images.

• Access to Document (DOI)

Saturated Reflection Detection for Reflection Removal Based on Convolutional Neural Network

Yoshida T., Funahashi I., Yamashita N., Ikehara M.

IEEE Access 10   39800 - 39809 2022年

　概要を見る

Single image reflection removal is a technique that removes undesirable reflections, which occur due to glass, from images. Various methods of reflection removal have been proposed, but unfortunately, they usually fail to remove reflections particularly with very high pixel values. In this paper, we define these saturated reflections and their characteristics, as well as discuss and propose a removal system. The proposed system detects areas of saturated reflections based on our proposed model of convolutional neural networks and restores them by a conventional method of image estimation. In our experiments, the proposed system shows better peak-signal-to-noise ratio scores and perceptual quality than conventional methods of reflection removal.

• Access to Document (DOI)

Principal Components of Neural Convolution Filters

Fukuzaki S., Ikehara M.

IEEE Access 10   104328 - 104336 2022年

　概要を見る

Convolutions in neural networks are still essential on various vision tasks. To develop neural convolutions, this study focuses on Structured Receptive Field (SRF), representing a convolution filter as a linear combination of widely acting designed components. Although SRF can represent convolution filters with fewer components than the number of filter bins, N-Jet, the sole component system implementation, requires ten trainable parameters per filter to improve accuracy even for 3×3 convolutions. Hence, we aim to formulate a new component system for SRF that can represent valid filters with fewer components. Our component system named 'OtX' is based on the Principal Component Analysis of well-trained filter weights because the extracted components will also be principal for neural convolution filters. In addition to proposing the component system, we develop a component scaling method to defuse massive scale differences among the coefficients in a linear combination of OtX components. In the experimental section, we train image classification models on CIFAR-100 dataset under the hyperparameters tuned for the original models with the standard convolutions. For NFNet-F0 classifier, OtX with six components performs 0.5% better than the standard convolution, 3.1% better than N-Jet with six components, and only 0.1% worse than N-Jet with ten components. Besides, OtX with nine components provides stabler training than N-Jet, performing 0.5% better than the standard for NFNet-F0. OtX suits when replacing standard convolutions because OtX performs at least comparably against N-Jet with further parameter efficiency and training stability.

• Access to Document (DOI)

People Removal Using Edge and Depth Information

Yae S., Ikehara M.

Digest of Technical Papers - IEEE International Conference on Consumer Electronics 2022-January 2022年

ISSN  0747668X

　概要を見る

In this paper, we propose a people removal method from a single image for privacy and other reasons using a three-stage network of depth estimation, semantic segmentation, and inpainting, as shown in Fig. 1. In this three-stage network, we improve semantic segmentation for detecting people. We focus on a special situation of a person and construct a network. It is known that the accuracy of conventional methods can be improved by using edge information. The accuracy of segmentation can be further improved by increasing the accuracy of the edge map. In addition, edge detection does not work well when the person and the background are of the similar color, because edge detects the brightness change of the image. Therefore, in this paper, an adversarial loss function for edge maps is proposed. In addition, since an image with people is expected to have a depth difference from the background image, we use a trained depth estimation network to include the depth image in the input. In this way, it is possible to construct a network for people removal with a high accuracy both quantitatively and qualitatively.

• Access to Document (DOI)

MULTI-STAGE FEATURE ALIGNMENT NETWORK FOR VIDEO SUPER-RESOLUTION

Suzuki K., Ikehara M.

Proceedings - International Conference on Image Processing, ICIP    2001 - 2005 2022年

ISSN  15224880

　概要を見る

Video super-resolution aims at generating high-resolution video frames using multiple adjacent low-resolution frames. An important aspect of video super-resolution is the alignment of neighboring frames to the reference frame. Previous methods directly align the frames either using optical flow or deformable convolution. However, directly estimating the motion from low-resolution inputs is hard since they often contain blur and noise that hinder the image quality. To address this problem, we propose to conduct feature alignment across multiple stages to more accurately align the frames. Furthermore, to fuse the aligned features, we introduce a novel Attentional Feature Fusion Block that applies a spatial attention mechanism to avoid areas with occlusion or misalignment. Experimental results show that the proposed method achieves competitive performance to other state-of-the-art super-resolution methods while reducing the network parameters.

• Access to Document (DOI)

Local Image Denoising Using RAISR

Zin T., Seta S., Nakahara Y., Yamaguchi T., Ikehara M.

IEEE Access 10   22420 - 22428 2022年

　概要を見る

Digital images are frequently degraded by Gaussian noise while capturing photos. This paper proposes a rapid and high accurate Gaussian noise removal method by applying the learned linear filter used in RAISR for super-resolution. The denoising methods are classified into local, nonlocal methods and deep-learning-based methods. The conventional local processing has a problem that high-frequency components of the original image are lost while reducing the noise. The nonlocal and deep-learning-based methods achieve higher denoising performance but take a long time for training and implementation. To solve these problems, we apply a super-resolution method to the local denoising method as post-processing because it can efficiently recover the high-frequency components. The super-resolution method uses a learned linear filter according to the feature of patches. The novelty of this paper is that the same processing as super-resolution is incorporated into denoising. The proposed algorithm is a rapid local denoising method and can achieve comparable performance to the high-accurate nonlocal denoising methods. Experimental results show that our proposed method provides accurate denoising performance with a low computational cost compared to nonlocal processing like BM3D.

• Access to Document (DOI)

Enhanced Multiscale Attention Network for Single Image Dehazing

Imai W., Ikehara M.

IEEE Access 10   93626 - 93635 2022年

　概要を見る

Under severe weather conditions, the quality of the images taken outside is directly affected by floating atmospheric particles. To keep the quality of the images, haze removal methods play a critical role. The most difficult part of haze removal is removing the haze that spreads over the entire image. Many CNN-based methods have been proposed to remove the haze, and can be divided into two types. One is to use a multi-scale structure and the other is to stack layers. The former causes image degradation due to the loss of some of the original information in an image and the latter increases computational complexity due to not reducing the resolution. In addition, a large number of parameters is required to secure the expressive power of the model, which leads to a huge amount of memory. To tackle these problems, we tried to 1) downsample the image while saving parameters and maintaining the quality of the generated image, and 2) consider the information in the entire image to remove the haze. For the first problem, we tried to solve this by using a feature extractor that has been used in other tasks, learning to optimize the output image in low-resolution, and preparing kernels with various dilation rates to expand the receptive fields. For the second problem, we use the attention structure to determine which part of the image features should be focused on from the entire feature map. By incorporating such modules, our method achieves better results on both synthetic and real-world images when compared with state-of-the-art methods.

• Access to Document (DOI)

Channel Attention GAN Trained with Enhanced Dataset for Single-Image Shadow Removal

Abiko R., Ikehara M.

IEEE Access 10   12322 - 12333 2022年

　概要を見る

Even today, where many deep-learning-based methods have been published, single-image shadow removal is a challenging task to achieve high accuracy. This is because the shadow changes depending on various conditions such as the target material or the light source, and it is difficult to estimate all the physical parameters. In this paper, we propose a new single-image shadow removal method (Channel Attention GAN: CANet) using two networks for detecting shadows and removing shadows. Intensity change in shadowed regions has different characteristics depending on the wavelength of light. In addition, the image acquisition system of the camera acquires an image in a state where the RGB values influence each other. Therefore, our method focused on the physical properties of shadows and the camera's image acquisition system. The proposed network has a structure considering the relationship between color channels. When training this network, we modified the color and added some artifacts to the training images in order to make the training dataset more complex. These image processing are based on the shadow model, considering the camera image acquisition system. With these new proposals, our method can remove shadows in all ISTD, ISTD+, SRD, and SRD+ datasets with higher accuracy than the state-of-the-art methods. The code is available on GitHub: https://github.com/ryo-abiko/CANet.

• Access to Document (DOI)

4K Real Time Image to Image Translation Network With Transformers

Shibasaki K., Fukuzaki S., Ikehara M.

IEEE Access 10   73057 - 73067 2022年

　概要を見る

CNNs have traditionally been applied in computer vision. Recently, applying Transformer networks, originally a technique in natural language processing, to computer vision has received much attention and produced superior results. However, Transformers and their derivation have drawbacks that the computational cost and memory usage increase rapidly with the image resolution. In this paper, we propose the Laplacian Pyramid Translation Transformer (LPTT) for image to image translation. The Laplacian Pyramid Translation Network, a previous study of this work, creates Laplacian pyramid of the input images and processes each component with CNNs. However, LPTT transforms the high-frequency components with CNNs and the low-frequency components with Axial Transformer blocks. LPTT can have Transformer's expressive power while reducing the computational cost and memory usage. LPTT significantly improves the quality of generated images and inference speed for high-resolution images over conventional methods. LPTT is the first network with a Transformer that can perform practical inference in real time on 4K resolution images. LPTT can also process 8K images in real time depending on the model conditions and the performance of the GPU. The ablation study in this paper suggests that even when processing high-resolution images, the performance is improved while maintaining the inference speed by computing the low-resolution component with a Transformer. LPTT improves PSNR value by 0.41 dB in MIT-Adobe FiveK dataset. The greater the number of layers in the Laplacian pyramid, the greater the improvement of LPTT over the Laplacian Pyramid Translation Network.

• Access to Document (DOI)

Improved image denoising via RAISR with fewer filters

Zin T., Nakahara Y., Yamaguchi T., Ikehara M.

Computational Visual Media 7 （ 4 ） 499 - 511 2021年12月

ISSN  20960433

　概要を見る

In recent years, accurate Gaussian noise removal has attracted considerable attention for mobile applications, as in smart phones. Accurate conventional denoising methods have the potential ability to improve denoising performance with no additional time. Therefore, we propose a rapid post-processing method for Gaussian noise removal in this paper. Block matching and 3D filtering and weighted nuclear norm minimization are utilized to suppress noise. Although these nonlocal image denoising methods have quantitatively high performance, some fine image details are lacking due to the loss of high frequency information. To tackle this problem, an improvement to the pioneering RAISR approach (rapid and accurate image super-resolution), is applied to rapidly post-process the denoised image. It gives performance comparable to state-of-the-art super-resolution techniques at low computational cost, preserving important image structures well. Our modification is to reduce the hash classes for the patches extracted from the denoised image and the pixels from the ground truth to 18 filters by two improvements: geometric conversion and reduction of the strength classes. In addition, following RAISR, the census transform is exploited by blending the image processed by noise removal methods with the filtered one to achieve artifact-free results. Experimental results demonstrate that higher quality and more pleasant visual results can be achieved than by other methods, efficiently and with low memory requirements. [Figure not available: see fulltext.].

• Access to Document (DOI)

Gan-based rain noise removal from single-image considering rain composite models

Matsui T., Ikehara M.

European Signal Processing Conference 2021-January   665 - 669 2021年01月

ISSN  22195491

　概要を見る

Under severe weather conditions, outdoor images or videos captured by cameras can be affected by heavy rain and fog. In this paper, we address a single-image rain removal problem (de-raining). As compared to video-based methods, single-image based methods are challenging because of the lack of temporal information. Although many existing methods have tackled these challenges, they suffer from overfitting, over-smoothing, and unnatural hue change. To solve these problems, we propose a GAN-based de-raining method. The optimal generator is determined by experimental comparisons. To train the generator, we learn the mapping between rainy and residual images from the training dataset. Besides, we synthesize a variety of rainy images to train our network. In particular, we focus on not only the orientations and scales of rain streaks but also the rainy image composite models. Our method also achieves better performance on both synthetic and real-world images than state-of-the-art methods in terms of quantitative and visual performances.

• Access to Document (DOI)

Weighted generalization of dark channel prior with adaptive color correction for defogging

Ueki Y., Ikehara M.

European Signal Processing Conference 2021-January   685 - 689 2021年01月

ISSN  22195491

　概要を見る

Images and video captured in water or fog suffer from low contrast and color distortion due to light scattering and absorption. An image formation model for hazy images is commonly used to restore both underwater images and hazy images because of the similarity between the two types of images. However, red light is attenuated faster than blue and green light in underwater, and underwater images are distorted by changes of color tone. Therefore, most current methods are specialized for either hazy images or underwater images. In this paper, we propose a novel defogging method which is efficient for both hazy images and underwater images. Our method is composed of adaptive color correction and weighted generalization of dark channel prior (WGDCP). Experimental results show that our algorithm can recover both underwater images and hazy images.

• Access to Document (DOI)

Shadow detection and removal using GAN

Nagae T., Abiko R., Yamaguchi T., Ikehara M.

European Signal Processing Conference 2021-January   630 - 634 2021年01月

ISSN  22195491

　概要を見る

To remove shadowed region in a single image, it is important to obtain high accuracy in both two processes, shadow detection and removal. In order to improve the results, recent methods perform these two processes simultaneously and use GAN for the training. However, since these methods do not try to maintain the luminance of non-shadowed regions, the output images tend to be faded. In this paper, to overcome fading problem, we proposed a new GAN structure based on shadow model. Since our GAN-based method focus on the variation of the illuminance, the illuminances of the shadowed regions, whose amount of change are large, are effectively estimated. In addition, non-shadowed regions remain slightly faded due to our new GAN structure and training method. Owing to our novel GAN structure and training method, our method outperforms state-of-the-art methods in PSNR and SSIM.

• Access to Document (DOI)

GAN-Based Image Deblurring Using DCT Loss with Customized Datasets

Tomosada H., Kudo T., Fujisawa T., Ikehara M.

IEEE Access 9   135224 - 135233 2021年

　概要を見る

In this paper, we propose a high quality image deblurring method that uses discrete cosine transform (DCT) and requires less computational complexity. We train our model on a new dataset which is customized to include images with large motion blurs. Recently, Convolutional Neural Network (CNN) and Generative Adversarial Network (GAN) based algorithms have been proposed for image deblurring. Moreover, multi-scale and multi-patch architectures of CNN restore blurred images clearly and suppress more ringing or blocking artifacts, but they take a longer time to process. To improve the quality of deblured images and reduce the computational time, we propose a method called 'DeblurDCTGAN' that preserves texture and suppresses ringing artifacts in the restored image without multi-scale or multi-patch architecture using DCT based loss. This loss compares the restored image and the ground truth image in the frequency domain. With this loss, DeblurDCTGAN can reduce block noise and ringing artifacts while maintaining deblurring performance. Our experimental results show that DeblurDCTGAN gets the highest performances in terms of PSNR, SSIM, and running time compared with conventional methods. In terms of real image datasets, DeblurDCTGAN shows a better performance by using a customized training dataset made from GoPro, DVD, NFS and HIDE training datasets. Experimented code with pre-trained weights, datasets and results are available at https://github.com/Hiroki-Tomosada/DCTGAN-master

• Access to Document (DOI)

Underwater Image Enhancement with Multi-Scale Residual Attention Network

Ueki Y., Ikehara M.

2021 International Conference on Visual Communications and Image Processing, VCIP 2021 - Proceedings 2021年

　概要を見る

Underwater images suffer from low contrast, color distortion and visibility degradation due to the light scattering and attenuation. Over the past few years, the importance of underwater image enhancement has increased because of ocean engineering and underwater robotics. Existing underwater image enhancement methods are based on various assumptions. However, it is almost impossible to define appropriate assumptions for underwater images due to the diversity of underwater images. Therefore, they are only effective for specific types of underwater images. Recently, underwater image enhancement algorisms using CNNs and GANS have been proposed, but they are not as advanced as other image processing methods due to the lack of suitable training data sets and the complexity of the issues. To solve the problems, we propose a novel underwater image enhancement method which combines the residual feature attention block and novel combination of multi-scale and multi-patch structure. Multi-patch network extracts local features to adjust to various underwater images which are often Non-homogeneous. In addition, our network includes multi-scale network which is often effective for image restoration. Experimental results show that our proposed method outperforms the conventional method for various types of images.

• Access to Document (DOI)

High Reflection Removal Using CNN with Detection and Estimation

Funahashi I., Yamashita N., Yoshida T., Ikehara M.

2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2021 - Proceedings    1381 - 1385 2021年

　概要を見る

In this paper, we propose a method of reflection removal that reduces high intensity reflection for single image. Various methods of reflection removal have been proposed, but they fail to reduce the high reflections due to their assumption. To tackle this issue, the proposed method detects the target areas with high reflections by the proposed convolutional neural network (CNN) model and estimates their background information by inpainting. It is observed that the reflection is strongly blurred because of its physical property, and hence the proposed CNN model utilizes edge features of pixels for the detection. In simulation, we compare state-of-the-art methods of reflection removal with and without the proposed method for natural images, and the proposed method improves peak signal-to-noise ratio and perceptual quality.