論文 - 川島　英之

Applying Pwrake Workflow System and Gfarm File System to Telescope Data Processing

Tanaka M., Tatebe O., Kawashima H.

Proceedings - IEEE International Conference on Cluster Computing, ICCC （Proceedings - IEEE International Conference on Cluster Computing, ICCC）  2018-September   124 - 133 2018年10月

ISSN  9781538683194

　概要を見る

In this paper, we describe a use case applying a scientific workflow system and a distributed file system to improve the performance of telescope data processing. The application is pipeline processing of data generated by Hyper Suprime-Cam (HSC) which is a focal plane camera mounted on the Subaru telescope. In this paper, we focus on the scalability of parallel I/O and core utilization. The IBM Spectrum Scale (GPFS) used for actual operation has a limit on scalability due to the configuration using storage servers. Therefore, we introduce the Gfarm file system which uses the storage of the worker node for parallel I/O performance. To improve core utilization, we introduce the Pwrake workflow system instead of the parallel processing framework developed for the HSC pipeline. Descriptions of task dependencies are necessary to further improve core utilization by overlapping different types of tasks. We discuss the usefulness of the workflow description language with the function of scripting language for defining complex task dependency. In the experiment, the performance of the pipeline is evaluated using a quarter of the observation data per night (input files: 80 GB, output files: 1.2 TB). Measurements on strong scaling from 48 to 576 cores show that the processing with Gfarm file system is more scalable than that with GPFS. Measurement using 576 cores shows that our method improves the processing speed of the pipeline by 2.2 times compared with the method used in actual operation.

• Access to Document (DOI)

Integration of parallel write ahead logging and cicada concurrency control method

Tanabe T., Kawashima H., Tatebe O.

Proceedings - 2018 IEEE International Conference on Smart Computing, SMARTCOMP 2018 （Proceedings - 2018 IEEE International Conference on Smart Computing, SMARTCOMP 2018）     291 - 296 2018年07月

ISSN  9781538647059

　概要を見る

We proposed the idea of combining the Cicada concurrency control mechanism with P-WAL (parallel write-Ahead logging) and experimentally evaluated the proposed control mechanism. Our results show that P-WAL can be combined suitably with Cicada without decreasing Cicada's performance. We then experimentally evaluated the most optimal parallel write-Ahead logging parameter among (1) conservative lock release; (2) early lock release, the early release of locks on database objects; and (3) group commit, which writes the transaction log to storage all at once. The results show that early lock release and group commit were the most optimal parameters for the combination of Cicada and P-WAL.

• Access to Document (DOI)

Performing external join operator on PostgreSQL with data transfer approach

Takizawa R., Kawashima H., Mitsuhashi R., Tatebe O.

ACM International Conference Proceeding Series （ACM International Conference Proceeding Series）     271 - 277 2018年01月

ISSN  9781450353724

　概要を見る

With the development of sensing devices, the size of data managed by human being has been rapidly increasing. To manage such huge data, relational database management system (RDBMS) plays a key role. RDBMS models the real world data as n-ary relational tables. Join operator is one of the most important relational operators, and its acceleration has been studied widely and deeply. How can an RDBMS provide such an efficient join operator? The performance improvement of join operator has been deeply studied for a decade, and many techniques are proposed already. The problem that we face is how to actually use such excellent techniques in real RDBMSs. We propose to implement an efficient join technique by the data transfer approach. The approach makes a hook point inside an RDBMS internal, and pulls data streams from the operator pipeline in the RDBMS, and applies our original join operator to the data, and finally returns the result to the operator pipeline in the RDBMS. The result of the experiment showed that our proposed method achieved 1.42x speedup compared with PostgreSQL. Our code is available on GitHub.

• Access to Document (DOI)