Massive AIS data storage and query based on Hadoop platform
Authors:
- Taizhi LvTLTaizhi Lv
- Chenyong HeCHChenyong He
- Juan ZhangJZJuan Zhang
- Zhiyang SongZSZhiyang Song
Open Access
- 1 June 2021
- conference paper
- Published by IOP Publishing in Journal of Physics: Conference Series
- Vol. 1948 (1)
Abstract
There are nearly 50 million AIS data generated every day in the Yangtze River Basin. Efficient storage and query of AIS data become the key problem of smart maritime administration. To overcome this problem, a distributed storage and query scheme based on Hadoop platform is proposed. Firstly, the original AIS message is parsed, the abnormal data is removed and the parsed data is pushed to the Kafka message queue. Secondly, the messages in the queue are pushed to different servers to complete the storage of real-time cache, historical trajectory and regional information. Finally, the query interface is provided to realize the real-time query of ship status, trajectory and regional ship in a certain period of time. The real applications show that compared with the traditional relational database, the solution can meet the requirements of real-time query and has high scalability.Keywords
This publication has 1 reference indexed in Scilit:
- The significance of incorporating unidentified vessels into AIS-based ship emission inventoryAtmospheric Environment, 2019