国产化环境下的海量小文件数据分布式存储技术
投稿时间:2022-07-06  修订日期:2022-09-20  点此下载全文
引用本文:
摘要点击次数: 25
全文下载次数: 0
作者单位邮编
梁懿 福建亿榕信息技术有限公司 350001
刘迪 国网信息通信产业集团有限公司 
陈又咏 福建亿榕信息技术有限公司 
董晓祺 福建亿榕信息技术有限公司 
许志毅 福建亿榕信息技术有限公司 
基金项目:信创公共应用关键技术研究及标准规范制定项目(53681121000900K0000000)
中文摘要:为缓解单一存储设备存储海量小文件的压力,提出一种国产化环境下的海量小文件数据分布式存储技术。利用聚类算法实现海量小文件合并。以均衡度最大为目标,在多项约束条件下利用人工鱼群算法求解分布式存储方案。按照分布式存储方案将海量小文件数据迁移到存储节点及其存储设备上,完成海量小文件数据分布式存储。结果表明:14个存储节点和28个存储设备的内存占用较为均衡,内存资源利用率较高。将小文件样本迁移并存储到节点的过程中,分布式存储均衡度整体波动均超过设定的阈值1.0,说明分布式存储均衡度较好,证明了所提存储技术的有效性。
中文关键词:国产化环境  海量小文件数据  数据合并  数据迁移  分布式存储技术
 
Distributed storage technology of massive small file data in localization environment
Abstract:In order to alleviate the pressure of a single storage device to store large amounts of small files, a distributed storage technology for large amounts of small file data in a domestic environment is proposed. Using clustering algorithm to merge large amount of small files. Taking the maximum degree of equilibrium as the goal, the artificial fish swarm algorithm is used to solve the distributed storage scheme under multiple constraints. According to the distributed storage scheme, the massive small file data is migrated to the storage nodes and their storage devices to complete the distributed storage of massive small file data. The results show that the memory occupation of 14 storage nodes and 28 storage devices is relatively balanced, and the utilization rate of memory resources is high. In the process of migrating and storing small file samples to nodes, the overall fluctuation of distributed storage balance exceeds the set threshold of 1.0, indicating that the distributed storage balance is good, which proves the effectiveness of the proposed storage technology.
keywords:Localization environment  Massive small file data  Data consolidation  Data migration  Distributed storage technology
查看全文   查看/发表评论   下载pdf阅读器