Cointime

Download App
iOS & Android

DeepSeek open-sources and releases 3FS, a high-speed parallel file system that optimizes AI data access

  • Wechat scan to share

On the fifth day of Open Source Week, its Fire-Flyer file system (3FS) was officially open sourced. As a high-performance parallel file system, 3FS can fully utilize modern SSDs and RDMA networks to achieve high-speed data access, improving the efficiency of AI model training and inference.

Key performance indicators of 3FS:

Achieving a total read throughput of 6.6 TiB/s in a 180-node cluster;

Achieving a throughput of 3.66 TiB/minute in a 25-node GraySort benchmark test;

Peak throughput of over 40+ GiB/s for single-node KVCache queries.

3FS adopts a modular architecture, supporting data preprocessing, dataset loading, checkpoint storage/recovery, embedded vector search, and inference KVCache queries, with strong consistency semantics. DeepSeek also synchronously launched the Smallpond data processing framework to further optimize 3FS data management capabilities.

Comments

All Comments

Recommended for you