On the fifth day of Open Source Week, its Fire-Flyer file system (3FS) was officially open sourced. As a high-performance parallel file system, 3FS can fully utilize modern SSDs and RDMA networks to achieve high-speed data access, improving the efficiency of AI model training and inference.
Key performance indicators of 3FS:
Achieving a total read throughput of 6.6 TiB/s in a 180-node cluster;
Achieving a throughput of 3.66 TiB/minute in a 25-node GraySort benchmark test;
Peak throughput of over 40+ GiB/s for single-node KVCache queries.
3FS adopts a modular architecture, supporting data preprocessing, dataset loading, checkpoint storage/recovery, embedded vector search, and inference KVCache queries, with strong consistency semantics. DeepSeek also synchronously launched the Smallpond data processing framework to further optimize 3FS data management capabilities.
All Comments