このエントリーをはてなブックマークに追加

s3fs vs s3fuse

Amazon S3をFUSEをつかってファイルシステムのように扱えるソフトウェアとしてs3fsが割と有名だけど、 s3fsにインスパイアされてできたs3fuseというものもある。

コードを見た感じs3fuseのほうが奇麗だったのでs3fuseいいなと思ったけど、 パフォーマンス悪いとアレなので簡単な試験をして比較をしてみた。

  • 必須条件
    • httpsを仕様
    • tokyo regionを使う
  • s3fs 1.61
# vi /etc/passwd-s3fs
accessKeyId:secretAccessKey
# /usr/local/bin/s3fs barbar -o url=https://s3-ap-northeast-1.amazonaws.com /mnt/
# ./perf.sh

prepare: removing all ...
--- create 1M file
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.319318 s, 3.3 MB/s

real    0m0.591s
user    0m0.002s
sys     0m0.010s
--- done
--- create 5M file
1+0 records in
1+0 records out
5242880 bytes (5.2 MB) copied, 0.714269 s, 7.3 MB/s

real    0m0.977s
user    0m0.001s
sys     0m0.026s
--- done
--- create 10M file
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 1.27239 s, 8.2 MB/s

real    0m1.542s
user    0m0.000s
sys     0m0.091s
--- done
--- create 50M file
1+0 records in
1+0 records out
52428800 bytes (52 MB) copied, 6.42343 s, 8.2 MB/s

real    0m6.676s
user    0m0.000s
sys     0m0.266s
--- done
--- create 100M file
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 13.2172 s, 7.9 MB/s

real    0m13.503s
user    0m0.000s
sys     0m0.722s
--- done
--- create 200M file
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 24.4316 s, 8.6 MB/s

real    0m24.688s
user    0m0.000s
sys     0m0.685s
--- done
--- cat 1M file

real    0m0.264s
user    0m0.000s
sys     0m0.002s
--- done
--- cat 5M file

real    0m0.685s
user    0m0.000s
sys     0m0.004s
--- done
--- cat 10M file

real    0m1.241s
user    0m0.000s
sys     0m0.009s
--- done
--- cat 50M file

real    0m5.526s
user    0m0.001s
sys     0m0.028s
--- done
--- cat 100M file

real    0m11.370s
user    0m0.000s
sys     0m0.054s
--- done
--- cat 200M file

real    0m21.721s
user    0m0.000s
sys     0m0.090s
--- done
prepare: removing all ...
--- create new non-zero byte 50 files

real    0m18.273s
user    0m0.031s
sys     0m0.078s
--- done
--- copy non-zero byte 50 files

real    0m22.181s
user    0m0.040s
sys     0m0.155s
--- done
--- cat non-zero byte 50 files

real    0m4.348s
user    0m0.041s
sys     0m0.093s
--- done
--- mv non-zero byte 50 files

real    0m16.025s
user    0m0.041s
sys     0m0.107s
--- done
--- rm non-zero byte 50 files

real    0m4.690s
user    0m0.051s
sys     0m0.070s
--- done
prepare: removing all ...
--- touch zero byte 50 files

real    0m22.865s
user    0m0.046s
sys     0m0.091s
--- done
--- touch already exist 50 files

real    0m18.976s
user    0m0.050s
sys     0m0.080s
--- done
prepare: create 50 files ...
--- mv directory

real    0m12.328s
user    0m0.001s
sys     0m0.002s
--- done
--- rm directory and 50 files

real    0m6.503s
user    0m0.001s
sys     0m0.013s
--- done
  • s3fuse 0.12
# umount /mnt
# vi /etc/s3fuse.conf
service=aws
bucket_name=barbar
auth_data=/etc/s3fuse/aws.key
aws_service_endpoint=s3-ap-northeast-1.amazonaws.com
# mkdir /etc/s3fuse
# vi /etc/s3fuse/aws.key
accessKeyId secretAccessKey
# /usr/local/bin/s3fuse -o config=/etc/s3fuse.conf  /mnt/
# ./perf.sh

prepare: removing all ...
--- create 1M file
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.2001 s, 5.2 MB/s

real    0m0.397s
user    0m0.000s
sys     0m0.011s
--- done
--- create 5M file
1+0 records in
1+0 records out
5242880 bytes (5.2 MB) copied, 0.765021 s, 6.9 MB/s

real    0m0.888s
user    0m0.001s
sys     0m0.040s
--- done
--- create 10M file
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 2.02609 s, 5.2 MB/s

real    0m2.142s
user    0m0.000s
sys     0m0.044s
--- done
--- create 50M file
1+0 records in
1+0 records out
52428800 bytes (52 MB) copied, 8.37144 s, 6.3 MB/s

real    0m8.499s
user    0m0.001s
sys     0m0.447s
--- done
--- create 100M file
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 16.3551 s, 6.4 MB/s

real    0m16.469s
user    0m0.001s
sys     0m0.798s
--- done
--- create 200M file
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 32.0709 s, 6.5 MB/s

real    0m32.183s
user    0m0.000s
sys     0m1.231s
--- done
--- cat 1M file

real    0m0.296s
user    0m0.001s
sys     0m0.002s
--- done
--- cat 5M file

real    0m1.499s
user    0m0.000s
sys     0m0.004s
--- done
--- cat 10M file

real    0m2.109s
user    0m0.000s
sys     0m0.008s
--- done
--- cat 50M file

real    0m6.392s
user    0m0.001s
sys     0m0.025s
--- done
--- cat 100M file

real    0m11.953s
user    0m0.000s
sys     0m0.046s
--- done
--- cat 200M file

real    0m23.393s
user    0m0.000s
sys     0m0.095s
--- done
prepare: removing all ...
--- create new non-zero byte 50 files

real    0m7.818s
user    0m0.029s
sys     0m0.090s
--- done
--- copy non-zero byte 50 files

real    0m13.207s
user    0m0.060s
sys     0m0.160s
--- done
--- cat non-zero byte 50 files

real    0m2.809s
user    0m0.055s
sys     0m0.071s
--- done
--- mv non-zero byte 50 files

real    0m10.916s
user    0m0.038s
sys     0m0.157s
--- done
--- rm non-zero byte 50 files

real    0m2.685s
user    0m0.049s
sys     0m0.082s
--- done
prepare: removing all ...
--- touch zero byte 50 files

real    0m8.230s
user    0m0.057s
sys     0m0.128s
--- done
--- touch already exist 50 files

real    0m5.916s
user    0m0.050s
sys     0m0.113s
--- done
prepare: create 50 files ...
--- mv directory

real    0m1.041s
user    0m0.000s
sys     0m0.003s
--- done
--- rm directory and 50 files

real    0m1.603s
user    0m0.003s
sys     0m0.002s
--- done

この2つを比較すると、大きいファイルの読み書きに関してはs3fsが早く、細かいファイルが複数ある場合はs3fuseの方が有利に見える。 複数のファイル処理にs3fuseが強いのは並列でリクエストを出しているのがうまく効いているためと思われる。 大きいファイル処理が遅いという点に関しては、実はs3fuseの方はupload時5Mをchunkサイズと見なし、 それ以上の場合にはamazon s3の定義するところのmultipart転送になる。(HTTPのmultipartとは無関係) そこで、s3fuseのchunkサイズをs3fsと同じ20Mにして再測定してみた。併せてダウンロードのチャンクサイズも20Mに変更した。

  • s3fuse 0.12 chunk 20M
# umount /mnt
# vi /etc/s3fuse.conf
service=aws
bucket_name=barbar
auth_data=/etc/s3fuse/aws.key
aws_service_endpoint=s3-ap-northeast-1.amazonaws.com
upload_chunk_size=20971520
download_chunk_size=20971520
# /usr/local/bin/s3fuse -o config=/etc/s3fuse.conf  /mnt/
# ./perf.sh

prepare: removing all ...
--- create 1M file
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.193304 s, 5.4 MB/s

real    0m0.375s
user    0m0.001s
sys     0m0.014s
--- done
--- create 5M file
1+0 records in
1+0 records out
5242880 bytes (5.2 MB) copied, 0.793856 s, 6.6 MB/s

real    0m0.910s
user    0m0.000s
sys     0m0.069s
--- done
--- create 10M file
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 1.3154 s, 8.0 MB/s

real    0m1.419s
user    0m0.000s
sys     0m0.093s
--- done
--- create 50M file
1+0 records in
1+0 records out
52428800 bytes (52 MB) copied, 8.77542 s, 6.0 MB/s

real    0m8.881s
user    0m0.000s
sys     0m0.617s
--- done
--- create 100M file
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 16.4549 s, 6.4 MB/s

real    0m16.559s
user    0m0.001s
sys     0m1.035s
--- done
--- create 200M file
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 34.1758 s, 6.1 MB/s

real    0m34.297s
user    0m0.000s
sys     0m2.148s
--- done
--- cat 1M file

real    0m0.361s
user    0m0.002s
sys     0m0.000s
--- done
--- cat 5M file

real    0m0.750s
user    0m0.001s
sys     0m0.007s
--- done
--- cat 10M file

real    0m1.289s
user    0m0.000s
sys     0m0.008s
--- done
--- cat 50M file

real    0m5.260s
user    0m0.000s
sys     0m0.025s
--- done
--- cat 100M file

real    0m10.232s
user    0m0.001s
sys     0m0.044s
--- done
--- cat 200M file

real    0m30.007s
user    0m0.000s
sys     0m0.095s
--- done
prepare: removing all ...
--- create new non-zero byte 50 files

real    0m7.427s
user    0m0.038s
sys     0m0.094s
--- done
--- copy non-zero byte 50 files

real    0m12.242s
user    0m0.045s
sys     0m0.169s
--- done
--- cat non-zero byte 50 files

real    0m2.525s
user    0m0.052s
sys     0m0.056s
--- done
--- mv non-zero byte 50 files

real    0m9.728s
user    0m0.057s
sys     0m0.131s
--- done
--- rm non-zero byte 50 files

real    0m2.547s
user    0m0.054s
sys     0m0.077s
--- done
prepare: removing all ...
--- touch zero byte 50 files

real    0m8.386s
user    0m0.049s
sys     0m0.104s
--- done
--- touch already exist 50 files

real    0m5.580s
user    0m0.047s
sys     0m0.110s
--- done
prepare: create 50 files ...
--- mv directory

real    0m1.116s
user    0m0.000s
sys     0m0.007s
--- done
--- rm directory and 50 files

real    0m1.619s
user    0m0.000s
sys     0m0.007s
--- done

パラメータは同じはずだけど、ほとんど変わらず。どうもs3fuseの方がmultipart実装が遅いと推測できる。 ただし、10Mの速度がs3fsより多少速くなっているので、multipartでなければs3fuseの方が速いと分かる。 そこで、いっそのことchunkサイズを200Mまであげて試してみた。

  • s3fuse 0.12 chunk 200M
# umount /mnt
# vi /etc/s3fuse.conf
service=aws
bucket_name=barbar
auth_data=/etc/s3fuse/aws.key
aws_service_endpoint=s3-ap-northeast-1.amazonaws.com
upload_chunk_size=209715200
download_chunk_size=209715200
# /usr/local/bin/s3fuse -o config=/etc/s3fuse.conf  /mnt/
# ./perf.sh

prepare: removing all ...
--- create 1M file
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.193559 s, 5.4 MB/s

real    0m0.368s
user    0m0.001s
sys     0m0.008s
--- done
--- create 5M file
1+0 records in
1+0 records out
5242880 bytes (5.2 MB) copied, 0.699863 s, 7.5 MB/s

real    0m0.847s
user    0m0.000s
sys     0m0.015s
--- done
--- create 10M file
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 1.32566 s, 7.9 MB/s

real    0m1.462s
user    0m0.000s
sys     0m0.076s
--- done
--- create 50M file
1+0 records in
1+0 records out
52428800 bytes (52 MB) copied, 6.20686 s, 8.4 MB/s

real    0m6.341s
user    0m0.000s
sys     0m0.306s
--- done
--- create 100M file
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 11.572 s, 9.1 MB/s

real    0m12.279s
user    0m0.000s
sys     0m0.351s
--- done
--- create 200M file
1+0 records in
1+0 records out
209715200 bytes (210 MB) copied, 23.506 s, 8.9 MB/s

real    0m23.693s
user    0m0.000s
sys     0m0.741s
--- done
--- cat 1M file

real    0m0.285s
user    0m0.000s
sys     0m0.001s
--- done
--- cat 5M file

real    0m0.728s
user    0m0.000s
sys     0m0.005s
--- done
--- cat 10M file

real    0m1.414s
user    0m0.000s
sys     0m0.010s
--- done
--- cat 50M file

real    0m5.520s
user    0m0.000s
sys     0m0.031s
--- done
--- cat 100M file

real    0m11.365s
user    0m0.001s
sys     0m0.044s
--- done
--- cat 200M file

real    0m22.288s
user    0m0.000s
sys     0m0.094s
--- done
prepare: removing all ...
--- create new non-zero byte 50 files

real    0m8.945s
user    0m0.030s
sys     0m0.094s
--- done
--- copy non-zero byte 50 files

real    0m15.052s
user    0m0.060s
sys     0m0.160s
--- done
--- cat non-zero byte 50 files

real    0m3.204s
user    0m0.049s
sys     0m0.061s
--- done
--- mv non-zero byte 50 files

real    0m12.038s
user    0m0.068s
sys     0m0.122s
--- done
--- rm non-zero byte 50 files

real    0m3.055s
user    0m0.046s
sys     0m0.068s
--- done
prepare: removing all ...
--- touch zero byte 50 files

real    0m8.934s
user    0m0.043s
sys     0m0.136s
--- done
--- touch already exist 50 files

real    0m6.170s
user    0m0.048s
sys     0m0.097s
--- done
prepare: create 50 files ...
--- mv directory

real    0m1.345s
user    0m0.002s
sys     0m0.004s
--- done
--- rm directory and 50 files

real    0m1.961s
user    0m0.000s
sys     0m0.007s
--- done

大きなファイルの処理に関しても、s3fsと遜色無い値になった。 ただし、multipart使わないというのはよくない場合があるかもしれない。

まとめると、s3fuseはmultipartの処理がs3fsより遅いようなので、 大きなファイルを扱う場合はs3fsの方が良さそう。 ただし、場合によってはs3fuseはチャンクサイズを適切に調整してやる事で、 大きなファイルでもs3fsと遜色ないパフォーマンスを出す事が可能。 複数の小さなファイルの処理に関してはs3fuseにアドバンテージがある。

最後にs3fuseの欠点としては、s3fuseはバケット単位でしかマウントポイントを割り当てられないのに対し、 s3fsはバケットの中のパスに対してマウントポイントを割り当てられる。 まぁ、この機能あまり需要無い気がする。

s3fuseのほうがいいけど、s3fuseのmultipart速くなると最高なのにな。

name:
email:
comment: