weedfs improvements / validation

goal

fork from weedfs
create certain improvements, allow author of weedfs to merge it back in
do some serious stress testing on larger scale
- deploy on 30 sata disks
- store billions of 16k objects (fill up the sata disks)
- check performance (document well)
- check what happens if certain disks get full faster than others (put some small disks in)
- remove disks (simulate broken disk), see how redundancy policy recovers (if it does that at all)
validation
- check the way how redundancy is done
  - will it pick up after disk was broken (so will it recover automatically)
  - is it a sound way to guarantee consistent redundancy
- document memory usage
  - size of keys in relation to memory usage (how many objects can be stored on storage node in relation to mem)
- document performance
  - 10 gbit / gbit network
  - small/bigger clusters
  - degradation over time when more objects
improvements
- variable key size, allow upto full sha1 keysize, this to use redis to store deduped information
- performance improvements
- redundancy improvements
- code review, fix minor issues
- implement caching on SSD (write cache) before it gets written to SATA disks at backend

how

document all in this repo
work with our professional services team in Egypt to get access to hardware which is configured properly
create some tools to do performance testing
tune the OS/filesystem to get best results (filesystems used underneath weedfs)

requirements

billion objects can be stored reliably
performance is predictable
- expressed in IOPS per system
- IOPS from total system should be not to far of of IOPS from sum of underlying SATA disks (read iops)
when node or disk down system keeps on working
when node or disk back system knows how to deal with it
when new node or new disk there is a way to let this new node & disk to become part of the cluster
when disks lost, there is a way to rebalance the data in such a way that the redundancy requirements are met !!!
when too much data (is ok), there is a way to remove it
there is a procedure to check validity of data on disks (crc check on disk?)

remarks

ignore the filesystem interface, we are only interested in a backend storage system