File Storage Engine
(FSEngine) is a write-optimized object storage library designed mainly for storing large, continuous streams on commodity hardware and Disk-Drives. This project includes APIs and primitives for storing raw data and fetching data from the storage. Use-cases of this library include storing live video streams, logs, metadata, or any write-optimized workload.
This library might be used with ArchiverMedia
wrapper library for easily transforming media objects into consumable binary packets for FSEngine
.
This library uses a pseudo log storage model for storing multiple data streams into a single file. It also supports a circular storage model for a fixed-sized store.
Currently, FSEngine has been offered as a Golang only library, and all APIs are usable within any Golang 1.12+ project.
- Write-optimized storage
- Local storage
- Circular storage
- Multi-instance storage
- Spare storage with ECC
- To install FSEngine library into and existing project, having Go installed, use the bellow Go command to install FSEngine.
$ go get github.com/fanap-infra/fsEngine
- Import it to your library
import "github.com/fanap-infra/fsEngine"
First you need to establish storage location on local system and initialize storage engine by maximum size of storage used by the engine:
package main
import (
"github.com/fanap-infra/fsEngine"
"github.com/fanap-infra/log"
)
type EventsTrigger struct {
fileID uint32
}
func main() {
// define storage location and size
const storagePath = "/var/fsEngine/volume1"
const storageSize = 1 << 32 // 4GB volume
fileSystem, err := fsEngine.CreateFileSystem(storagePath, storageSize,
fsEngine.BLOCKSIZE, &EventsTrigger{}, log.GetScope("Example"))
if err != nil {
log.Fatal(err)
}
return
}
// VirtualFileDeleted
// Implement event listener.
func (e EventsTrigger) VirtualFileDeleted(fileID uint32, message string) {
log.Warnf("File with ID:%v, deleted: %s", fileID, message)
return
}
This is the barebone definition for using FSEngine, however, to store objects you have to create a virtual identity for the object which here is called a virtualFile
.
fileSystem, err := fsEngine.CreateFileSystem(storagePath, storageSize,
fsEngine.BLOCKSIZE, &EventsTrigger{}, log.GetScope("Example"))
if err != nil {
log.Fatal(err)
}
const vfID = 1
const vfName = "Hello"
virtualFile, err := fileSystem.NewVirtualFile(vfID, vfName)
if err != nil {
log.Fatal(err)
}
_, err = virtualFile.Write([]byte("HelloStorage"))
if err != nil {
log.Fatal(err)
}
err = virtualFile.Close()
if err != nil {
log.Fatal(err)
}