Skip to content
/ fsEngine Public archive

A write-optimized object storage, designed for high load on commodity hardware.

Notifications You must be signed in to change notification settings

fanap-infra/fsEngine

Folders and files

NameName
Last commit message
Last commit date
Aug 11, 2021
Jun 29, 2021
Aug 23, 2021
Aug 23, 2021
Oct 11, 2021
May 15, 2021
Jul 6, 2021
Oct 12, 2021
Aug 23, 2021
Aug 14, 2021
Jul 7, 2021
Jul 20, 2021
Oct 12, 2021
Oct 10, 2021
May 16, 2021
Oct 11, 2021
Aug 23, 2021
Aug 23, 2021
Jul 6, 2021
Oct 12, 2021
Aug 14, 2021
Aug 23, 2021
Aug 7, 2021
Aug 23, 2021
Sep 2, 2021
Aug 23, 2021
Aug 14, 2021
Aug 7, 2021

Repository files navigation

File Storage Engine

File Storage Engine (FSEngine) is a write-optimized object storage library designed mainly for storing large, continuous streams on commodity hardware and Disk-Drives. This project includes APIs and primitives for storing raw data and fetching data from the storage. Use-cases of this library include storing live video streams, logs, metadata, or any write-optimized workload.

This library might be used with ArchiverMedia wrapper library for easily transforming media objects into consumable binary packets for FSEngine.

This library uses a pseudo log storage model for storing multiple data streams into a single file. It also supports a circular storage model for a fixed-sized store.

Currently, FSEngine has been offered as a Golang only library, and all APIs are usable within any Golang 1.12+ project.

Features

  • Write-optimized storage
  • Local storage
  • Circular storage
  • Multi-instance storage
  • Spare storage with ECC

Installing

  1. To install FSEngine library into and existing project, having Go installed, use the bellow Go command to install FSEngine.
$ go get github.com/fanap-infra/fsEngine
  1. Import it to your library
import "github.com/fanap-infra/fsEngine"

How to use

First you need to establish storage location on local system and initialize storage engine by maximum size of storage used by the engine:

package main
import (
	"github.com/fanap-infra/fsEngine"
	"github.com/fanap-infra/log"
)
type EventsTrigger struct {
	fileID uint32
}
func main() {
	// define storage location and size
	const storagePath = "/var/fsEngine/volume1"
	const storageSize = 1 << 32 // 4GB volume
	fileSystem, err := fsEngine.CreateFileSystem(storagePath, storageSize,
		fsEngine.BLOCKSIZE, &EventsTrigger{}, log.GetScope("Example"))
	if err != nil {
		log.Fatal(err)
	}
	
	return
}
// VirtualFileDeleted 
// Implement event listener.
func (e EventsTrigger) VirtualFileDeleted(fileID uint32, message string) {
	log.Warnf("File with ID:%v, deleted: %s", fileID, message)
	return
}

This is the barebone definition for using FSEngine, however, to store objects you have to create a virtual identity for the object which here is called a virtualFile.

	fileSystem, err := fsEngine.CreateFileSystem(storagePath, storageSize,
		fsEngine.BLOCKSIZE, &EventsTrigger{}, log.GetScope("Example"))
	if err != nil {
		log.Fatal(err)
	}
    const vfID = 1
    const vfName = "Hello"
    virtualFile, err := fileSystem.NewVirtualFile(vfID, vfName)
    if err != nil {
        log.Fatal(err)
    }
    _, err = virtualFile.Write([]byte("HelloStorage"))
    if err != nil {
        log.Fatal(err)
    }
    err = virtualFile.Close()
	if err != nil {
        log.Fatal(err)
    }