Skip to content

YashishDua/shipper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

shipper

Ship large files concurrently

Features

  1. Concurrent and high speed shipping
  2. Ships in the same machine (Non-TCP)
  3. Ships between two machines (TCP)

Setup

go get github.com/yashishdua/shipper

Note: Example code to run shipper is placed in the /example directory

1. Ship in the same machine (non-TCP)

Configuration

Config Description
Source Source file path
Destination Destination file path
BatchSize Total characters to be processed in 1 concurrent batch. Default is 10000
func main() {
	shipper := shipper.NewShipper(shipper.Config{
		Source:      "test.log",
		Destination: "test2.log",
		BatchSize:   5000, // Total characters to be processed in 1 concurrent batch
	})

	if err := shipper.ShipAndDock(); err != nil {
		fmt.Printf("Error in ShipAndDock: %s", err.Error())
	}
}

2. Ship between two machines (TCP)

Configuration

Config Description
TCP Host Destination host where to ship data
TCP Port Destination port where to ship data
Source Source file path
Destination Destination file path
BatchSize Total characters to be processed in 1 concurrent batch. Default is 10000

TCP Source

func main() {
	shipper := shipper.NewShipper(shipper.Config{
		Source:    "test.log",
		BatchSize: 5000, // Total characters to be processed in 1 concurrent batch
		TCP: shipper.TCP{
			Host: "127.0.0.1",
			Port: 8001,
		},
	})

	if err := shipper.Ship(); err != nil {
		fmt.Printf("Error in Shipping: %s", err.Error())
	}
}

TCP Destination

TCP Destination only requires file path and port

func main() {
	shipper := shipper.NewShipper(shipper.Config{
		Destination: "test2.log",
		TCP: shipper.TCP{
			Port: 8001,
		},
	})

	if err := shipper.Dock(); err != nil {
		fmt.Printf("Error in Docking: %s", err.Error())
	}
}