This repository was archived by the owner on Aug 2, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 110
LocalStore #1032
Closed
Closed
LocalStore #1032
Changes from 1 commit
Commits
Show all changes
81 commits
Select commit
Hold shift + click to select a range
d8acb12
swarm/storage/localstore: most basic database
janos 9ec535a
swarm/storage/localstore: fix typos and comments
janos 37205de
swarm/shed: add uint64 field Dec and DecInBatch methods
janos b1ded5a
swarm/storage/localstore: decrement size counter on ModeRemoval update
janos 572f3cb
swarm/storage/localstore: unexport modeAccess and modeRemoval
janos cbb510b
swarm/storage/localstore: add WithRetrievalCompositeIndex
janos c7beb22
swarm/storage/localstore: add TestModeSyncing
janos 391faa7
swarm/storage/localstore: fix test name
janos 58f3f86
swarm/storage/localstore: add TestModeUpload
janos 2d928bf
swarm/storage/localstore: add TestModeRequest
janos 4d58a6f
swarm/storage/localstore: add TestModeSynced
janos af1b137
swarm/storage/localstore: add TestModeAccess
janos 96409ff
swarm/storage/localstore: add TestModeRemoval
janos b782bfe
swarm/storage/localstore: add mock store option for chunk data
janos 35376d8
swarm/storage/localstore: add TestDB_pullIndex
janos e6a7196
swarm/storage/localstore: add TestDB_gcIndex
janos 58c7f11
swarm/storage/localstore: change how batches are written
janos 6e8b2ad
swarm/storage/localstore: add updateOnAccess function
janos 7b8510e
swarm/storage/localhost: add DB.gcSize
janos cf3ec30
swarm/storage/localstore: update comments
janos d58e1ee
swarm/storage/localstore: add BenchmarkNew
janos f2299f4
swarm/storage/localstore: add retrieval tests benchmarks
janos e6bdda7
swarm/storage/localstore: accessors redesign
janos b6f5b7a
swarm/storage/localstore: add semaphore for updateGC goroutine
janos 5732c38
swarm/storage/localstore: implement basic garbage collection
janos ac9d153
swarm/storage/localstore: optimize collectGarbage
janos e6e29f5
swarm/storage/localstore: add more garbage collection tests cases
janos 750268d
swarm/shed, swarm/storage/localstore: rename IndexItem to Item
janos a486a09
swarm/shed: add Index.CountFrom
janos e17fec2
swarm/storage/localstore: persist gcSize
janos 5605323
swarm/storage/localstore: remove composite retrieval index
janos 5ebbcb5
swarm/shed: IterateWithPrefix and IterateWithPrefixFrom Index functions
janos 80c269b
swarm/storage/localstore: writeGCSize function with leveldb batch
janos bca85ef
swarm/storage/localstore: unexport modeSetRemove
janos cee4004
swarm/storage/localstore: update writeGCSizeWorker comment
janos b5a0fd2
swarm/storage/localstore: add triggerGarbageCollection function
janos 4fca004
swarm/storage/localstore: call writeGCSize on DB Close
janos 5ad0c8d
swarm/storage/localstore: additional comment in writeGCSizeWorker
janos cb8078e
Merge branch 'master' into localstore
janos c9f5130
swarm/storage/localstore: add MetricsPrefix option
janos da9bab4
swarm/storage/localstore: fix a typo
janos d54d7ae
swamr/shed: only one Index Iterate function
janos 67473be
swarm/storage/localstore: use shed Iterate function
janos 2299147
swarm/shed: pass a new byte slice copy to index decode functions
janos 5488a2b
swarm/storage/localstore: implement feed subscriptions
janos 38bdf7f
swarm/storage/localstore: add more subscriptions tests
janos a74eae2
swarm/storage/localsore: add parallel upload test
janos 534009f
swarm/storage/localstore: use storage.MaxPO in subscription tests
janos 5edd22d
swarm/storage/localstore: subscription of addresses instead chunks
janos 95726d7
swarm/storage/localstore: lock item address in collectGarbage iterator
janos c5dcae3
swarm/storage/localstore: fix TestSubscribePull to include MaxPO
janos f3380ea
swarm/storage/localstore: improve subscriptions
janos 1d2d470
swarm/storage/localstore: add TestDB_SubscribePull_sinceAndUntil test
janos c26c979
swarm/storage/localstore: adjust pull sync tests
janos c188281
Merge branch 'master' into localstore
janos c5e4c61
swarm/storage/localstore: remove writeGCSizeDelay and use literal
janos 015d977
swarm/storage/localstore: adjust subscriptions tests delays and comments
janos abbb4a6
swarm/storage/localstore: add godoc package overview
janos fb0a822
swarm/storage/localstore: fix a typo
janos c423060
swarm/storage/localstore: update package overview
janos c5a6456
swarm/storage/localstore: remove repeated index change
janos 33726a4
swarm/storage/localstore: rename ChunkInfo to ChunkDescriptor
janos 25a068a
swarm/storage/localstore: add comment in collectGarbageWorker
janos ca1e24f
swarm/storage/localstore: replace atomics with mutexes for gcSize and…
janos c62fee3
Merge branch 'master' into localstore
janos 5ddc75f
swarm/storage/localstore: protect addrs map in pull subs tests
janos 87bbd61
swarm/storage/localstore: protect slices in push subs test
janos 6ad67d7
swarm/storage/localstore: protect chunks in TestModePutUpload_parallel
janos a550388
swarm/storage/localstore: fix a race in TestDB_updateGCSem defers
janos 1dae999
swarm/storage/localstore: remove parallel flag from tests
janos eda338a
swarm/storage/localstore: fix a race in testDB_collectGarbageWorker
janos ad5b329
swarm/storage/localstore: remove unused code
janos 8d15e82
swarm/storage/localstore: add more context to pull sub log messages
janos 3948044
swarm/storage/localstore: merge branch 'master' into localstore
janos 6c8208a
swarm/storage/localstore: BenchmarkPutUpload and global lock option
janos 6accc6b
swarm/storage/localstore: pre-generate chunks in BenchmarkPutUpload
janos 85cd349
swarm/storage/localstore: correct useGlobalLock in collectGarbage
janos 7fa1ba9
swarm/storage/localstore: fix typos and update comments
janos ebecd05
swarm/storage/localstore: update writeGCSize comment
janos f056e86
swarm/storage/localstore: remove global lock option
janos 40432d9
swarm/storage/localstore: add description for gc size counting
janos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| // Copyright 2018 The go-ethereum Authors | ||
| // This file is part of the go-ethereum library. | ||
| // | ||
| // The go-ethereum library is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU Lesser General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
| // | ||
| // The go-ethereum library is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU Lesser General Public License for more details. | ||
| // | ||
| // You should have received a copy of the GNU Lesser General Public License | ||
| // along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| package localstore | ||
|
|
||
| import ( | ||
| "context" | ||
|
|
||
| "github.com/syndtr/goleveldb/leveldb" | ||
|
|
||
| "github.com/ethereum/go-ethereum/swarm/shed" | ||
| "github.com/ethereum/go-ethereum/swarm/storage" | ||
| ) | ||
|
|
||
| // Accessor implements ChunkStore to manage data | ||
| // in DB with different modes of access and update. | ||
| type Accessor struct { | ||
| db *DB | ||
| mode Mode | ||
| } | ||
|
|
||
| // Accessor returns a new Accessor with a specified Mode. | ||
| func (db *DB) Accessor(mode Mode) *Accessor { | ||
| return &Accessor{ | ||
| mode: mode, | ||
| db: db, | ||
| } | ||
| } | ||
|
|
||
| // Put overwrites the underlying DB Put method for the specific mode of update. | ||
| func (u *Accessor) Put(ctx context.Context, ch storage.Chunk) error { | ||
| return u.db.update(ctx, u.mode, chunkToItem(ch)) | ||
| } | ||
|
|
||
| // Get overwrites the underlying DB Get method for the specific mode of access. | ||
|
zelig marked this conversation as resolved.
Outdated
|
||
| func (u *Accessor) Get(_ context.Context, addr storage.Address) (chunk storage.Chunk, err error) { | ||
| item := addressToItem(addr) | ||
| out, err := u.db.access(u.mode, item) | ||
| if err != nil { | ||
| if err == leveldb.ErrNotFound { | ||
| return nil, storage.ErrChunkNotFound | ||
| } | ||
| return nil, err | ||
| } | ||
| return storage.NewChunk(out.Address, out.Data), nil | ||
| } | ||
|
|
||
| // chunkToItem creates new IndexItem with data provided by the Chunk. | ||
| func chunkToItem(ch storage.Chunk) shed.IndexItem { | ||
| return shed.IndexItem{ | ||
| Address: ch.Address(), | ||
| Data: ch.Data(), | ||
| } | ||
| } | ||
|
|
||
| // addressToItem creates new IndexItem with a provided address. | ||
| func addressToItem(addr storage.Address) shed.IndexItem { | ||
| return shed.IndexItem{ | ||
| Address: addr, | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| // Copyright 2018 The go-ethereum Authors | ||
| // This file is part of the go-ethereum library. | ||
| // | ||
| // The go-ethereum library is free software: you can redistribute it and/or modify | ||
| // it under the terms of the GNU Lesser General Public License as published by | ||
| // the Free Software Foundation, either version 3 of the License, or | ||
| // (at your option) any later version. | ||
| // | ||
| // The go-ethereum library is distributed in the hope that it will be useful, | ||
| // but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| // GNU Lesser General Public License for more details. | ||
| // | ||
| // You should have received a copy of the GNU Lesser General Public License | ||
| // along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>. | ||
|
|
||
| package localstore | ||
|
|
||
| import ( | ||
| "bytes" | ||
| "context" | ||
| "testing" | ||
|
|
||
| "github.com/ethereum/go-ethereum/swarm/storage" | ||
| ) | ||
|
|
||
| // TestAccessors tests most basic Put and Get functionalities | ||
| // for different accessors. This test validates that the chunk | ||
| // is retrievable from the database, not if all indexes are set | ||
| // correctly. | ||
|
zelig marked this conversation as resolved.
Outdated
|
||
| func TestAccessors(t *testing.T) { | ||
| db, cleanupFunc := newTestDB(t) | ||
| defer cleanupFunc() | ||
|
|
||
| for _, m := range []Mode{ | ||
| ModeSyncing, | ||
| ModeUpload, | ||
| ModeRequest, | ||
| ModeSynced, | ||
| ModeAccess, | ||
| } { | ||
| t.Run(ModeName(m), func(t *testing.T) { | ||
| a := db.Accessor(m) | ||
|
|
||
| want := generateRandomChunk() | ||
|
|
||
| err := a.Put(context.Background(), want) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
|
|
||
| got, err := a.Get(context.Background(), want.Address()) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| if !bytes.Equal(got.Data(), want.Data()) { | ||
| t.Errorf("got chunk data %x, want %x", got.Data(), want.Data()) | ||
| } | ||
| }) | ||
| } | ||
|
|
||
| // Removal mode is a special case as it removes the chunk | ||
| // from the database. | ||
| t.Run(ModeName(ModeRemoval), func(t *testing.T) { | ||
| a := db.Accessor(ModeUpload) | ||
|
|
||
| want := generateRandomChunk() | ||
|
|
||
| // first put a random chunk to the database | ||
| err := a.Put(context.Background(), want) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
|
|
||
| got, err := a.Get(context.Background(), want.Address()) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| if !bytes.Equal(got.Data(), want.Data()) { | ||
| t.Errorf("got chunk data %x, want %x", got.Data(), want.Data()) | ||
| } | ||
|
|
||
| a = db.Accessor(ModeRemoval) | ||
|
|
||
| // removal accessor actually removes the chunk on Put | ||
| err = a.Put(context.Background(), want) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
|
|
||
| // chunk should not be found | ||
| wantErr := storage.ErrChunkNotFound | ||
| _, err = a.Get(context.Background(), want.Address()) | ||
| if err != wantErr { | ||
| t.Errorf("got error %v, expected %v", err, wantErr) | ||
| } | ||
| }) | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.