feat(explorer): make possible to run sync in a separate process #3224

mudler · 2024-08-12T17:12:28Z

Description

This PR allows to run the discovery as a separate process, and moves the informations cumulated of the network to the database.

The problem I have is that by going over the networks over and over, closing the nodes leaks data, and as such the node might exhaust RAM. Since discovery is something that have to happen periodically, it's good to move it to its own process that is short-lived.

Notes for Reviewers

Signed commits

Yes, I signed my commits.

Signed-off-by: Ettore Di Giacinto <[email protected]>

core/explorer/database.go

 	}
 	return db, db.load()
 }

 // Get retrieves a Token from the Database by its token.
 func (db *Database) Get(token string) (TokenData, bool) {
-	db.RLock()
-	defer db.RUnlock()
+	db.flock.Lock() // we are making sure that the file is not being written to


core/explorer/database.go

+	defer db.flock.Unlock()
+	db.Lock() // we are making sure that is safe if called by another instance in the same process
+	defer db.Unlock()
+	db.load()


core/explorer/database.go

 	t, ok := db.data[token]
 	return t, ok
 }

 // Set stores a Token in the Database by its token.
 func (db *Database) Set(token string, t TokenData) error {
+	db.flock.Lock()


core/explorer/database.go

 	db.Lock()
+	defer db.Unlock()
+	db.load()


core/explorer/database.go

 }

 // Delete removes a Token from the Database by its token.
 func (db *Database) Delete(token string) error {
+	db.flock.Lock()


core/explorer/database.go

 }

 func (db *Database) TokenList() []string {
-	db.RLock()
-	defer db.RUnlock()
+	db.flock.Lock()


core/explorer/database.go

+	defer db.flock.Unlock()
+	db.Lock()
+	defer db.Unlock()
+	db.load()


core/explorer/discovery.go

+			data, _ := s.database.Get(token)
+			(&data).Clusters = ledgerK
+			(&data).Failures = 0
+			s.database.Set(token, data)


core/explorer/discovery.go

-	s.failures[token]++
+	data, _ := s.database.Get(token)
+	(&data).Failures++
+	s.database.Set(token, data)


core/explorer/discovery.go

+		data, _ := s.database.Get(t)
+		if data.Failures > s.errorThreshold {
+			log.Info().Any("token", t).Msg("Token has been removed from the database")
+			s.database.Delete(t)


netlify · 2024-08-12T17:12:57Z

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	`085667d`
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/66ba427d79b4830008749bb3
😎 Deploy Preview	https://deploy-preview-3224--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

feat(explorer): make possible to run sync in a separate process

085667d

Signed-off-by: Ettore Di Giacinto <[email protected]>

github-advanced-security bot found potential problems Aug 12, 2024

View reviewed changes

mudler merged commit 9729d2a into master Aug 12, 2024
31 checks passed

mudler deleted the explorer_flock branch August 12, 2024 17:25

mudler added the enhancement New feature or request label Aug 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(explorer): make possible to run sync in a separate process #3224

feat(explorer): make possible to run sync in a separate process #3224

mudler commented Aug 12, 2024

netlify bot commented Aug 12, 2024 •

edited

Loading

feat(explorer): make possible to run sync in a separate process #3224

feat(explorer): make possible to run sync in a separate process #3224

Conversation

mudler commented Aug 12, 2024

netlify bot commented Aug 12, 2024 • edited Loading

✅ Deploy Preview for localai ready!

netlify bot commented Aug 12, 2024 •

edited

Loading