breaking-change(redis): change Redis Architecture

[MaineK00n]

NOTE
In this PR, we are changing the architecture of Redis.

// delete all old key
$ redis-cli keys "EXPLOIT#*" | xargs redis-cli del

What did you implement:

Changed the architecture of Redis.
The main changes are as follows
Changed EXPLOIT#C#$CVEID from Hash to Sets.
Also, EXPLOIT#DEP#$EXPLOITTYPE has been introduced to save the ExploitDBID and CVEID for each Insert in order to correctly update the Field in Hash and Member in Sets.
In addition, some unimplemented functions have been implemented.

Also, fixed a bug that prevented searching by ExploitUniqueID in Awesomepoc and Githubrepos because the ExploitUniqueID contained a URL.

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

$ make clean-integration

$ git checkout upstream/master
$ patch -p1 < exploit.go.patch
$ make build
$ mv go-exploitdb integration/exploitdb.old
$ git reset --hard
$ git checkout MaineK00n/change-redis-architecture
$ patch -p1 < exploit.go.patch
$ make build
$ mv go-exploitdb integration/exploitdb.new

$ patch -p1 < GNUMakefile.patch
$ make fetch-rdb && make fetch-redis
$ make diff-server-rdb && make diff-server-redis && make diff-server-rdb-redis

• exploit.go.patch

diff --git a/models/exploit.go b/models/exploit.go
index fbdd4bc..0d53d2e 100644
--- a/models/exploit.go
+++ b/models/exploit.go
@@ -23,7 +23,7 @@ var (
 type Exploit struct {
 	ID                int64              `json:"-"`
 	ExploitType       ExploitType        `gorm:"type:varchar(255)" json:"exploit_type"`
-	ExploitUniqueID   string             `gorm:"type:varchar(255);index:idx_exploit_exploit_unique_id" json:"exploit_unique_id"`
+	ExploitUniqueID   string             `gorm:"type:varchar(255);index:idx_exploit_exploit_unique_id" json:"-"`
 	URL               string             `gorm:"type:varchar(255)" json:"url"`
 	Description       string             `gorm:"type:text" json:"description"`
 	CveID             string             `gorm:"type:varchar(255);index:idx_exploit_cve_id" json:"cve_id"`

• GNUMakefile.patch

diff --git a/GNUmakefile b/GNUmakefile
index 5fe7b4b..7b68595 100644
--- a/GNUmakefile
+++ b/GNUmakefile
@@ -73,7 +73,7 @@ BRANCH := $(shell git symbolic-ref --short HEAD)
 build-integration:
 	@ git stash save "build-integration"
 	go buitld -ldflags "$(LDFLAGS)" -o integration/exploitdb.new
-	git checkout $(shell git describe --tags --abbrev=0)
+	git checkout upstream/master
 	@git reset --hard
 	go build -ldflags "$(LDFLAGS)" -o integration/exploitdb.old
 	git checkout $(BRANCH)
@@ -108,20 +108,20 @@ fetch-redis:
 	integration/exploitdb.new fetch githubrepos --dbtype redis --dbpath "redis://127.0.0.1:6380/0"
 
 diff-cveid:
-	@ python integration/diff_server_mode.py cveid --sample_rate 0.01 awesomepoc
-	@ python integration/diff_server_mode.py cveid --sample_rate 0.01 exploitdb
-	@ python integration/diff_server_mode.py cveid --sample_rate 0.01 githubrepos
+	@ python integration/diff_server_mode.py cveid --sample_rate 1.00 awesomepoc
+	@ python integration/diff_server_mode.py cveid --sample_rate 1.00 exploitdb
+	@ python integration/diff_server_mode.py cveid --sample_rate 1.00 githubrepos
 
 diff-uniqueid:
-	@ python integration/diff_server_mode.py uniqueid --sample_rate 0.01 awesomepoc
-	@ python integration/diff_server_mode.py uniqueid --sample_rate 0.01 exploitdb
-	@ python integration/diff_server_mode.py uniqueid --sample_rate 0.01 githubrepos
+	@ python integration/diff_server_mode.py uniqueid --sample_rate 1.00 awesomepoc
+	@ python integration/diff_server_mode.py uniqueid --sample_rate 1.00 exploitdb
+	@ python integration/diff_server_mode.py uniqueid --sample_rate 1.00 githubrepos
 
 diff-server-rdb:
 	integration/exploitdb.old server --dbpath=$(PWD)/integration/go-exploitdb.old.sqlite3 --port 1325 > /dev/null 2>&1 & 
 	integration/exploitdb.new server --dbpath=$(PWD)/integration/go-exploitdb.new.sqlite3 --port 1326 > /dev/null 2>&1 &
 	make diff-cveid
-	make diff-uniqueid
+	# make diff-uniqueid
 	pkill exploitdb.old
 	pkill exploitdb.new
 
@@ -129,7 +129,7 @@ diff-server-redis:
 	integration/exploitdb.old server --dbtype redis --dbpath "redis://127.0.0.1:6379/0" --port 1325 > /dev/null 2>&1 & 
 	integration/exploitdb.new server --dbtype redis --dbpath "redis://127.0.0.1:6380/0" --port 1326 > /dev/null 2>&1 &
 	make diff-cveid
-	make diff-uniqueid
+	# make diff-uniqueid
 	pkill exploitdb.old
 	pkill exploitdb.new

Checklist:

You don't have to satisfy all of the following.

• Write tests
• Write documentation
• Check that there aren't other open pull requests for the same issue/feature
• Format your source code by make fmt
• Pass the test by make test
• Provide verification config / commands
• Enable "Allow edits from maintainers" for this PR
• Update the messages below

Is this ready for review?: YES

Reference