Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add remote pinning to ipfs command #7661

Merged
merged 128 commits into from
Dec 9, 2020
Merged
Show file tree
Hide file tree
Changes from 81 commits
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
51235ee
first draft: ipfs pin remote add
petar Sep 8, 2020
7931918
first draft ls command
petar Sep 9, 2020
cd7bdc1
first draft rm cmd
petar Sep 9, 2020
4f717f9
connect to pin delegates after add
petar Sep 10, 2020
0eacb46
fix: handle errors in remote pins
aschmahmann Sep 11, 2020
ad92599
bump go-pinning-service
aschmahmann Sep 11, 2020
23a49e8
fix
petar Sep 11, 2020
ba49da3
fix
petar Sep 11, 2020
952efa2
fix
petar Sep 11, 2020
a89c16a
add pinning api service to sharness circle ci
petar Sep 18, 2020
e29c51c
ok
petar Sep 18, 2020
82abe0b
fix
petar Sep 18, 2020
bfd5b6b
ok
petar Sep 18, 2020
885c962
ok
petar Sep 18, 2020
a086d7c
Update core/commands/remotepin.go
petar Sep 18, 2020
af1994b
Update core/commands/remotepin.go
petar Sep 18, 2020
27e5dad
upgrades
petar Sep 21, 2020
8e07f26
improvements
petar Sep 22, 2020
c58e296
ok
petar Sep 22, 2020
c8f0743
sharness ready for a CI spin
petar Sep 22, 2020
4bb8dd6
ok
petar Sep 22, 2020
74a00b1
ok
petar Sep 22, 2020
a17a170
ok
petar Sep 23, 2020
3383e81
ok
petar Sep 23, 2020
e7d3d47
ok
petar Sep 23, 2020
cf4ae65
ok
petar Sep 24, 2020
125bd85
add ipfs pin remote service add
petar Sep 25, 2020
0c639f8
ok
petar Sep 25, 2020
d974b63
add ipfs pin remote servce add|update|rm|rename
petar Sep 28, 2020
4dd2806
flag fixes
petar Sep 29, 2020
d51f78c
execute ci using machine
petar Oct 14, 2020
8271449
rm setup_remote_docker
petar Oct 14, 2020
87babfe
rm ruby install
petar Oct 14, 2020
9a33faa
point remote pin test to service in ci
petar Oct 14, 2020
ce4e47c
ok
petar Oct 14, 2020
a723323
ok
petar Oct 14, 2020
2014d40
ok
petar Oct 14, 2020
f940ce2
ok
petar Oct 15, 2020
512d28a
ok
petar Oct 15, 2020
e5df8d4
fixes
petar Oct 15, 2020
6af2be2
ok
petar Oct 15, 2020
db63b3d
ok
petar Oct 15, 2020
482d0c6
fix go test
petar Oct 15, 2020
ae295f2
ok
petar Oct 15, 2020
9551188
fixes
petar Oct 15, 2020
b0b7936
ok
petar Oct 15, 2020
fd5977c
ok
petar Oct 15, 2020
269559b
update http client
petar Oct 16, 2020
5665beb
try sleeping after ipfs pin remote add
petar Oct 26, 2020
aa9be93
ci: try later ubuntu sharness image
aschmahmann Oct 27, 2020
1decef0
reduce make parallelism to 3
petar Oct 27, 2020
ef05661
change cut for CI Host IP, need something better
aschmahmann Oct 27, 2020
0484dac
-j 10 again
petar Oct 27, 2020
fba1fa3
hardcode ci host ip
petar Oct 27, 2020
2188111
add ipfs pin remote ls --status flag
petar Oct 27, 2020
d96e6d1
test pin remote ls --status
petar Oct 27, 2020
ebc3e7d
add --name --status --cid to pin remote rm
petar Oct 28, 2020
fb57add
add tests for all ls and remove flags
petar Oct 29, 2020
1c3d033
update client dep
petar Oct 30, 2020
2d9607a
fix context in lsRemote
petar Oct 30, 2020
e8d2d61
use remote pin service config in test; demote circle ci machine
petar Nov 2, 2020
5102a9f
remove nonce secret key for ruby pinning
petar Nov 2, 2020
8d3f998
fix comment in config.yml; remove IPFS_* env variables for pinning
petar Nov 2, 2020
fcaed78
ok
petar Nov 2, 2020
4ea4d91
add ipfs pin remote service ls
petar Nov 3, 2020
12398e8
-j 10
petar Nov 3, 2020
736afdf
add --background to pin add
petar Nov 3, 2020
eb57fbd
scrap update and rename
petar Nov 3, 2020
a463bc9
-j 5
petar Nov 3, 2020
8dd8220
-j 3
petar Nov 3, 2020
4322a2c
prevent 'ipfs config' from showing remote pinning service info
petar Nov 4, 2020
d56ec14
lsRemote
petar Nov 4, 2020
39a3539
fix
petar Nov 4, 2020
4283745
test pinning service registrygu
petar Nov 4, 2020
28d0614
test remote pinning service config operations
petar Nov 4, 2020
0047826
test --background
petar Nov 5, 2020
ff89659
add --force
petar Nov 5, 2020
7426095
add missing cid
petar Nov 5, 2020
b2d073c
service add --name --url --key
petar Nov 6, 2020
a3e500c
pin package
petar Nov 6, 2020
6511f42
fix
petar Nov 6, 2020
82142cc
fix(ci): use the latest docker images
lidel Nov 12, 2020
a16e847
fix(ci): deterministic rb-pinning-service-api
lidel Nov 12, 2020
23b559e
fix: pin remote improvements
lidel Nov 13, 2020
576b085
fix(pins): pass own multiaddrs in pin.origins
lidel Nov 13, 2020
a152a43
fix: output of pin remote service ls
lidel Nov 13, 2020
b7e0c08
feat: ipfs pin remote ls [<request-id>]
lidel Nov 14, 2020
bea8056
feat: ipfs pin remote ls [<request-id>]...
lidel Nov 14, 2020
531923b
chore: golint
lidel Nov 14, 2020
75bd021
use updated config structure
petar Nov 16, 2020
87d93a3
style: improved error messages
lidel Nov 17, 2020
235a099
chore: latest rb-pinning-service-api
lidel Nov 17, 2020
f75a209
fix: support --status=a,b and --cid=a,b
lidel Nov 17, 2020
cd0a273
refactor(ci): dynamic DOCKER_HOST
lidel Nov 17, 2020
322ec35
fix: 'config show' crash when no pin services
lidel Nov 17, 2020
c66f1dd
feat: switch to --background=false by default
lidel Nov 17, 2020
807b790
refactor: background=false by default and status
lidel Nov 17, 2020
eb992b4
chore: remove delegate list from plain-text output
lidel Nov 18, 2020
0bf0314
refactor: switch remote pin tests to inlined CIDs
lidel Nov 18, 2020
fb31dd3
feat: pin remote service ls --pin-count
lidel Nov 20, 2020
3293d1a
fix: partial config sanitization and tests
lidel Nov 24, 2020
a080d84
bump version of go-ipfs-cmds
aschmahmann Nov 25, 2020
c5a8451
check option types in pin remote commands more defensively
aschmahmann Nov 25, 2020
ca3b866
fix: remote remote services from config outputs
lidel Nov 26, 2020
9ff07fa
refactor: parallel remote service ls --pin-count
lidel Nov 26, 2020
8f89163
fix: escape HTML in error response body
lidel Nov 27, 2020
4f44ac3
refactor: remove mapstructure
lidel Nov 30, 2020
4eb3757
chore: go-pinning-service-http-client v0.1.0
lidel Nov 30, 2020
ef975b0
chore: golint
lidel Nov 30, 2020
5e9e89c
fix(ci): remove reference to missing commit
lidel Nov 30, 2020
965b709
use newest config structure
petar Dec 2, 2020
7d63db3
commands: pin remote add,ls, and rm now require a service flag to be …
aschmahmann Dec 3, 2020
8564996
add pin policy flag
petar Dec 3, 2020
5dfbe2f
update config dep
petar Dec 3, 2020
1706647
fix: avoid redundant /pins/pins
lidel Dec 4, 2020
e4e2b04
simplify inBlockScope(); now Pinning.MFSRepinInterval is settable wit…
petar Dec 4, 2020
a1ea3d6
refactor: remove requestid and delegates
lidel Dec 4, 2020
fa9f36f
refactor: sanitize entire Pinning block for now
lidel Dec 4, 2020
244b186
refactor: rename --pin-count to --stat
lidel Dec 4, 2020
808e74e
fix(ci): deterministic rb-pinning-service-api
lidel Dec 5, 2020
32f8950
refactor: no Stat without --stat
lidel Dec 5, 2020
eda4ea4
remove references to pinmfs
petar Dec 6, 2020
876abda
fix compile; fix config dependency
petar Dec 6, 2020
da85c74
remove mfs and policy from test
petar Dec 6, 2020
121ae86
fix(test): Api.ApiKey → Api.Key
lidel Dec 7, 2020
0cc482d
update deps
aschmahmann Dec 8, 2020
06709ac
Merge branch 'master' into petar/pincli
aschmahmann Dec 8, 2020
6beca40
add pinning service config docs
aschmahmann Dec 9, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,14 +118,37 @@ jobs:
- store_artifacts:
path: /tmp/circleci-test-results
sharness:
executor: golang
machine:
image: ubuntu-2004:202010-01
working_directory: ~/ipfs/go-ipfs
environment:
<<: *default_environment
GO111MODULE: "on"
TEST_NO_DOCKER: 1
TEST_NO_FUSE: 1
GOPATH: /home/circleci/go
TEST_VERBOSE: 1
steps:
- run: sudo apt install socat
- checkout

- run:
git clone https://github.com/ipfs-shipyard/rb-pinning-service-api.git
- run:
cd rb-pinning-service-api &&
docker-compose up -d &&
cd ..

- *make_out_dirs
- *restore_gomod

- run: make -O -j 10 coverage/sharness_tests.coverprofile test/sharness/test-results/sharness.xml TEST_GENERATE_JUNIT=1 CONTINUE_ON_S_FAILURE=1
- run:
name: Setup Environment Variables
# 172.17.0.1 is the docker host IP; all ports exported by child containers can be accessed there.
command: echo 'export CI_HOST_IP=172.17.0.1' >> $BASH_ENV
petar marked this conversation as resolved.
Show resolved Hide resolved
- run:
echo CI_HOST_IP=$CI_HOST_IP &&
make -O -j 3 coverage/sharness_tests.coverprofile test/sharness/test-results/sharness.xml TEST_GENERATE_JUNIT=1 CONTINUE_ON_S_FAILURE=1 CI_HOST_IP=$CI_HOST_IP
petar marked this conversation as resolved.
Show resolved Hide resolved

- run:
when: always
Expand Down
10 changes: 9 additions & 1 deletion core/commands/commands_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -177,11 +177,19 @@ func TestCommands(t *testing.T) {
"/p2p/stream/ls",
"/pin",
"/pin/add",
"/ping",
"/pin/ls",
"/pin/remote",
"/pin/remote/add",
"/pin/remote/ls",
"/pin/remote/rm",
"/pin/remote/service",
"/pin/remote/service/add",
"/pin/remote/service/ls",
"/pin/remote/service/rm",
"/pin/rm",
"/pin/update",
"/pin/verify",
"/ping",
"/pubsub",
"/pubsub/ls",
"/pubsub/peers",
Expand Down
12 changes: 9 additions & 3 deletions core/commands/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ import (
"github.com/ipfs/go-ipfs/repo/fsrepo"

"github.com/elgris/jsondiff"
"github.com/ipfs/go-ipfs-cmds"
"github.com/ipfs/go-ipfs-config"
cmds "github.com/ipfs/go-ipfs-cmds"
config "github.com/ipfs/go-ipfs-config"
)

// ConfigUpdateOutput is config profile apply command's output
Expand Down Expand Up @@ -83,6 +83,8 @@ Set the value of the 'Datastore.Path' key:
switch strings.ToLower(key) {
case "identity", "identity.privkey":
return errors.New("cannot show or change private key through API")
case "remotepinservices":
return errors.New("cannot show remote pinning services, use 'ipfs pin remote service ls'")
lidel marked this conversation as resolved.
Show resolved Hide resolved
default:
}

Expand Down Expand Up @@ -144,7 +146,7 @@ var configShowCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Output config file contents.",
ShortDescription: `
NOTE: For security reasons, this command will omit your private key. If you would like to make a full backup of your config (private key included), you must copy the config file from your repo.
NOTE: For security reasons, this command will omit your private key and remote services. If you would like to make a full backup of your config (private key included), you must copy the config file from your repo.
`,
},
Type: map[string]interface{}{},
Expand Down Expand Up @@ -174,6 +176,10 @@ NOTE: For security reasons, this command will omit your private key. If you woul
if err != nil {
return err
}
err = scrubValue(cfg, []string{config.RemotePinServicesTag})
if err != nil {
return err
}

return cmds.EmitOnce(res, &cfg)
},
Expand Down
3 changes: 2 additions & 1 deletion core/commands/pin.go → core/commands/pin/pin.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package commands
package pin

import (
"context"
Expand Down Expand Up @@ -35,6 +35,7 @@ var PinCmd = &cmds.Command{
"ls": listPinCmd,
"verify": verifyPinCmd,
"update": updatePinCmd,
"remote": remotePinCmd,
},
}

Expand Down
Loading