Add client retry or throttling/backoff

[brint]

Hey there!

Thank you for creating this tool! I've noticed when I use a hosted CI service like CircleCI that uploads a Jekyll site that sometimes it fails, and the client gets disconnected. When it does this, it looks like the push is screaming through (like >30 uploads in a second) and the client might be getting cut off and rate limited.

Example deploy command:

bundle exec neocities push ../_site

Error coming out, and retries seem to fail the same way after about 30 uploads that went through really fast:

Uploading tech/year/month/day/blah.html ... bundler: failed to load command: neocities (/home/circleci/project/neocities/vendor/bundle/ruby/3.1.0/bin/neocities)
/home/circleci/project/neocities/vendor/bundle/ruby/3.1.0/gems/httpclient-fixcerts-2.8.5/lib/httpclient/session.rb:813:in `rescue in block in parse_header': HTTPClient::KeepAliveDisconnected: Connection reset by peer (HTTPClient::KeepAliveDisconnected)

I haven't been able to replicate this issue from my laptop, where uploads of individual files are uploaded slower. If it is server-side rate limiting, it'd be nice if the client could jive with it. This doesn't fail every time with CI services. If things are a bit slower uploading from a CI service, it will sometimes work.

Steps to replicate

• Have a repo with ~100 small files that are part of a static website (ie, Jekyll, Hugo).
• Add a Gemfile that looks like this in the root directory of the repo:

source 'https://rubygems.org'

gem 'neocities'

• Setup the repo to work with CircleCI
• Add an environment variable to CircleCI project config called NEOCITIES_API_KEY with your API key
• Add a CircleCI config to the repo .circleci/config.yml that looks something like this (assumes static assets are in a directory called _site like with Jekyll):

---
version: 2.1
orbs:
  ruby: circleci/[email protected]
ruby_image: &ruby_image cimg/ruby:3.1

workflows:
  build_and_deploy:
    jobs:
      - deploy_neocities

jobs:
  deploy_neocities:
    resource_class: large
    docker:
      - image: *ruby_image
    steps:
      - checkout
      - ruby/install-deps
      - run:
          name: Deploy Jekyll from _site directory to neocities
          command: |
            bundle exec neocities push ./_site       

With failed tasks, you can retry with SSH to log in with CircleCI to manually run the deploy command and experiment.

[janseeger]

Any news? I have encountered the same issue and am currently looking into replacing the neocities-cli with WebDAV or switching away from neocities entirely because I'm fed up having to fix this every few months.

[stefan-burke]

I was having the same issue, so I forked the repo and added a sleep(2) before every attempted upload and up to three retries, and now my builds (via Sourcehut) upload successfully (albeit slowly)..!

Here's that changed bit: https://github.com/stefan-burke/neocities-ruby/blob/master/lib/neocities/client.rb#L73

[janseeger]

Saw this too late. I have since switched to nsc within a small alpine/curl container and so far it seems to work nicely. And as a bonus on top without artificial delays and with a way smaller footprint (sh+curl).

Edit: here's corresponding the GitLab pipeline script, if anybody is interested:

neocities:
  image: xxx
  stage: deploy
  environment: neocities
  variables:
    NSC_KEY: "$NEOCITIES_TOKEN"
    NSC_USER: "$NEOCITIES_USER"
    NSC_PASSCMD: "echo asdf"
  script:
    - cd _site
    - find -exec sh -c 'filetoupload="{}";nsc u "${filetoupload#./*}"' \;