From 1008a083678be2e137a1150e94e9802af2a9b656 Mon Sep 17 00:00:00 2001 From: NAKAI Ryosuke Date: Sat, 17 Dec 2022 12:50:18 +0900 Subject: [PATCH 01/79] =?UTF-8?q?=E3=82=A4=E3=83=B3=E3=83=9D=E3=83=BC?= =?UTF-8?q?=E3=83=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitignore | 3 + LICENSE | 176 ++++++++ Makefile | 20 + README.md | 79 ++++ cmd/sora-archive-uploader/main.go | 38 ++ config.example.toml | 64 +++ config.go | 56 +++ finder.go | 78 ++++ gatekeeper.go | 233 ++++++++++ go.mod | 32 ++ go.sum | 65 +++ logging.go | 68 +++ renovate.json | 19 + runner.go | 192 ++++++++ s3.go | 133 ++++++ sora-archive-uploader.service | 21 + sora-archive-uploader.timer | 9 + uploader.go | 708 ++++++++++++++++++++++++++++++ webhook.go | 144 ++++++ 19 files changed, 2138 insertions(+) create mode 100644 .gitignore create mode 100644 LICENSE create mode 100644 Makefile create mode 100644 README.md create mode 100644 cmd/sora-archive-uploader/main.go create mode 100644 config.example.toml create mode 100644 config.go create mode 100644 finder.go create mode 100644 gatekeeper.go create mode 100644 go.mod create mode 100644 go.sum create mode 100644 logging.go create mode 100644 renovate.json create mode 100644 runner.go create mode 100644 s3.go create mode 100644 sora-archive-uploader.service create mode 100644 sora-archive-uploader.timer create mode 100644 uploader.go create mode 100644 webhook.go diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..53a8a5e --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +bin/* +config.toml +*.jsonl \ No newline at end of file diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..2bb9ad2 --- /dev/null +++ b/LICENSE @@ -0,0 +1,176 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS \ No newline at end of file diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..3cf08aa --- /dev/null +++ b/Makefile @@ -0,0 +1,20 @@ +VERSION := 2022.1.0 +REVISION := $(shell git rev-parse --short HEAD) +BUILD_DATE := $(shell date -u "+%Y-%m-%dT%H:%M:%SZ") +LDFLAGS := "-X main.version=$(VERSION) -X main.revision=$(REVISION) -X main.buildDate=$(BUILD_DATE)" +LDFLAGS_PROD := "-s -w -X main.version=$(VERSION) -X main.revision=$(REVISION)" + +export GO1111MODULE=on +export CWD=$(dir $(abspath $(lastword $(MAKEFILE_LIST)))) + +.PHONY: all sora-archive-uploader-dev sora-archive-uploader-prod +all: sora-archive-uploader-dev + +sora-archive-uploader-dev: cmd/sora-archive-uploader/main.go + go build -race -ldflags $(LDFLAGS) -o bin/$@ $< + +sora-archive-uploader-prod: cmd/sora-archive-uploader/main.go + go build -ldflags $(LDFLAGS_PROD) -o bin/$@ $< + +test: + go test -v ./db/test diff --git a/README.md b/README.md new file mode 100644 index 0000000..ebe4342 --- /dev/null +++ b/README.md @@ -0,0 +1,79 @@ +# Sora Archive Uploader + + +[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) + + +## About Shiguredo's open source software + +We will not respond to PRs or issues that have not been discussed on Discord. Also, Discord is only available in Japanese. + +Please read https://github.com/shiguredo/oss/blob/master/README.en.md before use. + +## 時雨堂のオープンソースソフトウェアについて + +利用前に https://github.com/shiguredo/oss をお読みください。 + +## Sora Archive Uploader について + +Sora が出力する録画関連のファイルを S3 または S3 互換オブジェクトストレージにアップロードするツールです。 +systemd タイマーユニットを利用しての定期実行を想定しています。 + +[Sora Cloud](https://sora-cloud.shiguredo.jp/) で実際に利用しているツールから切り出して公開しています。 + +## 目的 + +Sora は録画を行った場合、録画ファイルを WebM 、録画メタデータ JSON ファイルで出力します。 +Sora Cloud では出力されたファイルをオブジェクトストレージにアップロード仕組みが必要となり開発しました。 + +## 特徴 + +- systemd の設定だけで利用できます +- 並列でオブジェクトストレージにアップロードできます +- アップロード完了時に指定された URL にウェブフックリクエストを通知します +- ウェブフックにはベーシック認証や mTLS が利用可能です +- アップロードに失敗した場合は設定ファイルで指定した隔離ディレクトリに移動します + +## まずは使ってみる + +config.toml に必要な情報を設定してください。 + +```console +$ cp config.example.com config.toml +``` + +make でビルドして実行します。 + +```console +$ make +$ ./bin/sora-archive-uploader-dev -C config.toml +``` + +## Discord + +最新の状況などは Discord で共有しています。質問や相談も Discord でのみ受け付けています。 + +https://discord.gg/shiguredo + +## 優先実装 + +- Amazon S3 SSE-S3 への対応 + +## ライセンス + +``` +Copyright 2022-2022, Takeshi Namao (Original Author) +Copyright 2022-2022, Shiguredo Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +``` diff --git a/cmd/sora-archive-uploader/main.go b/cmd/sora-archive-uploader/main.go new file mode 100644 index 0000000..fcfdcac --- /dev/null +++ b/cmd/sora-archive-uploader/main.go @@ -0,0 +1,38 @@ +package main + +import ( + "flag" + "fmt" + "log" + "os" + + archive "github.com/shiguredo/sora-archive-uploader" +) + +var ( + version string + revision string + buildDate string + + versionText = `sora-archive-uploader build info. +version: %s +revision: %s +build date: %s +` +) + +func main() { + configFilePath := flag.String("C", "config.toml", "Config file path") + var v bool + flag.BoolVar(&v, "version", false, "Show version") + flag.Parse() + + if v { + fmt.Printf(versionText, version, revision, buildDate) + os.Exit(0) + } + + log.Printf("sora-archive-uploader version:%s revision:%s build_date:%s", version, revision, buildDate) + log.Printf("config file path: %s", *configFilePath) + archive.Run(configFilePath) +} diff --git a/config.example.toml b/config.example.toml new file mode 100644 index 0000000..07893aa --- /dev/null +++ b/config.example.toml @@ -0,0 +1,64 @@ +debug = false + +# Sora の録画アーカイブディレクトリのフルパス +archive_dir_full_path = "" +# アップロードに失敗した際の待避ディレクトリのフルパス +evacuate_dir_full_path = "" + +# 同時アップロード数 +upload_workers = 4 + +# 起動中にアップロード処理を行なったファイルを記録するキャッシュの上限で、 +# 重複してアップロードしてしまうのを避けます。 +# 起動時にアップロードする録画ファイルが大量にある場合は、 +# 大きめのキャッシュサイズを設定することをおすすめします。 +uploaded_file_cache_size = 32 + +# [log] + +log_dir = "." +log_name = "sora-archive-uploader.jsonl" +log_std_out = true + +# MB +log_rotate_max_size = 200 +log_rotate_max_backups = 7 +# day +log_rotate_max_age = 30 +log_rotate_compress = false + +# [object_storage] + +# アップロード先の S3 または S3 互換オブジェクトストレージの設定 +object_storage_endpoint = "" +object_storage_bucket_name = "" +object_storage_access_key_id = "" +object_storage_secret_access_key = "" + +# オブジェクトストレージにアップロードが完了した際に通知するウェブフック +# [webhook] + +# 空文字列の場合はウェブフックは飛ばさない +webhook_endpoint_url = "" + +# ウェブフックリクエストのタイムアウト時間 (秒) +webhook_request_timeout_s = 30 + +# ウェブフックタイプが入ってくるヘッダー名 +webhook_type_header_name = "sora-archive-uploader-webhook-type" +webhook_type_archive_uploaded = "archive.uploaded" +webhook_type_split_archive_uploaded = "split-archive.uploaded" +webhook_type_split_archive_end_uploaded = "split-archive-end.uploaded" +webhook_type_report_uploaded = "recording-report.uploaded" + +# ウェブフックのベーシック認証 +# 空文字はベーシック認証を行わない +webhook_basic_auth_username = "" +webhook_basic_auth_password = "" + +# webhook で HTTPS を利用する場合にサーバーの証明書をベリファイする場合に指定 +# 指定しない場合は OS のものを利用し、サーバー名までは検証しません +webhook_tls_verify_cacert_path = "" +# webhook で mTLS を利用する場合に指定します +webhook_tls_fullchain_path = "" +webhook_tls_privkey_path = "" \ No newline at end of file diff --git a/config.go b/config.go new file mode 100644 index 0000000..4cb93b4 --- /dev/null +++ b/config.go @@ -0,0 +1,56 @@ +package archive + +import ( + "github.com/BurntSushi/toml" +) + +type Config struct { + Debug bool `toml:"debug"` + + LogDir string `toml:"log_dir"` + LogName string `toml:"log_name"` + LogStdOut bool `toml:"log_std_out"` + LogRotateMaxSize int `toml:"log_rotate_max_size"` + LogRotateMaxBackups int `toml:"log_rotate_max_backups"` + LogRotateMaxAge int `toml:"log_rotate_max_age"` + LogRotateCompress bool `toml:"log_rotate_compress"` + + ObjectStorageEndpoint string `toml:"object_storage_endpoint"` + ObjectStorageBucketName string `toml:"object_storage_bucket_name"` + ObjectStorageAccessKeyID string `toml:"object_storage_access_key_id"` + ObjectStorageSecretAccessKey string `toml:"object_storage_secret_access_key"` + + SoraArchiveDirFullPath string `toml:"archive_dir_full_path"` + SoraEvacuateDirFullPath string `toml:"evacuate_dir_full_path"` + + UploadWorkers int `toml:"upload_workers"` + + UploadedFileCacheSize int `toml:"uploaded_file_cache_size"` + + WebhookEndpointURL string `toml:"webhook_endpoint_url"` + WebhookEndpointHealthCheckURL string `toml:"webhook_endpoint_health_check_url"` + + WebhookTypeHeaderName string `toml:"webhook_type_header_name"` + WebhookTypeArchiveUploaded string `toml:"webhook_type_archive_uploaded"` + WebhookTypeSplitArchiveUploaded string `toml:"webhook_type_split_archive_uploaded"` + WebhookTypeSplitArchiveEndUploaded string `toml:"webhook_type_split_archive_end_uploaded"` + WebhookTypeReportUploaded string `toml:"webhook_type_report_uploaded"` + + WebhookBasicAuthUsername string `toml:"webhook_basic_auth_username"` + WebhookBasicAuthPassword string `toml:"webhook_basic_auth_password"` + + WebhookRequestTimeoutS int32 `toml:"webhook_request_timeout_s"` + + WebhookTlsVerifyCacertPath string `toml:"webhook_tls_verify_cacert_path"` + WebhookTlsFullchainPath string `toml:"webhook_tls_fullchain_path"` + WebhookTlsPrivkeyPath string `toml:"webhook_tls_privkey_path"` +} + +func initConfig(data []byte, config interface{}) error { + if err := toml.Unmarshal(data, config); err != nil { + return err + } + + // TODO: 初期値 + return nil +} diff --git a/finder.go b/finder.go new file mode 100644 index 0000000..32210a8 --- /dev/null +++ b/finder.go @@ -0,0 +1,78 @@ +package archive + +import ( + "os" + "path/filepath" + "regexp" + "strings" + + zlog "github.com/rs/zerolog/log" +) + +func runFileFinder(archiveDir string) ([]string, error) { + var result []string + zlog.Debug().Str("archive-dir", archiveDir).Msg("START-SCRAPE-DIRECTORY") + files, err := os.ReadDir(archiveDir) + if err != nil { + zlog.Err(err).Msg("ERROR-RUN-FILE-FINDER") + return result, err + } + replaceWebmPattern := regexp.MustCompile(`.json$`) + for _, f := range files { + if !f.IsDir() { + continue + } + dirPath := filepath.Join(archiveDir, f.Name()) + archiveFiles, err := os.ReadDir(dirPath) + if err != nil { + zlog.Err(err).Msg("ERROR-READ-DIRECTORY") + continue + } + var reportFile *string + for _, archiveFile := range archiveFiles { + fullpath := filepath.Join(dirPath, archiveFile.Name()) + filename := archiveFile.Name() + if !(strings.HasSuffix(filename, ".json")) { + zlog.Debug(). + Str("file_path", fullpath). + Msg("IGNORE-FILE-TYPE") + continue + } + // 以下の処理は .json ファイルであることが保証される + if strings.HasPrefix(filename, "report-") { + zlog.Debug(). + Str("file_path", fullpath). + Msg("FOUND-AT-FINDER") + reportFile = &fullpath + } else if strings.HasPrefix(filename, "split-archive-end-") { + zlog.Debug(). + Str("file_path", fullpath). + Msg("FOUND-AT-FINDER") + result = append(result, fullpath) + } else if strings.HasPrefix(filename, "archive-") || strings.HasPrefix(filename, "split-archive-") { + // webm ファイルの存在を確認し、ファイルが存在したら後続の処理にファイルパスを渡す + // webm ファイルが存在しない場合は、次回のスクレイピングのタイミングで処理する + webmFilename := replaceWebmPattern.ReplaceAllString(filename, ".webm") + webmFullpath := filepath.Join(dirPath, webmFilename) + if info, err := os.Stat(webmFullpath); err != nil || info.IsDir() { + continue + } + zlog.Debug(). + Str("file_path", fullpath). + Str("webm_file_path", webmFullpath). + Msg("FOUND-AT-FINDER") + result = append(result, fullpath) + } else { + zlog.Debug(). + Str("file_path", fullpath). + Msg("IGNORE-FILE") + } + } + // ディレクトリ内に report json ファイルが見つかった場合は、最後に流す + if reportFile != nil { + result = append(result, *reportFile) + } + } + zlog.Debug().Str("archive-dir", archiveDir).Msg("END-SCRAPE-DIRECTORY") + return result, nil +} diff --git a/gatekeeper.go b/gatekeeper.go new file mode 100644 index 0000000..3d1849c --- /dev/null +++ b/gatekeeper.go @@ -0,0 +1,233 @@ +package archive + +import ( + "context" + "os" + "path/filepath" + "strings" + "sync" + "sync/atomic" + + zlog "github.com/rs/zerolog/log" +) + +type RecordingUnit struct { + mutex sync.RWMutex + recordingID string + counter int32 + reportFile string +} + +func newRecordingUnit(recordingID string) *RecordingUnit { + return &RecordingUnit{ + recordingID: recordingID, + counter: 0, + } +} + +func (ru *RecordingUnit) run() { + ru.mutex.Lock() + defer ru.mutex.Unlock() + atomic.AddInt32(&ru.counter, 1) + zlog.Debug(). + Str("recording_id", ru.recordingID). + Int32("counter", ru.counter). + Msgf("RUN-RECORDING-UNIT") +} + +func (ru *RecordingUnit) done() { + atomic.AddInt32(&ru.counter, -1) + zlog.Debug(). + Str("recording_id", ru.recordingID). + Int32("counter", ru.counter). + Msgf("DONE-RECORDING-UNIT") +} + +func (ru *RecordingUnit) canProcessAndSetReportFile(reportFile string) bool { + ru.mutex.Lock() + defer ru.mutex.Unlock() + if atomic.LoadInt32(&ru.counter) == 0 { + return true + } + ru.reportFile = reportFile + return false +} + +func (ru *RecordingUnit) canProcessAndGetReportFile() (*string, bool) { + ru.mutex.Lock() + defer ru.mutex.Unlock() + if atomic.LoadInt32(&ru.counter) == 0 && ru.reportFile != "" { + return &ru.reportFile, true + } + return nil, false +} + +type GateKeeper struct { + mutex sync.RWMutex + config *Config + ctx context.Context + processingList sync.Map + processingCounter int64 + out chan string +} + +func newGateKeeper(config *Config) *GateKeeper { + g := &GateKeeper{ + config: config, + processingList: sync.Map{}, + out: make(chan string, 50), + } + return g +} + +func (g *GateKeeper) stop() { + close(g.out) + zlog.Debug().Msg("STOPPED-GATE-KEEPER") +} + +func (g *GateKeeper) run(ctx context.Context, infiles []string) <-chan string { + g.ctx = ctx + go func() { + defer g.stop() + + processArchiveFile := func(infile string) { + filename := filepath.Base(infile) + recordingID := filepath.Base(filepath.Dir(infile)) + atomic.AddInt64(&g.processingCounter, 1) + if strings.HasPrefix(filename, "report-") { + g.mutex.Lock() + ru, ok := g.getRecordingUnit(recordingID) + if !ok { + // report-* の前に他のファイルが処理されてない + g.mutex.Unlock() + go func() { + select { + case <-g.ctx.Done(): + case g.out <- infile: + } + }() + return + } + g.mutex.Unlock() + + if ru.canProcessAndSetReportFile(infile) { + go func() { + select { + case <-g.ctx.Done(): + case g.out <- infile: + } + }() + return + } + return + } + if strings.HasPrefix(filename, "split-archive-end-") { + g.processRun(recordingID) + select { + case <-g.ctx.Done(): + case g.out <- infile: + } + return + } + if strings.HasPrefix(filename, "archive-") || strings.HasPrefix(filename, "split-archive-") { + archiveID := strings.Split(filename, ".")[0] + g.processRun(recordingID) + select { + case <-g.ctx.Done(): + return + case g.out <- infile: + zlog.Debug().Str("archive_id", archiveID).Msg("RUN-ARCHIVE-FILE-PROCESS") + } + return + } + } + + for _, infile := range infiles { + processArchiveFile(infile) + } + <-g.ctx.Done() + }() + return g.out +} + +func (g *GateKeeper) getRecordingUnit(recordingID string) (*RecordingUnit, bool) { + ru, ok := g.processingList.Load(recordingID) + if ok { + return ru.(*RecordingUnit), ok + } else { + return nil, ok + } +} + +func (g *GateKeeper) processRun(recordingID string) { + g.mutex.Lock() + defer g.mutex.Unlock() + ru, ok := g.getRecordingUnit(recordingID) + if !ok { + ru = newRecordingUnit(recordingID) + g.processingList.Store(recordingID, ru) + } + ru.run() +} + +func (g *GateKeeper) processDone(infile string) { + g.mutex.Lock() + defer g.mutex.Unlock() + zlog.Debug().Str("infile", infile).Msg("PROCESS-DONE") + + recordingID := filepath.Base(filepath.Dir(infile)) + ru, ok := g.getRecordingUnit(recordingID) + if !ok { + atomic.AddInt64(&g.processingCounter, -1) + zlog.Error().Str("infile", infile).Msg("WAIT-GROUP-NOT-FOUND") + return + } + ru.done() + if reportFile, ok := ru.canProcessAndGetReportFile(); ok { + go func() { + select { + case <-g.ctx.Done(): + case g.out <- *reportFile: + } + }() + } + atomic.AddInt64(&g.processingCounter, -1) +} + +func (g *GateKeeper) recordingDone(infile string) { + zlog.Debug().Str("infile", infile).Msg("RECORDING-DONE") + + // Recording ID のディレクトリを削除するべきだが、いきなり削除せず、mv で監視対象外のパスに移動しておく + dirname := filepath.Dir(infile) + newDirPath := filepath.Join(g.config.SoraEvacuateDirFullPath, filepath.Base(dirname)) + var evacuatePath = g.config.SoraEvacuateDirFullPath + _, err := os.Stat(g.config.SoraEvacuateDirFullPath) + if err != nil { + err = os.Mkdir(evacuatePath, 0755) + if err != nil { + zlog.Error(). + Str("evacuate_dir_path", evacuatePath). + Str("old_path", dirname). + Str("new_path", newDirPath). + Msg("EVACUATE-DIRECTORY-CREATE-ERROR") + } + } + err = os.Rename(dirname, newDirPath) + if err != nil { + zlog.Error(). + Err(err). + Str("old_path", dirname). + Str("new_path", newDirPath). + Msg("RECORDING-DIRECTORY-MOVE-ERROR") + } else { + zlog.Debug(). + Str("old_path", dirname). + Str("new_path", newDirPath). + Msg("RECORDING-DIRECTORY-MOVE-SUCCESSFULLY") + } + atomic.AddInt64(&g.processingCounter, -1) +} + +func (g *GateKeeper) isFileUploadFinished() bool { + return atomic.LoadInt64(&g.processingCounter) == 0 +} diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..2514323 --- /dev/null +++ b/go.mod @@ -0,0 +1,32 @@ +module github.com/shiguredo/sora-archive-uploader + +go 1.19 + +require ( + github.com/BurntSushi/toml v1.2.1 + github.com/google/uuid v1.3.0 + github.com/minio/minio-go/v7 v7.0.45 + github.com/rs/zerolog v1.28.0 + github.com/shiguredo/lumberjack/v3 v3.0.0 + github.com/shogo82148/go-clockwork-base32 v1.0.0 +) + +require ( + github.com/dustin/go-humanize v1.0.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/klauspost/compress v1.15.9 // indirect + github.com/klauspost/cpuid/v2 v2.1.0 // indirect + github.com/mattn/go-colorable v0.1.12 // indirect + github.com/mattn/go-isatty v0.0.14 // indirect + github.com/minio/md5-simd v1.1.2 // indirect + github.com/minio/sha256-simd v1.0.0 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/rs/xid v1.4.0 // indirect + github.com/sirupsen/logrus v1.9.0 // indirect + golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa // indirect + golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect + golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect + golang.org/x/text v0.3.7 // indirect + gopkg.in/ini.v1 v1.66.6 // indirect +) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..5cbdff3 --- /dev/null +++ b/go.sum @@ -0,0 +1,65 @@ +github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak= +github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= +github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= +github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= +github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY= +github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU= +github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0= +github.com/klauspost/cpuid/v2 v2.1.0/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= +github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= +github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= +github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= +github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= +github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= +github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= +github.com/minio/minio-go/v7 v7.0.45 h1:g4IeM9M9pW/Lo8AGGNOjBZYlvmtlE1N5TQEYWXRWzIs= +github.com/minio/minio-go/v7 v7.0.45/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw= +github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= +github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY= +github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= +github.com/rs/zerolog v1.28.0 h1:MirSo27VyNi7RJYP3078AA1+Cyzd2GB66qy3aUHvsWY= +github.com/rs/zerolog v1.28.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= +github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= +github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= +github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= +github.com/shogo82148/go-clockwork-base32 v1.0.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= +github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= +github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c= +golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= +golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= +gopkg.in/ini.v1 v1.66.6/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/logging.go b/logging.go new file mode 100644 index 0000000..f261919 --- /dev/null +++ b/logging.go @@ -0,0 +1,68 @@ +package archive + +import ( + "fmt" + "io" + "os" + "strings" + "time" + + "github.com/rs/zerolog" + "github.com/rs/zerolog/log" + "github.com/shiguredo/lumberjack/v3" +) + +func initLogger(config Config) error { + if f, err := os.Stat(config.LogDir); os.IsNotExist(err) || !f.IsDir() { + return err + } + + logPath := fmt.Sprintf("%s/%s", config.LogDir, config.LogName) + + writer := &lumberjack.Logger{ + Filename: logPath, + MaxSize: config.LogRotateMaxSize, + MaxBackups: config.LogRotateMaxBackups, + MaxAge: config.LogRotateMaxAge, + Compress: false, + } + + // https://github.com/rs/zerolog/issues/77 + zerolog.TimestampFunc = func() time.Time { + return time.Now().UTC() + } + + zerolog.TimeFieldFormat = time.RFC3339Nano + + var writers io.Writer + writers = zerolog.MultiLevelWriter(writer) + + // ログレベル + if config.Debug { + zerolog.SetGlobalLevel(zerolog.DebugLevel) + } else { + zerolog.SetGlobalLevel(zerolog.InfoLevel) + } + + if config.LogStdOut { + consoleWriter := zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: "2006-01-02 15:04:05.000000Z"} + format(&consoleWriter) + writers = zerolog.MultiLevelWriter(writers, consoleWriter) + } + + log.Logger = zerolog.New(writers).With().Caller().Timestamp().Logger() + + return nil +} + +func format(w *zerolog.ConsoleWriter) { + w.FormatLevel = func(i interface{}) string { + return strings.ToUpper(fmt.Sprintf("[%s]", i)) + } + w.FormatFieldName = func(i interface{}) string { + return fmt.Sprintf("%s=", i) + } + w.FormatFieldValue = func(i interface{}) string { + return fmt.Sprintf("%s", i) + } +} diff --git a/renovate.json b/renovate.json new file mode 100644 index 0000000..5474339 --- /dev/null +++ b/renovate.json @@ -0,0 +1,19 @@ +{ + "$schema": "https://docs.renovatebot.com/renovate-schema.json", + "extends": [ + "config:base" + ], + "dependencyDashboard": false, + "packageRules": [ + { + "matchUpdateTypes": [ + "minor", + "patch", + "pin", + "digest" + ], + "platformAutomerge": true, + "automerge": true + } + ] +} \ No newline at end of file diff --git a/runner.go b/runner.go new file mode 100644 index 0000000..9638734 --- /dev/null +++ b/runner.go @@ -0,0 +1,192 @@ +package archive + +import ( + "context" + "log" + "os" + "os/signal" + "syscall" + "time" + + zlog "github.com/rs/zerolog/log" +) + +type Main struct { + config Config +} + +func newMain(config Config) *Main { + return &Main{ + config: config, + } +} + +func (m *Main) run(ctx context.Context, cancel context.CancelFunc) error { + var archiveDir = m.config.SoraArchiveDirFullPath + zlog.Info().Str("path", archiveDir).Msg("WATCHING-ROOT-DIR") + fileInfo, err := os.Stat(archiveDir) + if err != nil { + // 対象のディレクトリが存在しなければ終わる + zlog.Fatal().Err(err).Str("path", archiveDir).Msg("NOT-FOUND-TARGET-PATH") + } + if !fileInfo.IsDir() { + // 対象のパスが Directory でなければ終わる + zlog.Fatal().Str("path", archiveDir).Msg("TARGET-PATH-DOES-NOT-DIRECTORY") + } + // TODO: パーミッションチェック + + // ディレクトリ退避先を作成する + var evacuatePath = m.config.SoraEvacuateDirFullPath + _, err = os.Stat(m.config.SoraEvacuateDirFullPath) + if err != nil { + err = os.Mkdir(evacuatePath, 0755) + if err != nil { + zlog.Fatal(). + Str("evacuate_dir_path", evacuatePath). + Msg("CANT-CREATE-DIRECTORY") + } + } + + foundFiles, err := runFileFinder(archiveDir) + if err != nil { + return err + } + if len(foundFiles) == 0 { + // 処理対象のファイルが見つからなかったので終わる + cancel() + zlog.Info().Msg("ARCHIVE-FILE-NOT-FOUND") + return nil + } + + processContext, processContextCancel := context.WithCancel(context.Background()) + gateKeeper := newGateKeeper(&m.config) + recordingFileStream := gateKeeper.run(processContext, foundFiles) + + uploaderManager := newUploaderManager() + _, err = uploaderManager.run(processContext, &m.config, recordingFileStream) + if err != nil { + processContextCancel() + return err + } + + for { + select { + case <-ctx.Done(): + processContextCancel() + // 停止ログ出力待ちのため、500ms 待ってから停止している + <-time.After(500 * time.Millisecond) + return nil + case archiveFileResult := <-uploaderManager.ArchiveStream: + if !archiveFileResult.Success { + zlog.Warn(). + Str("archive_file", archiveFileResult.Filepath). + Msg("FAILED-UPLOAD-ARCHIVE-END") + } + // zlog.Info(). + // Str("archive_file", archiveFileResult.Filepath). + // Msg("UPLOADED-ARCHIVE-FILE") + gateKeeper.processDone(archiveFileResult.Filepath) + if gateKeeper.isFileUploadFinished() { + cancel() + } + case archiveEndFileResult := <-uploaderManager.ArchiveEndStream: + if !archiveEndFileResult.Success { + zlog.Warn(). + Str("archive_end_file", archiveEndFileResult.Filepath). + Msg("FAILED-UPLOAD-ARCHIVE-END") + } + // zlog.Info(). + // Str("archive_end_file", archiveEndFileResult.Filepath). + // Msg("UPLOADED-ARCHIVE-END-FILE") + gateKeeper.processDone(archiveEndFileResult.Filepath) + if gateKeeper.isFileUploadFinished() { + cancel() + } + case reportFileResult := <-uploaderManager.ReportStream: + if !reportFileResult.Success { + zlog.Warn(). + Str("report_file", reportFileResult.Filepath). + Msg("FAILED-UPLOAD-ARCHIVE-END") + } + // zlog.Info(). + // Str("report_file", reportFileResult.Filepath). + // Msg("UPLOADED-REPORT-FILE") + gateKeeper.recordingDone(reportFileResult.Filepath) + if gateKeeper.isFileUploadFinished() { + cancel() + } + } + } +} + +func Run(configFilePath *string) { + buf, err := os.ReadFile(*configFilePath) + if err != nil { + // 読み込めない場合 Fatal で終了 + log.Fatal("cannot open config file, err=", err) + } + + // toml をパース + var config Config + if err := initConfig(buf, &config); err != nil { + // パースに失敗した場合 Fatal で終了 + log.Fatal("cannot parse config file, err=", err) + } + + // ロガー初期化 + err = initLogger(config) + if err != nil { + // ロガー初期化に失敗したら Fatal で終了 + log.Fatal("cannot parse config file, err=", err) + } + + // もしあれば mTLS の設定確認と Webhook のヘルスチェック + if config.WebhookEndpointHealthCheckURL != "" { + client, err := createHttpClient(&config) + if err != nil { + zlog.Fatal().Err(err).Msg("FAILED-CREATE-RPC-CLIENT") + } + // ヘルスチェック URL で起動確認する + resp, err := client.Get(config.WebhookEndpointHealthCheckURL) + if err != nil { + zlog.Fatal().Err(err).Msg("WEBHOOK-SERVER-CONNECT-ERROR") + } + if resp.StatusCode != 200 { + zlog.Fatal().Err(err).Msg("WEBHOOK-SERVER-UNHEALTHY") + } + resp.Body.Close() + } + + zlog.Info().Msg("STARTED-SORA-ARCHIVE-UPLOADER") + + // シグナルをキャッチして停止処理 + trapSignals := []os.Signal{ + syscall.SIGINT, + syscall.SIGTERM, + } + signalChannel := make(chan os.Signal, 1) + signal.Notify(signalChannel, trapSignals...) + ctx, cancel := context.WithCancel(context.Background()) + doneShutdown := make(chan interface{}) + defer close(doneShutdown) + go func() { + sig := <-signalChannel + zlog.Debug().Str("signal", sig.String()).Msg("RECEIVED-SIGNAL") + + cancel() + doneShutdown <- struct{}{} + }() + + // ディレクトリ監視とアップロード処理 + m := newMain(config) + if err := m.run(ctx, cancel); err != nil { + zlog.Error().Err(err).Msg("FAILED-RUN") + os.Exit(1) + } else { + go func() { + doneShutdown <- struct{}{} + }() + } + <-doneShutdown + zlog.Info().Msg("STOPPED-SORA-ARCHIVE-UPLOADER") +} diff --git a/s3.go b/s3.go new file mode 100644 index 0000000..9860868 --- /dev/null +++ b/s3.go @@ -0,0 +1,133 @@ +package archive + +import ( + "context" + "fmt" + "io" + "net/url" + "os" + "path/filepath" + + "github.com/minio/minio-go/v7" + "github.com/minio/minio-go/v7/pkg/credentials" + + zlog "github.com/rs/zerolog/log" +) + +// config ではなくこちらにまとめる +type S3CompatibleObjectStorage struct { + Endpoint string + AccessKeyID string + SecretAccessKey string + BucketName string +} + +func uploadJSONFile( + ctx context.Context, + osConfig *S3CompatibleObjectStorage, + reader io.Reader, size int64, dst string, +) (string, error) { + s3Client, err := minio.New( + osConfig.Endpoint, + &minio.Options{ + Creds: credentials.NewStaticV4( + osConfig.AccessKeyID, + osConfig.SecretAccessKey, + "", + ), + Secure: true, + }) + if err != nil { + return "", err + } + + n, err := s3Client.PutObject(ctx, + osConfig.BucketName, dst, + reader, size, + minio.PutObjectOptions{ContentType: "application/octet-stream"}, + ) + if err != nil { + return "", err + } + zlog.Debug(). + Str("dst", dst). + Int64("size", n.Size). + Msg("UPLAOD-SUCCESSFULLY") + + reqParams := make(url.Values) + filename := filepath.Base(dst) + zlog.Debug(). + Str("filename", filename). + Msg("CREATE-CONTENT-DISPOSITION-FILENAME") + reqParams.Set( + "response-content-disposition", + fmt.Sprintf("attachment; filename=\"%s\"", filename), + ) + + objectUrl := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) + return objectUrl, nil +} + +func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, file *os.File, dst string) (string, error) { + s3Client, err := minio.New( + osConfig.Endpoint, + &minio.Options{ + Creds: credentials.NewStaticV4( + osConfig.AccessKeyID, + osConfig.SecretAccessKey, + "", + ), + Secure: true, + }) + if err != nil { + return "", err + } + + fileStat, err := file.Stat() + if err != nil { + return "", err + } + + zlog.Debug(). + Str("dst", dst). + Msg("WEB-UPLOAD-START") + n, err := s3Client.PutObject(ctx, + osConfig.BucketName, dst, + file, fileStat.Size(), + minio.PutObjectOptions{ContentType: "application/octet-stream"}, + ) + if err != nil { + return "", err + } + zlog.Debug(). + Str("dst", dst). + Int64("size", n.Size). + Msg("UPLOAD-WEBM-SUCCESSFULLY") + + reqParams := make(url.Values) + filename := filepath.Base(dst) + zlog.Debug(). + Str("filename", filename). + Msg("create content-disposition filename") + reqParams.Set( + "response-content-disposition", + fmt.Sprintf("attachment; filename=\"%s\"", filename), + ) + + objectUrl := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) + return objectUrl, nil +} + +// minio のエラーをレスポンスに復元して、リトライするためファイルを残すか対象のファイルを削除するか判断する +func isFileContinuous(err error) bool { + errResp := minio.ToErrorResponse(err) + switch errResp.Code { + case "NoSuchBucket": + return false + case "AccessDenied": + return false + case "InvalidRegion": + return false + } + return true +} diff --git a/sora-archive-uploader.service b/sora-archive-uploader.service new file mode 100644 index 0000000..1dc1ada --- /dev/null +++ b/sora-archive-uploader.service @@ -0,0 +1,21 @@ +[Unit] +Description=Sora Archive Uploader Service +RefuseManualStart=no +RefuseManualStop=yes +After=network-online.target + +[Service] +Type=oneshot +User=sora +Group=sora +PermissionsStartOnly=true +Restart=no + +WorkingDirectory=/home/sora/sora-archive-uploader +ExecStartPre=/bin/mkdir -p /var/log/sora-archive-uploader +ExecStartPre=/bin/chown -R sora:sora /var/log/sora-archive-uploader + +ExecStart=/home/sora/sora-archive-uploader/bin/sora-archive-uploader-prod -C /home/sora/sora-archive-uploader/config.toml + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/sora-archive-uploader.timer b/sora-archive-uploader.timer new file mode 100644 index 0000000..21909be --- /dev/null +++ b/sora-archive-uploader.timer @@ -0,0 +1,9 @@ +[Unit] +Description=Sora Archive Uploader Wake Timer + +[Timer] +OnBootSec=2min +OnUnitActiveSec=1min + +[Install] +WantedBy=timers.target \ No newline at end of file diff --git a/uploader.go b/uploader.go new file mode 100644 index 0000000..558edb8 --- /dev/null +++ b/uploader.go @@ -0,0 +1,708 @@ +package archive + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "time" + + "github.com/google/uuid" + base32 "github.com/shogo82148/go-clockwork-base32" + + zlog "github.com/rs/zerolog/log" +) + +type RecordingReport struct { + RecordingID string `json:"recording_id"` + ChannelID string `json:"channel_id"` + FilePath string `json:"file_path"` + Filename string `json:"filename"` +} + +type UploaderManager struct { + ArchiveStream chan UploaderResult + ArchiveEndStream chan UploaderResult + ReportStream chan UploaderResult + uploaders []Uploader +} + +type ArchiveMetadata struct { + RecordingID string `json:"recording_id"` + ChannelID string `json:"channel_id"` + SessionID string `json:"session_id"` + ClientID string `json:"client_id"` + ConnectionID string `json:"connection_id"` + FilePath string `json:"file_path"` + Filename string `json:"filename"` + MetadataFilePath string `json:"metadata_file_path"` + MetadataFilename string `json:"metadata_filename"` +} + +type ArchiveEndMetadata struct { + RecordingID string `json:"recording_id"` + ChannelID string `json:"channel_id"` + SessionID string `json:"session_id"` + ClientID string `json:"client_id"` + ConnectionID string `json:"connection_id"` + FilePath string `json:"file_path"` + Filename string `json:"filename"` +} + +func newUploaderManager() *UploaderManager { + var uploaders []Uploader + archiveStream := make(chan UploaderResult) + archiveEndStream := make(chan UploaderResult) + reportStream := make(chan UploaderResult) + return &UploaderManager{ + ArchiveStream: archiveStream, + ArchiveEndStream: archiveEndStream, + ReportStream: reportStream, + uploaders: uploaders, + } +} + +func (um *UploaderManager) run(ctx context.Context, config *Config, fileStream <-chan string) (*UploaderManager, error) { + for i := 0; i < config.UploadWorkers; i++ { + uploader, err := newUploader(i+1, config) + if err != nil { + return nil, err + } + uploader.run(fileStream, um.ArchiveStream, um.ArchiveEndStream, um.ReportStream) + um.uploaders = append(um.uploaders, *uploader) + } + go func() { + defer func() { + close(um.ArchiveStream) + close(um.ArchiveEndStream) + close(um.ReportStream) + }() + + <-ctx.Done() + zlog.Debug().Msg("STOP-UPLOADER-MANAGER") + for _, u := range um.uploaders { + u.Stop() + } + zlog.Debug().Msg("STOPPED-UPLOADER-MANAGER") + }() + return um, nil +} + +type UploaderResult struct { + Success bool + Filepath string +} + +type Uploader struct { + id int + config *Config + ctx context.Context + cancel context.CancelFunc + base32Encoder *base32.Encoding +} + +func newUploader(id int, config *Config) (*Uploader, error) { + u := &Uploader{ + id: id, + config: config, + base32Encoder: base32.NewEncoding(), + } + u.ctx, u.cancel = context.WithCancel(context.Background()) + return u, nil +} + +func (u Uploader) run( + fileStream <-chan string, + outArchive chan UploaderResult, + outArchiveEnd chan UploaderResult, + outReport chan UploaderResult, +) { + go func() { + for { + select { + case <-u.ctx.Done(): + zlog.Debug(). + Int("uploader_id", u.id). + Msg("STOPPED-UPLOADER") + return + case inputFilepath, ok := <-fileStream: + if !ok { + continue + } + filename := filepath.Base(inputFilepath) + if strings.HasPrefix(filename, "report-") { + zlog.Debug(). + Int("uploader_id", u.id). + Str("json_file_path", inputFilepath). + Msg("FOUND-AT-STARTUP") + ok := u.handleReport(inputFilepath) + select { + case <-u.ctx.Done(): + return + case outReport <- UploaderResult{ + Success: ok, + Filepath: inputFilepath, + }: + } + } else if strings.HasPrefix(filename, "split-archive-end-") { + zlog.Debug(). + Int("uploader_id", u.id). + Str("json_file_path", inputFilepath). + Msg("FOUND-AT-STARTUP") + ok := u.handleArchiveEnd(inputFilepath) + select { + case <-u.ctx.Done(): + return + case outArchiveEnd <- UploaderResult{ + Success: ok, + Filepath: inputFilepath, + }: + } + } else if strings.HasPrefix(filename, "archive-") { + zlog.Debug(). + Int("uploader_id", u.id). + Str("file_path", inputFilepath). + Msg("FOUND-AT-STARTUP") + ok := u.handleArchive(inputFilepath, false) + select { + case <-u.ctx.Done(): + return + case outArchive <- UploaderResult{ + Success: ok, + Filepath: inputFilepath, + }: + } + } else if strings.HasPrefix(filename, "split-archive-") { + zlog.Debug(). + Int("uploader_id", u.id). + Str("file_path", inputFilepath). + Msg("FOUND-AT-STARTUP") + ok := u.handleArchive(inputFilepath, true) + select { + case <-u.ctx.Done(): + return + case outArchive <- UploaderResult{ + Success: ok, + Filepath: inputFilepath, + }: + } + } + } + } + }() +} + +func (u Uploader) Stop() { + u.cancel() +} + +func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { + fileInfo, err := os.Stat(archiveJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Msg("JSON-NOT-ACCESSABLE") + return false + } + + // json をパースする + raw, err := os.ReadFile(archiveJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveJSONFilePath). + Msg("ARCHIVE-JSON-FILE-READ-ERROR") + return false + } + + var am ArchiveMetadata + if err := json.Unmarshal(raw, &am); err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveJSONFilePath). + Msg("ARCHIVE-JSON-PARSE-ERROR") + return false + } + + // ここで s3 ファイルをアップロード + // json がくればファイルパスもわかる + zlog.Debug(). + Int("uploader_id", u.id). + Str("recording_id", am.RecordingID). + Str("channel_id", am.ChannelID). + Str("connection_id", am.ConnectionID). + Msg("ARCHIVE-METADATA-INFO") + + // webm ファイルのパスを作っておく + webmFilename := filepath.Base(am.Filename) + webmFilepath := filepath.Join(filepath.Dir(archiveJSONFilePath), webmFilename) + // []byte を *bytes.Reader へ + reader := bytes.NewReader(raw) + + // metadata ファイル (json) をアップロード + metadataFilename := fileInfo.Name() + metadataObjectKey := fmt.Sprintf("%s/%s", am.RecordingID, metadataFilename) + osConfig := &S3CompatibleObjectStorage{ + Endpoint: u.config.ObjectStorageEndpoint, + BucketName: u.config.ObjectStorageBucketName, + AccessKeyID: u.config.ObjectStorageAccessKeyID, + SecretAccessKey: u.config.ObjectStorageSecretAccessKey, + } + + // webm ファイルを開いておく + f, err := os.Open(webmFilepath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveJSONFilePath). + Str("webm_filename", am.FilePath). + Msg("WEBM-FILE-OPEN-ERROR") + return false + } + defer f.Close() + + metadataFileURL, err := uploadJSONFile( + u.ctx, + osConfig, + reader, + reader.Size(), + metadataObjectKey, + ) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveJSONFilePath). + Str("metadata_filename", metadataFilename). + Str("metadata_object_key", metadataObjectKey). + Msg("METADATA-FILE-UPLOAD-ERROR") + if !isFileContinuous(err) { + // リトライしないエラーの場合は、ファイルを削除 + u.removeArchiveJSONFile(archiveJSONFilePath, webmFilepath) + u.removeArchiveWEBMFile(archiveJSONFilePath, webmFilepath) + } + return false + } + zlog.Debug(). + Int("uploader_id", u.id). + Str("uploaded_matadata", am.MetadataFilename). + Msg("UPLOAD-METADATA-FILE-SUCCESSFULLY") + + webmObjectKey := fmt.Sprintf("%s/%s", am.RecordingID, webmFilename) + fileURL, err := uploadWebMFile(u.ctx, osConfig, f, webmObjectKey) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveJSONFilePath). + Str("webm_filename", webmFilename). + Str("webm_object_key", webmObjectKey). + Msg("WEBM-FILE-UPLOAD-ERROR") + if !isFileContinuous(err) { + // リトライしないエラーの場合は、ファイルを削除 + u.removeArchiveJSONFile(archiveJSONFilePath, webmFilepath) + u.removeArchiveWEBMFile(archiveJSONFilePath, webmFilepath) + } + return false + } + zlog.Debug(). + Int("uploader_id", u.id). + Str("uploaded_webm", am.Filename). + Msg("UPLOAD-WEBM-FILE-SUCCESSFULLY") + + if u.config.WebhookEndpointURL != "" { + var archiveUploadedType string + if split { + archiveUploadedType = u.config.WebhookTypeSplitArchiveUploaded + } else { + archiveUploadedType = u.config.WebhookTypeArchiveUploaded + } + webhookID, err := u.generateWebhookID() + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("uploaded_webm", am.Filename). + Msg("WEBHOOK-ID-GENERATE-ERROR") + return false + } + var w = WebhookArchiveUploaded{ + ID: webhookID, + Type: archiveUploadedType, + Timestamp: time.Now().UTC(), + SessionID: am.SessionID, + ClientID: am.ClientID, + RecordingID: am.RecordingID, + ChannelID: am.ChannelID, + ConnectionID: am.ConnectionID, + Filename: webmFilename, + FileURL: fileURL, + MetadataFilename: metadataFilename, + MetadataFileURL: metadataFileURL, + } + buf, err := json.Marshal(w) + if err != nil { + zlog.Error(). + Int("uploader_id", u.id). + Err(err). + Msg("ARCHIVE-UPLOADED-WEBHOOK-MARSHAL-ERROR") + return false + } + if err := u.postWebhook( + archiveUploadedType, + buf, + ); err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("recording_id", w.RecordingID). + Str("channel_id", w.ChannelID). + Str("filename", w.Filename). + Str("metadata_filename", w.MetadataFilename). + Msg("ARCHIVE-UPLOADED-WEBHOOK-SEND-ERROR") + return false + } + } + + // 処理し終わったファイルを削除 + jsonError := u.removeArchiveJSONFile(archiveJSONFilePath, webmFilepath) + webmError := u.removeArchiveWEBMFile(archiveJSONFilePath, webmFilepath) + return jsonError == nil && webmError == nil +} + +func (u Uploader) handleReport(reportJSONFilePath string) bool { + fileInfo, err := os.Stat(reportJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Msg("JSON-NOT-ACCESSABLE") + return false + } + + // report- ファイルのアップロード + // json をパースする + raw, err := os.ReadFile(reportJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", reportJSONFilePath). + Msg("REPORT-JSON-FILE-READ-ERROR") + return false + } + var rr RecordingReport + if err := json.Unmarshal(raw, &rr); err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", reportJSONFilePath). + Msg("REPORT-JSON-FILE-UNMARSHAL-ERROR") + return false + } + + // []byte を *bytes.Reader へ + reader := bytes.NewReader(raw) + // report ファイル (json) をアップロード + filename := fileInfo.Name() + reportObjectKey := fmt.Sprintf("%s/%s", rr.RecordingID, filename) + osConfig := &S3CompatibleObjectStorage{ + Endpoint: u.config.ObjectStorageEndpoint, + BucketName: u.config.ObjectStorageBucketName, + AccessKeyID: u.config.ObjectStorageAccessKeyID, + SecretAccessKey: u.config.ObjectStorageSecretAccessKey, + } + + fileURL, err := uploadJSONFile( + u.ctx, + osConfig, + reader, reader.Size(), + reportObjectKey, + ) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("filename", filename). + Str("report_object_key", reportObjectKey). + Msg("REPORT-FILE-UPLOAD-ERROR") + if !isFileContinuous(err) { + // リトライしないエラーの場合は、ファイルを削除 + u.removeReportFile(reportJSONFilePath) + } + return false + } + zlog.Debug(). + Int("uploader_id", u.id). + Str("uplaoded_report", filename). + Msg("UPLOAD-REPORT-JSON-SUCCESSFULLY") + + if u.config.WebhookEndpointURL != "" { + webhookID, err := u.generateWebhookID() + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("uploaded_report", filename). + Msg("WEBHOOK-ID-GENERATE-ERROR") + return false + } + var w = WebhookReportUploaded{ + ID: webhookID, + Type: u.config.WebhookTypeReportUploaded, + Timestamp: time.Now().UTC(), + RecordingID: rr.RecordingID, + ChannelID: rr.ChannelID, + Filename: filename, + FileURL: fileURL, + } + buf, err := json.Marshal(w) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("recording_id", w.RecordingID). + Str("channel_id", w.ChannelID). + Str("filename", w.Filename). + Msg("REPORT-UPLOAD-WEBHOOK-MARSHAL-ERROR") + return false + } + if err := u.postWebhook( + u.config.WebhookTypeReportUploaded, + buf, + ); err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("recording_id", w.RecordingID). + Str("channel_id", w.ChannelID). + Str("filename", w.Filename). + Msg("REPORT-UPLOADED-WEBHOOK-SEND-ERROR") + return false + } + zlog.Debug(). + Int("uploader_id", u.id). + Str("recording_id", w.RecordingID). + Str("channel_id", w.ChannelID). + Str("filename", w.Filename). + Msg("REPORT-UPLOADED-WEBHOOK-SEND-SUCCESSFULLY") + } + + // 処理し終わったファイルを削除 + if err = u.removeReportFile(reportJSONFilePath); err != nil { + return false + } + return true +} + +func (u Uploader) handleArchiveEnd(archiveEndJSONFilePath string) bool { + fileInfo, err := os.Stat(archiveEndJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Msg("JSON-NOT-ACCESSABLE") + return false + } + + // json をパースする + raw, err := os.ReadFile(archiveEndJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveEndJSONFilePath). + Msg("archive json file read error") + return false + } + + var aem ArchiveEndMetadata + if err := json.Unmarshal(raw, &aem); err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveEndJSONFilePath). + Msg("ARCHIVE-END-JSON-FILE-PARSE-ERROR") + return false + } + + zlog.Debug(). + Int("uploader_id", u.id). + Str("recording_id", aem.RecordingID). + Str("channel_id", aem.ChannelID). + Str("connection_id", aem.ConnectionID). + Msg("ARCHIVE-END-METADATA-INFO") + + // []byte を *bytes.Reader へ + reader := bytes.NewReader(raw) + + // metadata ファイル (json) をアップロード + filename := fileInfo.Name() + objectKey := fmt.Sprintf("%s/%s", aem.RecordingID, filename) + osConfig := &S3CompatibleObjectStorage{ + Endpoint: u.config.ObjectStorageEndpoint, + BucketName: u.config.ObjectStorageBucketName, + AccessKeyID: u.config.ObjectStorageAccessKeyID, + SecretAccessKey: u.config.ObjectStorageSecretAccessKey, + } + + archiveEndURL, err := uploadJSONFile( + u.ctx, + osConfig, + reader, reader.Size(), + objectKey, + ) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("path", archiveEndJSONFilePath). + Str("filename", filename). + Str("object_key", objectKey). + Msg("METADATA-FILE-UPLOAD-ERROR") + if !isFileContinuous(err) { + u.removeArchiveEndFile(archiveEndJSONFilePath) + } + return false + } + zlog.Debug(). + Int("uploader_id", u.id). + Str("uploaded_archive_end", aem.Filename). + Str("archive_end_presigned_url", archiveEndURL). + Msg("UPLOAD-ARCHIVE-END-FILE-SUCCESSFULLY") + + if u.config.WebhookEndpointURL != "" { + webhookID, err := u.generateWebhookID() + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("uploaded_archive_end", aem.Filename). + Str("archive_end_presigned_url", archiveEndURL). + Msg("WEBHOOK-ID-GENERATE-ERROR") + return false + } + var w = WebhookArchiveEndUploaded{ + ID: webhookID, + Type: u.config.WebhookTypeSplitArchiveEndUploaded, + Timestamp: time.Now().UTC(), + RecordingID: aem.RecordingID, + SessionID: aem.SessionID, + ClientID: aem.ClientID, + ChannelID: aem.ChannelID, + ConnectionID: aem.ConnectionID, + Filename: filename, + FileURL: archiveEndURL, + } + buf, err := json.Marshal(w) + if err != nil { + zlog.Error(). + Int("uploader_id", u.id). + Err(err). + Msg("ARCHIVE-UPLOADED-WEBHOOK-MARSHAL-ERROR") + return false + } + if err := u.postWebhook( + u.config.WebhookTypeSplitArchiveEndUploaded, + buf, + ); err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("recording_id", w.RecordingID). + Str("channel_id", w.ChannelID). + Str("filename", w.Filename). + Msg("ARCHIVE-END-UPLOADED-WEBHOOK-SEND-ERROR") + return false + } + } + + if err = u.removeArchiveEndFile(archiveEndJSONFilePath); err != nil { + return false + } + return true +} + +func (u Uploader) removeArchiveJSONFile(metadataFilePath, webmFilepath string) error { + err := os.Remove(metadataFilePath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("metadata_filepath", metadataFilePath). + Str("archive_filepath", webmFilepath). + Msg("FAILED-REMOVE-METADATA-JSON-FILE") + } else { + zlog.Debug(). + Int("uploader_id", u.id). + Str("metadata_filepath", metadataFilePath). + Str("archive_filepath", webmFilepath). + Msg("REMOVED-METADATA-JSON-FILE") + } + return err +} + +func (u Uploader) removeArchiveWEBMFile(metadataFilePath, webmFilepath string) error { + err := os.Remove(webmFilepath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("archive_filepath", webmFilepath). + Msg("FAILED-REMOVE-ARCHIVE-WEBM-FILE") + } else { + zlog.Debug(). + Int("uploader_id", u.id). + Str("archive_filepath", webmFilepath). + Msg("remove archive webm file successfully.") + } + return err +} + +func (u Uploader) removeReportFile(reportJSONFilePath string) error { + err := os.Remove(reportJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("filepath", reportJSONFilePath). + Msg("FAILED-REMOVE-REPORT-JSON-FILE") + } else { + zlog.Debug(). + Int("uploader_id", u.id). + Str("filepath", reportJSONFilePath). + Msg("REMOVED-REPORT-JSON-FILE") + + } + return err +} + +func (u Uploader) removeArchiveEndFile(archiveEndJSONFilePath string) error { + err := os.Remove(archiveEndJSONFilePath) + if err != nil { + zlog.Error(). + Err(err). + Int("uploader_id", u.id). + Str("filepath", archiveEndJSONFilePath). + Msg("FAILED-REMOVE-ARCHIVE-END-FILE") + } else { + zlog.Debug(). + Int("uploader_id", u.id). + Str("filepath", archiveEndJSONFilePath). + Msg("REMOVED-ARCHIVE-END-FILE") + } + return err +} + +func (u Uploader) generateWebhookID() (string, error) { + id := uuid.New() + binaryUUID, err := id.MarshalBinary() + if err != nil { + return "", err + } + return u.base32Encoder.EncodeToString(binaryUUID), nil +} diff --git a/webhook.go b/webhook.go new file mode 100644 index 0000000..4601bca --- /dev/null +++ b/webhook.go @@ -0,0 +1,144 @@ +package archive + +import ( + "bytes" + "crypto/tls" + "crypto/x509" + "fmt" + "net/http" + "net/url" + "os" + "time" +) + +type WebhookReportUploaded struct { + ID string `json:"id"` + Type string `json:"type"` + Timestamp time.Time `json:"timestamp"` + RecordingID string `json:"recording_id"` + ChannelID string `json:"channel_id"` + Filename string `json:"filename"` + FileURL string `json:"file_url"` +} + +type WebhookArchiveUploaded struct { + ID string `json:"id"` + Type string `json:"type"` + Timestamp time.Time `json:"timestamp"` + RecordingID string `json:"recording_id"` + SessionID string `json:"session_id"` + ClientID string `json:"client_id"` + ChannelID string `json:"channel_id"` + ConnectionID string `json:"connection_id"` + Filename string `json:"filename"` + FileURL string `json:"file_url"` + MetadataFilename string `json:"metadata_filename"` + MetadataFileURL string `json:"metadata_file_url"` +} + +type WebhookArchiveEndUploaded struct { + ID string `json:"id"` + Type string `json:"type"` + Timestamp time.Time `json:"timestamp"` + RecordingID string `json:"recording_id"` + SessionID string `json:"session_id"` + ClientID string `json:"client_id"` + ChannelID string `json:"channel_id"` + ConnectionID string `json:"connection_id"` + Filename string `json:"filename"` + FileURL string `json:"file_url"` +} + +// mTLS を組み込んだ http.Client を構築する +func createHttpClient(config *Config) (*http.Client, error) { + e, err := url.Parse(config.WebhookEndpointURL) + if err != nil { + return nil, err + } + + // http または VerifyCacertPath 指定していない場合はそのまま投げる + if e.Scheme != "https" || config.WebhookTlsVerifyCacertPath == "" { + client := &http.Client{ + CheckRedirect: func(req *http.Request, via []*http.Request) error { + return http.ErrUseLastResponse + }, + Timeout: time.Duration(config.WebhookRequestTimeoutS) * time.Second, + } + + return client, nil + } + + CaCert, err := os.ReadFile(config.WebhookTlsVerifyCacertPath) + if err != nil { + return nil, err + } + caCertPool := x509.NewCertPool() + caCertPool.AppendCertsFromPEM(CaCert) + + var certificates []tls.Certificate + if config.WebhookTlsFullchainPath != "" && config.WebhookTlsPrivkeyPath != "" { + pair, err := tls.LoadX509KeyPair(config.WebhookTlsFullchainPath, config.WebhookTlsPrivkeyPath) + if err != nil { + return nil, err + } + certificates = append(certificates, pair) + } + + client := &http.Client{ + CheckRedirect: func(req *http.Request, via []*http.Request) error { + return http.ErrUseLastResponse + + }, + Timeout: time.Duration(config.WebhookRequestTimeoutS) * time.Second, + Transport: &http.Transport{ + TLSClientConfig: &tls.Config{ + // hostname はチェックする + ServerName: e.Hostname(), + RootCAs: caCertPool, + Certificates: certificates, + }, + // TODO: config へ + // ForceAttemptHTTP2: true, + }, + } + + return client, nil +} + +func (u Uploader) httpClientDo(client *http.Client, webhookType string, buf []byte) error { + req, err := http.NewRequest("POST", u.config.WebhookEndpointURL, bytes.NewBuffer(buf)) + if err != nil { + return err + } + + // 固有ヘッダーを追加する + req.Header.Set("Content-Type", "application/json") + req.Header.Add(u.config.WebhookTypeHeaderName, webhookType) + + // 設定があれば Basic 認証に対応する + if u.config.WebhookBasicAuthUsername != "" && u.config.WebhookBasicAuthPassword != "" { + req.SetBasicAuth(u.config.WebhookBasicAuthUsername, u.config.WebhookBasicAuthPassword) + } + + resp, err := client.Do(req) + if err != nil { + return err + } + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("status_code: %d", resp.StatusCode) + } + + return nil +} + +func (u Uploader) postWebhook(webhookType string, buf []byte) error { + client, err := createHttpClient(u.config) + if err != nil { + return err + } + if err := u.httpClientDo(client, webhookType, buf); err != nil { + return err + } + + return nil +} From 29ea88e3ca85c821b203db4f5cd626eb4702eb52 Mon Sep 17 00:00:00 2001 From: voluntas Date: Sat, 17 Dec 2022 13:19:10 +0900 Subject: [PATCH 02/79] =?UTF-8?q?=E5=AF=BE=E5=BF=9C=E3=82=AA=E3=83=96?= =?UTF-8?q?=E3=82=B8=E3=82=A7=E3=82=AF=E3=83=88=E3=82=B9=E3=83=88=E3=83=AC?= =?UTF-8?q?=E3=83=BC=E3=82=B8=E3=82=92=E8=BF=BD=E5=8A=A0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/README.md b/README.md index ebe4342..f50e146 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,16 @@ Sora Cloud では出力されたファイルをオブジェクトストレージ - ウェブフックにはベーシック認証や mTLS が利用可能です - アップロードに失敗した場合は設定ファイルで指定した隔離ディレクトリに移動します +### 対応オブジェクトストレージ + +- AWS S3 +- MinIO +- GCP GCS +- Vultr Object Storage +- Linode Object Storage +- DigitalOcean Spaces +- Cloudflare R2 + ## まずは使ってみる config.toml に必要な情報を設定してください。 From eea855d9524763bb5fd762b7b3f7dccf431c9691 Mon Sep 17 00:00:00 2001 From: voluntas Date: Sat, 17 Dec 2022 13:37:23 +0900 Subject: [PATCH 03/79] =?UTF-8?q?Azure=20Blob=20Storage=20=E5=AF=BE?= =?UTF-8?q?=E5=BF=9C=E3=82=92=E5=84=AA=E5=85=88=E5=AE=9F=E8=A3=85=E3=81=AB?= =?UTF-8?q?=E8=BF=BD=E5=8A=A0=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index f50e146..8e6f994 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,7 @@ https://discord.gg/shiguredo ## 優先実装 - Amazon S3 SSE-S3 への対応 +- [Azure Blob Storage](https://azure.microsoft.com/ja-jp/products/storage/blobs/) への対応 ## ライセンス From 29fd4264734ad7c5c2430f43a00e63ee03ea774f Mon Sep 17 00:00:00 2001 From: voluntas Date: Sat, 17 Dec 2022 13:37:48 +0900 Subject: [PATCH 04/79] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8e6f994..6fa8220 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ https://discord.gg/shiguredo ## 優先実装 -- Amazon S3 SSE-S3 への対応 +- [Amazon S3 SSE-S3](https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/UsingServerSideEncryption.html) への対応 - [Azure Blob Storage](https://azure.microsoft.com/ja-jp/products/storage/blobs/) への対応 ## ライセンス From 41d89e2cf7bf72db12cb3b1a06bc9a1105342c5c Mon Sep 17 00:00:00 2001 From: voluntas Date: Sat, 17 Dec 2022 13:41:10 +0900 Subject: [PATCH 05/79] =?UTF-8?q?=E6=97=A5=E6=9C=AC=E8=AA=9E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6fa8220..43263ef 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ systemd タイマーユニットを利用しての定期実行を想定してい ## 目的 Sora は録画を行った場合、録画ファイルを WebM 、録画メタデータ JSON ファイルで出力します。 -Sora Cloud では出力されたファイルをオブジェクトストレージにアップロード仕組みが必要となり開発しました。 +Sora Cloud では出力されたファイルをオブジェクトストレージにアップロードする仕組みが必要となり開発しました。 ## 特徴 From 85a2c1c15d8d7b5b9087e9521dc6101ca562daed Mon Sep 17 00:00:00 2001 From: voluntas Date: Sat, 17 Dec 2022 14:05:12 +0900 Subject: [PATCH 06/79] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 43263ef..c123050 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Please read https://github.com/shiguredo/oss/blob/master/README.en.md before use Sora が出力する録画関連のファイルを S3 または S3 互換オブジェクトストレージにアップロードするツールです。 systemd タイマーユニットを利用しての定期実行を想定しています。 -[Sora Cloud](https://sora-cloud.shiguredo.jp/) で実際に利用しているツールから切り出して公開しています。 +[Sora Cloud](https://sora-cloud.shiguredo.jp/) で実際に利用している仕組みからツールとして切り出して公開しています。 ## 目的 From 364fbcd6d9bf3810144c1901f6d4b6be8726dbb9 Mon Sep 17 00:00:00 2001 From: voluntas Date: Sat, 17 Dec 2022 14:13:01 +0900 Subject: [PATCH 07/79] Update README.md --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c123050..6466e11 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,15 @@ $ ./bin/sora-archive-uploader-dev -C config.toml https://discord.gg/shiguredo -## 優先実装 +## 有償での優先実装が可能な機能一覧 + +**詳細は Discord またはメールにてお問い合わせください** + +- オープンソースでの公開が前提 +- 可能であれば企業名の公開 + - 公開が難しい場合は `企業名非公開` と書かせていただきます + +### 機能 - [Amazon S3 SSE-S3](https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/UsingServerSideEncryption.html) への対応 - [Azure Blob Storage](https://azure.microsoft.com/ja-jp/products/storage/blobs/) への対応 From db9cca13b5e8492a4ee229b8c7488efc894e5556 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Wed, 28 Dec 2022 16:03:16 +0900 Subject: [PATCH 08/79] =?UTF-8?q?AccessKeyID,=20SecretAccessKey=20?= =?UTF-8?q?=E3=81=8C=E6=8C=87=E5=AE=9A=E3=81=95=E3=82=8C=E3=81=A6=E3=81=84?= =?UTF-8?q?=E3=81=AA=E3=81=84=E5=A0=B4=E5=90=88=E3=81=AF=E3=80=81=E7=92=B0?= =?UTF-8?q?=E5=A2=83=E5=A4=89=E6=95=B0=E3=81=AE=20AWS=5FACCESS=5FKEY=5FID,?= =?UTF-8?q?=20AWS=5FSECRET=5FACCESS=5FKEY=20=E3=82=92=E4=BD=BF=E7=94=A8?= =?UTF-8?q?=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/s3.go b/s3.go index 9860868..6c35aa7 100644 --- a/s3.go +++ b/s3.go @@ -27,14 +27,20 @@ func uploadJSONFile( osConfig *S3CompatibleObjectStorage, reader io.Reader, size int64, dst string, ) (string, error) { + var creds *credentials.Credentials + if (osConfig.AccessKeyID == "") || (osConfig.SecretAccessKey == "") { + creds = credentials.NewEnvAWS() + } else { + creds = credentials.NewStaticV4( + osConfig.AccessKeyID, + osConfig.SecretAccessKey, + "", + ) + } s3Client, err := minio.New( osConfig.Endpoint, &minio.Options{ - Creds: credentials.NewStaticV4( - osConfig.AccessKeyID, - osConfig.SecretAccessKey, - "", - ), + Creds: creds, Secure: true, }) if err != nil { @@ -69,14 +75,20 @@ func uploadJSONFile( } func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, file *os.File, dst string) (string, error) { + var creds *credentials.Credentials + if (osConfig.AccessKeyID == "") || (osConfig.SecretAccessKey == "") { + creds = credentials.NewEnvAWS() + } else { + creds = credentials.NewStaticV4( + osConfig.AccessKeyID, + osConfig.SecretAccessKey, + "", + ) + } s3Client, err := minio.New( osConfig.Endpoint, &minio.Options{ - Creds: credentials.NewStaticV4( - osConfig.AccessKeyID, - osConfig.SecretAccessKey, - "", - ), + Creds: creds, Secure: true, }) if err != nil { From 14c631ea323c36657e3f5ee66a7ff1d349887a07 Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 16 Jan 2023 21:48:27 +0900 Subject: [PATCH 09/79] =?UTF-8?q?IAM=20=E3=81=AB=E5=AF=BE=E5=BF=9C?= =?UTF-8?q?=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/s3.go b/s3.go index 6c35aa7..4445463 100644 --- a/s3.go +++ b/s3.go @@ -28,14 +28,16 @@ func uploadJSONFile( reader io.Reader, size int64, dst string, ) (string, error) { var creds *credentials.Credentials - if (osConfig.AccessKeyID == "") || (osConfig.SecretAccessKey == "") { - creds = credentials.NewEnvAWS() - } else { + if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( osConfig.AccessKeyID, osConfig.SecretAccessKey, "", ) + } else if (len(os.Getenv("AWS_ACCESS_KEY_ID")) > 0) && (len(os.Getenv("AWS_SECRET_ACCESS_KEY")) > 0) { + creds = credentials.NewEnvAWS() + } else { + creds = credentials.NewIAM("") } s3Client, err := minio.New( osConfig.Endpoint, @@ -76,14 +78,16 @@ func uploadJSONFile( func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, file *os.File, dst string) (string, error) { var creds *credentials.Credentials - if (osConfig.AccessKeyID == "") || (osConfig.SecretAccessKey == "") { - creds = credentials.NewEnvAWS() - } else { + if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( - osConfig.AccessKeyID, - osConfig.SecretAccessKey, - "", + osConfig.AccessKeyID, + osConfig.SecretAccessKey, + "", ) + } else if (len(os.Getenv("AWS_ACCESS_KEY_ID")) > 0) && (len(os.Getenv("AWS_SECRET_ACCESS_KEY")) > 0) { + creds = credentials.NewEnvAWS() + } else { + creds = credentials.NewIAM("") } s3Client, err := minio.New( osConfig.Endpoint, From 8b021c93b37f9881911a2cf416ecb1aa15ba2962 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 26 Jan 2023 20:27:53 +0000 Subject: [PATCH 10/79] Update module github.com/rs/zerolog to v1.29.0 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 2514323..33513fb 100644 --- a/go.mod +++ b/go.mod @@ -6,7 +6,7 @@ require ( github.com/BurntSushi/toml v1.2.1 github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.45 - github.com/rs/zerolog v1.28.0 + github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 ) diff --git a/go.sum b/go.sum index 5cbdff3..c45ff4d 100644 --- a/go.sum +++ b/go.sum @@ -38,6 +38,8 @@ github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY= github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= github.com/rs/zerolog v1.28.0 h1:MirSo27VyNi7RJYP3078AA1+Cyzd2GB66qy3aUHvsWY= github.com/rs/zerolog v1.28.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= +github.com/rs/zerolog v1.29.0 h1:Zes4hju04hjbvkVkOhdl2HpZa+0PmVwigmo8XoORE5w= +github.com/rs/zerolog v1.29.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= From 05d6336dddbf60bc6b8a270946ef8872c432a4c9 Mon Sep 17 00:00:00 2001 From: voluntas Date: Sun, 29 Jan 2023 19:57:14 +0900 Subject: [PATCH 11/79] =?UTF-8?q?endpoint=20=E3=81=AB=20http://=20?= =?UTF-8?q?=E3=82=84=20https://=20=E3=82=92=E6=8C=87=E5=AE=9A=E3=81=A7?= =?UTF-8?q?=E3=81=8D=E3=82=8B=E3=82=88=E3=81=86=E3=81=AB=E3=81=99=E3=82=8B?= =?UTF-8?q?=E3=80=81=E6=8C=87=E5=AE=9A=E3=81=97=E3=81=AA=E3=81=84=E5=A0=B4?= =?UTF-8?q?=E5=90=88=E3=81=AF=E4=BB=8A=E3=81=BE=E3=81=A7=E9=80=9A=E3=82=8A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Makefile | 2 +- go.mod | 7 +++++++ go.sum | 13 +++++++++++-- s3.go | 54 ++++++++++++++++++++++++++++++++++++++++++------------ s3_test.go | 21 +++++++++++++++++++++ 5 files changed, 82 insertions(+), 15 deletions(-) create mode 100644 s3_test.go diff --git a/Makefile b/Makefile index 3cf08aa..f3c1139 100644 --- a/Makefile +++ b/Makefile @@ -17,4 +17,4 @@ sora-archive-uploader-prod: cmd/sora-archive-uploader/main.go go build -ldflags $(LDFLAGS_PROD) -o bin/$@ $< test: - go test -v ./db/test + go test -v diff --git a/go.mod b/go.mod index 33513fb..cf8e7c1 100644 --- a/go.mod +++ b/go.mod @@ -9,6 +9,13 @@ require ( github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 + github.com/stretchr/testify v1.8.1 +) + +require ( + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect ) require ( diff --git a/go.sum b/go.sum index c45ff4d..a91b2f1 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,7 @@ github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= @@ -33,11 +34,10 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY= github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= -github.com/rs/zerolog v1.28.0 h1:MirSo27VyNi7RJYP3078AA1+Cyzd2GB66qy3aUHvsWY= -github.com/rs/zerolog v1.28.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= github.com/rs/zerolog v1.29.0 h1:Zes4hju04hjbvkVkOhdl2HpZa+0PmVwigmo8XoORE5w= github.com/rs/zerolog v1.29.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= @@ -47,8 +47,14 @@ github.com/shogo82148/go-clockwork-base32 v1.0.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQx github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0= @@ -61,7 +67,10 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9w golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= gopkg.in/ini.v1 v1.66.6/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/s3.go b/s3.go index 4445463..c570458 100644 --- a/s3.go +++ b/s3.go @@ -39,12 +39,8 @@ func uploadJSONFile( } else { creds = credentials.NewIAM("") } - s3Client, err := minio.New( - osConfig.Endpoint, - &minio.Options{ - Creds: creds, - Secure: true, - }) + + s3Client, err := newS3Client(osConfig.Endpoint, creds) if err != nil { return "", err } @@ -89,12 +85,7 @@ func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, fi } else { creds = credentials.NewIAM("") } - s3Client, err := minio.New( - osConfig.Endpoint, - &minio.Options{ - Creds: creds, - Secure: true, - }) + s3Client, err := newS3Client(osConfig.Endpoint, creds) if err != nil { return "", err } @@ -147,3 +138,42 @@ func isFileContinuous(err error) bool { } return true } + + +func maybeEndpointURL(endpoint string) (string, bool) { + // もし endpoint に指定されたのが endpoint_url だった場合、 + // scheme をチェックして http ならば secure = false にする + // さらに host だけを取り出して endpoint として扱う + var secure = true + u, err := url.Parse(endpoint) + // エラーがあっても無視してそのまま文字列として扱う + // エラーがないときだけ scheme チェックする + if err == nil { + switch (u.Scheme) { + case "http": + // http なので secure を false にする + secure = false + return u.Host, secure + case "https": + return u.Host, secure + default: + // parse 失敗はタダの文字列として扱う + } + } + return endpoint, secure +} + +func newS3Client(endpoint string, credentials *credentials.Credentials) (*minio.Client, error) { + newEndpoint, secure := maybeEndpointURL(endpoint) + s3Client, err := minio.New( + newEndpoint, + &minio.Options{ + Creds: credentials, + Secure: secure, + }) + if err != nil { + return nil, err + } + + return s3Client, nil +} \ No newline at end of file diff --git a/s3_test.go b/s3_test.go new file mode 100644 index 0000000..038ef52 --- /dev/null +++ b/s3_test.go @@ -0,0 +1,21 @@ +package archive + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestMaybeEndpointURL(t *testing.T) { + endpoint, secure := maybeEndpointURL("s3.example.com") + assert.Equal(t, endpoint, "s3.example.com") + assert.True(t, secure) + + endpoint, secure = maybeEndpointURL("http://s3.example.com") + assert.Equal(t, endpoint, "s3.example.com") + assert.False(t, secure) + + endpoint, secure = maybeEndpointURL("https://s3.example.com") + assert.Equal(t, endpoint, "s3.example.com") + assert.True(t, secure) +} \ No newline at end of file From ebcbf12fb4053ce8503c6c27b4d46beb869243c8 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Tue, 31 Jan 2023 16:13:37 +0900 Subject: [PATCH 12/79] =?UTF-8?q?=E6=9C=9F=E5=BE=85=E5=80=A4=E3=81=A8?= =?UTF-8?q?=E7=B5=90=E6=9E=9C=E3=82=92=20expected,=20actual=20=E3=81=AB?= =?UTF-8?q?=E5=90=88=E3=82=8F=E3=81=9B=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3_test.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/s3_test.go b/s3_test.go index 038ef52..bbaf14f 100644 --- a/s3_test.go +++ b/s3_test.go @@ -8,14 +8,14 @@ import ( func TestMaybeEndpointURL(t *testing.T) { endpoint, secure := maybeEndpointURL("s3.example.com") - assert.Equal(t, endpoint, "s3.example.com") + assert.Equal(t, "s3.example.com", endpoint) assert.True(t, secure) endpoint, secure = maybeEndpointURL("http://s3.example.com") - assert.Equal(t, endpoint, "s3.example.com") + assert.Equal(t, "s3.example.com", endpoint) assert.False(t, secure) endpoint, secure = maybeEndpointURL("https://s3.example.com") - assert.Equal(t, endpoint, "s3.example.com") + assert.Equal(t, "s3.example.com", endpoint) assert.True(t, secure) -} \ No newline at end of file +} From c60222918f7a3c2a5a908ab88a51d43c4788312a Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Tue, 31 Jan 2023 16:25:51 +0900 Subject: [PATCH 13/79] =?UTF-8?q?scheme=20=E3=81=AA=E3=81=97=E3=81=AE?= =?UTF-8?q?=E5=A0=B4=E5=90=88=E3=81=AE=E5=87=A6=E7=90=86=E3=82=92=E8=BF=BD?= =?UTF-8?q?=E5=8A=A0=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 37 ++++++++++++++++++++----------------- s3_test.go | 4 ++++ 2 files changed, 24 insertions(+), 17 deletions(-) diff --git a/s3.go b/s3.go index c570458..60674ee 100644 --- a/s3.go +++ b/s3.go @@ -39,7 +39,7 @@ func uploadJSONFile( } else { creds = credentials.NewIAM("") } - + s3Client, err := newS3Client(osConfig.Endpoint, creds) if err != nil { return "", err @@ -76,14 +76,14 @@ func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, fi var creds *credentials.Credentials if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( - osConfig.AccessKeyID, - osConfig.SecretAccessKey, - "", + osConfig.AccessKeyID, + osConfig.SecretAccessKey, + "", ) - } else if (len(os.Getenv("AWS_ACCESS_KEY_ID")) > 0) && (len(os.Getenv("AWS_SECRET_ACCESS_KEY")) > 0) { - creds = credentials.NewEnvAWS() - } else { - creds = credentials.NewIAM("") + } else if (len(os.Getenv("AWS_ACCESS_KEY_ID")) > 0) && (len(os.Getenv("AWS_SECRET_ACCESS_KEY")) > 0) { + creds = credentials.NewEnvAWS() + } else { + creds = credentials.NewIAM("") } s3Client, err := newS3Client(osConfig.Endpoint, creds) if err != nil { @@ -139,27 +139,30 @@ func isFileContinuous(err error) bool { return true } - func maybeEndpointURL(endpoint string) (string, bool) { // もし endpoint に指定されたのが endpoint_url だった場合、 // scheme をチェックして http ならば secure = false にする // さらに host だけを取り出して endpoint として扱う - var secure = true + var secure = false u, err := url.Parse(endpoint) // エラーがあっても無視してそのまま文字列として扱う // エラーがないときだけ scheme チェックする if err == nil { - switch (u.Scheme) { + switch u.Scheme { case "http": - // http なので secure を false にする - secure = false return u.Host, secure case "https": + // https なので secure を true にする + secure = true return u.Host, secure + case "": + // scheme なしの場合は secure を true にする + secure = true + return endpoint, secure default: - // parse 失敗はタダの文字列として扱う + // サポート外の scheme の場合はタダの文字列として扱う } - } + } return endpoint, secure } @@ -168,7 +171,7 @@ func newS3Client(endpoint string, credentials *credentials.Credentials) (*minio. s3Client, err := minio.New( newEndpoint, &minio.Options{ - Creds: credentials, + Creds: credentials, Secure: secure, }) if err != nil { @@ -176,4 +179,4 @@ func newS3Client(endpoint string, credentials *credentials.Credentials) (*minio. } return s3Client, nil -} \ No newline at end of file +} diff --git a/s3_test.go b/s3_test.go index bbaf14f..cd36b23 100644 --- a/s3_test.go +++ b/s3_test.go @@ -18,4 +18,8 @@ func TestMaybeEndpointURL(t *testing.T) { endpoint, secure = maybeEndpointURL("https://s3.example.com") assert.Equal(t, "s3.example.com", endpoint) assert.True(t, secure) + + endpoint, secure = maybeEndpointURL("ldap://s3.example.com") + assert.Equal(t, "ldap://s3.example.com", endpoint) + assert.False(t, secure) } From 451f598d17450bac6f16c3da1a07526334a8284b Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Tue, 31 Jan 2023 16:29:29 +0900 Subject: [PATCH 14/79] =?UTF-8?q?=E8=BF=94=E3=81=99=E5=80=A4=E3=81=AF?= =?UTF-8?q?=E5=90=8C=E3=81=98=E3=81=9F=E3=82=81=E3=80=81=E3=81=99=E3=81=90?= =?UTF-8?q?=E3=81=AB=20return=20=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/s3.go b/s3.go index 60674ee..c63d425 100644 --- a/s3.go +++ b/s3.go @@ -168,15 +168,10 @@ func maybeEndpointURL(endpoint string) (string, bool) { func newS3Client(endpoint string, credentials *credentials.Credentials) (*minio.Client, error) { newEndpoint, secure := maybeEndpointURL(endpoint) - s3Client, err := minio.New( + return minio.New( newEndpoint, &minio.Options{ Creds: credentials, Secure: secure, }) - if err != nil { - return nil, err - } - - return s3Client, nil } From fe2b864dce99423982f01056a91f837e7bb0c6db Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Sat, 18 Feb 2023 14:30:15 +0000 Subject: [PATCH 15/79] Update module github.com/minio/minio-go/v7 to v7.0.49 --- go.mod | 18 +++++++++--------- go.sum | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index cf8e7c1..523523e 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.2.1 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.45 + github.com/minio/minio-go/v7 v7.0.49 github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 @@ -19,10 +19,10 @@ require ( ) require ( - github.com/dustin/go-humanize v1.0.0 // indirect + github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.15.9 // indirect - github.com/klauspost/cpuid/v2 v2.1.0 // indirect + github.com/klauspost/compress v1.15.15 // indirect + github.com/klauspost/cpuid/v2 v2.2.3 // indirect github.com/mattn/go-colorable v0.1.12 // indirect github.com/mattn/go-isatty v0.0.14 // indirect github.com/minio/md5-simd v1.1.2 // indirect @@ -31,9 +31,9 @@ require ( github.com/modern-go/reflect2 v1.0.2 // indirect github.com/rs/xid v1.4.0 // indirect github.com/sirupsen/logrus v1.9.0 // indirect - golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa // indirect - golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect - golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect - golang.org/x/text v0.3.7 // indirect - gopkg.in/ini.v1 v1.66.6 // indirect + golang.org/x/crypto v0.6.0 // indirect + golang.org/x/net v0.7.0 // indirect + golang.org/x/sys v0.5.0 // indirect + golang.org/x/text v0.7.0 // indirect + gopkg.in/ini.v1 v1.67.0 // indirect ) diff --git a/go.sum b/go.sum index a91b2f1..da48afc 100644 --- a/go.sum +++ b/go.sum @@ -6,6 +6,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= +github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= @@ -14,10 +16,14 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY= github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU= +github.com/klauspost/compress v1.15.15 h1:EF27CXIuDsYJ6mmvtBRlEuB2UVOqHG1tAXgZ7yIO+lw= +github.com/klauspost/compress v1.15.15/go.mod h1:ZcK2JAFqKOpnBlxcLsJzYfrS9X1akm9fHZNnD9+Vo/4= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0= github.com/klauspost/cpuid/v2 v2.1.0/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= +github.com/klauspost/cpuid/v2 v2.2.3 h1:sxCkb+qR91z4vsqw4vGGZlDgPz3G7gjaLyK3V8y70BU= +github.com/klauspost/cpuid/v2 v2.2.3/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= @@ -26,6 +32,8 @@ github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= github.com/minio/minio-go/v7 v7.0.45 h1:g4IeM9M9pW/Lo8AGGNOjBZYlvmtlE1N5TQEYWXRWzIs= github.com/minio/minio-go/v7 v7.0.45/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw= +github.com/minio/minio-go/v7 v7.0.49 h1:dE5DfOtnXMXCjr/HWI6zN9vCrY6Sv666qhhiwUMvGV4= +github.com/minio/minio-go/v7 v7.0.49/go.mod h1:UI34MvQEiob3Cf/gGExGMmzugkM/tNgbFypNDy5LMVc= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -57,20 +65,30 @@ github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKs github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= +golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc= +golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= +golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= gopkg.in/ini.v1 v1.66.6/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= +gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= +gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= From 941b197dc2bc28229710e7313f83571ee4cdab54 Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 20 Feb 2023 11:59:19 +0900 Subject: [PATCH 16/79] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6466e11..aa8ff78 100644 --- a/README.md +++ b/README.md @@ -81,8 +81,8 @@ https://discord.gg/shiguredo ## ライセンス ``` -Copyright 2022-2022, Takeshi Namao (Original Author) -Copyright 2022-2022, Shiguredo Inc. +Copyright 2022-2023, Takeshi Namao (Original Author) +Copyright 2022-2023, Shiguredo Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. From 4c299bcde7f08e209b92144cc2f2f4fe7ad2b73b Mon Sep 17 00:00:00 2001 From: voluntas Date: Wed, 22 Feb 2023 16:19:13 +0900 Subject: [PATCH 17/79] =?UTF-8?q?FPutObject=20=E3=82=92=E5=88=A9=E7=94=A8?= =?UTF-8?q?=E3=81=99=E3=82=8B=E3=82=88=E3=81=86=E3=81=AB=E5=A4=89=E6=9B=B4?= =?UTF-8?q?=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 20 ++++++-------------- uploader.go | 21 ++++++--------------- 2 files changed, 12 insertions(+), 29 deletions(-) diff --git a/s3.go b/s3.go index c63d425..cfa6e44 100644 --- a/s3.go +++ b/s3.go @@ -3,7 +3,6 @@ package archive import ( "context" "fmt" - "io" "net/url" "os" "path/filepath" @@ -25,7 +24,7 @@ type S3CompatibleObjectStorage struct { func uploadJSONFile( ctx context.Context, osConfig *S3CompatibleObjectStorage, - reader io.Reader, size int64, dst string, + dst, filePath string, ) (string, error) { var creds *credentials.Credentials if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { @@ -45,9 +44,8 @@ func uploadJSONFile( return "", err } - n, err := s3Client.PutObject(ctx, - osConfig.BucketName, dst, - reader, size, + n, err := s3Client.FPutObject(ctx, + osConfig.BucketName, dst, filePath, minio.PutObjectOptions{ContentType: "application/octet-stream"}, ) if err != nil { @@ -72,7 +70,7 @@ func uploadJSONFile( return objectUrl, nil } -func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, file *os.File, dst string) (string, error) { +func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, dst, filePath string) (string, error) { var creds *credentials.Credentials if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( @@ -90,17 +88,11 @@ func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, fi return "", err } - fileStat, err := file.Stat() - if err != nil { - return "", err - } - zlog.Debug(). Str("dst", dst). Msg("WEB-UPLOAD-START") - n, err := s3Client.PutObject(ctx, - osConfig.BucketName, dst, - file, fileStat.Size(), + n, err := s3Client.FPutObject(ctx, + osConfig.BucketName, dst, filePath, minio.PutObjectOptions{ContentType: "application/octet-stream"}, ) if err != nil { diff --git a/uploader.go b/uploader.go index 558edb8..10636f6 100644 --- a/uploader.go +++ b/uploader.go @@ -1,7 +1,6 @@ package archive import ( - "bytes" "context" "encoding/json" "fmt" @@ -204,7 +203,7 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { if err != nil { zlog.Error(). Err(err). - Msg("JSON-NOT-ACCESSABLE") + Msg("JSON-NOT-ACCESSIBLE") return false } @@ -241,8 +240,6 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { // webm ファイルのパスを作っておく webmFilename := filepath.Base(am.Filename) webmFilepath := filepath.Join(filepath.Dir(archiveJSONFilePath), webmFilename) - // []byte を *bytes.Reader へ - reader := bytes.NewReader(raw) // metadata ファイル (json) をアップロード metadataFilename := fileInfo.Name() @@ -270,9 +267,8 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { metadataFileURL, err := uploadJSONFile( u.ctx, osConfig, - reader, - reader.Size(), metadataObjectKey, + archiveJSONFilePath, ) if err != nil { zlog.Error(). @@ -295,7 +291,7 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { Msg("UPLOAD-METADATA-FILE-SUCCESSFULLY") webmObjectKey := fmt.Sprintf("%s/%s", am.RecordingID, webmFilename) - fileURL, err := uploadWebMFile(u.ctx, osConfig, f, webmObjectKey) + fileURL, err := uploadWebMFile(u.ctx, osConfig, webmObjectKey, webmFilepath) if err != nil { zlog.Error(). Err(err). @@ -406,8 +402,6 @@ func (u Uploader) handleReport(reportJSONFilePath string) bool { return false } - // []byte を *bytes.Reader へ - reader := bytes.NewReader(raw) // report ファイル (json) をアップロード filename := fileInfo.Name() reportObjectKey := fmt.Sprintf("%s/%s", rr.RecordingID, filename) @@ -421,8 +415,8 @@ func (u Uploader) handleReport(reportJSONFilePath string) bool { fileURL, err := uploadJSONFile( u.ctx, osConfig, - reader, reader.Size(), reportObjectKey, + reportJSONFilePath, ) if err != nil { zlog.Error(). @@ -505,7 +499,7 @@ func (u Uploader) handleArchiveEnd(archiveEndJSONFilePath string) bool { if err != nil { zlog.Error(). Err(err). - Msg("JSON-NOT-ACCESSABLE") + Msg("JSON-NOT-ACCESSIBLE") return false } @@ -537,9 +531,6 @@ func (u Uploader) handleArchiveEnd(archiveEndJSONFilePath string) bool { Str("connection_id", aem.ConnectionID). Msg("ARCHIVE-END-METADATA-INFO") - // []byte を *bytes.Reader へ - reader := bytes.NewReader(raw) - // metadata ファイル (json) をアップロード filename := fileInfo.Name() objectKey := fmt.Sprintf("%s/%s", aem.RecordingID, filename) @@ -553,8 +544,8 @@ func (u Uploader) handleArchiveEnd(archiveEndJSONFilePath string) bool { archiveEndURL, err := uploadJSONFile( u.ctx, osConfig, - reader, reader.Size(), objectKey, + archiveEndJSONFilePath, ) if err != nil { zlog.Error(). From c12228d0fbb8725a7e8c77d43fe1d74b715e468d Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Sat, 25 Feb 2023 13:42:17 +0000 Subject: [PATCH 18/79] Update module github.com/stretchr/testify to v1.8.2 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 523523e..039a053 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,7 @@ require ( github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 - github.com/stretchr/testify v1.8.1 + github.com/stretchr/testify v1.8.2 ) require ( diff --git a/go.sum b/go.sum index da48afc..3d18474 100644 --- a/go.sum +++ b/go.sum @@ -63,6 +63,8 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= +github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc= From 19651fc2af47cbbe1658a0924aed80ce86b854e0 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Fri, 24 Mar 2023 07:56:54 +0000 Subject: [PATCH 19/79] Update module github.com/minio/minio-go/v7 to v7.0.50 --- go.mod | 6 +++--- go.sum | 6 ++++++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 039a053..466bab2 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.2.1 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.49 + github.com/minio/minio-go/v7 v7.0.50 github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 @@ -21,8 +21,8 @@ require ( require ( github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.15.15 // indirect - github.com/klauspost/cpuid/v2 v2.2.3 // indirect + github.com/klauspost/compress v1.16.0 // indirect + github.com/klauspost/cpuid/v2 v2.2.4 // indirect github.com/mattn/go-colorable v0.1.12 // indirect github.com/mattn/go-isatty v0.0.14 // indirect github.com/minio/md5-simd v1.1.2 // indirect diff --git a/go.sum b/go.sum index 3d18474..80655bb 100644 --- a/go.sum +++ b/go.sum @@ -18,12 +18,16 @@ github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQan github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU= github.com/klauspost/compress v1.15.15 h1:EF27CXIuDsYJ6mmvtBRlEuB2UVOqHG1tAXgZ7yIO+lw= github.com/klauspost/compress v1.15.15/go.mod h1:ZcK2JAFqKOpnBlxcLsJzYfrS9X1akm9fHZNnD9+Vo/4= +github.com/klauspost/compress v1.16.0 h1:iULayQNOReoYUe+1qtKOqw9CwJv3aNQu8ivo7lw1HU4= +github.com/klauspost/compress v1.16.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0= github.com/klauspost/cpuid/v2 v2.1.0/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= github.com/klauspost/cpuid/v2 v2.2.3 h1:sxCkb+qR91z4vsqw4vGGZlDgPz3G7gjaLyK3V8y70BU= github.com/klauspost/cpuid/v2 v2.2.3/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= +github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk= +github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= @@ -34,6 +38,8 @@ github.com/minio/minio-go/v7 v7.0.45 h1:g4IeM9M9pW/Lo8AGGNOjBZYlvmtlE1N5TQEYWXRW github.com/minio/minio-go/v7 v7.0.45/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw= github.com/minio/minio-go/v7 v7.0.49 h1:dE5DfOtnXMXCjr/HWI6zN9vCrY6Sv666qhhiwUMvGV4= github.com/minio/minio-go/v7 v7.0.49/go.mod h1:UI34MvQEiob3Cf/gGExGMmzugkM/tNgbFypNDy5LMVc= +github.com/minio/minio-go/v7 v7.0.50 h1:4IL4V8m/kI90ZL6GupCARZVrBv8/XrcKcJhaJ3iz68k= +github.com/minio/minio-go/v7 v7.0.50/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= From 8876aba1bdc6d10ccfadf454b2882fab734ae1d1 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Sat, 8 Apr 2023 09:48:58 +0000 Subject: [PATCH 20/79] Update module github.com/minio/minio-go/v7 to v7.0.51 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 466bab2..961966d 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.2.1 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.50 + github.com/minio/minio-go/v7 v7.0.51 github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 diff --git a/go.sum b/go.sum index 80655bb..59cdaaa 100644 --- a/go.sum +++ b/go.sum @@ -40,6 +40,8 @@ github.com/minio/minio-go/v7 v7.0.49 h1:dE5DfOtnXMXCjr/HWI6zN9vCrY6Sv666qhhiwUMv github.com/minio/minio-go/v7 v7.0.49/go.mod h1:UI34MvQEiob3Cf/gGExGMmzugkM/tNgbFypNDy5LMVc= github.com/minio/minio-go/v7 v7.0.50 h1:4IL4V8m/kI90ZL6GupCARZVrBv8/XrcKcJhaJ3iz68k= github.com/minio/minio-go/v7 v7.0.50/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= +github.com/minio/minio-go/v7 v7.0.51 h1:eSewrwc23TqUDEH8aw8Bwp4f+JDdozRrPWcKR7DZhmY= +github.com/minio/minio-go/v7 v7.0.51/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= From a375f2c66957664ada3318eb273f6fb267099db8 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Tue, 11 Apr 2023 23:38:21 +0000 Subject: [PATCH 21/79] Update module github.com/minio/minio-go/v7 to v7.0.52 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 961966d..c16c085 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.2.1 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.51 + github.com/minio/minio-go/v7 v7.0.52 github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 diff --git a/go.sum b/go.sum index 59cdaaa..9b946d3 100644 --- a/go.sum +++ b/go.sum @@ -42,6 +42,8 @@ github.com/minio/minio-go/v7 v7.0.50 h1:4IL4V8m/kI90ZL6GupCARZVrBv8/XrcKcJhaJ3iz github.com/minio/minio-go/v7 v7.0.50/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/minio-go/v7 v7.0.51 h1:eSewrwc23TqUDEH8aw8Bwp4f+JDdozRrPWcKR7DZhmY= github.com/minio/minio-go/v7 v7.0.51/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= +github.com/minio/minio-go/v7 v7.0.52 h1:8XhG36F6oKQUDDSuz6dY3rioMzovKjW40W6ANuN0Dps= +github.com/minio/minio-go/v7 v7.0.52/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= From e0d9f7fb6c2526ecfe14f5460747f61b6e612878 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 9 May 2023 01:32:02 +0900 Subject: [PATCH 22/79] =?UTF-8?q?s3=20=E3=81=AF=E7=89=B9=E5=88=A5=E6=89=B1?= =?UTF-8?q?=E3=81=84=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.mod | 7 ++--- go.sum | 30 -------------------- s3.go | 55 ++++--------------------------------- s3/s3.go | 53 +++++++++++++++++++++++++++++++++++ s3_test.go => s3/s3_test.go | 2 +- uploader.go | 7 +++-- 6 files changed, 65 insertions(+), 89 deletions(-) create mode 100644 s3/s3.go rename s3_test.go => s3/s3_test.go (97%) diff --git a/go.mod b/go.mod index c16c085..2474243 100644 --- a/go.mod +++ b/go.mod @@ -14,11 +14,6 @@ require ( require ( github.com/davecgh/go-spew v1.1.1 // indirect - github.com/pmezard/go-difflib v1.0.0 // indirect - gopkg.in/yaml.v3 v3.0.1 // indirect -) - -require ( github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/compress v1.16.0 // indirect @@ -29,6 +24,7 @@ require ( github.com/minio/sha256-simd v1.0.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect github.com/rs/xid v1.4.0 // indirect github.com/sirupsen/logrus v1.9.0 // indirect golang.org/x/crypto v0.6.0 // indirect @@ -36,4 +32,5 @@ require ( golang.org/x/sys v0.5.0 // indirect golang.org/x/text v0.7.0 // indirect gopkg.in/ini.v1 v1.67.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/go.sum b/go.sum index 9b946d3..33ac175 100644 --- a/go.sum +++ b/go.sum @@ -4,8 +4,6 @@ github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= -github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= @@ -14,18 +12,10 @@ github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= -github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY= -github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU= -github.com/klauspost/compress v1.15.15 h1:EF27CXIuDsYJ6mmvtBRlEuB2UVOqHG1tAXgZ7yIO+lw= -github.com/klauspost/compress v1.15.15/go.mod h1:ZcK2JAFqKOpnBlxcLsJzYfrS9X1akm9fHZNnD9+Vo/4= github.com/klauspost/compress v1.16.0 h1:iULayQNOReoYUe+1qtKOqw9CwJv3aNQu8ivo7lw1HU4= github.com/klauspost/compress v1.16.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= -github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0= -github.com/klauspost/cpuid/v2 v2.1.0/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= -github.com/klauspost/cpuid/v2 v2.2.3 h1:sxCkb+qR91z4vsqw4vGGZlDgPz3G7gjaLyK3V8y70BU= -github.com/klauspost/cpuid/v2 v2.2.3/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk= github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= @@ -34,14 +24,6 @@ github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9 github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= -github.com/minio/minio-go/v7 v7.0.45 h1:g4IeM9M9pW/Lo8AGGNOjBZYlvmtlE1N5TQEYWXRWzIs= -github.com/minio/minio-go/v7 v7.0.45/go.mod h1:nCrRzjoSUQh8hgKKtu3Y708OLvRLtuASMg2/nvmbarw= -github.com/minio/minio-go/v7 v7.0.49 h1:dE5DfOtnXMXCjr/HWI6zN9vCrY6Sv666qhhiwUMvGV4= -github.com/minio/minio-go/v7 v7.0.49/go.mod h1:UI34MvQEiob3Cf/gGExGMmzugkM/tNgbFypNDy5LMVc= -github.com/minio/minio-go/v7 v7.0.50 h1:4IL4V8m/kI90ZL6GupCARZVrBv8/XrcKcJhaJ3iz68k= -github.com/minio/minio-go/v7 v7.0.50/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= -github.com/minio/minio-go/v7 v7.0.51 h1:eSewrwc23TqUDEH8aw8Bwp4f+JDdozRrPWcKR7DZhmY= -github.com/minio/minio-go/v7 v7.0.51/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/minio-go/v7 v7.0.52 h1:8XhG36F6oKQUDDSuz6dY3rioMzovKjW40W6ANuN0Dps= github.com/minio/minio-go/v7 v7.0.52/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= @@ -71,34 +53,22 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= -github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= -github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c= -golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc= golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= -golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0= -golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s= -golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= -golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= -gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= -gopkg.in/ini.v1 v1.66.6/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/s3.go b/s3.go index cfa6e44..7ab4d2b 100644 --- a/s3.go +++ b/s3.go @@ -9,21 +9,13 @@ import ( "github.com/minio/minio-go/v7" "github.com/minio/minio-go/v7/pkg/credentials" - zlog "github.com/rs/zerolog/log" + "github.com/shiguredo/sora-archive-uploader/s3" ) -// config ではなくこちらにまとめる -type S3CompatibleObjectStorage struct { - Endpoint string - AccessKeyID string - SecretAccessKey string - BucketName string -} - func uploadJSONFile( ctx context.Context, - osConfig *S3CompatibleObjectStorage, + osConfig *s3.S3CompatibleObjectStorage, dst, filePath string, ) (string, error) { var creds *credentials.Credentials @@ -39,7 +31,7 @@ func uploadJSONFile( creds = credentials.NewIAM("") } - s3Client, err := newS3Client(osConfig.Endpoint, creds) + s3Client, err := s3.NewClient(osConfig.Endpoint, creds) if err != nil { return "", err } @@ -70,7 +62,7 @@ func uploadJSONFile( return objectUrl, nil } -func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, dst, filePath string) (string, error) { +func uploadWebMFile(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, dst, filePath string) (string, error) { var creds *credentials.Credentials if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( @@ -83,7 +75,7 @@ func uploadWebMFile(ctx context.Context, osConfig *S3CompatibleObjectStorage, ds } else { creds = credentials.NewIAM("") } - s3Client, err := newS3Client(osConfig.Endpoint, creds) + s3Client, err := s3.NewClient(osConfig.Endpoint, creds) if err != nil { return "", err } @@ -130,40 +122,3 @@ func isFileContinuous(err error) bool { } return true } - -func maybeEndpointURL(endpoint string) (string, bool) { - // もし endpoint に指定されたのが endpoint_url だった場合、 - // scheme をチェックして http ならば secure = false にする - // さらに host だけを取り出して endpoint として扱う - var secure = false - u, err := url.Parse(endpoint) - // エラーがあっても無視してそのまま文字列として扱う - // エラーがないときだけ scheme チェックする - if err == nil { - switch u.Scheme { - case "http": - return u.Host, secure - case "https": - // https なので secure を true にする - secure = true - return u.Host, secure - case "": - // scheme なしの場合は secure を true にする - secure = true - return endpoint, secure - default: - // サポート外の scheme の場合はタダの文字列として扱う - } - } - return endpoint, secure -} - -func newS3Client(endpoint string, credentials *credentials.Credentials) (*minio.Client, error) { - newEndpoint, secure := maybeEndpointURL(endpoint) - return minio.New( - newEndpoint, - &minio.Options{ - Creds: credentials, - Secure: secure, - }) -} diff --git a/s3/s3.go b/s3/s3.go new file mode 100644 index 0000000..c682aef --- /dev/null +++ b/s3/s3.go @@ -0,0 +1,53 @@ +package s3 + +import ( + "net/url" + + "github.com/minio/minio-go/v7" + "github.com/minio/minio-go/v7/pkg/credentials" +) + +// config ではなくこちらにまとめる +type S3CompatibleObjectStorage struct { + Endpoint string + AccessKeyID string + SecretAccessKey string + BucketName string +} + +func maybeEndpointURL(endpoint string) (string, bool) { + // もし endpoint に指定されたのが endpoint_url だった場合、 + // scheme をチェックして http ならば secure = false にする + // さらに host だけを取り出して endpoint として扱う + var secure = false + u, err := url.Parse(endpoint) + // エラーがあっても無視してそのまま文字列として扱う + // エラーがないときだけ scheme チェックする + if err == nil { + switch u.Scheme { + case "http": + return u.Host, secure + case "https": + // https なので secure を true にする + secure = true + return u.Host, secure + case "": + // scheme なしの場合は secure を true にする + secure = true + return endpoint, secure + default: + // サポート外の scheme の場合はタダの文字列として扱う + } + } + return endpoint, secure +} + +func NewClient(endpoint string, credentials *credentials.Credentials) (*minio.Client, error) { + newEndpoint, secure := maybeEndpointURL(endpoint) + return minio.New( + newEndpoint, + &minio.Options{ + Creds: credentials, + Secure: secure, + }) +} diff --git a/s3_test.go b/s3/s3_test.go similarity index 97% rename from s3_test.go rename to s3/s3_test.go index cd36b23..55e8d70 100644 --- a/s3_test.go +++ b/s3/s3_test.go @@ -1,4 +1,4 @@ -package archive +package s3 import ( "testing" diff --git a/uploader.go b/uploader.go index 10636f6..eeda33a 100644 --- a/uploader.go +++ b/uploader.go @@ -10,6 +10,7 @@ import ( "time" "github.com/google/uuid" + "github.com/shiguredo/sora-archive-uploader/s3" base32 "github.com/shogo82148/go-clockwork-base32" zlog "github.com/rs/zerolog/log" @@ -244,7 +245,7 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { // metadata ファイル (json) をアップロード metadataFilename := fileInfo.Name() metadataObjectKey := fmt.Sprintf("%s/%s", am.RecordingID, metadataFilename) - osConfig := &S3CompatibleObjectStorage{ + osConfig := &s3.S3CompatibleObjectStorage{ Endpoint: u.config.ObjectStorageEndpoint, BucketName: u.config.ObjectStorageBucketName, AccessKeyID: u.config.ObjectStorageAccessKeyID, @@ -405,7 +406,7 @@ func (u Uploader) handleReport(reportJSONFilePath string) bool { // report ファイル (json) をアップロード filename := fileInfo.Name() reportObjectKey := fmt.Sprintf("%s/%s", rr.RecordingID, filename) - osConfig := &S3CompatibleObjectStorage{ + osConfig := &s3.S3CompatibleObjectStorage{ Endpoint: u.config.ObjectStorageEndpoint, BucketName: u.config.ObjectStorageBucketName, AccessKeyID: u.config.ObjectStorageAccessKeyID, @@ -534,7 +535,7 @@ func (u Uploader) handleArchiveEnd(archiveEndJSONFilePath string) bool { // metadata ファイル (json) をアップロード filename := fileInfo.Name() objectKey := fmt.Sprintf("%s/%s", aem.RecordingID, filename) - osConfig := &S3CompatibleObjectStorage{ + osConfig := &s3.S3CompatibleObjectStorage{ Endpoint: u.config.ObjectStorageEndpoint, BucketName: u.config.ObjectStorageBucketName, AccessKeyID: u.config.ObjectStorageAccessKeyID, From 32e9751155f85e58982dc785843c282f28634a43 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 9 May 2023 10:14:14 +0900 Subject: [PATCH 23/79] =?UTF-8?q?=E3=83=AA=E3=83=95=E3=82=A1=E3=82=AF?= =?UTF-8?q?=E3=82=BF=E3=83=AA=E3=83=B3=E3=82=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- renovate.json => .github/renovate.json5 | 0 .github/workflows/ci.yaml | 30 +++++++++ .gitignore | 4 +- Makefile | 4 +- README.md | 10 +-- config.go | 77 ++++++++++++----------- config.example.toml => config_example.ini | 28 ++++----- gatekeeper.go | 4 +- go.mod | 23 ++++--- go.sum | 27 +++++++- logging.go | 2 +- runner.go | 20 +++--- s3.go | 8 +-- sora-archive-uploader.service | 2 +- staticcheck.conf | 1 + webhook.go | 12 ++-- 16 files changed, 151 insertions(+), 101 deletions(-) rename renovate.json => .github/renovate.json5 (100%) create mode 100644 .github/workflows/ci.yaml rename config.example.toml => config_example.ini (74%) create mode 100644 staticcheck.conf diff --git a/renovate.json b/.github/renovate.json5 similarity index 100% rename from renovate.json rename to .github/renovate.json5 diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml new file mode 100644 index 0000000..5ca22f8 --- /dev/null +++ b/.github/workflows/ci.yaml @@ -0,0 +1,30 @@ +name: ci +on: + push: + branches-ignore: + - "master" + tags-ignore: + - "*" + +jobs: + build: + name: ci + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: setup go + uses: actions/setup-go@v4 + with: + go-version-file: ./go.mod + cache: true + cache-dependency-path: ./go.sum + - run: go version + - run: go fmt . + - uses: dominikh/staticcheck-action@v1.3.0 + with: + version: "2023.1.2" + install-go: false + - name: Test + run: make test + - name: Build + run: make diff --git a/.gitignore b/.gitignore index 53a8a5e..dc62d1b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,3 @@ bin/* -config.toml -*.jsonl \ No newline at end of file +config.ini +*.jsonl diff --git a/Makefile b/Makefile index f3c1139..4d1648d 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -VERSION := 2022.1.0 +VERSION := 2023.1.0 REVISION := $(shell git rev-parse --short HEAD) BUILD_DATE := $(shell date -u "+%Y-%m-%dT%H:%M:%SZ") LDFLAGS := "-X main.version=$(VERSION) -X main.revision=$(REVISION) -X main.buildDate=$(BUILD_DATE)" @@ -17,4 +17,4 @@ sora-archive-uploader-prod: cmd/sora-archive-uploader/main.go go build -ldflags $(LDFLAGS_PROD) -o bin/$@ $< test: - go test -v + go test -v ./s3 diff --git a/README.md b/README.md index aa8ff78..94c404a 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,9 @@ # Sora Archive Uploader + [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) + ## About Shiguredo's open source software @@ -46,17 +48,17 @@ Sora Cloud では出力されたファイルをオブジェクトストレージ ## まずは使ってみる -config.toml に必要な情報を設定してください。 +config.ini に必要な情報を設定してください。 ```console -$ cp config.example.com config.toml +$ cp config_example.ini config.ini ``` make でビルドして実行します。 ```console $ make -$ ./bin/sora-archive-uploader-dev -C config.toml +$ ./bin/sora-archive-uploader-dev -C config.ini ``` ## Discord @@ -71,7 +73,7 @@ https://discord.gg/shiguredo - オープンソースでの公開が前提 - 可能であれば企業名の公開 - - 公開が難しい場合は `企業名非公開` と書かせていただきます + - 公開が難しい場合は `企業名非公開` と書かせていただきます ### 機能 diff --git a/config.go b/config.go index 4cb93b4..b304bd7 100644 --- a/config.go +++ b/config.go @@ -1,56 +1,57 @@ package archive -import ( - "github.com/BurntSushi/toml" -) +import "gopkg.in/ini.v1" type Config struct { - Debug bool `toml:"debug"` + Debug bool `ini:"debug"` - LogDir string `toml:"log_dir"` - LogName string `toml:"log_name"` - LogStdOut bool `toml:"log_std_out"` - LogRotateMaxSize int `toml:"log_rotate_max_size"` - LogRotateMaxBackups int `toml:"log_rotate_max_backups"` - LogRotateMaxAge int `toml:"log_rotate_max_age"` - LogRotateCompress bool `toml:"log_rotate_compress"` + LogDir string `ini:"log_dir"` + LogName string `ini:"log_name"` + LogStdOut bool `ini:"log_std_out"` + LogRotateMaxSize int `ini:"log_rotate_max_size"` + LogRotateMaxBackups int `ini:"log_rotate_max_backups"` + LogRotateMaxAge int `ini:"log_rotate_max_age"` + LogRotateCompress bool `ini:"log_rotate_compress"` - ObjectStorageEndpoint string `toml:"object_storage_endpoint"` - ObjectStorageBucketName string `toml:"object_storage_bucket_name"` - ObjectStorageAccessKeyID string `toml:"object_storage_access_key_id"` - ObjectStorageSecretAccessKey string `toml:"object_storage_secret_access_key"` + ObjectStorageEndpoint string `ini:"object_storage_endpoint"` + ObjectStorageBucketName string `ini:"object_storage_bucket_name"` + ObjectStorageAccessKeyID string `ini:"object_storage_access_key_id"` + ObjectStorageSecretAccessKey string `ini:"object_storage_secret_access_key"` - SoraArchiveDirFullPath string `toml:"archive_dir_full_path"` - SoraEvacuateDirFullPath string `toml:"evacuate_dir_full_path"` + SoraArchiveDirFullPath string `ini:"archive_dir_full_path"` + SoraEvacuateDirFullPath string `ini:"evacuate_dir_full_path"` - UploadWorkers int `toml:"upload_workers"` + UploadWorkers int `ini:"upload_workers"` - UploadedFileCacheSize int `toml:"uploaded_file_cache_size"` + UploadedFileCacheSize int `ini:"uploaded_file_cache_size"` - WebhookEndpointURL string `toml:"webhook_endpoint_url"` - WebhookEndpointHealthCheckURL string `toml:"webhook_endpoint_health_check_url"` + WebhookEndpointURL string `ini:"webhook_endpoint_url"` + WebhookEndpointHealthCheckURL string `ini:"webhook_endpoint_health_check_url"` - WebhookTypeHeaderName string `toml:"webhook_type_header_name"` - WebhookTypeArchiveUploaded string `toml:"webhook_type_archive_uploaded"` - WebhookTypeSplitArchiveUploaded string `toml:"webhook_type_split_archive_uploaded"` - WebhookTypeSplitArchiveEndUploaded string `toml:"webhook_type_split_archive_end_uploaded"` - WebhookTypeReportUploaded string `toml:"webhook_type_report_uploaded"` + WebhookTypeHeaderName string `ini:"webhook_type_header_name"` + WebhookTypeArchiveUploaded string `ini:"webhook_type_archive_uploaded"` + WebhookTypeSplitArchiveUploaded string `ini:"webhook_type_split_archive_uploaded"` + WebhookTypeSplitArchiveEndUploaded string `ini:"webhook_type_split_archive_end_uploaded"` + WebhookTypeReportUploaded string `ini:"webhook_type_report_uploaded"` - WebhookBasicAuthUsername string `toml:"webhook_basic_auth_username"` - WebhookBasicAuthPassword string `toml:"webhook_basic_auth_password"` + WebhookBasicAuthUsername string `ini:"webhook_basic_auth_username"` + WebhookBasicAuthPassword string `ini:"webhook_basic_auth_password"` - WebhookRequestTimeoutS int32 `toml:"webhook_request_timeout_s"` + WebhookRequestTimeoutS int32 `ini:"webhook_request_timeout_s"` - WebhookTlsVerifyCacertPath string `toml:"webhook_tls_verify_cacert_path"` - WebhookTlsFullchainPath string `toml:"webhook_tls_fullchain_path"` - WebhookTlsPrivkeyPath string `toml:"webhook_tls_privkey_path"` + WebhookTLSVerifyCacertPath string `ini:"webhook_tls_verify_cacert_path"` + WebhookTLSFullchainPath string `ini:"webhook_tls_fullchain_path"` + WebhookTLSPrivkeyPath string `ini:"webhook_tls_privkey_path"` } -func initConfig(data []byte, config interface{}) error { - if err := toml.Unmarshal(data, config); err != nil { - return err +func newConfig(configFilePath string) (*Config, error) { + config := new(Config) + iniConfig, err := ini.InsensitiveLoad(configFilePath) + if err != nil { + return nil, err } - - // TODO: 初期値 - return nil + if err := iniConfig.StrictMapTo(config); err != nil { + return nil, err + } + return config, nil } diff --git a/config.example.toml b/config_example.ini similarity index 74% rename from config.example.toml rename to config_example.ini index 07893aa..77fdbf5 100644 --- a/config.example.toml +++ b/config_example.ini @@ -1,9 +1,9 @@ debug = false # Sora の録画アーカイブディレクトリのフルパス -archive_dir_full_path = "" +# archive_dir_full_path = /path/to/archive # アップロードに失敗した際の待避ディレクトリのフルパス -evacuate_dir_full_path = "" +# evacuate_dir_full_path = /path/to/evacuate # 同時アップロード数 upload_workers = 4 @@ -16,8 +16,8 @@ uploaded_file_cache_size = 32 # [log] -log_dir = "." -log_name = "sora-archive-uploader.jsonl" +log_dir = . +log_name = sora-archive-uploader.jsonl log_std_out = true # MB @@ -30,16 +30,16 @@ log_rotate_compress = false # [object_storage] # アップロード先の S3 または S3 互換オブジェクトストレージの設定 -object_storage_endpoint = "" -object_storage_bucket_name = "" -object_storage_access_key_id = "" -object_storage_secret_access_key = "" +# object_storage_endpoint = https://s3.example.com +# object_storage_bucket_name = bucket-name +# object_storage_access_key_id = access-key-id +# object_storage_secret_access_key = secret-access-key # オブジェクトストレージにアップロードが完了した際に通知するウェブフック # [webhook] # 空文字列の場合はウェブフックは飛ばさない -webhook_endpoint_url = "" +# webhook_endpoint_url = https://example.com/webhook # ウェブフックリクエストのタイムアウト時間 (秒) webhook_request_timeout_s = 30 @@ -53,12 +53,12 @@ webhook_type_report_uploaded = "recording-report.uploaded" # ウェブフックのベーシック認証 # 空文字はベーシック認証を行わない -webhook_basic_auth_username = "" -webhook_basic_auth_password = "" +# webhook_basic_auth_username = username +# webhook_basic_auth_password = password # webhook で HTTPS を利用する場合にサーバーの証明書をベリファイする場合に指定 # 指定しない場合は OS のものを利用し、サーバー名までは検証しません -webhook_tls_verify_cacert_path = "" +# webhook_tls_verify_cacert_path = /path/to/cacert.pem # webhook で mTLS を利用する場合に指定します -webhook_tls_fullchain_path = "" -webhook_tls_privkey_path = "" \ No newline at end of file +# webhook_tls_fullchain_path = /path/to/fullchain.pem +# webhook_tls_privkey_path = /path/to/privkey.pem diff --git a/gatekeeper.go b/gatekeeper.go index 3d1849c..459a740 100644 --- a/gatekeeper.go +++ b/gatekeeper.go @@ -32,7 +32,7 @@ func (ru *RecordingUnit) run() { zlog.Debug(). Str("recording_id", ru.recordingID). Int32("counter", ru.counter). - Msgf("RUN-RECORDING-UNIT") + Msg("RUN-RECORDING-UNIT") } func (ru *RecordingUnit) done() { @@ -40,7 +40,7 @@ func (ru *RecordingUnit) done() { zlog.Debug(). Str("recording_id", ru.recordingID). Int32("counter", ru.counter). - Msgf("DONE-RECORDING-UNIT") + Msg("DONE-RECORDING-UNIT") } func (ru *RecordingUnit) canProcessAndSetReportFile(reportFile string) bool { diff --git a/go.mod b/go.mod index 2474243..22f1874 100644 --- a/go.mod +++ b/go.mod @@ -1,36 +1,35 @@ module github.com/shiguredo/sora-archive-uploader -go 1.19 +go 1.20 require ( - github.com/BurntSushi/toml v1.2.1 github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.52 - github.com/rs/zerolog v1.29.0 + github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 github.com/stretchr/testify v1.8.2 + gopkg.in/ini.v1 v1.67.0 ) require ( github.com/davecgh/go-spew v1.1.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.16.0 // indirect + github.com/klauspost/compress v1.16.5 // indirect github.com/klauspost/cpuid/v2 v2.2.4 // indirect - github.com/mattn/go-colorable v0.1.12 // indirect - github.com/mattn/go-isatty v0.0.14 // indirect + github.com/mattn/go-colorable v0.1.13 // indirect + github.com/mattn/go-isatty v0.0.18 // indirect github.com/minio/md5-simd v1.1.2 // indirect github.com/minio/sha256-simd v1.0.0 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect - github.com/rs/xid v1.4.0 // indirect + github.com/rs/xid v1.5.0 // indirect github.com/sirupsen/logrus v1.9.0 // indirect - golang.org/x/crypto v0.6.0 // indirect - golang.org/x/net v0.7.0 // indirect - golang.org/x/sys v0.5.0 // indirect - golang.org/x/text v0.7.0 // indirect - gopkg.in/ini.v1 v1.67.0 // indirect + golang.org/x/crypto v0.8.0 // indirect + golang.org/x/net v0.9.0 // indirect + golang.org/x/sys v0.8.0 // indirect + golang.org/x/text v0.9.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/go.sum b/go.sum index 33ac175..bae4423 100644 --- a/go.sum +++ b/go.sum @@ -1,6 +1,5 @@ -github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak= -github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= +github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -14,14 +13,22 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/klauspost/compress v1.16.0 h1:iULayQNOReoYUe+1qtKOqw9CwJv3aNQu8ivo7lw1HU4= github.com/klauspost/compress v1.16.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= +github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI= +github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk= github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= +github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= +github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= +github.com/mattn/go-isatty v0.0.16 h1:bq3VjFmv/sOjHtdEhmkEV4x1AJtvUvOJ2PFAZ5+peKQ= +github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= +github.com/mattn/go-isatty v0.0.18 h1:DOKFKCQ7FNG2L1rbrmstDN4QVRdS89Nkh85u68Uwp98= +github.com/mattn/go-isatty v0.0.18/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= github.com/minio/minio-go/v7 v7.0.52 h1:8XhG36F6oKQUDDSuz6dY3rioMzovKjW40W6ANuN0Dps= @@ -38,8 +45,12 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY= github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= +github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc= +github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= github.com/rs/zerolog v1.29.0 h1:Zes4hju04hjbvkVkOhdl2HpZa+0PmVwigmo8XoORE5w= github.com/rs/zerolog v1.29.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= +github.com/rs/zerolog v1.29.1 h1:cO+d60CHkknCbvzEWxP0S9K6KqyTjrCNUy1LdQLCGPc= +github.com/rs/zerolog v1.29.1/go.mod h1:Le6ESbR7hc+DP6Lt1THiV8CQSdkkNrd3R0XbEgp3ZBU= github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= @@ -57,16 +68,28 @@ github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc= golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= +golang.org/x/crypto v0.8.0 h1:pd9TJtTueMTVQXzk8E2XESSMQDj/U7OUu0PqJqPXQjQ= +golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.9.0 h1:aWJ/m6xSmxWBx+V0XRHTlrYrPG56jKsLdTFmsSsCzOM= +golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= +golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= diff --git a/logging.go b/logging.go index f261919..fc1f530 100644 --- a/logging.go +++ b/logging.go @@ -12,7 +12,7 @@ import ( "github.com/shiguredo/lumberjack/v3" ) -func initLogger(config Config) error { +func initLogger(config *Config) error { if f, err := os.Stat(config.LogDir); os.IsNotExist(err) || !f.IsDir() { return err } diff --git a/runner.go b/runner.go index 9638734..d177fa2 100644 --- a/runner.go +++ b/runner.go @@ -12,10 +12,10 @@ import ( ) type Main struct { - config Config + config *Config } -func newMain(config Config) *Main { +func newMain(config *Config) *Main { return &Main{ config: config, } @@ -59,11 +59,11 @@ func (m *Main) run(ctx context.Context, cancel context.CancelFunc) error { } processContext, processContextCancel := context.WithCancel(context.Background()) - gateKeeper := newGateKeeper(&m.config) + gateKeeper := newGateKeeper(m.config) recordingFileStream := gateKeeper.run(processContext, foundFiles) uploaderManager := newUploaderManager() - _, err = uploaderManager.run(processContext, &m.config, recordingFileStream) + _, err = uploaderManager.run(processContext, m.config, recordingFileStream) if err != nil { processContextCancel() return err @@ -120,15 +120,9 @@ func (m *Main) run(ctx context.Context, cancel context.CancelFunc) error { } func Run(configFilePath *string) { - buf, err := os.ReadFile(*configFilePath) + // INI をパース + config, err := newConfig(*configFilePath) if err != nil { - // 読み込めない場合 Fatal で終了 - log.Fatal("cannot open config file, err=", err) - } - - // toml をパース - var config Config - if err := initConfig(buf, &config); err != nil { // パースに失敗した場合 Fatal で終了 log.Fatal("cannot parse config file, err=", err) } @@ -142,7 +136,7 @@ func Run(configFilePath *string) { // もしあれば mTLS の設定確認と Webhook のヘルスチェック if config.WebhookEndpointHealthCheckURL != "" { - client, err := createHttpClient(&config) + client, err := createHTTPClient(config) if err != nil { zlog.Fatal().Err(err).Msg("FAILED-CREATE-RPC-CLIENT") } diff --git a/s3.go b/s3.go index 7ab4d2b..5dd176d 100644 --- a/s3.go +++ b/s3.go @@ -58,8 +58,8 @@ func uploadJSONFile( fmt.Sprintf("attachment; filename=\"%s\"", filename), ) - objectUrl := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) - return objectUrl, nil + objectURL := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) + return objectURL, nil } func uploadWebMFile(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, dst, filePath string) (string, error) { @@ -105,8 +105,8 @@ func uploadWebMFile(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, fmt.Sprintf("attachment; filename=\"%s\"", filename), ) - objectUrl := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) - return objectUrl, nil + objectURL := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) + return objectURL, nil } // minio のエラーをレスポンスに復元して、リトライするためファイルを残すか対象のファイルを削除するか判断する diff --git a/sora-archive-uploader.service b/sora-archive-uploader.service index 1dc1ada..01c4df8 100644 --- a/sora-archive-uploader.service +++ b/sora-archive-uploader.service @@ -15,7 +15,7 @@ WorkingDirectory=/home/sora/sora-archive-uploader ExecStartPre=/bin/mkdir -p /var/log/sora-archive-uploader ExecStartPre=/bin/chown -R sora:sora /var/log/sora-archive-uploader -ExecStart=/home/sora/sora-archive-uploader/bin/sora-archive-uploader-prod -C /home/sora/sora-archive-uploader/config.toml +ExecStart=/home/sora/sora-archive-uploader/bin/sora-archive-uploader-prod -C /home/sora/sora-archive-uploader/config.ini [Install] WantedBy=multi-user.target \ No newline at end of file diff --git a/staticcheck.conf b/staticcheck.conf new file mode 100644 index 0000000..5f0a348 --- /dev/null +++ b/staticcheck.conf @@ -0,0 +1 @@ +checks = ["all", "-ST1000"] diff --git a/webhook.go b/webhook.go index 4601bca..d7108da 100644 --- a/webhook.go +++ b/webhook.go @@ -50,14 +50,14 @@ type WebhookArchiveEndUploaded struct { } // mTLS を組み込んだ http.Client を構築する -func createHttpClient(config *Config) (*http.Client, error) { +func createHTTPClient(config *Config) (*http.Client, error) { e, err := url.Parse(config.WebhookEndpointURL) if err != nil { return nil, err } // http または VerifyCacertPath 指定していない場合はそのまま投げる - if e.Scheme != "https" || config.WebhookTlsVerifyCacertPath == "" { + if e.Scheme != "https" || config.WebhookTLSVerifyCacertPath == "" { client := &http.Client{ CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse @@ -68,7 +68,7 @@ func createHttpClient(config *Config) (*http.Client, error) { return client, nil } - CaCert, err := os.ReadFile(config.WebhookTlsVerifyCacertPath) + CaCert, err := os.ReadFile(config.WebhookTLSVerifyCacertPath) if err != nil { return nil, err } @@ -76,8 +76,8 @@ func createHttpClient(config *Config) (*http.Client, error) { caCertPool.AppendCertsFromPEM(CaCert) var certificates []tls.Certificate - if config.WebhookTlsFullchainPath != "" && config.WebhookTlsPrivkeyPath != "" { - pair, err := tls.LoadX509KeyPair(config.WebhookTlsFullchainPath, config.WebhookTlsPrivkeyPath) + if config.WebhookTLSFullchainPath != "" && config.WebhookTLSPrivkeyPath != "" { + pair, err := tls.LoadX509KeyPair(config.WebhookTLSFullchainPath, config.WebhookTLSPrivkeyPath) if err != nil { return nil, err } @@ -132,7 +132,7 @@ func (u Uploader) httpClientDo(client *http.Client, webhookType string, buf []by } func (u Uploader) postWebhook(webhookType string, buf []byte) error { - client, err := createHttpClient(u.config) + client, err := createHTTPClient(u.config) if err != nil { return err } From 76cc5f12e521fb348f10a4c39f430ee817424c4f Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 9 May 2023 10:21:50 +0900 Subject: [PATCH 24/79] =?UTF-8?q?=E4=B8=8D=E8=A6=81=E3=81=AA=E3=82=B3?= =?UTF-8?q?=E3=83=A1=E3=83=B3=E3=83=88=E5=89=8A=E9=99=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3/s3.go | 1 - 1 file changed, 1 deletion(-) diff --git a/s3/s3.go b/s3/s3.go index c682aef..612bec0 100644 --- a/s3/s3.go +++ b/s3/s3.go @@ -7,7 +7,6 @@ import ( "github.com/minio/minio-go/v7/pkg/credentials" ) -// config ではなくこちらにまとめる type S3CompatibleObjectStorage struct { Endpoint string AccessKeyID string From 1351d17072bc5d15bb3b7951e01e5082e4a5ec33 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 9 May 2023 10:45:12 +0900 Subject: [PATCH 25/79] =?UTF-8?q?workflow=20=E6=95=B4=E7=90=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/workflows/release.yml | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 .github/workflows/release.yml diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..1b03fae --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,30 @@ +name: release + +on: + push: + tags: + - "*" + +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Set up Go + uses: actions/setup-go@v4 + with: + go-version: "^1.20" + + - run: go install github.com/tcnksm/ghr@latest + + - name: Build + run: | + GOOS=linux GOARCH=amd64 go build -o dist/sora-archive-uploader_linux_amd64 cmd/sora-archive-uploader/main.go + GOOS=darwin GOARCH=amd64 go build -o dist/sora-archive-uploader_darwin_amd64 cmd/sora-archive-uploader/main.go + GOOS=darwin GOARCH=arm64 go build -o dist/sora-archive-uploader_darwin_arm64 cmd/sora-archive-uploader/main.go + gzip dist/* + + - name: Release + run: | + ghr -t "${{ secrets.GITHUB_TOKEN }}" -u "${{ github.repository_owner }}" -r "sora-archive-uploader" --replace "${GITHUB_REF##*/}" dist/ From fcaf523e1fc03fc0bebfe2c6cc54b10917331b97 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 9 May 2023 10:46:17 +0900 Subject: [PATCH 26/79] =?UTF-8?q?script/=20=E3=81=AB=E7=A7=BB=E5=8B=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../sora-archive-uploader.service | 0 sora-archive-uploader.timer => script/sora-archive-uploader.timer | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename sora-archive-uploader.service => script/sora-archive-uploader.service (100%) rename sora-archive-uploader.timer => script/sora-archive-uploader.timer (100%) diff --git a/sora-archive-uploader.service b/script/sora-archive-uploader.service similarity index 100% rename from sora-archive-uploader.service rename to script/sora-archive-uploader.service diff --git a/sora-archive-uploader.timer b/script/sora-archive-uploader.timer similarity index 100% rename from sora-archive-uploader.timer rename to script/sora-archive-uploader.timer From 16ccbe8ac0a0c8a5b507e772074bef1ccd615c20 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 9 May 2023 14:18:19 +0900 Subject: [PATCH 27/79] =?UTF-8?q?renovate=20=E6=9B=B4=E6=96=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/renovate.json5 | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/.github/renovate.json5 b/.github/renovate.json5 index 5474339..c960bfa 100644 --- a/.github/renovate.json5 +++ b/.github/renovate.json5 @@ -1,17 +1,19 @@ { "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ - "config:base" + "config:base", + ":timezone(Asia/Tokyo)", + ":combinePatchMinorReleases", + ":prHourlyLimitNone", + ":prConcurrentLimit10", + "group:recommended", + "group:allNonMajor", + "schedule:weekly" ], "dependencyDashboard": false, "packageRules": [ { - "matchUpdateTypes": [ - "minor", - "patch", - "pin", - "digest" - ], + "matchUpdateTypes": ["minor", "patch", "pin", "digest"], "platformAutomerge": true, "automerge": true } From 88aaa3e64f5b3c4aab346cf32134298cd7cd3bcb Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Sat, 20 May 2023 21:13:10 +0000 Subject: [PATCH 28/79] Update module github.com/minio/minio-go/v7 to v7.0.53 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index c16c085..3cc92d0 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.2.1 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.52 + github.com/minio/minio-go/v7 v7.0.53 github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 diff --git a/go.sum b/go.sum index 9b946d3..6d1d71f 100644 --- a/go.sum +++ b/go.sum @@ -44,6 +44,8 @@ github.com/minio/minio-go/v7 v7.0.51 h1:eSewrwc23TqUDEH8aw8Bwp4f+JDdozRrPWcKR7DZ github.com/minio/minio-go/v7 v7.0.51/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/minio-go/v7 v7.0.52 h1:8XhG36F6oKQUDDSuz6dY3rioMzovKjW40W6ANuN0Dps= github.com/minio/minio-go/v7 v7.0.52/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= +github.com/minio/minio-go/v7 v7.0.53 h1:qtPyQ+b0Cc1ums3LsnVMAYULPNdAGz8qdX8R2zl9XMU= +github.com/minio/minio-go/v7 v7.0.53/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= From 42748e1a21579c6e3afb02efa95a0426fff73921 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Tue, 30 May 2023 17:35:48 +0000 Subject: [PATCH 29/79] Update module github.com/BurntSushi/toml to v1.3.0 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 3cc92d0..a0dbbdb 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/shiguredo/sora-archive-uploader go 1.19 require ( - github.com/BurntSushi/toml v1.2.1 + github.com/BurntSushi/toml v1.3.0 github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.53 github.com/rs/zerolog v1.29.0 diff --git a/go.sum b/go.sum index 6d1d71f..71d2fb6 100644 --- a/go.sum +++ b/go.sum @@ -1,5 +1,7 @@ github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak= github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= +github.com/BurntSushi/toml v1.3.0 h1:Ws8e5YmnrGEHzZEzg0YvK/7COGYtTC5PbaH9oSSbgfA= +github.com/BurntSushi/toml v1.3.0/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= From b97d8e2fede23e5b5d0654ff9b562f1cf24bb63a Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Tue, 6 Jun 2023 12:12:43 +0000 Subject: [PATCH 30/79] Update module github.com/BurntSushi/toml to v1.3.1 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index a0dbbdb..77aca26 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/shiguredo/sora-archive-uploader go 1.19 require ( - github.com/BurntSushi/toml v1.3.0 + github.com/BurntSushi/toml v1.3.1 github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.53 github.com/rs/zerolog v1.29.0 diff --git a/go.sum b/go.sum index 71d2fb6..b72c547 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,8 @@ github.com/BurntSushi/toml v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/BurntSushi/toml v1.3.0 h1:Ws8e5YmnrGEHzZEzg0YvK/7COGYtTC5PbaH9oSSbgfA= github.com/BurntSushi/toml v1.3.0/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= +github.com/BurntSushi/toml v1.3.1 h1:rHnDkSK+/g6DlREUK73PkmIs60pqrnuduK+JmP++JmU= +github.com/BurntSushi/toml v1.3.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= From 94440f80b5c82b66c86e883a2d267ab5e58bcc79 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 8 Jun 2023 12:28:50 +0000 Subject: [PATCH 31/79] Update module github.com/BurntSushi/toml to v1.3.2 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 77aca26..c7d02f3 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/shiguredo/sora-archive-uploader go 1.19 require ( - github.com/BurntSushi/toml v1.3.1 + github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.53 github.com/rs/zerolog v1.29.0 diff --git a/go.sum b/go.sum index b72c547..485d406 100644 --- a/go.sum +++ b/go.sum @@ -4,6 +4,8 @@ github.com/BurntSushi/toml v1.3.0 h1:Ws8e5YmnrGEHzZEzg0YvK/7COGYtTC5PbaH9oSSbgfA github.com/BurntSushi/toml v1.3.0/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/BurntSushi/toml v1.3.1 h1:rHnDkSK+/g6DlREUK73PkmIs60pqrnuduK+JmP++JmU= github.com/BurntSushi/toml v1.3.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= +github.com/BurntSushi/toml v1.3.2 h1:o7IhLm0Msx3BaB+n3Ag7L8EVlByGnpq14C4YWiu/gL8= +github.com/BurntSushi/toml v1.3.2/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= From 2977430b9c57097460a19b8a269cd954fb093852 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 8 Jun 2023 16:28:36 +0000 Subject: [PATCH 32/79] Update module github.com/rs/zerolog to v1.29.1 --- go.mod | 2 +- go.sum | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index c7d02f3..b9e7ed7 100644 --- a/go.mod +++ b/go.mod @@ -6,7 +6,7 @@ require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.53 - github.com/rs/zerolog v1.29.0 + github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 github.com/stretchr/testify v1.8.2 diff --git a/go.sum b/go.sum index 485d406..3dd7be6 100644 --- a/go.sum +++ b/go.sum @@ -7,6 +7,7 @@ github.com/BurntSushi/toml v1.3.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbi github.com/BurntSushi/toml v1.3.2 h1:o7IhLm0Msx3BaB+n3Ag7L8EVlByGnpq14C4YWiu/gL8= github.com/BurntSushi/toml v1.3.2/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= +github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -66,6 +67,8 @@ github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY= github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= github.com/rs/zerolog v1.29.0 h1:Zes4hju04hjbvkVkOhdl2HpZa+0PmVwigmo8XoORE5w= github.com/rs/zerolog v1.29.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= +github.com/rs/zerolog v1.29.1 h1:cO+d60CHkknCbvzEWxP0S9K6KqyTjrCNUy1LdQLCGPc= +github.com/rs/zerolog v1.29.1/go.mod h1:Le6ESbR7hc+DP6Lt1THiV8CQSdkkNrd3R0XbEgp3ZBU= github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= From 224e770aac287d183ff98fb6dfabea512de8b8fd Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 8 Jun 2023 16:28:50 +0000 Subject: [PATCH 33/79] Update module github.com/stretchr/testify to v1.8.4 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index c7d02f3..3dd99b4 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,7 @@ require ( github.com/rs/zerolog v1.29.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 - github.com/stretchr/testify v1.8.2 + github.com/stretchr/testify v1.8.4 ) require ( diff --git a/go.sum b/go.sum index 485d406..47a3f83 100644 --- a/go.sum +++ b/go.sum @@ -83,6 +83,8 @@ github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKs github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJbGPo6sKVl54f/TVDObg1c= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc= From d5394b402fed76c31eaaf0a4ec043825a95fe102 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Fri, 9 Jun 2023 04:37:56 +0000 Subject: [PATCH 34/79] Update module github.com/minio/minio-go/v7 to v7.0.56 --- go.mod | 18 +++++++++--------- go.sum | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index 46d4826..ce01b13 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.53 + github.com/minio/minio-go/v7 v7.0.56 github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 @@ -21,19 +21,19 @@ require ( require ( github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.16.0 // indirect + github.com/klauspost/compress v1.16.5 // indirect github.com/klauspost/cpuid/v2 v2.2.4 // indirect github.com/mattn/go-colorable v0.1.12 // indirect github.com/mattn/go-isatty v0.0.14 // indirect github.com/minio/md5-simd v1.1.2 // indirect - github.com/minio/sha256-simd v1.0.0 // indirect + github.com/minio/sha256-simd v1.0.1 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect - github.com/rs/xid v1.4.0 // indirect - github.com/sirupsen/logrus v1.9.0 // indirect - golang.org/x/crypto v0.6.0 // indirect - golang.org/x/net v0.7.0 // indirect - golang.org/x/sys v0.5.0 // indirect - golang.org/x/text v0.7.0 // indirect + github.com/rs/xid v1.5.0 // indirect + github.com/sirupsen/logrus v1.9.2 // indirect + golang.org/x/crypto v0.9.0 // indirect + golang.org/x/net v0.10.0 // indirect + golang.org/x/sys v0.8.0 // indirect + golang.org/x/text v0.9.0 // indirect gopkg.in/ini.v1 v1.67.0 // indirect ) diff --git a/go.sum b/go.sum index 3d629c4..8a32e26 100644 --- a/go.sum +++ b/go.sum @@ -27,6 +27,8 @@ github.com/klauspost/compress v1.15.15 h1:EF27CXIuDsYJ6mmvtBRlEuB2UVOqHG1tAXgZ7y github.com/klauspost/compress v1.15.15/go.mod h1:ZcK2JAFqKOpnBlxcLsJzYfrS9X1akm9fHZNnD9+Vo/4= github.com/klauspost/compress v1.16.0 h1:iULayQNOReoYUe+1qtKOqw9CwJv3aNQu8ivo7lw1HU4= github.com/klauspost/compress v1.16.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= +github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI= +github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0= @@ -53,8 +55,12 @@ github.com/minio/minio-go/v7 v7.0.52 h1:8XhG36F6oKQUDDSuz6dY3rioMzovKjW40W6ANuN0 github.com/minio/minio-go/v7 v7.0.52/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/minio-go/v7 v7.0.53 h1:qtPyQ+b0Cc1ums3LsnVMAYULPNdAGz8qdX8R2zl9XMU= github.com/minio/minio-go/v7 v7.0.53/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= +github.com/minio/minio-go/v7 v7.0.56 h1:pkZplIEHu8vinjkmhsexcXpWth2tjVLphrTZx6fBVZY= +github.com/minio/minio-go/v7 v7.0.56/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= +github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= +github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -65,6 +71,8 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rs/xid v1.4.0 h1:qd7wPTDkN6KQx2VmMBLrpHkiyQwgFXRnkOLacUiaSNY= github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= +github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc= +github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= github.com/rs/zerolog v1.29.0 h1:Zes4hju04hjbvkVkOhdl2HpZa+0PmVwigmo8XoORE5w= github.com/rs/zerolog v1.29.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= github.com/rs/zerolog v1.29.1 h1:cO+d60CHkknCbvzEWxP0S9K6KqyTjrCNUy1LdQLCGPc= @@ -75,6 +83,8 @@ github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yA github.com/shogo82148/go-clockwork-base32 v1.0.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y= +github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= @@ -92,10 +102,14 @@ golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa h1:zuSxTR4o9y82ebqCUJYNGJ golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc= golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= +golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g= +golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0= golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= +golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -104,10 +118,14 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9w golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= +golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= From b39f6a00ca8e98aad88fab35faeac435ae1552cd Mon Sep 17 00:00:00 2001 From: voluntas Date: Fri, 9 Jun 2023 13:41:54 +0900 Subject: [PATCH 35/79] =?UTF-8?q?lumberjack=20=E3=82=92=E6=9C=AC=E5=AE=B6?= =?UTF-8?q?=E3=81=AB=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.mod | 2 +- go.sum | 4 ++-- logging.go | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/go.mod b/go.mod index 79655c1..679ef9b 100644 --- a/go.mod +++ b/go.mod @@ -6,10 +6,10 @@ require ( github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.56 github.com/rs/zerolog v1.29.1 - github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 github.com/stretchr/testify v1.8.4 gopkg.in/ini.v1 v1.67.0 + gopkg.in/natefinch/lumberjack.v2 v2.2.1 ) require ( diff --git a/go.sum b/go.sum index 685300d..459f9b8 100644 --- a/go.sum +++ b/go.sum @@ -41,8 +41,6 @@ github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc= github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= github.com/rs/zerolog v1.29.1 h1:cO+d60CHkknCbvzEWxP0S9K6KqyTjrCNUy1LdQLCGPc= github.com/rs/zerolog v1.29.1/go.mod h1:Le6ESbR7hc+DP6Lt1THiV8CQSdkkNrd3R0XbEgp3ZBU= -github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= -github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= github.com/shogo82148/go-clockwork-base32 v1.0.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y= @@ -70,6 +68,8 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= +gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= +gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/logging.go b/logging.go index fc1f530..6930edf 100644 --- a/logging.go +++ b/logging.go @@ -9,7 +9,7 @@ import ( "github.com/rs/zerolog" "github.com/rs/zerolog/log" - "github.com/shiguredo/lumberjack/v3" + "gopkg.in/natefinch/lumberjack.v2" ) func initLogger(config *Config) error { From ad5c6e37c9a041513a09788e4d18d07423b5fd22 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 15 Jun 2023 22:37:20 +0000 Subject: [PATCH 36/79] Update module github.com/minio/minio-go/v7 to v7.0.57 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index ce01b13..a274b36 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.56 + github.com/minio/minio-go/v7 v7.0.57 github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 diff --git a/go.sum b/go.sum index 8a32e26..0f2288c 100644 --- a/go.sum +++ b/go.sum @@ -57,6 +57,8 @@ github.com/minio/minio-go/v7 v7.0.53 h1:qtPyQ+b0Cc1ums3LsnVMAYULPNdAGz8qdX8R2zl9 github.com/minio/minio-go/v7 v7.0.53/go.mod h1:IbbodHyjUAguneyucUaahv+VMNs/EOTV9du7A7/Z3HU= github.com/minio/minio-go/v7 v7.0.56 h1:pkZplIEHu8vinjkmhsexcXpWth2tjVLphrTZx6fBVZY= github.com/minio/minio-go/v7 v7.0.56/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= +github.com/minio/minio-go/v7 v7.0.57 h1:xsFiOiWjpC1XAGbFEUOzj1/gMXGz7ljfxifwcb/5YXU= +github.com/minio/minio-go/v7 v7.0.57/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= From 1c659329dec3abd136a3d3a88c0d3a929d3da658 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Fri, 23 Jun 2023 10:01:34 +0000 Subject: [PATCH 37/79] Update module github.com/minio/minio-go/v7 to v7.0.58 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index a274b36..f4ee5de 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.57 + github.com/minio/minio-go/v7 v7.0.58 github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 diff --git a/go.sum b/go.sum index 0f2288c..5644c52 100644 --- a/go.sum +++ b/go.sum @@ -59,6 +59,8 @@ github.com/minio/minio-go/v7 v7.0.56 h1:pkZplIEHu8vinjkmhsexcXpWth2tjVLphrTZx6fB github.com/minio/minio-go/v7 v7.0.56/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/minio-go/v7 v7.0.57 h1:xsFiOiWjpC1XAGbFEUOzj1/gMXGz7ljfxifwcb/5YXU= github.com/minio/minio-go/v7 v7.0.57/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= +github.com/minio/minio-go/v7 v7.0.58 h1:B9/8Az8Om/2kX8Ys2ai2PZbBTokRE5W6P5OaqnAs6po= +github.com/minio/minio-go/v7 v7.0.58/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= From 01d0081111324e63f88799c656237dccf22b6ec4 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 29 Jun 2023 07:00:28 +0000 Subject: [PATCH 38/79] Update module github.com/minio/minio-go/v7 to v7.0.59 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index f4ee5de..9425c49 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.58 + github.com/minio/minio-go/v7 v7.0.59 github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 diff --git a/go.sum b/go.sum index 5644c52..0bdac5c 100644 --- a/go.sum +++ b/go.sum @@ -61,6 +61,8 @@ github.com/minio/minio-go/v7 v7.0.57 h1:xsFiOiWjpC1XAGbFEUOzj1/gMXGz7ljfxifwcb/5 github.com/minio/minio-go/v7 v7.0.57/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/minio-go/v7 v7.0.58 h1:B9/8Az8Om/2kX8Ys2ai2PZbBTokRE5W6P5OaqnAs6po= github.com/minio/minio-go/v7 v7.0.58/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= +github.com/minio/minio-go/v7 v7.0.59 h1:lxIXwsTIcQkYoEG25rUJbzpmSB/oWeVDmxFo/uWUUsw= +github.com/minio/minio-go/v7 v7.0.59/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= From 93e8d02b349901717615defd9c369286897e080e Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Tue, 11 Jul 2023 22:34:42 +0000 Subject: [PATCH 39/79] Update module github.com/minio/minio-go/v7 to v7.0.60 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 9425c49..e3d6018 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.59 + github.com/minio/minio-go/v7 v7.0.60 github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.0.0 diff --git a/go.sum b/go.sum index 0bdac5c..b9b18c5 100644 --- a/go.sum +++ b/go.sum @@ -63,6 +63,8 @@ github.com/minio/minio-go/v7 v7.0.58 h1:B9/8Az8Om/2kX8Ys2ai2PZbBTokRE5W6P5OaqnAs github.com/minio/minio-go/v7 v7.0.58/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/minio-go/v7 v7.0.59 h1:lxIXwsTIcQkYoEG25rUJbzpmSB/oWeVDmxFo/uWUUsw= github.com/minio/minio-go/v7 v7.0.59/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= +github.com/minio/minio-go/v7 v7.0.60 h1:iHkrmWyHFs/eZiWc2F/5jAHtNBAFy+HjdhMX6FkkPWc= +github.com/minio/minio-go/v7 v7.0.60/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= From cb3f9dc091c38979bb181e8537f57f55b4c0f3f8 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Mon, 17 Jul 2023 07:53:06 +0000 Subject: [PATCH 40/79] Update module github.com/shogo82148/go-clockwork-base32 to v1.1.0 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index e3d6018..6892081 100644 --- a/go.mod +++ b/go.mod @@ -8,7 +8,7 @@ require ( github.com/minio/minio-go/v7 v7.0.60 github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 - github.com/shogo82148/go-clockwork-base32 v1.0.0 + github.com/shogo82148/go-clockwork-base32 v1.1.0 github.com/stretchr/testify v1.8.4 ) diff --git a/go.sum b/go.sum index b9b18c5..833f821 100644 --- a/go.sum +++ b/go.sum @@ -89,6 +89,8 @@ github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wE github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= github.com/shogo82148/go-clockwork-base32 v1.0.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= +github.com/shogo82148/go-clockwork-base32 v1.1.0 h1:PEqSTiyVEKdl5ar5RHlUQjFJyJTx9XZD5dUVCLG08K0= +github.com/shogo82148/go-clockwork-base32 v1.1.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0= github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y= From 520a314997e4608b0645980773c740e1af253317 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Mon, 17 Jul 2023 17:08:32 +0000 Subject: [PATCH 41/79] Update module github.com/minio/minio-go/v7 to v7.0.61 --- go.mod | 16 ++++++++-------- go.sum | 16 ++++++++++++++++ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/go.mod b/go.mod index 6892081..6f49c4f 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.60 + github.com/minio/minio-go/v7 v7.0.61 github.com/rs/zerolog v1.29.1 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.1.0 @@ -21,8 +21,8 @@ require ( require ( github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.16.5 // indirect - github.com/klauspost/cpuid/v2 v2.2.4 // indirect + github.com/klauspost/compress v1.16.7 // indirect + github.com/klauspost/cpuid/v2 v2.2.5 // indirect github.com/mattn/go-colorable v0.1.12 // indirect github.com/mattn/go-isatty v0.0.14 // indirect github.com/minio/md5-simd v1.1.2 // indirect @@ -30,10 +30,10 @@ require ( github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/rs/xid v1.5.0 // indirect - github.com/sirupsen/logrus v1.9.2 // indirect - golang.org/x/crypto v0.9.0 // indirect - golang.org/x/net v0.10.0 // indirect - golang.org/x/sys v0.8.0 // indirect - golang.org/x/text v0.9.0 // indirect + github.com/sirupsen/logrus v1.9.3 // indirect + golang.org/x/crypto v0.11.0 // indirect + golang.org/x/net v0.12.0 // indirect + golang.org/x/sys v0.10.0 // indirect + golang.org/x/text v0.11.0 // indirect gopkg.in/ini.v1 v1.67.0 // indirect ) diff --git a/go.sum b/go.sum index 833f821..8cecbe1 100644 --- a/go.sum +++ b/go.sum @@ -29,6 +29,8 @@ github.com/klauspost/compress v1.16.0 h1:iULayQNOReoYUe+1qtKOqw9CwJv3aNQu8ivo7lw github.com/klauspost/compress v1.16.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI= github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= +github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I= +github.com/klauspost/compress v1.16.7/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.1.0 h1:eyi1Ad2aNJMW95zcSbmGg7Cg6cq3ADwLpMAP96d8rF0= @@ -37,6 +39,8 @@ github.com/klauspost/cpuid/v2 v2.2.3 h1:sxCkb+qR91z4vsqw4vGGZlDgPz3G7gjaLyK3V8y7 github.com/klauspost/cpuid/v2 v2.2.3/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk= github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= +github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/qbZg= +github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws= github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= @@ -65,6 +69,8 @@ github.com/minio/minio-go/v7 v7.0.59 h1:lxIXwsTIcQkYoEG25rUJbzpmSB/oWeVDmxFo/uWU github.com/minio/minio-go/v7 v7.0.59/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/minio-go/v7 v7.0.60 h1:iHkrmWyHFs/eZiWc2F/5jAHtNBAFy+HjdhMX6FkkPWc= github.com/minio/minio-go/v7 v7.0.60/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= +github.com/minio/minio-go/v7 v7.0.61 h1:87c+x8J3jxQ5VUGimV9oHdpjsAvy3fhneEBKuoKEVUI= +github.com/minio/minio-go/v7 v7.0.61/go.mod h1:BTu8FcrEw+HidY0zd/0eny43QnVNkXRPXrLXFuQBHXg= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= @@ -95,6 +101,8 @@ github.com/sirupsen/logrus v1.9.0 h1:trlNQbNUG3OdDrDil03MCb1H2o9nJ1x4/5LYw7byDE0 github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y= github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= +github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= @@ -114,12 +122,16 @@ golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc= golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g= golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0= +golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA= +golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio= golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= +golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50= +golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -130,12 +142,16 @@ golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA= +golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4= +golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= From d9a72d043cade2283391dc0e347d0ad8f84f5e61 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Sun, 30 Jul 2023 01:09:33 +0000 Subject: [PATCH 42/79] Update module github.com/rs/zerolog to v1.30.0 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 6f49c4f..9c5a2e3 100644 --- a/go.mod +++ b/go.mod @@ -6,7 +6,7 @@ require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 github.com/minio/minio-go/v7 v7.0.61 - github.com/rs/zerolog v1.29.1 + github.com/rs/zerolog v1.30.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.1.0 github.com/stretchr/testify v1.8.4 diff --git a/go.sum b/go.sum index 8cecbe1..2020147 100644 --- a/go.sum +++ b/go.sum @@ -91,6 +91,8 @@ github.com/rs/zerolog v1.29.0 h1:Zes4hju04hjbvkVkOhdl2HpZa+0PmVwigmo8XoORE5w= github.com/rs/zerolog v1.29.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= github.com/rs/zerolog v1.29.1 h1:cO+d60CHkknCbvzEWxP0S9K6KqyTjrCNUy1LdQLCGPc= github.com/rs/zerolog v1.29.1/go.mod h1:Le6ESbR7hc+DP6Lt1THiV8CQSdkkNrd3R0XbEgp3ZBU= +github.com/rs/zerolog v1.30.0 h1:SymVODrcRsaRaSInD9yQtKbtWqwsfoPcRff/oRXLj4c= +github.com/rs/zerolog v1.30.0/go.mod h1:/tk+P47gFdPXq4QYjvCmT5/Gsug2nagsFWBWhAiSi1w= github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= From bdc1c743d62387fc5038c4a1305676374da63ef2 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 1 Aug 2023 10:51:11 +0900 Subject: [PATCH 43/79] =?UTF-8?q?=E3=83=A9=E3=82=A4=E3=83=96=E3=83=A9?= =?UTF-8?q?=E3=83=AA=E3=82=A2=E3=83=83=E3=83=97=E3=83=87=E3=83=BC=E3=83=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.mod | 22 +++++++++++----------- go.sum | 47 +++++++++++++++++++++++------------------------ 2 files changed, 34 insertions(+), 35 deletions(-) diff --git a/go.mod b/go.mod index 679ef9b..2ec1c1e 100644 --- a/go.mod +++ b/go.mod @@ -4,9 +4,9 @@ go 1.20 require ( github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.56 - github.com/rs/zerolog v1.29.1 - github.com/shogo82148/go-clockwork-base32 v1.0.0 + github.com/minio/minio-go/v7 v7.0.61 + github.com/rs/zerolog v1.30.0 + github.com/shogo82148/go-clockwork-base32 v1.1.0 github.com/stretchr/testify v1.8.4 gopkg.in/ini.v1 v1.67.0 gopkg.in/natefinch/lumberjack.v2 v2.2.1 @@ -16,20 +16,20 @@ require ( github.com/davecgh/go-spew v1.1.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.16.5 // indirect - github.com/klauspost/cpuid/v2 v2.2.4 // indirect + github.com/klauspost/compress v1.16.7 // indirect + github.com/klauspost/cpuid/v2 v2.2.5 // indirect github.com/mattn/go-colorable v0.1.13 // indirect - github.com/mattn/go-isatty v0.0.18 // indirect + github.com/mattn/go-isatty v0.0.19 // indirect github.com/minio/md5-simd v1.1.2 // indirect github.com/minio/sha256-simd v1.0.1 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.2 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect github.com/rs/xid v1.5.0 // indirect - github.com/sirupsen/logrus v1.9.2 // indirect - golang.org/x/crypto v0.9.0 // indirect - golang.org/x/net v0.10.0 // indirect - golang.org/x/sys v0.8.0 // indirect - golang.org/x/text v0.9.0 // indirect + github.com/sirupsen/logrus v1.9.3 // indirect + golang.org/x/crypto v0.11.0 // indirect + golang.org/x/net v0.12.0 // indirect + golang.org/x/sys v0.10.0 // indirect + golang.org/x/text v0.11.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/go.sum b/go.sum index 459f9b8..9dfd544 100644 --- a/go.sum +++ b/go.sum @@ -10,22 +10,22 @@ github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= -github.com/klauspost/compress v1.16.5 h1:IFV2oUNUzZaz+XyusxpLzpzS8Pt5rh0Z16For/djlyI= -github.com/klauspost/compress v1.16.5/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= +github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I= +github.com/klauspost/compress v1.16.7/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= -github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk= -github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY= +github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/qbZg= +github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws= github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= -github.com/mattn/go-isatty v0.0.18 h1:DOKFKCQ7FNG2L1rbrmstDN4QVRdS89Nkh85u68Uwp98= -github.com/mattn/go-isatty v0.0.18/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA= +github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= -github.com/minio/minio-go/v7 v7.0.56 h1:pkZplIEHu8vinjkmhsexcXpWth2tjVLphrTZx6fBVZY= -github.com/minio/minio-go/v7 v7.0.56/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= +github.com/minio/minio-go/v7 v7.0.61 h1:87c+x8J3jxQ5VUGimV9oHdpjsAvy3fhneEBKuoKEVUI= +github.com/minio/minio-go/v7 v7.0.61/go.mod h1:BTu8FcrEw+HidY0zd/0eny43QnVNkXRPXrLXFuQBHXg= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -36,34 +36,33 @@ github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjY github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc= github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= -github.com/rs/zerolog v1.29.1 h1:cO+d60CHkknCbvzEWxP0S9K6KqyTjrCNUy1LdQLCGPc= -github.com/rs/zerolog v1.29.1/go.mod h1:Le6ESbR7hc+DP6Lt1THiV8CQSdkkNrd3R0XbEgp3ZBU= -github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= -github.com/shogo82148/go-clockwork-base32 v1.0.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= -github.com/sirupsen/logrus v1.9.2 h1:oxx1eChJGI6Uks2ZC4W1zpLlVgqB8ner4EuQwV4Ik1Y= -github.com/sirupsen/logrus v1.9.2/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= +github.com/rs/zerolog v1.30.0 h1:SymVODrcRsaRaSInD9yQtKbtWqwsfoPcRff/oRXLj4c= +github.com/rs/zerolog v1.30.0/go.mod h1:/tk+P47gFdPXq4QYjvCmT5/Gsug2nagsFWBWhAiSi1w= +github.com/shogo82148/go-clockwork-base32 v1.1.0 h1:PEqSTiyVEKdl5ar5RHlUQjFJyJTx9XZD5dUVCLG08K0= +github.com/shogo82148/go-clockwork-base32 v1.1.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= +github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= +github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g= -golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0= -golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= -golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= +golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA= +golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio= +golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50= +golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= -golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= -golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA= +golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4= +golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= From a983720b08ff782f10bae5b00d70fe590745394c Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 1 Aug 2023 10:54:16 +0900 Subject: [PATCH 44/79] =?UTF-8?q?go.sum=20=E3=82=A2=E3=83=83=E3=83=97?= =?UTF-8?q?=E3=83=87=E3=83=BC=E3=83=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.sum | 5 ----- 1 file changed, 5 deletions(-) diff --git a/go.sum b/go.sum index 9dfd544..bc6f54a 100644 --- a/go.sum +++ b/go.sum @@ -1,6 +1,5 @@ github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= @@ -34,7 +33,6 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc= github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= @@ -47,7 +45,6 @@ github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVs github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA= golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio= @@ -63,12 +60,10 @@ golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA= golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4= golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= -gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= From b53c4ca3dd6aad26b57fff9baf4912a67c2d7741 Mon Sep 17 00:00:00 2001 From: voluntas Date: Thu, 17 Aug 2023 13:51:02 +0900 Subject: [PATCH 45/79] =?UTF-8?q?go=201.21=20=E3=81=AB=E4=B8=8A=E3=81=92?= =?UTF-8?q?=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/workflows/release.yml | 3 ++- go.mod | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 1b03fae..11a7d8e 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -14,7 +14,8 @@ jobs: - name: Set up Go uses: actions/setup-go@v4 with: - go-version: "^1.20" + go-version-file: ./go.mod + cache: false - run: go install github.com/tcnksm/ghr@latest diff --git a/go.mod b/go.mod index 2ec1c1e..9e111ea 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/shiguredo/sora-archive-uploader -go 1.20 +go 1.21 require ( github.com/google/uuid v1.3.0 From 4c40f0ac348f6461c94a25f2c4e6c613b21b2893 Mon Sep 17 00:00:00 2001 From: voluntas Date: Thu, 17 Aug 2023 14:02:37 +0900 Subject: [PATCH 46/79] =?UTF-8?q?go.sum=20=E6=9B=B4=E6=96=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.sum | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/go.sum b/go.sum index bc6f54a..0510827 100644 --- a/go.sum +++ b/go.sum @@ -1,5 +1,6 @@ github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= @@ -33,6 +34,7 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc= github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= @@ -45,6 +47,7 @@ github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVs github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA= golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio= @@ -66,4 +69,5 @@ gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= From 8145bf9d8b4e64a12cf70e64696d2b5684c9bb99 Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Thu, 17 Aug 2023 22:15:09 +0000 Subject: [PATCH 47/79] Update module github.com/minio/minio-go/v7 to v7.0.62 --- go.mod | 10 +++++----- go.sum | 10 ++++++++++ 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/go.mod b/go.mod index 9c5a2e3..8d679c1 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.61 + github.com/minio/minio-go/v7 v7.0.62 github.com/rs/zerolog v1.30.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.1.0 @@ -31,9 +31,9 @@ require ( github.com/modern-go/reflect2 v1.0.2 // indirect github.com/rs/xid v1.5.0 // indirect github.com/sirupsen/logrus v1.9.3 // indirect - golang.org/x/crypto v0.11.0 // indirect - golang.org/x/net v0.12.0 // indirect - golang.org/x/sys v0.10.0 // indirect - golang.org/x/text v0.11.0 // indirect + golang.org/x/crypto v0.12.0 // indirect + golang.org/x/net v0.14.0 // indirect + golang.org/x/sys v0.11.0 // indirect + golang.org/x/text v0.12.0 // indirect gopkg.in/ini.v1 v1.67.0 // indirect ) diff --git a/go.sum b/go.sum index 2020147..b20587e 100644 --- a/go.sum +++ b/go.sum @@ -71,6 +71,8 @@ github.com/minio/minio-go/v7 v7.0.60 h1:iHkrmWyHFs/eZiWc2F/5jAHtNBAFy+HjdhMX6Fkk github.com/minio/minio-go/v7 v7.0.60/go.mod h1:NUDy4A4oXPq1l2yK6LTSvCEzAMeIcoz9lcj5dbzSrRE= github.com/minio/minio-go/v7 v7.0.61 h1:87c+x8J3jxQ5VUGimV9oHdpjsAvy3fhneEBKuoKEVUI= github.com/minio/minio-go/v7 v7.0.61/go.mod h1:BTu8FcrEw+HidY0zd/0eny43QnVNkXRPXrLXFuQBHXg= +github.com/minio/minio-go/v7 v7.0.62 h1:qNYsFZHEzl+NfH8UxW4jpmlKav1qUAgfY30YNRneVhc= +github.com/minio/minio-go/v7 v7.0.62/go.mod h1:Q6X7Qjb7WMhvG65qKf4gUgA5XaiSox74kR1uAEjxRS4= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= @@ -126,6 +128,8 @@ golang.org/x/crypto v0.9.0 h1:LF6fAI+IutBocDJ2OT0Q1g8plpYljMZ4+lty+dsqw3g= golang.org/x/crypto v0.9.0/go.mod h1:yrmDGqONDYtNj3tH8X9dzUun2m2lzPa9ngI6/RUPGR0= golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA= golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio= +golang.org/x/crypto v0.12.0 h1:tFM/ta59kqch6LlvYnPa0yx5a83cL2nHflFhYKvv9Yk= +golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= @@ -134,6 +138,8 @@ golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50= golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA= +golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= +golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -146,6 +152,8 @@ golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA= golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= +golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= @@ -154,6 +162,8 @@ golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4= golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= +golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.66.6 h1:LATuAqN/shcYAOkv3wl2L4rkaKqkcgTBQjOyYDvcPKI= From fa48c0b8f987903831820f1c98d3e605f714fb5c Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Mon, 21 Aug 2023 17:42:57 +0000 Subject: [PATCH 48/79] Update module github.com/google/uuid to v1.3.1 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 8d679c1..2f498b1 100644 --- a/go.mod +++ b/go.mod @@ -4,7 +4,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 - github.com/google/uuid v1.3.0 + github.com/google/uuid v1.3.1 github.com/minio/minio-go/v7 v7.0.62 github.com/rs/zerolog v1.30.0 github.com/shiguredo/lumberjack/v3 v3.0.0 diff --git a/go.sum b/go.sum index b20587e..3f8c8ba 100644 --- a/go.sum +++ b/go.sum @@ -19,6 +19,8 @@ github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5x github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4= +github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/klauspost/compress v1.15.9 h1:wKRjX6JRtDdrE9qwa4b/Cip7ACOshUI4smpCQanqjSY= From 52878d7b583e1d83a05b41405403697cc970cd4c Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Mon, 28 Aug 2023 22:13:46 +0000 Subject: [PATCH 49/79] Update module github.com/minio/minio-go/v7 to v7.0.63 --- go.mod | 2 +- go.sum | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/go.mod b/go.mod index 2f498b1..39744f9 100644 --- a/go.mod +++ b/go.mod @@ -5,7 +5,7 @@ go 1.19 require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.1 - github.com/minio/minio-go/v7 v7.0.62 + github.com/minio/minio-go/v7 v7.0.63 github.com/rs/zerolog v1.30.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.1.0 diff --git a/go.sum b/go.sum index 3f8c8ba..43d3149 100644 --- a/go.sum +++ b/go.sum @@ -75,6 +75,8 @@ github.com/minio/minio-go/v7 v7.0.61 h1:87c+x8J3jxQ5VUGimV9oHdpjsAvy3fhneEBKuoKE github.com/minio/minio-go/v7 v7.0.61/go.mod h1:BTu8FcrEw+HidY0zd/0eny43QnVNkXRPXrLXFuQBHXg= github.com/minio/minio-go/v7 v7.0.62 h1:qNYsFZHEzl+NfH8UxW4jpmlKav1qUAgfY30YNRneVhc= github.com/minio/minio-go/v7 v7.0.62/go.mod h1:Q6X7Qjb7WMhvG65qKf4gUgA5XaiSox74kR1uAEjxRS4= +github.com/minio/minio-go/v7 v7.0.63 h1:GbZ2oCvaUdgT5640WJOpyDhhDxvknAJU2/T3yurwcbQ= +github.com/minio/minio-go/v7 v7.0.63/go.mod h1:Q6X7Qjb7WMhvG65qKf4gUgA5XaiSox74kR1uAEjxRS4= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= From bcbb5df9c0a5d6663060fa57e8cd37f6c83afae8 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 29 Aug 2023 22:36:51 +0900 Subject: [PATCH 50/79] =?UTF-8?q?=E3=83=A9=E3=82=A4=E3=83=96=E3=83=A9?= =?UTF-8?q?=E3=83=AA=E3=82=A2=E3=83=83=E3=83=97=E3=83=87=E3=83=BC=E3=83=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.mod | 12 ++++++------ go.sum | 25 +++++++++++++------------ 2 files changed, 19 insertions(+), 18 deletions(-) diff --git a/go.mod b/go.mod index 9e111ea..17edf81 100644 --- a/go.mod +++ b/go.mod @@ -3,8 +3,8 @@ module github.com/shiguredo/sora-archive-uploader go 1.21 require ( - github.com/google/uuid v1.3.0 - github.com/minio/minio-go/v7 v7.0.61 + github.com/google/uuid v1.3.1 + github.com/minio/minio-go/v7 v7.0.63 github.com/rs/zerolog v1.30.0 github.com/shogo82148/go-clockwork-base32 v1.1.0 github.com/stretchr/testify v1.8.4 @@ -27,9 +27,9 @@ require ( github.com/pmezard/go-difflib v1.0.0 // indirect github.com/rs/xid v1.5.0 // indirect github.com/sirupsen/logrus v1.9.3 // indirect - golang.org/x/crypto v0.11.0 // indirect - golang.org/x/net v0.12.0 // indirect - golang.org/x/sys v0.10.0 // indirect - golang.org/x/text v0.11.0 // indirect + golang.org/x/crypto v0.12.0 // indirect + golang.org/x/net v0.14.0 // indirect + golang.org/x/sys v0.11.0 // indirect + golang.org/x/text v0.12.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/go.sum b/go.sum index 0510827..3c42a32 100644 --- a/go.sum +++ b/go.sum @@ -6,8 +6,8 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= -github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= -github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4= +github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I= @@ -24,8 +24,8 @@ github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APP github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= -github.com/minio/minio-go/v7 v7.0.61 h1:87c+x8J3jxQ5VUGimV9oHdpjsAvy3fhneEBKuoKEVUI= -github.com/minio/minio-go/v7 v7.0.61/go.mod h1:BTu8FcrEw+HidY0zd/0eny43QnVNkXRPXrLXFuQBHXg= +github.com/minio/minio-go/v7 v7.0.63 h1:GbZ2oCvaUdgT5640WJOpyDhhDxvknAJU2/T3yurwcbQ= +github.com/minio/minio-go/v7 v7.0.63/go.mod h1:Q6X7Qjb7WMhvG65qKf4gUgA5XaiSox74kR1uAEjxRS4= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -49,20 +49,21 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -golang.org/x/crypto v0.11.0 h1:6Ewdq3tDic1mg5xRO4milcWCfMVQhI4NkqWWvqejpuA= -golang.org/x/crypto v0.11.0/go.mod h1:xgJhtzW8F9jGdVFWZESrid1U1bjeNy4zgy5cRr/CIio= -golang.org/x/net v0.12.0 h1:cfawfvKITfUsFCeJIHJrbSxpeu/E81khclypR0GVT50= -golang.org/x/net v0.12.0/go.mod h1:zEVYFnQC7m/vmpQFELhcD1EWkZlX69l4oqgmer6hfKA= +golang.org/x/crypto v0.12.0 h1:tFM/ta59kqch6LlvYnPa0yx5a83cL2nHflFhYKvv9Yk= +golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= +golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= +golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA= -golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4= -golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= +golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= +golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= From e50c38eab45f4d94a8ebb22d4ce3ed0e2b83999f Mon Sep 17 00:00:00 2001 From: voluntas Date: Thu, 31 Aug 2023 12:27:22 +0900 Subject: [PATCH 51/79] =?UTF-8?q?=E3=83=AD=E3=82=B0=E5=91=A8=E3=82=8A?= =?UTF-8?q?=E3=82=92=E6=95=B4=E5=82=99?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGES.md | 12 ++++++ config.go | 26 ++++++++---- config_example.ini | 3 +- logging.go | 100 ++++++++++++++++++++++++++++++++++----------- 4 files changed, 109 insertions(+), 32 deletions(-) create mode 100644 CHANGES.md diff --git a/CHANGES.md b/CHANGES.md new file mode 100644 index 0000000..71a72b0 --- /dev/null +++ b/CHANGES.md @@ -0,0 +1,12 @@ +# 変更履歴 + +- CHANGE + - 下位互換のない変更 +- UPDATE + - 下位互換がある変更 +- ADD + - 下位互換がある追加 +- FIX + - バグ修正 + +## develop diff --git a/config.go b/config.go index b304bd7..90719d0 100644 --- a/config.go +++ b/config.go @@ -2,16 +2,28 @@ package archive import "gopkg.in/ini.v1" +const ( + DefaultLogDir = "." + DefaultLogName = "sora-archive-uploader.jsonl" + + // megabytes + DefaultLogRotateMaxSize = 200 + DefaultLogRotateMaxBackups = 7 + // days + DefaultLogRotateMaxAge = 30 +) + type Config struct { Debug bool `ini:"debug"` - LogDir string `ini:"log_dir"` - LogName string `ini:"log_name"` - LogStdOut bool `ini:"log_std_out"` - LogRotateMaxSize int `ini:"log_rotate_max_size"` - LogRotateMaxBackups int `ini:"log_rotate_max_backups"` - LogRotateMaxAge int `ini:"log_rotate_max_age"` - LogRotateCompress bool `ini:"log_rotate_compress"` + LogDir string `ini:"log_dir"` + LogName string `ini:"log_name"` + LogStdout bool `ini:"log_stdout"` + + LogRotateMaxSize int `ini:"log_rotate_max_size"` + LogRotateMaxBackups int `ini:"log_rotate_max_backups"` + LogRotateMaxAge int `ini:"log_rotate_max_age"` + LogRotateCompress bool `ini:"log_rotate_compress"` ObjectStorageEndpoint string `ini:"object_storage_endpoint"` ObjectStorageBucketName string `ini:"object_storage_bucket_name"` diff --git a/config_example.ini b/config_example.ini index 77fdbf5..82c8f04 100644 --- a/config_example.ini +++ b/config_example.ini @@ -15,10 +15,9 @@ upload_workers = 4 uploaded_file_cache_size = 32 # [log] - log_dir = . log_name = sora-archive-uploader.jsonl -log_std_out = true +log_stdout = true # MB log_rotate_max_size = 200 diff --git a/logging.go b/logging.go index 6930edf..d090f56 100644 --- a/logging.go +++ b/logging.go @@ -2,8 +2,8 @@ package archive import ( "fmt" - "io" "os" + "path/filepath" "strings" "time" @@ -19,49 +19,103 @@ func initLogger(config *Config) error { logPath := fmt.Sprintf("%s/%s", config.LogDir, config.LogName) - writer := &lumberjack.Logger{ - Filename: logPath, - MaxSize: config.LogRotateMaxSize, - MaxBackups: config.LogRotateMaxBackups, - MaxAge: config.LogRotateMaxAge, - Compress: false, - } - // https://github.com/rs/zerolog/issues/77 zerolog.TimestampFunc = func() time.Time { return time.Now().UTC() } - zerolog.TimeFieldFormat = time.RFC3339Nano - - var writers io.Writer - writers = zerolog.MultiLevelWriter(writer) + zerolog.TimeFieldFormat = "2006-01-02T15:04:05.000000Z" - // ログレベル if config.Debug { zerolog.SetGlobalLevel(zerolog.DebugLevel) } else { zerolog.SetGlobalLevel(zerolog.InfoLevel) } - if config.LogStdOut { - consoleWriter := zerolog.ConsoleWriter{Out: os.Stdout, TimeFormat: "2006-01-02 15:04:05.000000Z"} - format(&consoleWriter) - writers = zerolog.MultiLevelWriter(writers, consoleWriter) - } + if config.Debug && config.LogStdout { + writer := zerolog.ConsoleWriter{ + Out: os.Stdout, + FormatTimestamp: func(i interface{}) string { + darkGray := "\x1b[90m" + reset := "\x1b[0m" + return strings.Join([]string{darkGray, i.(string), reset}, "") + }, + NoColor: false, + } + prettyFormat(&writer) + log.Logger = zerolog.New(writer).With().Caller().Timestamp().Logger() + } else if config.LogStdout { + writer := os.Stdout + log.Logger = zerolog.New(writer).With().Caller().Timestamp().Logger() + } else { + var logRotateMaxSize, logRotateMaxBackups, logRotateMaxAge int + if config.LogRotateMaxSize == 0 { + logRotateMaxSize = DefaultLogRotateMaxSize + } + if config.LogRotateMaxBackups == 0 { + logRotateMaxBackups = DefaultLogRotateMaxBackups + } + if config.LogRotateMaxAge == 0 { + logRotateMaxAge = DefaultLogRotateMaxAge + } - log.Logger = zerolog.New(writers).With().Caller().Timestamp().Logger() + writer := &lumberjack.Logger{ + Filename: logPath, + MaxSize: logRotateMaxSize, + MaxBackups: logRotateMaxBackups, + MaxAge: logRotateMaxAge, + Compress: false, + } + log.Logger = zerolog.New(writer).With().Caller().Timestamp().Logger() + } return nil } -func format(w *zerolog.ConsoleWriter) { +// 現時点での prettyFormat +// 2023-04-17 12:51:56.333485Z [INFO] config.go:102 > CONF | debug=true +func prettyFormat(w *zerolog.ConsoleWriter) { + const Reset = "\x1b[0m" + w.FormatLevel = func(i interface{}) string { - return strings.ToUpper(fmt.Sprintf("[%s]", i)) + var color, level string + // TODO: 各色を定数に置き換える + // TODO: 他の logLevel が必要な場合は追加する + switch i.(string) { + case "info": + color = "\x1b[32m" + case "error": + color = "\x1b[31m" + case "warn": + color = "\x1b[33m" + case "debug": + color = "\x1b[34m" + default: + color = "\x1b[37m" + } + + level = strings.ToUpper(i.(string)) + return fmt.Sprintf("%s[%s]%s", color, level, Reset) + } + w.FormatCaller = func(i interface{}) string { + return fmt.Sprintf("[%s]", filepath.Base(i.(string))) + } + // TODO: Caller をファイル名と行番号だけの表示で出力する + // 以下のようなフォーマットにしたい + // 2023-04-17 12:50:09.334758Z [INFO] [config.go:102] CONF | debug=true + // TODO: name=value が無い場合に | を消す方法がわからなかった + w.FormatMessage = func(i interface{}) string { + if i == nil { + return "" + } else { + return fmt.Sprintf("%s |", i) + } } w.FormatFieldName = func(i interface{}) string { - return fmt.Sprintf("%s=", i) + const Cyan = "\x1b[36m" + return fmt.Sprintf("%s%s=%s", Cyan, i, Reset) } + // TODO: カンマ区切りを同実現するかわからなかった w.FormatFieldValue = func(i interface{}) string { return fmt.Sprintf("%s", i) } From 04303a48c3e22c8dfeee1600e12e526a87002943 Mon Sep 17 00:00:00 2001 From: voluntas Date: Thu, 31 Aug 2023 12:29:48 +0900 Subject: [PATCH 52/79] 2023.1.5 --- .github/workflows/ci.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 5ca22f8..02c07f9 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -20,9 +20,9 @@ jobs: cache-dependency-path: ./go.sum - run: go version - run: go fmt . - - uses: dominikh/staticcheck-action@v1.3.0 + - uses: dominikh/staticcheck-action@v1 with: - version: "2023.1.2" + version: "2023.1.5" install-go: false - name: Test run: make test From 9e8420d3325c1f0479f068dad5389deaf3a7998e Mon Sep 17 00:00:00 2001 From: voluntas Date: Thu, 31 Aug 2023 12:30:17 +0900 Subject: [PATCH 53/79] config.ini --- cmd/sora-archive-uploader/main.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/sora-archive-uploader/main.go b/cmd/sora-archive-uploader/main.go index fcfdcac..c49b6cb 100644 --- a/cmd/sora-archive-uploader/main.go +++ b/cmd/sora-archive-uploader/main.go @@ -22,7 +22,7 @@ build date: %s ) func main() { - configFilePath := flag.String("C", "config.toml", "Config file path") + configFilePath := flag.String("C", "config.ini", "Config file path") var v bool flag.BoolVar(&v, "version", false, "Show version") flag.Parse() From 1f251be67885168493c9d06159a56c8b5c13033a Mon Sep 17 00:00:00 2001 From: voluntas Date: Fri, 8 Sep 2023 23:38:46 +0900 Subject: [PATCH 54/79] =?UTF-8?q?VERSION=20=E3=83=95=E3=82=A1=E3=82=A4?= =?UTF-8?q?=E3=83=AB=E3=82=92=E5=B0=8E=E5=85=A5=E3=81=97=E3=81=A6=20Makefi?= =?UTF-8?q?le=20=E3=82=92=E6=95=B4=E7=90=86=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Makefile | 21 ++++----------------- VERSION | 1 + cmd/sora-archive-uploader/main.go | 28 ++++++++-------------------- config.go | 9 ++++++++- 4 files changed, 21 insertions(+), 38 deletions(-) create mode 100644 VERSION diff --git a/Makefile b/Makefile index 4d1648d..25d8e79 100644 --- a/Makefile +++ b/Makefile @@ -1,20 +1,7 @@ -VERSION := 2023.1.0 -REVISION := $(shell git rev-parse --short HEAD) -BUILD_DATE := $(shell date -u "+%Y-%m-%dT%H:%M:%SZ") -LDFLAGS := "-X main.version=$(VERSION) -X main.revision=$(REVISION) -X main.buildDate=$(BUILD_DATE)" -LDFLAGS_PROD := "-s -w -X main.version=$(VERSION) -X main.revision=$(REVISION)" +.PHONY: all test -export GO1111MODULE=on -export CWD=$(dir $(abspath $(lastword $(MAKEFILE_LIST)))) - -.PHONY: all sora-archive-uploader-dev sora-archive-uploader-prod -all: sora-archive-uploader-dev - -sora-archive-uploader-dev: cmd/sora-archive-uploader/main.go - go build -race -ldflags $(LDFLAGS) -o bin/$@ $< - -sora-archive-uploader-prod: cmd/sora-archive-uploader/main.go - go build -ldflags $(LDFLAGS_PROD) -o bin/$@ $< +all: + go build -o bin/sora-archive-uploader cmd/sora-archive-uploader/main.go test: - go test -v ./s3 + go test -race -v ./s3 \ No newline at end of file diff --git a/VERSION b/VERSION new file mode 100644 index 0000000..57c6e87 --- /dev/null +++ b/VERSION @@ -0,0 +1 @@ +2023.1.0 diff --git a/cmd/sora-archive-uploader/main.go b/cmd/sora-archive-uploader/main.go index c49b6cb..fcdb834 100644 --- a/cmd/sora-archive-uploader/main.go +++ b/cmd/sora-archive-uploader/main.go @@ -4,35 +4,23 @@ import ( "flag" "fmt" "log" - "os" archive "github.com/shiguredo/sora-archive-uploader" ) -var ( - version string - revision string - buildDate string - - versionText = `sora-archive-uploader build info. -version: %s -revision: %s -build date: %s -` -) - func main() { - configFilePath := flag.String("C", "config.ini", "Config file path") - var v bool - flag.BoolVar(&v, "version", false, "Show version") + // /bin/sora-archive-uploader -V + showVersion := flag.Bool("V", false, "バージョン") + + // /bin/sora-archive-uploader -C ./config.ini + configFilePath := flag.String("C", "./config.ini", "Config file path") flag.Parse() - if v { - fmt.Printf(versionText, version, revision, buildDate) - os.Exit(0) + if *showVersion { + fmt.Printf("Sora Archive Uploader version %s\n", archive.Version) + return } - log.Printf("sora-archive-uploader version:%s revision:%s build_date:%s", version, revision, buildDate) log.Printf("config file path: %s", *configFilePath) archive.Run(configFilePath) } diff --git a/config.go b/config.go index 90719d0..1216b32 100644 --- a/config.go +++ b/config.go @@ -1,6 +1,13 @@ package archive -import "gopkg.in/ini.v1" +import ( + _ "embed" + + "gopkg.in/ini.v1" +) + +//go:embed VERSION +var Version string const ( DefaultLogDir = "." From 9904859d8ee89121a755608b0ac8f50fa545d60b Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Mon, 25 Sep 2023 11:02:15 +0000 Subject: [PATCH 55/79] Update module github.com/rs/zerolog to v1.31.0 --- go.mod | 8 ++++---- go.sum | 11 +++++++++++ 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/go.mod b/go.mod index 39744f9..e17395b 100644 --- a/go.mod +++ b/go.mod @@ -6,7 +6,7 @@ require ( github.com/BurntSushi/toml v1.3.2 github.com/google/uuid v1.3.1 github.com/minio/minio-go/v7 v7.0.63 - github.com/rs/zerolog v1.30.0 + github.com/rs/zerolog v1.31.0 github.com/shiguredo/lumberjack/v3 v3.0.0 github.com/shogo82148/go-clockwork-base32 v1.1.0 github.com/stretchr/testify v1.8.4 @@ -23,8 +23,8 @@ require ( github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/compress v1.16.7 // indirect github.com/klauspost/cpuid/v2 v2.2.5 // indirect - github.com/mattn/go-colorable v0.1.12 // indirect - github.com/mattn/go-isatty v0.0.14 // indirect + github.com/mattn/go-colorable v0.1.13 // indirect + github.com/mattn/go-isatty v0.0.19 // indirect github.com/minio/md5-simd v1.1.2 // indirect github.com/minio/sha256-simd v1.0.1 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect @@ -33,7 +33,7 @@ require ( github.com/sirupsen/logrus v1.9.3 // indirect golang.org/x/crypto v0.12.0 // indirect golang.org/x/net v0.14.0 // indirect - golang.org/x/sys v0.11.0 // indirect + golang.org/x/sys v0.12.0 // indirect golang.org/x/text v0.12.0 // indirect gopkg.in/ini.v1 v1.67.0 // indirect ) diff --git a/go.sum b/go.sum index 43d3149..cc6ee5e 100644 --- a/go.sum +++ b/go.sum @@ -45,8 +45,13 @@ github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/q github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws= github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= +github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= +github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= +github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= +github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA= +github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= github.com/minio/minio-go/v7 v7.0.45 h1:g4IeM9M9pW/Lo8AGGNOjBZYlvmtlE1N5TQEYWXRWzIs= @@ -99,6 +104,8 @@ github.com/rs/zerolog v1.29.1 h1:cO+d60CHkknCbvzEWxP0S9K6KqyTjrCNUy1LdQLCGPc= github.com/rs/zerolog v1.29.1/go.mod h1:Le6ESbR7hc+DP6Lt1THiV8CQSdkkNrd3R0XbEgp3ZBU= github.com/rs/zerolog v1.30.0 h1:SymVODrcRsaRaSInD9yQtKbtWqwsfoPcRff/oRXLj4c= github.com/rs/zerolog v1.30.0/go.mod h1:/tk+P47gFdPXq4QYjvCmT5/Gsug2nagsFWBWhAiSi1w= +github.com/rs/zerolog v1.31.0 h1:FcTR3NnLWW+NnTwwhFWiJSZr4ECLpqCm6QsEnyvbV4A= +github.com/rs/zerolog v1.31.0/go.mod h1:/7mN4D5sKwJLZQ2b/znpjC3/GQWY/xaDXUM0kKWRHss= github.com/shiguredo/lumberjack/v3 v3.0.0 h1:IkhnjVTpg7Fl2fNAfsMn7R+No9SrnFdN5wEWBY2KOT8= github.com/shiguredo/lumberjack/v3 v3.0.0/go.mod h1:zrQSStRcwpVudSkotZINDGYm8x03gnEQd7M+Y0OzPp0= github.com/shogo82148/go-clockwork-base32 v1.0.0 h1:tSLEf+tCChBr5Evbg812hYi2b+yAaZZQb9w7v5gxOn4= @@ -150,14 +157,18 @@ golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA= golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.12.0 h1:CM0HF96J0hcLAwsHPJZjfdNzs0gftsLfgKt57wWHJ0o= +golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= From cabecfccc0bcb5609d958ca845821da2fa22d5d5 Mon Sep 17 00:00:00 2001 From: voluntas Date: Thu, 12 Oct 2023 10:40:06 +0900 Subject: [PATCH 56/79] =?UTF-8?q?rate=20limit=20=E3=81=AE=E3=83=97?= =?UTF-8?q?=E3=83=AD=E3=83=88=E3=82=BF=E3=82=A4=E3=83=97=E3=82=92=E8=BF=BD?= =?UTF-8?q?=E5=8A=A0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config.go | 2 ++ go.mod | 1 + go.sum | 2 ++ s3.go | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++ uploader.go | 9 ++++++- 5 files changed, 90 insertions(+), 1 deletion(-) diff --git a/config.go b/config.go index 1216b32..baad143 100644 --- a/config.go +++ b/config.go @@ -42,6 +42,8 @@ type Config struct { UploadWorkers int `ini:"upload_workers"` + UploadFileRateLimitMbps float64 `ini:"upload_file_rate_limit_mbps"` + UploadedFileCacheSize int `ini:"uploaded_file_cache_size"` WebhookEndpointURL string `ini:"webhook_endpoint_url"` diff --git a/go.mod b/go.mod index 17edf81..5498eff 100644 --- a/go.mod +++ b/go.mod @@ -31,5 +31,6 @@ require ( golang.org/x/net v0.14.0 // indirect golang.org/x/sys v0.11.0 // indirect golang.org/x/text v0.12.0 // indirect + golang.org/x/time v0.3.0 gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/go.sum b/go.sum index 3c42a32..b9a61ac 100644 --- a/go.sum +++ b/go.sum @@ -63,6 +63,8 @@ golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= +golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= diff --git a/s3.go b/s3.go index 5dd176d..e3b5662 100644 --- a/s3.go +++ b/s3.go @@ -3,10 +3,13 @@ package archive import ( "context" "fmt" + "io" "net/url" "os" "path/filepath" + "golang.org/x/time/rate" + "github.com/minio/minio-go/v7" "github.com/minio/minio-go/v7/pkg/credentials" zlog "github.com/rs/zerolog/log" @@ -122,3 +125,77 @@ func isFileContinuous(err error) bool { } return true } + +func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, dst, filePath string, + rateLimitMpbs float64) (string, error) { + var creds *credentials.Credentials + if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { + creds = credentials.NewStaticV4( + osConfig.AccessKeyID, + osConfig.SecretAccessKey, + "", + ) + } else if (len(os.Getenv("AWS_ACCESS_KEY_ID")) > 0) && (len(os.Getenv("AWS_SECRET_ACCESS_KEY")) > 0) { + creds = credentials.NewEnvAWS() + } else { + creds = credentials.NewIAM("") + } + s3Client, err := s3.NewClient(osConfig.Endpoint, creds) + if err != nil { + return "", err + } + + fileReader, err := os.Open(filePath) + if err != nil { + return "", err + } + defer fileReader.Close() + + // Save the file stat. + fileStat, err := fileReader.Stat() + if err != nil { + return "", err + } + + // Save the file size. + fileSize := fileStat.Size() + + // bit を byte にする + rateLimitMByteps := rateLimitMpbs / 8 + limiter := rate.NewLimiter(rate.Limit(rateLimitMByteps*1024*1024), int(rateLimitMByteps*1024*1024)) + + // リミッタを適用したReaderを作成 + rateLimitedFileReader := NewRateLimitedReader(fileReader, limiter) + + n, err := s3Client.PutObject(ctx, osConfig.BucketName, dst, rateLimitedFileReader, fileSize, + minio.PutObjectOptions{ContentType: "application/octet-stream"}) + if err != nil { + return "", err + } + + objectURL := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) + return objectURL, nil +} + +type RateLimitedReader struct { + reader io.Reader + limiter *rate.Limiter +} + +func (r *RateLimitedReader) Read(p []byte) (int, error) { + n, err := r.reader.Read(p) + if err != nil { + return n, err + } + + // ここで制限をかける + err = r.limiter.WaitN(nil, n) + return n, err +} + +func NewRateLimitedReader(r io.Reader, l *rate.Limiter) *RateLimitedReader { + return &RateLimitedReader{ + reader: r, + limiter: l, + } +} diff --git a/uploader.go b/uploader.go index eeda33a..a1be4f5 100644 --- a/uploader.go +++ b/uploader.go @@ -292,7 +292,14 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { Msg("UPLOAD-METADATA-FILE-SUCCESSFULLY") webmObjectKey := fmt.Sprintf("%s/%s", am.RecordingID, webmFilename) - fileURL, err := uploadWebMFile(u.ctx, osConfig, webmObjectKey, webmFilepath) + + var fileURL string + if u.config.UploadFileRateLimitMbps == 0 { + fileURL, err = uploadWebMFile(u.ctx, osConfig, webmObjectKey, webmFilepath) + } else { + fileURL, err = uploadWebMFileWithRateLimit(u.ctx, osConfig, webmObjectKey, webmFilepath, u.config.UploadFileRateLimitMbps) + } + if err != nil { zlog.Error(). Err(err). From 8411f6ef57c7b6e56ed0947062f94c5eca15c7f6 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Thu, 12 Oct 2023 13:08:55 +0900 Subject: [PATCH 57/79] =?UTF-8?q?nil=20Context=20=E3=82=92=E3=82=84?= =?UTF-8?q?=E3=82=81=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/s3.go b/s3.go index e3b5662..8cfc8c9 100644 --- a/s3.go +++ b/s3.go @@ -189,7 +189,7 @@ func (r *RateLimitedReader) Read(p []byte) (int, error) { } // ここで制限をかける - err = r.limiter.WaitN(nil, n) + err = r.limiter.WaitN(context.TODO(), n) return n, err } From 880e326d94a824eab9e44d04498a29d4832e6a18 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Thu, 12 Oct 2023 16:42:09 +0900 Subject: [PATCH 58/79] =?UTF-8?q?UploaderManager=20=E3=81=AB=20limiter=20?= =?UTF-8?q?=E3=82=92=E6=8C=81=E3=81=9F=E3=81=9B=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- runner.go | 9 ++++++++- s3.go | 6 +----- uploader.go | 16 ++++++++++------ 3 files changed, 19 insertions(+), 12 deletions(-) diff --git a/runner.go b/runner.go index d177fa2..32a981b 100644 --- a/runner.go +++ b/runner.go @@ -9,6 +9,7 @@ import ( "time" zlog "github.com/rs/zerolog/log" + "golang.org/x/time/rate" ) type Main struct { @@ -62,7 +63,13 @@ func (m *Main) run(ctx context.Context, cancel context.CancelFunc) error { gateKeeper := newGateKeeper(m.config) recordingFileStream := gateKeeper.run(processContext, foundFiles) - uploaderManager := newUploaderManager() + rateLimitMpbs := m.config.UploadFileRateLimitMbps + // bit を byte にする + rateLimitMByteps := rateLimitMpbs / 8 + limiter := rate.NewLimiter(rate.Limit(rateLimitMByteps*1024*1024), int(rateLimitMByteps*1024*1024)) + + // rate.Limiter は共通 + uploaderManager := newUploaderManager(limiter) _, err = uploaderManager.run(processContext, m.config, recordingFileStream) if err != nil { processContextCancel() diff --git a/s3.go b/s3.go index 8cfc8c9..ac435ac 100644 --- a/s3.go +++ b/s3.go @@ -127,7 +127,7 @@ func isFileContinuous(err error) bool { } func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, dst, filePath string, - rateLimitMpbs float64) (string, error) { + limiter *rate.Limiter) (string, error) { var creds *credentials.Credentials if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( @@ -160,10 +160,6 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO // Save the file size. fileSize := fileStat.Size() - // bit を byte にする - rateLimitMByteps := rateLimitMpbs / 8 - limiter := rate.NewLimiter(rate.Limit(rateLimitMByteps*1024*1024), int(rateLimitMByteps*1024*1024)) - // リミッタを適用したReaderを作成 rateLimitedFileReader := NewRateLimitedReader(fileReader, limiter) diff --git a/uploader.go b/uploader.go index a1be4f5..dceb034 100644 --- a/uploader.go +++ b/uploader.go @@ -12,6 +12,7 @@ import ( "github.com/google/uuid" "github.com/shiguredo/sora-archive-uploader/s3" base32 "github.com/shogo82148/go-clockwork-base32" + "golang.org/x/time/rate" zlog "github.com/rs/zerolog/log" ) @@ -28,6 +29,7 @@ type UploaderManager struct { ArchiveEndStream chan UploaderResult ReportStream chan UploaderResult uploaders []Uploader + Limiter *rate.Limiter } type ArchiveMetadata struct { @@ -52,7 +54,7 @@ type ArchiveEndMetadata struct { Filename string `json:"filename"` } -func newUploaderManager() *UploaderManager { +func newUploaderManager(limiter *rate.Limiter) *UploaderManager { var uploaders []Uploader archiveStream := make(chan UploaderResult) archiveEndStream := make(chan UploaderResult) @@ -62,6 +64,7 @@ func newUploaderManager() *UploaderManager { ArchiveEndStream: archiveEndStream, ReportStream: reportStream, uploaders: uploaders, + Limiter: limiter, } } @@ -71,7 +74,7 @@ func (um *UploaderManager) run(ctx context.Context, config *Config, fileStream < if err != nil { return nil, err } - uploader.run(fileStream, um.ArchiveStream, um.ArchiveEndStream, um.ReportStream) + uploader.run(fileStream, um.ArchiveStream, um.ArchiveEndStream, um.ReportStream, um.Limiter) um.uploaders = append(um.uploaders, *uploader) } go func() { @@ -119,6 +122,7 @@ func (u Uploader) run( outArchive chan UploaderResult, outArchiveEnd chan UploaderResult, outReport chan UploaderResult, + limiter *rate.Limiter, ) { go func() { for { @@ -166,7 +170,7 @@ func (u Uploader) run( Int("uploader_id", u.id). Str("file_path", inputFilepath). Msg("FOUND-AT-STARTUP") - ok := u.handleArchive(inputFilepath, false) + ok := u.handleArchive(inputFilepath, false, limiter) select { case <-u.ctx.Done(): return @@ -180,7 +184,7 @@ func (u Uploader) run( Int("uploader_id", u.id). Str("file_path", inputFilepath). Msg("FOUND-AT-STARTUP") - ok := u.handleArchive(inputFilepath, true) + ok := u.handleArchive(inputFilepath, true, limiter) select { case <-u.ctx.Done(): return @@ -199,7 +203,7 @@ func (u Uploader) Stop() { u.cancel() } -func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { +func (u Uploader) handleArchive(archiveJSONFilePath string, split bool, limiter *rate.Limiter) bool { fileInfo, err := os.Stat(archiveJSONFilePath) if err != nil { zlog.Error(). @@ -297,7 +301,7 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { if u.config.UploadFileRateLimitMbps == 0 { fileURL, err = uploadWebMFile(u.ctx, osConfig, webmObjectKey, webmFilepath) } else { - fileURL, err = uploadWebMFileWithRateLimit(u.ctx, osConfig, webmObjectKey, webmFilepath, u.config.UploadFileRateLimitMbps) + fileURL, err = uploadWebMFileWithRateLimit(u.ctx, osConfig, webmObjectKey, webmFilepath, limiter) } if err != nil { From 957fc330c5ea2b1bb3bc4e2463be5a483e50c9a3 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Fri, 13 Oct 2023 11:15:33 +0900 Subject: [PATCH 59/79] =?UTF-8?q?Revert=20"UploaderManager=20=E3=81=AB=20l?= =?UTF-8?q?imiter=20=E3=82=92=E6=8C=81=E3=81=9F=E3=81=9B=E3=82=8B"?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 880e326d94a824eab9e44d04498a29d4832e6a18. --- runner.go | 9 +-------- s3.go | 6 +++++- uploader.go | 16 ++++++---------- 3 files changed, 12 insertions(+), 19 deletions(-) diff --git a/runner.go b/runner.go index 32a981b..d177fa2 100644 --- a/runner.go +++ b/runner.go @@ -9,7 +9,6 @@ import ( "time" zlog "github.com/rs/zerolog/log" - "golang.org/x/time/rate" ) type Main struct { @@ -63,13 +62,7 @@ func (m *Main) run(ctx context.Context, cancel context.CancelFunc) error { gateKeeper := newGateKeeper(m.config) recordingFileStream := gateKeeper.run(processContext, foundFiles) - rateLimitMpbs := m.config.UploadFileRateLimitMbps - // bit を byte にする - rateLimitMByteps := rateLimitMpbs / 8 - limiter := rate.NewLimiter(rate.Limit(rateLimitMByteps*1024*1024), int(rateLimitMByteps*1024*1024)) - - // rate.Limiter は共通 - uploaderManager := newUploaderManager(limiter) + uploaderManager := newUploaderManager() _, err = uploaderManager.run(processContext, m.config, recordingFileStream) if err != nil { processContextCancel() diff --git a/s3.go b/s3.go index ac435ac..8cfc8c9 100644 --- a/s3.go +++ b/s3.go @@ -127,7 +127,7 @@ func isFileContinuous(err error) bool { } func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, dst, filePath string, - limiter *rate.Limiter) (string, error) { + rateLimitMpbs float64) (string, error) { var creds *credentials.Credentials if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( @@ -160,6 +160,10 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO // Save the file size. fileSize := fileStat.Size() + // bit を byte にする + rateLimitMByteps := rateLimitMpbs / 8 + limiter := rate.NewLimiter(rate.Limit(rateLimitMByteps*1024*1024), int(rateLimitMByteps*1024*1024)) + // リミッタを適用したReaderを作成 rateLimitedFileReader := NewRateLimitedReader(fileReader, limiter) diff --git a/uploader.go b/uploader.go index dceb034..a1be4f5 100644 --- a/uploader.go +++ b/uploader.go @@ -12,7 +12,6 @@ import ( "github.com/google/uuid" "github.com/shiguredo/sora-archive-uploader/s3" base32 "github.com/shogo82148/go-clockwork-base32" - "golang.org/x/time/rate" zlog "github.com/rs/zerolog/log" ) @@ -29,7 +28,6 @@ type UploaderManager struct { ArchiveEndStream chan UploaderResult ReportStream chan UploaderResult uploaders []Uploader - Limiter *rate.Limiter } type ArchiveMetadata struct { @@ -54,7 +52,7 @@ type ArchiveEndMetadata struct { Filename string `json:"filename"` } -func newUploaderManager(limiter *rate.Limiter) *UploaderManager { +func newUploaderManager() *UploaderManager { var uploaders []Uploader archiveStream := make(chan UploaderResult) archiveEndStream := make(chan UploaderResult) @@ -64,7 +62,6 @@ func newUploaderManager(limiter *rate.Limiter) *UploaderManager { ArchiveEndStream: archiveEndStream, ReportStream: reportStream, uploaders: uploaders, - Limiter: limiter, } } @@ -74,7 +71,7 @@ func (um *UploaderManager) run(ctx context.Context, config *Config, fileStream < if err != nil { return nil, err } - uploader.run(fileStream, um.ArchiveStream, um.ArchiveEndStream, um.ReportStream, um.Limiter) + uploader.run(fileStream, um.ArchiveStream, um.ArchiveEndStream, um.ReportStream) um.uploaders = append(um.uploaders, *uploader) } go func() { @@ -122,7 +119,6 @@ func (u Uploader) run( outArchive chan UploaderResult, outArchiveEnd chan UploaderResult, outReport chan UploaderResult, - limiter *rate.Limiter, ) { go func() { for { @@ -170,7 +166,7 @@ func (u Uploader) run( Int("uploader_id", u.id). Str("file_path", inputFilepath). Msg("FOUND-AT-STARTUP") - ok := u.handleArchive(inputFilepath, false, limiter) + ok := u.handleArchive(inputFilepath, false) select { case <-u.ctx.Done(): return @@ -184,7 +180,7 @@ func (u Uploader) run( Int("uploader_id", u.id). Str("file_path", inputFilepath). Msg("FOUND-AT-STARTUP") - ok := u.handleArchive(inputFilepath, true, limiter) + ok := u.handleArchive(inputFilepath, true) select { case <-u.ctx.Done(): return @@ -203,7 +199,7 @@ func (u Uploader) Stop() { u.cancel() } -func (u Uploader) handleArchive(archiveJSONFilePath string, split bool, limiter *rate.Limiter) bool { +func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { fileInfo, err := os.Stat(archiveJSONFilePath) if err != nil { zlog.Error(). @@ -301,7 +297,7 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool, limiter if u.config.UploadFileRateLimitMbps == 0 { fileURL, err = uploadWebMFile(u.ctx, osConfig, webmObjectKey, webmFilepath) } else { - fileURL, err = uploadWebMFileWithRateLimit(u.ctx, osConfig, webmObjectKey, webmFilepath, limiter) + fileURL, err = uploadWebMFileWithRateLimit(u.ctx, osConfig, webmObjectKey, webmFilepath, u.config.UploadFileRateLimitMbps) } if err != nil { From a93c9eee30d0d7a1184fa897022af54a19a10176 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Fri, 13 Oct 2023 11:16:34 +0900 Subject: [PATCH 60/79] =?UTF-8?q?=E8=AA=AC=E6=98=8E=E3=82=92=E8=BF=BD?= =?UTF-8?q?=E8=A8=98=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config.go | 1 + 1 file changed, 1 insertion(+) diff --git a/config.go b/config.go index baad143..7ac15d3 100644 --- a/config.go +++ b/config.go @@ -42,6 +42,7 @@ type Config struct { UploadWorkers int `ini:"upload_workers"` + // 1 ファイルあたりのアップロードレート制限 UploadFileRateLimitMbps float64 `ini:"upload_file_rate_limit_mbps"` UploadedFileCacheSize int `ini:"uploaded_file_cache_size"` From 43ba0899c2033c0fdb25a5e734ac861735ff61dd Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Tue, 17 Oct 2023 16:19:50 +0900 Subject: [PATCH 61/79] =?UTF-8?q?=E3=82=A2=E3=83=83=E3=83=97=E3=83=AD?= =?UTF-8?q?=E3=83=BC=E3=83=89=E9=80=9F=E5=BA=A6=E3=81=AE=E5=88=B6=E9=99=90?= =?UTF-8?q?=E3=82=92=E8=BF=BD=E5=8A=A0=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config.go | 2 +- go.mod | 2 ++ go.sum | 2 ++ runner.go | 5 +++++ s3.go | 64 ++++++++++++++++++++++--------------------------------- s3/s3.go | 17 ++++++++++++--- 6 files changed, 49 insertions(+), 43 deletions(-) diff --git a/config.go b/config.go index 7ac15d3..a16a3d9 100644 --- a/config.go +++ b/config.go @@ -43,7 +43,7 @@ type Config struct { UploadWorkers int `ini:"upload_workers"` // 1 ファイルあたりのアップロードレート制限 - UploadFileRateLimitMbps float64 `ini:"upload_file_rate_limit_mbps"` + UploadFileRateLimitMbps int `ini:"upload_file_rate_limit_mbps"` UploadedFileCacheSize int `ini:"uploaded_file_cache_size"` diff --git a/go.mod b/go.mod index 5498eff..2c3f291 100644 --- a/go.mod +++ b/go.mod @@ -12,6 +12,8 @@ require ( gopkg.in/natefinch/lumberjack.v2 v2.2.1 ) +require github.com/conduitio/bwlimit v0.1.0 // indirect + require ( github.com/davecgh/go-spew v1.1.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect diff --git a/go.sum b/go.sum index b9a61ac..83f2556 100644 --- a/go.sum +++ b/go.sum @@ -1,3 +1,5 @@ +github.com/conduitio/bwlimit v0.1.0 h1:x3ijON0TSghQob4tFKaEvKixFmYKfVJQeSpXluC2JvE= +github.com/conduitio/bwlimit v0.1.0/go.mod h1:E+ASZ1/5L33MTb8hJTERs5Xnmh6Ulq3jbRh7LrdbXWU= github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= diff --git a/runner.go b/runner.go index d177fa2..74a7dda 100644 --- a/runner.go +++ b/runner.go @@ -151,6 +151,11 @@ func Run(configFilePath *string) { resp.Body.Close() } + // 指定は 0 (制限なし) または 10 Mbps 以上 + if (config.UploadFileRateLimitMbps > 0) && (config.UploadFileRateLimitMbps < 10) { + zlog.Fatal().Msg("UPLOAD-FILE-RATE-LIMIT-MBPS-ERROR") + } + zlog.Info().Msg("STARTED-SORA-ARCHIVE-UPLOADER") // シグナルをキャッチして停止処理 diff --git a/s3.go b/s3.go index 8cfc8c9..ff76b63 100644 --- a/s3.go +++ b/s3.go @@ -3,17 +3,19 @@ package archive import ( "context" "fmt" - "io" + "net" + "net/http" "net/url" "os" "path/filepath" - - "golang.org/x/time/rate" + "time" "github.com/minio/minio-go/v7" "github.com/minio/minio-go/v7/pkg/credentials" zlog "github.com/rs/zerolog/log" "github.com/shiguredo/sora-archive-uploader/s3" + + "github.com/conduitio/bwlimit" ) func uploadJSONFile( @@ -34,7 +36,7 @@ func uploadJSONFile( creds = credentials.NewIAM("") } - s3Client, err := s3.NewClient(osConfig.Endpoint, creds) + s3Client, err := s3.NewClient(osConfig.Endpoint, creds, nil) if err != nil { return "", err } @@ -78,7 +80,7 @@ func uploadWebMFile(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, } else { creds = credentials.NewIAM("") } - s3Client, err := s3.NewClient(osConfig.Endpoint, creds) + s3Client, err := s3.NewClient(osConfig.Endpoint, creds, nil) if err != nil { return "", err } @@ -127,7 +129,7 @@ func isFileContinuous(err error) bool { } func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, dst, filePath string, - rateLimitMpbs float64) (string, error) { + rateLimitMpbs int) (string, error) { var creds *credentials.Credentials if (osConfig.AccessKeyID != "") || (osConfig.SecretAccessKey != "") { creds = credentials.NewStaticV4( @@ -140,7 +142,20 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO } else { creds = credentials.NewIAM("") } - s3Client, err := s3.NewClient(osConfig.Endpoint, creds) + + // bit を byte にする + rateLimitMByteps := (bwlimit.Byte(rateLimitMpbs) * bwlimit.MiB) / 8 + + // 受信には制限をかけない + dialer := bwlimit.NewDialer(&net.Dialer{ + Timeout: 30 * time.Second, + KeepAlive: 30 * time.Second, + }, rateLimitMByteps, 0) + + transport := http.DefaultTransport + transport.(*http.Transport).DialContext = dialer.DialContext + + s3Client, err := s3.NewClient(osConfig.Endpoint, creds, &transport) if err != nil { return "", err } @@ -160,15 +175,9 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO // Save the file size. fileSize := fileStat.Size() - // bit を byte にする - rateLimitMByteps := rateLimitMpbs / 8 - limiter := rate.NewLimiter(rate.Limit(rateLimitMByteps*1024*1024), int(rateLimitMByteps*1024*1024)) - - // リミッタを適用したReaderを作成 - rateLimitedFileReader := NewRateLimitedReader(fileReader, limiter) - - n, err := s3Client.PutObject(ctx, osConfig.BucketName, dst, rateLimitedFileReader, fileSize, - minio.PutObjectOptions{ContentType: "application/octet-stream"}) + // 制限時にはマルチパートアップロードを行わない + n, err := s3Client.PutObject(ctx, osConfig.BucketName, dst, fileReader, fileSize, + minio.PutObjectOptions{ContentType: "application/octet-stream", DisableMultipart: true}) if err != nil { return "", err } @@ -176,26 +185,3 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO objectURL := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) return objectURL, nil } - -type RateLimitedReader struct { - reader io.Reader - limiter *rate.Limiter -} - -func (r *RateLimitedReader) Read(p []byte) (int, error) { - n, err := r.reader.Read(p) - if err != nil { - return n, err - } - - // ここで制限をかける - err = r.limiter.WaitN(context.TODO(), n) - return n, err -} - -func NewRateLimitedReader(r io.Reader, l *rate.Limiter) *RateLimitedReader { - return &RateLimitedReader{ - reader: r, - limiter: l, - } -} diff --git a/s3/s3.go b/s3/s3.go index 612bec0..593dcc3 100644 --- a/s3/s3.go +++ b/s3/s3.go @@ -1,6 +1,7 @@ package s3 import ( + "net/http" "net/url" "github.com/minio/minio-go/v7" @@ -41,12 +42,22 @@ func maybeEndpointURL(endpoint string) (string, bool) { return endpoint, secure } -func NewClient(endpoint string, credentials *credentials.Credentials) (*minio.Client, error) { +func NewClient(endpoint string, credentials *credentials.Credentials, transport *http.RoundTripper) (*minio.Client, error) { newEndpoint, secure := maybeEndpointURL(endpoint) + if transport == nil { + return minio.New( + newEndpoint, + &minio.Options{ + Creds: credentials, + Secure: secure, + }) + } + return minio.New( newEndpoint, &minio.Options{ - Creds: credentials, - Secure: secure, + Creds: credentials, + Secure: secure, + Transport: *transport, }) } From 31c9e5466e12104c46785f28473c6ad2ee6a1a59 Mon Sep 17 00:00:00 2001 From: voluntas Date: Tue, 17 Oct 2023 17:56:30 +0900 Subject: [PATCH 62/79] =?UTF-8?q?go.mod=20=E3=82=A2=E3=83=83=E3=83=97?= =?UTF-8?q?=E3=83=87=E3=83=BC=E3=83=88?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.mod | 19 +++++++++---------- go.sum | 32 +++++++++++++++----------------- 2 files changed, 24 insertions(+), 27 deletions(-) diff --git a/go.mod b/go.mod index 2c3f291..fe3c3f7 100644 --- a/go.mod +++ b/go.mod @@ -3,25 +3,24 @@ module github.com/shiguredo/sora-archive-uploader go 1.21 require ( + github.com/conduitio/bwlimit v0.1.0 github.com/google/uuid v1.3.1 github.com/minio/minio-go/v7 v7.0.63 - github.com/rs/zerolog v1.30.0 + github.com/rs/zerolog v1.31.0 github.com/shogo82148/go-clockwork-base32 v1.1.0 github.com/stretchr/testify v1.8.4 gopkg.in/ini.v1 v1.67.0 gopkg.in/natefinch/lumberjack.v2 v2.2.1 ) -require github.com/conduitio/bwlimit v0.1.0 // indirect - require ( github.com/davecgh/go-spew v1.1.1 // indirect github.com/dustin/go-humanize v1.0.1 // indirect github.com/json-iterator/go v1.1.12 // indirect - github.com/klauspost/compress v1.16.7 // indirect + github.com/klauspost/compress v1.17.1 // indirect github.com/klauspost/cpuid/v2 v2.2.5 // indirect github.com/mattn/go-colorable v0.1.13 // indirect - github.com/mattn/go-isatty v0.0.19 // indirect + github.com/mattn/go-isatty v0.0.20 // indirect github.com/minio/md5-simd v1.1.2 // indirect github.com/minio/sha256-simd v1.0.1 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect @@ -29,10 +28,10 @@ require ( github.com/pmezard/go-difflib v1.0.0 // indirect github.com/rs/xid v1.5.0 // indirect github.com/sirupsen/logrus v1.9.3 // indirect - golang.org/x/crypto v0.12.0 // indirect - golang.org/x/net v0.14.0 // indirect - golang.org/x/sys v0.11.0 // indirect - golang.org/x/text v0.12.0 // indirect - golang.org/x/time v0.3.0 + golang.org/x/crypto v0.14.0 // indirect + golang.org/x/net v0.17.0 // indirect + golang.org/x/sys v0.13.0 // indirect + golang.org/x/text v0.13.0 // indirect + golang.org/x/time v0.3.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect ) diff --git a/go.sum b/go.sum index 83f2556..d56410b 100644 --- a/go.sum +++ b/go.sum @@ -12,18 +12,17 @@ github.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4= github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= -github.com/klauspost/compress v1.16.7 h1:2mk3MPGNzKyxErAw8YaohYh69+pa4sIQSC0fPGCFR9I= -github.com/klauspost/compress v1.16.7/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= +github.com/klauspost/compress v1.17.1 h1:NE3C767s2ak2bweCZo3+rdP4U/HoyVXLv/X9f2gPS5g= +github.com/klauspost/compress v1.17.1/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE= github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/qbZg= github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws= -github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= -github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= -github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA= github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= github.com/minio/minio-go/v7 v7.0.63 h1:GbZ2oCvaUdgT5640WJOpyDhhDxvknAJU2/T3yurwcbQ= @@ -40,8 +39,8 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc= github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= -github.com/rs/zerolog v1.30.0 h1:SymVODrcRsaRaSInD9yQtKbtWqwsfoPcRff/oRXLj4c= -github.com/rs/zerolog v1.30.0/go.mod h1:/tk+P47gFdPXq4QYjvCmT5/Gsug2nagsFWBWhAiSi1w= +github.com/rs/zerolog v1.31.0 h1:FcTR3NnLWW+NnTwwhFWiJSZr4ECLpqCm6QsEnyvbV4A= +github.com/rs/zerolog v1.31.0/go.mod h1:/7mN4D5sKwJLZQ2b/znpjC3/GQWY/xaDXUM0kKWRHss= github.com/shogo82148/go-clockwork-base32 v1.1.0 h1:PEqSTiyVEKdl5ar5RHlUQjFJyJTx9XZD5dUVCLG08K0= github.com/shogo82148/go-clockwork-base32 v1.1.0/go.mod h1:iMSFoMZCgiXZbDyIpEjQxr4T9dH5q8agqxt9HECoiEQ= github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ= @@ -51,20 +50,19 @@ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UV github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= -golang.org/x/crypto v0.12.0 h1:tFM/ta59kqch6LlvYnPa0yx5a83cL2nHflFhYKvv9Yk= -golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= -golang.org/x/net v0.14.0 h1:BONx9s002vGdD9umnlX1Po8vOZmrgH34qlHcD1MfK14= -golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= -golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/crypto v0.14.0 h1:wBqGXzWJW6m1XrIKlAH0Hs1JJ7+9KBwnIO8v66Q9cHc= +golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4= +golang.org/x/net v0.17.0 h1:pVaXccu2ozPjCXewfr1S7xza/zcXTity9cCdXQYSjIM= +golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= -golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= -golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE= +golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= +golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= From 9382f92ef923dbd8e42fc0de1d5a8727d072a587 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Wed, 18 Oct 2023 17:11:06 +0900 Subject: [PATCH 63/79] =?UTF-8?q?config=20=E3=81=AB=20upload=5Ffile=5Frate?= =?UTF-8?q?=5Flimit=5Fmbps=20=E3=82=92=E8=BF=BD=E5=8A=A0=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config_example.ini | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/config_example.ini b/config_example.ini index 82c8f04..c75e5a2 100644 --- a/config_example.ini +++ b/config_example.ini @@ -14,6 +14,11 @@ upload_workers = 4 # 大きめのキャッシュサイズを設定することをおすすめします。 uploaded_file_cache_size = 32 +# 1 ファイルあたりのアップロード速度制限 +# 0 または 10 以上 +# 0 の場合は制限しません +# upload_file_rate_limit_mbps = 0 + # [log] log_dir = . log_name = sora-archive-uploader.jsonl From 10d1b6dff519addfa12579613a2f3a09aff95db9 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Thu, 19 Oct 2023 10:11:04 +0900 Subject: [PATCH 64/79] =?UTF-8?q?=E6=94=B9=E8=A1=8C=E3=82=92=E5=8F=96?= =?UTF-8?q?=E3=82=8A=E9=99=A4=E3=81=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- VERSION | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/VERSION b/VERSION index 57c6e87..f970d5b 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2023.1.0 +2023.1.0 \ No newline at end of file From 3e5ee15f025ee4780ccec4574ebda9da628e8d83 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Fri, 20 Oct 2023 12:49:14 +0900 Subject: [PATCH 65/79] =?UTF-8?q?=E5=88=B6=E9=99=90=E6=99=82=E3=81=AB?= =?UTF-8?q?=E3=82=82=E3=83=AD=E3=82=B0=E3=82=92=E5=87=BA=E5=8A=9B=E3=81=99?= =?UTF-8?q?=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/s3.go b/s3.go index ff76b63..f5c7f96 100644 --- a/s3.go +++ b/s3.go @@ -175,6 +175,10 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO // Save the file size. fileSize := fileStat.Size() + zlog.Debug(). + Str("dst", dst). + Msg("WEB-UPLOAD-START") + // 制限時にはマルチパートアップロードを行わない n, err := s3Client.PutObject(ctx, osConfig.BucketName, dst, fileReader, fileSize, minio.PutObjectOptions{ContentType: "application/octet-stream", DisableMultipart: true}) @@ -182,6 +186,11 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO return "", err } + zlog.Debug(). + Str("dst", dst). + Int64("size", n.Size). + Msg("UPLOAD-WEBM-SUCCESSFULLY") + objectURL := fmt.Sprintf("s3://%s/%s", n.Bucket, n.Key) return objectURL, nil } From 738cc1882c66b2e083204d420079e3662cfd1a20 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Fri, 20 Oct 2023 12:52:53 +0900 Subject: [PATCH 66/79] =?UTF-8?q?=E5=88=B6=E9=99=90=E3=81=AE=E4=B8=8B?= =?UTF-8?q?=E9=99=90=E3=82=92=E5=8F=96=E3=82=8A=E6=89=95=E3=81=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config_example.ini | 1 - runner.go | 5 ----- 2 files changed, 6 deletions(-) diff --git a/config_example.ini b/config_example.ini index c75e5a2..62a35bf 100644 --- a/config_example.ini +++ b/config_example.ini @@ -15,7 +15,6 @@ upload_workers = 4 uploaded_file_cache_size = 32 # 1 ファイルあたりのアップロード速度制限 -# 0 または 10 以上 # 0 の場合は制限しません # upload_file_rate_limit_mbps = 0 diff --git a/runner.go b/runner.go index 74a7dda..d177fa2 100644 --- a/runner.go +++ b/runner.go @@ -151,11 +151,6 @@ func Run(configFilePath *string) { resp.Body.Close() } - // 指定は 0 (制限なし) または 10 Mbps 以上 - if (config.UploadFileRateLimitMbps > 0) && (config.UploadFileRateLimitMbps < 10) { - zlog.Fatal().Msg("UPLOAD-FILE-RATE-LIMIT-MBPS-ERROR") - } - zlog.Info().Msg("STARTED-SORA-ARCHIVE-UPLOADER") // シグナルをキャッチして停止処理 From 4b99e9b7335f2a00f546c83971afae21484df434 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Fri, 20 Oct 2023 12:54:32 +0900 Subject: [PATCH 67/79] =?UTF-8?q?=E4=B8=80=E9=83=A8=E3=81=AE=E3=83=AD?= =?UTF-8?q?=E3=82=B0=E3=81=AE=E3=83=AD=E3=82=B0=E3=83=AC=E3=83=99=E3=83=AB?= =?UTF-8?q?=E3=82=92=E5=A4=89=E6=9B=B4=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- runner.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/runner.go b/runner.go index d177fa2..882fb41 100644 --- a/runner.go +++ b/runner.go @@ -23,7 +23,7 @@ func newMain(config *Config) *Main { func (m *Main) run(ctx context.Context, cancel context.CancelFunc) error { var archiveDir = m.config.SoraArchiveDirFullPath - zlog.Info().Str("path", archiveDir).Msg("WATCHING-ROOT-DIR") + zlog.Debug().Str("path", archiveDir).Msg("WATCHING-ROOT-DIR") fileInfo, err := os.Stat(archiveDir) if err != nil { // 対象のディレクトリが存在しなければ終わる @@ -54,7 +54,7 @@ func (m *Main) run(ctx context.Context, cancel context.CancelFunc) error { if len(foundFiles) == 0 { // 処理対象のファイルが見つからなかったので終わる cancel() - zlog.Info().Msg("ARCHIVE-FILE-NOT-FOUND") + zlog.Debug().Msg("ARCHIVE-FILE-NOT-FOUND") return nil } @@ -151,7 +151,7 @@ func Run(configFilePath *string) { resp.Body.Close() } - zlog.Info().Msg("STARTED-SORA-ARCHIVE-UPLOADER") + zlog.Debug().Msg("STARTED-SORA-ARCHIVE-UPLOADER") // シグナルをキャッチして停止処理 trapSignals := []os.Signal{ @@ -182,5 +182,5 @@ func Run(configFilePath *string) { }() } <-doneShutdown - zlog.Info().Msg("STOPPED-SORA-ARCHIVE-UPLOADER") + zlog.Debug().Msg("STOPPED-SORA-ARCHIVE-UPLOADER") } From 0bee93baa7f979f5e0294ef00817d196e393270c Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Fri, 20 Oct 2023 16:03:14 +0900 Subject: [PATCH 68/79] =?UTF-8?q?=E5=AF=BE=E8=B1=A1=E3=81=AE=20webm=20?= =?UTF-8?q?=E3=83=95=E3=82=A1=E3=82=A4=E3=83=AB=E3=81=AE=E3=83=91=E3=82=B9?= =?UTF-8?q?=E3=82=92=E3=83=AD=E3=82=B0=E3=81=AB=E5=87=BA=E5=8A=9B=E3=81=99?= =?UTF-8?q?=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- uploader.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/uploader.go b/uploader.go index a1be4f5..994a704 100644 --- a/uploader.go +++ b/uploader.go @@ -265,6 +265,10 @@ func (u Uploader) handleArchive(archiveJSONFilePath string, split bool) bool { } defer f.Close() + zlog.Info(). + Str("path", webmFilepath). + Msg("WEBM-FILE-PATH") + metadataFileURL, err := uploadJSONFile( u.ctx, osConfig, From fb1d357a19bb3edb4f56f97349ff0941df3c4ca3 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Fri, 20 Oct 2023 16:16:29 +0900 Subject: [PATCH 69/79] =?UTF-8?q?=E3=82=A2=E3=83=83=E3=83=97=E3=83=AD?= =?UTF-8?q?=E3=83=BC=E3=83=89=E3=81=AE=E9=96=8B=E5=A7=8B=E3=81=A8=E6=88=90?= =?UTF-8?q?=E5=8A=9F=E3=81=AE=E3=83=AD=E3=82=B0=E3=81=AE=E3=83=AD=E3=82=B0?= =?UTF-8?q?=E3=83=AC=E3=83=99=E3=83=AB=E3=82=92=20info=20=E3=81=AB?= =?UTF-8?q?=E5=A4=89=E6=9B=B4=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/s3.go b/s3.go index f5c7f96..444818a 100644 --- a/s3.go +++ b/s3.go @@ -85,7 +85,7 @@ func uploadWebMFile(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, return "", err } - zlog.Debug(). + zlog.Info(). Str("dst", dst). Msg("WEB-UPLOAD-START") n, err := s3Client.FPutObject(ctx, @@ -95,7 +95,7 @@ func uploadWebMFile(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, if err != nil { return "", err } - zlog.Debug(). + zlog.Info(). Str("dst", dst). Int64("size", n.Size). Msg("UPLOAD-WEBM-SUCCESSFULLY") @@ -175,7 +175,7 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO // Save the file size. fileSize := fileStat.Size() - zlog.Debug(). + zlog.Info(). Str("dst", dst). Msg("WEB-UPLOAD-START") @@ -186,7 +186,7 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO return "", err } - zlog.Debug(). + zlog.Info(). Str("dst", dst). Int64("size", n.Size). Msg("UPLOAD-WEBM-SUCCESSFULLY") From f51a8a35ee836590dd5d77fc1dc6fefc7712979d Mon Sep 17 00:00:00 2001 From: "renovate[bot]" <29139614+renovate[bot]@users.noreply.github.com> Date: Sun, 22 Oct 2023 15:51:26 +0000 Subject: [PATCH 70/79] Update actions/checkout action to v4 --- .github/workflows/ci.yaml | 2 +- .github/workflows/release.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 02c07f9..b8a495c 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -11,7 +11,7 @@ jobs: name: ci runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: setup go uses: actions/setup-go@v4 with: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 11a7d8e..71ab422 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -9,7 +9,7 @@ jobs: release: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Set up Go uses: actions/setup-go@v4 From f356d204440bcca74e919d23392478c07662438d Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Mon, 23 Oct 2023 14:49:25 +0900 Subject: [PATCH 71/79] =?UTF-8?q?=E5=8F=AF=E8=83=BD=E3=81=AA=E9=99=90?= =?UTF-8?q?=E3=82=8A=20minio=20=E3=81=AE=20DefaultTransport=20=E3=81=AE?= =?UTF-8?q?=E8=A8=AD=E5=AE=9A=E3=82=92=E4=BD=BF=E7=94=A8=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- s3.go | 14 ++++++++------ s3/s3.go | 24 ++++++++++++++---------- 2 files changed, 22 insertions(+), 16 deletions(-) diff --git a/s3.go b/s3.go index 444818a..1464f97 100644 --- a/s3.go +++ b/s3.go @@ -4,7 +4,6 @@ import ( "context" "fmt" "net" - "net/http" "net/url" "os" "path/filepath" @@ -36,7 +35,7 @@ func uploadJSONFile( creds = credentials.NewIAM("") } - s3Client, err := s3.NewClient(osConfig.Endpoint, creds, nil) + s3Client, err := s3.NewClient(osConfig.Endpoint, creds) if err != nil { return "", err } @@ -80,7 +79,7 @@ func uploadWebMFile(ctx context.Context, osConfig *s3.S3CompatibleObjectStorage, } else { creds = credentials.NewIAM("") } - s3Client, err := s3.NewClient(osConfig.Endpoint, creds, nil) + s3Client, err := s3.NewClient(osConfig.Endpoint, creds) if err != nil { return "", err } @@ -152,10 +151,13 @@ func uploadWebMFileWithRateLimit(ctx context.Context, osConfig *s3.S3CompatibleO KeepAlive: 30 * time.Second, }, rateLimitMByteps, 0) - transport := http.DefaultTransport - transport.(*http.Transport).DialContext = dialer.DialContext + transport, err := s3.DefaultTransport(osConfig.Endpoint) + if err != nil { + return "", err + } + transport.DialContext = dialer.DialContext - s3Client, err := s3.NewClient(osConfig.Endpoint, creds, &transport) + s3Client, err := s3.NewClientWithTransport(osConfig.Endpoint, creds, transport) if err != nil { return "", err } diff --git a/s3/s3.go b/s3/s3.go index 593dcc3..bc2cad7 100644 --- a/s3/s3.go +++ b/s3/s3.go @@ -42,22 +42,26 @@ func maybeEndpointURL(endpoint string) (string, bool) { return endpoint, secure } -func NewClient(endpoint string, credentials *credentials.Credentials, transport *http.RoundTripper) (*minio.Client, error) { - newEndpoint, secure := maybeEndpointURL(endpoint) - if transport == nil { - return minio.New( - newEndpoint, - &minio.Options{ - Creds: credentials, - Secure: secure, - }) +func NewClient(endpoint string, credentials *credentials.Credentials) (*minio.Client, error) { + transport, err := DefaultTransport(endpoint) + if err != nil { + return nil, err } + return NewClientWithTransport(endpoint, credentials, transport) +} +func NewClientWithTransport(endpoint string, credentials *credentials.Credentials, transport http.RoundTripper) (*minio.Client, error) { + newEndpoint, secure := maybeEndpointURL(endpoint) return minio.New( newEndpoint, &minio.Options{ Creds: credentials, Secure: secure, - Transport: *transport, + Transport: transport, }) } + +func DefaultTransport(endpoint string) (*http.Transport, error) { + _, secure := maybeEndpointURL(endpoint) + return minio.DefaultTransport(secure) +} From 7f6ab4c6e9b3b3ead6d922cb14013986368676d6 Mon Sep 17 00:00:00 2001 From: Yoshida Hiroshi Date: Mon, 23 Oct 2023 16:03:08 +0900 Subject: [PATCH 72/79] =?UTF-8?q?go.mod=20=E3=82=92=E6=9B=B4=E6=96=B0?= =?UTF-8?q?=E3=81=99=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- go.mod | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/go.mod b/go.mod index fe3c3f7..04837a3 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/shiguredo/sora-archive-uploader -go 1.21 +go 1.21.3 require ( github.com/conduitio/bwlimit v0.1.0 From ea9b48fa64109fcd8e19074f48a5f0a75a4cff2c Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 23 Oct 2023 16:09:17 +0900 Subject: [PATCH 73/79] 2023.1.0 --- CHANGES.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index 71a72b0..a970540 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -10,3 +10,7 @@ - バグ修正 ## develop + +## 2023.1.0 + +**祝いリリース** From fcb3600bcadfe287cf6325320f05511d87b03a22 Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 23 Oct 2023 16:09:26 +0900 Subject: [PATCH 74/79] =?UTF-8?q?=E3=82=BB=E3=82=AF=E3=82=B7=E3=83=A7?= =?UTF-8?q?=E3=83=B3=E3=81=AF=E8=AA=A4=E8=A7=A3=E3=82=92=E6=8B=9B=E3=81=8F?= =?UTF-8?q?=E3=81=AE=E3=81=A7=E5=89=8A=E9=99=A4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config_example.ini | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/config_example.ini b/config_example.ini index 62a35bf..6880ad1 100644 --- a/config_example.ini +++ b/config_example.ini @@ -18,7 +18,7 @@ uploaded_file_cache_size = 32 # 0 の場合は制限しません # upload_file_rate_limit_mbps = 0 -# [log] +# ログ log_dir = . log_name = sora-archive-uploader.jsonl log_stdout = true @@ -30,9 +30,8 @@ log_rotate_max_backups = 7 log_rotate_max_age = 30 log_rotate_compress = false -# [object_storage] - # アップロード先の S3 または S3 互換オブジェクトストレージの設定 + # object_storage_endpoint = https://s3.example.com # object_storage_bucket_name = bucket-name # object_storage_access_key_id = access-key-id From cb070bf4d7f21dcf29a09e606281d852f62cf649 Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 23 Oct 2023 16:09:56 +0900 Subject: [PATCH 75/79] =?UTF-8?q?=E3=82=B3=E3=82=B9=E3=83=A1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- config_example.ini | 2 -- 1 file changed, 2 deletions(-) diff --git a/config_example.ini b/config_example.ini index 6880ad1..0934b9a 100644 --- a/config_example.ini +++ b/config_example.ini @@ -31,14 +31,12 @@ log_rotate_max_age = 30 log_rotate_compress = false # アップロード先の S3 または S3 互換オブジェクトストレージの設定 - # object_storage_endpoint = https://s3.example.com # object_storage_bucket_name = bucket-name # object_storage_access_key_id = access-key-id # object_storage_secret_access_key = secret-access-key # オブジェクトストレージにアップロードが完了した際に通知するウェブフック -# [webhook] # 空文字列の場合はウェブフックは飛ばさない # webhook_endpoint_url = https://example.com/webhook From 6a3d5759f11669b646883d9ad5ca0b4cea1c8756 Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 23 Oct 2023 16:17:59 +0900 Subject: [PATCH 76/79] =?UTF-8?q?-prod=20=E4=B8=8D=E8=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- script/sora-archive-uploader.service | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/script/sora-archive-uploader.service b/script/sora-archive-uploader.service index 01c4df8..ff23c29 100644 --- a/script/sora-archive-uploader.service +++ b/script/sora-archive-uploader.service @@ -15,7 +15,7 @@ WorkingDirectory=/home/sora/sora-archive-uploader ExecStartPre=/bin/mkdir -p /var/log/sora-archive-uploader ExecStartPre=/bin/chown -R sora:sora /var/log/sora-archive-uploader -ExecStart=/home/sora/sora-archive-uploader/bin/sora-archive-uploader-prod -C /home/sora/sora-archive-uploader/config.ini +ExecStart=/home/sora/sora-archive-uploader/bin/sora-archive-uploader -C /home/sora/sora-archive-uploader/config.ini [Install] WantedBy=multi-user.target \ No newline at end of file From 80a62a6fc73afb523d973580466dc69733242a9c Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 23 Oct 2023 16:21:04 +0900 Subject: [PATCH 77/79] =?UTF-8?q?-dev=20=E3=81=A8=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 94c404a..23eb643 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ make でビルドして実行します。 ```console $ make -$ ./bin/sora-archive-uploader-dev -C config.ini +$ ./bin/sora-archive-uploader -C config.ini ``` ## Discord From eb5829f7562a9b23db5543bed897bcebc6c54a64 Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 23 Oct 2023 16:21:42 +0900 Subject: [PATCH 78/79] =?UTF-8?q?=E3=82=A2=E3=83=94=E3=83=BC=E3=83=AB?= =?UTF-8?q?=E8=BF=BD=E5=8A=A0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 23eb643..677e511 100644 --- a/README.md +++ b/README.md @@ -35,6 +35,7 @@ Sora Cloud では出力されたファイルをオブジェクトストレージ - アップロード完了時に指定された URL にウェブフックリクエストを通知します - ウェブフックにはベーシック認証や mTLS が利用可能です - アップロードに失敗した場合は設定ファイルで指定した隔離ディレクトリに移動します +- アップロードの帯域制限を設定できます ### 対応オブジェクトストレージ From 988b73f83f9175634fedad02f667508425f93ec2 Mon Sep 17 00:00:00 2001 From: voluntas Date: Mon, 23 Oct 2023 16:22:06 +0900 Subject: [PATCH 79/79] =?UTF-8?q?console=20=E3=82=84=E3=82=81=E3=82=8B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 677e511..31c5f78 100644 --- a/README.md +++ b/README.md @@ -51,13 +51,13 @@ Sora Cloud では出力されたファイルをオブジェクトストレージ config.ini に必要な情報を設定してください。 -```console +```bash $ cp config_example.ini config.ini ``` make でビルドして実行します。 -```console +```bash $ make $ ./bin/sora-archive-uploader -C config.ini ```