Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

エクスポートしたカスタム絵文字のzipが大きいと読み込めない #10778

Closed
CyberRex0 opened this issue May 6, 2023 · 9 comments · Fixed by #10780
Closed

エクスポートしたカスタム絵文字のzipが大きいと読み込めない #10778

CyberRex0 opened this issue May 6, 2023 · 9 comments · Fixed by #10780
Labels
packages/backend Server side specific issue/PR ❓needs more investigation A bug whose causes are unknown

Comments

@CyberRex0
Copy link
Contributor

CyberRex0 commented May 6, 2023 •
edited
Loading

💡 Summary

絵文字をエクスポートした際、zipファイルのサイズが大きいと、インポート時の解凍で何らかの理由で読み込めない、または壊れたファイルを読み込むような挙動をする。

🥰 Expected Behavior

正常にインポートできる

🤬 Actual Behavior

meta.jsonが正しいJSONとして読み込まれずに、インポートが失敗してしまう。

パターン1

SyntaxError: Unexpected non-whitespace character after JSON at position 1
     at JSON.parse (<anonymous>)
     at DuplexWrapper.<anonymous> (file:///home/misskey/misskey/packages/backend/built/queue/processors/ImportCustomEmojisProcessorService.js:68:31)
     at DuplexWrapper.emit (node:events:513:28)
     at Writable.<anonymous> (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/extract.js:44:15)
    at Writable.emit (node:events:525:35)
     at finish (node:internal/streams/writable:750:10)
     at finishMaybe (node:internal/streams/writable:735:9)
     at afterWrite (node:internal/streams/writable:507:3)
     at onwrite (node:internal/streams/writable:480:7)
     at ProxyWriter.emit (node:events:525:35)
     at FileWriter.emit (node:events:513:28)
     at end (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:324:14)
     at /home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:314:34
     at endUtimes (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:238:46)
     at setProps (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:297:5)
     at /home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:287:7
     at callback (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/graceful-fs/polyfills.js:306:20)
     at FSReqCallback.oncomplete (node:fs:196:5)

パターン2

SyntaxError: Unexpected token 'p', "pdatedAt":"... is not valid JSON
     at JSON.parse (<anonymous>)
    at DuplexWrapper.<anonymous> (file:///home/misskey/misskey/packages/backend/built/queue/processors/ImportCustomEmojisProcessorService.js:68:31)
     at DuplexWrapper.emit (node:events:513:28)
     at Writable.<anonymous> (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/unzipper/lib/extract.js:44:15)
     at Writable.emit (node:events:525:35)
     at finish (node:internal/streams/writable:750:10)
     at finishMaybe (node:internal/streams/writable:735:9)
     at afterWrite (node:internal/streams/writable:507:3)
     at onwrite (node:internal/streams/writable:480:7)
     at ProxyWriter.emit (node:events:525:35)
     at FileWriter.emit (node:events:513:28)
     at end (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:324:14)
     at /home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:314:34
     at endUtimes (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:238:46)
     at setProps (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:297:5)
     at /home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/fstream/lib/writer.js:287:7
     at callback (/home/misskey/misskey/node_modules/.pnpm/[email protected]/node_modules/graceful-fs/polyfills.js:306:20)
     at FSReqCallback.oncomplete (node:fs:196:5)

📝 Steps to Reproduce

  1. エクスポートしたzipを用意する
  2. インポートする

Memo

別なサーバーそれぞれ2つ (v13.10.3、v13.12.0-beta.3)からエクスポートしたzipを試したが両方ダメだった。
両方とも10MBを超えている。

一方で、1MB未満のzipでは正常にインポートできた。

なお、正常にインポートできたzipも、できなかったzipもunzipコマンドやファイルマネージャーで解凍可能。
meta.jsonも破損しておらず、読み込むことが可能。

📌 Environment

Misskey version: 13.12.0-beta.3
Node.js version: v19.9.0
Your OS: macOS Ventura 13.3.1(a)
Your browser: Chrome 112.0.5615.137 (Official Build) （arm64）
@CyberRex0 CyberRex0 added the ⚠️bug? This might be a bug label May 6, 2023
@acid-chicken
Copy link
Member

原因の切り分けができないので、可能であれば読み込めないデータを添付ください

@CyberRex0
Copy link
Contributor Author

インポートできなかったもの
https://pub-560e65a56a294433907ae17133312204.r2.dev/custom-emojis-2023-05-04-22-36-37.zip (32.87MB)
https://pub-560e65a56a294433907ae17133312204.r2.dev/custom-emojis-2023-05-06-17-50-17.zip (14.39MB)

インポートできたもの
https://pub-560e65a56a294433907ae17133312204.r2.dev/custom-emojis-2023-05-06-17-33-15.zip (11.76KB)

@acid-chicken
Copy link
Member

メモ

Node.js の挙動が怪しい

Node.js Version Result
v16.20.0 ⭕
v17.9.1 ⭕
v18.8.0 ⭕
v18.10.0 ⭕
v18.12.1 ⭕
v18.14.2 ⭕
v18.15.0 ⭕
v18.16.0 ❌
v19.0.1 ⭕
v19.6.1 ⭕
v19.7.0 ⭕
v19.8.0 ❌
v19.8.1 ❌
v19.9.0 ❌
v20.1.0 ❌

@acid-chicken
Copy link
Member

とりあえずワークアラウンドとしては Node.js v18.15.0 (v18.16.0 ではない) で起動していると回避できそう

@CyberRex0
Copy link
Contributor Author

CyberRex0 commented May 6, 2023 •
edited
Loading

問題が起きたのはv19.9.0、v18.16.0なので図星かもしれない

@acid-chicken acid-chicken added ❓needs more investigation A bug whose causes are unknown and removed ⚠️bug? This might be a bug labels May 6, 2023
@acid-chicken
Copy link
Member

ストリームの頭が切れるみたいな挙動になるっぽい
どれが原因か特定面倒だし unzipper やめるか

@acid-chicken acid-chicken mentioned this issue May 6, 2023
Merged
5 tasks
@acid-chicken acid-chicken added the packages/backend Server side specific issue/PR label May 7, 2023
@yuriha-chan
Copy link
Contributor

参考: ZJONSSON/node-unzipper#271
zip-streamを使うというworkaroundでもよいらしい

@acid-chicken acid-chicken mentioned this issue Jul 6, 2023
Closed
@kanasaki15
Copy link

どうなりました？

@acid-chicken
Copy link
Member

どうなりました？

#10780 出したけど、当時絵文字のバルクインポート自体がライセンスの観点で問題視されていて誰もレビューする気がなかったので、それから壊れたままの状態を意図的に維持されてしまっている状況

@tamaina tamaina mentioned this issue Jul 20, 2023
Open
@kanade kanade mentioned this issue Nov 23, 2023
Closed
@acid-chicken acid-chicken mentioned this issue Jun 2, 2024
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
packages/backend Server side specific issue/PR ❓needs more investigation A bug whose causes are unknown
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(backend): use slacc instead of unzipper misskey-dev/misskey
4 participants
@kanasaki15 @acid-chicken @CyberRex0 @yuriha-chan