Payload v3 accepts only single file uploads, export fetchAPIFileUpload as a workaround

[max-degterev]

Describe the Bug

Payload v2 used to be able to handle any shape of multipart data and parse files accordingly. v3 is only able to find a single file named file.

Ideally as a long term solution it would be great to have payload automatically figure out data types passed in a multipart body and create matching data structure. So just req.files = files and done I guess...

As a quick and dirty fix it would be also great to just expose fetchAPIFileUpload so that end users can easily create custom body parsers while still using global payload upload configuration.

Link to the code that reproduces this issue

https://github.com/payloadcms/payload/blob/main/packages/next/src/fetchAPI-multipart/index.ts

Reproduction Steps

upgrade to v3
make sad noises attempting to migrate file uploads

Which area(s) are affected? (Select all that apply)

area: core

Environment Info

v3

[max-degterev]

Temporary hackfix:

import fs from 'node:fs/promises';

const packages = {
  "@payloadcms/next": {
    "exports": {
      "./fetchAPI-multipart": {
        "import": "./dist/fetchAPI-multipart/index.js",
        "types": "./dist/fetchAPI-multipart/index.d.ts",
        "default": "./dist/fetchAPI-multipart/index.js"
      },
    },
  },
};

const deepMerge = (target, overrides) => {
  for (const [key, value] of Object.entries(overrides)) {
    if (!target[key]) target[key] = {};
    if (typeof value === 'object') return deepMerge(target[key], value);
    target[key] = value;
  }
};

const postinstall = async () => Object.entries(packages).forEach(async ([name, patches]) => {
  const packageJsonPath = `node_modules/${name}/package.json`;

  const packageString = await fs.readFile(packageJsonPath, 'utf8');
  const packageJson = JSON.parse(packageString);

  deepMerge(packageJson, patches);

  await fs.writeFile(packageJsonPath, JSON.stringify(packageJson, null, 2));
});

postinstall();
console.log(`Applied patches to ${Object.keys(packages).length} packages`);

run this in "postinstall"

[max-degterev]

And this brings back the old style of body parsing, automatically finding data field and safely parsing as JSON as a bonus

import { set } from 'lodash-es';
import { APIError, type PayloadRequest } from 'payload';
import { fetchAPIFileUpload, type FetchAPIFileUploadResponse } from '@payloadcms/next/fetchAPI-multipart';
import { v4 as uuidv4 } from 'uuid';

type UploadedFile = FetchAPIFileUploadResponse['files'][string];
type NestedFiles = {
  // Not using Record because TS fails to handle circular references
  [key: string]: UploadedFile | NestedFiles,
};

export type PayloadRequestWithFiles = Omit<PayloadRequest, 'file'> & {
  files?: NestedFiles,
};

const createFilesStructure = (files: FetchAPIFileUploadResponse['files']) => {
  const result: Record<string, UploadedFile> = {};
  Object.entries(files).forEach(([key, file]) => {
    if (!file || !file.size || !file.data?.length) return; // Uploading empty file

    const parts = (file.name || 'file').split('.');
    const random = uuidv4().slice(0, 8);
    const extension = parts.pop();

    // Patch file name to include random string, avoiding collisions
    const mutable = file;
    mutable.name = `${parts.join('.')}-${random}.${extension}`;

    set(result, key, mutable);
  });
  return result;
};

export const addDataAndFilesToRequest = async (req: PayloadRequest) => {
  const { headers, method, payload } = req;
  const mutableReq = req as PayloadRequestWithFiles;
  const bodyByteSize = parseInt(req.headers.get('Content-Length') || '0', 10);

  if (method && ['PATCH', 'POST', 'PUT', 'DELETE'].includes(method.toUpperCase()) && bodyByteSize) {
    const [contentType] = (headers.get('Content-Type') || '').split(';');

    if (contentType === 'application/json') {
      const bodyText = await req.text?.();
      const data = tryParse<JsonObject>(bodyText, {})!;
      mutableReq.data = data;
      mutableReq.json = () => Promise.resolve(data);
    } else if (contentType.includes('multipart/')) {
      const { error, fields, files } = await fetchAPIFileUpload({
        options: payload.config.upload,
        request: req as Request,
      });

      if (error) throw new APIError(error.message);
      if (files) mutableReq.files = createFilesStructure(files);

      // Guessing that the string field is JSON compatible, not throwing an error if it's not
      const maybeJSON = Object.values(fields).find((value) => typeof value === 'string');
      const data = tryParse<JsonObject>(maybeJSON, {})!;
      mutableReq.data = data;
      mutableReq.json = () => Promise.resolve(data);
    }
  }
};

[max-degterev]

Additional issue: if filename contains a # it will not be processed by s3 correctly resulting in a failed upload. Try uploading attached file: Trollface # 31337.png.zip

To summarize all issues with addDataAndFileToRequest:

1. Only accepts 1 file
2. Doesn't find JSON data unless it's _payload
3. Only allows 'PATCH'/'POST'/'PUT' although it's also possible to attach body to 'DELETE' (See https://stackoverflow.com/questions/299628/is-an-entity-body-allowed-for-an-http-delete-request) (tbh I'd just remove the check entirely)
4. Explodes if 'application/json' is sent with empty body
5. Doesn't catch if multipart is sent with 0 byte file
6. Doesn't sanitize filenames resulting in failed uploads to s3
7. (?) Not sure if a bug, but collisions are not checked here, perhaps it's also better to check in the cloud plugin so just FYI

[max-degterev]

Hey guys any chance to get fetchAPI-multipart exported in the meantime?

[RickGeersing]

What is the status of this issue? This blocks an upgrade for one of our customers

[RickGeersing]

Got mine to work without the need of a post install script, but with a direct node_modules import. I also implemented custom logic that layers form data like v2 did, with the added bonus that adds files directly alongside those fields.

import { APIError, PayloadRequest } from "payload"
import { fetchAPIFileUpload } from "node_modules/payload/dist/uploads/fetchAPI-multipart"

export const addDataAndFilesToRequest = async (req: PayloadRequest): Promise<void> => {
    const { body, headers, method, payload  } = req

  if (method && ['PATCH', 'POST', 'PUT'].includes(method.toUpperCase()) && body) {
    const [contentType] = (headers.get('Content-Type') || '').split(';')
    const bodyByteSize = parseInt(req.headers.get('Content-Length') || '0', 10)

    if (contentType === 'application/json') {
        let data = {}
        try {
          data = await req?.json?.()
        } catch (error) {
          req.payload.logger.error(error)
        } finally {
          req.data = data
          req.json = () => Promise.resolve(data)
        }
    } else if (bodyByteSize && contentType.includes('multipart/')) {
      const { error, fields, files } = await fetchAPIFileUpload({
        options: payload.config.upload,
        request: req as Request,
      })

      if (error) {
        throw new APIError(error.message)
      }

      const data = {
        ...fields || {},
        ...files || {},
      }
      
      req.data = Object.entries(data).reduce((acc, [key, value]) => {
        const parts = key.split('.')

        parts.reduce((acc: Record<string, any>, part, index) => {
          if (index === parts.length - 1) {
            acc[part] = value
            return
          }

          if (!acc[part]) {
            acc[part] = {}
          }

          return acc[part]
        }, acc)

        return acc
      }, {})
    }
  }
}

[max-degterev]

Nice one, I only take a single JSON field, no reason not to parse all. though stripe tends to shoot 0 length POST at my endpoints from time to time, keep that in mind (will crash)