Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e51d77a
feat(cli): Add --split-output option
Dango233 Dec 18, 2025
3216450
docs(cli): Document --split-output
Dango233 Dec 18, 2025
eb62d4b
refactor(cli): Abstract conflicting options validation
yamadashy Dec 18, 2025
f143aeb
docs(readme): Add Splitting Output for Large Codebases section
yamadashy Dec 21, 2025
bc93d72
docs(website): Enhance --split-output documentation
yamadashy Dec 21, 2025
4303b07
chore(gitignore): Add pattern for split output files
yamadashy Dec 21, 2025
db146b9
fix(cli): Address PR review comments for --split-output
yamadashy Dec 21, 2025
2227230
fix(cli): Address additional PR review comments
yamadashy Dec 21, 2025
a49a8f9
test(cli): Add generateSplitOutputParts error case tests
yamadashy Dec 21, 2025
5f9c22e
refactor(core): Extract output generation logic from packager
yamadashy Dec 21, 2025
e780bca
test(core): Add produceOutput unit tests
yamadashy Dec 21, 2025
375da20
refactor(core): Extract inner functions from generateSplitOutputParts
yamadashy Dec 21, 2025
dd7717b
feat(shared): Support decimal values in size parsing
yamadashy Dec 21, 2025
09e8d39
docs(readme,website): Add decimal size examples for --split-output
yamadashy Dec 21, 2025
b27d1c3
perf(output): Cache compiled Handlebars templates
yamadashy Dec 21, 2025
fd84483
docs(cli): Add decimal size example to --split-output help text
yamadashy Dec 21, 2025
aacc4fa
feat(output): Improve split output progress display
yamadashy Dec 21, 2025
5f8be42
perf(output): Cache git file change counts in sortOutputFiles
yamadashy Dec 21, 2025
0364f3c
test(output): Add split output success pattern tests
yamadashy Dec 21, 2025
05c3245
test(output): Add getRootEntry edge case tests
yamadashy Dec 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ repomix-output.txt
repomix-output.xml
repomix-output.md
repomix-output.json
repomix-output.*

# ESLint cache
.eslintcache
Expand Down
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -626,6 +626,7 @@ Instruction
- `--truncate-base64`: Truncate long base64 data strings to reduce output size
- `--header-text <text>`: Custom text to include at the beginning of the output
- `--instruction-file-path <path>`: Path to file containing custom instructions to include in output
- `--split-output <size>`: Split output into multiple numbered files (e.g., repomix-output.1.xml, repomix-output.2.xml); size like 500kb, 2mb, or 1.5mb
- `--include-empty-directories`: Include folders with no files in directory structure
- `--include-full-directory-structure`: Show complete directory tree in output, including files not matched by --include patterns
- `--no-git-sort-by-changes`: Don't sort files by git change frequency (default: most changed files first)
Expand Down Expand Up @@ -682,6 +683,9 @@ repomix --compress
# Process specific files
repomix --include "src/**/*.ts" --ignore "**/*.test.ts"

# Split output into multiple files (max size per part)
repomix --split-output 20mb

# Remote repository with branch
repomix --remote https://github.com/user/repo/tree/main

Expand Down Expand Up @@ -837,6 +841,24 @@ This helps you:
- **Plan compression strategies** by targeting the largest contributors
- **Balance content vs. context** when preparing code for AI analysis

### Splitting Output for Large Codebases

When working with large codebases, the packed output may exceed file size limits imposed by some AI tools (e.g., Google AI Studio's 1MB limit). Use `--split-output` to automatically split the output into multiple files:

```bash
repomix --split-output 1mb
```

This generates numbered files like:
- `repomix-output.1.xml`
- `repomix-output.2.xml`
- `repomix-output.3.xml`

Size can be specified with units: `500kb`, `1mb`, `2mb`, `1.5mb`, etc. Decimal values are supported.

> [!NOTE]
> Files are grouped by top-level directory to maintain context. A single file or directory will never be split across multiple output files.

### MCP Server Integration

Repomix supports the [Model Context Protocol (MCP)](https://modelcontextprotocol.io), allowing AI assistants to directly interact with your codebase. When run as an MCP server, Repomix provides tools that enable AI assistants to package local or remote repositories for analysis without requiring manual file preparation.
Expand Down
64 changes: 53 additions & 11 deletions src/cli/actions/defaultAction.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,19 +52,11 @@ export const runDefaultAction = async (
const config: RepomixConfigMerged = mergeConfigs(cwd, fileConfig, cliConfig);
logger.trace('Merged config:', config);

// Validate conflicting options
validateConflictingOptions(config);

// Validate skill generation options and prompt for location
if (config.skillGenerate !== undefined) {
if (config.output.stdout) {
throw new RepomixError(
'--skill-generate cannot be used with --stdout. Skill output requires writing to filesystem.',
);
}
if (config.output.copyToClipboard) {
throw new RepomixError(
'--skill-generate cannot be used with --copy. Skill output is a directory and cannot be copied to clipboard.',
);
}

// Resolve skill name: use pre-computed name (from remoteAction) or generate from directory
cliOptions.skillName ??=
typeof config.skillGenerate === 'string'
Expand Down Expand Up @@ -274,6 +266,13 @@ export const buildCliConfig = (options: CliOptions): RepomixConfigCli => {
};
}

if (options.splitOutput !== undefined) {
cliConfig.output = {
...cliConfig.output,
splitOutput: options.splitOutput,
};
}

// Only apply gitSortByChanges setting if explicitly set to false
if (options.gitSortByChanges === false) {
cliConfig.output = {
Expand Down Expand Up @@ -367,3 +366,46 @@ const waitForWorkerReady = async (taskRunner: {
logger.debug('All Worker ping attempts failed, proceeding anyway...');
}
};

/**
* Validates that conflicting CLI options are not used together.
* Throws RepomixError if incompatible options are detected.
*/
const validateConflictingOptions = (config: RepomixConfigMerged): void => {
const isStdoutMode = config.output.stdout || config.output.filePath === '-';

// Define option states for conflict checking
const options = {
splitOutput: {
enabled: config.output.splitOutput !== undefined,
name: '--split-output',
},
skillGenerate: {
enabled: config.skillGenerate !== undefined,
name: '--skill-generate',
},
stdout: {
enabled: isStdoutMode,
name: '--stdout',
},
copy: {
enabled: config.output.copyToClipboard,
name: '--copy',
},
};

// Define conflicts: [optionA, optionB, errorMessage]
const conflicts: [keyof typeof options, keyof typeof options, string][] = [
['splitOutput', 'stdout', 'Split output requires writing to filesystem.'],
['splitOutput', 'skillGenerate', 'Skill output is a directory.'],
['splitOutput', 'copy', 'Split output generates multiple files.'],
['skillGenerate', 'stdout', 'Skill output requires writing to filesystem.'],
['skillGenerate', 'copy', 'Skill output is a directory and cannot be copied to clipboard.'],
];

for (const [optionA, optionB, message] of conflicts) {
if (options[optionA].enabled && options[optionB].enabled) {
throw new RepomixError(`${options[optionA].name} cannot be used with ${options[optionB].name}. ${message}`);
}
}
};
17 changes: 14 additions & 3 deletions src/cli/cliReport.ts
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,20 @@ export const reportSummary = (
const displayPath = getDisplayPath(options.skillDir, cwd);
logger.log(`${pc.white(' Output:')} ${pc.white(displayPath)} ${pc.dim('(skill directory)')}`);
} else {
const outputPath = path.resolve(cwd, config.output.filePath);
const displayPath = getDisplayPath(outputPath, cwd);
logger.log(`${pc.white(' Output:')} ${pc.white(displayPath)}`);
if (packResult.outputFiles && packResult.outputFiles.length > 0) {
const first = packResult.outputFiles[0];
const last = packResult.outputFiles[packResult.outputFiles.length - 1];
const firstDisplayPath = getDisplayPath(path.resolve(cwd, first), cwd);
const lastDisplayPath = getDisplayPath(path.resolve(cwd, last), cwd);

logger.log(
`${pc.white(' Output:')} ${pc.white(firstDisplayPath)} ${pc.dim('…')} ${pc.white(lastDisplayPath)} ${pc.dim(`(${packResult.outputFiles.length} parts)`)}`,
);
} else {
const outputPath = path.resolve(cwd, config.output.filePath);
const displayPath = getDisplayPath(outputPath, cwd);
logger.log(`${pc.white(' Output:')} ${pc.white(displayPath)}`);
}
}
logger.log(`${pc.white(' Security:')} ${pc.white(securityCheckMessage)}`);

Expand Down
7 changes: 7 additions & 0 deletions src/cli/cliRun.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import pc from 'picocolors';
import { getVersion } from '../core/file/packageJsonParse.js';
import { handleError, RepomixError } from '../shared/errorHandle.js';
import { logger, repomixLogLevels } from '../shared/logger.js';
import { parseHumanSizeToBytes } from '../shared/sizeParse.js';
import { runDefaultAction } from './actions/defaultAction.js';
import { runInitAction } from './actions/initAction.js';
import { runMcpAction } from './actions/mcpAction.js';
Expand Down Expand Up @@ -116,6 +117,12 @@ export const run = async () => {
.option('--truncate-base64', 'Truncate long base64 data strings to reduce output size')
.option('--header-text <text>', 'Custom text to include at the beginning of the output')
.option('--instruction-file-path <path>', 'Path to file containing custom instructions to include in output')
.addOption(
new Option(
'--split-output <size>',
'Split output into multiple numbered files (e.g., repomix-output.1.xml, repomix-output.2.xml); size like 500kb, 2mb, or 2.5mb',
).argParser(parseHumanSizeToBytes),
)
.option('--include-empty-directories', 'Include folders with no files in directory structure')
.option(
'--include-full-directory-structure',
Expand Down
1 change: 1 addition & 0 deletions src/cli/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export interface CliOptions extends OptionValues {
instructionFilePath?: string;
includeEmptyDirectories?: boolean;
includeFullDirectoryStructure?: boolean;
splitOutput?: number; // bytes
gitSortByChanges?: boolean;
includeDiffs?: boolean;
includeLogs?: boolean;
Expand Down
2 changes: 2 additions & 0 deletions src/config/configSchema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ export const repomixConfigBaseSchema = z.object({
copyToClipboard: z.boolean().optional(),
includeEmptyDirectories: z.boolean().optional(),
includeFullDirectoryStructure: z.boolean().optional(),
splitOutput: z.number().int().min(1).optional(),
tokenCountTree: z.union([z.boolean(), z.number(), z.string()]).optional(),
git: z
.object({
Expand Down Expand Up @@ -100,6 +101,7 @@ export const repomixConfigDefaultSchema = z.object({
copyToClipboard: z.boolean().default(false),
includeEmptyDirectories: z.boolean().optional(),
includeFullDirectoryStructure: z.boolean().default(false),
splitOutput: z.number().int().min(1).optional(),
tokenCountTree: z.union([z.boolean(), z.number(), z.string()]).default(false),
git: z.object({
sortByChanges: z.boolean().default(true),
Expand Down
19 changes: 15 additions & 4 deletions src/core/metrics/calculateMetrics.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import type { RepomixProgressCallback } from '../../shared/types.js';
import type { ProcessedFile } from '../file/fileTypes.js';
import type { GitDiffResult } from '../git/gitDiffHandle.js';
import type { GitLogResult } from '../git/gitLogHandle.js';
import { buildSplitOutputFilePath } from '../output/outputSplit.js';
import { calculateGitDiffMetrics } from './calculateGitDiffMetrics.js';
import { calculateGitLogMetrics } from './calculateGitLogMetrics.js';
import { calculateOutputMetrics } from './calculateOutputMetrics.js';
Expand All @@ -22,7 +23,7 @@ export interface CalculateMetricsResult {

export const calculateMetrics = async (
processedFiles: ProcessedFile[],
output: string,
output: string | string[],
progressCallback: RepomixProgressCallback,
config: RepomixConfigMerged,
gitDiffResult: GitDiffResult | undefined,
Expand All @@ -47,6 +48,7 @@ export const calculateMetrics = async (
});

try {
const outputParts = Array.isArray(output) ? output : [output];
// For top files display optimization: calculate token counts only for top files by character count
// However, if tokenCountTree is enabled, calculate for all files to avoid double calculation
const topFilesLength = config.output.topFilesLength;
Expand All @@ -62,21 +64,30 @@ export const calculateMetrics = async (
.slice(0, Math.min(processedFiles.length, Math.max(topFilesLength * 10, topFilesLength)))
.map((file) => file.path);

const [selectiveFileMetrics, totalTokens, gitDiffTokenCount, gitLogTokenCount] = await Promise.all([
const [selectiveFileMetrics, outputTokenCounts, gitDiffTokenCount, gitLogTokenCount] = await Promise.all([
deps.calculateSelectiveFileMetrics(
processedFiles,
metricsTargetPaths,
config.tokenCount.encoding,
progressCallback,
{ taskRunner },
),
deps.calculateOutputMetrics(output, config.tokenCount.encoding, config.output.filePath, { taskRunner }),
Promise.all(
outputParts.map(async (part, index) => {
const partPath =
outputParts.length > 1
? buildSplitOutputFilePath(config.output.filePath, index + 1)
: config.output.filePath;
return await deps.calculateOutputMetrics(part, config.tokenCount.encoding, partPath, { taskRunner });
}),
),
deps.calculateGitDiffMetrics(config, gitDiffResult, { taskRunner }),
deps.calculateGitLogMetrics(config, gitLogResult, { taskRunner }),
]);

const totalTokens = outputTokenCounts.reduce((sum, count) => sum + count, 0);
const totalFiles = processedFiles.length;
const totalCharacters = output.length;
const totalCharacters = outputParts.reduce((sum, part) => sum + part.length, 0);

// Build character counts for all files
const fileCharCounts: Record<string, number> = {};
Expand Down
46 changes: 30 additions & 16 deletions src/core/output/outputGenerate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,35 @@ import { getMarkdownTemplate } from './outputStyles/markdownStyle.js';
import { getPlainTemplate } from './outputStyles/plainStyle.js';
import { getXmlTemplate } from './outputStyles/xmlStyle.js';

// Cache for compiled Handlebars templates to avoid recompilation on every call
const compiledTemplateCache = new Map<string, Handlebars.TemplateDelegate>();

const getCompiledTemplate = (style: string): Handlebars.TemplateDelegate => {
const cached = compiledTemplateCache.get(style);
if (cached) {
return cached;
}

let template: string;
switch (style) {
case 'xml':
template = getXmlTemplate();
break;
case 'markdown':
template = getMarkdownTemplate();
break;
case 'plain':
template = getPlainTemplate();
break;
default:
throw new RepomixError(`Unsupported output style for handlebars template: ${style}`);
}

const compiled = Handlebars.compile(template);
compiledTemplateCache.set(style, compiled);
return compiled;
};

const calculateMarkdownDelimiter = (files: ReadonlyArray<ProcessedFile>): string => {
const maxBackticks = files
.flatMap((file) => file.content.match(/`+/g) ?? [])
Expand Down Expand Up @@ -190,23 +219,8 @@ const generateHandlebarOutput = async (
renderContext: RenderContext,
processedFiles?: ProcessedFile[],
): Promise<string> => {
let template: string;
switch (config.output.style) {
case 'xml':
template = getXmlTemplate();
break;
case 'markdown':
template = getMarkdownTemplate();
break;
case 'plain':
template = getPlainTemplate();
break;
default:
throw new RepomixError(`Unsupported output style for handlebars template: ${config.output.style}`);
}

try {
const compiledTemplate = Handlebars.compile(template);
const compiledTemplate = getCompiledTemplate(config.output.style);
return `${compiledTemplate(renderContext).trim()}\n`;
} catch (error) {
if (error instanceof RangeError && error.message === 'Invalid string length') {
Expand Down
Loading
Loading