fallow dupes

fallow dupes finds copy-pasted code blocks across your whole codebase. Each clone group shows you code that you can extract into one shared function. You can also fail CI when duplication is more than a threshold. Fallow uses suffix-array analysis to find the clones.

fallow dupes

Options

Detection

Output

Incremental

When a quarter or more of the saved entries match no current clone group, the run prints a warning on stderr with the command to save the baseline again. A run narrowed by --changed-since or --production prints no warning, because it cannot judge a whole-project baseline. --workspace, --changed-workspaces, --diff-file and a positional path narrow only the rendered report. The comparison, and a new save, still cover the whole project.

When the run loaded a baseline, the JSON output of an analysis run has a baseline_staleness object. It contains:

• the counts
• current_findings
• change_scoped
• the advisory result as warning
• gate_trips, which applies the rule of --fail-on-stale-baseline, so a CI integration reads one boolean and does not need to know the rule

fallow report --from renders a saved envelope again, and it includes the object only when that envelope already had it. Read change_scoped before you divide the counts: a narrowed run can report matched_entries: 0 on a healthy baseline.

scope_reasons names what narrowed the run, so you do not have to guess which input caused it. It is present and not empty exactly when change_scoped is true. dupes compares before the report filters run, so it reports only the inputs that narrow the analysis: changed-files for a resolved changed-file set, and production. Each command reports what it can see, so the names differ per command. Expect new names in the set.

A run can load a baseline with entries and use only --changed-since. Such a run also adds a recheck-baseline entry to next_steps[] with the unscoped command. That command reads the baseline again and reports. It never saves the baseline again. A run in production mode adds no entry. Production mode can come from the config and the environment as well as from a flag, so the command in the entry would be just as narrow.

Each command writes its own baseline format, and a saved baseline names that format. Every file that --save-baseline writes has a top-level kind of dead-code, dupes or health. A baseline from an earlier release has no kind and still loads. For such a file, its keys decide the format.

fallow dupes --baseline loads a baseline of another kind as zero clone groups, so no entry stays unmatched. Such a run prints a warning on stderr, with or without --quiet, that names both kinds and the path. It reports baseline_staleness.unrecognised_format: true and suppresses nothing. Next to it, saved_by names the command that saved the file: dead-code, dupes, or health. The saved_by field is absent for an empty file, for a baseline saved before files named their writer, and for a writer that this version does not know. The field is present only when it is true. An absent field means false. Read this field and not baseline_entries == 0, because a baseline from a project with no duplication is correctly empty. The baseline gate trips on such a file, so --fail-on-stale-baseline exits 1. A run with no gate turned on keeps its exit code.

Token cache

On large projects, fallow dupes caches tokenized source, so later runs do not tokenize unchanged files again. The cache is on for projects with at least duplicates.minCorpusSizeForTokenCache source files (default 5000). It is stored per project under <root>/.fallow/cache/dupes-tokens-vN/. On the first save, the cache writes a .gitignore file with * next to it, so git status does not show its contents.

• To disable the cache for one run, pass --no-cache.
• To keep the cache off on larger projects, set a higher minCorpusSizeForTokenCache.

Below the threshold, the cache stays off: on small projects, loading and saving the cache costs more time than it saves.

With --changed-since, on a project with at least duplicates.minCorpusSizeForShingleFilter files (default 1024), fallow also drops some unchanged files before it builds the suffix array. It drops each unchanged file whose k-token shingles do not overlap any changed file. You set both options in the [duplicates] section of .fallowrc.json, .fallowrc.jsonc, fallow.toml, or .fallow.toml.

Default generated-output ignores

By default, fallow dupes skips common framework output: .next, .nuxt, .svelte-kit, .turbo, .parcel-cache, .vite, .cache, out, and storybook-static. Fallow adds these defaults to duplicates.ignore. They do not replace your configured ignores.

By default, fallow also skips test and mock files: *.test.*, *.spec.*, __tests__/, and __mocks__/ paths. Test setup is often repetitive on purpose, and it causes much noise on the first run of a hand-coded project. These patterns are part of the same ignoreDefaults set as the generated-output ignores.

{
  "duplicates": {
    "ignoreDefaults": true,
    "ignore": ["**/__generated__/**"]
  }
}

To opt out and use only your duplicates.ignore list, set ignoreDefaults to false.

Reviewed clone groups

To accept one clone group and keep all other duplication findings, add the group to duplicates.ignoredClones:

{
  "duplicates": {
    "ignoredClones": ["dup:6f12ab34:2"]
  }
}

The key is <fingerprint>:<instance_count>. Copy the fingerprint from the human listing or from clone_groups[].fingerprint in JSON. When the tokens change or a new occurrence appears, fallow reports the group again. An edit that changes only the formatting keeps the same fingerprint.

Collision handles use dup:<16hex>-rN. Ordinary short and widened handles do not change. Older dup:<16hex>-N handles no longer select or suppress a group.

• To replace a saved key, run fallow dupes again and review the reported group. Do not insert r into the old handle.
• Upgrade all runners before you add -rN keys to a shared ignoredClones configuration.

For the baseline behavior, see migrating collision handles.

Monorepo scoping

Both flags keep only the clone groups with at least one instance under a selected workspace root. Fallow still builds the full cross-workspace index. It then computes the reported groups and the duplication_percentage stat again from the scoped part.

Debugging

Detection modes

Ranking and JSON fields

The default report and --top rank clone groups by duplicated token count times occurrence count, with a spread boost. spread is the maximum directory-tree distance between instances. In a same-file group, each 250 lines between instances count as one spread step. The boost is at most 15%.

Each JSON clone_groups[] entry has these fields:

• spread: maximum directory or same-file line distance
• similarity: lowest all-pairs similarity for near-miss groups, omitted for exact groups

Each entry in clone_groups[].instances[] can also have is_symlink: true. See Symlinked files.

Shown versus measured

stats always describes the corpus that the run measured. A presentation cap such as --top shortens the arrays and does not change stats. Four counters on the envelope report the split:

clone_groups_shown + clone_groups_omitted always equals stats.clone_groups. In the same way, the family counters add up to stats.clone_families. Read clone_groups_shown for the number of groups in the array. Read stats.clone_groups for the number of groups in the project.

Scope filters work differently from presentation caps. --changed-since, --workspace, and a duplication baseline narrow the corpus itself. Fallow computes stats again for the narrowed corpus, and reports nothing as omitted.

The human output gives the same information in words. A capped run ends the listing with the counts that it withheld:

  ... and 63 more clone groups
  ... and 52 more clone families withheld by the same display limit

With --summary, the counts block shows the corpus totals next to the shown counts. This makes it clear that the numbers cover two scopes:

       2  Clone families
       2  Clone groups
   2,605  Duplicated lines
    9.5%  Duplication rate
  ... and 63 more clone groups and 52 more clone families withheld by a display limit
  Corpus totals: 65 clone groups, 54 clone families; duplicated lines and rate cover all of it

The stats object can also include these fields. Fallow omits them when they are zero.

• clone_groups_ignored, when duplicates.ignoredClones hides groups
• near_candidates_skipped, when bounded-work limits skip near-miss comparisons

Symlinked files

A clone instance can come from a file whose path is a symlink, or lies under a symlinked directory, inside the project root. Such a clone can be one file under two paths, not copied code. Fallow marks these instances in each output format:

• JSON: is_symlink: true on the instance. Fallow omits the field when it is false.
• Compact: symlink=true on the record.
• Human: a (symlink) marker after the location.

By default, the report keeps symlinked instances. To report only duplication between real files, pass --ignore-symlinks or set duplicates.ignoreSymlinks: true in the config. Fallow then removes the symlinked files from the duplication corpus. A clone group with fewer than two remaining instances is not reported. The file and line totals in stats and the duplication percentage that --threshold checks count only real files. --no-ignore-symlinks overrides a config value of true.

Exit codes

Without a gate flag, fallow dupes exits 0, also when it finds clone groups. These flags turn on a gate:

The duplication-findings entry is present only when you pass --fail-on-issues or --ci. It is always enforced. observed is the number of clone groups and threshold is 0. The bare run with --only dupes --fail-on-issues uses the same gate. Invalid input exits 2. For the parse error gate, see global flags.

Examples

# Mild mode (default)
fallow dupes

Example output

● Duplicates (3 clone groups)

  5,433 lines  2 instances
    deno/lib/types.ts:45-5477
    src/types.ts:45-5477

    914 lines  2 instances
    deno/lib/__tests__/string.test.ts:8-921
    src/__tests__/string.test.ts:7-920

     42 lines  3 instances
    src/features/forecasting/server/procedures/analytics.ts:141-181
    src/features/forecasting/server/procedures/cashflow.ts:153-194
    src/features/forecasting/server/procedures/income.ts:590-631

  Identical code blocks detected via suffix-array analysis, https://docs.fallow.tools/explanations/duplication#clone-groups

✓ 27,255 lines (19.4%) duplicated across 398 files (0.23s)

● Duplicates (3 clone groups)
  ...

✗ 27,255 lines (19.4%) duplicated across 398 files (0.23s)
Duplication (19.4%) exceeds threshold (15%)

JSON output

mirrored_directories

When fallow detects two directories that are mostly duplicates (for example, a deno/ mirror of src/), the JSON output includes a mirrored_directories array:

{
  "mirrored_directories": [
    {
      "dir_a": "deno/lib",
      "dir_b": "src",
      "shared_files": ["client.ts", "handlers.ts", "types.ts", "utils.ts"],
      "total_lines": 6347
    }
  ]
}

Each entry names the two directories, the shared filenames, and the total duplicated lines in those files.

See also