# Indexing, caching & performance

How Projectile discovers, lists, orders and caches the files in a project, and how to tune all of that for speed (including over TRAMP).

## Project indexing method

Projectile has three modes of operation - one is portable and is implemented in Emacs Lisp (therefore it’s native to Emacs and is known as the `native indexing method`) and the other two (`hybrid` and `alien`) rely on external commands like `find`, `git`, etc to obtain the list of files in a project.

The `alien` indexing method trades features for raw speed: Projectile does no processing or sorting of the files returned by the external commands, so you get the maximum performance possible. This behaviour makes a lot of sense for most people, as they’d typically be putting ignores in their VCS config (e.g. `.gitignore`) and won’t care about any additional ignores/unignores/sorting that Projectile might also provide.

Note | By default the `alien` method is used on all operating systems except Windows.

To force the use of native indexing in all operating systems:

    (setq projectile-indexing-method 'native)

To force the use of hybrid indexing in all operating systems:

    (setq projectile-indexing-method 'hybrid)

To force the use of alien indexing in all operating systems:

    (setq projectile-indexing-method 'alien)

Alien indexing is significantly faster, especially on big projects, but it relies on external Unix utilities. That’s why it isn’t the default on Windows; if you run into trouble there, fall back to the `native` indexing method.

### Alien indexing

The alien indexing works in a pretty simple manner - it simply shells out to a command that returns the list of files within a project. For version-controlled projects by default Projectile will use the VCS itself to obtain the list of files. As an example, here is the command that Projectile uses for Git projects:

    git ls-files -zco --exclude-standard

For every supported VCS there’s a matching Projectile defcustom holding the command to invoke for it:

Variable | VCS
--- | ---
`projectile-git-command` | Git
`projectile-hg-command` | Mercurial
`projectile-svn-command` | Subversion
`projectile-bzr-command` | Bazaar
`projectile-darcs-command` | Darcs
`projectile-fossil-command` | Fossil
`projectile-pijul-command` | Pijul
`projectile-sapling-command` | Sapling
`projectile-jj-command` | Jujutsu

There are also two Git-specific commands for listing submodules and ignored files:

    ;; Command to list git submodules (set to nil to disable)
    (setq projectile-git-submodule-command
      "git submodule --quiet foreach 'echo $displaypath' | tr '\\n' '\\0'")

    ;; Command to get git-ignored files
    (setq projectile-git-ignored-command "git ls-files -zcoi --exclude-standard")

Warning | If you ever decide to tweak those keep in mind that the command should always be returning the list of files relative to the project root and the resulting file list should be 0-delimited (as opposed to newline delimited).

For non-VCS projects Projectile will invoke whatever is in `projectile-generic-command`. The default chooses `fd` when it’s installed and falls back to `find`:

    ;; Effective default value of projectile-generic-command, picked at load time:
    ;;   when fd is on PATH:
    "fd . -0 --type f --color=never --strip-cwd-prefix"
    ;;   otherwise:
    "find . -type f | cut -c3- | tr '\\n' '\\0'"

Tip | It’s a great idea to install fd which is much faster than `find`. If `fd` is found, projectile will use it as a replacement for `find` for non-VCS projects.

Warning | The `find` fallback does not exclude common build/cache directories (`.git`, `node_modules`, `target`, `build`, …) itself, so it walks the whole tree even though Projectile then drops the ignored entries from its output (see below). Install `fd` or override `projectile-generic-command` with a tighter recipe if that walk is slow.

By default, `fd` is also used inside Git repositories (instead of `git ls-files`), because `git ls-files` has the limitation that it lists deleted files until the deletions are staged. With `fd`, deleted files disappear from the listing immediately; with `git ls-files`, Projectile post-filters the listing against `git ls-files -zd` to hide deletions until they’re staged. You can control this with `projectile-git-use-fd`:

    ;; Disable fd in git repos (use git ls-files instead)
    (setq projectile-git-use-fd nil)

You can also customize the path to the `fd` executable and the arguments passed to it:

    ;; Use a custom fd path
    (setq projectile-fd-executable "/usr/local/bin/fd")

    ;; Customize the fd arguments used in git repos
    (setq projectile-git-fd-args "-H -0 -E .git -tf --strip-cwd-prefix -c never")

### Hybrid indexing

The `hybrid` method runs the same external command as `alien` and then post-processes the result with Projectile’s filtering rules. In other words, it’s `alien` plus a second pass that applies:

- `.projectile` (dirconfig) ignore/keep/ensure entries

- `projectile-globally-ignored-files`, `projectile-globally-ignored-directories` and `projectile-globally-ignored-file-suffixes`

- `projectile-globally-unignored-files` and `projectile-globally-unignored-directories`

- The configured sort order from `projectile-sort-order`

This is the right choice when your VCS lists more files than you want Projectile to surface (e.g. you want to drop `*.elc` even though they’re checked in), or when you rely on `.projectile` to keep/ignore subtrees inside an otherwise large repository. It’s slower than `alien` because of the extra pass, but it remains substantially faster than `native` for big projects since the file list itself is still produced by the external tool.

### Indexing method comparison

Not all Projectile features apply to every indexing method. The table below summarises which configuration knobs take effect under which mode:

Feature | `native` | `hybrid` | `alien`
--- | --- | --- | ---
Project file listing speed | Slow (Elisp) | Fast | Fastest
Honors `.projectile` (dirconfig) `-` ignore rules | Yes | Yes | Yes
Honors `.projectile` (dirconfig) `+` keep rules | Yes | Yes | Yes
Honors `.projectile` (dirconfig) `!` unignore rules | Yes | Yes | Yes
Honors `projectile-globally-ignored-files` / `-directories` / `-file-suffixes` | Yes | Yes | Yes
Honors `projectile-globally-ignored-file-regexps` (Emacs regexps) | Yes | No | No
Honors `projectile-globally-unignored-*` | Yes | Yes | Yes
Re-adds files the VCS itself ignores (`projectile-globally-unignored-*`, `!` entries) | n/a | Yes | No
Sorts via `projectile-sort-order` | Yes | Yes | No (returned in the external tool’s order)
File caching enabled by default | Yes | No | No
Works on Windows out of the box | Yes | Requires Unix utilities | Requires Unix utilities

### How `alien` applies the ignore rules

`alien` honors the ignore rules without giving up its speed: instead of filtering the tool’s output in Emacs, it hands the rules to the tool as exclusion arguments. Which tools can do that varies, so the ones that can’t get their output filtered in Emacs Lisp after all. Every path applies the same gitignore rules to the same pattern list, so the resulting file set is the same - the difference is only where the work happens:

Indexing command | How the rules are applied | Cost
--- | --- | ---
`git ls-files` | `:(exclude,glob)` pathspecs | In the tool
`fd` (including `projectile-git-use-fd`) | repeated `--exclude` globs | In the tool
`hg locate`, `sl locate`, `jj file list` | output filtered in Emacs Lisp | One pass over the file list
`svn`, `fossil`, `bzr`, `darcs`, `pijul` | output filtered in Emacs Lisp | One pass over the file list
the plain `find` fallback | output filtered in Emacs Lisp | One pass over the file list

Set `projectile-alien-honors-ignores` to nil to get the raw listing back and defer entirely to the tool’s own ignore rules (`.gitignore` and friends).

Note | An exclusion argument can’t be taken back, so a project whose dirconfig has `!` ensure entries is filtered in Emacs Lisp under `alien` too, rather than pushed down - the rules are the same, only the fast path is given up. Dirconfig `+` keep entries are passed to the tool as path arguments where it takes them, and applied to its output where it doesn’t.

Note | No indexing method can add back files the external tool never listed in the first place (because `.gitignore` excludes them); only `hybrid` does that, via its VCS-aware unignore pass.

Note | Ignore matching is case-sensitive under every indexing method. With `.elc` in `projectile-globally-ignored-file-suffixes`, a file named `BUILD.ELC` is not ignored - spell out both cases if you want both. Before Projectile 3.3 `native` and `hybrid` folded case here (a side effect of the Emacs functions they matched with), which `alien` had no way to reproduce in the external tools.

Note | Since Projectile 3.3 every ignore option speaks gitignore patterns and every indexing method matches them the same way, so a bare entry in `projectile-globally-ignored-directories` matches at any depth everywhere. It used to be top-level-only under `hybrid`, and a `*` prefix was a marker rather than a wildcard; both quirks are gone.

Tip | For the details of each ignore mechanism (`.projectile`, the indexing tools, and the global ignore/unignore variables), see Ignoring files.

## Sorting

You can choose how Projectile sorts files by customizing `projectile-sort-order`.

Note | If Alien indexing is set, files are not sorted by Projectile at all.

The default is to not sort files:

    (setq projectile-sort-order 'default)

To sort files by recently opened:

    (setq projectile-sort-order 'recentf)

To sort files by recently active buffers and then recently opened files:

    (setq projectile-sort-order 'recently-active)

To sort files by modification time (mtime):

    (setq projectile-sort-order 'modification-time)

To sort files by access time (atime):

    (setq projectile-sort-order 'access-time)

Finally, `projectile-sort-order` can be set to a function that receives the list of project files (as relative paths) and returns them in the desired order (added in Projectile 3.1). For example, to list files with shorter paths first:

    (setq projectile-sort-order
          (lambda (files) (seq-sort-by #'length #'< files)))

## File ranking (frecency)

Since Projectile 3.1, `projectile-find-file` (and its dwim variant) rank the files you actually work with first: Projectile records file visits per project and orders completion candidates by "frecency", a combination of visit frequency and recency (frequent visits decay with a half-life of two weeks). Files you’ve never visited keep their original order after the ranked ones.

Unlike `projectile-sort-order`, the ranking is applied through completion metadata (`display-sort-function`), so it works with any completion UI that honors it (the default completion UI, Vertico, Icomplete, …​) and under every indexing method, including `alien`.

The feature is enabled by default. To turn it off:

    (setq projectile-enable-frecency nil)

The visit history is persisted across sessions in `projectile-frecency-file` and capped per project by `projectile-frecency-max-files` (200 by default). Concurrent Emacs sessions merge their histories on save rather than overwriting each other. Remote (TRAMP) projects are not tracked.

Note | For tracked files the frecency order takes precedence over `projectile-sort-order`, since completion UIs apply the metadata sort to whatever candidate order they receive.

## Caching

### Project files

Since indexing a big project is not exactly quick (especially in Emacs Lisp), Projectile supports caching of the project’s files. The caching is enabled by default whenever native indexing is enabled.

To enable caching unconditionally use this snippet of code:

    (setq projectile-enable-caching t)

At this point you can try out a Projectile command such as kbd:[s-p f] (kbd:[M-x] `projectile-find-file` kbd:[RET]).

Running kbd:[C-u s-p f] will invalidate the cache prior to prompting you for a file to jump to.

Pressing kbd:[s-p z] will add the currently visited file to the cache for current project. Generally files created outside Emacs will be added to the cache automatically the first time you open them.

A file the project ignores isn’t added, whether it’s your `.projectile` and the global ignore settings that exclude it or - under `alien` and `hybrid` indexing, where the file list comes from the VCS - your `.gitignore`. Otherwise opening, say, a compiled artifact would put it in a cache that indexing would never have produced.

Normally the cache lasts for the duration of your Emacs session. If you want to cache to persist between Emacs sessions you should set this option to `'persistent`.

    (setq projectile-enable-caching 'persistent)

Now the project cache is persistent and will be preserved during Emacs restarts. Each project gets its own cache file, that will be placed in the root folder of the project. The name of the cache file is `.projectile-cache.eld` by default, but you can tweak it if you want to:

    (setq projectile-cache-file "foo.eld")

The cache file will be loaded automatically in memory the first time you trigger a "find file" operation for the project it belongs to.

You can purge an individual file from the cache with `M-x projectile-purge-file-from-cache` or an entire directory with `M-x projectile-purge-dir-from-cache`.

If the caches of many projects have gone stale at once (something you’d typically notice with commands operating on several projects, like `projectile-find-file-in-known-projects`), you can invalidate them all in one go with `M-x projectile-invalidate-cache-all`. Remote (TRAMP) projects are skipped to avoid a potentially slow connection round-trip per project; invalidate those individually with `projectile-invalidate-cache`.

Note | Prior to Projectile 2.9 the cache for all projects was serialized to the same file. In Projectile 2.9 this was changed and now each project has its own cache file relative to the project’s root directory.

When `projectile-mode` is enabled Projectile will auto-update the project cache when files within are added or deleted from within Emacs (via file hooks). This behavior can be disabled like this:

    (setq projectile-auto-update-cache nil)

You can also set the project files cache to expire after a given number of seconds:

    ;; Expire the project files cache after 5 minutes
    (setq projectile-files-cache-expire (* 5 60))

By default `projectile-files-cache-expire` is `nil`, meaning the cache never expires automatically.

One last thing - the project cache will be auto-invalidated if you’re using `.projectile` and its last modification time is more recent than the time at which the cache file was last updated.

### Automatic cache updates

Note | This feature is experimental and was added in Projectile 3.2.

The file hooks described above only see what happens inside Emacs. Files created, deleted or renamed by external tools (a `git pull`, a code generator, another editor) still leave the cache stale until you run `projectile-invalidate-cache` by hand. If you’d rather have Projectile notice such changes itself, it can watch the directories of your cached projects using filesystem notifications:

    (setq projectile-auto-update-cache-with-watches t)

Whenever a project’s file list is cached, Projectile then registers a `file-notify` watch for each of the project’s directories and applies the resulting events to the cached list: new files are run through the usual ignore filtering before being added, deleted files (and directories) are removed, renames are handled as both. Events are debounced, so bursts (say, a branch switch touching hundreds of files) are folded into a single update. Anything that can’t be applied incrementally with confidence falls back to invalidating the project’s cache, which simply gets rebuilt on the next file listing.

Be aware of the trade-offs before enabling it:

- Emacs file notifications are not recursive, so a watched project costs one watch, and typically one file descriptor, per directory. Projects spanning more directories than `projectile-watch-directory-limit` (512 by default) are not watched at all; set `projectile-verbose` to be told when that happens.

- Remote (TRAMP) projects are never watched.

- Watched directories are derived from the cached file list, so a directory that contains no cached files (e.g. an empty directory) isn’t watched until the next full re-index.

- Under `alien` and `hybrid` indexing the VCS is consulted too, so a watched project doesn’t gain files your `.gitignore` excludes. That’s one `git check-ignore` per batch of events rather than one per file, since a batch can cover a whole directory that was moved in. Other version control systems aren’t asked, and `native` indexing lists such files anyway.

Watches are dropped automatically when a project’s cache is invalidated (and re-armed when it’s next filled), when the option is disabled via Customize or `setopt`, when `projectile-mode` is turned off (turning the mode back on re-arms them for all cached projects), when a project is removed from the known projects and on Emacs shutdown.

### Background indexing

The first "find file" in a large or remote project has to index it. With the `alien` and `hybrid` indexing methods Projectile does this without freezing Emacs: the indexing command runs asynchronously and is awaited in a way that keeps redisplay going and leaves kbd:[C-g] live, so you can abort a slow index (for instance one stuck on an unresponsive remote host) instead of staring at a frozen editor. The resulting file list is exactly what the synchronous indexer would produce.

This is on by default and controlled by `projectile-async-indexing`:

    ;; index synchronously, as in older Projectile versions
    (setq projectile-async-indexing nil)

Note | It has no effect under `native` indexing (the Emacs Lisp directory walk can’t run off the main thread), in batch mode, or while a keyboard macro is executing - those index synchronously.

The result of the indexing command reaches Projectile from the process’s sentinel, and Emacs doesn’t guarantee it runs that promptly while a command is waiting on the process. If it hasn’t run `projectile-async-index-sentinel-timeout` seconds after the command finished, Projectile parses the output itself instead of waiting any longer. You shouldn’t ever need to touch that option; it exists because waiting forever was once possible.

You can also warm the cache ahead of time, without blocking, using kbd:[M-x] `projectile-index-project-async`:

    ;; index the current project in the background
    (projectile-index-project-async)

It runs the project’s indexing command via an asynchronous process and populates the same cache that a regular `projectile-find-file` reads, so a later "find file" finds the cache already warm instead of indexing on the spot. You could, for instance, warm a project right after switching to it:

    (add-hook 'projectile-after-switch-project-hook
              #'projectile-index-project-async)

Note | Background indexing only works with the `alien` and `hybrid` indexing methods - the `native` (pure Emacs Lisp) walk can’t run off the main thread. It also requires caching to be enabled, since the warmed result is stored in the project files cache.

The asynchronous building blocks it’s made of - `projectile-files-via-ext-command-async` and `projectile-dir-files-alien-async` - are public, so a streaming file finder (for example one built on top of `consult`) can drive Projectile’s indexing command asynchronously and feed results into its own UI. See the next section.

### Integrating a streaming file finder

If you’d rather have your own asynchronous/streaming UI (à la `fzf`, `consult` or `affe`) list a project’s files instead of `projectile-find-file`, `projectile-project-files-producer` hands you everything you need to run Projectile’s own indexing command yourself:

    (projectile-project-files-producer)
    ;; => (:directory "/path/to/project/"
    ;;     :vcs git
    ;;     :command "git ls-files -zco --exclude-standard"
    ;;     :separator "\0")

Run `:command` in `:directory` and split its output on `:separator`. A minimal (non-streaming) example built on the public asynchronous runner:

    (defun my/project-find-file ()
      "Pick a file from the current project, indexing without blocking."
      (interactive)
      (let* ((producer (projectile-project-files-producer))
             (default-directory (plist-get producer :directory)))
        (projectile-files-via-ext-command-async
         default-directory (plist-get producer :command)
         (lambda (files _err)
           (find-file (expand-file-name (completing-read "File: " files)
                                        default-directory))))))

A real streaming finder would instead consume the process output incrementally (filtering as you type) rather than collecting it. For git projects, note that `:command` lists the main worktree only; if you want byte-for-byte the same set as `projectile-find-file` (submodule files folded in, deleted-but-unstaged files removed) drive `projectile-dir-files-alien-async` instead.

#### Bundled Consult integration

Projectile ships such a finder for Consult in `projectile-consult.el`. It’s an optional module with a soft dependency on `consult` (Projectile core never loads it), so you opt in by requiring it:

    (require 'projectile-consult)
    (define-key projectile-command-map (kbd "f") #'projectile-consult-find-file)

`projectile-consult-find-file` lists the project with Projectile’s own indexing command and streams the candidates into Consult as they arrive, so there’s no wait for the whole project to be indexed before you can start typing. Because it goes through `projectile-project-files-producer`, it stays VCS-aware and honours the project’s indexing configuration. The module needs Consult 2.0+, which in turn requires Emacs 29.1+.

### File exists cache

Projectile does many file existence checks since that is how it identifies a project root. Normally this is fine, however in some situations the file system speed is much slower than usual and can make Emacs "freeze" for extended periods of time when opening files and browsing directories.

The most common example would be interfacing with remote systems using TRAMP/ssh. By default all remote file existence checks are cached.

To disable the remote file-exists cache, use this snippet of code:

    (setq projectile-file-exists-remote-cache-expire nil)

To change the remote file exists cache expire to 10 minutes use this snippet of code:

    (setq projectile-file-exists-remote-cache-expire (* 10 60))

You can also enable the cache for local file systems, that is normally not needed but possible:

    (setq projectile-file-exists-local-cache-expire (* 5 60))

## TRAMP performance

Projectile is TRAMP-aware and the hot paths cache aggressively, so once the caches are warm, navigating a remote project should feel similar to a local one. The first command on a fresh remote project pays a one-time setup cost (one VCS-detection listing, one project-root walk, one indexing shell-out); after that everything is served from memory until you invalidate it.

The following knobs are most relevant for remote use:

- `projectile-indexing-method` - leave at the default (`alien`) or set to `hybrid`. Don’t use `native` over TRAMP: native indexing walks the tree with one `directory-files` call per subdirectory, which is one remote round-trip per subdirectory. `hybrid` and `alien` use a single `git ls-files` / `fd` shell-out.

- `projectile-enable-caching` - enables the per-project file list cache. Set to `'persistent` if you want the cache to survive Emacs restarts, so reopening a remote project doesn’t re-shell-out.

- `projectile-file-exists-remote-cache-expire` (default 5 minutes) - TTL for cached remote file-existence results. Lower it if you frequently create new project markers (`.projectile`, `.git`, …​) on the remote and want them detected sooner; raise it if your remotes are stable and you’d rather avoid the round-trip.

After creating, removing, or moving a project marker on a remote, run `projectile-discard-root-cache` (or `projectile-invalidate-cache`) - both also clear the file-existence cache, so the new state takes effect on the next call instead of waiting for the TTL.

A few things that don’t require configuration but are worth knowing:

- `fd` is auto-detected per remote host (cached for the session). The `projectile-fd-executable` defcustom is the local detection result and isn’t used remotely - whatever Projectile finds at `executable-find` time on the remote (`fd` or `fdfind`) is what runs there. No `fd` on the remote means the indexing path falls back to `git ls-files`.

- If indexing fails (e.g. `git` or `fd` isn’t on the remote `PATH`), Projectile signals a `user-error` pointing at `*projectile-files-errors*` instead of returning an empty file list. Older versions silently returned `nil`, which manifested as unexplained empty completion lists or `stringp, nil` crashes downstream.