Git LFS (Large File Storage)
Why a 2GB model checkpoint doesn't belong directly in git — and what to do instead.
Intermediate
TopNotchNote
Notes that matter
the problem: git never forgets
Every version of every file you've ever committed stays in
.git/objects
forever (see Git Internals). That's fine for text — diffs of
source code are small. It's a disaster for a binary file that changes often: a 500MB
dataset re-saved 20 times isn't 500MB of history, it's up to 10GB, and none of it
compresses or diffs well since it's binary.
what lfs actually does
Git LFS replaces a large file's content, inside your git history, with a small text
pointer file (a hash + size, a few hundred bytes). The real file content is stored
separately, on an LFS server, and only downloaded on demand when you check out a commit
that needs it.
git lfs install | https://git-lfs.com/ | one-time setup per machine — registers LFS with your local git |'lfs_install1'
tracking files
git lfs track "*.pt" | | tells LFS to intercept any file matching this pattern from now on |'lfs_track1'
git lfs track "*.ckpt" "*.safetensors" "datasets/**" | | track can take several patterns at once |'lfs_track2'
This writes the patterns to a
.gitattributes file at the repo root —
that file is what actually needs to be committed and shared; git lfs track
is just the command that edits it for you correctly.
git add .gitattributes | | this file must be committed, or nobody else's clone will know to use LFS |'lfs_track3'
day to day, nothing changes
Once a pattern is tracked,
git add, git commit, and
git push work exactly as before — LFS intercepts matching files
transparently. The only visible difference is what a clone downloads.
git lfs pull | https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-pull.adoc | downloads the actual LFS file content for the currently checked-out commit |'lfs_pull1'
GIT_LFS_SKIP_SMUDGE=1 git clone <url> | | clones without downloading any LFS content up
front (only the small LFS pointer files), useful for huge repos. Note: git's own
--filter=blob:none (partial clone) is a different, unrelated mechanism for deferring ordinary
git blobs — it does NOT by itself skip LFS payloads, since those are fetched by git-lfs's
smudge filter independently of that flag |'lfs_pull2'
git lfs ls-files | | lists which files in the current checkout are tracked by LFS |'lfs_pull3'
migrating files already in history
Adding a track pattern only affects new commits. If a large file is already
sitting in your history from before LFS was set up, it's still there, uncompressed,
bloating every future clone —
track alone won't fix that.
git lfs migrate import --include="*.pt" --everything | https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-migrate.adoc | rewrites history, moving matching files into LFS retroactively |'lfs_mig1'
This rewrites every commit that touched a matching file — same caveat as any history
rewrite (see Undoing Things in Git): coordinate with anyone else
who has the repo cloned, since their history will no longer match after a force-push.
why this matters for ML work
Model checkpoints, tokenizer files, and datasets are exactly the files LFS exists
for: large, binary, and re-saved often during training. Tracking them with LFS from day
one keeps a research repo's actual git history — the part that's meant to be small and
diffable — usable, instead of turning every clone into a multi-gigabyte download.
related topics
Git Internals — why large binary blobs are expensive: nothing in git history is ever removed automatically.
Undoing Things in Git — the same history-rewrite caveats apply to
Machine Learning — the checkpoints and datasets this page is usually reached for.
Undoing Things in Git — the same history-rewrite caveats apply to
git lfs migrate.Machine Learning — the checkpoints and datasets this page is usually reached for.