Getting Git repositories corrupted in QubesOS (and fixing it)
Table of Contents
I use Git for everything, and because I run things virtualized in Qubes, sometimes I get my repos corrupted from a unexpected shutdown.
Just now I had this issue. The battery was draining fast (as regular with Qubes) and when it was 6% I put it to sleep. When waking again with the charger, it froze and didn't shown the login screen. (This happens sometimes, I didn't investigated further.)
With this soft-lock, my options were force a shutdown.
The problem: it slept it immediately after had a lot of new commits:
- Asciicasts sessions recording
- Bash history (yes, also a Git repo)
- A feature in a personal project
Waking it again I saw the blast: 2 qubes had their asciicasts sessions and bash history repos corrupted, plus this side project.
Troubleshooting
A couple of empty objects, and the last commit wasn't synced in the disk yet.
$ pwd /home/user/Documents/repos/bash_history $ git s error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty fatal: bad object HEAD # [ble: exit 128] $ git log error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty fatal: bad object HEAD [ble: exit 128] $ git reset error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty error: object file .git/objects/07/cffcc5423aef821b968bebf5d49288a2ba2a9a is empty fatal: Could not parse object 'HEAD'. # [ble: exit 128]
git fsck --full also reported a couple of nonexistent objects. So the first step
is to backup the repo.
Empty objects
Then I deleted all empty objects:
find .git -size 0 -print -delete
The fsck now only reports the invalid commit:
$ git fsck --full Checking ref database: 100% (1/1), done. Checking object directories: 100% (256/256), done. Checking objects: 100% (2879/2879), done. error: refs/heads/qube-ix5: invalid sha1 pointer ec68ff937df5fe1d79c2ba969768a6343e17e841 error: HEAD: invalid sha1 pointer ec68ff937df5fe1d79c2ba969768a6343e17e841 # [...]
Solution? Need to drop this corrupted commit…
Dropping corrupted commit
The task is simple, set branch head to the previous commit, not corrupted.
We can check the log of commits in .git/logs/HEAD:
$ tail -n2 .git/logs/HEAD c0a44d9d72588652d988ace4e69d76356e4e69db 958f4c7e642e55130a25f56d5bf9bb6b810711f9 thisago <thisago@disroot.org> 1777933122 +0000 commit: chore(history): session start
Corrupted as well… Last line was full of ^@ chars (fn:what-cursor-position tells: Char: C-@ (0, #o0, #x0) …):
$ tail -n2 .git/logs/HEAD | bat --style=plain # escaping the control chars c0a44d9d72588652d988ace4e69d76356e4e69db 958f4c7e642e55130a25f56d5bf9bb6b810711f9 thisago <thisago@disroot.org> 1777933122 +0000 commit: chore(history): session start ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Deleted last lines with nano, cleaned.
$ tail -n2 .git/logs/HEAD 438ead2c9a35789ebbb48dd36bebcccd2ca5c507 c0a44d9d72588652d988ace4e69d76356e4e69db thisago <thisago@disroot.org> 1777933052 +0000 commit: chore(history): session end c0a44d9d72588652d988ace4e69d76356e4e69db 958f4c7e642e55130a25f56d5bf9bb6b810711f9 thisago <thisago@disroot.org> 1777933122 +0000 commit: chore(history): session start
Now, grabbing the last commit I can replace the current branch HEAD manually:
$ cat .git/HEAD ref: refs/heads/qube-ix5 $ cat .git/refs/heads/qube-ix5 ec68ff937df5fe1d79c2ba969768a6343e17e841 $ echo 958f4c7e642e55130a25f56d5bf9bb6b810711f9>.git/refs/heads/qube-ix5
git status now works:
$ git s MM .bash_history
Unstaging
The repos that had staged changes still had corrupted objects, in the case of
asciicasts repos and my personal project. It shown as ? in
git status --short:
$ git s # [...] MM workflow/timestamps.jsonl ?? workflow/20260504/1777937875.cast $ git diff fatal: unable to read a39a0900d65e7e8221ad81695e899eff90e1c724
The solution is simple, just unstage.
$ git reset HEAD -- workflow/20260504/1777937875.cast
$ git diff >/dev/null && echo not corrupted
not corrupted
Sanity check
$ git fsck --full Checking ref database: 100% (1/1), done. Checking object directories: 100% (256/256), done. Checking objects: 100% (2882/2882), done. Verifying commits in commit graph: 100% (481/481), done. # [ble: elapsed 52.239s (CPU 98.0%)] git fsck --full
All done!
Outro
Previous attempts drove me to search internet across solutions, it shown me clearer how Git internals works. It's good to solve problems by your own.
This post was really a dump of commands and thoughts so I can:
- Post something in this blog
- Keep notes of how I solved
Nevertheless, better content will soon come.
Fun fact: add a C-@ control char in a file and Git will consider it as
binary. Try it yourself, press C-q C-@ in your Emacs!