Tips & Tricks of the trade: sanitize secrets from dirty commits

Tips & Tricks of the Trade: sanitize secrets from dirty commits

Scan a repo for senstive keys or secretes. Did you accidently commit a api key.

Tools like gitleaks or trufflehog can be very effective aid in cleaning up dirty commits.

GitLeaks

Lets cover gitleaks to remove an api key from history. Go install

1
brew install gitleaks

Setup a config file .gitleaks.toml in the root dir of the repo. This example excludes some rules from a git directory that is being used as a static asset store.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
title = "Custom Gitleaks Configuration"

[extend]

useDefault = true

tags = ["data_dir"]

[[rules]]

id = "generic-api-key"

[[rules.allowlists]]

paths = [

'''^data/raw-events-.*\.(json|parquet)$''',
]

[[rules]]

id = "github-app-token"

[[rules.allowlists]]

paths = [

'''^data/raw-events-.*\.(json|parquet)$'''
]

Run the CLI command or better yet setup a pre-commit to always check.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
❯ gitleaks detect --source . --report-format json --report-path gitleaks-report.json

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks

2:24PM INF 373 commits scanned.
2:24PM INF scanned ~4469777469 bytes (4.47 GB) in 1m7.7s
2:24PM WRN leaks found: 6

Resolve the leaks

Github has an excellent guide using git filter-repo . The tldr is you need to clone a fresh repo and use a file outside of that repo to replace text in side commits

1
❯ git filter-repo --sensitive-data-removal --replace-text ../replacements.txt

You can check the changes with grep

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
❯ grep -c '^refs/pull/.*/head$' .git/filter-repo/changed-refs
11

❯ grep '^refs/pull/.*/head$' .git/filter-repo/changed-refs
refs/pull/123/head
refs/pull/37/head
refs/pull/372/head
refs/pull/379/head
refs/pull/42/head
refs/pull/433/head
refs/pull/48/head
refs/pull/57/head
refs/pull/72/head
refs/pull/73/head
refs/pull/76/head
# {{title}}
{{date}}
Status: #Blog 
Category: 
Volume: 
Difficulty: 
KW: 
## Should I Write This?
1. How does {{title}} fit into my deal? What offer or funnel is it promoting?

2. What's the key takeaway, action or CTA?

## Content Plan

### Intro - What Makes It Interesting?
- Data Point or Fact
- Hook
- Anecdote
- Personal Story
- Examples
- Screenshots
- Links
- Quotes/Interview
- Imagery
- Embedded demo
- Real Life application
- Connection to something timely or well known

### Main Content

#### What's my point of view on {{title}}? Do I have a background in {{title}}? Do I want to exapnd on an old idea?

#### What Questions do I get asked about this or would I ask about {{title}}

#### Problems

#### Solutions

#### What do I want them to know, feel or do?
- Core Message
- CTA


## Blog
# {{title}}
❯ git show refs/pull/37/head

Next setup pre-commit to prevent this in the future.