-
Notifications
You must be signed in to change notification settings - Fork 179
Description
Describe the bug
I am having an issue with a git-repo-scanner + gitleaks working in concert to scan my company's org, which has over 2k repos. The scan is ending around 900 repos with the following error:
INFO:git_repo_scanner: 901 - crp-legacy-ui-dhtmlWins
INFO:git_repo_scanner: Github API Exception: 403 -> You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.
This is causing CrashLoopBackoff events with the pod. Ultimately the job fails with error.
I understand what the error is telling me. My main question is: What is the largest number of repos does this scanner support? I would like to scan my entire org in one run, but I have not been successful, with the number of repos and the GH API throttle limits. The --ignore-repo is a start, but seems not too effective given my large list of repos.
Other questions:
Is there a way to throttle api calls?
Is there a recommended number of batches I should run with?
Would there be a way to pass an argument to break up the scans?
To Reproduce
Steps to reproduce the behavior:
- Install via helm: see second screenshot below.
- Create a git-repo-scanner yaml file.
- Run the git-repo-scanner job on a github org with over 2000 repos and with cascadingrules triggering gitleaks.
- Tail the created job's log, for example:
k logs job/scan-gh-repo-scan-org-coretech-wdbns -c git-repo-scanner -f - See 403 error in the first screenshot below.
Expected behavior
System (please complete the following information):
- secureCodeBox 2.4.0
- OS: Linux
- Kubernetes Version [command:
kubectl version]
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-21T20:23:45Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
- Docker Version [command:
docker -v]
Docker version 20.10.2, build 2291f61
- Browser [e.g. chrome, safari, firefox,...]
N/A
Screenshots / Logs
Additional context

