Skip to content

🐞 The git_repo_scanner triggers an abuse detection mechanism at GitHub for large organisations  #325

@glermaidt

Description

@glermaidt

Describe the bug

I am having an issue with a git-repo-scanner + gitleaks working in concert to scan my company's org, which has over 2k repos. The scan is ending around 900 repos with the following error:

INFO:git_repo_scanner: 901 - crp-legacy-ui-dhtmlWins
INFO:git_repo_scanner: Github API Exception: 403 -> You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.

This is causing CrashLoopBackoff events with the pod. Ultimately the job fails with error.

I understand what the error is telling me. My main question is: What is the largest number of repos does this scanner support? I would like to scan my entire org in one run, but I have not been successful, with the number of repos and the GH API throttle limits. The --ignore-repo is a start, but seems not too effective given my large list of repos.

Other questions:
Is there a way to throttle api calls?
Is there a recommended number of batches I should run with?
Would there be a way to pass an argument to break up the scans?

To Reproduce

Steps to reproduce the behavior:

  1. Install via helm: see second screenshot below.
  2. Create a git-repo-scanner yaml file.
  3. Run the git-repo-scanner job on a github org with over 2000 repos and with cascadingrules triggering gitleaks.
  4. Tail the created job's log, for example: k logs job/scan-gh-repo-scan-org-coretech-wdbns -c git-repo-scanner -f
  5. See 403 error in the first screenshot below.

Expected behavior

System (please complete the following information):

  • secureCodeBox 2.4.0
  • OS: Linux
  • Kubernetes Version [command: kubectl version]

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-21T20:23:45Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

  • Docker Version [command: docker -v]

Docker version 20.10.2, build 2291f61

  • Browser [e.g. chrome, safari, firefox,...]
    N/A

Screenshots / Logs

image

image

Additional context

Metadata

Metadata

Assignees

Labels

bugBugspythonIssues based on python implementations

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions