Description

Implements a new rate limit configuration

Implements #325

This PR if applied adds a new rate limit configuration to the git-repo-scanner to obey the GitHub and GitLeaks server rate limits. By default this setting is true.

Example

Works for GitHub and Gitlab:

--obey-rate-limit: True to obey the rate limit of the GitHub server (default), otherwise False

• For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the user making requests.
  • With --obey-rate-limit=True the scanner will hit the GitHub API every 60 seconds at max (3600s / 60 requests per hour = 60s seconds delay between two request).
• OAuth applications authorized by a user share the same quota of 5,000 requests per hour when they authenticate with different tokens owned by the same user.
  • With --obey-rate-limit=True the scanner will hit the GitHub API every 0,7 seconds at max (3600s / 5000 requests per hour = 0,7s seconds delay between two request).

Implements a new git repository filter based on latest activity

Implements #320

Example

Works for GitHub and Gitlab:

--activity-since-duration: Return git repo findings with repo activity (e.g. commits) more recent than a specific date expresed by an duration (now + duration). A duration string is a possibly signed sequence of decimal numbers, each
                               with optional fraction and a unit suffix, such as '1h' or '2h45m'. Valid time units are 'm', 'h', 'd', 'w'.
--activity-until-duration: Return git repo findings with repo activity (e.g. commits) older than a specific date expresed by an duration (now + duration). A duration string is a possibly signed sequence of decimal numbers, each with
                               optional fraction and a unit suffix, such as '1h' or '2h45m'. Valid time units are 'm', 'h', 'd', 'w'.

Testing

If you want to test this PR please reference the following docker image: docker pull securecodebox/scanner-git-repo-scanner:git-repo-rate-limit

image:
  # image.repository -- Container Image to run the scan
  repository: docker.io/securecodebox/scanner-git-repo-scanner
  # image.tag -- defaults to the charts version
  tag: git-repo-rate-limit

Example HelmChart Install

You can use the following command to install and test this PR in your environment. For more general informations please check the scanner documentation : https://docs.securecodebox.io/docs/scanners/git-repo-scanner

# Optional: Add the secureCodeBox Helm Repo
helm repo add secureCodeBox https://charts.securecodebox.io
# Install HelmChart (use -n to configure another namespace)
helm upgrade --install git-repo-scanner secureCodeBox/git-repo-scanner \
  --set="image.repository=docker.io/securecodebox/scanner-git-repo-scanner" \
  --set="image.tag=git-repo-rate-limit"

Checklist

• Test your changes as thoroughly as possible before you commit them. Preferably, automate your test by unit/integration tests.
• Make sure npm test runs for the whole project.
• Cleanup the code to make codeclimate happy