Skip to content

Ensure Findings are validated in Integration Tests (closes #546) #603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Weltraumschaf merged 12 commits into from
Aug 31, 2021
Merged

Ensure Findings are validated in Integration Tests (closes #546) #603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
c946711
add env variable to crash on failed validation
JohannesZahn Aug 20, 2021
d703be1
log crash on failed validation envb variable
JohannesZahn Aug 20, 2021
d0b54ab
Revert "add env variable to crash on failed validation"
JohannesZahn Aug 20, 2021
cf2d130
add crash on failed validation to parse definition in CI
JohannesZahn Aug 20, 2021
6145e05
use test scan to make sure invalid findings make tests fail
JohannesZahn Aug 23, 2021
0d67530
fix code climate issues
JohannesZahn Aug 23, 2021
ea448d2
copy findings schema to docker container
JohannesZahn Aug 23, 2021
858d89c
validate the findings with ids and dates
JohannesZahn Aug 24, 2021
0722646
validate findings in all tests where scans are involved
JohannesZahn Aug 24, 2021
33e0047
fix string formatting issues
JohannesZahn Aug 24, 2021
56ab4c8
fix bug with invalid findings in test-scann
JohannesZahn Aug 24, 2021
cb1d9ce
fix bug due to naming error
JohannesZahn Aug 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 70 additions & 17 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -825,6 +825,23 @@ jobs:

# This steps should include Integration tests which are not related to a Specific Scanner

# ---- Gerneral Testing | Findings Validation ----

- name: "Parser should fail on invalid Findings"
run: |
helm -n integration-tests install test-scan ./scanners/test-scan/ \
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-test-scan" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-test-scan" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true" \
--set="parser.env[1].name=PRODUCE_INVALID_FINDINGS" \
--set-string="parser.env[1].value=true"
cd tests/integration/
npx jest --ci --color generic/findings-validation.test.js
helm -n integration-tests uninstall test-scan

# ---- General Testing | ReadAndWrite Hook ----

- name: "Throws NoScanDefinition Error Integration Tests"
Expand All @@ -847,7 +864,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-test-scan" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-test-scan" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)"
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color generic/read-write-hook.test.js
helm -n integration-tests uninstall test-scan update-category update-severity
Expand All @@ -860,7 +879,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-test-scan" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-test-scan" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)"
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
helm -n integration-tests install http-webhook ./demo-targets/http-webhook
helm -n integration-tests install ro-hook ./hooks/generic-webhook/ \
--set="hook.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/hook-generic-webhook" \
Expand Down Expand Up @@ -899,7 +920,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-gitleaks" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-gitleaks" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/gitleaks.test.js

Expand All @@ -911,7 +934,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-kube-hunter" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-kube-hunter" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/kube-hunter.test.js

Expand All @@ -926,7 +951,9 @@ jobs:
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-kubeaudit" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="kubeauditScope=cluster"
--set="kubeauditScope=cluster" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/kubeaudit.test.js
kubectl delete namespace kubeaudit-tests
Expand All @@ -940,7 +967,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-ncrack" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-ncrack" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/ncrack.test.js

Expand All @@ -953,7 +982,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-nikto" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-nikto" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/nikto.test.js

Expand All @@ -966,7 +997,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-nmap" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-nmap" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/nmap.test.js

Expand All @@ -978,7 +1011,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-test-scan" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-test-scan" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)"
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
helm -n integration-tests install http-webhook ./demo-targets/http-webhook --wait

helm -n integration-tests install notification-hook ./hooks/notification --values tests/integration/hooks/__testFiles__/notification-values.yaml \
Expand All @@ -996,7 +1031,9 @@ jobs:
kubectl -n integration-tests delete scans --all
helm -n integration-tests install ssh-scan ./scanners/ssh-scan/ \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-ssh-scan"
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-ssh-scan" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/ssh-scan.test.js

Expand All @@ -1007,7 +1044,9 @@ jobs:
kubectl -n integration-tests delete scans --all
helm -n integration-tests install sslyze ./scanners/sslyze/ \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-sslyze"
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-sslyze" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/sslyze.test.js

Expand All @@ -1018,7 +1057,9 @@ jobs:
kubectl -n integration-tests delete scans --all
helm -n integration-tests install wpscan ./scanners/wpscan/ \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-wpscan" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/wpscan.test.js

Expand All @@ -1029,7 +1070,9 @@ jobs:
kubectl -n integration-tests delete scans --all
helm -n integration-tests install zap ./scanners/zap/ \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-zap"
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-zap" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
cd tests/integration/
npx jest --ci --color scanner/zap.test.js

Expand All @@ -1044,7 +1087,9 @@ jobs:
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-zap" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-zap-advanced" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)"
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
kubectl apply -f ./scanners/zap-advanced/examples/integration-tests/scantype-configMap.yaml -n integration-tests
cd tests/integration/
npx jest --ci --color scanner/zap-advanced.test.js
Expand All @@ -1065,7 +1110,9 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-nmap" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-nmap" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
# Install ncrack
printf "root\nadmin\n" > users.txt
printf "THEPASSWORDYOUCREATED\n123456\npassword\n" > passwords.txt
Expand All @@ -1075,6 +1122,8 @@ jobs:
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-ncrack" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true" \
--values -
scanner:
extraVolumes:
Expand Down Expand Up @@ -1110,11 +1159,15 @@ jobs:
--set="scanner.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/scanner-nmap" \
--set="scanner.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-nmap" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
# Install sslyze
helm -n cascading-tests install sslyze ./scanners/sslyze/ --wait \
--set="parser.image.repository=docker.io/${{ env.DOCKER_NAMESPACE }}/parser-sslyze" \
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)"
--set="parser.image.tag=sha-$(git rev-parse --short HEAD)" \
--set="parser.env[0].name=CRASH_ON_FAILED_VALIDATION" \
--set-string="parser.env[0].value=true"
# Actually run the tests
cd tests/integration/
npx jest --ci --color scanner/cascade-nmap-sslyze.test.js
Expand Down
1 change: 1 addition & 0 deletions parser-sdk/nodejs/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ WORKDIR /home/app/parser-wrapper/
COPY --from=build --chown=app:app /home/app/node_modules/ ./node_modules/
COPY --chown=app:app ./parser-wrapper.js ./parser-wrapper.js
COPY --chown=app:app ./parser-utils.js ./parser-utils.js
COPY --chown=app:app ./findings-schema.json ./findings-schema.json
USER 1001
ENV NODE_ENV ${NODE_ENV:-production}
ENTRYPOINT ["node", "/home/app/parser-wrapper/parser-wrapper.js"]
10 changes: 6 additions & 4 deletions parser-sdk/nodejs/parser-wrapper.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,13 +135,15 @@ async function main() {
console.log("Adding UUIDs and Dates to the findings");
const findingsWithIdsAndDates = addIdsAndDates(findings);

console.log("Validating Findings");
const crash_on_failed_validation = process.env["CRASH_ON_FAILED_VALIDATION"] === "true"
console.log("Validating Findings. Environment variable CRASH_ON_FAILED_VALIDATION is set to %s", crash_on_failed_validation);
try {
await validate(findings);
await validate(findingsWithIdsAndDates);
console.log("The Findings were successfully validated")
} catch (error) {
console.error("Findings Validation failed with error:");
console.error("The Findings Validation failed with error(s):");
console.error(error);
if (process.env["CRASH_ON_FAILED_VALIDATION"] === "true") {
if (crash_on_failed_validation) {
process.exit(1);
}
}
Expand Down
71 changes: 47 additions & 24 deletions scanners/test-scan/parser/parser.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,16 @@
// SPDX-License-Identifier: Apache-2.0

async function parse() {
if (process.env["PRODUCE_INVALID_FINDINGS"] === "true")
return getInvalidFindings()
else
return getValidFindings()
}

function getInvalidFindings(){
return [
{
name: "Test read-write-hook service",
description: `Port is using protocol.`,
category: "Open Port",
location: `tcp://rw-hook-test:80`,
osi_layer: "NETWORK",
severity: "INFORMATIONAL",
attributes: {
port: 80,
state: "Open",
ip_address: "host ip address",
mac_address: "hostInfo.mac",
protocol: "openPort.protocol",
hostname: "hostInfo.hostname",
method: "openPort.method",
operating_system: "hostInfo.osNmap",
service: "openPort.service",
serviceProduct: "openPort.serviceProduct",
serviceVersion: "openPort.serviceVersion",
scripts: "openPort.scriptOutputs",
},
},
{
name: `Host: hostname`,
category: "Host",
//missing name and category to be a valid finding
description: "Found a host",
location: "hostname",
severity: "INFORMATIONAL",
Expand All @@ -38,7 +22,46 @@ async function parse() {
hostname: "hostname",
operating_system: "osNmap",
},
}
]
}

function getValidFindings(){
return [{
name: "Test read-write-hook service",
description: `Port is using protocol.`,
category: "Open Port",
location: `tcp://rw-hook-test:80`,
osi_layer: "NETWORK",
severity: "INFORMATIONAL",
attributes: {
port: 80,
state: "Open",
ip_address: "host ip address",
mac_address: "hostInfo.mac",
protocol: "openPort.protocol",
hostname: "hostInfo.hostname",
method: "openPort.method",
operating_system: "hostInfo.osNmap",
service: "openPort.service",
serviceProduct: "openPort.serviceProduct",
serviceVersion: "openPort.serviceVersion",
scripts: "openPort.scriptOutputs",
},
},
{
name: `Host: hostname`,
category: "Host",
description: "Found a host",
location: "hostname",
severity: "INFORMATIONAL",
osi_layer: "NETWORK",
attributes: {
ip_address: "ip address",
hostname: "hostname",
operating_system: "osNmap",
},
},
];
}

Expand Down
19 changes: 19 additions & 0 deletions tests/integration/generic/findings-validation.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
// SPDX-FileCopyrightText: 2021 iteratec GmbH
//
// SPDX-License-Identifier: Apache-2.0

const { scan } = require("../helpers");

jest.retryTimes(3);

test(
"Parser must fail on invalid findings",
async () => {
await expect(
scan("invalid-findings-test-scan", "test-scan", [], 90)
).rejects.toThrow(
`Scan failed with description "Failed to run the Parser. This is likely a Bug, we would like to know about. Please open up a Issue on GitHub."`
);
},
3 * 60 * 1000
);