Skip to content

deepthought42/CrawlerAPI

Repository files navigation

Codacy Badge

Getting Started

Prerequisites

  • Java 21
  • Maven 3.x
  • Neo4j database instance
  • Auth0 account (for authentication)

Build

mvn clean install

Launch Jar locally

Command Line Interface(CLI)

mvn clean install
java -ea -jar target/crawlerApi-#.#.#.jar

NOTE: The -ea flag tells the java compiler to run the program with assertions enabled

Neo4j application setup

Note that this section will need to be replaced once we have an Anthos or Terraform script.

Step 1: setup firewall for neo4j

```bash
gcloud compute firewall-rules create allow-neo4j-bolt-http-https --allow tcp:7473,tcp:7474,tcp:7687 --source-ranges 0.0.0.0/0 --target-tags neo4j
```

Step 2: Get image name for Community version 1.4

```bash
gcloud compute images list --project launcher-public | grep --extended-regexp "neo4j-community-1-4-.*"
```

Step 3: create new instance

```bash
gcloud config set project cosmic-envoy-280619
gcloud compute instances create neo4j-prod --machine-type e2-medium --image-project launcher-public --image neo4j-community-1-4-3-6-apoc --tags neo4j,http-server,https-server
gcloud compute instances add-tags neo4j-stage --tags http-server,https-server
```

Step 4 : SSH to server and check status

```bash
gcloud compute ssh neo4j-stage
sudo systemctl status neo4j
```

Follow step 3 from this webpage to configure neo4j server - https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-neo4j-on-ubuntu-20-04

Step 6: Delete neo4j instance

```bash
gcloud compute instances delete neo4j-stage
```

Docker

```bash
maven clean install
docker build --tag crawlerApi .
docker run -p 80:80 -p 8080:8080 -p 9080:9080 -p 443:443 --name crawlerApi crawlerApi
```

Deploy Docker container to GCP Artifact Registry

```bash
gcloud auth print-access-token | sudo docker login -u oauth2accesstoken --password-stdin https://us-central1-docker.pkg.dev
sudo docker build --no-cache -t us-central1-docker.pkg.dev/cosmic-envoy-280619/api/#.#.# .
sudo docker push us-central1-docker.pkg.dev/cosmic-envoy-280619/api/#.#.#
```

Security

Configuring SSL - This should be replaced with API Gateway endpoint and ssl certificate

Generating a new PKCS12 certificate for SSL

Run the following command in Linux to create a keystore called api_key with a privateKeyEntry

  • Using CRT

     openssl pkcs12 -export -inkey private.key -in certificate.crt -out api_key.p12
  • Using PEM instead

     openssl pkcs12 -export -inkey crawlerApi.com.key -in crawlerApi.com-2022-chain.pem -out api_key.p12

Testing

Running Tests

Run the full test suite with Maven:

mvn test

Code Coverage

This project uses JaCoCo for code coverage reporting. Coverage reports are generated automatically when tests run.

To generate the coverage report:

mvn test

The HTML report is available at target/site/jacoco/index.html after the test run completes.

Test Structure

Tests are organized to mirror the main source tree under src/test/java/com/crawlerApi/:

Package Description
api/ Controller and API annotation tests
config/ Configuration class tests (CORS, Auth0)
generators/ Field generator tests (String, Integer, Decimal)
integrations/ Integration framework tests (facade, encryption, providers, factory)
models/ Domain model tests
security/ Security configuration and handler tests
service/ Service layer tests (Auth0Service)

Testing Frameworks

  • JUnit 5 (Jupiter) — test framework
  • Mockito — mocking and stubbing
  • JaCoCo — code coverage reporting

Design by Contract

This project follows Design by Contract (DbC) principles throughout the codebase. All public methods enforce their contracts at runtime:

Preconditions

  • Null checks: Objects.requireNonNull() is used at method entry for all required parameters (request objects, domain entities, identifiers, configuration objects).
  • Range checks: IllegalArgumentException is thrown for invalid numeric arguments (e.g., non-positive IDs, negative counts).
  • Constructor validation: All service constructors and value objects validate their dependencies and required fields on creation.

Postconditions

  • Assertions: assert statements verify return values and internal state after computation (e.g., generated values are non-null, sort specifications are valid).
  • Run the application with the -ea JVM flag to enable assertion checks.

Class Invariants

  • Immutable configuration: Integration metadata and provider configuration are validated at construction time and remain consistent throughout their lifecycle.
  • Repository contracts: Repository interfaces define clear query contracts via Spring Data method naming conventions.

Conventions

Layer Enforcement
API Controllers Objects.requireNonNull for request/body params; IllegalArgumentException for invalid IDs
Services Objects.requireNonNull for constructor dependencies and method arguments
Integration Providers Constructor-validated metadata; validateConfig() contract on all providers
Models / DTOs Constructor preconditions for required fields
Generators Postcondition assertions on generated values
Utility / Broadcaster Objects.requireNonNull on all static method parameters

About

A system to navigate, map and generate tests for your website.

Resources

License

Contributing

Stars

Watchers

Forks

Packages

 
 
 

Contributors

Languages