cve-mitigation

---
name: cve-mitigation
description: >
  Scan, triage, and remediate CVEs in the Gitpod monorepo. Use when asked to
  fix vulnerabilities, update dependencies for security, respond to CVE scan
  failures, or validate that dependency updates resolve known CVEs. Triggers on
  "fix CVE", "vulnerability scan", "security update", "dependency CVE",
  "grype scan", "leeway sbom", "CVE remediation", "update vulnerable dependency",
  "daily scan failure", "critical vulnerability".
---

# CVE Mitigation

Fix and validate CVE findings in the Gitpod monorepo. This repo has ~80 Go modules
under `components/`, `dev/`, `install/`, and `test/`, plus TypeScript/Node packages.

## Scanning

### Fast local scan (grype)

Install grype if missing, then scan individual modules or the whole repo:

```bash
# Install grype (one-time)
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin

# Scan a single module (fastest — ~2s per module)
grype file:components/common-go/go.mod --only-fixed --output table

# Scan a single module directory (slower — ~40s, also picks up binaries/lockfiles)
grype dir:components/common-go --only-fixed --output table

# JSON output for programmatic use
grype file:components/common-go/go.mod --only-fixed --output json 2>/dev/null | \
  jq '[.matches[] | {id: .vulnerability.id, severity: .vulnerability.severity, package: .artifact.name, installed: .artifact.version, fixedIn: .vulnerability.fix.versions}]'

# Scan all Go modules, show only Critical/High with fixes available
find components -name "go.mod" -not -path "*/vendor/*" \
  -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \;
```

Use `--fail-on critical` or `--fail-on high` to get a non-zero exit code.

### CI scan (leeway sbom)

CI uses `leeway sbom export` + `leeway sbom scan` against `components:needs-vuln-scan`.
Leeway embeds both syft (SBOM generation) and grype (vulnerability matching) internally.
The CI scan is slow (requires full build cache) and runs on every push to `main` and on PRs.

The daily scheduled run (`build.yml`, cron `0 0 * * *`) checks `vulnerability-stats.json`
for critical counts and sends Slack alerts via `notify-scheduled-failure`.

Read `references/ci-scanning.md` for the full CI pipeline details.

### Trivy image scanning

A separate Trivy-based scan (`scripts/trivy/trivy-scan-images.sh`) scans built Docker
images. It uses `scripts/trivy/trivyignore.yaml` for suppressions. This is distinct from
the Go dependency scanning.

## Triage

When a CVE is reported:

1. **Check severity and CVSS score.** Focus on Critical (CVSS ≥ 9.0) and High (CVSS ≥ 7.0) first.
2. **Check if a fix exists.** Use `--only-fixed` with grype. If no fix exists, document and defer.
3. **Check exploitability in context.** Not all CVEs are exploitable in Gitpod's usage. If a CVE
   is a false positive or not exploitable, add it to `WORKSPACE.yaml` under `sbom.ignoreVulnerabilities`
   with a `vulnerability` ID and a `reason` explaining why.
4. **Check for breaking API changes.** Some upgrades (e.g., `runc` v1.1.x → v1.2.x) change
   exported APIs. These require code migration, not just `go get`.

## Remediation (Go dependencies)

### Standard update flow

For each affected module:

```bash
cd components/<module>

# Update a specific dependency
go get google.golang.org/grpc@v1.79.3

# Update transitive dependencies that need bumping
go get golang.org/x/net@latest

# Clean up
go mod tidy

# Verify it builds
go build ./...
```

### Multi-module coordination

This repo has many Go modules that share dependencies via `replace` directives in `go.mod`.
When updating a dependency:

1. **Start with leaf modules** (APIs, protocol packages) and work up to service modules.
2. **Check replace directives.** Many modules use `replace` to point to local paths:
   ```
   replace github.com/gitpod-io/gitpod/common-go => ../common-go // leeway
   ```
   If `common-go` bumps a dependency, all modules that replace-reference it will
   transitively pick up the new version.
3. **Update all modules that directly depend on the vulnerable package.** Use:
   ```bash
   grep -rl "google.golang.org/grpc" components/*/go.mod | sort
   ```
4. **Don't forget `dev/` and `test/` modules.** CI builds these too. They often lag behind.
5. **Run `go mod tidy` in every updated module.** Missing this causes CI failures.

### OpenTelemetry replace directives

Some modules (notably `image-builder-bob`) use `replace` directives to pin OpenTelemetry
versions. When upgrading grpc, the OTel SDK version requirements may change. Update the
replace directives to match:

```go
replace (
    go.opentelemetry.io/otel => go.opentelemetry.io/otel v1.39.0
    go.opentelemetry.io/otel/sdk => go.opentelemetry.io/otel/sdk v1.39.0
    // ... etc
)
```

### Deprecated package migration

When a package is deprecated and replaced (e.g., `dgrijalva/jwt-go` → `golang-jwt/jwt/v5`):

1. Update `go.mod`: remove old dependency, add new one.
2. Update import paths in `.go` files.
3. Verify API compatibility — most maintained forks keep the same API surface.
4. Run `go build ./...` and `go test ./...` to confirm.

### Packages with breaking changes

Some upgrades cannot be done with a simple version bump. Document these in the PR
under "Not addressed (requires follow-up)" with the specific breaking change.
Example: `opencontainers/runc` v1.1.x → v1.2.x made `libcontainer/cgroups/ebpf`
functions unexported.

## Suppressing false positives

Two suppression mechanisms exist:

### WORKSPACE.yaml (leeway sbom scan)

Add entries under `sbom.ignoreVulnerabilities` in `WORKSPACE.yaml`:

```yaml
sbom:
  enabled: true
  ignoreVulnerabilities:
    - vulnerability: GHSA-xxxx-xxxx-xxxx
      reason: |
        Explanation of why this CVE is not exploitable in Gitpod's usage.
```

### Trivy ignore (image scanning)

Add entries to `scripts/trivy/trivyignore.yaml`:

```yaml
vulnerabilities:
  - id: CVE-YYYY-NNNNN
    paths:
      - "app/node_modules/package/package.json"
    statement: |
      Explanation.
```

## Validation

After making changes, validate locally before pushing:

```bash
# Quick: scan only the modules you changed
grype file:components/<module>/go.mod --only-fixed --output table

# Thorough: scan all components
find components -name "go.mod" -not -path "*/vendor/*" \
  -exec sh -c 'echo "=== {} ===" && grype file:"{}" --only-fixed -q 2>/dev/null' \; \
  2>&1 | grep -E "^===|Critical|High"

# Build verification for key modules
for mod in common-go supervisor public-api-server ws-proxy ws-manager-mk2 \
           registry-facade blobserve content-service image-builder-mk3 \
           gitpod-cli local-app ide-metrics ide-service node-labeler \
           openvsx-proxy service-waiter ws-daemon image-builder-bob usage; do
  echo "=== Building components/$mod ===" && \
  (cd "components/$mod" && go build ./...) || echo "FAILED: $mod"
done
```

## PR conventions

- Title: "Fix critical CVEs from daily vulnerability scan" or similar.
- Body: Include a table of updated packages with From/To versions and CVE IDs.
- Tag with `team: staff-engineers` and `team: team-enterprise`.
- Reference the Linear issue (e.g., "Fixes: CLC-XXXX").
- List any CVEs that could NOT be addressed and why under "Not addressed (requires follow-up)".
- Co-author: `Co-authored-by: Ona <no-reply@ona.com>`

## Reference files

- `references/ci-scanning.md` — CI pipeline details for vulnerability scanning.
- `references/go-modules.md` — List of all Go modules and their scan-relevant groupings.