The Problem: Mock Responses Are a Manual Nightmare

Anyone who has ever built or tested a client application — mobile app, desktop agent, or web frontend — against a remote server knows the drill. The server is down. The network is unavailable. The staging environment is shared and broken again.

The standard solution is mitmproxy: a transparent HTTP proxy you run locally. Point your app at it, load a JSON file of canned responses, and your client happily fetches "live" data that is actually served from that file. No real server required.

The catch? Someone has to write that JSON file. For every endpoint, every status code, every edge case. That means manually curating HTTP captures, extracting bodies, categorising 200s from 404s, inventing plausible error payloads — and doing it all again when the API changes.

The Idea: Automate It — and Let AI Fill the Gaps

HARvest (HAR + harvest) automates the entire pipeline:

1. Capture real HTTP traffic with Charles Proxy, Chrome DevTools, or mitmproxy itself — export as a HAR file.
2. Feed the HAR file to HARvest.
3. HARvest parses every request/response pair, classifies them, and converts them into a mitmproxy-ready mock JSON.
4. With the --enrich flag, an AI model analyses every endpoint and synthesises edge cases — 401 Unauthorised, 503 Service Unavailable, malformed payloads, quota-exceeded errors — for scenarios that never appeared in real traffic.
5. Two files are produced: _positive.json for happy-path responses and _negative.json for errors and edge cases.
6. Drop either file into mitmproxy and test.

How HARvest Works

1. Parsing — HAR and mitmproxy flows, both welcome

HARvest auto-detects the input format. If the file looks like JSON it delegates to har_parser.py; otherwise it falls back to flow_parser.py for binary mitmproxy dump files. Both produce the same HarEntry dataclass: method, URL, status, MIME type, headers, and request/response bodies. Base64-encoded response bodies (exported by Chrome DevTools) are transparently decoded.

2. Classification — positive vs. negative

ai_enricher.py walks every entry and splits them into two buckets:

• Positive: HTTP 2xx and 3xx — successful, redirect, or cached responses.
• Negative: HTTP 4xx, 5xx, and status 0 (no response / network failure).

Real captured negatives always take precedence. If a 404 was observed for /api/session/start in real traffic, that exact response is kept. Synthetic cases only fill the gaps.

3. Mock generation — anonymised, plug-and-play JSON

mock_generator.py converts entries into the flat key → record schema consumed by the mitmproxy interceptor. To make the mock portable across customers and devices, it detects billing IDs (numeric segments ≥ 6 digits, or /customer/<id> patterns) and device IDs (/device/<id> patterns) and replaces them with {billing_id} / {device_id} placeholders. At runtime the interceptor extracts the real values from the inbound URL and substitutes them back into the response body — so a single mock file works for every customer.

{
  "api/v1/customer/{billing_id}/config": {
    "status": 200,
    "body": {
      "customerId": "{billing_id}",
      "features": ["reporting", "alerts"]
    }
  }
}

4. AI enrichment — synthetic edge cases Needs full integration

The --enrich flag triggers a synthetic negative generation pass. The intended target for this is IBM WatsonX (with IBM Granite as the candidate model) — the idea being to send each discovered endpoint to the LLM and ask it to generate realistic error responses that were never observed in real traffic: expired tokens, rate limits, malformed bodies, service outages, and so on.

As a working stand-in for the hackathon, synthetic edge cases were pre-generated manually (with AI assistance at development time) and stored in synthetic_negatives.py as a static Python dict. The merge logic is already in place: any path covered by a real captured negative is not overwritten; the LLM cases only fill the genuine gaps.

5. The mitmproxy interceptor — live traffic interception Working

harvest/interceptors/mitm.py is loaded as a mitmproxy addon script. It has two jobs:

• Mock serving: On every outbound request, it scans the URL against the compiled regex patterns from the mock JSON. If a match is found — accounting for {billing_id} / {device_id} placeholders — it returns the canned response immediately, bypassing the real server entirely. The Date header is refreshed to the current time so clients don't see stale timestamps.
• Upload decoding: For multipart upload endpoints (/bulkupload/, /payloadupload/), it silently decodes the binary protobuf-like payload, decompresses it with zlib, and writes each DataPacket to a timestamped XML file in the output directory — enabling offline analysis of upload traffic without needing the real server.

The CLI in Action

# Parse any capture file — auto-detects HAR or mitmproxy flow
python -m harvest parse-capture input/example.har

# Save all entries as a single flat mock JSON
python -m harvest parse-capture input/example.har --output parsed.json

# Enriched split — positive + negative files (uses bundled static edge cases today;
# will call WatsonX once the integration is validated)
python -m harvest parse-capture input/example.har --output parsed.json --enrich

Running the command shows a rich terminal table of every captured entry with method, status, MIME type, and URL. The --verbose flag prints full request and response bodies for deep inspection.

HARvest — parsing input/example.har

 Detected format: HAR

 ┏━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
 ┃ #  ┃ Method ┃ Status┃ MIME Type              ┃ URL                               ┃
 ┣━━━━╋━━━━━━━━╋━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
 ┃  1 ┃ POST   ┃   200 ┃ application/json       ┃ https://api.example.com/auth      ┃
 ┃  2 ┃ GET    ┃   200 ┃ application/json       ┃ https://api.example.com/user/42   ┃
 ┃  3 ┃ POST   ┃   401 ┃ application/json       ┃ https://api.example.com/refresh   ┃
 ┗━━━━┻━━━━━━━━┻━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

OK Positive cases -> output/parsed_positive.json  (2 entries)
OK Negative cases -> output/parsed_negative.json  (1 entries, 8 AI-generated)

Running the Interceptor

# macOS / Linux
mitmweb -s "/path/to/harvest/interceptors/mitm.py" \
        --mode local:YourApp \
        --set mock_file="/path/to/output/parsed_positive.json"

# Windows
mitmweb -s "C:\path\to\harvest\interceptors\mitm.py" `
        --mode local:YourApp.exe `
        --set output_dir="C:\Users\you\Desktop\payloads"

Both output_dir (where decoded upload XMLs land) and mock_file (which mock JSON to serve) are runtime options that can be changed without restarting mitmproxy. The interceptor prints a confirmation on startup and logs every mock hit and decoded upload file.

Architecture at a Glance

Key Design Decisions

Real traffic takes precedence over AI

If a 429 Too Many Requests was actually observed for /api/data/stream, that exact response is kept verbatim. Synthetic cases only fill endpoints where no real negative was captured. This means the mock is grounded in reality first and extended by AI second — regardless of whether the AI step is a static file today or a live WatsonX call tomorrow.

Placeholders for portable mocks

Hardcoding a specific billing ID or device serial into a mock file makes it useless for anyone else. HARvest detects IDs at parse time, anonymises them in both the path key and the response body, and lets the interceptor re-inject the real values from the live URL at request time.

Inlined payload decoder

mitmproxy loads addon scripts in its own embedded Python environment, which has no knowledge of the project package structure. Rather than fighting with PYTHONPATH, the payload decoder is inlined directly into mitm.py. This makes the script fully self-contained and drop-in deployable: copy one file, run one command.

Static edge cases as a stepping stone

Rather than blocking the whole tool on a WatsonX integration that was not yet testable, the synthetic negative cases were pre-authored (with AI assistance) and committed as a static dict in synthetic_negatives.py. This let us validate the full parsing → classification → mock generation → mitmproxy pipeline end-to-end. Swapping in live WatsonX calls is now a contained, well-defined task — replace one function, keep everything else.

What HARvest Solves for Teams

Beyond that, HARvest addresses several common pain points:

• Regression testing offline: Run your full test suite on a plane. The mock file captures the exact API contract your app was built against.
• Error handling confidence: Synthetic edge cases cover status codes that may never appear in a normal capture — ensuring your app handles 503s and 429s before they hit production.
• Secrets and PII scrubbing: Billing IDs and device identifiers are replaced automatically. The committed mock file contains no customer data.
• Protocol debugging: The upload payload decoder gives visibility into binary multipart request bodies — something that is otherwise invisible in network tools.

Getting Started

git clone https://[to be updated]/harvest.git
cd harvest
pip install -r requirements.txt

# Run against your own capture (uses bundled static edge cases)
python -m harvest parse-capture input/my_capture.har --output mocks.json --enrich

# Future: set WatsonX credentials once the live integration is validated
export WATSONX_API_KEY=your_key
export WATSONX_PROJECT_ID=your_project_id

The only runtime dependencies are click, rich, and mitmproxy. The ibm-watsonx-ai package is needed for the live enrichment path but is not required for the tool to function today.

What's Next

1. WatsonX / Granite PoC — exercise watsonx_client.py against a live WatsonX endpoint, validate the prompt format and JSON output, and wire it into the --enrich pipeline. This is the single highest-priority item before a broader release.
2. OpenAPI spec parser — parse YAML/JSON OpenAPI specifications to extract endpoint schemas and response codes, letting HARvest generate mocks for endpoints that never appeared in real traffic captures.
3. Full WatsonX integration — once the PoC is validated, replace synthetic_negatives.py with a live, on-demand LLM call so the edge-case library stays current with every new capture.

HARvest was built for the IBM WatsonX Hackathon but solves a problem that every team shipping client applications faces. The core pipeline — parse, classify, anonymise, mock — is solid and working today. The AI enrichment layer via WatsonX is the piece we are most excited to close out.