theHarvester 5.0: A New Era

Development preview

This post covers work in progress toward theHarvester 5.0. Features and interfaces may change before release, and the public v5.0.0 release is not available yet.

Why this release matters

It has been a few years since I worked seriously on theHarvester. After graduating college and coding professionally, I found it much harder to want to code when I was not working. The rise of increasingly capable AI tools has reignited my passion for development. It makes me wonder what the tool would look like now if those tools had been available when I was porting it to Python 3 or making it fully asynchronous with Stack Overflow as my copilot.

I can now apply the skills I have learned over the past few months to a tool many people use and take it to the next level. This is the biggest release since the tool's inception, so this post focuses on four changes:

• richer, source-attributed JSONL and SQLite evidence;
• more precise source selection and discovery controls;
• new discovery capabilities, including vhost, PTR, and bounded recursive DNS discovery; and
• HarvestView, a web app for running searches and examining saved results.

There is much more than I can cover here. The full list is in the changelog. I preserved legacy behavior where it still made sense, and the documented migration path should make the necessary convention changes straightforward. If anything is unclear, feedback is only a GitHub issue away. With that, let's dive in.

Enhanced Output and DB Storage

One of the main goals of this release was to enrich the results theHarvester obtains and preserve them more effectively. Previously, there were three storage options: a SQLite database, a JSON output file, and an XML output file. An older HTML output option had already been removed; HarvestView now provides a much more capable way to inspect results in a browser.

Each option had its own flaws. The biggest were missing source attribution, incomplete storage for some sources and modes, and formats that were difficult to parse or understand at a glance.

That has changed. The database has been rebuilt with SQLAlchemy, and every run now receives a UUID before being stored in one unified database.

The -f flag now produces a third output format: JSONL. It carries richer data while remaining straightforward to stream and parse. The legacy JSON and XML files are still generated for compatibility, but they do not contain the same level of detail. The following figures compare the legacy JSON and SQLite formats with the new evidence-oriented models.

JSON preserves compatibility; JSONL provides independently processable, source-attributed evidence.Scroll to inspect

SQLite 5.0 stores a reusable run record instead of an isolated list of findings.Scroll to inspect

These diagrams were created with GPT-5.6 Sol and the Diagram Design skill.

Source Capability Mapping and Easier Source Selection

Speaking of data storage, it is important to talk about the sources themselves. Each source is now categorized by a ResultRoute that describes the type of data it can return.

theHarvester/result_routes.py

class ResultRoute(Enum):
    """Normalized result collections a source can contribute.

    ``SUBDOMAINS`` contains in-scope descendant names reported by a source. It
    does not imply DNS resolution or current addressability. Legacy adapters
    and output formats may still call these values hosts for compatibility.
    """

    SUBDOMAINS = auto()
    EMAILS = auto()
    IPS = auto()
    ASNS = auto()
    PEOPLE = auto()
    LINKS = auto()
    URLS = auto()
    INTERESTING_URLS = auto()
    BREACHES = auto()

_ROUTE_CAPABILITIES = {
    ResultRoute.SUBDOMAINS: 'subdomains',
    ResultRoute.EMAILS: 'emails',
    ResultRoute.IPS: 'ips',
    ResultRoute.ASNS: 'asns',
    ResultRoute.PEOPLE: 'people',
    ResultRoute.LINKS: 'urls',
    ResultRoute.URLS: 'urls',
    ResultRoute.INTERESTING_URLS: 'urls',
    ResultRoute.BREACHES: 'breaches',
}

This enables more precise source selection than -b all or -b source1,source2,source3. For example, -b subdomains or -b asns runs only sources with those route capabilities. Routes and sources can also be combined, as in -b subdomains,asns or -b subdomains,asns,hunter,ips. -b all remains available, but selecting the information you actually need can save time and resources. The README contains a complete source-capability table.

Additional Discovery Capabilities

I suspect most people ran uv run theHarvester -d example.com -f results -b all and called it a day. However, theHarvester has many additional features. Screenshot capture, DNS brute forcing, API route discovery, DNS resolution, and subdomain takeover checks have all been refactored and upgraded.

This release also introduces vhost discovery, PTR record discovery, neighboring IP CIDR analysis, RouteViews enrichment, and bounded recursive DNS discovery. New capabilities were not the release's only focus, but they extend the kinds of investigations theHarvester can support. Every source has also been reviewed: new sources were added, API routes were upgraded, and existing integrations were reworked where they could return better results. The wiki has been rebuilt with more detail about each capability.

Results from these features flow into the SQLite database and JSONL output alongside the rest of the run evidence.

The following diagram shows how discovery and enrichment now converge on a normalized evidence contract.

Discovery sources and explicit actions converge on one attributable evidence contract.Scroll to inspect

HarvestView

Last but certainly not least is the feature I am most excited to discuss: HarvestView, a full web app for running theHarvester and analyzing its results. It replaces the old REST API with a more complete interface. Under the hood, the frontend is deliberately straightforward: a single HTML file and a single JavaScript file use Tabulator to display tables backed by FastAPI. HarvestView can also schedule scans for continuous coverage.

When I created the original REST API, I mainly wanted a reason to use FastAPI after being impressed by its first release. FastAPI has since become widely adopted, and I have used it for years. This time I wanted to build an accompanying web app with a proper API, parity with the CLI, and a much easier way to use theHarvester. Existing runs can also be exported and imported through JSONL or SQLite files.

HarvestView carries an authorized plan through durable execution, inspection, and interchange.Scroll to inspect

The web app's styling was developed mainly with the Impeccable skill. No longer having to visit Stack Overflow to center a div manually is still one of my favorite uses of AI: frontend development no longer has to be a headache.

Updating theHarvester for the AI Era of Development

TheHarvester now includes AGENTS.md and CONTEXT.md files that make it easier for developers and coding agents to work with the repository. The wiki, README, and issue templates have also been revised. While vibe coding is a useful way to prototype features, the resulting code must still be readable and maintainable. I will discuss the development process in part two, including how these changes were completed in such a short time.

One of the biggest obstacles was trying to add features or fix bugs with frontier AI models without a proper harness. That led to overengineered code, implementations that missed the bigger picture, and fixes that introduced new bugs. Skills such as Matt Pocock's engineering skills and Ponytail helped me stay on top of the work by grilling assumptions and mapping features before implementation. TheHarvester is now much more developer-friendly, and it is embracing AI-assisted development without treating generated code as automatically correct. This is only the beginning.

Follow theHarvester releases for the eventual 5.0.0 release and final notes.

Thank you for taking the time to read this. If you encounter a bug or have an idea for a new feature, create an issue. I look forward to hearing how the new version feels to use.