theHarvester 5.0: A Renaissance in Reconnaissance

Development preview

This post covers work in progress toward theHarvester 5.0. Features and interfaces may change before release, and the public v5.0.0 release is not available yet.

Why this release matters

It has been a few years since I worked seriously on theHarvester. After graduating college and coding professionally, I found it much harder to want to code when I was not working. The rise of increasingly capable AI models has reignited my passion for development. It makes me wonder what the tool would look like now if those models had been available when I was porting it to Python 3 or making it fully asynchronous with Stack Overflow as my copilot.

I can now apply what I've learned over the past few months to a tool many people use and take it to the next level. This is the biggest release since the tool's inception, so this post focuses on four changes:

• richer, source-attributed JSONL and SQLite evidence;
• more precise source selection and discovery controls;
• new discovery capabilities, including vhost, PTR, and bounded recursive DNS discovery; and
• HarvestView, a web app for running searches and examining saved results.

There is much more than I can cover here. The full list is in the changelog. I preserved legacy behavior where it still made sense, and the documented migration path should make the necessary convention changes straightforward. If anything is unclear, feedback is only a GitHub issue away. With that, let's dive in.

Enhanced Output and DB Storage

One of the main goals of this release was to enrich the results theHarvester obtains and preserve them more effectively. Previously, there were three storage options: a SQLite database, a JSON output file, and an XML output file. An older HTML output option had already been removed; HarvestView now provides a much more capable way to inspect results in a browser.

Each option had its own flaws. The biggest were missing source attribution, incomplete storage for some sources and modes, and formats that were difficult to parse or understand at a glance.

That has changed. The database has been rebuilt with SQLAlchemy, and every run now receives a UUID before being stored in one unified database.

The -f flag now produces a third file format option: JSONL. It carries richer data while remaining straightforward to stream and parse. The legacy JSON and XML files are still generated for compatibility, but they do not contain the same level of detail. The following figures compare the legacy JSON and SQLite formats with the new evidence-oriented models.

JSON preserves compatibility; JSONL provides independently processable, source-attributed evidence.Scroll to inspect

SQLite 5.0 stores a reusable run record instead of an isolated list of findings.Scroll to inspect

These diagrams were created with GPT-5.6 Sol and the Diagram Design skill.

You may also have noticed P0, P1, and P2. These new theHarvester classifications describe how each activity interacts with providers, infrastructure, and targets.

Class	Meaning
P0	Queries an existing provider or dataset
P1	Makes resolver-facing DNS requests about authorized targets
P2	Contacts a target endpoint or causes equivalent direct interaction

Source Capability Mapping and Easier Source Selection

Speaking of data storage, it is important to talk about the sources themselves. Each source is now associated with a single or multiple ResultRoute that describes the types of data it can return.

theHarvester/lib/source_catalog.py

class ResultRoute(Enum):
    """Result types a source can return.

    ``SUBDOMAINS`` contains names found below the target domain. A name can
    appear here even if it does not currently resolve.
    """

    SUBDOMAINS = auto()
    EMAILS = auto()
    IPS = auto()
    ASNS = auto()
    PEOPLE = auto()
    URLS = auto()
    BREACHES = auto()

This makes source selection more targeted than just doing a blanket -b all. For example, -b subdomains or -b asns runs only sources with those specific route capabilities. Routes and sources can also be combined, such as with -b subdomains,asns or -b subdomains,asns,hunter,ips. Of course -b all remains available, but in certain cases it's useful to just query for the information you need which leads to saved time and resources. The README contains a complete source-capability table.

Additional Discovery Capabilities

I suspect most people ran uv run theHarvester -d example.com -f results -b all and called it a day. However, theHarvester has many additional features. Screenshot capture, DNS brute forcing, API route discovery, DNS resolution, and subdomain takeover checks have all been refactored and upgraded.

This release also introduces vhost discovery, PTR record discovery, neighboring CIDR and IP analysis, RouteViews enrichment, and bounded recursive DNS discovery. New capabilities were not the release's only focus, but they extend the kinds of investigations theHarvester can support. Every source has also been reviewed: new sources were added, API routes were upgraded, and existing integrations were reworked where they could return better results. The wiki has been rebuilt with more detail about each capability.

Results from these features flow into the SQLite database and JSONL output alongside the rest of the run evidence.

The following diagram shows how discovery and enrichment now converge on a normalized evidence contract.

Discovery sources and explicit actions converge on one attributable evidence contract.Scroll to inspect

HarvestView

Last but certainly not least is the feature I am most excited to discuss: HarvestView, a full web app for running theHarvester and analyzing its results. It replaces the old standalone REST API experience with a richer REST API with many more options with an accompanying feature rich web app. Under the hood, the frontend is deliberately straightforward: a single HTML file and a single JavaScript file use Tabulator to display tables backed by FastAPI. HarvestView can also schedule scans for continuous coverage.

When I created the original REST API, I mainly wanted a reason to use FastAPI after being impressed by its first release. FastAPI has since become widely adopted, and I have used it for years. This time I wanted to build an accompanying web app with a proper API, parity with the CLI, and a much easier way to use theHarvester. Existing runs can also be exported and imported through JSONL or SQLite files.

HarvestView carries an authorized run from execution through inspection, export, and reuse.Scroll to inspect

The web app's styling was developed mainly with the Impeccable skill. No longer having to visit Stack Overflow to center a div manually is still one of my favorite uses of AI: frontend development no longer has to be a headache.

HarvestView's dashboard after scanning example.com, with the completed run and collected results ready for review.

HarvestView's scheduling view for configuring and managing scans from the web app.

Updating theHarvester for the AI Era of Development

theHarvester now includes AGENTS.md and CONTEXT.md files that make it easier for developers and coding agents to work with the repository. The wiki, README, and issue templates have also been revised. While vibe coding is a useful way to prototype features, the resulting code must still be readable and maintainable. I will discuss the development process in part two, including how these changes were completed in such a short time.

One of the biggest obstacles was trying to add features or fix bugs with frontier AI models without a proper harness. That led to overengineered code, implementations that missed the bigger picture, and fixes that introduced new bugs. Tools such as Matt Pocock's engineering skills and Ponytail helped me stay on top of the work by challenging assumptions and helping map out complex features before implementation. TheHarvester is now much more developer-friendly, and it is embracing AI-assisted development without treating generated code as automatically correct. This is only the beginning.

Follow theHarvester releases for the eventual 5.0.0 release and final notes.

Thank you for taking the time to read this. If you encounter a bug or have an idea for a new feature, create an issue. I look forward to hearing how the new version feels to use.