See how ignore list works

The guide explains how the ignore list works in Adverse Media Monitoring (AMM): what it is, where it lives, how it is applied in the pipeline, and what kinds of links are ignored and why.

What is ignore list

The ignore list is a configurable list of website patterns and hosts that AMM uses to accomplish the following:

• Exclude links that are not useful sources of Adverse Media (for instance, social media, information aggregators, company home pages)
• Avoid downloading content from websites that are behind paywalls or known to block or forbid automated access
• Block unsafe domains known to host malware or otherwise risky

An entry in the ignore list includes the following:

• web_address: a pattern or host to match
• mode: the UNSAFE, IGNORE, or DO_NOT_DOWNLOAD flag
• note: a short reason or category description, for instance, Paywall, Information provider

Where ignore list is applied

Within the AMM solution pipeline, the ignore list is applied as follows:

• Search results filtering, for instance, with the Google provider

  Any links flagged as UNSAFE and IGNORE are removed from further processing.

• Content downloads and validation

  The DO_NOT_DOWNLOAD flag and the special Paywall subset stop the downloader from fetching content, and Evan returns a corresponding error code.

How ignore list entries are checked

AMM normalizes URLs to lowercase before matching. The matching patterns are as follows:

• UNSAFE mode (isSafe)

  • The URL host (domain) is extracted and compared for exact matches with entries in the UNSAFE list.

  • The check covers host-only values (for instance, x.co). Path fragments, wildcards, or leading and trailing dots are not matched.

• IGNORE (isNotIgnored) and DO_NOT_DOWNLOAD modes (isInDoNotDownloadList and isInBehindThePaywallList)

  • The full URL string is checked using the simple contains() substring against each entry.

  • Use case-insensitive substrings that uniquely identify the target (for instance, .bloomberg., https://docs., /maps/).

• Paywall subset

  • The subset is technically part of DO_NOT_DOWNLOAD with the additional == "Paywall" note.

  • Produces the WEBSITE_BEHIND_PAYWALL error before a download is attempted.

What are ignore list modes

UNSAFE

• What happens: a link is dropped early as unsafe.

• Why: security and content risk, for instance, malware, known malicious shorteners.

• How to use: set the mode to UNSAFE and use an exact host (for instance, x.co).

IGNORE

• What happens: a link is removed from results and not processed further.

• Why: the source is not considered a valid Adverse Media source (for instance, social networks, directories, info aggregators, maps).

• How to use: set the mode to IGNORE and provide a specific substring from the full URL.

DO_NOT_DOWNLOAD

• What happens: before fetching an article content, AMM checks it and stops the download if a URL matches specific patterns.

• Why: sites block bots, have paywalls, forbid scraping, or provide content not suitable for automated parsing.

Paywall subset

This is a corner case for the DO_NOT_DOWNLOAD mode and works the same way. However, such entries are assigned the additional Paywall flag to identify the cause more clearly.

What links are ignored and why

note

The examples below are based on the categories used in ignore_list.csv (see the note column).

Source type	Mode	Rationale	Examples
Information providers and aggregators	IGNORE	Not original news articles, typically profile or aggregated data (for instance, people directories, salary or HR sites, Q&A, code or package registries).	beenverified.com, spokeo.com, ziprecruiter.com, stackoverflow.com, npmjs.com, github.com, wiktionary, urbandictionary
Company or organization home and career sites	IGNORE	PR and marketing pages or corporate data that are not Adverse Media news.	santander.*, textron.com, farmers.com, casella.com
Social networks and media sharing	IGNORE	User-generated content, unreliable for automated Adverse Media extraction; often requires login or JS.	facebook, youtube, twitter, instagram, tumblr, linkedin, pinterest, tiktok, vimeo, soundcloud
Map providers	IGNORE	Location or map pages are not news articles.	google.com/maps, mapquest.com, bing.com/maps, yahoo.com/maps, waze.com
Review and listing sites	IGNORE	Business listings or ratings, not news articles.	yelp.com, manta.com, zaubee.com, realtor.com, zillow.com
Media hosting, Content Delivery Network (CDN), file mirrors	IGNORE	File repositories or embedded media pages rather than articles.	s3.amazonaws.com, issuu.com, jstor.org, books.google
Paywalled outlets	DO_NOT_DOWNLOAD; Paywall subset	Content cannot be fetched by the AI Agent.	.bloomberg., .ft.com, .lexology.com
Other non-downloadable content sites	DO_NOT_DOWNLOAD	Known to block or forbid automated download or not compatible with parsers.	supermarketnews.com, naturalgasintel.com
Unsafe websites	UNSAFE	Security risk (malware, hacked sites, malicious shorteners).	Host-only entries such as x.co (an exact host match required).

Tips for adding ignore list entries

• For the UNSAFE mode, use the exact domain or host. No protocol, path, or wildcards. Example: x.co.

• For the IGNORE and DO_NOT_DOWNLOAD modes, use a minimal unambiguous substring that appears in the full URL. Examples:

  • A domain fragment with dots to avoid False Positives, such as .bloomberg

  • A path fragment for specific sections: /maps/ or https://docs

• Use the note field to categorize the reason, for instance, Paywall, Information provider, or Social Media.

• Keep patterns lowercase as they are anyway normalized to lowercase in the code.

Error codes and reporting

• If a site is behind a paywall, AMM returns WEBSITE_BEHIND_PAYWALL.

• If a site is flagged as DO_NOT_DOWNLOAD (non-paywall), AMM returns WEBSITE_FORBIDDEN_DOWNLOAD.

• Some download errors can be suppressed as ignorable in search tasks (for instance, EXCLUDED_FORMAT, ARTICLE_NOT_FOUND), but this is a case separate from the ignore list usage.

info

See also Ignore list FAQs.