Show HN: A blocklist to remove spam and bad websites from search results

Even Google is plagued by spam, I've tried all sorts of search techniques and alternative engines but I feel like the only solution seems to be doing things manually. I was already starting to block things by myself but I thought it'd be more productive to make the list public and try crowdsourcing. Even now, searching "how to partition a hard disk" would often drive you to low-effort sites telling you to use their software.

> Unfortunately, this kind of blocklist is highly subjective. This list blocks MSN.com! That's hardly what I would have chosen.

It's definitely a bit opinionated, but it's open to discussion - you can create an unblock request issue (if you care enough to do so, of course!). The reason I blocked MSN is that it just re-hosts articles from other websites, so I'd rather see the official source than be tricked into Microsoft's site which is very annoying, like how it opens another article if you scroll too fast down.

Afaik DDG is just Bing, whereas Kagi is using Google, Bing, (Yandex?) among others - https://help.kagi.com/kagi/search-details/search-sources.htm...

As a Kagi user I actually haven’t encountered much search result spam, surprised you’re seeing enough there to drive you back to Google!

> Unfortunately, this kind of blocklist is highly subjective. This list blocks MSN.com! That's hardly what I would have chosen.

I'm wondering how much the blacklist can be broken down into categories of spam. Sponsorblock for YouTube has a lot options around the types of things it'll skip through and the user has choice in how they're handled (skipped automatically, prompted to skip, simply highlighted in the scrubbar) at the category level.

I get tons when looking up recipes and cooking related information. Things that will say "X can be refrigerated for up to two weeks" then in the next paragraph "X is fine to refrigerate and eat for 2-3 days" or similar.

I'd block them but there seem to be infinite. They're probably buying 10+ character domains using random words/names/phrases in bulk.

You can use ublacklist without a list and just block shit sites as you see them.

I'm loving being able to search for something without getting results from garbage sites like howtogeek, stackoverflow, MSN, Pinterest, etc.

Yes and no.

Power-law relations mean that a small number of domains will account for the lion's share of low-relevance results, and filtering those out will result in dramatic improvements in relevance.

That small set is probably fairly dynamic, however, and will likely change at a fairly high rate over time.

Penny-ante sites are less likely to appear in generic results, but might well be whatever the spam/phish term is for junk general Web search results.

We may well come to rely more on whitelisting, but I think at least for now that's not necessary, largely due to the dynamics of publishing / attention economies themselves.

This. I think, well curated web directories (by humans and machines) deserve a comeback.

You could get a step closer to that and integrate it into your DNS: https://github.com/StevenBlack/hosts

The upside is that it would go beyond your browser to anything on your machine that makes a DNS request.

> Another great function (not for this plugin) should be the option to "bundle" all search results from the same domain. Stuff them under one collapsible entry.

That would be really cool. Just zip it up if you don't want to see that domain for that specific search.

I think this would block you from visiting the websites, but they'd still show up on search results. uBlacklist doesn't block them, but rather just hides them for search engines which IMO is a better approach.

Hi! I'm mostly going through them manually. Not all of the domains in the list are literally spam - most of the list also includes misleading sites like corporate blogs that trick people into downloading their software.

You might be interested in the AI spam/low effort section though, one that tops DDG often are these AI generated tech articles: https://github.com/popcar2/BadWebsiteBlocklist/issues/1

They're the same site under different domains, you can tell it's AI by its writing style, how much they churn out per day, how little info there is about who's writing it, how similarly the about pages are written, and how the same article is suspiciously also in similar-sounding sites.

Another one I just caught today that was on top of page 1: https://github.com/popcar2/BadWebsiteBlocklist/issues/84

I'll be sure to report these sites as I'm adding to the list, thanks.

I feel like lots of sites that are censored by the bing and google are also being censored by DDG. Is there really any point of filtering out more sites? Search results are so meaningless now. All results simply point to the same 200 links page after page.

That was my thought as well. Their UI is great for one-at-time operations, but an API endpoint I could curl and sync with a local file I keep in git would be killer.

Although, using this via the extension would make it cross-platform so the block affects kagi and google, which could be nice.

Although, that would require manual syncing between devices, which would not be nice.

Although, uploading it to kagi through API doesn't mean I have to not use the extension, so having the cake and eating it too may be possible.

Was thinking of that as I was browsing this doc! I just did the ole' reddit.com -> old.reddit.com redirect via kagi yesterday.

Some kind of democratic process. Where membership and blacklist are both something arrived at democratically.

It could be simple.

Good?

Some kind of community notes consensus system could make sense here to find common ground. When a diverse set of people agree that a site should belong to the list, only then it is added.

They aren't, that's why they're blocked. You can see more detail in the issues but the blocked corporate blogs often make clickbait to advertise their product, like https://www.ccleaner.com/knowledge/windows-11-problems-how-t...

Because corporate blogs are predominantly nothing but marketing fluff that dominates search results so thoroughly that they drown out any actual useful information.

Google used to do this years ago but clawed it back around the time they started removing _all_ customizations under the premise of, "we know how to customize your results better than you do."

DuckDuckGo has site blocking. The problem is that there are so many SEO-optimized blogspam, referral link, and other "garbage" sites that you could spend a lifetime blocking each one individually before you get any actual work done. And it's only getting worse now that LLMs can generate a whole web site for you in a matter of minutes. I imagine a dedicated individual could provision several thousand websites/blogs per day, just chock full of ads and referral links.

Why Medium?

She really put my thoughts into words regarding this. The “who is this for” part really hit home.

I get Kagi since you pay for it, but doesn't using a logged in account with DDG defeat the purpose of using DDG? How does having a search history associated with your DDG account improve things? You're just moving who knows your search history from one org to another

I guess the author's "boiling hatred to bad tech support articles" leads to some overreach

DNS blacklists work fine for blocking access to sites or certain known-sketchy FQDNs/domains but do nothing to hide low-quality search engine results, which is what this is all about.

I'm sorry but you're missing the obvious fact that $0.000000001 per user per financial quarter is simply worth incomparably more than the future of humanity's access to knowledge and learning.

Have you paused even for a moment to think about what would happen to the poor shareholders if they put that one dude on this job..?

I'll manually review all the block/unblock requests, and you can open a discussion in the issues page. There's a template and tag for discussion posts.

I don't think AI search engines are a proper replacement for regular ones. The summaries are almost always worse than finding a proper article, which gives better context and editing (and doesn't make many subtle mistakes).