[fix] results: display internationalized domain names human-friendly

Some search engines encode domains that contain special characters
in the IDN format, i.e. by using Punycode (https://en.wikipedia.org/wiki/Punycode).

This means that domains formatted like that are not human readable
and thus very unintuive.

Also, as only some engines use Punycode, this often causes a result
to appear multiple times, once with a Punycode domain (e.g. xn--allestrungen-9ib.de),
and once with a normal domain (e.g. allestörungen.de).

Always formatting the domains nicely has the caveat that the official sites
are harder to distinguish from malicious clones that just swapped out some
characters by using Punycode, e.g. replaced `a` with `á`. I don't think
that will be much of an issue though - I don't think Punycode results
previously stopped users from clicking the link, and I also think that
most search engines filter out such bad results or don't even have them
indexed.

1 files changed, 6 insertions, 0 deletions

diff --git a/searx/result_types/_base.py b/searx/result_types/_base.py
index a08675d5f..94a55337c 100644
--- a/searx/result_types/_base.py
+++ b/searx/result_types/_base.py
@@ -52,6 +52,12 @@ def _normalize_url_fields(result: "Result | LegacyResult"):
             result.parsed_url = urllib.parse.urlparse(result.url)
 
     if result.parsed_url:
+        # properly format special characters (e.g. "ä", "ö") in IDN domains
+        # e.g. "xn--strung-xxa.de" becomes "störung.de"
+        if result.parsed_url.netloc.startswith("xn--"):
+            netloc = result.parsed_url.netloc.encode().decode("idna")
+            result.parsed_url = result.parsed_url._replace(netloc=netloc)
+
         result.parsed_url = result.parsed_url._replace(
             # if the result has no scheme, use http as default
             scheme=result.parsed_url.scheme or "http",