diff options
| author | Brock Vojkovic <git@vojk.au> | 2026-03-06 22:40:44 +0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2026-03-06 15:40:44 +0100 |
| commit | 68ff08f2244c6fa9e95255eb6733c157be2f46a7 (patch) | |
| tree | 59cbe50ff98dc9cb60f83f87c48c03cbcc2d9e4b /searxng_extra | |
| parent | c7ba2158f9062f8f5ee3b9f72b5da61d3370ad23 (diff) | |
[mod] drop fasttext-predict (#5795)
Removes the `fasttext-predict` dependency and the language detection code.
If a user now selects `auto` for the search language, the detected language now
falls back directly to the `Accept-Language` header sent by the browser (which was already the fallback when fasttext returned no result).
- fasttext's [language detection is unreliable](https://github.com/searxng/searxng/issues/4195) for some languages, especially short search queries, and in particular for queries containing proper names which is a common case.
- `fasttext-predict` consumes [significant memory](https://github.com/searxng/searxng/pull/1969#issuecomment-1345366676) without offering users much real value.
- the upstream fasttext project was archived by Meta in 2024
- users already have two better alternatives: the `Accept-Language` header and the search-syntax language prefix (e.g. `:fr` or `:de`).
Related: https://github.com/searxng/searxng/issues/4195
Closes: https://github.com/searxng/searxng/issues/5790
Diffstat (limited to 'searxng_extra')
| -rwxr-xr-x | searxng_extra/update/update_engine_descriptions.py | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/searxng_extra/update/update_engine_descriptions.py b/searxng_extra/update/update_engine_descriptions.py index 769af128a..d200a70bb 100755 --- a/searxng_extra/update/update_engine_descriptions.py +++ b/searxng_extra/update/update_engine_descriptions.py @@ -19,7 +19,7 @@ from searx.engines import wikidata, set_loggers from searx.utils import extract_text, searxng_useragent from searx.locales import LOCALE_NAMES, locales_initialize, match_locale from searx import searx_dir -from searx.utils import gen_useragent, detect_language +from searx.utils import gen_useragent import searx.search import searx.network from searx.data import data_dir @@ -169,7 +169,7 @@ def get_website_description(url, lang1, lang2=None): lang = extract_text(html.xpath('/html/@lang')) if lang is None and len(lang1) > 0: lang = lang1 - lang = detect_language(description) or lang or 'en' + lang = lang or 'en' lang = lang.split('_')[0] lang = lang.split('-')[0] return (lang, description) |