[refactor] typification of SearXNG (initial) / result items (part 1)

Typification of SearXNG
=======================

This patch introduces the typing of the results.  The why and how is described
in the documentation, please generate the documentation ..

    $ make docs.clean docs.live

and read the following articles in the "Developer documentation":

- result types --> http://0.0.0.0:8000/dev/result_types/index.html

The result types are available from the `searx.result_types` module.  The
following have been implemented so far:

- base result type: `searx.result_type.Result`
  --> http://0.0.0.0:8000/dev/result_types/base_result.html

- answer results
  --> http://0.0.0.0:8000/dev/result_types/answer.html

including the type for translations (inspired by #3925).  For all other
types (which still need to be set up in subsequent PRs), template documentation
has been created for the transition period.

Doc of the fields used in Templates
===================================

The template documentation is the basis for the typing and is the first complete
documentation of the results (needed for engine development).  It is the
"working paper" (the plan) with which further typifications can be implemented
in subsequent PRs.

- https://github.com/searxng/searxng/issues/357

Answer Templates
================

With the new (sub) types for `Answer`, the templates for the answers have also
been revised, `Translation` are now displayed with collapsible entries (inspired
by #3925).

    !en-de dog

Plugins & Answerer
==================

The implementation for `Plugin` and `Answer` has been revised, see
documentation:

- Plugin: http://0.0.0.0:8000/dev/plugins/index.html
- Answerer: http://0.0.0.0:8000/dev/answerers/index.html

With `AnswerStorage` and `AnswerStorage` to manage those items (in follow up
PRs, `ArticleStorage`, `InfoStorage` and .. will be implemented)

Autocomplete
============

The autocompletion had a bug where the results from `Answer` had not been shown
in the past.  To test activate autocompletion and try search terms for which we
have answerers

- statistics: type `min 1 2 3` .. in the completion list you should find an
  entry like `[de] min(1, 2, 3) = 1`

- random: type `random uuid` .. in the completion list, the first item is a
  random UUID

Extended Types
==============

SearXNG extends e.g. the request and response types of flask and httpx, a module
has been set up for type extensions:

- Extended Types
  --> http://0.0.0.0:8000/dev/extended_types.html

Unit-Tests
==========

The unit tests have been completely revised.  In the previous implementation,
the runtime (the global variables such as `searx.settings`) was not initialized
before each test, so the runtime environment with which a test ran was always
determined by the tests that ran before it.  This was also the reason why we
sometimes had to observe non-deterministic errors in the tests in the past:

- https://github.com/searxng/searxng/issues/2988 is one example for the Runtime
  issues, with non-deterministic behavior ..

- https://github.com/searxng/searxng/pull/3650
- https://github.com/searxng/searxng/pull/3654
- https://github.com/searxng/searxng/pull/3642#issuecomment-2226884469
- https://github.com/searxng/searxng/pull/3746#issuecomment-2300965005

Why msgspec.Struct
==================

We have already discussed typing based on e.g. `TypeDict` or `dataclass` in the past:

- https://github.com/searxng/searxng/pull/1562/files
- https://gist.github.com/dalf/972eb05e7a9bee161487132a7de244d2
- https://github.com/searxng/searxng/pull/1412/files
- https://github.com/searxng/searxng/pull/1356

In my opinion, TypeDict is unsuitable because the objects are still dictionaries
and not instances of classes / the `dataclass` are classes but ...

The `msgspec.Struct` combine the advantages of typing, runtime behaviour and
also offer the option of (fast) serializing (incl. type check) the objects.

Currently not possible but conceivable with `msgspec`: Outsourcing the engines
into separate processes, what possibilities this opens up in the future is left
to the imagination!

Internally, we have already defined that it is desirable to decouple the
development of the engines from the development of the SearXNG core / The
serialization of the `Result` objects is a prerequisite for this.

HINT: The threads listed above were the template for this PR, even though the
implementation here is based on msgspec.  They should also be an inspiration for
the following PRs of typification, as the models and implementations can provide
a good direction.

Why just one commit?
====================

I tried to create several (thematically separated) commits, but gave up at some
point ... there are too many things to tackle at once / The comprehensibility of
the commits would not be improved by a thematic separation. On the contrary, we
would have to make multiple changes at the same places and the goal of a change
would be vaguely recognizable in the fog of the commits.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

7 files changed, 256 insertions, 0 deletions

diff --git a/docs/dev/result_types/answer.rst b/docs/dev/result_types/answer.rst
new file mode 100644
index 000000000..72368be39
--- /dev/null
+++ b/docs/dev/result_types/answer.rst
@@ -0,0 +1,7 @@
+.. _result_types.answer:
+
+==============
+Answer Results
+==============
+
+.. automodule:: searx.result_types.answer
diff --git a/docs/dev/result_types/base_result.rst b/docs/dev/result_types/base_result.rst
new file mode 100644
index 000000000..24fd96a9d
--- /dev/null
+++ b/docs/dev/result_types/base_result.rst
@@ -0,0 +1,5 @@
+======
+Result
+======
+
+.. automodule:: searx.result_types._base
diff --git a/docs/dev/result_types/correction.rst b/docs/dev/result_types/correction.rst
new file mode 100644
index 000000000..51cdd45d2
--- /dev/null
+++ b/docs/dev/result_types/correction.rst
@@ -0,0 +1,34 @@
+.. _result_types.corrections:
+
+==================
+Correction Results
+==================
+
+.. hint::
+
+   There is still no typing for these result items. The templates can be used as
+   orientation until the final typing is complete.
+
+The corrections area shows the user alternative search terms.
+
+A result of this type is a very simple dictionary with only one key/value pair
+
+.. code:: python
+
+   {"correction" : "lorem ipsum .."}
+
+From this simple dict another dict is build up:
+
+.. code:: python
+
+   # use RawTextQuery to get the corrections URLs with the same bang
+   {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum .." }
+
+and used in the template :origin:`corrections.html
+<searx/templates/simple/elements/corrections.html>`:
+
+title : :py:class:`str`
+  Corrected search term.
+
+url : :py:class:`str`
+  Not really an URL, its the value to insert in a HTML form for a SearXNG query.
diff --git a/docs/dev/result_types/index.rst b/docs/dev/result_types/index.rst
new file mode 100644
index 000000000..e90dba849
--- /dev/null
+++ b/docs/dev/result_types/index.rst
@@ -0,0 +1,95 @@
+.. _result types:
+
+============
+Result Types
+============
+
+To understand the typification of the results, let's take a brief look at the
+structure of SearXNG ..  At its core, SearXNG is nothing more than an aggregator
+that aggregates the results from various sources, renders them via templates and
+displays them to the user.
+
+The **sources** can be:
+
+1. :ref:`engines <engine implementations>`
+2. :ref:`plugins <dev plugin>`
+3. :ref:`answerers <dev answerers>`
+
+The sources provide the results, which are displayed in different **areas**
+depending on the type of result. The areas are:
+
+main results:
+  It is the main area in which -- as is typical for search engines -- the
+  results that a search engine has found for the search term are displayed.
+
+answers:
+  This area displays short answers that could be found for the search term.
+
+info box:
+  An area in which additional information can be displayed, e.g. excerpts from
+  wikipedia or other sources such as maps.
+
+suggestions:
+  Suggestions for alternative search terms can be found in this area.  These can
+  be clicked on and a search is carried out with these search terms.
+
+corrections:
+  Results in this area are like the suggestion of alternative search terms,
+  which usually result from spelling corrections
+
+At this point it is important to note that all **sources** can contribute
+results to all of the areas mentioned above.
+
+In most cases, however, the :ref:`engines <engine implementations>` will fill
+the *main results* and the :ref:`answerers <dev answerers>` will generally
+provide the contributions for the *answer* area.  Not necessary to mention here
+but for a better understanding: the plugins can also filter out or change
+results from the main results area (e.g. the URL of the link).
+
+The result items are organized in the :py:obj:`results.ResultContainer` and
+after all sources have delivered their results, this container is passed to the
+templating to build a HTML output.  The output is usually HTML, but it is also
+possible to output the result lists as JSON or RSS feed. Thats quite all we need
+to know before we dive into typification of result items.
+
+.. hint::
+
+   Typification of result items: we are at the very first beginng!
+
+The first thing we have to realize is that there is no typification of the
+result items so far, we have to build it up first .. and that is quite a big
+task, which we will only be able to accomplish gradually.
+
+The foundation for the typeless results was laid back in 2013 in the very first
+commit :commit:`ae9fb1d7d`, and the principle has not changed since then.  At
+the time, the approach was perfectly adequate, but we have since evolved and the
+demands on SearXNG increase with every feature request.
+
+**Motivation:** in the meantime, it has become very difficult to develop new
+features that require structural changes and it is especially hard for newcomers
+to find their way in this typeless world.  As long as the results are only
+simple key/value dictionaries, it is not even possible for the IDEs to support
+the application developer in his work.
+
+**Planning:** The procedure for subsequent typing will have to be based on the
+circumstances ..
+
+.. attention::
+
+   As long as there is no type defined for a kind of result the HTML template
+   specify what the properties of a type are.
+
+   In this sense, you will either find a type definition here in the
+   documentation or, if this does not yet exist, a description of the HTML
+   template.
+
+
+.. toctree::
+   :maxdepth: 2
+
+   base_result
+   main_result
+   answer
+   correction
+   suggestion
+   infobox
diff --git a/docs/dev/result_types/infobox.rst b/docs/dev/result_types/infobox.rst
new file mode 100644
index 000000000..428dc8db7
--- /dev/null
+++ b/docs/dev/result_types/infobox.rst
@@ -0,0 +1,60 @@
+.. _result_types.infobox:
+
+===============
+Infobox Results
+===============
+
+.. hint::
+
+   There is still no typing for these result items. The templates can be used as
+   orientation until the final typing is complete.
+
+The infobox is an area where addtional infos shown to the user.
+
+Fields used in the :origin:`infobox.html
+<searx/templates/simple/elements/infobox.html>`:
+
+img_src: :py:class:`str`
+  URL of a image or thumbnail that is displayed in the infobox.
+
+infobox: :py:class:`str`
+  Title of the info box.
+
+content: :py:class:`str`
+  Text of the info box.
+
+The infobox has additional subsections for *attributes*, *urls* and
+*relatedTopics*:
+
+attributes: :py:class:`List <list>`\ [\ :py:class:`dict`\ ]
+  A list of attributes.  An *attribute* is a dictionary with keys:
+
+  - label :py:class:`str`: (mandatory)
+
+  - value :py:class:`str`: (mandatory)
+
+  - image :py:class:`List <list>`\ [\ :py:class:`dict`\ ] (optional)
+
+    A list of images.  An *image* is a dictionary with keys:
+
+    - src :py:class:`str`: URL of an image/thumbnail (mandatory)
+    - alt :py:class:`str`: alternative text for the image (mandatory)
+
+urls: :py:class:`List <list>`\ [\ :py:class:`dict`\ ]
+  A list of links.  An *link* is a dictionary with keys:
+
+  - url :py:class:`str`: URL of the link (mandatory)
+  - title :py:class:`str`: Title of the link (mandatory)
+
+relatedTopics: :py:class:`List <list>`\ [\ :py:class:`dict`\ ]
+  A list of topics.  An *topic* is a dictionary with keys:
+
+  - name: :py:class:`str`: (mandatory)
+
+  - suggestions: :py:class:`List <list>`\ [\ :py:class:`dict`\ ] (optional)
+
+    A list of suggestions.  A *suggestion* is simple dictionary with just one
+    key/value pair:
+
+    - suggestion: :py:class:`str`: suggested search term (mandatory)
+
diff --git a/docs/dev/result_types/main_result.rst b/docs/dev/result_types/main_result.rst
new file mode 100644
index 000000000..1f178cbd1
--- /dev/null
+++ b/docs/dev/result_types/main_result.rst
@@ -0,0 +1,17 @@
+============
+Main Results
+============
+
+There is still no typing for the items in the :ref:`main result list`.  The
+templates can be used as orientation until the final typing is complete.
+
+- :ref:`template default`
+- :ref:`template images`
+- :ref:`template videos`
+- :ref:`template torrent`
+- :ref:`template map`
+- :ref:`template paper`
+- :ref:`template packages`
+- :ref:`template code`
+- :ref:`template files`
+- :ref:`template products`
diff --git a/docs/dev/result_types/suggestion.rst b/docs/dev/result_types/suggestion.rst
new file mode 100644
index 000000000..52e8a05a8
--- /dev/null
+++ b/docs/dev/result_types/suggestion.rst
@@ -0,0 +1,38 @@
+.. _result_types.suggestion:
+
+==================
+Suggestion Results
+==================
+
+.. hint::
+
+   There is still no typing for these result items. The templates can be used as
+   orientation until the final typing is complete.
+
+The suggestions area shows the user alternative search terms.
+
+A result of this type is a very simple dictionary with only one key/value pair
+
+.. code:: python
+
+   {"suggestion" : "lorem ipsum .."}
+
+From this simple dict another dict is build up:
+
+.. code:: python
+
+   {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" }
+
+and used in the template :origin:`suggestions.html
+<searx/templates/simple/elements/suggestions.html>`:
+
+.. code:: python
+
+   # use RawTextQuery to get the suggestion URLs with the same bang
+   {"url" : "!bang lorem ipsum ..", "title": "lorem ipsum" }
+
+title : :py:class:`str`
+  Suggested search term
+
+url : :py:class:`str`
+  Not really an URL, its the value to insert in a HTML form for a SearXNG query.