Unverified Commit 55aa7f96 authored by Markus Heiser's avatar Markus Heiser Committed by GitHub

Merge branch 'master' into docker/opencontainers

parents 24472ce7 8c260977
......@@ -100,3 +100,26 @@ generally made searx better:
- @ZEROF
- Ivan Skytte Jørgensen @isj-privacore
- @miicha
- Étienne Deparis @milouse
- @pelag0s
- Denis Wernert @d-tux
- Robin Hallabro-Kokko @hallabro
- Jonas Zohren @jfowl
- Elias Ojala @theel0ja
- @brunob
- Nick Espig @nachtalb
- Rachmadani Haryono @rachmadaniHaryono
- Frank de Lange @yetangitu
- Émilien Devos @unifox
- Nicolas Gelot @nfk
- @volth
- Mathieu Brunot @madmath03
- @lorddavidiii
- @x250
- Robby O'Connor @robbyoconnor
- Finn @0xhtml
- @tmikaeld
- @hobbestigrou
- Vipul @finn0
- @CaffeinatedTech
- Robin Schneider @ypid
0.16.0 2020.01.30
=================
- New engines
- Splash
- Apkmirror
- NPM search
- Invidious
- Seedpeer
- New languages
- Estonian
- Interlingua
- Lithuanian
- Tibetian
- Occitan
- Tamil
- Engine fixes (wolframalpha, google scholar, youtube, google images, seznam, google, soundcloud, google cloud, duden, btdigg, google play, bing images, flickr noapi, wikidata, dailymotion, google videos, arxiv, dictzone, fdroid, etymonline, bing, duckduckgo, startpage, voat, 1x, deviantart, digg, gigablast, mojeek, duckduckgo definitions, spotify, libgen, qwant, openstreetmap, wikipedia, ina, microsoft academic, scanr structures)
- Dependency updates
- Speed optimizations
- Initial support for offline engines
- Image format display
- Inline js scripts removed
- Infinite scroll plugin fix
- Simple theme bugfixes
- Docker image updates
- Bang expression fixes
- Result merging fixes
- New environment variable added: SEARX_BIND_ADDRESS
News
~~~~
- @return42 joined the maintainer team
- This is the last release with Python2 support
0.15.0 2019.01.06
=================
......
......@@ -7,7 +7,7 @@ engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__.
Pronunciation: səːks
List of `running
instances <https://github.com/asciimoo/searx/wiki/Searx-instances>`__.
instances <https://searx.space/>`__.
See the `documentation <https://asciimoo.github.io/searx>`__ and the `wiki <https://github.com/asciimoo/searx/wiki>`__ for more information.
......@@ -18,17 +18,17 @@ Installation
~~~~~~~~~~~~
With Docker
------
-----------
Go to the `searx-docker <https://github.com/searx/searx-docker>`__ project.
Without Docker
------
--------------
For all of the details, follow this `step by step installation <https://asciimoo.github.io/searx/dev/install/installation.html>`__.
Note: the documentation needs to be updated.
If you are in a hurry
------
---------------------
- clone the source:
``git clone https://github.com/asciimoo/searx.git && cd searx``
- install dependencies: ``./manage.sh update_packages``
......
......@@ -31,3 +31,7 @@ touch-logrotate = /run/uwsgi-logrotate
unique-cron = 15 0 -1 -1 -1 { touch /run/uwsgi-logrotate }
log-backupname = /var/log/uwsgi/uwsgi.log.1
logto = /var/log/uwsgi/uwsgi.log
# No keep alive
# See https://github.com/searx/searx-docker/issues/24
add-header = Connection: close
......@@ -11,7 +11,7 @@ DOCS_URL = os.environ.get("DOCS_URL", "https://asciimoo.github.io/searx/")
# Project --------------------------------------------------------------
project = u'searx'
copyright = u'2015-2019, Adam Tauber, Noémi Ványi'
copyright = u'2015-2020, Adam Tauber, Noémi Ványi'
author = u'Adam Tauber'
release, version = VERSION_STRING, VERSION_STRING
highlight_language = 'none'
......@@ -94,7 +94,7 @@ html_context = {
"project_links": [
ProjectLink("Source", GIT_URL),
ProjectLink("Wiki", "https://github.com/asciimoo/searx/wiki"),
ProjectLink("Public instances", "https://asciimoo.github.io/searx/user/public_instances.html"),
ProjectLink("Public instances", "https://searx.space/"),
ProjectLink("Twitter", "https://twitter.com/Searx_engine"),
]
}
......
......@@ -285,7 +285,7 @@ content becomes smart.
:rst:role:`pep` :pep:`8` ``:pep:`8```
sphinx.ext.extlinks_
--------------------------------------------------------------------------------------------------
project's wiki article :wiki:`Searx-instances` ``:wiki:`Searx-instances```
project's wiki article :wiki:`Offline-engines` ``:wiki:`Offline-engines```
to docs public URL :docs:`dev/reST.html` ``:docs:`dev/reST.html```
files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst```
pull request :pull:`1756` ``:pull:`1756```
......
......@@ -20,8 +20,8 @@ Searx is a free internet metasearch engine which aggregates results from more
than 70 search services. Users are neither tracked nor profiled. Additionally,
searx can be used over Tor for online anonymity.
Get started with searx by using one of the :wiki:`Searx-instances`. If you
don't trust anyone, you can set up your own, see :ref:`installation`.
Get started with searx by using one of the Searx-instances_. If you don't trust
anyone, you can set up your own, see :ref:`installation`.
.. toctree::
:maxdepth: 2
......@@ -30,3 +30,5 @@ don't trust anyone, you can set up your own, see :ref:`installation`.
admin/index
dev/index
blog/index
.. _Searx-instances: https://searx.space
......@@ -5,6 +5,5 @@ User documentation
.. toctree::
:maxdepth: 1
public_instances
search_syntax
own-instance
This diff is collapsed.
certifi==2019.3.9
babel==2.7.0
flask-babel==0.12.2
flask-babel==1.0.0
flask==1.0.2
idna==2.8
jinja2==2.10.1
......
......@@ -38,6 +38,7 @@ def check_settings_yml(file_name):
else:
return None
# find location of settings.yml
if 'SEARX_SETTINGS_PATH' in environ:
# if possible set path to settings using the
......
......@@ -54,7 +54,8 @@ engine_default_args = {'paging': False,
'suspend_end_time': 0,
'continuous_errors': 0,
'time_range_support': False,
'offline': False}
'offline': False,
'tokens': []}
def load_engine(engine_data):
......@@ -160,7 +161,7 @@ def to_percentage(stats, maxvalue):
return stats
def get_engines_stats():
def get_engines_stats(preferences):
# TODO refactor
pageloads = []
engine_times = []
......@@ -171,8 +172,12 @@ def get_engines_stats():
max_pageload = max_engine_times = max_results = max_score = max_errors = max_score_per_result = 0 # noqa
for engine in engines.values():
if not preferences.validate_token(engine):
continue
if engine.stats['search_count'] == 0:
continue
results_num = \
engine.stats['result_count'] / float(engine.stats['search_count'])
......
"""
Dummy Offline
@results one result
@stable yes
"""
def search(query, request_params):
return [{
'result': 'this is what you get',
}]
"""
eTools (Web)
@website https://www.etools.ch
@provide-api no
@using-api no
@results HTML
@stable no (HTML can change)
@parse url, title, content
"""
from lxml import html
from searx.engines.xpath import extract_text
from searx.url_utils import quote
from searx.utils import eval_xpath
categories = ['general']
paging = False
language_support = False
safesearch = True
base_url = 'https://www.etools.ch'
search_path = '/searchAdvancedSubmit.do'\
'?query={search_term}'\
'&pageResults=20'\
'&safeSearch={safesearch}'
def request(query, params):
if params['safesearch']:
safesearch = 'true'
else:
safesearch = 'false'
params['url'] = base_url + search_path.format(search_term=quote(query), safesearch=safesearch)
return params
def response(resp):
results = []
dom = html.fromstring(resp.text)
for result in eval_xpath(dom, '//table[@class="result"]//td[@class="record"]'):
url = eval_xpath(result, './a/@href')[0]
title = extract_text(eval_xpath(result, './a//text()'))
content = extract_text(eval_xpath(result, './/div[@class="text"]//text()'))
results.append({'url': url,
'title': title,
'content': content})
return results
......@@ -72,6 +72,7 @@ def parse_album(hit):
result.update({'content': 'Released: {}'.format(year)})
return result
parse = {'lyric': parse_lyric, 'song': parse_lyric, 'artist': parse_artist, 'album': parse_album}
......
......@@ -4,6 +4,7 @@ from sys import version
from searx import settings, autocomplete
from searx.languages import language_codes as languages
from searx.utils import match_language
from searx.url_utils import parse_qs, urlencode
if version[0] == '3':
......@@ -104,9 +105,38 @@ class MultipleChoiceSetting(EnumStringSetting):
resp.set_cookie(name, ','.join(self.value), max_age=COOKIE_MAX_AGE)
class SetSetting(Setting):
def _post_init(self):
if not hasattr(self, 'values'):
self.values = set()
def get_value(self):
return ','.join(self.values)
def parse(self, data):
if data == '':
self.values = set()
return
elements = data.split(',')
for element in elements:
self.values.add(element)
def parse_form(self, data):
elements = data.split(',')
self.values = set(elements)
def save(self, name, resp):
resp.set_cookie(name, ','.join(self.values), max_age=COOKIE_MAX_AGE)
class SearchLanguageSetting(EnumStringSetting):
"""Available choices may change, so user's value may not be in choices anymore"""
def _validate_selection(self, selection):
if not match_language(selection, self.choices, fallback=None) and selection != "":
raise ValidationException('Invalid language code: "{0}"'.format(selection))
def parse(self, data):
if data not in self.choices and data != self.value:
# hack to give some backwards compatibility with old language cookies
......@@ -243,8 +273,8 @@ class Preferences(object):
super(Preferences, self).__init__()
self.key_value_settings = {'categories': MultipleChoiceSetting(['general'], choices=categories + ['none']),
'language': SearchLanguageSetting(settings['search']['language'],
choices=LANGUAGE_CODES),
'language': SearchLanguageSetting(settings['search']['default_lang'],
choices=list(LANGUAGE_CODES) + ['']),
'locale': EnumStringSetting(settings['ui']['default_locale'],
choices=list(settings['locales'].keys()) + ['']),
'autocomplete': EnumStringSetting(settings['search']['autocomplete'],
......@@ -272,6 +302,7 @@ class Preferences(object):
self.engines = EnginesSetting('engines', choices=engines)
self.plugins = PluginsSetting('plugins', choices=plugins)
self.tokens = SetSetting('tokens')
self.unknown_params = {}
def get_as_url_params(self):
......@@ -288,6 +319,8 @@ class Preferences(object):
settings_kv['disabled_plugins'] = ','.join(self.plugins.disabled)
settings_kv['enabled_plugins'] = ','.join(self.plugins.enabled)
settings_kv['tokens'] = ','.join(self.tokens.values)
return urlsafe_b64encode(compress(urlencode(settings_kv).encode('utf-8'))).decode('utf-8')
def parse_encoded_data(self, input_data):
......@@ -307,6 +340,8 @@ class Preferences(object):
elif user_setting_name == 'disabled_plugins':
self.plugins.parse_cookie((input_data.get('disabled_plugins', ''),
input_data.get('enabled_plugins', '')))
elif user_setting_name == 'tokens':
self.tokens.parse(user_setting)
elif not any(user_setting_name.startswith(x) for x in [
'enabled_',
'disabled_',
......@@ -328,6 +363,8 @@ class Preferences(object):
enabled_categories.append(user_setting_name[len('category_'):])
elif user_setting_name.startswith('plugin_'):
disabled_plugins.append(user_setting_name)
elif user_setting_name == 'tokens':
self.tokens.parse_form(user_setting)
else:
self.unknown_params[user_setting_name] = user_setting
self.key_value_settings['categories'].parse_form(enabled_categories)
......@@ -346,6 +383,18 @@ class Preferences(object):
user_setting.save(user_setting_name, resp)
self.engines.save(resp)
self.plugins.save(resp)
self.tokens.save('tokens', resp)
for k, v in self.unknown_params.items():
resp.set_cookie(k, v, max_age=COOKIE_MAX_AGE)
return resp
def validate_token(self, engine):
valid = True
if hasattr(engine, 'tokens') and engine.tokens:
valid = False
for token in self.tokens.values:
if token in engine.tokens:
valid = True
break
return valid
......@@ -177,7 +177,8 @@ class RawTextQuery(object):
class SearchQuery(object):
"""container for all the search parameters (query, language, etc...)"""
def __init__(self, query, engines, categories, lang, safesearch, pageno, time_range, timeout_limit=None):
def __init__(self, query, engines, categories, lang, safesearch, pageno, time_range,
timeout_limit=None, preferences=None):
self.query = query.encode('utf-8')
self.engines = engines
self.categories = categories
......@@ -186,6 +187,7 @@ class SearchQuery(object):
self.pageno = pageno
self.time_range = None if time_range in ('', 'None', None) else time_range
self.timeout_limit = timeout_limit
self.preferences = preferences
def __str__(self):
return str(self.query) + ";" + str(self.engines)
......@@ -407,7 +407,7 @@ def get_search_query_from_webapp(preferences, form):
return (SearchQuery(query, query_engines, query_categories,
query_lang, query_safesearch, query_pageno,
query_time_range, query_timeout),
query_time_range, query_timeout, preferences),
raw_text_query)
......@@ -459,6 +459,9 @@ class Search(object):
engine = engines[selected_engine['name']]
if not search_query.preferences.validate_token(engine):
continue
# skip suspended engines
if engine.suspend_end_time >= time():
logger.debug('Engine currently suspended: %s', selected_engine['name'])
......
......@@ -5,7 +5,7 @@ general:
search:
safe_search : 0 # Filter results. 0: None, 1: Moderate, 2: Strict
autocomplete : "" # Existing autocomplete backends: "dbpedia", "duckduckgo", "google", "startpage", "wikipedia" - leave blank to turn it off by default
language : "en-US"
default_lang : "" # Default search language - leave blank to detect from browser information or use codes from 'languages.py'
ban_time_on_fail : 5 # ban time in seconds after engine errors
max_ban_time_on_fail : 120 # max ban time in seconds after engine errors
......@@ -203,6 +203,11 @@ engines:
timeout: 3.0
disabled : True
- name : etools
engine : etools
shortcut : eto
disabled : True
- name : etymonline
engine : xpath
paging : True
......@@ -776,6 +781,7 @@ locales:
en : English
ar : العَرَبِيَّة (Arabic)
bg : Български (Bulgarian)
bo : བོད་སྐད་ (Tibetian)
ca : Català (Catalan)
cs : Čeština (Czech)
cy : Cymraeg (Welsh)
......@@ -784,6 +790,7 @@ locales:
el_GR : Ελληνικά (Greek_Greece)
eo : Esperanto (Esperanto)
es : Español (Spanish)
et : Eesti (Estonian)
eu : Euskara (Basque)
fa_IR : (fārsī) فارسى (Persian)
fi : Suomi (Finnish)
......@@ -796,8 +803,10 @@ locales:
ia : Interlingua (Interlingua)
it : Italiano (Italian)
ja : 日本語 (Japanese)
lt : Lietuvių (Lithuanian)
nl : Nederlands (Dutch)
nl_BE : Vlaams (Dutch_Belgium)
oc : Lenga D'òc (Occitan)
pl : Polski (Polish)
pt : Português (Portuguese)
pt_BR : Português (Portuguese_Brazil)
......@@ -808,6 +817,7 @@ locales:
sr : српски (Serbian)
sv : Svenska (Swedish)
te : తెలుగు (telugu)
ta : தமிழ் (Tamil)
tr : Türkçe (Turkish)
uk : українська мова (Ukrainian)
vi : tiếng việt (Vietnamese)
......
......@@ -5,12 +5,18 @@ $(document).ready(function() {
var formData = $('#pagination form:last').serialize();
if (formData) {
$('#pagination').html('<div class="loading-spinner"></div>');
$.post('./', formData, function (data) {
var body = $(data);
$('#pagination').remove();
$('#main_results').append('<hr/>');
$('#main_results').append(body.find('.result'));
$('#main_results').append(body.find('#pagination'));
$.ajax({
type: "POST",
url: './',
data: formData,
dataType: 'html',
success: function(data) {
var body = $(data);
$('#pagination').remove();
$('#main_results').append('<hr/>');
$('#main_results').append(body.find('.result'));
$('#main_results').append(body.find('#pagination'));
}
});
}
}
......
......@@ -22,7 +22,7 @@ Searx can be added to your browser's search bar; moreover, it can be set as the
<h2>How can I make it my own?</h2>
<p>Searx appreciates your concern regarding logs, so take the <a href="https://github.com/asciimoo/searx">code</a> and run it yourself! <br />Add your Searx to this <a href="https://github.com/asciimoo/searx/wiki/Searx-instances">list</a> to help other people reclaim their privacy and make the Internet freer!
<p>Searx appreciates your concern regarding logs, so take the <a href="https://github.com/asciimoo/searx">code</a> and run it yourself! <br />Add your Searx to this <a href="https://searx.space/">list</a> to help other people reclaim their privacy and make the Internet freer!
<br />The more decentralized the Internet is, the more freedom we have!</p>
......
......@@ -88,7 +88,7 @@
{{ _('Powered by') }} <a href="https://asciimoo.github.io/searx/">searx</a> - {{ searx_version }} - {{ _('a privacy-respecting, hackable metasearch engine') }}<br/>
<a href="https://github.com/asciimoo/searx">{{ _('Source code') }}</a> |
<a href="https://github.com/asciimoo/searx/issues">{{ _('Issue tracker') }}</a> |
<a href="https://github.com/asciimoo/searx/wiki/Searx-instances">{{ _('Public instances') }}</a>
<a href="https://searx.space/">{{ _('Public instances') }}</a>
</small>
</p>
</div>
......
......@@ -131,6 +131,12 @@
{% endfor %}
</select>
{{ preferences_item_footer(info, label, rtl) }}
{% set label = _('Engine tokens') %}
{% set info = _('Access tokens for private engines') %}
{{ preferences_item_header(info, label, rtl) }}
<input class="form-control" id='tokens' name='tokens' value='{{ preferences.tokens.get_value() }}'/>
{{ preferences_item_footer(info, label, rtl) }}
</div>
</fieldset>
</div>
......
......@@ -53,7 +53,7 @@
{{ _('Powered by') }} <a href="{{ url_for('about') }}">searx</a> - {{ searx_version }} - {{ _('a privacy-respecting, hackable metasearch engine') }}<br/>
<a href="https://github.com/asciimoo/searx">{{ _('Source code') }}</a> |
<a href="https://github.com/asciimoo/searx/issues">{{ _('Issue tracker') }}</a> |
<a href="https://github.com/asciimoo/searx/wiki/Searx-instances">{{ _('Public instances') }}</a>
<a href="https://searx.space/">{{ _('Public instances') }}</a>
</p>
</footer>
<!--[if gte IE 9]>-->
......
This diff is collapsed.
This diff is collapsed.
......@@ -3,14 +3,14 @@
# This file is distributed under the same license as the PROJECT project.
#
# Translators:
# Guimarães Mello <maeslor@cryptolab.net>, 2017
# Guimarães Mello <matheus.mello@disroot.org>, 2017,2019
msgid ""
msgstr ""
"Project-Id-Version: searx\n"
"Report-Msgid-Bugs-To: EMAIL@ADDRESS\n"
"POT-Creation-Date: 2017-10-07 19:18+0200\n"
"PO-Revision-Date: 2019-02-23 17:39+0000\n"
"Last-Translator: Guimarães Mello <maeslor@cryptolab.net>\n"
"PO-Revision-Date: 2019-09-30 14:07+0000\n"
"Last-Translator: Guimarães Mello <matheus.mello@disroot.org>\n"
"Language-Team: Interlingua (http://www.transifex.com/asciimoo/searx/language/ia/)\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
......@@ -21,15 +21,15 @@ msgstr ""
#: searx/search.py:137 searx/search.py:182
msgid "timeout"
msgstr ""
msgstr "tempore finite"
#: searx/search.py:144
msgid "request exception"
msgstr ""
msgstr "requesta un exception"
#: searx/search.py:151
msgid "unexpected crash"
msgstr ""
msgstr "crash impreviste"
#: searx/webapp.py:135
msgid "files"
......@@ -77,7 +77,7 @@ msgstr "Configurationes non valide, per favor, modifica tu preferentias."
#: searx/webapp.py:410
msgid "Invalid settings"
msgstr ""
msgstr "Configurationes invalide"
#: searx/webapp.py:444 searx/webapp.py:488
msgid "search error"
......@@ -625,11 +625,11 @@ msgstr "per proxy"
#: searx/templates/oscar/macros.html:92
msgid "supported"
msgstr ""
msgstr "supportate"
#: searx/templates/oscar/macros.html:96
msgid "not supported"
msgstr ""
msgstr "non supportate"
#: searx/templates/oscar/preferences.html:13
#: searx/templates/oscar/preferences.html:22
......@@ -714,7 +714,7 @@ msgstr "Via breve"
#: searx/templates/oscar/preferences.html:152
#: searx/templates/oscar/preferences.html:162
msgid "Selected language"
msgstr ""
msgstr "Lingua selectionate"
#: searx/templates/oscar/preferences.html:154
#: searx/templates/oscar/preferences.html:160
......@@ -774,13 +774,13 @@ msgstr "Valor"
#: searx/templates/oscar/preferences.html:289
msgid "Search URL of the currently saved preferences"
msgstr ""
msgstr "URL de Recerca del preferentias actualmente salvate"
#: searx/templates/oscar/preferences.html:289
msgid ""
"Note: specifying custom settings in the search URL can reduce privacy by "
"leaking data to the clicked result sites."
msgstr ""
msgstr "Nota: specificar configurationes personalisate in le URL de Recerca pote reducer le confidentialitate per lassar escappar datos al sitos cliccate in le resultatos."
#: searx/templates/oscar/results.html:17
msgid "Search results"
......@@ -789,12 +789,12 @@ msgstr "Resultatos de recerca"
#: searx/templates/oscar/results.html:21
#: searx/templates/simple/results.html:84
msgid "Try searching for:"
msgstr ""
msgstr "Essaya recercar pro:"
#: searx/templates/oscar/results.html:100
#: searx/templates/simple/results.html:25
msgid "Engines cannot retrieve results"
msgstr ""
msgstr "Le motores non poteva obtener resultatos"
#: searx/templates/oscar/results.html:131
msgid "Links"
......@@ -859,12 +859,12 @@ msgstr "Actualmente, il non ha datos disponibile."
#: searx/templates/oscar/messages/no_results.html:4
#: searx/templates/simple/messages/no_results.html:4
msgid "Engines cannot retrieve results."
msgstr ""
msgstr "Le motores non poteva obtener resultatos."
#: searx/templates/oscar/messages/no_results.html:10
#: searx/templates/simple/messages/no_results.html:10
msgid "Please, try again later or find another searx instance."
msgstr ""
msgstr "Per favor, essaya de novo plus tarde o trova un altere instantia de searx"
#: searx/templates/oscar/messages/no_results.html:14
#: searx/templates/simple/messages/no_results.html:14
......@@ -983,7 +983,7 @@ msgstr "Cargar plus..."
#: searx/templates/simple/base.html:31
msgid "No item found"
msgstr ""
msgstr "Nulle item trovate"
#: searx/templates/simple/preferences.html:89
msgid "Supports selected language"
......@@ -991,8 +991,8 @@ msgstr "Supporta le lingua selectionate"
#: searx/templates/simple/preferences.html:118
msgid "User interface"
msgstr ""
msgstr "Interfacie del usator"
#: searx/templates/simple/preferences.html:154
msgid "Privacy"
msgstr ""
msgstr "Confidentialitate"
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment