[Fix #4268] Adding Documentation for upgraded Search #4467

safwanrahman · 2018-08-03T11:36:49Z

I have added instruction for preparing the local development for search. Its first step towards documentation of search as proposed in #4268.
@ericholscher r?

ericholscher

A good start. Needs some spell checking, and I think I touched on the grammar issues I found.

ericholscher · 2018-08-06T17:13:56Z

docs/development/search.rst

+Search
+============
+
+Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search result. Documentations are indexed in Elasticsearch index and the search is made through API. All the Search Code is Opensource and lives in `Github Repository`_. Currently we are using `Elasticsearch 6.3`_ version.


result -> results

Documentations -> Documents

ericholscher · 2018-08-06T17:14:46Z

docs/development/search.rst

+
+Installing and running Elasticsearch
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+You must need to install and run `Elasticsearch 6.3`_ version in your local development machine. You can get installation instuction `here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_.


instuction is misspelled.

ericholscher · 2018-08-06T17:15:07Z

docs/development/search.rst

+^^^^^^^^
+For using search, you need to index data to Elasticsearch Index. Run `reindex_elasticsearch` management command::
+
+    ./manage.py reindex_elasticsearch


Doesn't this take arguments?

./manage.py search_index which is provided by DED takes argument. our implementation of management command do not take arguments.

safwanrahman · 2018-08-07T19:46:39Z

@ericholscher fixed the grammer and added some docs about architecture. Is it ready to merge?

RichardLitt · 2018-08-07T21:12:37Z

docs/development/search.rst

+Search
+============
+
+Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search


...instead of the built-in Sphinx search...

RichardLitt · 2018-08-07T21:13:03Z

docs/development/search.rst

+============
+
+Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search
+results. Documents are indexed in Elasticsearch index and the search is made through API.


...in the Eleasticearch index...through the API.

RichardLitt · 2018-08-07T21:13:28Z

docs/development/search.rst

+
+Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search
+results. Documents are indexed in Elasticsearch index and the search is made through API.
+All the Search Code is Opensource and lives in `Github Repository`_.


...the GitHub Repository.

And "open source", not Opensource

RichardLitt · 2018-08-07T21:14:07Z

docs/development/search.rst

+Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search
+results. Documents are indexed in Elasticsearch index and the search is made through API.
+All the Search Code is Opensource and lives in `Github Repository`_.
+Currently we are using `Elasticsearch 6.3`_ version.


using the Elasticsearch 6.3 version.

RichardLitt · 2018-08-07T21:15:08Z

docs/development/search.rst

+
+Installing and running Elasticsearch
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+You must need to install and run `Elasticsearch 6.3`_ version in your local development machine.


You need to install and run the Elasticsearch 6.3 version on your local development machine. You can get the installation instructions here.

RichardLitt · 2018-08-07T21:15:17Z

docs/development/search.rst

+You must need to install and run `Elasticsearch 6.3`_ version in your local development machine.
+You can get installation instruction
+`here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_.
+Otherwise, you can also start a Elasticsearch Docker container by running following command::


running the following command

RichardLitt · 2018-08-07T21:15:34Z

docs/development/search.rst

+
+Auto Indexing
+^^^^^^^^^^^^^
+By default, Auto Indexing is turned off in development mode. To turn in on, change the


To turn it on

RichardLitt · 2018-08-07T21:15:42Z

docs/development/search.rst

+^^^^^^^^^^^^^
+By default, Auto Indexing is turned off in development mode. To turn in on, change the
+`ELASTICSEARCH_DSL_AUTOSYNC` settings to `True` in `readthedocs/settings/dev.py` file.
+After that, whenever a documentation successfully build, or project gets added,


RichardLitt · 2018-08-07T21:37:21Z

docs/development/search.rst

+------------
+The search architecture is devided into 2 parts.
+One part is responsible for **indexing** the documents and projects and
+other part is responsible to query through the Index for showing proper result to users.


RichardLitt · 2018-08-07T21:37:26Z

docs/development/search.rst

+The search architecture is devided into 2 parts.
+One part is responsible for **indexing** the documents and projects and
+other part is responsible to query through the Index for showing proper result to users.
+We use `django-elasticsearch-dsl`_ package mostly for the keep the search working.


to keep the

RichardLitt · 2018-08-07T21:37:39Z

docs/development/search.rst

+
+Indexing
+^^^^^^^^
+All the Sphinx documents are indexed into elasticsearch after build gets successfully finish.


the build is successful.

RichardLitt · 2018-08-07T21:38:08Z

docs/development/search.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After any build gets successfully finished, `HTMLFile` objects are created for each of the
+`HTML` file of the build version and delete the old version's `HTMLFile` object. Signal_


files and the old version's HTMLFile object is deleted.

RichardLitt · 2018-08-07T21:38:26Z

docs/development/search.rst

+files. Both of the signals are dispatched with the list of the instances of `HTMLFile`
+in `instance_list` parameter.
+
+We listen the `bulk_post_create` and `bulk_post_delete` signals in our `Search` application and


listen to the

RichardLitt · 2018-08-07T21:38:44Z

docs/development/search.rst

+
+How we index projects
+~~~~~~~~~~~~~~~~~~~~~
+We also index project informations in our search index so that user can search for projects


information ... so that they user can...

RichardLitt · 2018-08-07T21:38:56Z

docs/development/search.rst

+~~~~~~~~~~~~~~~~~~~~~~
+
+`elasticsearch-dsl`_ provide model like wrapper for `Elasticsearch document`_.
+As per requirements of `django-elasticsearch-dsl`_, its stored in


it's stored

RichardLitt · 2018-08-07T21:39:08Z

docs/development/search.rst

+Elasticsearch Document
+~~~~~~~~~~~~~~~~~~~~~~
+
+`elasticsearch-dsl`_ provide model like wrapper for `Elasticsearch document`_.


provides a model-like wrapper for the

RichardLitt · 2018-08-07T21:39:14Z

docs/development/search.rst

+As per requirements of `django-elasticsearch-dsl`_, its stored in
+`readthedocs/search/documents.py` file.
+
+    **ProjectDocument:** Its used for indexing projects. Signal listener of


RichardLitt · 2018-08-07T21:39:22Z

docs/development/search.rst

+    `django-elasticsearch-dsl`_ listen the `post_save` singal of `Project` model and
+    then index/delete into Elasticsearch.
+
+    **PageDocument**: Its used for indexing documentation of projects. By default, the auto


RichardLitt · 2018-08-07T21:42:21Z

docs/development/search.rst

+    **PageDocument**: Its used for indexing documentation of projects. By default, the auto
+    indexing is turned off by `ignore_signals = settings.ES_PAGE_IGNORE_SIGNALS`.
+    `settings.ES_PAGE_IGNORE_SIGNALS` is `False` both in development and production.
+    As mentioned above, our `Search` app listens the `bulk_post_create` and `bulk_post_delete`


listens to the

RichardLitt · 2018-08-07T21:42:45Z

docs/development/search.rst

+    As mentioned above, our `Search` app listens the `bulk_post_create` and `bulk_post_delete`
+    signals and index/delete documentations into Elasticsearch. The signal listeners are in
+    the `readthedocs/search/signals.py` file. Both of the signals are dispatched
+    after successful documentation build.


after a successful

safwanrahman · 2018-08-08T18:43:58Z

@RichardLitt I have fixed all the issue you have mentioned. can you check again?

RichardLitt · 2018-08-08T20:30:09Z

docs/development/search.rst

+
+Installing and running Elasticsearch
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+You need to install and run the `Elasticsearch 6.3`_ version on your local development machine.


...run Elastichsearch version 6.3_ on your...

This might be a bit more fluent.

RichardLitt · 2018-08-08T20:30:19Z

docs/development/search.rst

+You need to install and run the `Elasticsearch 6.3`_ version on your local development machine.
+You can get the installation instructions
+`here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_.
+Otherwise, you can also start a Elasticsearch Docker container by running the following command::


start an Elasticsearch

RichardLitt · 2018-08-08T20:30:37Z

docs/development/search.rst

+
+Indexing into Elasticsearch
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For using search, you need to index data to Elasticsearch Index. Run `reindex_elasticsearch`


data to the Elasticsearch

RichardLitt · 2018-08-08T20:30:58Z

docs/development/search.rst

+Auto Indexing
+^^^^^^^^^^^^^
+By default, Auto Indexing is turned off in development mode. To turn it on, change the
+`ELASTICSEARCH_DSL_AUTOSYNC` settings to `True` in `readthedocs/settings/dev.py` file.


in the readthedocs... file

RichardLitt · 2018-08-08T20:31:36Z

docs/development/search.rst

+------------
+The search architecture is devided into 2 parts.
+One part is responsible for **indexing** the documents and projects and
+other part is responsible to query through the Index for showing proper results to users.


the other part is responsible for querying the Index to show the proper results to users.

RichardLitt · 2018-08-08T20:31:50Z

docs/development/search.rst

+The search architecture is devided into 2 parts.
+One part is responsible for **indexing** the documents and projects and
+other part is responsible to query through the Index for showing proper results to users.
+We use `django-elasticsearch-dsl`_ package mostly to the keep the search working.


RichardLitt · 2018-08-08T20:32:04Z

docs/development/search.rst

+
+Indexing
+^^^^^^^^
+All the Sphinx documents are indexed into elasticsearch after the build is successful.


Elasticsearch should always be capitalized.

RichardLitt · 2018-08-08T20:32:22Z

docs/development/search.rst

+How we index documentations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After any build gets successfully finished, `HTMLFile` objects are created for each of the


any build is successfully finished

RichardLitt · 2018-08-08T20:32:36Z

docs/development/search.rst

+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After any build gets successfully finished, `HTMLFile` objects are created for each of the
+`HTML` files and the old version's `HTMLFile` object is deleted. Signal_


The Signal_

RichardLitt · 2018-08-08T20:33:24Z

docs/development/search.rst

+~~~~~~~~~~~~~~~~~~~~~
+We also index project information in our search index so that the user can search for projects
+from the main site. `django-elasticsearch-dsl`_ listen `post_create` and `post_delete` signals of
+`Project` model and index/delte into Elasticsearch accordingly.


RichardLitt · 2018-08-08T20:33:35Z

docs/development/search.rst

+Elasticsearch Document
+~~~~~~~~~~~~~~~~~~~~~~
+
+`elasticsearch-dsl`_ provides model like wrapper for the `Elasticsearch document`_.


provides a model-like wrapper

RichardLitt · 2018-08-08T20:33:46Z

docs/development/search.rst

+
+`elasticsearch-dsl`_ provides model like wrapper for the `Elasticsearch document`_.
+As per requirements of `django-elasticsearch-dsl`_, it is stored in
+`readthedocs/search/documents.py` file.


stored in a

I think it should be the

RichardLitt · 2018-08-08T20:34:03Z

docs/development/search.rst

+`readthedocs/search/documents.py` file.
+
+    **ProjectDocument:** It is used for indexing projects. Signal listener of
+    `django-elasticsearch-dsl`_ listen the `post_save` singal of `Project` model and


listens to the

RichardLitt · 2018-08-08T20:34:33Z

docs/development/search.rst

+    indexing is turned off by `ignore_signals = settings.ES_PAGE_IGNORE_SIGNALS`.
+    `settings.ES_PAGE_IGNORE_SIGNALS` is `False` both in development and production.
+    As mentioned above, our `Search` app listens to the `bulk_post_create` and `bulk_post_delete`
+    signals and index/delete documentations into Elasticsearch. The signal listeners are in


indexes/deleted documentation into...

RichardLitt

Getting there! Sorry about all of these comments.

safwanrahman · 2018-08-10T09:00:04Z

@RichardLitt I have updated the issues you have mentioned.

ericholscher

This looks great. We could definitely add a bit more detail in some places, but I think this is a solid start to work from. 👍

ericholscher · 2018-08-10T09:27:20Z

docs/development/search.rst

+For using search, you need to index data to the Elasticsearch Index. Run `reindex_elasticsearch`
+management command::
+
+    ./manage.py reindex_elasticsearch


It's probably worth noting here why we implemented our own version, instead of using the detail one provided by django-elasticsearch-dsl.

ericholscher · 2018-08-10T09:28:04Z

docs/development/search.rst

+in `instance_list` parameter.
+
+We listen to the `bulk_post_create` and `bulk_post_delete` signals in our `Search` application and
+index/delete the documentation content from the `HTMLFile` instances.


Again, it's probably worth mentioning why we designed it this way, because by default it was doing an HTTP request per object.

ericholscher · 2018-08-10T09:28:35Z

docs/development/search.rst

+    the `readthedocs/search/signals.py` file. Both of the signals are dispatched
+    after a successful documentation build.
+
+


It's probably useful to mention how we parse the data from the JSON files to get the actual indexed data.

RichardLitt · 2018-08-10T14:34:26Z

@safwanrahman Thanks. :) Sorry for being so persnickety!

safwanrahman · 2018-08-10T23:08:12Z

@ericholscher updated with fixes. Possible to merge?

[Fix readthedocs#4268] Adding Documentation for upgraded Search

Adding Documentation for Search

879b59c

safwanrahman requested a review from ericholscher August 3, 2018 11:36

safwanrahman added 2 commits August 3, 2018 17:37

fixup

5a3d9c8

remove unneeded management command

bbf0973

ericholscher reviewed Aug 6, 2018

View reviewed changes

safwanrahman added 2 commits August 7, 2018 23:54

fixup as per review

e51d580

adding docs for architecture

c752e44

safwanrahman force-pushed the search_docs branch from c25d9db to c752e44 Compare August 7, 2018 19:35

safwanrahman changed the title ~~Adding Documentation for upgraded Search~~ [Fix #4268] Adding Documentation for upgraded Search Aug 7, 2018

RichardLitt reviewed Aug 7, 2018

View reviewed changes

fixup

568d8c6

RichardLitt reviewed Aug 8, 2018

View reviewed changes

RichardLitt suggested changes Aug 8, 2018

View reviewed changes

more fixup

e923884

ericholscher approved these changes Aug 10, 2018

View reviewed changes

fixup

2a726db

agjohnson added the PR: work in progress Pull request is not ready for full review label Aug 27, 2018

agjohnson added this to the Search improvements milestone Aug 27, 2018

ericholscher merged commit 88ff413 into readthedocs:search_upgrade Sep 7, 2018

safwanrahman pushed a commit to safwanrahman/readthedocs.org that referenced this pull request Sep 15, 2018

Merge pull request readthedocs#4467 from safwanrahman/search_docs

36bb8cd

[Fix readthedocs#4268] Adding Documentation for upgraded Search

		the `readthedocs/search/signals.py` file. Both of the signals are dispatched
		after a successful documentation build.

[Fix #4268] Adding Documentation for upgraded Search #4467

[Fix #4268] Adding Documentation for upgraded Search #4467

Conversation

safwanrahman commented Aug 3, 2018

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

safwanrahman commented Aug 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

safwanrahman commented Aug 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RichardLitt left a comment

Choose a reason for hiding this comment

safwanrahman commented Aug 10, 2018

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RichardLitt commented Aug 10, 2018

safwanrahman commented Aug 10, 2018