Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A lot of changes #2

Merged
merged 29 commits into from
Dec 14, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
990c6d3
ability to have a default ES connection
rochacbruno Nov 24, 2015
1d219ba
pep
rochacbruno Nov 24, 2015
9a77584
fix readme
rochacbruno Nov 24, 2015
b987ede
installation with extras depending on ES version
rochacbruno Nov 26, 2015
2e2bcc3
ID field handling
rochacbruno Nov 26, 2015
0e4f7b6
fix tests, better autoid handling, strict mode added
rochacbruno Nov 26, 2015
ce55931
tests for utils, 100% coverage again
rochacbruno Nov 27, 2015
58a8967
Naming convention not messing up with Python standards
rochacbruno Nov 27, 2015
c10ff8a
bump version, client interface needs a get method
rochacbruno Nov 27, 2015
0e3f545
pep8 and release to PyPI
rochacbruno Nov 27, 2015
1a7814a
added BooleanField, GeoField and GeoField
rochacbruno Nov 27, 2015
7799027
Fix casting mechanism
rochacbruno Dec 2, 2015
1877853
added Makefile and test.req, wrote tests for future
rochacbruno Dec 2, 2015
cb24bd6
get returns only one doc and filter returns a list
rochacbruno Dec 7, 2015
e1cb611
Mapping and docstings
rochacbruno Dec 8, 2015
6c22173
Dealing with ResultSet operations and refresh
rochacbruno Dec 9, 2015
5dff5d3
Fixed reload() method (tests is broken :sad:)
rochacbruno Dec 9, 2015
996cb2c
added tests configs and new tests
rochacbruno Dec 11, 2015
04b5e61
updated LICENSe to MIT
rochacbruno Dec 14, 2015
25bd811
changes make test script
rochacbruno Dec 14, 2015
bd1ddb2
updated README
rochacbruno Dec 14, 2015
11f6c45
Better example and docstrings
rochacbruno Dec 14, 2015
98903ef
adjust tests
rochacbruno Dec 14, 2015
eeb29a7
added badges
rochacbruno Dec 14, 2015
ec82d0f
octorized
rochacbruno Dec 14, 2015
c8e6f78
Update README.md
rochacbruno Dec 14, 2015
667fd00
released public to PyPI (pip install esengine)
rochacbruno Dec 14, 2015
e1d867a
added coveralls
rochacbruno Dec 14, 2015
fa38153
added noqa
rochacbruno Dec 14, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[run]
source = esengine

[report]
omit =
*/python?.?/*
*/site-packages/nose/*
1 change: 1 addition & 0 deletions .coveralls.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
repo_token: DzZ30nm43hTFokYZPwlYbIBLGju5D0QI4
22 changes: 22 additions & 0 deletions .landscape.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
pylint:
disable:
- bare-except
- unused-argument
- pointless-string-statement
- too-many-locals
- too-many-arguments
- protected-access
- unused-variable
- super-on-old-class

pep8:
disable:
- E1002

ignore-paths:
- tests

requirements:
- test.req

#max-line-length: 120
15 changes: 15 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
language: python
before_install:
- curl -O https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.3.deb && sudo dpkg -i --force-confnew elasticsearch-1.7.3.deb
before_script:
- sleep 10
python:
- "2.7"
services: elasticsearch
install:
- "pip install --upgrade -r test.req"
script: make test
after_success:
- coveralls
notifications:
slack: catholabs:9yCjbY6Jgn3Xdy9hwq6PyLEJ
362 changes: 22 additions & 340 deletions LICENSE

Large diffs are not rendered by default.

21 changes: 21 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
.PHONY: test
test: pep8
py.test -v --cov=esengine -l --tb=short --maxfail=1 tests/

.PHONY: install
install:
python setup.py develop

.PHONY: pep8
pep8:
@flake8 esengine --ignore=F403

.PHONY: sdist
sdist: test
@python setup.py sdist upload

.PHONY: clean
clean:
@find ./ -name '*.pyc' -exec rm -f {} \;
@find ./ -name 'Thumbs.db' -exec rm -f {} \;
@find ./ -name '*~' -exec rm -f {} \;
322 changes: 322 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,322 @@
[![Travis CI](http://img.shields.io/travis/catholabs/esengine.svg)](https://travis-ci.org/catholabs/esengine)
[![Coverage Status](http://img.shields.io/coveralls/catholabs/esengine.svg)](https://coveralls.io/r/catholabs/esengine)
[![Code Health](https://landscape.io/github/catholabs/esengine/development/landscape.svg?style=flat)](https://landscape.io/github/catholabs/esengine/development)
<a href="http://smallactsmanifesto.org" title="Small Acts Manifesto"><img src="http://smallactsmanifesto.org/static/images/smallacts-badge-80x15-blue.png" style="border: none;" alt="Small Acts Manifesto" /></a>

# ESEngine - ElasticSearch ODM
## (Object Document Mapper) inspired by MongoEngine

<p align="left" style="float:left" >
<img src="octosearch.gif" alt="EsEngine" width="300" />
</p>


# install

ESengine depends on elasticsearch-py (Official E.S Python library) so the instalation
depends on the version of elasticsearch you are using.


## Elasticsearch 2.x

```bash
pip install esengine[es2]
```

## Elasticsearch 1.x

```bash
pip install esengine[es1]
```

## Elasticsearch 0.90.x

```bash
pip install esengine[es0]
```

The above command will install esengine and the elasticsearch library specific for you ES version.


> Alternatively you can install elasticsearch library before esengine

pip install ``<version-specific-es>``

- for 2.0 + use "elasticsearch>=2.0.0,<3.0.0"
- for 1.0 + use "elasticsearch>=1.0.0,<2.0.0"
- under 1.0 use "elasticsearch<1.0.0"

Then install esengine

```bash
pip install esengine
```

# Getting started

```python
from elasticsearch import ElasticSearch
from esengine import Document, StringField

es = ElasticSearch(host='host', port=port)
```

# Defining a document

```python
class Person(Document):
_doctype = "person"
_index = "universe"

name = StringField()

```

> If you do not specify an "id" field, ESEngine will automatically add "id" as StringField. It is recommended that when specifying you use StringField for ids.

# Indexing

```python
person = Person(id=1234, name="Gonzo")
person.save(es=es)
```

# Getting by id

```python
Person.get(id=1234, es=es)
```

# filtering by IDS

```python
ids = [1234, 5678, 9101]
power_trio = Person.filter(ids=ids)
```


# filtering by fields

```python
Person.filter(name="Gonzo", es=es)
```

# Searching

ESengine does not try to create abstraction for query building,
by default ESengine only implements search transport receiving a raw ES query
in form of a Python dictionary.

```python
query = {
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"ids": {
"values": list(ids)
}
}
}
}
}
Person.search(query, size=10, es=es)
```

# Default connection

By default ES engine does not try to implicit create a connection for you,
but you can easily achieve this overwriting the **get_es** method and returning a
default connection or using any kind of technique as RoundRobin or Mocking for tests
Also you can set the **_es** attribute pointing to a function generating the connection client
or the client instance as the following example:

```python

from elasticsearch import ElasticSearch
from esengine import Document, StringField
from esengine.utils import validate_client


class Person(Document):
_doctype = "person"
_index = "universe"
_es = Elasticsearch(host='10.0.0.0')

name = StringField()

```

# Now you can use the document transport methods ommiting ES instance


```python
person = Person(id=1234, name="Gonzo")
person.save()

Person.get(id=1234)

Person.filter(name="Gonzo")
```


# Updating

## A single document

A single document can be updated simply using the **.save()** method

```python

person = Person.get(id=1234)
person.name = "Another Name"
person.save()

```

## Updating a Resultset

The Document methods **.get**, **.filter** and **.search** will return an instance
of **ResultSet** object. This object is an Iterator containing the **hits** reached by
the filtering or search process and exposes some CRUD methods[ **update**, **delete** and **reload** ]
to deal with its results.


```python
people = Person.filter(field='value')
people.update(another_field='another_value')
```

> When updating documents sometimes you need the changes done in the E.S index reflected in the objects
of the **ResultSet** iterator, so you can use **.reload** method to perform that action.


## The use of **reload** method

```python
people = Person.filter(field='value')
print people
... <Resultset: [{'field': 'value', 'another_field': None},
{'field': 'value', 'another_field': None}]>

# Updating another field on both instances
people.update(another_field='another_value')
print people
... <Resultset: [{'field': 'value', 'another_field': None}, {'field': 'value', 'another_field': None}]>

# Note that in E.S index the values weres changed but the current ResultSet is not updated by defaul
# you have to fire an update
people.reload()

print people
... <Resultset: [{'field': 'value', 'another_field': 'another_value'},
{'field': 'value', 'another_field': 'another_value'}]>


```

## Deleting documents


### A ResultSet

```python
people = Person.all()
people.delete()
```

### A single document

```python
Person.get(id=123).delete()
```

# Bulk operations

ESEngine takes advantage of elasticsearch-py helpers for bulk actions,
the **ResultSet** object uses **bulk** melhod to **update** and **delete** documents.

But you can use it in a explicit way using Document's **update_all**, **save__all** and **delete_all** methods.

### Lets create a bunch of document instances


```python
top_5_racing_bikers = []

for name in ['Eddy Merckx',
'Bernard Hinault',
'Jacques Anquetil',
'Sean Kelly',
'Lance Armstrong']:
top_5_racing_bikers.append(Person(name=name))
```

### Save it all

```python
Person.save_all(top_5_racing_bikers)
```

### Using the **create** shortcut

The above could be achieved using **create** shortcut


#### A single

```python
Person.create(name='Eddy Merckx', active=False)
```

> Create will return the instance of the indexed Document

#### All using list comprehension

```python
top_5_racing_bikers = [
Person.create(name=name, active=False)
for name in ['Eddy Merckx',
'Bernard Hinault',
'Jacques Anquetil',
'Sean Kelly',
'Lance Armstrong']
]

```
> NOTE: **.create** method will automatically save the document to the index, and
will not raise an error if there is a document with the same ID (if specified), it will update it acting as upsert.

### Updating all

Turning the field **active** to **True** for all documents

```python
Person.update_all(top_5_racing_bikes, active=True)
```

### Deleting all

```python
Person.delete_all(top_5_racing_bikes)
```


### Chunck size

chunk_size is number of docs in one chunk sent to ES (default: 500)
you can change using **meta** argument.

```python
Person.update_all(
top_5_racing_bikes, # the documents
active=True, # values to be changed
metal={'chunk_size': 200} # meta data passed to **bulk** operation
)
```

# Contribute

ESEngine is OpenSource! join us!
Loading