Merge Development #79

domodwyer · 2018-01-15T17:22:38Z

Includes:

Reduced memory in bulk operations (Reduce memory allocation in bulk op #56)
Native x509 authentication (Add proper DN construction #55)
Better connection recovery (Recover from persistent "i/o timeout" or "Closed explicitly" pool errors #69)
Example usage (auth: add an example for x509 authentication #75 and session: add example concurrent usage #78)

Thanks to:

Select throughput has increased by ~600 requests/second with slightly increased variance:

x => r2017.11.06-select-zipfian-throughput.log 
y => 9acbd68815499fc7f8fbb5931d643b8fd7804a80-select-zipfian-throughput.log 

     n   min   max median  average   stddev      p99
x 3600 49246 71368  66542 66517.26 2327.675 70927.01
y 3600 53304 72005  67151 67145.36 2448.534 71630.00

          62000       64000       66000        68000       70000       72000    
 |----------+-----------+-----------+------------+-----------+-----------+-----|
                              +---------+--------+                              
1          -------------------|         |        |--------------------          
                              +---------+--------+                              
                                 +---------+---------+                          
2    ----------------------------|         |         |--------------------      
                                 +---------+---------+                          
Legend: 1=data$x, 2=data$y

At 95% probablitiy:
===> average is statistically significant     (p=0.000000, diff ~628.094444)
===> variance is statistically significant    (p=0.002398)

Note: latencies are approximations calculated from grouped data

* master: Add contribution guidelines. Add more changes to README.

* master: Run integration tests against 3.2.16 (#24)

Create a new method to drop all the indexes of a collection in a single call

fix [#484](https://github.com/go-mgo/mgo/issues/484) Annotate connections with metadata provided by the connecting client. informations send: { "aplication": { // optional "name": "myAppName" } "driver": { "name": "mgo", "version": "v2" }, "os": { "type": runtime.GOOS, "architecture": runtime.GOARCH } } to set "application.name", add `appname` param in options of string connection URI, for example : "mongodb://localhost:27017?appname=myAppName"

* docs: elaborate on what appName does * readme: add appName to changes

@feliixx

Fix #30. Thanks to @feliixx for the time and effort.

@mapete94

* Stop all db instances after tests (go-mgo#462) If all tests pass, the builds for mongo earlier than 2.6 are still failing. Running a clean up fixes the issue. * fixing int64 type failing when getting indexes and trying to type them * requested changes relating to case statement and panic * Update README.md to credit @mapete94. * tests: ensure indexed int64 fields do not cause a panic in Indexes() See: * #23 * https://github.com/go-mgo/mgo/issues/475 * go-mgo#476

- Allow specifying the default collation for the collection when creating it. - Add some documentation to query.Collation() method. fix #29

* test against MongoDB 3.4.x * tests: use listIndexes to assert index state for 3.4+ * make test pass against v3.4.x - skip `TestViewWithCollation` because of SERVER-31049, cf: https://jira.mongodb.org/browse/SERVER-31049 - add versionAtLeast() method in init.js script to better detect server version fixes #31

@feliixx

* Adds missing collation feature description (by @feliixx). * Adds missing 3.4 tests description (by @feliixx). * Adds BSON constants description (by @bozaro). * Adds UTC time.Time unmarshalling (by @gazoon).

readme: add missing feature descriptions / credit

Fixes #43

* Fix GetBSON() method usage Original issue --- You can't use type with custom GetBSON() method mixed with structure field type and structure field reference type. For example, you can't create custom GetBSON() for Bar type: ``` struct Foo { a Bar b *Bar } ``` Type implementation (`func (t Bar) GetBSON()` ) would crash on `Foo.b = nil` value encoding. Reference implementation (`func (t *Bar) GetBSON()` ) would not call on `Foo.a` value encoding. After this change --- For type implementation `func (t Bar) GetBSON()` would not call on `Foo.b = nil` value encoding. In this case `nil` value would be seariazied as `nil` BSON value. For reference implementation `func (t *Bar) GetBSON()` would call even on `Foo.a` value encoding. * Minor refactoring

This change remove full BSON decoding on: - parsing to `bson.Raw` and `bson.DocElem` fields; - skipping unused BSON fields.

@bozaro

* readme: credit @bozaro and @idy * readme: add @idy to contributor list

fix #51

* Add proper DN construction * Added openssl check to test * Addressed code review comments * Changes to RDNformatting, and test * type/value fields to tests * Changes to RDN formatting * Corrected comment in getRFC2253NameString * Corrected comment in getRFC2253NameString * Changes to login and rdn formatting * Changed escaping of #

use memory pooling to reuse bulkActions and avoid some allocations

* test against 3.6 * update go and mongodb version - use last minor version for each major serie of mongodb - use travis 'go' param instead of eval "$(gimme $GO)" as it fails to install correctly last minor version ( 1.8.x notation for example) * test fixes on 3.2.17 also re-enable TestFindIterSnapshot as it was fixed a long time ago * fix X509 test fix TestAuthX509CredRDNConstruction test: need to create an user with {"username": subject} before trying to login * Fix auth test on 3.6-rc3 Make sure that "rs3/127.0.0.1/40031" is elected at primary. Create user before running 'addShard' command as it requires root access Also add a retry mechanism to make sure that shard are correctly added cf https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/ * implement OP_MSG wire protocole require maxWireVersion >= 6 on server side, and `experimental=opmsg` in the connection string - get `MaxMessageSizeBytes` and `MaxWriteBatchSize` from handshake - allow unacknowledged writes with `moreToCome` flag - split bulk operations in batch of `maxWriteBatchSize` - add 'experimental' param in URL. To enable an experimental feature, add `experimental=featureName` in the connection URL flush logout directly Previously, `flushLogout()` was called at the beginning of each Query to the database. To avoid these useless calls, we flush logout directly when `Logout()` or `LogoutAll()` is called * re-enable TestViewWithCollation SERVER-31049 is fixed in 3.4.10, so re-enable it * refactor memory pooling use the same pool for send and received messages. Slices are returned to the pool without being resized. Default allocation size might need to be updated (currently 256, no benchmarks available yet) * update to 3.6.0 stable

* master: Merge Development (#57) Revert "do not lock while writing to a socket (#52)" do not lock while writing to a socket (#52) Merge Development (#48) Merge development (#39) # Conflicts: # .travis.yml # cluster_test.go # harness/daemons/.env # harness/mongojs/init.js # session.go # session_internal_test.go # session_test.go

…ors (#69)

* Revert "MongoDB 3.6: implement the new wire protocol (#61)" This reverts commit 90c056c. * test against 3.6 * update go and mongodb version - use last minor version for each major serie of mongodb - use travis 'go' param instead of eval "$(gimme $GO)" as it fails to install correctly last minor version ( 1.8.x notation for example) * test fixes on 3.2.17 also re-enable TestFindIterSnapshot as it was fixed a long time ago * fix X509 test fix TestAuthX509CredRDNConstruction test: need to create an user with {"username": subject} before trying to login * Fix auth test on 3.6-rc3 Make sure that "rs3/127.0.0.1/40031" is elected at primary. Create user before running 'addShard' command as it requires root access Also add a retry mechanism to make sure that shard are correctly added cf https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/ * update to 3.6.0 stable * tests: cherry pick missing 3.6+ support changes

* auth: add an example for x509 authentication

szank · 2018-01-15T17:39:24Z

.travis.yml

+        - MONGODB=x86_64-ubuntu1404-3.0.15
+        - MONGODB=x86_64-ubuntu1404-3.2.17
+        - MONGODB=x86_64-ubuntu1404-3.4.10
+        - MONGODB=x86_64-ubuntu1404-3.6.0


Shall we upgrade to 3.6.2 while we are at it ?

@bachue

Includes: * Reduced memory in bulk operations (globalsign#56) * Native x509 authentication (globalsign#55) * Better connection recovery (globalsign#69) * Example usage (globalsign#75 and globalsign#78) Thanks to: * @bachue * @csucu * @feliixx --- [Throughput overview](https://user-images.githubusercontent.com/9275968/34954403-3d3253dc-fa18-11e7-8eef-0f2b0f21edc3.png) Select throughput has increased by ~600 requests/second with slightly increased variance: ``` x => r2017.11.06-select-zipfian-throughput.log y => 9acbd68-select-zipfian-throughput.log n min max median average stddev p99 x 3600 49246 71368 66542 66517.26 2327.675 70927.01 y 3600 53304 72005 67151 67145.36 2448.534 71630.00 62000 64000 66000 68000 70000 72000 |----------+-----------+-----------+------------+-----------+-----------+-----| +---------+--------+ 1 -------------------| | |-------------------- +---------+--------+ +---------+---------+ 2 ----------------------------| | |-------------------- +---------+---------+ Legend: 1=data$x, 2=data$y At 95% probablitiy: ===> average is statistically significant (p=0.000000, diff ~628.094444) ===> variance is statistically significant (p=0.002398) ``` * [insert-latency.txt](https://github.com/globalsign/mgo/files/1632474/insert-latency.txt) * [insert-throughput.txt](https://github.com/globalsign/mgo/files/1632475/insert-throughput.txt) * [select-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632476/select-zipfian-latency.txt) * [select-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632477/select-zipfian-throughput.txt) * [update-zipfian-latency.txt](https://github.com/globalsign/mgo/files/1632478/update-zipfian-latency.txt) * [update-zipfian-throughput.txt](https://github.com/globalsign/mgo/files/1632479/update-zipfian-throughput.txt) Note: latencies are approximations calculated from grouped data

domodwyer and others added 30 commits August 8, 2017 11:00

Merge branch 'master' into development

de872a6

* master: Add contribution guidelines. Add more changes to README.

Merge branch 'master' into development

1519fd3

* master: Run integration tests against 3.2.16 (#24)

add DropAllIndexes() method (#25)

454da02

Create a new method to drop all the indexes of a collection in a single call

readme: credit @feliixx for #25 (#26)

93aaa6e

Update README to add appName (#32)

a76b1a0

* docs: elaborate on what appName does * readme: add appName to changes

add method CreateView() (#33)

25200e4

Fix #30. Thanks to @feliixx for the time and effort.

readme: credit @feliixx in the README (#36)

1f4c10f

Add collation option to collection.Create() (#37)

b37e3c1

- Allow specifying the default collation for the collection when creating it. - Add some documentation to query.Collation() method. fix #29

Introduce constants for BSON element types (#41)

950ed5a

bson.Unmarshal returns time in UTC (#42)

d21a525

readme: add missing features / credit

9d743b4

* Adds missing collation feature description (by @feliixx). * Adds missing 3.4 tests description (by @feliixx). * Adds BSON constants description (by @bozaro). * Adds UTC time.Time unmarshalling (by @gazoon).

Merge pull request #45 from globalsign/feature/update-readme

c86ed84

readme: add missing feature descriptions / credit

fix golint, go vet and gofmt warnings (#44)

97bd0cd

Fixes #43

readme: credit @feliixx (#46)

fd79249

readme: credit @bozaro (#47)

12fb1c2

Improve cursorData struct unmarshaling speed (#49)

199dc25

This change remove full BSON decoding on: - parsing to `bson.Raw` and `bson.DocElem` fields; - skipping unused BSON fields.

readme: credit @bozaro and @idy (#53)

345ab0b

* readme: credit @bozaro and @idy * readme: add @idy to contributor list

do not lock while writing to a socket (#52) (#54)

0454966

fix #51

reduce memory allocation in bulk op (#56)

7cd0b89

use memory pooling to reuse bulkActions and avoid some allocations

readme: credit @feliixx (#58)

ea8e8e6

Recover from persistent "i/o timeout" or "Closed explicitly" pool err…

a104bfb

…ors (#69)

readme: credit @bachue (#74)

138ba2f

domodwyer added 2 commits January 9, 2018 14:05

auth: add an example for x509 authentication (#75)

9acbd68

* auth: add an example for x509 authentication

session: add example concurrent usage (#78)

eeedc17

domodwyer added bug enhancement release labels Jan 15, 2018

domodwyer self-assigned this Jan 15, 2018

domodwyer requested review from szank, brknstrngz, weiishann, csucu and eminano January 15, 2018 17:22

tadukurow approved these changes Jan 15, 2018

View reviewed changes

csucu approved these changes Jan 15, 2018

View reviewed changes

weiishann approved these changes Jan 15, 2018

View reviewed changes

domodwyer merged commit 896bbb8 into master Jan 15, 2018

szank reviewed Jan 15, 2018

View reviewed changes

domodwyer mentioned this pull request Feb 12, 2018

Resync development & master branches #102

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge Development #79

Merge Development #79

domodwyer commented Jan 15, 2018 •

edited

Loading

szank Jan 15, 2018

Merge Development #79

Merge Development #79

Conversation

domodwyer commented Jan 15, 2018 • edited Loading

szank Jan 15, 2018

Choose a reason for hiding this comment

domodwyer commented Jan 15, 2018 •

edited

Loading