Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breadcrumbs/Mongo integration: Handle 'any_of' cases #522

Merged
merged 8 commits into from
Jan 11, 2019

Conversation

Cawllec
Copy link
Contributor

@Cawllec Cawllec commented Jan 8, 2019

Goal

Enables mongo breadcrumb filters to include nested keys, such as in any_of cases.

@Cawllec Cawllec requested review from tobyhs and a team January 8, 2019 12:04
def sanitize_filter(filter_hash)
filter_hash.map do |key, value|
if value && value.is_a?(Array)
filtered_or = value.map { |filter| sanitize_filter(filter) }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable could be slightly misleading because an Array value doesn't necessarily mean it was an $or (it could have been $and or something else).

# @param filter_hash [Hash] the filter hash for the mongo transaction
#
# @return [Hash] the filtered hash
def sanitize_filter(filter_hash)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method fails on queries such as SomeModel.where(tags: ['a', 'b']):

irb(main):001:0> def sanitize_filter(filter_hash)
irb(main):002:1>   filter_hash.map do |key, value|
irb(main):003:2*     if value && value.is_a?(Array)
irb(main):004:3>       filtered_or = value.map { |filter| sanitize_filter(filter) }
irb(main):005:3>       [key, filtered_or]
irb(main):006:3>     else
irb(main):007:3*       [key, '?']
irb(main):008:3>     end
irb(main):009:2>   end.to_h
irb(main):010:1> end
=> :sanitize_filter
irb(main):011:0> sanitize_filter({'tags' => [1, 2]})
NoMethodError: undefined method `map' for 1:Integer
Did you mean?  tap
	from (irb):2:in `sanitize_filter'
	from (irb):4:in `block (2 levels) in sanitize_filter'
	from (irb):4:in `map'
	from (irb):4:in `block in sanitize_filter'
	from (irb):2:in `each'
	from (irb):2:in `map'
	from (irb):2:in `sanitize_filter'
	from (irb):11

I would change the method/algorithm to something like:

def sanitize_filter_element(element)
  if element.is_a?(Array)
    element.map { |e| sanitize_filter_element(e) }
  elsif element.is_a?(Hash)
    element.map { |k, v| [k, sanitize_filter_element(v)] }.to_h
  else
    '?'
  end
end

The only concern I have about the above is that there could be giant arrays for $in, but I'll pull the old college textbook approach of writing "This is left as an exercise for the reader".

@@ -2,4 +2,5 @@ class MongoModel
include Mongoid::Document

field :string_field, type: String
field :numeric_field, type: Numeric
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Numeric type isn't in the valid types of https://docs.mongodb.com/mongoid/current/tutorials/mongoid-documents/#fields so I would just use Integer here instead.

# Removes values from filter hashes, replacing them with '?'
#
# @param filter_hash [Hash] the filter hash for the mongo transaction
# @param depth [Numeric] the current filter depth
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Integer] (as the depth isn't going to be something that isn't a whole number)

#
# @return [Hash] the filtered hash
def sanitize_filter_hash(filter_hash, depth = 0)
filter_hash.each_with_object({}) do |args, output|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would change the block params to |(key, value), output| so you don't need the next line

module ::Mongo
module Monitoring
COMMAND = 'command'
class Global
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be module instead of class to match the mongo gem

expect(subscriber.respond_to?(:succeeded)).to be true
expect(subscriber.respond_to?(:failed)).to be true
end
require './lib/bugsnag/integrations/mongo'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be load (and you might need to end the argument with ".rb") instead of require. The reason is that if the file was required before from some other test, then the require won't load the file.

end

describe "#succeeded" do
it "calls #leave_mongo_beradcrumb with the event_name and event" do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leave_mongo_breadcrumb (spelling)

subscriber.send(:leave_mongo_breadcrumb, event_name, event)
end

it "adds the correct colleciton name for 'getMore' commands" do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collection (spelling)

:request_id => "123456",
:duration => "123.456",
:collection => "collection_name_command",
:filter => "{\"a\":\"?\",\"b\":\"?\",\"$or\":[{\"c\":\"?\"},{\"d\":\"?\"}]}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use single quotes for the outer quotes here so that the blackslashes aren't needed and it is easier to read this string literal


describe "#sanitize_filter_hash" do
it "calls into #sanitize_filter_value with the value from each {k,v} pair" do
expect(subscriber).to receive(:sanitize_filter_value).with(1, 0).ordered.and_return('?')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would omit all your sanitize_filter_value method/message expectations since your equality expectations should be good enough. Method/message expectations tend to be for things that have side effects that you don't want run (e.g. remote/network calls).

it "removes the command from the request_data" do
subscriber.send(:pop_command, request_id)
command_hash = Bugsnag.configuration.request_data[Bugsnag::MongoBreadcrumbSubscriber::MONGO_COMMAND_KEY]
expect(command_hash).not_to include(request_id => command)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would change this to not_to have_key(request_id) as the key shouldn't be in at all and the value is not relevant to the test.

Copy link
Contributor

@tobyhs tobyhs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few more comments, otherwise lgtm

@mocked_mongo = true
module ::Mongo
module Monitoring
COMMAND = 'command'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'Command' (capital C) to match the mongo gem

expect(subscriber.send(:sanitize_filter_hash, {:a => 1, :b => 2})).to eq({:a => '?', :b => '?'})
end

it "defaults the depth to 0" do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests on lines 185-192 no longer provide much value, so I would remove them.

end

describe "#failed" do
it "calls #leave_mongo_beradcrumb with the event_name and event" do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leave_mongo_breadcrumb (spelling)

subscriber.send(:sanitize_filter_value, {:a => 1}, 0)
end

it "increments the depth for each call" do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc string is no longer accurate; I would replace line 203 with line 212 as the input on line 212 recurses and remove this block (so the "is recursive and iterative for array values" test uses [1, [2, [3]]])

Copy link
Contributor

@tobyhs tobyhs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@Cawllec Cawllec merged commit 77dece6 into breadcrumbs/base Jan 11, 2019
@Cawllec Cawllec deleted the breadcrumbs/mongo-filters branch January 11, 2019 13:48
Cawllec added a commit that referenced this pull request Jan 17, 2019
* Breadcrumbs/Circular buffer: Added circular buffer class

* Breadcrumbs/Circular buffer: Added initial buffer unit tests

* Breadcrumbs/Circular buffer: Ensure test file is named correctly

* Breadcrumbs/Circular buffer: Code tidy up

* Breadcrumbs / Breadcrumb class: Add first pass breadcrumb class with unit tests

* Breadcrumbs/Circular buffer rubocop fixes

* Breadcrumbs / Breadcrumb class: Rubocop fixes

* Breadcrumbs/Cricular buffer: Improve test readability

* Breadcrumbs/Breadcrumb: Simplified timestamp, improved tests

* Breadcrumbs/Breadcrumb: Rubocop requirements

* Breadcrumbs/Breadcrumb: Added doc comments

* Breadcrumbs/Circular buffer: Fixed RSpec/Rspec typo

* Breadcrumbs/Breadcrumb: Using explicit RSpec.describe call

* Breadcrumbs/Validator: Added validator and tests

* Breadcrumbs/Validator: Rubocop fixes

* Breadcrumbs/Validator: Tidying up from initial review

* Breadcrumbs/Validator: Re-add renamed types file

* Breadcrumbs/Breadcrumb: Change message => name, add yard comments

* Breadcrumbs/Circular buffer: Added yard docs

* Breadcrumbs/Breadcrumb: Improve tests, convert to utc sooner

* Breadcrumbs/Configuration: Initial configuration changes

* Breadcrumbs/Report: Initial report changes to support breadcrumbs

* Breadcrumbs/Breadcrumb: Test timestamps match up

* Breadcrumbs/Validator: Change 'message' to 'name', use name and meta_data clones

* Breadcrumbs/Validator: Refactor toward instance_double & let

* Breadcrumbs/Configuration: Add in missing import statement

* Breadcrumbs/Circular buffer docs: Improve readability

* Breadcrumbs/Configuration: Add configuration tests

* Breadcrumbs/Configuration: Added yardoc comments

* Breadcrumbs/Configuration: Simplify before_breadcrumb_callbacks, allow automatic_breadcrumb_types editting

* Breadcrumbs/Configuration: Add automatic breadcrumbs types editablilty test

* Breadcrumbs/Configuration: Add types to Yardoc arrays

* Breadcrumbs/Validator: Use an instance_double to reflect Configuration better

* Breadcrumbs/Validator: Add Yardoc comments

* Breadcrumbs/Validator: Use select over each when filtering

* Breadcrumbs/Validator: Clean up tests

* Breadcrumbs/Validator: Use double for configuration

* Breadcrumbs/Validator: Improved Yardoc

* Breadcrumbs/Leave breadcrumb: Add implementation and tests

* Breadcrumbs/Middleware: Add breadcrumb middleware

* Breadcrumbs/Report: Rubify get_summary method name

* Breadcrumbs/Report: Ensure message is set correctly

* Breadcrumbs/Report: Change default breadcrumbs from Hash to Array

* Breadcrumbs/Report: Ensure breadcrumb meta_data is filtered correctly

* Breadcrumbs/Report: Fixed typo

* Bugsnag/Report: Added tests

* Breadcrumbs/Leave breadcrumb: Update rubocop to allow increased Bugsnag module length

* Breadcrumbs/middleware: Rubocop fixes

* Breadcrumbs/Leave-breadcrumb: Rubocop improvements/early return

* Breadcrumbs/Leave-breadcrumb: Better block styling

* Breadcrumbs/Report: Ensure report summary is consistent with dashboard

* Breadcrumbs/Report: Better Rubocop report exclusions

* Breadcrumbs/Report: Rspec improvements

* Breadcrumbs/Notify-breadcrumb: Initial implementation and tests

* Breadcrumbs/Notify breadcrumb: Add test against notifying a string

* Breadcrumbs/Report: Better checking against Java error

* Breadcrumbs/Report: Make summary more accurate

* Breadcrumbs/Report update: make summary guard more readable

* Breadcrumbs/Update automatic_breadcrumb_types to enabled_automatic_breadcrumb_types

* Breadcrumbs: Rails Breadcrumbs (#514)

* Breadcrumbs/Rails integration: Initial integration implemenetation

* Breadcrumbs/Rails: Minor fixes

* Breadcrumbs/Rails: Add initial maze tests

* Breadcrumbs/Rails breadcrumbs: Remove plural event data

* Breadcrumbs/Rails breadcrumbs: Ammend breadcrumb namespace

* Breadcrumbs/Rails breadcrumbs: Remove event_id from allowed keys, change name to event_name

* Breadcrumbs/Rails breadcrumbs: Require breadcrumb file when necessary to avoid namespace clashes

* Breadcrumbs/Rails breadcrumbs: Fix rubocop issues

* Breadcrumbs/Rails breadcrumbs: Clarify namespaces better

* Breadcrumbs/Rails breadcrumbs: Remove invalid data from captured breadcrumb definitions

* Breadcrumbs/Rails breadcrumbs: Added ActiveJob breadcrumb test

* Breadcrumbs/Rails breadcrumbs: Ammended active_job event id

* Breadcrumbs/Ruby breadcrumbs: Removed complex breadcrumb metadata

* Breadcrumbs/Ruby: Added cache breadcrumb

* Breadcrumbs/Ruby breadcrumbs: Fleshed out maze-test expectations where possible

* Breadcrumbs/Rails breadcrumbs: Fixed broken test fixture, made fixtures more rails-y

* Breadcrumbs: Add Mongo integration (#516)

* Breadcrumbs/Rails integration: Initial integration implemenetation

* Breadcrumbs/Rails: Minor fixes

* Breadcrumbs/Rails: Add initial maze tests

* Breadcrumbs/Mongo: Initial subscriber implementation

* Breadcrumbs/Rails breadcrumbs: Remove plural event data

* Breadcrumbs/Rails breadcrumbs: Ammend breadcrumb namespace

* Breadcrumbs/Rails breadcrumbs: Remove event_id from allowed keys, change name to event_name

* Breadcrumbs/Rails breadcrumbs: Require breadcrumb file when necessary to avoid namespace clashes

* Breadcrumbs/Mongo integration: Fix module name issue

* Breadcrumbs/Mongo integration: Maintain consistency with ActiveSupport arguments

* Breadcrumbs/Rails breadcrumbs: Fix rubocop issues

* Breadcrumbs/Rails breadcrumbs: Clarify namespaces better

* Breadcrumbs/Mongo integration: Add integration tests

* Breadcrumbs/Mongo integration: Remove example commits

* Breadcrumbs/Rails breadcrumbs: Remove invalid data from captured breadcrumb definitions

* Breadcrumbs/Mongo integration: Rubocop fixes

* Breadcrumbs/Mongo integration: Better doc comments

* Breadcrumbs/Mongo integration: Improve tests, add 'failure' test

* Breadcrumbs/Mongo integration: Re-add 'success' test

* Breadcrumbs/Rails breadcrumbs: Added ActiveJob breadcrumb test

* Breadcrumbs/Rails breadcrumbs: Ammended active_job event id

* Breadcrumbs/Ruby breadcrumbs: Removed complex breadcrumb metadata

* Breadcrumbs/Ruby: Added cache breadcrumb

* Breadcrumbs/Ruby breadcrumbs: Fleshed out maze-test expectations where possible

* Breadcrumbs/Mongo integration: Rubocop fixes

* Breadcrumbs/Rails breadcrumbs: Fixed broken test fixture, made fixtures more rails-y

* Breadcrumbs/Mongo integration: Maze fixes

* Breadcrumbs/Validator: Allow nil as an acceptable meta_data type (#520)

* Breadcrumbs/Mongo: Remove 'started' breadcrumb, add collection detail (#519)

* Breadcrumbs/Mongo integration: Remove 'started' event, add 'collection' data

* Breadcrumbs/Mongo: Add filter keys

* Breadcrumbs/Mongo integration: Made filters consistent with other breadcrumb implementations

* Breadcrumbs/Mongo integration: Rubocop fixes

* Breadcrumbs/Mongo integration: Address feedback

* Breadcrumbs: Additional Rails ActiveRecord data (#521)

* Breadcrumbs/Rails breadcrumbs: Capture event_id

* Breadcrumbs/Rails breadcrumbs: Capture 'sql' string and redacted bindings

* Breadcrumbs/Rails integration: Redact SQL string values

* Breadcrumbs/Rails integration: Remove SQL string acquisition

* Breadcrumbs/Rails improvements: Remove unnecessary require

* Breadcrumbs/Rails integration: Add integration tests for bindings

* Breadcrumbs/Ruby integration: lock nokogiri version

* Breadcrumbs/Ruby integration: Better meta_data naming, Ruby < 2.1 compatability

* Breadcrumbs/Mongo integration: Handle 'any_of' cases (#522)

* Breadcrumbs/Mongo integration: Handle 'any_of' cases

* Breadcrumbs/Mongo Integration: Improve filter sanitization process and add unit tests

* Breadcrumbs/Mongo integration: Ensure tests are notated correctly

* Breadcrumbs/Mongo filters: Add 1.9.3 & 2.0.0 compatability

* Breadcrumbs/Mongo integration: Appease rubocop

* Breadcrumbs/Mongo integration: Minor test and efficieny improvements

* Breadcrumbs/Mongo integration: Test improvements

* Breadcrumbs/Mongo integrations: Test spelling

* Breadcrumbs: Fix rubocop issues (excluding configuration from ClassLength until refactor)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants