How can I use stub_responses with Aws::S3::Resource instead of Aws::S3::Client?

[apurvis]

I have in the past been able to make very good use of stub_responses with Aws::S3::Client; however when it comes to Aws::S3::Resource I am at a bit of a loss as to how to actually stub anything.

As an example, I would like to stub a particular key as a response to Aws::S3::Bucket.objects. I have managed to get things to work in the sense of not exceptioning out with something like

s3 = Aws::S3::Resource.new(stub_responses: true)
bucket = Aws::S3::Bucket.new(name: 'bucket', stub_responses: true)

But this doesn't work:

bucket.stub_responses(objects: ['file'])

NoMethodError: undefined method `stub_responses' for #<Aws::S3::Bucket name="x">
	from (irb):140
	from /data/warehouse/shared/bundle/ruby/2.3.0/gems/railties-4.0.13/lib/rails/commands/console.rb:90:in `start'
	from /data/warehouse/shared/bundle/ruby/2.3.0/gems/railties-4.0.13/lib/rails/commands/console.rb:9:in `start'
	from /data/warehouse/shared/bundle/ruby/2.3.0/gems/railties-4.0.13/lib/rails/commands.rb:62:in `<top (required)>'
	from script/rails:6:in `require'
	from script/rails:6:in `<main>'

and neither does

s3_stub.stub_responses(bucket: bucket_stub)

i tried to create a stub for Aws::S3::Client ahead of time with stubbed responses and pass that into the Resource constructor but that also didn't work:

client_stub = Aws::S3::Client.new(region: 'region', stub_responses: true)
client_stub.stub_responses(:list_buckets, {
  buckets: [{ name: 'my-bucket' }]
})
resource_stub = Aws::S3::Resource.new(client_stub)
resource_stub.buckets
=> #<Aws::Resources::Collection type="Aws::S3::Bucket" limit=nil params={}>

and when i tried

resource_stub.list_buckets

i didn't actually get a stubbed response - i got a real response.

[awood45]

I was able to get your example to work with a slight change:

client_stub = Aws::S3::Client.new(stub_responses: true)
client_stub.stub_responses(:list_buckets, {
  buckets: [{ name: 'my-bucket' }]
})
resource_stub = Aws::S3::Resource.new(client: client_stub)
resource_stub.buckets # Note that this won't actually make a stubbed call, but buckets.first works.

[awood45]

I'll check the documentation to see if we could improve clarity, but let me know if this works for you or if you have further issues with stubbing resources.

[apurvis]

That's helpful and yeah, the documentation is completely unhelpful on this subject, which is a bit of a problem as users of the gem are encouraged more and more to use the Resource classes instead of the Client classes.

My question then becomes: "How could I stub a particular set of objects being returned from a particular bucket?" e.g. I just want to be able to stub that [s3://bucket/some/path/file1, s3://bucket/some/path/file2] exist in a bucket. I'm playing with this now but any help is appreciated.

[awood45]

So, consider the client calls being made by your resource calls. In this case, you would stub the response to :list_objects. It gets complex if you are checking multiple buckets, but if you'll only call it a single way, you could stub the response to :list_objects to similarly return those keys as you did for :list_buckets.

[apurvis]

i got it to work; would be awesome if you could update the documentation to include this but i'm good for now.

[awood45]

Thanks for the update, glad things are working well now. We do have an entry for this in the developer guide. Let me know if that would have helped as well (and we can make sure it's more prominently linked to).

[awood45]

Closing as the issue at hand is resolved, but still interested in your feedback on the guide.

[apurvis]

that part of the developer guide is basically useless for understanding how to stub a Resource; that's why I created this issue.

[ghost]

I agree that that section of the guide was not really helpful when trying to stub when using a Resource instead of a Client. One example that used a Resource would have been very helpful to me. This is what I eventually came up with after seeing the example above:

ec2 = Aws::EC2::Resource.new(stub_responses: Rails.env.test?)
ec2.client.stub_responses(:describe_instances, {
  reservations: [{
    instances: [ { instance_id: 'instance-42' } ]
  }]
})
ec2.instances.first
#=> #<Aws::EC2::Instance id="instance-42">

In hindsight it makes sense that to use "Client Response Stubs" requires using the client, but the documentation did not help me make that mental connection and I slogged through a few dead-ends before I found this thread. Thanks :-)

[DavidRagone]

Has there been any consideration to providing the ability to stub on resource w/o having to understand the underlying implementation on the client?

So, consider the client calls being made by your resource calls. In this case, you would stub the response to :list_objects. It gets complex if you are checking multiple buckets, but if you'll only call it a single way, you could stub the response to :list_objects to similarly return those keys as you did for :list_buckets.

What you're saying is that in order to test the API I'm using (Resource), I need to understand how it is implemented in terms of the Client. That feels like an interface that is not fully implemented.

[papadeltasierra]

Ditto DavidRagone's comments. I'm using BUCKETS and perhaps I could stub these but I don't want to have to learn the internals of the AWS SDK in order to do this. I'm guessing that thre is no other simple way to do this though - the example above needs me to know that RESOURCE.instances maps to CLIENT.described_instances[reservations][instances]! Sigh.

[thromera]

Bump. It's still unclear whether you can stub a resource or not.
Is Aws::EC2::Resource.new(stub_responses: true) gonna work?

I found it counter intuitive to create a stubbed client and pass it to the resource, since it can false the specs. I currently do that whenever I want to upload something to S3:

Aws::S3::Resource.new(region: region).bucket(bucket).object(file_name).tap do |object|
  object.put(body: file_data, **options)
end

[nzifnab]

Further bumping this. I'm doing something similar to @erowlin:

@s3_bucket ||= Aws::S3::Resource.new.bucket(ENV['S3_BUCKET'])
object = @s3_bucket.object(key)

And then I'm acting on that object (Specifically in this case calling presigned_url), but it's very unclear to me how best to stub out the response from bucket.object(key).

In my spec I have:

resource_stub = Aws::S3::Resource.new(stub_responses: true)
expect(Aws::S3::Resource).to receive(:new).and_return(resource_stub)

But again, I don't know where to go from here. Do I really need to know the underlying implementation of .bucket() and .object() and how they are calling Aws::S3::Client? That seems incredibly counter-intuitive.

Probably gonna go for stubbing the methods out with rspec, heh. But that doesn't give me a whole lot of confidence because then I can't be 100% sure that the methods I'm calling or the API I'm relying on is giving me the results I'm expecting.

[yuvmendel]

Have been wondering the exact same thing as @nzifnab. I see this ticket is closed but I wonder if there is a good answer to this, and if not, I would suggest opening a ticket for it.

[anabrs]

Hello! I'm trying to stub some Resource responses as well and I haven't been able to do so. Is it absolutely necessary to go through the Client?

[rokumatsumoto]

I was looking for a way to test my code that relies on AWS S3. I needed to stub bucket.object and bucket.object.exists? methods for my test cases.

After some digging, I came across these;
amazon-archives/aws-sdk-core-ruby#187 (comment)
#1289
#1058
https://gitter.im/aws/aws-sdk-ruby/archives/2016/01/30

Hello! I'm trying to stub some Resource responses as well and I haven't been able to do so. Is it absolutely necessary to go through the Client?

If I understand correctly, yes.
I see three ways of stubbing for the AWS SDK;

1. https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/ClientStubs.html#stub_data-instance_method
2. https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/ClientStubs.html#stub_responses-instance_method
3. https://github.com/rokumatsumoto/boyutluseyler/blob/4585c45afb02da7162eeaa8eed5c2131fcfeddc3/spec/services/blueprints/build_service_spec.rb#L39

You can supply a client object with custom configuration that will be used for all resource
operations. If you do not pass :client, a default client will be constructed

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Resource.html

But again, I don't know where to go from here. Do I really need to know the underlying implementation of .bucket() and .object() and how they are calling Aws::S3::Client? That seems incredibly counter-intuitive.

If I understand correctly, yes.
For the bucket.object(key) and bucket.object.exists? methods, I followed the implementation of these methods in source code.

1. aws-sdk-ruby/gems/aws-sdk-s3/lib/aws-sdk-s3/bucket.rb

   Line 655 in 31ae529

   def object(key)

2. aws-sdk-ruby/gems/aws-sdk-s3/lib/aws-sdk-s3/object.rb

   Line 356 in 31ae529

   def exists?(options = {})

Stubbing Aws::S3::Object and exists? method

context 'when the remote object exists' do
  it 'builds a blueprint without saving it' do
    direct_upload_bucket = class_double(ObjectStorage::DirectUpload::Bucket).as_stubbed_const
    remote_object = double(key: 'model.stl', size: 1, content_type: 'model/stl',
                            public_url: 'http://foo.com/model.stl', exists?: true)
    allow(direct_upload_bucket).to receive(:object).and_return(remote_object)

    result = service.execute

    expect(result).to be_valid
    expect(result).not_to be_persisted
  end
end

Stubbing Aws::S3::Object exists? responses

module StubAWSResponses
  def stub_direct_upload_bucket_object_is_not_exist
    DIRECT_UPLOAD_RESOURCE.client.stub_responses(:head_object,
                                                 status_code: 404,
                                                 headers: {},
                                                 body: '')
  end

  def stub_direct_upload_bucket_object_exists
    DIRECT_UPLOAD_RESOURCE.client.stub_responses(:head_object,
                                                 status_code: 200,
                                                 headers: {},
                                                 body: '')
  end
end

Testing response behavior

context 'when the remote object is not exist' do
  before do
    stub_direct_upload_bucket_object_is_not_exist
  end

  after do
    stub_direct_upload_bucket_object_exists # default stub behavior
  end

  it 'raises a file not found error' do
    expect { service.execute }.to raise_error(validation_error, 'File not found')
  end
end

I made a gist for this
https://gist.github.com/rokumatsumoto/e5f2e9ac027e2c8e2f88a3e6a5a5065d

Related repository
https://github.com/rokumatsumoto/boyutluseyler/tree/tests