Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add google storage support for remote uploads #326

Merged
merged 3 commits into from
Dec 26, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,6 +374,27 @@ directory.

Some documentation exists [on the wiki page][remote_hosts].

* `SitemapGenerator::GoogleStorageAdapter`

Uses `Google::Cloud::Storage` to upload to Google Cloud storage.

You must `require 'google-cloud-storage'` in your sitemap config before using this adapter.

An example of using this adapter in your sitemap configuration with options:

```ruby
SitemapGenerator::Sitemap.adapter = SitemapGenerator::GoogleStorageAdapter.new(
keyfile: 'path/to/keyfile.json',
project_id: 'google_account_project_id',
bucket: 'name_of_bucket'
)
```
Also, inline with Google Authentication options, it can also pick credentials from environment variables. Variables required to be set for proper authentication with google are `GOOGLE_CLOUD_PROJECT` and `GOOGLE_APPLICATION_CREDENTIALS`. An example of using this adapter with the environment variables is:

```ruby
SitemapGenerator::Sitemap.adapter = SitemapGenerator::GoogleStorageAdapter.new( bucket: 'name_of_bucket' )
```

#### An Example of Using an Adapter

1. Please see [this wiki page][remote_hosts] for more information about setting up SitemapGenerator to upload to a
Expand Down
17 changes: 9 additions & 8 deletions lib/sitemap_generator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@
require 'sitemap_generator/sitemap_location'

module SitemapGenerator
autoload(:Interpreter, 'sitemap_generator/interpreter')
autoload(:FileAdapter, 'sitemap_generator/adapters/file_adapter')
autoload(:S3Adapter, 'sitemap_generator/adapters/s3_adapter')
autoload(:AwsSdkAdapter, 'sitemap_generator/adapters/aws_sdk_adapter')
autoload(:WaveAdapter, 'sitemap_generator/adapters/wave_adapter')
autoload(:FogAdapter, 'sitemap_generator/adapters/fog_adapter')
autoload(:BigDecimal, 'sitemap_generator/core_ext/big_decimal')
autoload(:Numeric, 'sitemap_generator/core_ext/numeric')
autoload(:Interpreter, 'sitemap_generator/interpreter')
autoload(:FileAdapter, 'sitemap_generator/adapters/file_adapter')
autoload(:S3Adapter, 'sitemap_generator/adapters/s3_adapter')
autoload(:AwsSdkAdapter, 'sitemap_generator/adapters/aws_sdk_adapter')
autoload(:WaveAdapter, 'sitemap_generator/adapters/wave_adapter')
autoload(:FogAdapter, 'sitemap_generator/adapters/fog_adapter')
autoload(:GoogleStorageAdapter, 'sitemap_generator/adapters/google_storage_adapter')
autoload(:BigDecimal, 'sitemap_generator/core_ext/big_decimal')
autoload(:Numeric, 'sitemap_generator/core_ext/numeric')

SitemapError = Class.new(StandardError)
SitemapFullError = Class.new(SitemapError)
Expand Down
32 changes: 32 additions & 0 deletions lib/sitemap_generator/adapters/google_storage_adapter.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
if !defined?(Google::Cloud::Storage)
raise "Error: `Google::Cloud::Storage` is not defined.\n\n"\
"Please `require 'google/cloud/storage'` - or another library that defines this class."
end

module SitemapGenerator
# Class for uploading sitemaps to a Google Storage using google-cloud-storage gem.
class GoogleStorageAdapter
# Requires Google::Cloud::Storage to be defined.
#
# Options:
# :credentials [String] Path to the google service account keyfile.json
# :project_id [String] Google Accounts project_id where the storage bucket resides
# :bucket [String] Name of Google Storage Bucket where the file is to be uploaded

# @param [Hash] opts Google::Cloud::Storage configuration options
def initialize(opts = {})
@credentials = opts[:keyfile] || ENV['GOOGLE_CLOUD_PROJECT']
@project_id = opts[:project_id] || ENV['GOOGLE_APPLICATION_CREDENTIALS']
@bucket = opts[:bucket]
end

# Call with a SitemapLocation and string data
def write(location, raw_data)
SitemapGenerator::FileAdapter.new.write(location, raw_data)

storage = Google::Cloud::Storage.new(project_id: @project_id, credentials: @credentials)
bucket = storage.bucket @bucket
bucket.create_file location.path, location.path_in_public, acl: 'public'
end
end
end
1 change: 1 addition & 0 deletions sitemap_generator.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,6 @@ Gem::Specification.new do |s|
s.add_development_dependency 'rake'
s.add_development_dependency 'aws-sdk-core'
s.add_development_dependency 'aws-sdk-s3'
s.add_development_dependency 'google-cloud-storage'
s.files = Dir.glob('{lib,rails,templates}/**/*') + %w(CHANGES.md MIT-LICENSE README.md VERSION)
end
25 changes: 25 additions & 0 deletions spec/sitemap_generator/adapters/google_storage_adapter_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# encoding: UTF-8
require 'spec_helper'
require 'google/cloud/storage'

describe SitemapGenerator::GoogleStorageAdapter do
let(:location) { SitemapGenerator::SitemapLocation.new }
let(:options) { {:credentials=>nil, :project_id=>nil} }
let(:google_bucket) { 'bucket' }
let(:adapter) { SitemapGenerator::GoogleStorageAdapter.new(options.merge(bucket: google_bucket)) }

describe 'write' do
it 'it writes the raw data to a file and then uploads that file to Google Storage' do
bucket = double(:bucket)
storage = double(:storage)
bucket_resource = double(:bucket_resource)
expect(Google::Cloud::Storage).to receive(:new).with(options).and_return(storage)
expect(storage).to receive(:bucket).with('bucket').and_return(bucket_resource)
expect(location).to receive(:path_in_public).and_return('path_in_public')
expect(location).to receive(:path).and_return('path')
expect(bucket_resource).to receive(:create_file).with('path', 'path_in_public', acl: 'public').and_return(nil)
expect_any_instance_of(SitemapGenerator::FileAdapter).to receive(:write).with(location, 'raw_data')
adapter.write(location, 'raw_data')
end
end
end