Change Fog's development configuration and add begin rescue to the importer.

[abuiles]

Hi Guys,

Turns out that the error reported in issue #293 is mostly given because a configuration issue.
The following are the things that I found out:

1. Vault#directory was returning nil, because server/gems/development.s3.rubygems.org didn't exit. (nothing new until here )

2. Creating development.s3.rubygems.org under /gems ( see Fix Issue #293 #294 ) apparently was fixing the issue, after all It wouldn't throw the error we were getting in 1, and indeed when I checked, the gems were imported under server/gems/development.s3.rubygems.org/gems/

3. Here's when I noticed that something was wrong, because even when I had the gems and they were displaying in the application running in localhost, each time I clicked download, I was getting nothing, I checked the log and there was a get to https://localhost:3000/downloads/test-gem.gem, which lead me to hostess.rb, where I noticed (line 10)

   send_file(Pusher.server_path(request.path_info))
   

Which led me to believe that by default in development, we were expecting the gems to be under /gems/gem-name.gem

So that's were I noticed that in order to get it working like that I would need to change fog's local_root to Pusher.server_path, and then use the same local_root as the directory returned by "directory".

After doing that, I cleaned /server/ and run again the importer, this time it put all the gems under server/gems and then clicking download in the webpage was working.

As a side note, we still need to use the begin rescue clause in the importer, since for some gems we are getting:

     "\x9C" from ASCII-8BIT to UTF-8

Hopefully this fix #293 and #292.

[sferik]

This looks good to me, but I'm not very familiar with this portion of the codebase. @qrush or @cldwalker, would you mind giving this a second set of eyes and merging if everything looks okay to you.

[abuiles]

@sferik How's your workflow when working locally? Do you push gems to your localhost or do you use amazon? I kind of feel a bit meehh about:

    s3_bucket: .

but having

   s3_bucket: development.s3.rubygems.org

gives troubles too ;(. cc @qrush @cldwalker

[qrush]

holy crap, this needs to be rebased cleanly. no idea what this patch is for or does.

[qrush]

Is this the same issue as #294?

[abuiles]

@qrush yeap it is the same as #294, but if you check my comments I explained why #294 wasn't "really" solving the issue.

Bad news is that I now fuck my commits too ;(. I'll fix it again. One question though, when you are in development mode do you use S3 (development.s3.rubygems.org) or do you store the gems locally ?

The problem is mostly related with where are the gems expected to be located in dev-mode.

See https://github.com/rubygems/rubygems.org/blob/master/app/middleware/hostess.rb#L10

  send_file(Pusher.server_path(request.path_info))

That search for /gems/gem-name.gem, but if development.s3.rubygems.org is used, the gems are stored under server/gems/development.s3.rubygems.org.

[elskwid]

@abuiles, I know this is kind of old but these issues have been addressed in #494.