proj, postgres, spring, fork, Big Sur faulting

[mjy]

Spamming keywords to others might find this. I realize this this is spring centric, but there is high potential for some pg element to it, see below. This issue is, maybe, alluded to in various other tickets in various other places, none of which seem to come up with a concrete answer, crossreferencing:

• Mac fork() fatal error rails/spring#606
• Spring taking minutes to run on MacOS with M1 Chip rails/spring#635
• https://bugs.ruby-lang.org/issues/14009
• "Internal Proj Error: [...] database disk image is malformed" when multiprocessing since pyproj 2.3 pyproj4/pyproj#426
• Using GDAL in forking multiprocessing environment causes SQLite failure OSGeo/gdal#2221

When we run spring then we get faults on both M1 and Intel under Big Sur.

• The issue has existed for months
• Anecdotally this is related to some fork, somewhere
• Most people who seem to hit this have a postgres related element in their env
• This may be related to new mac binary protections, one of the errors leads with proj_create_operations: SQLite error on SELECT allowed_authorities FROM authority_to_authority_preference WHERE source_auth_name = ? AND target_auth_name = ?: disk I/O error
• Replicated on numerous machines, with clean installs of many of the potential culprits.
• When running the same command, 1 of 2 errors, apparently completley at random (fork?) are raised:

1

matt@Matts-MacBook-Pro taxonworks (dwc_fields *$%)$ spring rspec spec/models/otu_spec.rb 
Running via Spring preloader in process 3654
proj_create_operations: SQLite error on SELECT allowed_authorities FROM authority_to_authority_preference WHERE source_auth_name = ? AND target_auth_name = ?: disk I/O error

An error occurred while loading ./spec/models/otu_spec.rb.
Failure/Error:
  ILLINOIS = RSPEC_GEO_FACTORY.parse_wkt(
    'MULTIPOLYGON (((-88.02362060546875 38.08888244628906 0.0, ' \
      '-87.99749755859375 38.091102600097656 0.0, -87.98167419433594 38.091102600097656 0.0, ' \
      ' -87.975830078125 38.08971405029297 0.0, -87.97193908691406 38.08610534667969 0.0, ' \
      ' -87.96945190429688 38.078880310058594 0.0, -87.9727783203125 38.074440002441406 0.0, ' \
      ' -87.97721862792969 38.06999206542969 0.0, -87.9869384765625 38.06471252441406 0.0, ' \
      ' -88.03916931152344 38.03777313232422 0.0, -88.03388977050781 37.97221374511719 0.0, ' \
      ' -88.03443908691406 37.903045654296875 0.0, -88.03611755371094 37.89833068847656 0.0, ' \
      ' -88.040283203125 37.89471435546875 0.0, -88.05110168457031 37.888885498046875 0.0, ' \
      ' -88.05804443359375 37.888885498046875 0.0, -88.06388854980469 37.889991760253906 0.0, ' \

NoMethodError:
  undefined method `size' for nil:NilClass
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/feature/types.rb:206:in `cast'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/proj4_projector.rb:23:in `project'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/factory.rb:247:in `project'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/projected_feature_methods.rb:17:in `projection'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/projected_feature_methods.rb:39:in `equals?'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/feature/geometry.rb:600:in `=='
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/impl_helper/basic_line_string_methods.rb:184:in `!='
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/impl_helper/basic_line_string_methods.rb:184:in `validate_geometry'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/projected_feature_methods.rb:165:in `validate_geometry'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/impl_helper/basic_line_string_methods.rb:19:in `initialize'
# /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/factory.rb:338:in `new'

2

matt@Matts-MacBook-Pro ~$ tw
matt@Matts-MacBook-Pro taxonworks (dwc_fields *$%)$ spring rspec spec/models/otu_spec.rb 
DEPRECATION WARNING: ActionView::Base instances should be constructed with a lookup context, assignments, and a controller. (called from <top (required)> at /Users/matt/data/src/github/species_file_group/taxonworks/config/environment.rb:5)
Running via Spring preloader in process 2842
/Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-proj4-3.0.1/lib/rgeo/coord_sys/proj4.rb:80: [BUG] Bus Error at 0x000000010b6b7ad4
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-darwin20]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:                    
     * ~/Library/Logs/DiagnosticReports                                     
     * /Library/Logs/DiagnosticReports                                      
   for more details.                                                        
Don't forget to include the above Crash Report log file in bug reports.     

-- Control frame information -----------------------------------------------
c:0065 p:---- s:0358 e:000357 CFUNC  :_canonical_str
c:0064 p:0010 s:0354 e:000353 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-proj4-3.0.1/lib/rgeo/coord_sys/proj4.rb:80
c:0063 p:0015 s:0350 e:000349 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-proj4-3.0.1/lib/rgeo/coord_sys/proj4.rb:93
c:0062 p:0017 s:0346 e:000345 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-proj4-3.0.1/lib/rgeo/coord_sys/proj4.rb:45
c:0061 p:0077 s:0341 e:000340 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/factory.rb:87
c:0060 p:0136 s:0336 e:000335 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/feature/types.rb:193
c:0059 p:0029 s:0316 e:000315 BLOCK  /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/impl_helper/basic_line_string_methods.rb:15 [FINISH]
c:0058 p:---- s:0312 e:000311 CFUNC  :map
c:0057 p:0011 s:0308 e:000307 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/impl_helper/basic_line_string_methods.rb:14 [FINISH]
c:0056 p:---- s:0302 e:000301 CFUNC  :new
c:0055 p:0008 s:0296 e:000295 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/factory.rb:326
c:0054 p:0041 s:0291 e:000290 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/wkrep/wkt_parser.rb:287
c:0053 p:0037 s:0286 e:000285 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/wkrep/wkt_parser.rb:297
c:0052 p:0006 s:0280 e:000278 BLOCK  /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/wkrep/wkt_parser.rb:369 [FINISH]
c:0051 p:---- s:0276 e:000275 CFUNC  :loop
c:0050 p:0028 s:0272 e:000271 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/wkrep/wkt_parser.rb:368
c:0049 p:0416 s:0267 e:000266 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/wkrep/wkt_parser.rb:214
c:0048 p:0111 s:0258 e:000257 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/wkrep/wkt_parser.rb:133
c:0047 p:0007 s:0252 e:000251 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/rgeo-2.3.0/lib/rgeo/geographic/factory.rb:308
c:0046 p:0013 s:0247 e:000246 CLASS  /Users/matt/data/src/github/species_file_group/taxonworks/spec/support/shared_contexts/geo/illinois.rb:3
c:0045 p:0007 s:0244 e:000243 TOP    /Users/matt/data/src/github/species_file_group/taxonworks/spec/support/shared_contexts/geo/illinois.rb:1 [FINISH]
c:0044 p:---- s:0241 e:000240 CFUNC  :require
c:0043 p:0011 s:0236 e:000235 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/polyglot-0.3.5/lib/polyglot.rb:65
c:0042 p:0007 s:0228 e:000227 BLOCK  /Users/matt/.rvm/gems/ruby-2.7.4/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:324
c:0041 p:0068 s:0225 e:000224 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:291
c:0040 p:0010 s:0218 e:000217 METHOD /Users/matt/.rvm/gems/ruby-2.7.4/gems/activesupport-6.0.4.1/lib/active_support/dependencies.rb:324
c:0039 p:0006 s:0212 e:000211 BLOCK  /Users/matt/data/src/github/species_file_group/taxonworks/spec/rails_helper.rb:24 [FINISH]
c:0038 p:---- s:0208 e:000207 CFUNC  :each

[mjy]

We have discovered that if we exercise our Factory early in the test process, then we can at least run tests using spring, though we can various get raises after they complete:

#/config/initializers/gis.rb

module Gis

  SPHEROID = 'SPHEROID["WGS-84", 6378137, 298.257223563]'.freeze

  FACTORY = RGeo::Geographic.projected_factory(srid:                    4326,
                                               projection_srid:         4326,
                                               projection_proj4:        "EPSG:4326",
                                               uses_lenient_assertions: true,
                                               has_z_coordinate:        true,
                                               wkb_parser:              {support_ewkb: true},
                                               wkb_generator:           {hex_format: true, emit_ewkb_srid: true})

  # Warm up RGeo/Proj before any forking may occur (solves some problems with spring on Mac)
  FACTORY.parse_wkt('POLYGON ((0 0, 1 0, 1 1, 0 1, 0 0))')
end

[mjy]

Perhaps final followup.

Following this:

• MAC M1 Big Sur ruby compile issue for all rubies. Error running '__rvm_make -j8', rvm/rvm#5033 (comment) (i.e. completely clean start under Rosetta terminal)

Then:

• brew install postgis
• brew install proj
• bundle pristine

And when Factory is warmed up as above,

only then spring calls pass without a (fork?) issue. [Edit, not all all are removed]

[keithdoggett]

Thanks for the detailed write-up. I don't know how much I'll be able to help as more info comes out about the root cause since I don't have access to a system with an M1 chip.

If there's anything I can do or if you want to ask questions/discuss something about how the library is set up that might be causing it I'd be happy to help.

[mjy]

Thanks for the offer. Thought we had things resolved but we're still getting raise errors on rails 6.1 in some scenarios. If we can leave this open for a bit I'll add further insights if/when we track them down. There is an outside chance it's related to table truncation/forking, but not worth sending people off on wild goose chases till we can nail things down further.