'Repo.insert' throw 'IndexError' sometimes #95

metacortex · 2017-05-31T15:39:23Z

Sometimes simple insert query result is empty array [],
though all data were inserted to table correctly.

Index out of bounds (IndexError)
0x10a325535: *CallStack::new:CallStack at ??
0x10a32848e: *raise<IndexError>:NoReturn at ??

https://github.com/Crecto/crecto/blob/master/src/crecto/repo.cr#L184

The error is coming from first.

query = config.adapter.run_on_instance(tx || config.get_connection, :insert, changeset)
...
new_instance = changeset.instance.class.from_rs(query.as(DB::ResultSet)).first

I couldn't find any pattern, it looks like randomly occur.

mysql-5.6

The text was updated successfully, but these errors were encountered:

metacortex · 2017-05-31T16:16:15Z

#96

#95 use 'first?' to prevent 'IndexError' when insert/update

fridgerator · 2017-06-01T02:02:48Z

fixed by #96

onemanstartup · 2017-10-13T05:12:22Z

@fridgerator I'm still experience IndexError on master. And "Unexpected byte 0xff in UTF-8 byte sequence" as well. I don't know how to debug this, I'm new in Crystal.

fridgerator · 2017-10-13T12:59:06Z

You get the unexpected byte sequence only on insert statements with mysql?

onemanstartup · 2017-10-13T21:29:38Z

@fridgerator I'm experience this with postgresql. I'm parsing json and then insert changeset.
Only 3 records will be saved out of 6, it will stuck after Unexpected byte error.
And DB log will be only the first record.

 1.77ms INSERT INTO tickers (market, currency, ticker, price, bid, ask, low_24h, high_24h, avg_24h, volume_24h, volume_30d) VALUES ('coinone', 'bch', 'KRW', '360500.0', '', '', '350100.0', '369500.0', '', '34075.8113', '') RETURNING *

Very confusing. I'm already successfully record other records.

With insert

#<Crecto::Changeset::Changeset(Ticker):0x10c569000 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "bch", :ticker => "KRW", :price => 360500.0, :bid => nil, :ask => nil, :low_24h => 350100.0, :high_24h => 369500.0, :avg_24h => nil, :volume_24h => 34075.1407, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "bch"}, {:ticker => "KRW"}, {:price => 360500.0}, {:low_24h => 350100.0}, {:high_24h => 369500.0}, {:volume_24h => 34075.1407}], @valid=true, @instance=#<Ticker:0x10ee11f20 @initial_values=nil, @market="coinone", @currency="bch", @ticker="KRW", @price=360500.0, @bid=nil, @ask=nil, @low_24h=350100.0, @high_24h=369500.0, @avg_24h=nil, @volume_24h=34075.1407, @volume_30d=nil>, @initial_values={}>
insert error - Index out of bounds
#<Crecto::Changeset::Changeset(Ticker):0x10ee23900 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "qtum", :ticker => "KRW", :price => 12490.0, :bid => nil, :ask => nil, :low_24h => 11990.0, :high_24h => 13440.0, :avg_24h => nil, :volume_24h => 1920171.575, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "qtum"}, {:ticker => "KRW"}, {:price => 12490.0}, {:low_24h => 11990.0}, {:high_24h => 13440.0}, {:volume_24h => 1920171.575}], @valid=true, @instance=#<Ticker:0x10ee11dc0 @initial_values=nil, @market="coinone", @currency="qtum", @ticker="KRW", @price=12490.0, @bid=nil, @ask=nil, @low_24h=11990.0, @high_24h=13440.0, @avg_24h=nil, @volume_24h=1920171.575, @volume_30d=nil>, @initial_values={}>
insert error - Unexpected byte 0xff in UTF-8 byte sequence
#<Crecto::Changeset::Changeset(Ticker):0x10ee294e0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "etc", :ticker => "KRW", :price => 13910.0, :bid => nil, :ask => nil, :low_24h => 13260.0, :high_24h => 14170.0, :avg_24h => nil, :volume_24h => 1039445.4814, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "etc"}, {:ticker => "KRW"}, {:price => 13910.0}, {:low_24h => 13260.0}, {:high_24h => 14170.0}, {:volume_24h => 1039445.4814}], @valid=true, @instance=#<Ticker:0x10ee11d10 @initial_values=nil, @market="coinone", @currency="etc", @ticker="KRW", @price=13910.0, @bid=nil, @ask=nil, @low_24h=13260.0, @high_24h=14170.0, @avg_24h=nil, @volume_24h=1039445.4814, @volume_30d=nil>, @initial_values={}>

Without insert

#<Crecto::Changeset::Changeset(Ticker):0x10e76b000 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "bch", :ticker => "KRW", :price => 360500.0, :bid => nil, :ask => nil, :low_24h => 350100.0, :high_24h => 369500.0, :avg_24h => nil, :volume_24h => 34075.1407, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "bch"}, {:ticker => "KRW"}, {:price => 360500.0}, {:low_24h => 350100.0}, {:high_24h => 369500.0}, {:volume_24h => 34075.1407}], @valid=true, @instance=#<Ticker:0x111013f20 @initial_values=nil, @market="coinone", @currency="bch", @ticker="KRW", @price=360500.0, @bid=nil, @ask=nil, @low_24h=350100.0, @high_24h=369500.0, @avg_24h=nil, @volume_24h=34075.1407, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x111014900 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "qtum", :ticker => "KRW", :price => 12490.0, :bid => nil, :ask => nil, :low_24h => 11990.0, :high_24h => 13440.0, :avg_24h => nil, :volume_24h => 1920056.2903, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "qtum"}, {:ticker => "KRW"}, {:price => 12490.0}, {:low_24h => 11990.0}, {:high_24h => 13440.0}, {:volume_24h => 1920056.2903}], @valid=true, @instance=#<Ticker:0x111013e70 @initial_values=nil, @market="coinone", @currency="qtum", @ticker="KRW", @price=12490.0, @bid=nil, @ask=nil, @low_24h=11990.0, @high_24h=13440.0, @avg_24h=nil, @volume_24h=1920056.2903, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x1110142a0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "etc", :ticker => "KRW", :price => 13910.0, :bid => nil, :ask => nil, :low_24h => 13260.0, :high_24h => 14170.0, :avg_24h => nil, :volume_24h => 1039445.4814, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "etc"}, {:ticker => "KRW"}, {:price => 13910.0}, {:low_24h => 13260.0}, {:high_24h => 14170.0}, {:volume_24h => 1039445.4814}], @valid=true, @instance=#<Ticker:0x111013dc0 @initial_values=nil, @market="coinone", @currency="etc", @ticker="KRW", @price=13910.0, @bid=nil, @ask=nil, @low_24h=13260.0, @high_24h=14170.0, @avg_24h=nil, @volume_24h=1039445.4814, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x111018c00 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "btc", :ticker => "KRW", :price => 6445000.0, :bid => nil, :ask => nil, :low_24h => 6080500.0, :high_24h => 6989500.0, :avg_24h => nil, :volume_24h => 18770.8274, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "btc"}, {:ticker => "KRW"}, {:price => 6445000.0}, {:low_24h => 6080500.0}, {:high_24h => 6989500.0}, {:volume_24h => 18770.8274}], @valid=true, @instance=#<Ticker:0x111013d10 @initial_values=nil, @market="coinone", @currency="btc", @ticker="KRW", @price=6445000.0, @bid=nil, @ask=nil, @low_24h=6080500.0, @high_24h=6989500.0, @avg_24h=nil, @volume_24h=18770.8274, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x1110185a0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "eth", :ticker => "KRW", :price => 386950.0, :bid => nil, :ask => nil, :low_24h => 349250.0, :high_24h => 395900.0, :avg_24h => nil, :volume_24h => 282213.7917, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "eth"}, {:ticker => "KRW"}, {:price => 386950.0}, {:low_24h => 349250.0}, {:high_24h => 395900.0}, {:volume_24h => 282213.7917}], @valid=true, @instance=#<Ticker:0x111013c60 @initial_values=nil, @market="coinone", @currency="eth", @ticker="KRW", @price=386950.0, @bid=nil, @ask=nil, @low_24h=349250.0, @high_24h=395900.0, @avg_24h=nil, @volume_24h=282213.7917, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x11101cf00 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "xrp", :ticker => "KRW", :price => 294.0, :bid => nil, :ask => nil, :low_24h => 271.0, :high_24h => 298.0, :avg_24h => nil, :volume_24h => 170112858.829, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "xrp"}, {:ticker => "KRW"}, {:price => 294.0}, {:low_24h => 271.0}, {:high_24h => 298.0}, {:volume_24h => 170112858.829}], @valid=true, @instance=#<Ticker:0x111013bb0 @initial_values=nil, @market="coinone", @currency="xrp", @ticker="KRW", @price=294.0, @bid=nil, @ask=nil, @low_24h=271.0, @high_24h=298.0, @avg_24h=nil, @volume_24h=170112858.829, @volume_30d=nil>, @initial_values={}>

I found that if I replace nil values with values then all 6 will be recorded. But rescue will print out
PG::ResultSet#read returned a PG::Numeric. A (Float64 | Nil) was expected.

                begin
                  Repo.insert(changeset)
                rescue ex
                  puts "insert error - #{ex}"
                end

fridgerator · 2017-10-17T04:45:37Z

@onemanstartup are you able to share your schema and json that you are parsing? I tried replicating locally and couldn't

onemanstartup · 2017-10-17T14:23:39Z

@fridgerator This is the code. I don't understand why records persists but throw errors.. strange..

class Ticker < Crecto::Model
  schema "tickers", primary_key: false do
    field :market, String
    field :currency, String
    field :ticker, String
    field :price, Float64
    field :bid, Float64
    field :ask, Float64
    field :low_24h, Float64
    field :high_24h, Float64
    field :avg_24h, Float64
    field :volume_24h, Float64
    field :volume_30d, Float64

    set_created_at_field nil
    set_updated_at_field nil
  end

  validate_required [:market]
  validate_format :market, /^[a-zA-Z]*$/
end

CREATE TABLE tickers (
  time       TIMESTAMPTZ       NOT NULL DEFAULT now(),
  market     TEXT              NOT NULL,
  currency   TEXT              NOT NULL,
  ticker     TEXT              NOT NULL,
  price      numeric           NOT NULL,
  bid        numeric           NULL,
  ask        numeric           NULL,
  low_24h    numeric           NULL,
  high_24h   numeric           NULL,
  avg_24h    numeric           NULL,
  volume_24h numeric           NULL,
  volume_30d numeric           NULL
);

{"bch":{"volume":"214659.4839","last":"418000","yesterday_last":"352400","yesterday_low":"348000","high":"451500","currency":"bch","low":"349700","yesterday_first":"353500","yesterday_volume":"12689.9568","yesterday_high":"360300","first":"351400"},"qtum":{"volume":"2417040.9742","last":"12830","yesterday_last":"11960","yesterday_low":"11790","high":"13200","currency":"qtum","low":"11950","yesterday_first":"12060","yesterday_volume":"657970.4149","yesterday_high":"12360","first":"11980"},"errorCode":"0","etc":{"volume":"983336.4399","last":"13700","yesterday_last":"13430","yesterday_low":"13230","high":"14050","currency":"etc","low":"13260","yesterday_first":"13470","yesterday_volume":"477616.7548","yesterday_high":"13690","first":"13440"},"result":"success","btc":{"volume":"4240.6907","last":"6372000","yesterday_last":"6347000","yesterday_low":"6200000","high":"6486500","currency":"btc","low":"6300000","yesterday_first":"6332500","yesterday_volume":"5541.9419","yesterday_high":"6499500","first":"6375000"},"timestamp":"1508249874","eth":{"volume":"83368.7347","last":"367100","yesterday_last":"380400","yesterday_low":"368500","high":"383000","currency":"eth","low":"365000","yesterday_first":"375000","yesterday_volume":"165312.9340","yesterday_high":"394900","first":"381400"},"xrp":{"volume":"306820375.5870","last":"288","yesterday_last":"319","yesterday_low":"293","high":"323","currency":"xrp","low":"278","yesterday_first":"296","yesterday_volume":"334760612.5759","yesterday_high":"327","first":"323"}}

            JSON.parse(response.body).each do |k, v|
              next if k == "errorCode" || k == "result" || k == "timestamp"
              ticker = Mapping::Ticker.from_json(v.to_json)
              new_ticker            = Ticker.new
              new_ticker.market     = @market
              new_ticker.currency   = k.to_s
              new_ticker.ticker     = "KRW"
              new_ticker.price      = ticker.last.to_f
              new_ticker.low_24h    = ticker.low.to_f
              new_ticker.high_24h   = ticker.high.to_f
              new_ticker.volume_24h = ticker.volume.to_f
              new_ticker.avg_24h = 0.to_f
              new_ticker.volume_30d = 0.to_f
              new_ticker.bid = 0.to_f
              new_ticker.ask = 0.to_f
              changeset = Ticker.changeset(new_ticker)
              if changeset.valid?
                begin
                  Repo.insert(changeset)
                rescue ex
                  puts "insert error - #{ex}"
                end
              else
                puts changeset.errors
              end
            end

fridgerator · 2017-10-18T00:41:01Z

If you change the numeric fields to float it works

onemanstartup · 2017-10-19T07:54:50Z

@fridgerator I tried set types to PG::Numeric in Crecto::Model, but it's doesn't work. It seems either I don't understand how casting works in pg shard or crystal lacks decimal field leads to this. Anyway thanks for helping out, will save in floats for now, until figure out how this works.

fridgerator · 2017-10-19T12:15:54Z

yeah you'll have to use float right now, PG::Numeric wont work in the model right now. I'll add an issue to support this type. Thanks!

timkendall · 2018-02-23T16:50:39Z

I'm still getting this error with Crecto 0.8.4. Here's my Postgres schema:

CREATE TABLE foods(
  id SERIAL PRIMARY KEY,
  upc VARCHAR,
  usda_db_id VARCHAR,
  name VARCHAR NOT NULL,
  
  description VARCHAR,
  
  protein  FLOAT,
  fat  FLOAT,
  carbohydrates FLOAT,

  serving_unit VARCHAR,
  serving_label VARCHAR,
  serving_value INTEGER,

  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

timkendall · 2018-02-24T18:35:09Z

Looks like it was bad permissions settings on my part. Solved by making my dev postgres user a superuser with:

ALTER USER postgres WITH SUPERUSER;

timkendall · 2018-02-24T18:51:06Z

Nevermind, still get the IndexError when using Repo.get_by/2 or Repo.insert/1

fridgerator · 2018-02-28T00:00:20Z

@timkendall Does this happen all the time or randomly? It would be great to get a log of the sql generated

timkendall · 2018-03-02T22:13:08Z

All of the time - I'll add that when I get back to my personal computer.

fridgerator · 2018-03-13T12:34:16Z

@timkendall Creating a table with withe sql you posted, the following is working for me (crystal 0.24.2, crecto master branch)

require "pg"
require "./src/crecto"

Crecto::DbLogger.set_handler(STDOUT)

module Repo
  extend Crecto::Repo

  config do |conf|
    conf.adapter = Crecto::Adapters::Postgres
    conf.database = "issue_95"
    conf.hostname = "localhost"
    conf.port = 5432
  end
end

class Food < Crecto::Model
  schema "foods" do
    field :upc, String
    field :usda_db_id, String
    field :name, String
    field :description, String
    field :protein, Float64
    field :fat, Float64
    field :carbohydrates, Float64
    field :serving_unit, String
    field :serving_label, String
    field :serving_value, Int32
  end
end

food = Food.new
food.upc = "123"
food.usda_db_id = "456"
food.name = "hey"
food.description = "description"
food.protein = 342.222
food.fat = 123.234
food.carbohydrates = 54.234
food.serving_unit = "g"
food.serving_label = "good food"
food.serving_value = 123

changeset = Repo.insert(food)
puts Repo.get_by(Food, id: changeset.instance.id)

watzon · 2023-12-22T18:20:03Z

@timkendall did that fix end up working for you? Or can you let me know if you're still getting this error? I know this is an old issue.

fridgerator added a commit that referenced this issue Jun 1, 2017

Merge pull request #96 from metacortex/master

42dca74

#95 use 'first?' to prevent 'IndexError' when insert/update

fridgerator closed this as completed Jun 1, 2017

fridgerator reopened this Feb 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'Repo.insert' throw 'IndexError' sometimes #95

'Repo.insert' throw 'IndexError' sometimes #95

metacortex commented May 31, 2017

metacortex commented May 31, 2017

fridgerator commented Jun 1, 2017

onemanstartup commented Oct 13, 2017

fridgerator commented Oct 13, 2017

onemanstartup commented Oct 13, 2017 •

edited

Loading

fridgerator commented Oct 17, 2017

onemanstartup commented Oct 17, 2017 •

edited

Loading

fridgerator commented Oct 18, 2017

onemanstartup commented Oct 19, 2017

fridgerator commented Oct 19, 2017

timkendall commented Feb 23, 2018 •

edited

Loading

timkendall commented Feb 24, 2018 •

edited

Loading

timkendall commented Feb 24, 2018

fridgerator commented Feb 28, 2018

timkendall commented Mar 2, 2018

fridgerator commented Mar 13, 2018

watzon commented Dec 22, 2023

'Repo.insert' throw 'IndexError' sometimes #95

'Repo.insert' throw 'IndexError' sometimes #95

Comments

metacortex commented May 31, 2017

metacortex commented May 31, 2017

fridgerator commented Jun 1, 2017

onemanstartup commented Oct 13, 2017

fridgerator commented Oct 13, 2017

onemanstartup commented Oct 13, 2017 • edited Loading

fridgerator commented Oct 17, 2017

onemanstartup commented Oct 17, 2017 • edited Loading

fridgerator commented Oct 18, 2017

onemanstartup commented Oct 19, 2017

fridgerator commented Oct 19, 2017

timkendall commented Feb 23, 2018 • edited Loading

timkendall commented Feb 24, 2018 • edited Loading

timkendall commented Feb 24, 2018

fridgerator commented Feb 28, 2018

timkendall commented Mar 2, 2018

fridgerator commented Mar 13, 2018

watzon commented Dec 22, 2023

onemanstartup commented Oct 13, 2017 •

edited

Loading

onemanstartup commented Oct 17, 2017 •

edited

Loading

timkendall commented Feb 23, 2018 •

edited

Loading

timkendall commented Feb 24, 2018 •

edited

Loading