SQL error on first re-import of eddblink

[ultimatespirit]

I think this may end up being a consistency issue from my still using the same database from a month or so ago so some interim updates messed it up. However, every time I run the eddblink import the first execution will fail with the following error. The only time it hasn't done that was when I updated again within 24 hours or so.

$ ./trade.py import -P eddblink
NOTE: Checking for update to 'System.csv'.
NOTE: Downloading file 'System.csv'.
NOTE: Requesting https://elite.tromador.com/files/System.csv
NOTE: Downloaded   2.3MB of gziped data   3.5MB/s
NOTE: Checking for update to 'Station.csv'.
NOTE: Downloading file 'Station.csv'.
NOTE: Requesting https://elite.tromador.com/files/Station.csv
NOTE: Downloaded  28.9MB of gziped data  14.3MB/s
NOTE: Checking for update to 'Item.csv'.
NOTE: Downloading file 'Item.csv'.
NOTE: Requesting https://elite.tromador.com/files/Item.csv
NOTE: Downloaded  19.1KB of gziped data   4.5MB/s
NOTE: Checking for update to 'Category.csv'.
NOTE: Downloading file 'Category.csv'.
NOTE: Requesting https://elite.tromador.com/files/Category.csv
NOTE: Downloaded 250.0B  of gziped data  78.1KB/s
NOTE: Rebuilding cache file: this may take a few moments.
Traceback (most recent call last):
  File ".../Trade-Dangerous/./trade.py", line 44, in <module>
    cli.main(sys.argv)
  File ".../Trade-Dangerous/tradedangerous/cli.py", line 66, in main
    trade(argv)
  File ".../Trade-Dangerous/tradedangerous/cli.py", line 121, in trade
    results = cmdenv.run(tdb)
              ^^^^^^^^^^^^^^^
  File ".../Trade-Dangerous/tradedangerous/commands/commandenv.py", line 84, in run
    return self._cmd.run(results, self, tdb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../Trade-Dangerous/tradedangerous/commands/import_cmd.py", line 124, in run
    if not plugin.run():
           ^^^^^^^^^^^^
  File ".../Trade-Dangerous/tradedangerous/plugins/eddblink_plug.py", line 469, in run
    self.tdb.reloadCache()
  File ".../Trade-Dangerous/tradedangerous/tradedb.py", line 700, in reloadCache
    cache.buildCache(self, self.tdenv)
  File ".../Trade-Dangerous/tradedangerous/cache.py", line 1015, in buildCache
    processPricesFile(tdenv, tempDB, pricesPath)
  File ".../Trade-Dangerous/tradedangerous/cache.py", line 656, in processPricesFile
    stations, items, zeros, newItems, updtItems, ignItems, numSys = processPrices(
                                                                    ^^^^^^^^^^^^^^
  File ".../Trade-Dangerous/tradedangerous/cache.py", line 611, in processPrices
    changeStation(matches)
  File ".../Trade-Dangerous/tradedangerous/cache.py", line 471, in changeStation
    inscur.execute("""
sqlite3.IntegrityError: NOT NULL constraint failed: Station.station_id

The second execution will skip that step I'm guessing and work fine, at least insofar as not hitting this integrity error.

[eyeonus]

I don't know what is causing your problem.
Please run with -wwwvvv and post the output of a run with the problem occurring to a gist.

Until this is fixed, you can use -O clean as a work around.

[nepomuk16321]

Wasn't that the problem with the moving fleet carriers?
The second run always worked for me.
I have just updated to 11.5.0. Everything is ok, runs perfectly.

[gerbilOFdoom]

I have some verbose debug output that identifies a special case of data that seems to break Trade Dangerous with this exact error.

#Selected station: MEROPE/ALCAZAR'S HOPE, ID=128760620
#Selected station: SOLITUDE/KNIGHT'S RETREAT, ID=128770348
#Selected station: RODENTIA PETRAM/ROCK RATS REACH, ID=128770604
#Selected station: HELGOLAND/STOERTEBEKER DOCK, ID=128770860
#Selected station: SANTOS DUMONT/BRAZILIAN DREAM, ID=128771116
#Selected station: SOVEREIGN'S REACH/HAROLD'S RESPITE, ID=128771372
#Selected station: KOJEARA/TOLAGARF'S JUNKYARD, ID=128771628
#Selected station: CALIFORNIA SECTOR JH-V C2-12/REDMARCH LABORATORY, ID=128771885
#Selected station: MAIA/MONI'S HUB, ID=128772653
#Selected station: PYTHEAS/HIPPARQUE - CARTOGRAPHERS' UNIVERSITY, ID=128773933
#Selected station: EOL PROCUL CENTAURI/FORT MUG, ID=128774189
#Selected station: AMATSUBOSHI/HAYABUSA LANDING, ID=128774445
#Selected station: HIP 17862/THE GNOSIS, ID=-1
#Key value: "EBOR/EBOR XENOBIOLOGICAL FACILITY"
data\TradeDangerous.prices:99510 ERROR Unrecognized STAR/Station: "HIP 17862/THE GNOSIS"

Checking INARA, The Gnosis has a permanent route through HIP 17862 but appears to be at Epsilon Indi at the time of this comment.

This suggests that the error will crop up around every Thursday.

Edit: Looks like it'll happen more often. Had this error come up yesterday or so:

#Selected station: LHS 184/D0K-6VK, ID=3700089600
#Selected station: HAMLET'S HARMONY/Q8M-14B, ID=-1
#Key value: "EBOR/EBOR XENOBIOLOGICAL FACILITY"
.\data\TradeDangerous.prices:3850161 ERROR Unrecognized STAR/Station: "HAMLET'S HARMONY/Q8M-14B"

Edit 2: Another one of these events, another moving station other than fleet carrier

.\tradedangerous\cache.p │
│ y:471 in changeStation                                                                           │
│                                                                                                  │
│    469 │   │   │   name = utils.titleFixup(stationName)                                          │
│    470 │   │   │   inscur = db.cursor()                                                          │
│ ❱  471 │   │   │   inscur.execute("""                                                            │
│    472 │   │   │   │   INSERT INTO Station (                                                     │
│    473 │   │   │   │   │   system_id, name,                                                      │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │           altName = None                                                                     │ │
│ │        altStation = None                                                                     │ │
│ │         corrected = True                                                                     │ │
│ │                db = <sqlite3.Connection object at 0x0000020D30D22D40>                        │ │
│ │           DELETED = -111                                                                     │ │
│ │          facility = 'TELIN/RESCUE SHIP HUTNER'                                               │ │
│ │     ignoreUnknown = True                                                                     │ │
│ │            inscur = <sqlite3.Cursor object at 0x0000020D7015BB40>                            │ │
│ │            lineNo = 153973                                                                   │ │
│ │          localAdd = 0                                                                        │ │
│ │           matches = <re.Match object; span=(0, 26), match='@ TELIN/Rescue Ship Hutner'>      │ │
│ │              name = 'Rescue Ship Hutner'                                                     │ │
│ │             newID = -1                                                                       │ │
│ │         priceFile = <_io.TextIOWrapper name='.\\data\\TradeDangerous.prices' │ │
│ │                     mode='r' encoding='utf-8'>                                               │ │
│ │    processedItems = {                                                                        │ │
│ │                     │   128673851: 153946,                                                   │ │
│ │                     │   128049166: 153947,                                                   │ │
│ │                     │   128049241: 153949,                                                   │ │
│ │                     │   128672314: 153950,                                                   │ │
│ │                     │   128673856: 153952,                                                   │ │
│ │                     │   128673855: 153953,                                                   │ │
│ │                     │   128673857: 153954,                                                   │ │
│ │                     │   128673858: 153955,                                                   │ │
│ │                     │   128049210: 153957,                                                   │ │
│ │                     │   128672811: 153959,                                                   │ │
│ │                     │   ... +9                                                               │ │
│ │                     }                                                                        │ │
│ │ processedStations = {                                                                        │ │
│ │                     │   128000000: 17,                                                       │ │
│ │                     │   128000512: 166,                                                      │ │
│ │                     │   128000768: 531,                                                      │ │
│ │                     │   128001536: 669,                                                      │ │
│ │                     │   128002048: 733,                                                      │ │
│ │                     │   128003584: 844,                                                      │ │
│ │                     │   128004352: 1209,                                                     │ │
│ │                     │   128004608: 1279,                                                     │ │
│ │                     │   128005376: 1343,                                                     │ │
│ │                     │   128005632: 1709,                                                     │ │
│ │                     │   ... +864                                                             │ │
│ │                     }                                                                        │ │
│ │  processedSystems = {                                                                        │ │
│ │                     │   'CEOS',                                                              │ │
│ │                     │   'PLEIADES SECTOR HR-W D1-57',                                        │ │
│ │                     │   'VIRACOCHA',                                                         │ │
│ │                     │   'ARTEMIS',                                                           │ │
│ │                     │   '55 LEONIS',                                                         │ │
│ │                     │   'HIP 4420',                                                          │ │
│ │                     │   'LAEDLA',                                                            │ │
│ │                     │   'HIP 18077',                                                         │ │
│ │                     │   'LFT 992',                                                           │ │
│ │                     │   'HAVEN',                                                             │ │
│ │                     │   ... +612                                                             │ │
│ │                     }                                                                        │ │
│ │     stationByName = {                                                                        │ │
│ │                     │   'PHUA FRAAE AA-A H0/T2X-02X': 3708716800,                            │ │
│ │                     │   'KYLOALL AA-A H0/L2Y-1VZ': 3706314496,                               │ │
│ │                     │   'HYPOAE AESCS AA-A H0/X4X-28J': 3703943424,                          │ │
│ │                     │   'BD+50 886/V2Q-BHY': 3702332416,                                     │ │
│ │                     │   'HERSCHEL 36/K4Z-8QY': 3704961024,                                   │ │
│ │                     │   'GREAT ANNIHILATOR/XNT-08K': 3700017408,                             │ │
│ │                     │   'GREAT ANNIHILATOR/KHK-96Z': 3700957440,                             │ │
│ │                     │   'GREAT ANNIHILATOR/K4K-T9Z': 3700980480,                             │ │
│ │                     │   'GREAT ANNIHILATOR/T3K-0VV': 3701015808,                             │ │
│ │                     │   'GREAT ANNIHILATOR/T9Z-WXT': 3701775104,                             │ │
│ │                     │   ... +293513                                                          │ │
│ │                     }                                                                        │ │
│ │         stationID = -111                                                                     │ │
│ │  stationItemDates = {}                                                                       │ │
│ │       stationName = 'RESCUE SHIP HUTNER'                                                     │ │
│ │     stationNameIn = 'Rescue Ship Hutner'                                                     │ │
│ │    stnCorrections = {}                                                                       │ │
│ │    sysCorrections = {                                                                        │ │
│ │                     │   'PANDAMONIUM': 'PANDEMONIUM',                                        │ │
│ │                     │   'ARGETLÁMH': 'ARGETLAMH',                                            │ │
│ │                     │   'LíFTHRUTI': 'LIFTHRUTI',                                            │ │
│ │                     │   'MANTóAC': 'MANTOAC',                                                │ │
│ │                     │   'NANTóAC': 'NANTOAC'                                                 │ │
│ │                     }                                                                        │ │
│ │      systemByName = {                                                                        │ │
│ │                     │   'PHUA FRAAE AA-A H0': 819519,                                        │ │
│ │                     │   'TRADGOI AA-A H0': 1866855,                                          │ │
│ │                     │   'KYLOALL AA-A H0': 2129183,                                          │ │
│ │                     │   'BD+55 191': 2194559,                                                │ │
│ │                     │   'HYPOAE AESCS AA-A H0': 2195559,                                     │ │
│ │                     │   'HD 191612': 2261159,                                                │ │
│ │                     │   'BD+50 886': 2325607,                                                │ │
│ │                     │   'HD 2083': 2392207,                                                  │ │
│ │                     │   'HD 34656': 2457687,                                                 │ │
│ │                     │   'X PERSEI': 2522255,                                                 │ │
│ │                     │   ... +45658                                                           │ │
│ │                     }                                                                        │ │
│ │          systemID = 2557820736186                                                            │ │
│ │        systemName = 'TELIN'                                                                  │ │
│ │      systemNameIn = 'TELIN'                                                                  │ │
│ │             tdenv = <tradedangerous.commands.commandenv.CommandEnv object at                 │ │
│ │                     0x0000020D33D6A1E0>                                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IntegrityError: NOT NULL constraint failed: Station.station_id