Merged kfsone/tradedangerous into master

CHANGES.txt

@@ -2,6 +2,74 @@
 TradeDangerous, Copyright (C) Oliver "kfsone" Smith, July 2014
 ==============================================================================
 
+v6.13.3 Mar 11 2015
+. (kfsone) Fixed "--blackmarket" and "--trading" in local,
+. (Dry411s) Resolves #200 New ships for 1.2,
+. (kfsone) +103 Systems
+. (kfsone) OCR Derp fixes
+. (kfsone) Station cleanup
+
+v6.13.2 Mar 10 2015
+. (Tromador) Added "Painite" to Item.csv
+. (kfsone) import plugin now allows you to separate --opt values with ,
+   e.g. trade.py import --plug=maddavo --opt=csvs,csvonly
+   will import mad's csv files without importing his prices
+. (kfsone) 112 new systems
+
+v6.13.1 Mar 06 2015
+. (kfsone) "shipvendor" command:
+  - Default action is now to list ships at given station, e.g.
+    trade.py shipvendor galileo
+  - Added "--name-sort" (--name) to sort list by ship name,
+. (kfsone) Fixed some problems with maddavo's import,
+. (kfsone) Added "WARNING" level events (use -qq to disable),
+. (kfsone) Fixed #195 "local" ValueError when no stations listed,
++ Dry411S: ShipVendors
+
+v6.13.0 Mar 03 2015
+. (kfsone) Added "modified" column to ShipVendor table,
+. (kfsone) "maddavo" import plugin:
+  - Added "--opt=shipvendors" which imports his ShipVendor.csv,
+    - Entries with a 'modified' of "DELETED" will be
+  - Added "--opt=csvonly" to stop after importing any csv files (no prices),
+  - Added "--opt=csvs" to import all the csvs
+    (equivalent to typing --opt=systems --opt=stations --opt=shipvendor)
+  e.g.
+    trade.py import --plug=maddavo --opt=csvs --opt=csvonly
+    trade.py import --plug=maddavo --opt=stations
+. (kfsone) "market" sub-command:
+  - Default behavior is equivalent to "--buy --sell",
+  - You now only need to specify --buy (-B) or --sell (-S) to list ONLY
+    those columns, so "trade.py market SOL -vv" now shows both sets of data.
+. (kfsone) "local" sub-command:
+  - Added "--stations" option: only list systems with stations,
+  - Added "--trading" option: only list stations that are flagged as having
+    a market or have trade data available
+  - Added "--blackmarket" option: only list stations with a black market,
+  - Added "--shipyard" option: only list stations with a ship yard,
+. (kfsone) "run" sub-command:
+  - check from stations for whether they have anything that can be bought,
+  - better feedback on some edge-cases where a route cannot be found,
+  - fixed some problems with --via,
+  - better feedback when using --jumps=0 or --ly=0,
+
+v6.12.4 Mar 02 2015
+. (kfsone) Added 175 Systems,
+. (kfsone) Fixed #193 "run" was ignoring --ls-max,
++ DRy411S : ShipVendors
+
+v6.12.3 Mar 01 2015
+. (kfsone) Improved how we handle some edge cases with --from and --to,
+. (kfsone) Improved feedback when requesting an unreachable journey,
+   e.g. --from selianciens --to eravate --ly 9.23 --hops 2 --jumps 2
+   (the journey requires at least 9 jumps but the options only allow 4),
+. (kfsone) Fixed #185 --from and --to giving a "set" error,
+. (kfsone) Fixed #188 Allow multiple ships per "shipvendor",
+. (kfsone) Fixed #190 "station -r" wasn't updated the .csv file,
+. (kfsone) Cleaned up the output of "local" command when showing stations,
+. (kfsone) Fixed #191 'set' does not support indexing
++ DRy411S : ShipVendors
+
 v6.12.2 Feb 26 2015
 . (kfsone) "run" command:
   - added "--direct" option:
@@ -506,7 +574,7 @@ v6.0.3 Nov 21 2014
 . (kfsone) Fixed "local" command not showing stations when using "--detail",
 . (kfsone) Renamed "buildcache.py" to "cache.py"
 
-BETA 3: 
+BETA 3:
 CAUTION: So many systems changed with the Beta 3 update that old prices files
 won't stand a chance of containing many valid systems. You may want to obtain
 a fresh file from one of the online sources, e.g.
@@ -737,7 +805,7 @@ v3.7 Sep 15/2014
     removes them.
   emdn-tap now tries harder to honor commit intervals.
 API Changes:
-  makeSubParser now takes an epilog  
+  makeSubParser now takes an epilog
 
 v3.6 Sep 12/2014
   Added DB support for tracking item stock/demand levels,

README.txt

@@ -213,7 +213,7 @@ RUN sub-command:
        How many credits to start with
        e.g.
          --credits 20000
-       
+
      --ly-per N.NN
      --ly N.NN
        Maximum distance your ship can jump between systems at full capacity.
@@ -325,7 +325,7 @@ RUN sub-command:
        Limit results to stations that match one of the pad sizes
        specified.
          --pad ML?            (med, lrg or unknown only)
-         -o ML?                 ""    ""      ""    ""
+         -p ML?                 ""    ""      ""    ""
          --pad ?              (unknown only),
          --pad L              (large only, ignores unknown)
 
@@ -549,11 +549,14 @@ IMPORT sub-command:
       This has also additional options:
       --option=<option> where option is one of the following:
         systems:      Merge maddavo's System data into local db,
-        station:      Merge maddavo's Station data into local db,
+        stations:     Merge maddavo's Station data into local db,
+        shipvendors:  Merge maddavo's ShipVendor data into local db,
+        csvs:         Merge all of the above
         exportcsv:    Regenerate System and Station .csv files after
                       merging System/Station data.
+        csvonly:      Stop after importing CSV files, no prices,
         skipdl:       Skip doing any downloads.
-        force:        Process prices even if timestamps suggest 
+        force:        Process prices even if timestamps suggest
                       there is no new data.
         use3h:        Force download of the 3-hours .prices file
         use2d:        Force download of the 2-days .prices file
@@ -662,18 +665,18 @@ MARKET sub-command:
   Lists items bought / sold at a given station; with --detail (-v) also
   includes the average price for those items.
 
-  trade.py market <station> [--buy] [--sell] [--detail]
+  trade.py market <station> [--buy | --sell] [--detail]
 
     station
       Name of the station to list, e.g. "paes/ramon" or "ramoncity",
 
     --buy
     -B
-      List items bought by the station (listed as 'SELL' in-game)
+      List only items bought by the station (listed as 'SELL' in-game)
 
     --sell
     -S
-      List items sold by the station (listed as 'BUY' in-game)
+      List only items sold by the station (listed as 'BUY' in-game)
 
     --detail
     -v
@@ -776,13 +779,24 @@ LOCAL sub-command:
     --pad-size SML?
     --pad SML?
     -p
-      Limit results to stations that match one of the pad sizes
-      specified.
+      Limit stations to those that match one of the pad sizes specified.
         --pad ML?            (med, lrg or unknown only)
-        -o ML?                 ""    ""      ""    ""
+        -p ML?                 ""    ""      ""    ""
         --pad ?              (unknown only),
         --pad L              (large only, ignores unknown)
 
+    --stations
+      Limit results to systems which have stations
+
+    --trading
+      Limit stations to those which which have markets or trade data.
+
+    --shipyard
+      Limit stations to those known to have a shipyard.
+
+    --blackmarket
+      Limit stations to those known to have a black market.
+
     -v
       Show stations + their distance from star
 
@@ -823,13 +837,29 @@ LOCAL sub-command:
     Adding detail ('-vv' or '-v -v' or '--detail --detail') would add
     a count of the number of items we have prices for at each station.
 
+    > trade.py local LAVE --trading --ly 4 -vv
+    System    Dist
+      /  Station            StnLs Age/days Mkt BMk Shp Pad Itms
+    -----------------------------------------------------------
+    LAVE      0.00
+      /  Castellan Station  2.34K     2.57 Yes  No  No Med   37
+      /  Lave Station         299     7.79 Yes Yes Yes Lrg   33
+      /  Warinus              863     7.76 Yes Yes  No Med   38
+    DISO      3.59
+      /  Shifnalport          284     0.57 Yes Yes Yes Lrg   34
+    LEESTI    3.91
+      /  George Lucas         255     0.58 Yes Yes Yes Lrg   52
+      /  Kolmogorov Hub     2.96K     1.61 Yes Yes  No Med   53
+
+    > trade.py local SOL --blackmarket --ly 6 -vv
+
 
 BUY sub-command:
 
   Finds stations that are selling / where you can buy, a named list of
   items or ships.
-  
-  trade.py buy 
+
+  trade.py buy
         [-q | -v] [--quantity Q] [--near N] [--ly-per N]
         [-P | -S] [--limit]
         [--one-stop | -1]

cache.py

@@ -95,6 +95,7 @@
 
 
 ocrDerp = re.compile(r'''(
+    ^.$ |
     LAN[O0]ING |
     [O0][O0]CK |
     [O0]INEILL |
@@ -140,6 +141,7 @@
     \bVAR[O0]E |
     EN[^T]?ERPRISE |
     EN..ERPRISE |
+    [E38]NT[E38]F[I1']?PR[I1'][S5][E38] |
     \bMUR[O0]O |
     \bBAR[O0]E |
     \bBALLAR[O0] |
@@ -163,7 +165,8 @@
         VISION |
         REFINERY
     )$ |
-    ORBIDAL |
+    [O0]RB[I1]DAL |
+    [O0]R[DL][I1]TAL |
     \bBRIOGER |
     \bJUOSON |
     LANOER |
@@ -198,10 +201,11 @@
     \bRD[DO0][DO0]EN |
     \bR[O0]([O0]D|D[O0])EN |
     (?<!BR)[O0]ECK |
-    SETTL(FMENT|EMFNT|FMFNT) |
-    SETTLMENT |
-    SETTLE\sMENT |
-    SETTLEMNT |
+    SE?TTL(FMENT|EMFNT|FMFNT) |
+    SE?TTLMENT |
+    STTL[E38]?M[E38]?NT |
+    S\s?[E38]\s?T\s?T\s?L\s?[E38](\sM|M\s)[E38]\s?N\s?T$ |
+    S[E38]TT[38]?L[E38]MNT |
     MARKFT |
     HANGFR |
     CL(EVF|FVE|FVF) |
@@ -212,7 +216,7 @@
     ` |
     -- |
     \bREILLI\b |
-    RINF\b |
+    \bRIN[FC]\b |
     \bOL[E3]ARY |
     ‹ |
     \bSATION\b |
@@ -229,7 +233,7 @@
     \bHORNUCH\b |
     \bKLU\sDZE |
     ^[DR]HN\b |
-    SU\sI?RVEY\b |
+    SU\sI?RVEY$ |
     H[DO0]L[O0]ING |
     H[D0]LDING |
     M[DO0]HMAN[O0] |
@@ -251,7 +255,28 @@
     \bC[O0]LCNY\b |
     \bOOCTE\b |
     \bBULGAFIIN\b |
-    \bWH\sIEEL
+    \bWH\sIEEL |
+    \bBR8NNAN\b |
+    \b(ID)?ING$ |
+    GATEVVAY$ |
+    [HI\s]U\sI?B$ |
+    CLAI\sI?M$ |
+    \bUITY$ |
+    \bDING$ |
+    BTOP$ |
+    B'I'OP$ |
+    TRANQUNUTY$ |
+    C[O0]LUNY$ |
+    \bMAGN\sI?US\b |
+    \s-[A-Z] |
+    ^NAKAM(\sIU|U\sIR) |
+    ^THOM\sI?PSON |
+    ^STEPH\sI?ENSON |
+    \bCQRK\b |
+    ^AN\sI?DREW |
+    ^WATSO\sIN |
+    ^QSSW |
+    ^RTZEN
 )''', flags=re.X)
 
 
@@ -452,7 +477,7 @@ def getItemByNameIndex(cur):
 
 
 def checkForOcrDerp(tdenv, systemName, stationName):
-    match = ocrDerp.search(stationName)
+    match = ocrDerp.search(stationName.upper())
     if match:
         tdenv.NOTE(
             "Ignoring '{}/{}' because it looks like OCR derp."

commands/exceptions.py

@@ -37,16 +37,13 @@ def __init__(self, errorStr):
         self.errorStr = errorStr
     def __str__(self):
         return "Error: {}\n".format(self.errorStr) + ("""
-This could be due to a lack of price or station data. You
-may want to consult the "local -vv" sub-command to see if
-there are stations in the area with price data.
+Possible causes:
+- No profitable runs or routes meet your criteria,
+- Missing Systems or Stations along the route (see "local -vv"),
+- Missing price data (see "market -vv" or "update -h"),
 
-It can also be caused by a lack of any profitable runs
-that match the criteria you specified.
-
-See '{} update -h' for help entering/updating prices, or
-obtain a crowd-sourced '.prices' file from the web, such
-as maddavo's (http://www.davek.com.au/td/)".
+If you are not sure where to get data from, consider using a crowd-sourcing
+project such as "maddavo's" (http://www.davek.com.au/td/).
 
 For more help, see the TradeDangerous Wiki:
     http://kfs.org/td/wiki

commands/import_cmd.py

@@ -2,6 +2,7 @@
 
 from commands.exceptions import *
 from commands.parsing import MutuallyExclusiveGroup, ParseArgument
+from itertools import chain
 from pathlib import Path
 
 import cache
@@ -26,7 +27,12 @@
     "is removed, but other stations are not affected."
 )
 name='import'
-epilog=None
+epilog=(
+    "This sub-command provides a plugin infrastructure, and comes "
+    "with a module to import data from Maddavo's Market Share "
+    "(http://www.davek.com.au/td/).\n"
+    "See \"import --plug=maddavo --opt=help\" for more help."
+)
 wantsTradeDB=False
 arguments = [
 ]
@@ -82,6 +88,10 @@
 def run(results, cmdenv, tdb):
     # If we're using a plugin, initialize that first.
     if cmdenv.plug:
+        if cmdenv.pluginOptions:
+            cmdenv.pluginOptions = chain.from_iterable(
+                opt.split(',') for opt in cmdenv.pluginOptions
+            )
         try:
             pluginClass = plugins.load(cmdenv.plug, "ImportPlugin")
         except plugins.PluginException as e: