LUCENE-9211 Add compression for Binary doc value fields by markharwood · Pull Request #1234 · apache/lucene-solr

markharwood · 2020-02-03T17:31:55Z

This PR stores groups of 32 doc values in LZ4 compressed blocks.

Write performance

Test results for loading 680mb of log data (1.8m docs) are as follows:

Branch	Load time (seconds, single thread)	Resulting index size (mb)
master	16	680
this PR	11	78

Read performance

Time taken to read 5,000 random doc IDs from above indices

Branch	Read time (milliseconds, single thread)
master	284
this PR	63

On this particular test, the read + write speeds and resulting index size were all improved over the current master implementation. Obviously performance will vary with other tests with the main factors for changes being:

size of fields,
compress-ability of field contents,
read access patterns (hitting same compressed blocks vs different ones)
variation from doc-to-doc in field value sizes,

markharwood · 2020-02-04T10:31:18Z

FYI - related Jira issue is pending. I can't log in until Jira admins recover my identity which looks to have been accidentally merged with a different "markh".

mikemccand · 2020-02-04T15:30:48Z

my identity which looks to have been accidentally merged with a different "markh".

LOL, that's crazy -- you should go introduce yourself to the other markh ;)

mikemccand · 2020-02-04T15:34:24Z

@markharwood how can we reproduce these benchmarks? What were the log data documents storing as BINARY doc values fields?

And how can indexing and searching get so much faster when compress/decompress is in the path! These numbers seem a little too good to believe ... but if they hold up, that's incredible. Lucene's default codec doesn't do any compression for BINARY doc values so it'd be great to add that, especially with fun issues looking to give vectors first class support in Lucene (backed by BINARY doc values, I think).

mikemccand

Thanks @markharwood!

I think our testing of BINARY doc values may not be great ... maybe add a randomized test that sometimes stores very compressible and very incompressible, large, BINARY doc values?

Separately, another way we could increase test coverage is to make a mock codec that implements doc values formats on top of the BINARY format of the underlying codec. I.e. it'd implement NUMERIC by converting the longs to 8 bytes, and save to a separate BINARY field. This should massively improve our test coverage of BINARY ... I'll open an issue for that.

mikemccand · 2020-02-04T15:34:50Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

-      data.writeBytes(v.bytes, v.offset, v.length);
-      minLength = Math.min(length, minLength);
-      maxLength = Math.max(length, maxLength);
+  class CompressedBinaryBlockWriter  implements Closeable {


Extra space character before implements?

mikemccand · 2020-02-04T15:36:19Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

-      start = data.getFilePointer();
-      meta.writeLong(start);
+    private void flushData() throws IOException {
+      if(numDocsInCurrentBlock > 0) {


Add space after if before (?

mikemccand · 2020-02-04T15:38:02Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+
+
+      // already read and uncompressed?
+      if ( blockId != lastBlockId) {


Remove space after ( before blockId?

mikemccand · 2020-02-04T15:38:23Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+        }
+      }
+
+      // Position the Bytes ref to the relevant part of the uncompressed block


s/Bytes ref/BytesRef?

mikemccand · 2020-02-04T15:40:31Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

-      meta.writeLong(data.getFilePointer() - offset); // docsWithFieldLength
-      meta.writeShort(jumpTableEntryCount);
-      meta.writeByte(IndexedDISI.DEFAULT_DENSE_RANK_POWER);
+    void  addDoc(int doc, BytesRef v) throws IOException {


Remove one of the spaces after void before addDoc?

mikemccand · 2020-02-04T15:41:46Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+        totalChunks++;
+        long thisBlockStartPointer = data.getFilePointer();
+        data.writeVInt(numDocsInCurrentBlock);
+        for (int i = 0; i < numDocsInCurrentBlock; i++) {


Probably we could (later, separate issue) optimize writing these lengths -- often all docs will have the same length?

mikemccand · 2020-02-04T15:43:10Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+            fp += filePointersIn.readVLong();
+          }
+          if (maxPointer < fp) {
+            throw new CorruptIndexException("File pointers don't add up", filePointersIn);


Can you include maxPointer and fp in this exception message?

markharwood · 2020-02-05T11:36:31Z

Thanks for looking at this, Mike.

LOL, that's crazy -- you should go introduce yourself to the other markh ;)

I already reached out and we're working out the divorce proceedings :)

@markharwood how can we reproduce these benchmarks? What were the log data documents storing as BINARY doc values fields?

These were elasticsearch log file entries - so each value was a string which could be something short like [instance-0000000048] users file [/app/config/users] changed. updating users... ) or an error with a whole stack trace.
My test rig is here if you want to try with some other data files

And how can indexing and searching get so much faster when compress/decompress is in the path!

This was a test on my macbook with SSD and encrypted FS so perhaps not the best benchmarking setup. Maybe just writing more bytes = more overhead with the OS-level encryption?

I think our testing of BINARY doc values may not be great ... maybe add a randomized test that sometimes stores very compressible and very incompressible, large, BINARY doc values?

Will do. @jimczi has suggested adding support for storing without compression when the content doesn't compress well. I guess that can be a combination of :

A fast heuristic - e.g. if max value length for each of the docs in a block <=2 then store without compression and
"Try it and see" compression - buffer compression output to byte array and only write compressed form to disk if size is less than the uncompressed input

markharwood · 2020-02-06T11:42:07Z

And how can indexing and searching get so much faster when compress/decompress is in the path!

I tried benchmarking some straight-forward file read and write operations (no Lucene) and couldn't show LZ4 compression being faster (although it wasn't that much slower).

Maybe the rate-limited merging in Lucene plays a part and size therefore matters in that context?

jpountz · 2020-02-05T13:25:54Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java


  IndexOutput data, meta;
  final int maxDoc;
+  private SegmentWriteState state;


make it final?

jpountz · 2020-02-05T13:27:14Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+    int uncompressedBlockLength = 0;
+    int maxUncompressedBlockLength = 0;
+    int numDocsInCurrentBlock = 0;
+    int [] docLengths = new int[Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK]; 


we usually don't let spaces between the type of array elements and []

Suggested change

int [] docLengths = new int[Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK];

int[] docLengths = new int[Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK];

jpountz · 2020-02-05T13:27:23Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+    int maxUncompressedBlockLength = 0;
+    int numDocsInCurrentBlock = 0;
+    int [] docLengths = new int[Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK]; 
+    byte [] block = new byte [1024 * 16];


Suggested change

byte [] block = new byte [1024 * 16];

byte[] block = new byte [1024 * 16];

jpountz · 2020-02-05T14:06:22Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+        totalChunks++;
+        long thisBlockStartPointer = data.getFilePointer();
+        data.writeVInt(numDocsInCurrentBlock);
+        for (int i = 0; i < numDocsInCurrentBlock; i++) {


jpountz · 2020-02-05T14:07:52Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+      } catch (Throwable exception) {
+        IOUtils.closeWhileHandlingException(this); //self-close because constructor caller can't 
+        throw exception;
+      }


we usually do this like that instead, which helps avoid catching Throwable

boolean success = false; try { // write header } finally { if (success == false) { // close } }

What was the "+1" comment for line 407 about?
I've seen encoding elsewhere that have n+1 offsets to record start of each value and the last offset is effectively the end of the last value. In this scenario I'm writing n value lengths.

It was about optimizing for the case that all values have the same length. In that case we could still one bit of the first length to mean that all values have the same length for instance?

jpountz · 2020-02-05T14:09:36Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+    void addDoc(int doc, BytesRef v) throws IOException {
+      if (blockAddressesStart < 0) {
+        blockAddressesStart = data.getFilePointer();
+      }


it looks like we could set blockAddressesStart in the constructor instead?

I tried that and it didn't work - something else was writing to data in between constructor and addDoc calls

Have you found what this something else is?

jpountz · 2020-02-05T14:13:59Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+          data.writeVInt(docLengths[i]);
+        }
+        maxUncompressedBlockLength = Math.max(maxUncompressedBlockLength, uncompressedBlockLength);
+        LZ4.compress(block,  0, uncompressedBlockLength, data, ht);


Suggested change

LZ4.compress(block, 0, uncompressedBlockLength, data, ht);

LZ4.compress(block, 0, uncompressedBlockLength, data, ht);

jpountz · 2020-02-05T14:14:49Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+    }
+
+    void writeMetaData() throws IOException {
+      if (blockAddressesStart < 0 ) {


this only happens if there are no values? when do we run into this condition?

Looks to be when merges clear out deleted docs leaving no values.

This makes sense, can you leave a comment about it?

jpountz · 2020-02-06T13:18:18Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+      this.compressedData = compressedData;
+      // pre-allocate a byte array large enough for the biggest uncompressed block needed.
+      this.uncompressedBlock = new byte[biggestUncompressedBlockSize];
+


we could initialize uncompressedBytesRef from the uncompressed block:
uncompressedBytesRef = new BytesRef(uncompressedBlock)
and avoid creating new BytesRefs over and over in decode

markharwood · 2020-02-07T09:45:33Z

I've reclaimed my Jira log-in and opened https://issues.apache.org/jira/browse/LUCENE-9211

markharwood · 2020-02-07T15:21:09Z

There was a suggestion from @jimczi that we fall back to writing raw data if content doesn't compress well. I'm not sure this logic is worth developing for the reasons outlined below:

I wrote a compression buffer to see what the compression algo outputs before deciding whether to write the compressed or raw data to disk.
I tested with the most uncompressible content I could imagine:

public static void fillRandom(byte[] buffer, int length) {
    for (int i = 0; i < length; i++) {
        buffer[i] =  (byte) (Math.random() * Byte.MAX_VALUE);
    }
}

The LZ4 compressed versions of this content were only marginally bigger than their raw counterparts (adding 0.4% overhead to the original content e.g. 96,921 compressed vs 96,541 raw bytes).
On that basis I'm not sure if it's worth doubling the memory costs of the indexing logic (we would require a temporary output buffer that is at least the same size as the raw data being compressed) and additional byte shuffling.

msokolov · 2020-02-07T17:37:37Z

The LZ4 compressed versions of this content were only marginally bigger than their raw counterparts

Did you also test read performance in this incompressible case?

jpountz · 2020-02-07T17:58:36Z

In the case of content that can't be compressed, the compressed data will consist of the number of bytes, followed by the bytes. So decompressing consists of decoding the length and then reading the bytes. The only overhead compared to reading bytes directly is the decoding of the number of bytes, so I would believe that the overhead is rather small.

I don't have a strong preference regarding whether this case should be handled explicitly or not. It's true that not having a special "not-compressed" case helps keep the logic simpler.

markharwood · 2020-02-07T17:59:13Z

Did you also test read performance in this incompressible case?

Just tried it and it does look 4x faster reading raw random bytes Vs compressed random bytes

jpountz · 2020-02-07T18:02:14Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+              uncompressedBlockLength += onlyLength;
+            }
+          }
+          uncompressedDocEnds[i] = uncompressedBlockLength;


maybe we could call it uncompressedDocStarts and set the index at i+1 which would then help below to remove the else block of the docInBlockId > 0 condition below?

jpountz · 2020-02-07T18:03:15Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+        long blockStartOffset = addresses.get(blockId);
+        compressedData.seek(blockStartOffset);
+
+        numDocsInBlock = compressedData.readVInt();


do we really need to record the number of documents in the block? It should be 32 for all blocks except for the last one? Maybe at index-time we could append dummy values to the last block to make sure it has 32 values too, and we wouldn't need this vInt anymore?

jpountz · 2020-02-07T18:05:45Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+            // all other values are the same length
+            int lengthPlusSameInd = compressedData.readVInt();
+            int sameIndicator = lengthPlusSameInd & 1;
+            int firstValLength = lengthPlusSameInd >>1;


Since you are stealing a bit, we should do an unsigned shift (>>>) instead.

This would never be a problem in practice, but imagine than the length was a 31-bits integer. Shifting by one bit on the left at index time would make this number negative. So here we need an unsigned shift rather than a signed shift that preserves the sign.

jpountz · 2020-02-07T18:12:12Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+    private final IndexInput compressedData;
+    // Cache of last uncompressed block 
+    private long lastBlockId = -1;
+    private int []uncompressedDocEnds = new int[Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK];


in the past we've put these constants in the meta file and BinaryEntry so that it's easier to change values over time

@jpountz we should use the same structure while writing the data, in that case you will see all the properties of the class instead of adding comments in the code

jpountz · 2020-02-07T18:12:46Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

+
+        numDocsInBlock = compressedData.readVInt();
+        assert numDocsInBlock <= Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK;
+        uncompressedDocEnds = new int[numDocsInBlock];


can we reuse the same array across blocks?

jpountz · 2020-02-07T18:15:28Z

@msokolov FWIW LZ4 only removes duplicate strings from a stream: when it finds one it inserts a reference to a previous sequence of bytes. In the special case that the content in incompressible, the LZ4 compressed data just consists of the number of bytes followed by the bytes, so the only overhead compared to reading the bytes directly is the decoding of the number of bytes, which should be rather low.

I don't have a preference regarding whether we should have an explicit "not-compressed" case, but I understand how not having one helps keep things simpler.

msokolov · 2020-02-07T18:19:38Z

Strange that Mark would measure 4x slowdown from decoding the lengths... Perhaps the random bytes are not totally incompressible, just barely compressible?

markharwood · 2020-02-07T18:21:26Z

Strange that Mark would measure 4x slowdown from decoding the lengths... Perhaps the random bytes are not totally incompressible, just barely compressible?

I may have been too hasty in that reply - I've not been able to reproduce that and the raw vs compressed timings are very similar in the additional tests I've done so echo what @jpountz expects. My first (faster) run had random bytes selected in the range 0-20 and not the 0-127 range where I'm seeing parity

juanka588 · 2020-02-11T10:18:04Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+    try (CompressedBinaryBlockWriter blockWriter = new CompressedBinaryBlockWriter()){
+      BinaryDocValues values = valuesProducer.getBinary(field);
+      long start = data.getFilePointer();
+      meta.writeLong(start); // dataOffset


I think we should use the BinaryEntry object here, and the just make the object "Writable" to a given DataOutput and "Readable" from a DataInput (which is already the case: readBinaryEntry). This will avoid the comments in the code -2 == docsWithFieldOffset etc.

I like the idea but would prefer doing it in a separate PR.

juanka588 · 2020-02-11T10:20:40Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+      if (numDocsWithField == 0) {
+        meta.writeLong(-2); // docsWithFieldOffset
+        meta.writeLong(0L); // docsWithFieldLength
+        meta.writeShort((short) -1); // jumpTableEntryCount
+        meta.writeByte((byte) -1);   // denseRankPower
+      } else if (numDocsWithField == maxDoc) {
+        meta.writeLong(-1); // docsWithFieldOffset
+        meta.writeLong(0L); // docsWithFieldLength
+        meta.writeShort((short) -1); // jumpTableEntryCount
+        meta.writeByte((byte) -1);   // denseRankPower
+      } else {
+        long offset = data.getFilePointer();
+        meta.writeLong(offset); // docsWithFieldOffset
+        values = valuesProducer.getBinary(field);
+        final short jumpTableEntryCount = IndexedDISI.writeBitSet(values, data, IndexedDISI.DEFAULT_DENSE_RANK_POWER);
+        meta.writeLong(data.getFilePointer() - offset); // docsWithFieldLength
+        meta.writeShort(jumpTableEntryCount);
+        meta.writeByte(IndexedDISI.DEFAULT_DENSE_RANK_POWER);
+      }


Currently I'm working in a refactor of this code by having a doc id set iterator serializer capable to provide the correct instance based on the stored metadata. As you might see this is quite repetitive for the other fields

juanka588 · 2020-02-11T11:02:17Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesFormat.java

  static final String META_EXTENSION = "dvm";
  static final int VERSION_START = 0;
-  static final int VERSION_CURRENT = VERSION_START;
+  static final int VERSION_BIN_COMPRESSED = 1;  


This could be potentially in the BinaryDocValuesFormat class

jpountz

I left some minor comments, but I'm +1 on this PR, it looks great.

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

jpountz · 2020-02-14T08:16:02Z

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

+    void addDoc(int doc, BytesRef v) throws IOException {
+      if (blockAddressesStart < 0) {
+        blockAddressesStart = data.getFilePointer();
+      }


Have you found what this something else is?

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesProducer.java

lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java

…ues in LZ4 compressed blocks

… metadata, remove per-chunk storage of num docs and assume always same number of values. Change doc end pointers to be doc starts.

…emccand

…emccand suggestion

…sesStart in CompressedBinaryBlockWriter constructor, serialise shift for number of docs in block,

Stores groups of 32 binary doc values in LZ4-compressed blocks.

Stores groups of 32 binary doc values in LZ4-compressed blocks. (cherry picked from commit f549ee3)

The binary doc values implementation from LUCENE-9211 (apache/lucene-solr#1234) is used here, which stores the values in LZ4 compressed blocks, in order to reduce storage usage (in the Lucene default doc values coded binary doc values are stored without any compression). Follow up from elastic#105301

markharwood requested review from jimczi and jpountz February 3, 2020 17:32

mikemccand reviewed Feb 4, 2020

View reviewed changes

jpountz reviewed Feb 6, 2020

View reviewed changes

markharwood mentioned this pull request Feb 7, 2020

Wildcard field optimised for wildcard queries elastic/elasticsearch#49993

Merged

jpountz reviewed Feb 7, 2020

View reviewed changes

juanka588 reviewed Feb 11, 2020

View reviewed changes

jpountz approved these changes Feb 14, 2020

View reviewed changes

markharwood force-pushed the compressedBinDV branch from b1048ba to a20da17 Compare February 14, 2020 09:43

janhoy changed the title ~~Add compression for Binary doc value fields~~ LUCENE-9211 Add compression for Binary doc value fields Feb 15, 2020

markharwood added 4 commits February 18, 2020 09:45

Added compression for binary doc values - stores groups of 32 doc val…

1dcbb2e

…ues in LZ4 compressed blocks

Formatting fixes (Thanks, Mike!)

1b0c227

Addressing review comments (thanks, jpountz!)

1b08f9b

Optimisation - only write one value if all docs share the same length

6dd00af

markharwood added 3 commits February 18, 2020 09:45

Addressing @jpountz review comments - make num docs per chunk part of…

f69f990

… metadata, remove per-chunk storage of num docs and assume always same number of values. Change doc end pointers to be doc starts.

Added variously compressible examples of content to tests as per @mik…

289d549

…emccand suggestion

Addressing review comments. Fields made final, initialise blockAddres…

7d0db34

…sesStart in CompressedBinaryBlockWriter constructor, serialise shift for number of docs in block,

markharwood force-pushed the compressedBinDV branch from e830eda to 7d0db34 Compare February 18, 2020 09:58

Added note to changes.txt

3a3c30f

markharwood merged commit ce2959f into apache:master Feb 18, 2020

markharwood added a commit to markharwood/lucene-solr that referenced this pull request Feb 18, 2020

LUCENE-9211 Add compression for Binary doc value fields (apache#1234)

d0d9a4c

Stores groups of 32 binary doc values in LZ4-compressed blocks.

markharwood added a commit that referenced this pull request Feb 18, 2020

LUCENE-9211 Add compression for Binary doc value fields (#1234)

bcdc21a

Stores groups of 32 binary doc values in LZ4-compressed blocks. (cherry picked from commit f549ee3)

markharwood mentioned this pull request Mar 12, 2020

Range field performance regression after Lucene compression change elastic/elasticsearch#53498

Closed

asfimport mentioned this pull request May 21, 2020

Adding compression to BinaryDocValues storage [LUCENE-9211] apache/lucene#10251

Closed

martijnvg mentioned this pull request Feb 9, 2024

Change PerFieldMapperCodec to use tsdb doc values codec for all fields. elastic/elasticsearch#105301

Merged

martijnvg mentioned this pull request Feb 12, 2024

Update tsdb binary doc value format to use compression from LUCENE-9211. elastic/elasticsearch#105419

Closed

bugmakerrrrrr mentioned this pull request Dec 15, 2024

[Feature Request] Use of Binary DocValue for high cardinality fields to improve aggregations performance opensearch-project/OpenSearch#16837

Open



		// already read and uncompressed?
		if ( blockId != lastBlockId) {

	int [] docLengths = new int[Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK];
	int[] docLengths = new int[Lucene80DocValuesFormat.BINARY_DOCS_PER_COMPRESSED_BLOCK];

	byte [] block = new byte [1024 * 16];
	byte[] block = new byte [1024 * 16];

	LZ4.compress(block, 0, uncompressedBlockLength, data, ht);
	LZ4.compress(block, 0, uncompressedBlockLength, data, ht);

Conversation

markharwood commented Feb 3, 2020

Write performance

Read performance

Uh oh!

markharwood commented Feb 4, 2020

Uh oh!

mikemccand commented Feb 4, 2020

Uh oh!

mikemccand commented Feb 4, 2020

Uh oh!

mikemccand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markharwood commented Feb 5, 2020

Uh oh!

markharwood commented Feb 6, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markharwood commented Feb 7, 2020

Uh oh!

markharwood commented Feb 7, 2020

Uh oh!

msokolov commented Feb 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jpountz commented Feb 7, 2020

Uh oh!

markharwood commented Feb 7, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

msokolov commented Feb 7, 2020 •

edited

Loading

markharwood commented Feb 7, 2020 •

edited

Loading