ARM64: flatten/trim operations produce incorrect output from 16-bit RGBA input #1651

ossdev07 · 2019-04-08T12:12:52Z

uname -a
Linux c5475310be72 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:33:57 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

Facing below issues while testing sharp package test cases in arm64 platform.

npm test output:

Alpha transparency
Flatten 16-bit PNG with transparency to orange:
Uncaught Error: Expected maximum absolute distance of 25, actual 92.25032043457031
at Object.assertMaxColourDistance (test/fixtures/index.js:191:13)
at /pruthvi/sharp/test/unit/alpha.js:59:18
Trim borders
16-bit PNG with alpha channel:

Uncaught AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:

expected - actual

-2

-4
+ expected - actual

--2
+-4

at /pruthvi/sharp/test/unit/trim.js:55:16

Please guide me on solving these issues.
Thanks in advance.

The text was updated successfully, but these errors were encountered:

ossdev07 · 2019-04-09T06:56:33Z

Tested in x86 platform, all tests are getting passed. But 2 tests are getting failed while running in arm64 platform.
Steps to reproduce issues in arm64 platform:

git clone https://github.com/lovell/sharp
cd sharp
npm install-test

libvips is getting installed in arm64 platform, but unable to find any specific solution for above issues.

Please guide me on the same.

Thanks in advance.

lovell · 2019-04-10T19:13:33Z

Thanks for the report, I'll try to reproduce this via a QEMU-based VM. (I've been meaning to add an ARM64 CI job that might have caught this, not that I'm entirely sure what's causing it yet.)

ossdev07 · 2019-04-11T15:30:35Z

@lovell
Thanks for the reply.

We can use drone.io CI to build and run tests in arm64 platform. If you agree on the same, i will raise a PR with drone.io CI support.

lovell · 2019-04-11T19:48:14Z

I've been able to reproduce this via ARM64 QEMU emulation. Are you seeing this on real ARM64 hardware?

Both the failing tests use 16-bit RGBA input and involve a flatten operation.

Here is the state of the image just before and after the call to vips_flatten using an orange-coloured 16-bit background ink:

The first (input) image has max pixel values of 0xFFFF.

If this was a signed vs unsigned short problem then I'd expect the max pixel values of the second (output) image to erroneously be 0x7FFF, but instead it is 0x7F7F.

This suggests the top bit of each byte is being erroneously cleared, which is unexpected, perhaps even a compiler bug? The prebuild libvips binaries for ARM64 use the -Os flag.

The next step is to compile libvips from source inside the same ARM64 QEMU container and debug the internal state of libvips during the flatten operation.

ossdev07 · 2019-04-12T05:25:43Z

@lovell Thanks for the report. Yes, I have observed this on my local ARM machine.

ossdev07 · 2019-04-15T11:42:21Z

@lovell Followed link https://github.com/libvips/libvips/wiki/Build-for-Ubuntu to build libvips from source and set environment variables and ran tests. But getting same result, as these 2 cases are getting failed.

lovell · 2019-04-15T15:14:35Z

Thanks, which compiler (gcc?) version and flags were used?

ossdev07 · 2019-04-16T05:59:37Z

@lovell
Have used below gcc version.
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/8/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 8.2.0-7ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=aarch64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --disable-libphobos --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu
Thread model: posix
gcc version 8.2.0 (Ubuntu 8.2.0-7ubuntu1)

I have not added any extra flags, just executed make in source to build libvips.
Please revert if anything need to be added. Thanks in advance.

ossdev07 · 2019-04-24T07:09:49Z

Hi @lovell,

I can try if any suggestion or alternate method you have for this issue.

Thanks in advance.

ossdev07 · 2019-07-24T06:46:34Z

Hi @lovell

I have cloned the new release v0.22.1 and tested the package in aarch64 platform using command npm install-test with libvips version v8.8.0.

Facing below issues in aarch64 platform:

Alpha transparency
Flatten 16-bit PNG with transparency to orange:
Uncaught Error: Expected maximum absolute distance of 25, actual 92.25032043457031
at Object.assertMaxColourDistance (test/fixtures/index.js:191:13)
at /root/sharp/test/unit/alpha.js:57:18
Image metadata
JPEG with IPTC/XMP:

Uncaught AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:

expected - actual

12495

12466
+ expected - actual
```
-12495
+12466

at /root/sharp/test/unit/metadata.js:78:14
```
1. Negate
  negate (webp, trans):
  Error: Expected maximum similarity distance: 5. Actual: 29.
  at /root/sharp/test/fixtures/index.js:170:27
  at /root/sharp/test/fixtures/index.js:32:9
2. Resize fit=cover
  Attention strategy
  JPEG:
  
  Uncaught AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:
expected - actual

-143

-107
+ expected - actual
```
--143
+-107

at /root/sharp/test/unit/resize-cover.js:341:18
```
1. Threshold
  threshold default webp transparency:
  Error: Expected maximum similarity distance: 5. Actual: 6.
  at /root/sharp/test/fixtures/index.js:170:27
  at /root/sharp/test/fixtures/index.js:32:9
2. Trim borders
  16-bit PNG with alpha channel:
  
  Uncaught AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:
expected - actual

-2

-4
+ expected - actual

--2
+-4

at /root/sharp/test/unit/trim.js:55:16

Do we need to install any extra dependencies?

lovell · 2019-07-25T10:48:36Z

Thanks for continuing to investigate this, I've not had the time, nor do I have any ARMv8 hardware to hand (perhaps an excuse here to get the new Pi 4).

The latest work-in-progress code in the vision branch has updates to the unit tests so they pass with libvips v8.8.0+.

ossdev07 · 2019-07-25T12:26:07Z

Thanks for continuing to investigate this, I've not had the time, nor do I have any ARMv8 hardware to hand (perhaps an excuse here to get the new Pi 4).

The latest work-in-progress code in the vision branch has updates to the unit tests so they pass with libvips v8.8.0+.

Thanks for the reply.
To verify in ARMv8 or ARM64, I can provide drone.yml to test sharp package in drone.io CI which supports ARM64.
I will test with vision branch and will update the same.

ossdev07 · 2019-07-26T06:27:15Z

Hi @lovell ,

Tested with vision branch in ARM64 platform and it is failing in build stage.

Please find below logs for the same:

[email protected] install /root/sharp
(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Detected globally-installed libvips v8.8.0
info sharp Building from source via node-gyp
make: Entering directory '/root/sharp/build'
TOUCH Release/obj.target/libvips-cpp.stamp
CXX(target) Release/obj.target/sharp/src/common.o
CXX(target) Release/obj.target/sharp/src/metadata.o
CXX(target) Release/obj.target/sharp/src/stats.o
CXX(target) Release/obj.target/sharp/src/operations.o
CXX(target) Release/obj.target/sharp/src/pipeline.o
In file included from /usr/local/include/vips/vips.h:114,
from /usr/local/include/vips/VError8.h:38,
from /usr/local/include/vips/vips8:52,
from ../src/pipeline.cc:27:
../src/pipeline.cc: In member function 'virtual void PipelineWorker::Execute()':
../src/pipeline.cc:800:44: error: 'class vips::VImage' has no member named 'heifsave_buffer'; did you mean 'tiffsave_buffer'?
VipsArea area = VIPS_AREA(image.heifsave_buffer(VImage::option()
^~~~~~~~~~~~~~~
../src/pipeline.cc:921:17: error: 'class vips::VImage' has no member named 'heifsave'; did you mean 'tiffsave'?
image.heifsave(const_cast<char>(baton->fileOut.data()), VImage::option()
^~~~~~~~
tiffsave
make: *** [sharp.target.mk:169: Release/obj.target/sharp/src/pipeline.o] Error 1
make: Leaving directory '/root/sharp/build'
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2

Do we need to install any specific dependencies or run any specific command before testing?

lovell · 2019-07-26T08:41:46Z

That's interesting. Are you using libvips built from source? Was it compiled from a tarball or git? The heifsave_buffer method is definitely included in the libvips v8.8.0 release (even if libvips is compiled without support for libheif).

ossdev07 · 2019-07-26T09:22:46Z

That's interesting. Are you using libvips built from source? Was it compiled from a tarball or git? The heifsave_buffer method is definitely included in the libvips v8.8.0 release (even if libvips is compiled without support for libheif).

Yes i have compiled from source. I have used git to install and was facing this issue. I have re-installed libvips from source v8.8.0-rc1 release and tested the sharp package again and now heifsave_buffer issue is resolved. But facing previous issues with vision branch.

Please find below issues in ARM64 platform:

Alpha transparency
Flatten 16-bit PNG with transparency to orange:
Uncaught Error: Expected maximum absolute distance of 25, actual 92.25032043457031
at Object.assertMaxColourDistance (test/fixtures/index.js:188:13)
at /root/sharp/test/unit/alpha.js:57:18
Trim borders
16-bit PNG with alpha channel:

Uncaught AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:

expected - actual

-2

-4
+ expected - actual

--2
+-4

at /root/sharp/test/unit/trim.js:55:16

lovell · 2019-07-26T10:25:57Z

Thanks for confirming, it's the same two tests failing, both of which use the vips flatten operation on a 16-bit RGBA input. Are you still using gcc version 8.2.0?

ossdev07 · 2019-07-26T10:31:20Z

Thanks for confirming, it's the same two tests failing, both of which use the vips flatten operation on a 16-bit RGBA input. Are you still using gcc version 8.2.0?

Yes, I am still using the same gcc version (gcc version 8.2.0 (Ubuntu 8.2.0-7ubuntu1)).

rhenwood-arm · 2019-07-26T13:39:41Z

@ossdev07 : can you reproduce this problem with QEMU on x86?

ossdev07 · 2019-07-28T20:47:16Z

@rhenwood-arm ,

Built in qemu:xenial container on x86 and installed libvips version v8.8.0 from source.
Worked with sharp package vision branch and facing issues in alpha-transparency cases which we are getting with ARM64 local machine and facing more issues in svg to png conversion cases.
Worked with master branch and facing similar issues to issues facing with ARM64 local machine.

lovell · 2019-09-22T11:10:09Z

I've made some progress on this. Binaries compiled using gcc v9.2.1, as provided by Debian 11, produce the correct output both with and without the -Os flag.

This would fit with the idea that we were seeing the effects of a aarch64 compiler bug, with https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88469 a possible candidate.

See lovell/sharp#1651

pruthvireddypuresoftware · 2019-09-25T14:02:08Z

Hi @lovell ,

I have tried with docker container with debian:bullseye image and gcc 9.2.1 and built libvips from source. I was facing issue libvips/libvips#1426, so configured with flag '--without-orc' and built libvips and installed with version 8.9.0. Facing below issue while running tests:

Alpha transparency
/usr/local/bin/node: symbol lookup error: /pruthvi/sharp/build/Release/sharp.node: undefined symbol: vips_foreign_heif_compression_get_type

Installed libheif-dev as well, but still facing the same issue.

pruthvireddypuresoftware · 2019-09-25T16:37:53Z

@pruthvireddy1991 This issue relates to problems on ARM64 machines when flattening 16-bit input images. Does your question relate to this also?

yes, facing the same issue.

lovell · 2019-09-25T17:25:28Z

Thanks for confirming, it looks like sharp was compiled against a libvips with libheif but at runtime has found another libvips without libheif. Perhaps you need to run ldconfig to rebuild the shared library config. If that doesn't work, please can you share the Dockerfile+commands you're using.

ossdev07 · 2019-09-30T15:14:07Z

@lovell ,

I have created a docker container with debian:bullseye image in local arm64 machine and installed gcc version 9.2.1. Downloaded the libvips 8.8.3 tar and tried to install, but was facing ORCTargetPowerPCFlags issue. Installed it with flag "--without-orc", but facing issues in executing the tests.

Test Output:

[email protected] test /pruthvi/sharp
semistandard && cc && npm run test-unit && npm run test-licensing && prebuild-ci

[email protected] test-unit /pruthvi/sharp
nyc --reporter=lcov --branches=99 mocha --slow=5000 --timeout=60000 ./test/unit/*.js

HTTP agent
✓ Without proxy
✓ HTTPS proxy with auth from HTTPS_PROXY
✓ HTTP proxy without auth from npm_config_proxy

Alpha transparency
/usr/local/bin/node: symbol lookup error: /pruthvi/sharp/build/Release/sharp.node: undefined symbol: _ZN4vips6VImage15new_from_bufferEPKvmPKcPNS_7VOptionE

Do we need to install any extra dependencies?

lovell · 2019-10-01T09:28:15Z

Are you able to share a Dockerfile with the RUN commands to reproduce this?

ossdev07 · 2019-10-07T05:17:07Z

Are you able to share a Dockerfile with the RUN commands to reproduce this?

I have not used Dockerfile exactly, but i have pulled the image in local arm64 system.

Steps followed:

docker pull debian:bullseye
docker run -it --name=xxx debian:bullseye bash
apt-get install gcc-9
wget https://github.com/libvips/libvips/releases/download/v8.8.3/vips-8.8.3.tar.gz
./configure --without-orc
make
make install
git clone https://github.com/lovell/sharp
cd sharp
npm install-test

Please revert if any more information is required.

lovell · 2019-10-07T08:58:04Z

You'll need to run ldconfig after make install to update the shared library cache.

ossdev07 · 2019-10-07T11:17:02Z

You'll need to run ldconfig after make install to update the shared library cache.

Thanks for the reply.
I have run ldconfig and facing below issue in running tests

[email protected] test-unit /root/sharp
nyc --reporter=lcov --branches=99 mocha --slow=5000 --timeout=60000 ./test/unit/*.js
HTTP agent
✓ Without proxy
✓ HTTPS proxy with auth from HTTPS_PROXY
✓ HTTP proxy without auth from npm_config_proxy
Alpha transparency
/usr/local/bin/node: symbol lookup error: /root/sharp/build/Release/sharp.node: undefined symbol: vips_foreign_heif_compression_get_type
npm ERR! code ELIFECYCLE
npm ERR! syscall spawn
npm ERR! file sh
npm ERR! errno ENOENT
npm ERR! [email protected] test-unit: nyc --reporter=lcov --branches=99 mocha --slow=5000 --timeout=60000 ./test/unit/*.js
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] test-unit script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2019-10-07T11_00_36_316Z-debug.log
npm ERR! Test failed. See above for more details.

lovell · 2019-10-07T13:31:35Z

The vips_foreign_heif_compression_get_type function is not part of libvips v8.8.3. It currently exists only on the master branch, having been added recently in commit libvips/libvips@42f9f78

I suspect there are multiple, conflicting versions of libvips on the machine where this error is occurring.

ossdev07 · 2019-10-07T14:04:47Z

The vips_foreign_heif_compression_get_type function is not part of libvips v8.8.3. It currently exists only on the master branch, having been added recently in commit libvips/libvips@42f9f78

I suspect there are multiple, conflicting versions of libvips on the machine where this error is occurring.

I have freshly downloaded and built libvips version 8.8.3 and did ldconfig after installing libvips. Facing below issue:

[email protected] test-unit /pruthvi/sharp
nyc --reporter=lcov --branches=99 mocha --slow=5000 --timeout=60000 ./test/unit/*.js
HTTP agent
✓ Without proxy
✓ HTTPS proxy with auth from HTTPS_PROXY
✓ HTTP proxy without auth from npm_config_proxy
Alpha transparency
/usr/local/bin/node: symbol lookup error: /pruthvi/sharp/build/Release/sharp.node: undefined symbol: _ZN4vips6VImage15new_from_bufferEPKvmPKcPNS_7VOptionE
npm ERR! code ELIFECYCLE
npm ERR! syscall spawn
npm ERR! file sh
npm ERR! errno ENOENT
npm ERR! [email protected] test-unit: nyc --reporter=lcov --branches=99 mocha --slow=5000 --timeout=60000 ./test/unit/*.js
npm ERR! spawn ENOENT
npm ERR!
npm ERR! Failed at the [email protected] test-unit script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2019-10-07T14_02_14_064Z-debug.log
npm ERR! Test failed. See above for more details.

lovell · 2019-10-08T19:05:25Z

You'll need to rebuild/reinstall sharp after updating the global version of libvips.

ossdev07 · 2019-10-10T15:24:24Z

@lovell ,

I have created new container and followed the process mentioned in above comments with libvips version 8.8.1 and tests are running now, but the same 16-bit png tests are getting failed.

Do we need to install any extra dependencies?

lovell · 2019-10-12T09:12:05Z

Are you able to share the exact Dockerfile, commands and/or script you are now using?

ossdev07 · 2019-10-15T15:13:12Z

As mentioned in comment #1651 (comment), i have followed same steps with 8.8.1 libvips version and ldconfig after installing libvips.

Steps followed:
docker pull debian:bullseye
docker run -it --name=xxx debian:bullseye bash
apt-get install gcc-9
wget https://github.com/libvips/libvips/releases/download/v8.8.1/vips-8.8.1.tar.gz
./configure
make
make install
ldconfig
git clone https://github.com/lovell/sharp
cd sharp
npm install-test

Do I need to run any extra steps or commands before running test?

lovell · 2019-10-16T13:55:36Z

If there is more than one version of gcc on this machine then you may need to force the desired version via something like:

CC="gcc-9" CXX="g++-9" ./configure ...

ossdev07 · 2019-10-16T14:21:22Z

If there is more than one version of gcc on this machine then you may need to force the desired version via something like:
CC="gcc-9" CXX="g++-9" ./configure ...

I have tried with this command, but facing the same issues in 16-bit tests.

lovell · 2019-10-25T09:15:18Z

I've added an experimental ARM64 build to the Travis CI matrix e.g. https://travis-ci.org/lovell/sharp/jobs/602711582 but there's currently a known problem with https://travis-ci.community/t/no-ruby-for-c-build/5601

lovell · 2019-11-07T10:49:58Z

I've created #1953 to track the addition of ARM64 to the CI matrix. The expected 16-bit image test failures are still present at the moment, but these should hopefully go away after the next release of the prebuilt libvips binaries.

lovell · 2019-12-23T12:20:26Z

As hoped/expected, the two test failures under discussion here are now passing with the latest work-in-progress ARM64 build - see https://travis-ci.org/lovell/sharp/jobs/628697268

(There's a different, unrelated test failure around SVG rendering, which is probably a librsvg packaging thing.)

ossdev07 · 2019-12-23T15:42:57Z

@lovell ,

Thanks for the update. I have tested the same and the two test failures are now passing in the arm64 platform.

May I know how can we resolve SVG rendering issue?

ossdev07 · 2019-12-24T06:54:40Z

@lovell ,

I have followed the below steps with libvips version 8.9.0-rc1.

Steps followed:
docker pull debian:bullseye
docker run -it --name=xxx debian:bullseye bash
apt-get install gcc-9
wget https://github.com/libvips/libvips/archive/v8.9.0-rc1.tar.gz
tar -xf v8.9.0-rc1.tar.gz
cd libvips-8.9.0-rc1/
apt-get install librsvg2-bin librsvg2-dev librsvg2-doc
./autogen.sh
make
make install
ldconfig
git clone https://github.com/lovell/sharp -b wit
cd sharp
npm install-test --unsafe-perm=true

I am getting below output:
768 passing (3m)

[email protected] test-licensing /pruthvi/sharp
license-checker --production --summary --onlyAllow="Apache-2.0;BSD;ISC;MIT"

├─ MIT: 45
├─ ISC: 20
├─ Apache-2.0: 3
├─ (MIT OR WTFPL): 1
└─ (BSD-2-Clause OR MIT OR Apache-2.0): 1

All tests are passing after installing librsvg2-bin and installing libvips manually.

lovell · 2019-12-26T15:44:24Z

The failure of the "Convert SVG with embedded images to PNG, respecting dimensions, autoconvert to PNG" test is due to a meson dependency-discovery problem when cross-compiling all the ARM binaries and will be fixed by lovell/sharp-libvips@5f86843

lovell · 2019-12-29T12:22:04Z

ARM64v8 CI now passing - https://travis-ci.org/lovell/sharp/jobs/630581137

ossdev07 · 2019-12-30T05:40:36Z

Thanks @lovell . When can we expect the release with these updates?

lovell · 2019-12-30T11:21:35Z

This issue is currently tagged with the v0.24.0 milestone.

lovell · 2020-01-16T20:38:48Z

v0.24.0 now available, plus the docs have been updated to provide more info about of prebuilt libvips binaries.

https://sharp.pixelplumbing.com/install#prebuilt-binaries

MacroMan mentioned this issue Apr 9, 2019

flatten() ignoring background option #1652

Closed

lovell changed the title ~~npm test fails with error in alpha transparency functionality~~ ARM64: npm test fails with error in alpha transparency functionality Apr 10, 2019

lovell added the triage label Apr 10, 2019

lovell changed the title ~~ARM64: npm test fails with error in alpha transparency functionality~~ ARM64: flatten/trim operations produce incorrect output from 16-bit RGBA input Apr 12, 2019

lovell added a commit to lovell/sharp-libvips that referenced this issue Sep 22, 2019

Build ARM64 binaries using gcc 9 (Debian 11)

2e4202d

See lovell/sharp#1651

lovell added blocked-upstream-dependency and removed triage labels Sep 22, 2019

lovell added this to the v0.24.0 milestone Sep 22, 2019

lovell added the bug label Sep 22, 2019

lovell added ready-to-ship and removed blocked-upstream-dependency labels Dec 30, 2019

lovell closed this as completed Jan 16, 2020

narFnarF mentioned this issue May 17, 2020

vips_foreign_heif_compression_get_type error on Raspberry Pi #2210

Closed

milkcocoa mentioned this issue Feb 2, 2021

Build linux-arm64v8 with Ubuntu Bionic lovell/sharp-libvips#86

Closed

ARM64: flatten/trim operations produce incorrect output from 16-bit RGBA input #1651

ARM64: flatten/trim operations produce incorrect output from 16-bit RGBA input #1651

Comments

ossdev07 commented Apr 8, 2019

ossdev07 commented Apr 9, 2019

lovell commented Apr 10, 2019

ossdev07 commented Apr 11, 2019

lovell commented Apr 11, 2019

ossdev07 commented Apr 12, 2019

ossdev07 commented Apr 15, 2019

lovell commented Apr 15, 2019

ossdev07 commented Apr 16, 2019

ossdev07 commented Apr 24, 2019

ossdev07 commented Jul 24, 2019

lovell commented Jul 25, 2019

ossdev07 commented Jul 25, 2019

ossdev07 commented Jul 26, 2019

lovell commented Jul 26, 2019

ossdev07 commented Jul 26, 2019

lovell commented Jul 26, 2019

ossdev07 commented Jul 26, 2019

rhenwood-arm commented Jul 26, 2019

ossdev07 commented Jul 28, 2019 • edited Loading

lovell commented Sep 22, 2019

pruthvireddypuresoftware commented Sep 25, 2019

pruthvireddypuresoftware commented Sep 25, 2019 • edited Loading

lovell commented Sep 25, 2019

ossdev07 commented Sep 30, 2019

lovell commented Oct 1, 2019

ossdev07 commented Oct 7, 2019

lovell commented Oct 7, 2019

ossdev07 commented Oct 7, 2019 • edited Loading

lovell commented Oct 7, 2019

ossdev07 commented Oct 7, 2019 • edited Loading

lovell commented Oct 8, 2019

ossdev07 commented Oct 10, 2019

lovell commented Oct 12, 2019

ossdev07 commented Oct 15, 2019

lovell commented Oct 16, 2019

ossdev07 commented Oct 16, 2019

lovell commented Oct 25, 2019

lovell commented Nov 7, 2019

lovell commented Dec 23, 2019

ossdev07 commented Dec 23, 2019

ossdev07 commented Dec 24, 2019

lovell commented Dec 26, 2019

lovell commented Dec 29, 2019

ossdev07 commented Dec 30, 2019

lovell commented Dec 30, 2019

lovell commented Jan 16, 2020

ossdev07 commented Jul 28, 2019 •

edited

Loading

pruthvireddypuresoftware commented Sep 25, 2019 •

edited

Loading

ossdev07 commented Oct 7, 2019 •

edited

Loading

ossdev07 commented Oct 7, 2019 •

edited

Loading