glLinkProgram crashes on Adreno 3xx #3640

paulorrfilho · 2021-03-11T23:13:46Z

Describe the bug
Lit-cube app crashes on start.
Hello-triangle example shows a very similar logcat and it does not crash but it does not render anything. Every other example behaves the same way.
Surprisingly, the transparent view example is the only one that works, some dropped frames though.

To Reproduce
Run the stock lit-cube example

Expected behavior
Cube rendered

Smartphone (please complete the following information):

Device: Moto E5
OS: Android 8.0
GPU: Adreno 308

Additional context
Logcat lit-cube:
2021-03-11 18:00:54.566 2510-2510/? I/zygote: Late-enabling -Xcheck:jni
2021-03-11 18:00:55.037 2510-2510/com.google.android.filament.litcube I/Filament: FEngine (32 bits) created at 0x94200000 (threading is enabled)
2021-03-11 18:00:55.038 2510-2567/com.google.android.filament.litcube D/Filament: FEngine resolved backend: OpenGL
2021-03-11 18:00:55.039 2510-2567/com.google.android.filament.litcube I/Adreno-EGL: <qeglDrvAPI_eglInitialize:379>: QUALCOMM Build: 02/27/18, 3273d9c, I0b59f3a7cf
2021-03-11 18:00:55.149 2510-2569/com.google.android.filament.litcube D/OpenGLRenderer: HWUI GL Pipeline
2021-03-11 18:00:55.189 2510-2569/com.google.android.filament.litcube I/OpenGLRenderer: Initialized EGL, version 1.4
2021-03-11 18:00:55.189 2510-2569/com.google.android.filament.litcube D/OpenGLRenderer: Swap behavior 2
2021-03-11 18:00:55.194 2510-2569/com.google.android.filament.litcube W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-11 18:00:55.579 2510-2567/com.google.android.filament.litcube W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-11 18:00:55.641 2510-2567/com.google.android.filament.litcube A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0x60 in tid 2567 (FEngine::loop)

Logcat hello-triangle:
2021-03-11 18:07:27.444 2770-2770/? I/zygote: Late-enabling -Xcheck:jni
2021-03-11 18:07:27.851 2770-2770/com.google.android.filament.hellotriangle I/Filament: FEngine (32 bits) created at 0x94200000 (threading is enabled)
2021-03-11 18:07:27.853 2770-2797/com.google.android.filament.hellotriangle D/Filament: FEngine resolved backend: OpenGL
2021-03-11 18:07:27.854 2770-2797/com.google.android.filament.hellotriangle I/Adreno-EGL: <qeglDrvAPI_eglInitialize:379>: QUALCOMM Build: 02/27/18, 3273d9c, I0b59f3a7cf
2021-03-11 18:07:27.965 2770-2799/com.google.android.filament.hellotriangle D/OpenGLRenderer: HWUI GL Pipeline
2021-03-11 18:07:28.011 2770-2799/com.google.android.filament.hellotriangle I/OpenGLRenderer: Initialized EGL, version 1.4
2021-03-11 18:07:28.011 2770-2799/com.google.android.filament.hellotriangle D/OpenGLRenderer: Swap behavior 2
2021-03-11 18:07:28.017 2770-2799/com.google.android.filament.hellotriangle W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-11 18:07:28.241 2770-2797/com.google.android.filament.hellotriangle W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-11 18:07:32.749 2770-2775/com.google.android.filament.hellotriangle I/zygote: Do partial code cache collection, code=31KB, data=25KB
2021-03-11 18:07:32.750 2770-2775/com.google.android.filament.hellotriangle I/zygote: After code cache collection, code=31KB, data=25KB
2021-03-11 18:07:32.751 2770-2775/com.google.android.filament.hellotriangle I/zygote: Increasing code cache capacity to 128KB

romainguy · 2021-03-11T23:16:41Z

Thank you. Adreno 30x GPUs have old drivers that we know had many issues with OpenGL ES 3.0 support. According to those traces we can't really go past the driver initialization.

paulorrfilho · 2021-03-11T23:25:31Z

Yes, I noticed that the Adreno 30x series is buggy based on the issues, just find it interesting that one of the examples work, so there is path.

paulorrfilho · 2021-03-12T04:56:06Z

Ok, I took a look at the Android Play Store device catalog, and in my opinion, one should assume that Filament supports SDK 24 and up, by exclusion. It is the cut-off SDK that removes all these crappy Adreno GPUs and still supports decent older devices. The Adreno 30x family dominates the device list under that SDK.
I have the data in Excel, so if you are interested please let me know.

pixelflinger · 2021-03-14T07:25:54Z

I don't have a device I can test. But I will add some debug code so we at least get better logging than a crash on the driver. We'll go from there.

pixelflinger · 2021-03-15T18:41:35Z

Actually I thought the problem was because of the EGL_BAD_ATTRIBUTE, but it's not.
There should be more info in the log after
2021-03-11 18:00:55.641 2510-2567/com.google.android.filament.litcube A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0x60 in tid 2567 (FEngine::loop)

Please attach the section of the log with the crash report.

paulorrfilho · 2021-03-15T20:37:37Z

Sorry, sometimes the logcat does not show the whole crash.

`2021-03-15 15:35:35.318 14245-14245/? D/AndroidRuntime: >>>>>> START com.android.internal.os.RuntimeInit uid 2000 <<<<<<
2021-03-15 15:35:35.375 14265-14265/com.google.android.filament.litcube I/Filament: FEngine (32 bits) created at 0x94100000 (threading is enabled)
2021-03-15 15:35:35.376 14265-14293/com.google.android.filament.litcube D/Filament: FEngine resolved backend: OpenGL
2021-03-15 15:35:35.377 14265-14293/com.google.android.filament.litcube I/Adreno-EGL: <qeglDrvAPI_eglInitialize:379>: QUALCOMM Build: 02/27/18, 3273d9c, I0b59f3a7cf
2021-03-15 15:35:35.526 14265-14295/com.google.android.filament.litcube D/OpenGLRenderer: HWUI GL Pipeline
2021-03-15 15:35:35.604 14245-14245/? D/AndroidRuntime: Calling main entry com.android.commands.wm.Wm
2021-03-15 15:35:35.608 14265-14295/com.google.android.filament.litcube I/OpenGLRenderer: Initialized EGL, version 1.4
2021-03-15 15:35:35.608 14265-14295/com.google.android.filament.litcube D/OpenGLRenderer: Swap behavior 2
2021-03-15 15:35:35.608 2403-6546/? W/WindowManager: Permission Denial: dismissKeyguard from pid=14245, uid=2000 requires android.permission.CONTROL_KEYGUARD
2021-03-15 15:35:35.609 14245-14245/? D/AndroidRuntime: Shutting down VM
2021-03-15 15:35:35.614 14265-14295/com.google.android.filament.litcube W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-15 15:35:35.974 14265-14293/com.google.android.filament.litcube W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-15 15:35:36.020 2403-2590/? I/LaunchCheckinHandler: Displayed com.google.android.filament.litcube/.MainActivity,cp,ca,1103
2021-03-15 15:35:36.020 2403-2590/? I/ActivityManager: Displayed com.google.android.filament.litcube/.MainActivity: +1s85ms

--------- beginning of crash

2021-03-15 15:35:36.037 14265-14293/com.google.android.filament.litcube A/libc: Fatal signal 11 (SIGSEGV), code 1, fault addr 0x60 in tid 14293 (FEngine::loop)
2021-03-15 15:35:36.061 27639-13281/? I/PBSessionCacheImpl: Deleted sessionId[121559223124468769] from persistence.
2021-03-15 15:35:36.107 27639-27864/? W/SearchServiceCore: Abort, client detached.
2021-03-15 15:35:36.108 27639-27864/? I/chatty: uid=10067(u0_a67) EventBus0 identical 1 line
2021-03-15 15:35:36.111 27639-27864/? W/SearchServiceCore: Abort, client detached.
2021-03-15 15:35:36.119 14312-14312/? I/crash_dump32: obtaining output fd from tombstoned
2021-03-15 15:35:36.119 2122-2122/? I//system/bin/tombstoned: received crash request for pid 14265
2021-03-15 15:35:36.121 14312-14312/? I/crash_dump32: performing dump of process 14265 (target tid = 14293)
2021-03-15 15:35:36.121 14312-14312/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: Build fingerprint: 'motorola/nora_row/nora_8917:8.0.0/OPPS27.91-176-11-16/20:user/release-keys'
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: Revision: 'p400'
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: ABI: 'arm'
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: pid: 14265, tid: 14293, name: FEngine::loop >>> com.google.android.filament.litcube <<<
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x60
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: Cause: null pointer dereference
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: r0 00000001 r1 9f9952dd r2 00000000 r3 929597a8
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: r4 00000030 r5 00000000 r6 00000006 r7 93506330
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: r8 9288a808 r9 92888d64 sl 92888ec4 fp 92dde4b8
2021-03-15 15:35:36.122 14312-14312/? A/DEBUG: ip 00000000 sp 93506160 lr 9f9a3083 pc 9f9a309c cpsr 60070030
2021-03-15 15:35:36.262 14312-14312/? A/DEBUG: backtrace:
2021-03-15 15:35:36.262 14312-14312/? A/DEBUG: #00 pc 0004209c /vendor/lib/libsc-a3xx.so (_ZN8Compiler7CompileEP9ILProgram+28475)
2021-03-15 15:35:36.262 14312-14312/? A/DEBUG: #1 pc 00047911 /vendor/lib/libsc-a3xx.so (_ZN8Compiler13CompileShaderEPhS0_PKjP16CompilerExternal+228)
2021-03-15 15:35:36.262 14312-14312/? A/DEBUG: #2 pc 001d023b /vendor/lib/libsc-a3xx.so (_ZN8sclState4linkEPK18sclInputShaderPairRK17sclCompilerParams+15898)
2021-03-15 15:35:36.263 14312-14312/? A/DEBUG: #3 pc 001d440f /vendor/lib/libsc-a3xx.so (__link_shaders+330)
2021-03-15 15:35:36.263 14312-14312/? A/DEBUG: #4 pc 00095ae5 /vendor/lib/egl/libRBGLESv2_adreno.so (core_glLinkProgram+1404)
2021-03-15 15:35:36.263 14312-14312/? A/DEBUG: #5 pc 000db38d /data/app/com.google.android.filament.litcube-H1uylqvsnvynlw8G0bKHxg==/lib/arm/libfilament-jni.so
2021-03-15 15:35:37.677 2122-2122/? E//system/bin/tombstoned: Tombstone written to: /data/tombstones//tombstone_03
2021-03-15 15:35:37.679 2403-14314/? W/ActivityManager: Force finishing activity com.google.android.filament.litcube/.MainActivity
2021-03-15 15:35:37.683 2403-2585/? I/BootReceiver: Copying /data/tombstones/tombstone_03 to DropBox (SYSTEM_TOMBSTONE)
2021-03-15 15:35:37.689 2403-2528/? I/ActivityManager: Showing crash dialog for package com.google.android.filament.litcube u0
2021-03-15 15:35:37.698 2403-2523/? W/BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.stats.service.DropBoxEntryAddedReceiver
2021-03-15 15:35:37.698 2403-2523/? W/BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.chimera.GmsIntentOperationService$PersistentTrustedReceiver
2021-03-15 15:35:37.699 563-563/? E/lowmemorykiller: Error writing /proc/14265/oom_score_adj; errno=22
2021-03-15 15:35:37.746 570-570/? D/SurfaceFlinger: duplicate layer name: changing Application Error: com.google.android.filament.litcube to Application Error: com.google.android.filament.litcube#4
2021-03-15 15:35:37.755 570-1630/? E/SurfaceFlinger: Failed to find layer (SurfaceView - com.google.android.filament.litcube/com.google.android.filament.litcube.MainActivity#0) in layer parent (no-parent).
2021-03-15 15:35:37.755 570-1630/? E/SurfaceFlinger: Failed to find layer (Background for - SurfaceView - com.google.android.filament.litcube/com.google.android.filament.litcube.MainActivity#0) in layer parent (no-parent).
2021-03-15 15:35:37.756 2403-20688/? I/ActivityManager: Process com.google.android.filament.litcube (pid 14265) has died: vis +99TOP
2021-03-15 15:35:37.761 2403-23791/? I/WindowManager: WIN DEATH: Window{95c083 u0 com.google.android.filament.litcube/com.google.android.filament.litcube.MainActivity}
2021-03-15 15:35:37.771 2403-9686/? W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-15 15:35:37.774 570-690/? E/SurfaceFlinger: Failed to find layer (com.google.android.filament.litcube/com.google.android.filament.litcube.MainActivity#0) in layer parent (no-parent).
2021-03-15 15:35:37.783 2026-2026/? I/Zygote: Process 14265 exited due to signal (11)
2021-03-15 15:35:37.794 2403-20688/? I/ActivityManager: moveHomeStack, setupComplete:true
2021-03-15 15:35:37.807 2403-2590/? W/ActivityManager: setHasOverlayUi called on unknown pid: 14265
2021-03-15 15:35:37.808 2403-2523/? W/BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.stats.service.DropBoxEntryAddedReceiver
2021-03-15 15:35:37.809 2403-2523/? W/BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.chimera.GmsIntentOperationService$PersistentTrustedReceiver
2021-03-15 15:35:37.841 27639-965/? W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-15 15:35:37.895 528-14323/? I/Thermal-Lib: Thermal-Lib-Client: Client request sent
2021-03-15 15:35:37.896 2023-2096/? I/ThermalEngine: Thermal-Server: Thermal received msg from override
2021-03-15 15:35:37.898 3269-3285/? E/RecentsTaskLoader: Unexpected null component name or activity info: ComponentInfo{com.google.android.apps.messaging/com.google.android.apps.messaging.ui.conversationlist.ConversationListActivity}, null
2021-03-15 15:35:37.898 570-570/? I/SFPerfTracer: triggers: (rate: 65:9372) (compose: 0:11) (post: 0:41) (render: 0:76) (2:106724 frames) (3:125265)
2021-03-15 15:35:37.898 570-570/? D/SFPerfTracer: layers: (9:18) (StatusBar#0 (0xadc9e000): 0:13843) (RoundedOverlay#0 (0xadd27000): 0:845) (RoundedOverlay#1 (0xadca6000): 0:853) (com.android.systemui.ImageWallpaper#0 (0xadd31000): 0:1555)* (DimLayerController/Stack=0#0 (0xadca9000): 0:1573) (animation background stackId=1#0 (0xadc88000): 0:374)* (Sprite#0 (0xadcc9000): 0:8)* (NavigationBar#0 (0xadcbf000): 0:624) (Application Error: com.google.android.filament.litcube#0 (0xadd42000): 0:16) (Application Error: com.google.android.filament.litcube#1 (0xadcec000): 0:16) (Application Error: com.google.android.filament.litcube#2 (0xadcc5000): 0:12) (com.motorola.launcher3/com.android.launcher3.CustomizationPanelLauncher#0 (0xadccb000): 0:8)- (com.motorola.launcher3/com.android.launcher3.CustomizationPanelLauncher#1 (0xadcd1000): 0:14)- (Application Error: com.google.android.filament.litcube#3 (0xadcd3000): 0:17) (Splash Screen com.google.android.filament.litcube#0 (0xadce9000): 0:31)-
2021-03-15 15:35:37.898 3269-3285/? I/chatty: uid=10066(u0_a66) Binder:3269_1 identical 1 line
2021-03-15 15:35:37.899 3269-3285/? E/RecentsTaskLoader: Unexpected null component name or activity info: ComponentInfo{com.google.android.apps.messaging/com.google.android.apps.messaging.ui.conversationlist.ConversationListActivity}, null
2021-03-15 15:35:37.915 570-570/? D/SurfaceFlinger: duplicate layer name: changing com.motorola.launcher3/com.android.launcher3.CustomizationPanelLauncher to com.motorola.launcher3/com.android.launcher3.CustomizationPanelLauncher#1
2021-03-15 15:35:37.927 3946-8450/? W/Adreno-EGL: <qeglDrvAPI_eglGetConfigAttrib:612>: EGL_BAD_ATTRIBUTE
2021-03-15 15:35:37.972 27639-27864/? W/ThreadPoolDumper: Queue length for executor EventBus is now 11. Perhaps some tasks are too long, or the pool is too small.
2021-03-15 15:35:38.034 2403-2515/? W/MayaPerformanceService: resetBlockRules no packageName,ui:99008
2021-03-15 15:35:38.050 27639-27639/? I/BgTaskExecutorImpl: Starting EXCLUSIVE background task TNG_MINUS_ONE_SYNC.
2021-03-15 15:35:38.052 27639-27639/? I/BgTaskExecutorImpl: Starting EXCLUSIVE background task TNG_VOICE_SEARCH_SYNC.
2021-03-15 15:35:38.057 27639-27864/? I/TngMinusOneSync: Syncing TNG:-1
2021-03-15 15:35:38.060 27639-27864/? I/TngVoiceSearchSync: Syncing TNG:VoiceSearch
2021-03-15 15:35:38.260 27639-27652/? I/zygote: Background concurrent copying GC freed 98697(4MB) AllocSpace objects, 9(216KB) LOS objects, 49% free, 14MB/28MB, paused 327us total 180.234ms
2021-03-15 15:35:38.456 3946-8450/? W/OpenGLRenderer: Incorrectly called buildLayer on View: ShortcutAndWidgetContainer, destroying layer...
2021-03-15 15:35:38.457 3946-8450/? W/OpenGLRenderer: Incorrectly called buildLayer on View: ShortcutAndWidgetContainer, destroying layer...
2021-03-15 15:35:40.498 3418-3418/? D/QtiCarrierConfigHelper: WARNING, no carrier configs on phone Id: 0
2021-03-15 15:35:40.499 3751-4020/? D/HeadsetStateMachine: Disconnected process message: 10, size: 0
2021-03-15 15:35:40.499 3751-4020/? D/HeadsetStateMachine: Enter processIntentBatteryChanged()
2021-03-15 15:35:40.499 3751-4020/? D/HeadsetStateMachine: Exit processIntentBatteryChanged()
2021-03-15 15:35:40.499 3751-4020/? D/HeadsetStateMachine: Exit Disconnected processMessage()
2021-03-15 15:35:40.507 2091-3527/? V/LocSvc_HIDL_IzatProvider: [onLocationChanged][67] [HS] =>>>> [HC]
2021-03-15 15:35:40.508 2403-3140/? V/LocSvc_HIDL_IzatProvider_jni: [onLocationChanged][136] [HC] <<= [HS]
2021-03-15 15:35:40.561 570-570/? D/SurfaceFlinger: Backpressure trigger, skipping transaction & refresh!`

romainguy · 2021-03-15T20:49:45Z

Looks like a crash when linking a GLSL program.

pixelflinger · 2021-03-15T21:36:35Z

I'm afraid there's not much we can do here :-\

paulorrfilho · 2021-03-15T22:39:24Z

Thank you, I don't think it makes sense to try to fix these issues at this point. Anyway, I will require SDK 24 and up to download my app as it will remove 70-80% of the bad devices.

paulorrfilho · 2021-03-17T02:01:09Z

Ok, some interesting findings after some testing:

Filament does work with Adreno 3xx devices. My app uses the pre-built Filament libraries from mavenCentral and runs fine.
The crashes only happen when I build Filament myself and run the samples. The weird thing is that these same samples run fine in other devices.

paulorrfilho · 2021-03-17T02:22:35Z

I am building Filament using this command ./build.sh -p desktop,android -i release
I am afraid there is some sort of inconsistency between pre-built library and locally built.

bejado · 2021-03-17T17:22:53Z

The pre-built libraries are also built with the -v flag to turn off Vulkan. I don't know why that would affect OpenGL though.

hzzengxiaoqi · 2022-12-05T10:05:15Z

I recently met a glProgram crash problem on an android phone(vivo Y71A, Qualcomm, Adreno (TM) 308).
The backtrace is the same as paulorrfilho.
And I tried to fix it. If anyone met the same problem, you can try my method.

The glsl shader code(fragment shader in my case) that caused the crash is like this:

do{
if(xx){xx;break;}
else{xx;break;}
}while(false);

To fix it, just put another break at the end of the while loop. like this:

do{
if(xx){xx;break;}
else{xx;break;}
break;
}while(false);

In my case I used UbershaderLoader，so the shader is generated using matc tool, you need to change the source code of spirv_glsl.cpp
in spirv_glsl.cpp, search this:

end_scope_decl("while(false)");

before it, add

statement("break;");

and they should be within the same "if" control block of course.

and then build matc again. I turned to build an matc project myself, but I think you can just generate the tools again :

./build.sh -p desktop -i debug

At last, use the new tools to generate the materials.

romainguy added android Issue/feature request for Android only bug Something isn't working gpu specific This bug occurs only with a single GPU vendor. opengl Issue/request specific to OpenGL labels Mar 11, 2021

pixelflinger self-assigned this Mar 14, 2021

pixelflinger changed the title ~~EGL_BAD_ATTRIBUTE and no rendering~~ glLinkProgram crashes on Adreno 3xx Mar 15, 2021

pixelflinger closed this as completed Mar 16, 2021

joined72 mentioned this issue Feb 11, 2024

Fix Compatibility Rendering (GLES3) on old and low budget devices. godotengine/godot#87352

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

glLinkProgram crashes on Adreno 3xx #3640

glLinkProgram crashes on Adreno 3xx #3640

paulorrfilho commented Mar 11, 2021

romainguy commented Mar 11, 2021

paulorrfilho commented Mar 11, 2021

paulorrfilho commented Mar 12, 2021

pixelflinger commented Mar 14, 2021

pixelflinger commented Mar 15, 2021

paulorrfilho commented Mar 15, 2021

romainguy commented Mar 15, 2021

pixelflinger commented Mar 15, 2021

paulorrfilho commented Mar 15, 2021

paulorrfilho commented Mar 17, 2021

paulorrfilho commented Mar 17, 2021

bejado commented Mar 17, 2021

hzzengxiaoqi commented Dec 5, 2022 •

edited

Loading

glLinkProgram crashes on Adreno 3xx #3640

glLinkProgram crashes on Adreno 3xx #3640

Comments

paulorrfilho commented Mar 11, 2021

romainguy commented Mar 11, 2021

paulorrfilho commented Mar 11, 2021

paulorrfilho commented Mar 12, 2021

pixelflinger commented Mar 14, 2021

pixelflinger commented Mar 15, 2021

paulorrfilho commented Mar 15, 2021

romainguy commented Mar 15, 2021

pixelflinger commented Mar 15, 2021

paulorrfilho commented Mar 15, 2021

paulorrfilho commented Mar 17, 2021

paulorrfilho commented Mar 17, 2021

bejado commented Mar 17, 2021

hzzengxiaoqi commented Dec 5, 2022 • edited Loading

hzzengxiaoqi commented Dec 5, 2022 •

edited

Loading