Memory leak of Tier 1 handle when OpenJPEG fails to set it as TLS...

[sebras]

While testing what happens when opj_calloc() returns NULL using a custom allocator in OpenJPEG 2.3.0 I found a memory leak that doesn't seem to have been addressed in OpenJPEG on master.

The relevant part of backtrace for the memory leak is :

    #0 0x55aae2b06223 in opj_calloc
    #1 0x55aae347aad8 in opj_t1_create openjpeg/src/lib/openjp2/t1.c:1541
    #2 0x55aae347b4b8 in opj_t1_clbl_decode_processor openjpeg/src/lib/openjp2/t1.c:1660
    #3 0x55aae340a8f3 in opj_thread_pool_submit_job openjpeg/src/lib/openjp2/thread.c:835
    #4 0x55aae347da25 in opj_t1_decode_cblks openjpeg/src/lib/openjp2/t1.c:1908
    #5 0x55aae3400fa0 in opj_tcd_t1_decode openjpeg/src/lib/openjp2/tcd.c:1985
    #6 0x55aae33fedf4 in opj_tcd_decode_tile openjpeg/src/lib/openjp2/tcd.c:1639
    #7 0x55aae33b82d6 in opj_j2k_decode_tile openjpeg/src/lib/openjp2/j2k.c:8947
    #8 0x55aae33c51f7 in opj_j2k_decode_tiles openjpeg/src/lib/openjp2/j2k.c:10735
    #9 0x55aae33b28fd in opj_j2k_exec openjpeg/src/lib/openjp2/j2k.c:8105
    #10 0x55aae33c7dae in opj_j2k_decode openjpeg/src/lib/openjp2/j2k.c:11081
    #11 0x55aae316517d in opj_decode openjpeg/src/lib/openjp2/openjpeg.c:483

This is the backtrace of where I had opj_realloc() return NULL later causing the leak above:

#0  0x0000555555833281 in opj_realloc (ptr=0x0, size=24) at source/fitz/load-jpx.c:573
#1  0x000055555613663a in opj_tls_set (tls=0x603000004518, key=0, value=0x612000000948, opj_free_func=0x5555561a7cec <opj_t1_destroy_wrapper>) at openjpeg/src/lib/openjp2/thread.c:559
#2  0x00005555561a84de in opj_t1_clbl_decode_processor (user_data=0x608000009228, tls=0x603000004518) at openjpeg/src/lib/openjp2/t1.c:1661
#3  0x00005555561378f4 in opj_thread_pool_submit_job (tp=0x6080000091a8, job_fn=0x5555561a7d07 <opj_t1_clbl_decode_processor>, user_data=0x608000009228) at openjpeg/src/lib/openjp2/thread.c:835
#4  0x00005555561aa944 in opj_t1_decode_cblks (tcd=0x60b000000258, pret=0x7fffffff9820, tilec=0x613000001e08, tccp=0x61f000007088,  p_manager=0x6100000005a8, p_manager_mutex=0x0, check_pterm=0) at openjpeg/src/lib/openjp2/t1.c:1901
#5 0x000055555612dfa1 in opj_tcd_t1_decode (p_tcd=0x60b000000258, p_manager=0x6100000005a8) at openjpeg/src/lib/openjp2/tcd.c:1985
#6 0x000055555612bdf5 in opj_tcd_decode_tile (p_tcd=0x60b000000258, win_x0=0, win_y0=0, win_x1=640, win_y1=480, numcomps_to_decode=0, comps_indices=0x0, p_src=0x7ffff3e9f808 "H\241\343\345\355\213\035\211\031\210ZU\357T0R\355\363Y\376\242\355(X\256\071\177V\003e\276ڬ%\322g\357\032\351\016\244\065\357\254\027ߠcL\223P\366\253L\026\372\r\033\372e\237U\242\210\216\233w {\b\275\377\n\257\306\376\202[\247\v%\243\327\034\200\216n*\210\f\004\026\004:\nE\352il}\236\254\275\356\211\327\346\273\373\210\t\343\031ki\261\220\223Lnoè\234\065\373\003~yd,E\002C\034\223\366E)\313\346@iD\343\027k\244J\350n\331\356<]b\356\253\023\066\262 \357\067\313\336\035i\301(\250\022\006^!w\366J\033\320\016vIN\216\317\316\032\023L`\362\306ȧZ\243\312\305Y"..., p_max_length=259641, p_tile_no=0, p_cstr_index=0x606000007a08, p_manager=0x6100000005a8) at openjpeg/src/lib/openjp2/tcd.c:1639
#7 0x00005555560e52d7 in opj_j2k_decode_tile (p_j2k=0x613000001548, p_tile_index=0, p_data=0x0, p_data_size=0, p_stream=0x60c000008808, p_manager=0x6100000005a8) at openjpeg/src/lib/openjp2/j2k.c:8947
#8 0x00005555560f21f8 in opj_j2k_decode_tiles (p_j2k=0x613000001548, p_stream=0x60c000008808, p_manager=0x6100000005a8) at openjpeg/src/lib/openjp2/j2k.c:10735
#9 0x00005555560df8fe in opj_j2k_exec (p_j2k=0x613000001548, p_procedure_list=0x6030000044e8, p_stream=0x60c000008808, p_manager=0x6100000005a8) at openjpeg/src/lib/openjp2/j2k.c:8105
#10 0x00005555560f4daf in opj_j2k_decode (p_j2k=0x613000001548, p_stream=0x60c000008808, p_image=0x606000007ac8, p_manager=0x6100000005a8) at openjpeg/src/lib/openjp2/j2k.c:11081
#11 0x0000555555e9217e in opj_decode (p_codec=0x610000000548,  p_stream=0x60c000008808, p_image=0x606000007ac8) at openjpeg/src/lib/openjp2/openjpeg.c:483

Looking at the source code it is obvious that the return code of opj_tls_set() is never checked, hence the code can't know it should free the t1 handle and return an error code.