Gradient accumulation not properly implemented

[clemsgrs]

Hi, based on the following lines, it seems gradient accumulation is not properly implemented:

HIPT/2-Weakly-Supervised-Subtyping/utils/core_utils.py

Lines 285 to 290 in a9b5bb8

	loss = loss / gc
	loss.backward()
	# step
	optimizer.step()
	optimizer.zero_grad()

A proper implementation should look like the following:

loss = loss / gc
loss.backward()

if (batch_idx + 1) % gc == 0:
      optimizer.step()
      optimizer.zero_grad()

[vildesboe]

Hi, based on the following lines, it seems gradient accumulation is not properly implemented:

HIPT/2-Weakly-Supervised-Subtyping/utils/core_utils.py

Lines 285 to 290 in a9b5bb8

	loss = loss / gc
	loss.backward()
	# step
	optimizer.step()
	optimizer.zero_grad()

A proper implementation should look like the following:

loss = loss / gc
loss.backward()

if (batch_idx + 1) % gc == 0:
      optimizer.step()
      optimizer.zero_grad()

Hi! I'm also working on reproducing this HIPT paper. Would you be interested in some discussion?

[clemsgrs]

sure, happy to chat.
I’ve made my own version of the code here: https://github.com/clemsgrs/hipt

you can contact me at: clement (dot) grisi (at) radboudumc (dot) nl