Skip to content

Conversation

@milancurcic
Copy link
Member

First attempt at defining the concrete optimizer procedure as a method of SGD optimizer type.

Currently defining the minimize subroutine as elemental to allow a scalar/array/rank-agnostic interface. It's possible that this won't work for all cases if we discover new requirements but let's try this for the time being.

@milancurcic milancurcic marked this pull request as ready for review June 21, 2023 20:55
@milancurcic milancurcic added the enhancement New feature or request label Jun 21, 2023
@milancurcic milancurcic requested a review from Spnetic-5 June 21, 2023 20:56
@milancurcic
Copy link
Member Author

This now works. There's an API change to the network % train() and network % update() methods which now require an argument of class(optimizer_base_type). (I wonder if it's possible to make this optional so we can default to sgd).

Once an optimizer is passed to network % update, it's passed to layer % update() for all layers. In layer % update, the weights and biases are accessed from the internal layer representation and passed to optimizer % minimize(). I borrowed the name minimize from Keras. optimizer % optimize() would be appropriate but sounds weird due to repetition. How about optimizer % update()?

The optimizer step for conv2d is currently not implemented but it may be easy to do so even in this PR (but convolutional training is broken anyway as explained in #142).

@Spnetic-5
Copy link
Collaborator

Spnetic-5 commented Jun 22, 2023

Thanks for bringing up the API change regarding the network methods. Making the optimizer argument optional and setting up SGD as default sounds like a good idea.

Regarding the naming, I think optimizer % minimize() is good, as it captures the essence of the operation. Furthermore, I'll study all the updations in the code.

@milancurcic milancurcic merged commit 31fc061 into modern-fortran:main Jun 22, 2023
@milancurcic milancurcic deleted the sgd-optimizer-stub branch June 22, 2023 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants