-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How can I confirm that the deterministic environment variable is working? #2
Comments
Hi @cklsoft, there is currently nothing printed to the logs by TensorFlow that confirms that the environment variable has been acted upon. I have made a note to potentially add this in the future. |
Will |
That's a great idea. Yes, if you observe the number of graph nodes when running with Note that |
The ultimate test is whether your weights at the end of training change from run to run. For Keras models, you can call the following at the end of training, and make sure it produces the same result on two consecutive runs:
If you're not using Keras, it would look something like this:
It's also good to confirm that your weights are the same, on both runs, before training starts. Please note that while the above code is based on code I've used, the code as given above has not been tested. It may contain bugs and/or may not work on more recent versions of TensorFlow or Keras. |
This question has been answered, and there is nothing else to be done here. Closing. |
I've use the newest NGC container, and specify
os.environ['TF_DETERMINISTIC_OPS'] = '1'
at the begin of mymain
entry. But I don't know wether the environ is work or not. I have set tf log level to debug and didn't find any tf determinism relate log.The text was updated successfully, but these errors were encountered: