Log console output during replay to file #985 #76

cogutvalera · 2018-09-17T06:50:19Z

PR for "Log console output during replay to file #985 bitshares/bitshares-core#985"

abitmore · 2018-09-17T14:53:01Z

Can you please briefly explain why change like this? There is no comment in your code.

cogutvalera · 2018-09-17T15:31:18Z

I can add comments right into code if required.
After debugging and researching code I've found the reason of this issue.

During replay we can see logs in console but not in file because of next reason: fc::file_appender::impl::out (ofstream) is opened after rotate_files.
So this means that ofstream will be opened after replay is completed and finished. Thus after replay we can see the same logs in console and file.
Now after my changes ofstream is opened before replay happens. So this solution makes possible to see logs not only in console during replay but also we can see the same logs in file according to configuration in config.ini or logging.ini.

jmjatlanta · 2018-09-17T20:10:01Z

src/log/file_appender.cpp

-         if(!my->cfg.rotate)
-            my->out.open( my->cfg.filename, std::ios_base::out | std::ios_base::app);
+         {
+            fc::scoped_lock<boost::mutex> lock( my->slock );


As this is a constructor, I do not see the need for a lock here.

we have different threads, that's why I've added lock here, IMHO slock mutex is supposed to be used for log files within different threads, maybe I'm wrong.

Thanks !

jmjatlanta · 2018-09-17T20:14:52Z

src/log/file_appender.cpp

@@ -141,9 +160,11 @@ namespace fc {
      {
         fc::create_directories(my->cfg.filename.parent_path());

-         if(!my->cfg.rotate)


I haven't walked all of the code, but if rotate == true, will removing this cause open() to be called twice (once here, and once in rotate_files()) ?

if rotate check was removed, and twice won't be open because there is a check in open_log_filename method

abitmore

Need better abstraction / modularization.

abitmore · 2018-09-18T20:40:10Z

src/log/file_appender.cpp

@@ -57,21 +69,30 @@ namespace fc {
            }
         }

+         void open_log_filename()


This function name is confusing. I'd recommend rename it to open_log_file() or something else that can describe the logic better.

In addition, the check for file name inside the function made it even harder to understand, I think it would be better to move out the check, e.g. have an open() simply opens a file or perhaps close then reopen, and only call it when necessary (aka with a check).

In short, try to simplify the code.

abitmore · 2018-09-18T20:43:32Z

src/log/file_appender.cpp

+         fc::time_point             _now;
+         fc::time_point_sec         _start_time;
+         string                     _timestamp_string;
+         fc::path                   _old_log_filename, _log_filename, _link_filename;


For member variables, please declare only one variable per line.

abitmore · 2018-09-18T20:58:18Z

src/log/file_appender.cpp

-      line << std::setw( 21 ) << (m.get_context().get_thread_name().substr(0,9) + string(":") + m.get_context().get_task_name()).c_str() << " ";
+      line << std::setw( 21 ) << 
+         (m.get_context().get_thread_name().substr(0,9) + string(":") + m.get_context().get_task_name()).c_str() 
+      << " ";


The code style here looks a bit ugly. I'd prefer

line << "something" << "something else" << "the end";

abitmore · 2018-09-18T21:20:28Z

src/log/file_appender.cpp

@@ -141,9 +160,7 @@ namespace fc {
      {
         fc::create_directories(my->cfg.filename.parent_path());


Why not move the whole logic into constructor of impl class for simplicity? All operation is done on my anyway.

I thought the same earlier, I agree it would be better ! Thanks !

cogutvalera · 2018-09-19T07:39:08Z

pushed new commit "more improvements"

Thanks !

pmconrad · 2018-09-19T15:55:29Z

So this means that ofstream will be opened after replay is completed and finished.

Why is this so?
file_appender::impl calls rotate_files asynchronously from the constructor. Either there's something wrong with the call, or the async excution happens only after replay. In the latter case, the obvious (and much simpler) fix would be to make the initial rotate_files call synchronous.

abitmore · 2018-09-19T16:27:10Z

src/log/file_appender.cpp

+            if( cfg.rotate )
+            {
+               FC_ASSERT( cfg.rotation_interval >= seconds( 1 ) );
+               FC_ASSERT( cfg.rotation_limit >= cfg.rotation_interval );


I think these assertions should be put before creating log file, even before setup_log_filenames.

abitmore · 2018-09-19T16:32:39Z

src/log/file_appender.cpp

+            }
+            catch( ... )
+            {
+               std::cerr << "error opening log file: " << cfg.filename.preferred_string() << "\n";


Is the appender usable after caught an exception here? What will happen when trying to write?

abitmore · 2018-09-19T16:36:58Z

src/log/file_appender.cpp

+                  _old_log_filename = _log_filename;
+
+                  open_log_file();
+               }


The logic seems a bit strange.

Is there a chance that the file got closed but not reopened?

Is there a chance that the file be opened again when not closed?

abitmore · 2018-09-19T17:04:17Z

the obvious (and much simpler) fix would be to make the initial rotate_files call synchronous.

Probably because rotate_files() also removes old files that could take some time, thus not good to call on initialization. However, even it's the case, we can skip the removal on first call.

By the way, I guess the rotate_files(1) call is defensive? Looks like it will never be called.

cogutvalera · 2018-09-21T07:58:00Z

(just for the info) I've pushed 2 new commits:

reverted all changes made before to current master branch
Log console output during replay to file #985 (2nd simpler solution)

pmconrad

If we decide to merge the second version, you should rebase -i to get rid of the first version. But not yet.

pmconrad · 2018-09-21T08:07:57Z

src/log/file_appender.cpp

@@ -137,18 +134,7 @@ namespace fc {
   file_appender::file_appender( const variant& args ) :
     my( new impl( args.as<config>( FC_MAX_LOG_OBJECT_DEPTH ) ) )
   {
-      try
-      {
-         fc::create_directories(my->cfg.filename.parent_path());


I think this must be moved to impl::impl(), not simply removed.

pmconrad · 2018-09-21T08:08:19Z

src/log/file_appender.cpp

-      {
-         fc::create_directories(my->cfg.filename.parent_path());
-
-         if(!my->cfg.rotate)


...and this shouldn't be removed.

cogutvalera · 2018-09-21T09:25:10Z

I've cleaned up my previous changes with 1st solution and forced new push commit

abitmore · 2018-09-21T16:24:15Z

@cogutvalera: Peter said "not yet". He mean you should NOT rebase by now. Then you did it regardless :-/

cogutvalera · 2018-09-21T16:34:37Z

If we decide to merge the second version, you should rebase -i to get rid of the first version. But not yet.

IMHO the 2nd solution is simpler and it works as required, then why to rebase "not yet" ?

pmconrad · 2018-09-24T18:31:13Z

src/log/file_appender.cpp


+            try
+            {
+               fc::create_directories(cfg.filename.parent_path());


create_directories should happen before rotate_files.

pmconrad · 2018-09-24T18:32:30Z

The idea was to preserve the original version in case my ideas didn't work. I'm ok with the rebase if it does work as intended.

pmconrad · 2018-09-24T20:37:12Z

Thanks! @abitmore @jmjatlanta please re-review.

jmjatlanta · 2018-09-25T18:19:19Z

There may be an edge case here that we may want to fix, but the chances of it happening are very remote. Scenario: Destructor called, then rotate_files gets the lock.

I am okay with that being there, as the file_appender is typically only destroyed at program exit, and the problem has probably been there from before this issue was opened.

cogutvalera · 2018-09-28T18:05:32Z

@pmconrad @jmjatlanta Thanks guys !

stale

cogutvalera · 2018-10-01T13:14:00Z

Thanks !

cogutvalera mentioned this pull request Sep 17, 2018

Log console output during replay to file bitshares/bitshares-core#985

Closed

9 tasks

jmjatlanta suggested changes Sep 17, 2018

View reviewed changes

abitmore previously requested changes Sep 18, 2018

View reviewed changes

abitmore reviewed Sep 19, 2018

View reviewed changes

pmconrad suggested changes Sep 21, 2018

View reviewed changes

Log console output during replay to file #985

c544b38

cogutvalera force-pushed the issue_985 branch from c720d64 to c544b38 Compare September 21, 2018 09:22

pmconrad reviewed Sep 24, 2018

View reviewed changes

fc::create_directories before rotate_files

fa7faee

pmconrad approved these changes Sep 24, 2018

View reviewed changes

pmconrad mentioned this pull request Sep 24, 2018

Change replay percentage to total block size processed #1289 bitshares/bitshares-core#1335

Merged

jmjatlanta approved these changes Sep 25, 2018

View reviewed changes

pmconrad merged commit 8e1b47a into bitshares:master Oct 1, 2018

TheTaconator mentioned this pull request Oct 15, 2018

Release Notes for BitShares Core 201810 bitshares/bitshares-core#1381

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log console output during replay to file #985 #76

Log console output during replay to file #985 #76

cogutvalera commented Sep 17, 2018

abitmore commented Sep 17, 2018

cogutvalera commented Sep 17, 2018

jmjatlanta Sep 17, 2018

cogutvalera Sep 17, 2018

jmjatlanta Sep 17, 2018 •

edited

Loading

cogutvalera Sep 17, 2018

abitmore left a comment

abitmore Sep 18, 2018

abitmore Sep 18, 2018

abitmore Sep 18, 2018

abitmore Sep 18, 2018

cogutvalera Sep 19, 2018

cogutvalera commented Sep 19, 2018

pmconrad commented Sep 19, 2018

abitmore Sep 19, 2018

abitmore Sep 19, 2018

abitmore Sep 19, 2018

abitmore commented Sep 19, 2018

cogutvalera commented Sep 21, 2018

pmconrad left a comment

pmconrad Sep 21, 2018

pmconrad Sep 21, 2018

cogutvalera commented Sep 21, 2018

abitmore commented Sep 21, 2018

cogutvalera commented Sep 21, 2018

pmconrad Sep 24, 2018

pmconrad commented Sep 24, 2018

pmconrad commented Sep 24, 2018

jmjatlanta commented Sep 25, 2018 •

edited

Loading

cogutvalera commented Sep 28, 2018

cogutvalera commented Oct 1, 2018

		@@ -141,9 +160,7 @@ namespace fc {
		{
		fc::create_directories(my->cfg.filename.parent_path());

Log console output during replay to file #985 #76

Log console output during replay to file #985 #76

Conversation

cogutvalera commented Sep 17, 2018

abitmore commented Sep 17, 2018

cogutvalera commented Sep 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmjatlanta Sep 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abitmore left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cogutvalera commented Sep 19, 2018

pmconrad commented Sep 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abitmore commented Sep 19, 2018

cogutvalera commented Sep 21, 2018

pmconrad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cogutvalera commented Sep 21, 2018

abitmore commented Sep 21, 2018

cogutvalera commented Sep 21, 2018

Choose a reason for hiding this comment

pmconrad commented Sep 24, 2018

pmconrad commented Sep 24, 2018

jmjatlanta commented Sep 25, 2018 • edited Loading

cogutvalera commented Sep 28, 2018

cogutvalera commented Oct 1, 2018

jmjatlanta Sep 17, 2018 •

edited

Loading

jmjatlanta commented Sep 25, 2018 •

edited

Loading