Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/164/json output #240

Closed
wants to merge 12 commits into from
Closed

Feat/164/json output #240

wants to merge 12 commits into from

Conversation

morrieinmaas
Copy link
Contributor

closes #164

Moriarty added 5 commits December 13, 2022 15:21
Signed-off-by: Moriarty <[email protected]>
TODO:
* refactor and rename a a few things

Signed-off-by: Moriarty <[email protected]>
Signed-off-by: Moriarty <[email protected]>
NOTE: This is branched off the wallet endpoints already so that needs to
be merged first

Signed-off-by: Moriarty <[email protected]>
@morrieinmaas
Copy link
Contributor Author

needs #162 to be merged first

Signed-off-by: Moriarty <[email protected]>
@morrieinmaas morrieinmaas marked this pull request as ready for review December 14, 2022 13:40
Copy link
Member

@berendsliedrecht berendsliedrecht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bit difficult to review, but it does not matter that much.

The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.

Input:
siera --json connection invite
Output:

{
  "invitation_url": "https://...=",
  "connection_id": "abcd...xyz"
}

Comment on lines 22 to 24
/// Log level json and nothing else
Json,

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Json should not be a log level. Outputting JSON should not collide with the existing verbosity logs.

if ::siera_logger::LogLevel::Json ==
::siera_logger::STATE.read().unwrap().level
{
println!($($arg)*);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice here if we format the data, so the user does not have to do this themselves.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah true. actually better to do this here. what I did now is pass a formatted data blob that is json to this macro by using the pretty_stringify_obj function

@morrieinmaas
Copy link
Contributor Author

morrieinmaas commented Dec 14, 2022

bit difficult to review, but it does not matter that much.

The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.

Input: siera --json connection invite Output:

{
  "invitation_url": "https://...=",
  "connection_id": "abcd...xyz"
}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?

So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

@berendsliedrecht
Copy link
Member

bit difficult to review, but it does not matter that much.
The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.
Input: siera --json connection invite Output:

{
  "invitation_url": "https://...=",
  "connection_id": "abcd...xyz"
}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?

So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.

So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

@morrieinmaas
Copy link
Contributor Author

bit difficult to review, but it does not matter that much.
The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.
Input: siera --json connection invite Output:

{
  "invitation_url": "https://...=",
  "connection_id": "abcd...xyz"
}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?
So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.

So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

Ok I understand what you mean (I think). so then you'd get the normal logs according to verbosity wrapped in JSON and the result is under a key in there like 'result' or sth like that?

@berendsliedrecht
Copy link
Member

bit difficult to review, but it does not matter that much.
The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.
Input: siera --json connection invite Output:

{
  "invitation_url": "https://...=",
  "connection_id": "abcd...xyz"
}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?
So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.
So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

Ok I understand what you mean (I think). so then you'd get the normal logs according to verbosity wrapped in JSON and the result is under a key in there like 'result' or sth like that?

No, just the normal logs like the invitation url is in json, the verbosity logs will always stay the same.

Just the output that we always log (with log!()) will be json.

@morrieinmaas
Copy link
Contributor Author

bit difficult to review, but it does not matter that much.
The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.
Input: siera --json connection invite Output:

{
  "invitation_url": "https://...=",
  "connection_id": "abcd...xyz"
}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?
So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.
So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

Ok I understand what you mean (I think). so then you'd get the normal logs according to verbosity wrapped in JSON and the result is under a key in there like 'result' or sth like that?

No, just the normal logs like the invitation url is in json, the verbosity logs will always stay the same.

Just the output that we always log (with log!()) will be json.

Sorry if I'm being daft here. Then we still have the log output in there if doing json? then one would still need to filter somehow the verbosity stuff and separate that from the desired json?! I thought the point was to just get sth that is definitely JSON so it can just be used in unix pipe or some program?

@berendsliedrecht
Copy link
Member

bit difficult to review, but it does not matter that much.

The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.

Input: siera --json connection invite Output:

{

"invitation_url": "https://...=",

"connection_id": "abcd...xyz"

}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?

So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.

So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

Ok I understand what you mean (I think). so then you'd get the normal logs according to verbosity wrapped in JSON and the result is under a key in there like 'result' or sth like that?

No, just the normal logs like the invitation url is in json, the verbosity logs will always stay the same.

Just the output that we always log (with log!()) will be json.

Sorry if I'm being daft here. Then we still have the log output in there if doing json? then one would still need to filter somehow the verbosity stuff and separate that from the desired json?! I thought the point was to just get sth that is definitely JSON so it can just be used in unix pipe or some program?

Well just don't have a verbosity flag :). The regular command just displays the "pipable" information. We also have a quiet flag, for stuff like this.

@morrieinmaas
Copy link
Contributor Author

bit difficult to review, but it does not matter that much.

The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.

Input: siera --json connection invite Output:

{

"invitation_url": "https://...=",

"connection_id": "abcd...xyz"

}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?

So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.

So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

Ok I understand what you mean (I think). so then you'd get the normal logs according to verbosity wrapped in JSON and the result is under a key in there like 'result' or sth like that?

No, just the normal logs like the invitation url is in json, the verbosity logs will always stay the same.

Just the output that we always log (with log!()) will be json.

Sorry if I'm being daft here. Then we still have the log output in there if doing json? then one would still need to filter somehow the verbosity stuff and separate that from the desired json?! I thought the point was to just get sth that is definitely JSON so it can just be used in unix pipe or some program?

Well just don't have a verbosity flag :). The regular command just displays the "pipable" information. We also have a quiet flag, for stuff like this.

Alright I think where this is going. What about the loading spinner then? How do you imagine handling that. As in I'd assume this is a nice feature when not using this to pipe, but when using this for pipe that - if I'm not mistaken - will be part of the output (and undesired) when wanting json.

@berendsliedrecht
Copy link
Member

bit difficult to review, but it does not matter that much.

The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.

Input: siera --json connection invite Output:

{

"invitation_url": "https://...=",

"connection_id": "abcd...xyz"

}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?

So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.

So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

Ok I understand what you mean (I think). so then you'd get the normal logs according to verbosity wrapped in JSON and the result is under a key in there like 'result' or sth like that?

No, just the normal logs like the invitation url is in json, the verbosity logs will always stay the same.

Just the output that we always log (with log!()) will be json.

Sorry if I'm being daft here. Then we still have the log output in there if doing json? then one would still need to filter somehow the verbosity stuff and separate that from the desired json?! I thought the point was to just get sth that is definitely JSON so it can just be used in unix pipe or some program?

Well just don't have a verbosity flag :). The regular command just displays the "pipable" information. We also have a quiet flag, for stuff like this.

Alright I think where this is going. What about the loading spinner then? How do you imagine handling that. As in I'd assume this is a nice feature when not using this to pipe, but when using this for pipe that - if I'm not mistaken - will be part of the output (and undesired) when wanting json.

The loader is send to stderr as a convention within unix.

@morrieinmaas
Copy link
Contributor Author

bit difficult to review, but it does not matter that much.

The JSON logger MUST not collide with the verbosity logs and it should just be "log the normal ouput in a json format with a nice key/value, thats it.

Input: siera --json connection invite Output:

{

"invitation_url": "https://...=",

"connection_id": "abcd...xyz"

}

hmmm ok. Wouldn't it make sense to only log the json if json is passed? Was thinking that if only a json value is printed one can easily use siera in a context where you get sth that is json. So if you pipe the output somewhere or use its output to be read by some other program it can load that into whatever datastructure represents json in that lang without having to do any sort of cleaning. Or what is your opinion on this?

So what's happening now is that --json | -j are mutually exclusive with other logging flags plus when using -j it only prints json and nothing else

I understand the point of only showing JSON, and it should not give the default log output, but it should not remove the verbosity logs if specified.

So basically the normal logger should check the state and if it is json we can prettify it (also give it custom keys) and log it like JSOn instead of normal.

Ok I understand what you mean (I think). so then you'd get the normal logs according to verbosity wrapped in JSON and the result is under a key in there like 'result' or sth like that?

No, just the normal logs like the invitation url is in json, the verbosity logs will always stay the same.

Just the output that we always log (with log!()) will be json.

Sorry if I'm being daft here. Then we still have the log output in there if doing json? then one would still need to filter somehow the verbosity stuff and separate that from the desired json?! I thought the point was to just get sth that is definitely JSON so it can just be used in unix pipe or some program?

Well just don't have a verbosity flag :). The regular command just displays the "pipable" information. We also have a quiet flag, for stuff like this.

Alright I think where this is going. What about the loading spinner then? How do you imagine handling that. As in I'd assume this is a nice feature when not using this to pipe, but when using this for pipe that - if I'm not mistaken - will be part of the output (and undesired) when wanting json.

The loader is send to stderr as a convention within unix.

ahhh okayyyyy - thanks for clarifying.

@morrieinmaas
Copy link
Contributor Author

Alright, so changed to your suggestions ( if I got them coeectly). no printing JSON for by default for applicable. Quiet frankly, preferred the previous way (or a version of it) with the flag.

Copy link
Member

@berendsliedrecht berendsliedrecht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you understood my point.

Normally we do not want JSON output and we just want it seperated by a new line, like with the connections(uuid \n invitationURL).

When we pass the --json flag, we want the following:

{
  "connection_id": "foo-bar",
  "invitation_url": "https://..."
}

the default behaviour will not change, but it is just swapped with the JSOn logger.

crates/cli/src/modules/credential.rs Outdated Show resolved Hide resolved
@berendsliedrecht
Copy link
Member

Is this ready for review?

Copy link
Member

@berendsliedrecht berendsliedrecht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite interested if the macro works. Did you test it? Do you have some output I can see?

crates/cli/src/cli.rs Outdated Show resolved Hide resolved
crates/cli/src/modules/credential.rs Outdated Show resolved Hide resolved
crates/cli/src/modules/wallet.rs Outdated Show resolved Hide resolved
crates/cli/src/modules/wallet.rs Outdated Show resolved Hide resolved
crates/cloudagent-python/src/cloudagent/json_ld.rs Outdated Show resolved Hide resolved
crates/logger/src/lib.rs Outdated Show resolved Hide resolved
crates/logger/src/lib.rs Outdated Show resolved Hide resolved
crates/logger/src/macros.rs Outdated Show resolved Hide resolved
@morrieinmaas
Copy link
Contributor Author

cheers. Thanks for all the feedback. Will address asap

@morrieinmaas
Copy link
Contributor Author

@blu3beri addressed

@@ -141,7 +141,7 @@ pub async fn parse_wallet_args(
};
agent.create_local_did(options).await.map(|response| {
loader.stop();
log_info!("Successfully created local DID: ",);
log_info!("Successfully created local DID: {:?}", response.did);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need debug printing for the did (I assume it is a string). If it is not a string we need to convert it.

Copy link
Member

@berendsliedrecht berendsliedrecht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more comment.

@TimoGlastra
Copy link
Member

I'm with @morrieinmaas here. I think it's quite normal for CLIs / tools that when you enable json mode, ALL messages will be printed in json. This way you can parse the logs and make sense of it (structured logging). If you enable verbosity logs, these should also be logged as json imho.

Some examples:

for example tsed has the following structure:

{"startTime":"2017-06-05T22:23:08.479Z","categoryName":"json-test","data":["this is just a test"],"level":"INFO","context":{}},
{"startTime":"2017-06-05T22:23:08.483Z","categoryName":"json-test","data":["of a custom appender"],"level":"ERROR","context":{}},
{"startTime":"2017-06-05T22:23:08.483Z","categoryName":"json-test","data":["that outputs json"],"level""WARN","context":{}},

Also leaving this: https://stackify.com/what-is-structured-logging-and-why-developers-need-it/

@berendsliedrecht
Copy link
Member

berendsliedrecht commented Dec 20, 2022

I'm with @morrieinmaas here. I think it's quite normal for CLIs / tools that when you enable json mode, ALL messages will be printed in json. This way you can parse the logs and make sense of it (structured logging). If you enable verbosity logs, these should also be logged as json imho.

Some examples:

for example tsed has the following structure:


{"startTime":"2017-06-05T22:23:08.479Z","categoryName":"json-test","data":["this is just a test"],"level":"INFO","context":{}},

{"startTime":"2017-06-05T22:23:08.483Z","categoryName":"json-test","data":["of a custom appender"],"level":"ERROR","context":{}},

{"startTime":"2017-06-05T22:23:08.483Z","categoryName":"json-test","data":["that outputs json"],"level""WARN","context":{}},

Also leaving this: https://stackify.com/what-is-structured-logging-and-why-developers-need-it/

Yes, makes sense. I am not opposed to json verbosity logging, but my point was mainly that logging JSON must not mute the verbosity logs. I think we can address this in a separate PR as it would require reworking the whole logger. We would probably need to always pass a json / struct and create a normal message from that or log it as JSON.

@morrieinmaas
Copy link
Contributor Author

I'm with @morrieinmaas here. I think it's quite normal for CLIs / tools that when you enable json mode, ALL messages will be printed in json. This way you can parse the logs and make sense of it (structured logging). If you enable verbosity logs, these should also be logged as json imho.
Some examples:

for example tsed has the following structure:


{"startTime":"2017-06-05T22:23:08.479Z","categoryName":"json-test","data":["this is just a test"],"level":"INFO","context":{}},

{"startTime":"2017-06-05T22:23:08.483Z","categoryName":"json-test","data":["of a custom appender"],"level":"ERROR","context":{}},

{"startTime":"2017-06-05T22:23:08.483Z","categoryName":"json-test","data":["that outputs json"],"level""WARN","context":{}},

Also leaving this: https://stackify.com/what-is-structured-logging-and-why-developers-need-it/

Yes, makes sense. I am not opposed to json verbosity logging, but my point was mainly that logging JSON must not mute the verbosity logs. I think we can address this in a separate PR as it would require reworking the whole logger. We would probably need to always pass a json / struct and create a normal message from that or log it as JSON.

Or not pass json/struct and create json/struct on the fly with eg a timestamp if the/a key is missing but value is present.

I'll have a look into how this can be 'formally' handled as in sticking to some convention

@berendsliedrecht
Copy link
Member

@morrieinmaas what is the status of this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow json output format
3 participants