An extra CR is added to CSV during export on Windows

[kalupator]

Hello!
In the prepared jiraone csv files there are all line breaks (in fields such as Description).
But when loading the file into a new jira, the line breaks are lost and the field becomes single-line. Because of this, the formatting is lost.
I would like to know if this is possibly affected by some settings when preparing the export file in jiraone or something else?
Thanks!

If I manually return all line breaks, then the formatting returns. But this is very labor-intensive if there are thousands of tasks.

[princenyeche]

Are you importing between a Jira DC and Jira Cloud? One of the issues I'm aware of that would cause this, is the difference between Markup and Markdown. I believe DC uses a variant of Markup while Cloud uses Markdown but I would require additional context here about the type of Jira environment to expand on that issue.

[kalupator]

Import from Jira Server 8 to Jira Data Center 10.
In both versions, the formatting of descriptions and comments is exactly the same (I think, Markup).

[kalupator]

@princenyeche
I finally found the reason - an extra CR is added to csv during export.
I changed the issue title.
Perhaps this can be resolved by the export configuration?

Like this:
https://stackoverflow.com/questions/3191528/csv-in-python-adding-an-extra-carriage-return-on-windows
But how to fix this?

[kalupator]

@princenyeche

I was able to fix this by editing the code in reporting.py

As described here:

The official csv documentation recommends opening the file with newline='' on all platforms to disable universal newlines translation

[princenyeche]

@kalupator The script actually has this solved by default. SLOC 7897 - 7904 is the condition that is used (as shown in your image). So there was no actual need to edit the code and add additional conditions, as that would cause more problems when performing attachment transfers. You have to look into how you call the arguments for the extraction you're performing which triggers the extra CRs on window devices.

[kalupator]

@princenyeche

You have to look into how you call the arguments for the extraction you're performing which triggers the extra CRs on window devices.

Can you clarify what arguments you are talking about?
My code is extremely simple:

from jiraone import LOGIN, issue_export
import json  

file = "config-jira.json" 
config = json.load(open(file)) 
LOGIN.api = False
LOGIN(**config)
jql = "project in (TEST) order by created ASC"
issue_export(jql=jql, extension="csv", allow_media=True)

But additional CRs still appear (they disappeared only after adding the line highlighted in red on my screenshot).

[princenyeche]

Hmm, that's strange as you should only get to that point in the code if your system is not Windows and you want to open a non-text file.
This portion of the code

else open(
            file,
            mode,
            encoding=encoding,
            errors=errors,
        )

Should only be reached if you are not on Windows or if you're on Windows and the mode argument is file. Your script is basic enough, so probably did you modify the file_writer mark value in the export_issues function to set the mark = file?

[kalupator]

@princenyeche

I output the values ​​of the variables (system and mark) in file_writer function on clear installation of jiraone-0.8.6 and got the following result:

Defaulting to comma as CSV separator.
Downloading issue export in CSV format.
<Response [200]>  ::downloading issues at page: 0 of 0
system = Windows
mark = file
system = Windows
mark = many
Processing. Current progress: 100%
Reconstructing file headers
system = Windows
mark = many
Applying updated data into the CSV file
system = Windows
mark = many
Export Completed.File located at c:\temp\jira_imp\EXPORT\final_file.csv
The default CSV separator is a comma for the delimit argument, no action taken.

I didn't correct the source code in any other way.
I determined that mark="file" comes from this line:
https://github.com/princenyeche/jiraone/blob/main/src/jiraone/reporting.py#L4083C1-L4083C33
But if I change this line to mark="single", then I get an error:

Defaulting to comma as CSV separator.
Downloading issue export in CSV format.
<Response [200]>  ::downloading issues at page: 0 of 0
system = Windows
mark = single
Traceback (most recent call last):
  File "C:\temp\jira_imp\jira4-py\script-jira.py", line 12, in <module>
    issue_export(jql=jql, extension="csv", allow_media=True)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python313\Lib\site-packages\jiraone\reporting.py", line 4093, in export_issues
    download_csv() if not merge_files else config.update(
    ~~~~~~~~~~~~^^
  File "C:\Program Files\Python\Python313\Lib\site-packages\jiraone\reporting.py", line 4076, in download_csv
    file_writer(
    ~~~~~~~~~~~^
        folder,
        ^^^^^^^
    ...<6 lines>...
        mode="w+",
        ^^^^^^^^^^
    )
    ^
  File "C:\Program Files\Python\Python313\Lib\site-packages\jiraone\reporting.py", line 7931, in file_writer
    write.writerow(data)
    ~~~~~~~~~~~~~~^^^^^^
_csv.Error: iterable expected, not _AnyMeta

Tell me what I can diagnose or send you something else?

[princenyeche]

That first one from line is expected, so you shouldn't be changing that. It doesn't affect how the file should work because the content being received is actually in bytes. I think for most of it, the export based on the output you've shown did you still get the CRs added or was it not there?

If it's still there based on current ver 0.8.6 can you try something, can you change this line at your end for mode = "wb+" leave mark = "file" and let me know if its works and removes any CRs

[kalupator]

@princenyeche

When setting the mode = "wb+", an error occurs:

Traceback (most recent call last):
  File "C:\temp\jira_imp\jira4-py\script-jira4.py", line 12, in <module>
    issue_export(jql=jql, extension="csv", allow_media=True)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\Python313\Lib\site-packages\jiraone\reporting.py", line 4093, in export_issues
    download_csv() if not merge_files else config.update(
    ~~~~~~~~~~~~^^
  File "C:\Program Files\Python\Python313\Lib\site-packages\jiraone\reporting.py", line 4076, in download_csv
    file_writer(
    ~~~~~~~~~~~^
        folder,
        ^^^^^^^
    ...<6 lines>...
        mode="wb+",
        ^^^^^^^^^^^
    )
    ^
  File "C:\Program Files\Python\Python313\Lib\site-packages\jiraone\reporting.py", line 7917, in file_writer
    else open(
         ~~~~^
        file,
        ^^^^^
    ...<2 lines>...
        errors=errors,
        ^^^^^^^^^^^^^^
    )
    ^
ValueError: binary mode doesn't take an encoding argument

[princenyeche]

That validates that's not the issue. This is the closest similarity I've seen about this issue JRACLOUD-67140. For the most part, most people who use jiraone and on Windows haven't brought up this problem, so I'm guessing this is that edge occurrence. The workaround which you used shouldn't be the case though as what it does is remove the \r lines which by default it should do for Windows devices as it rewrites the entire cells into a new file. Maybe it could be a localized issue, have you tried on different Windows devices to see if the same problem occurs?

[kalupator]

But I understand correctly that this block of code should not be executed in my case at all?
https://github.com/princenyeche/jiraone/blob/main/src/jiraone/reporting.py#L7914C1-L7919C10
But for some reason it is executed (that's why adding newline="" in this block helps me).

I tried changing the locale, country and regional settings of Windows - it hasn't helped yet.

[princenyeche]

The problem here is that it's defaulting to that line when using mark = file which means the CR is added there in the first instance of the file when it is extracted from Jira (that is expected). Afterwards, mark is != file again.

<Response [200]>  ::downloading issues at page: 0 of 0
system = Windows
mark = file

This brings me to the second factor, which is the constant rewrites that are happening, if CR is added at the first instance when the file is created, this should have been resolved midflight during rewrites which don't happen in your case whereas other Windows user it works. While adding that line helps you, it might cause other issues which is unknown to me. Anyway, if it works for you then I have no complaints.

[kalupator]

I'm already interested in getting to the bottom of the truth.
What's interesting is that if I open the file in a standard Windows notepad and just save file (ctrl+s), all the extra CRs disappear.

[princenyeche]

Yeah, that's what the rewrites should have done, but it seems this doesn't just happen in your case. I can't provide any answers to that but I guess that's a much easier workaround to just ctrl + s on a notepad prior to using it.

[kalupator]

@princenyeche
Thank you, I think I will do that, since migration can’t wait any longer.