Skip to content

Add a troubleshooting doc #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
henrydavidge merged 3 commits into from
Oct 22, 2019
Merged

Add a troubleshooting doc #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d6e4fa6
Update version.sbt to 0.2.0 (#20)
henrydavidge Oct 17, 2019
38f6fb6
Merge branch 'master' of github.com:projectglow/glow into troubleshoo…
henrydavidge Oct 18, 2019
fb1dfaa
comments
henrydavidge Oct 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/source/_static/css/glow.css
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
ul.simple li p:last-of-type {
margin-bottom: 0px;
}
2 changes: 2 additions & 0 deletions docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

html_css_files = ['css/glow.css']

html_logo = '../../static/glow_logo_horiz_color_dark_bg.png'

html_favicon = '../../static/favicon.ico'
Expand Down
1 change: 1 addition & 0 deletions docs/source/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ machine learning, enabling the scale of the cloud for genomics workflows.
getting-started
etl/index
tertiary/index
troubleshooting
additional-resources

.. modules
20 changes: 20 additions & 0 deletions docs/source/troubleshooting.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
Troubleshooting
===============

- Job is slow or OOMs (throws an ``OutOfMemoryError``) while using an aggregate like ``collect_list`` or
``sample_call_summary_stats``

* Try disabling the `ObjectHashAggregate
<https://github.com/apache/spark/commit/27daf6bcde782ed3e0f0d951c90c8040fd47e985>`_ by setting
``spark.sql.execution.useObjectHashAggregateExec`` to ``false``

- Job is slow or OOMs while writing to partitioned table
Copy link
Collaborator

@kianfar77 kianfar77 Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the line spacing is awkward on rendering. The solution is closer to the next item than the item.

Copy link
Contributor Author

@henrydavidge henrydavidge Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link
Contributor Author

@henrydavidge henrydavidge Oct 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed by the css change

Copy link
Collaborator

@kianfar77 kianfar77 Oct 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice


* This error can occur when reading from highly compressed files. Try decreasing
``spark.files.maxPartitionBytes`` to a smaller value like ``33554432`` (32MB)

- My VCF looks weird after merging VCFs and saving with ``bigvcf``

* When saving to a VCF, the samples in the genotypes array must be in the same order for each row.
This ordering is not guaranteed when using ``collect_list`` to join multiple VCFs. Try sorting
the array using ``sort_array``.