Skip to content

Commit

Permalink
Fixes formatting for with_columns docs page
Browse files Browse the repository at this point in the history
  • Loading branch information
elijahbenizzy committed Apr 11, 2024
1 parent e92f3f8 commit 180d704
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 8 deletions.
11 changes: 8 additions & 3 deletions docs/reference/decorators/with_columns.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,18 @@
with_columns
=======================

** Overview **

--------
Overview
--------

This is part of the hamilton pyspark integration. To install, run:

`pip install sf-hamilton[pyspark]`
``pip install sf-hamilton[pyspark]``

**Reference Documentation**
-----------------------
Reference Documentation
-----------------------

.. autoclass:: hamilton.plugins.h_spark.with_columns
:special-members: __init__
10 changes: 5 additions & 5 deletions hamilton/plugins/h_spark.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ class SparkKoalasGraphAdapter(base.HamiltonGraphAdapter, base.ResultMixin):
using the \
`Pandas API on Spark <https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html>`__
Use `pip install sf-hamilton[spark]` to get the dependencies required to run this.
Use ``pip install sf-hamilton[spark]`` to get the dependencies required to run this.
Currently, this class assumes you're running SPARK 3.2+. You'd generally use this if you have an existing spark \
cluster running in your workplace, and you want to scale to very large data set sizes.
Expand Down Expand Up @@ -759,7 +759,7 @@ def transform_node(
Note that, at this point, we don't actually know which columns will come from the
base dataframe, and which will come from the upstream nodes. This is handled in the
`with_columns` decorator, so for now, we need to give it enough information to topologically
``with_columns`` decorator, so for now, we need to give it enough information to topologically
sort/assign dependencies.
:param node_: Node to transform
Expand Down Expand Up @@ -950,10 +950,10 @@ def __init__(
"""Initializes a with_columns decorator for spark. This allows you to efficiently run
groups of map operations on a dataframe, represented as pandas/primitives UDFs. This
effectively "linearizes" compute -- meaning that a DAG of map operations can be run
as a set of `.withColumn` operations on a single dataframe -- ensuring that you don't have
to do a complex `extract` then `join` process on spark, which can be inefficient.
as a set of ``.withColumn`` operations on a single dataframe -- ensuring that you don't have
to do a complex ``extract`` then ``join`` process on spark, which can be inefficient.
Here's an example of calling it -- if you've seen `@subdag`, you should be familiar with
Here's an example of calling it -- if you've seen :py:class:`@subdag <hamilton.function_modifiers.recursive>`, you should be familiar with
the concepts:
.. code-block:: python
Expand Down

0 comments on commit 180d704

Please sign in to comment.