FileDF Writer¶

Bases: FrozenModel

Allows you to write Spark DataFrame as files in a target path of specified file connection with parameters. |support_hooks|

Parameters:

connection (:obj:BaseFileDFConnection <onetl.base.base_file_df_connection.BaseFileDFConnection>) –

File DataFrame connection. See :ref:file-df-connections section.
format (:obj:BaseWritableFileFormat <onetl.base.base_file_format.BaseWritableFileFormat>) –

File format to write.
target_path (PathLike | str) –

Directory path to write data to.
options (:obj:FileDFWriterOptions <onetl.file.file_df_writer.options.FileDFWriterOptions>) –

Common writing options.

Examples:

.. tabs::

.. code-tab:: py Write CSV files to local filesystem

    from onetl.connection import SparkLocalFS
    from onetl.file import FileDFWriter
    from onetl.file.format import CSV

    local_fs = SparkLocalFS(spark=spark)

    writer = FileDFWriter(
        connection=local_fs,
        format=CSV(delimiter=","),
        target_path="/path/to/directory",
    )

.. code-tab:: py All supported options

    from onetl.connection import SparkLocalFS
    from onetl.file import FileDFWriter
    from onetl.file.format import CSV

    csv = CSV(delimiter=",")
    local_fs = SparkLocalFS(spark=spark)

    writer = FileDFWriter(
        connection=local_fs,
        format=csv,
        target_path="/path/to/directory",
        options=FileDFWriter.Options(if_exists="replace_entire_directory"),
    )

`run(df)` ¶

Method for writing DataFrame as files. |support_hooks|

.. note :: Method does support only batching DataFrames.

Parameters:

df (DataFrame) –

Spark dataframe

Examples:

Write df to target:

.. code:: python

writer.run(df)

FileDF Writer¶

run(df) ¶

`run(df)` ¶