Skip to content

Options

Bases: FileDFReadOptions, GenericOptions

Options for :obj:FileDFReader <onetl.file.file_df_reader.file_df_reader.FileDFReader>.

.. versionadded:: 0.9.0

Examples:

.. note::

You can pass any value `supported by Spark <https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html>`_,
even if it is not mentioned in this documentation. **Option names should be in** ``camelCase``!

The set of supported options depends on Spark version.

.. code:: python

from onetl.file import FileDFReader

options = FileDFReader.Options(recursive=True)

recursive = Field(default=None, alias='recursiveFileLookup') class-attribute instance-attribute

If True, perform recursive file lookup.

.. warning::

This disables partition inferring using file paths.

.. warning::

Can be used only in Spark 3+. See `SPARK-27990 <https://issues.apache.org/jira/browse/SPARK-27990>`_.

apply_to_reader(reader)

Apply provided format to :obj:pyspark.sql.DataFrameReader. |support_hooks|

Returns:

  • obj:`pyspark.sql.DataFrameReader` with options applied –