gokart package

Submodules

gokart.file_processor module

File processor module with support for multiple DataFrame backends.

class gokart.file_processor.BinaryFileProcessor

Bases: FileProcessor

Pass bytes to this processor

` figure_binary = io.BytesIO() plt.savefig(figure_binary) figure_binary.seek(0) BinaryFileProcessor().dump(figure_binary.read()) `

dump(obj, file)

format()

load(file)

class gokart.file_processor.CsvFileProcessor(sep: str = ',', encoding: str = 'utf-8', dataframe_type: Literal['pandas', 'polars', 'polars-lazy'] = 'pandas')

Bases: FileProcessor

CSV file processor with automatic backend selection based on dataframe_type.

dump(obj, file)

format()

load(file)

class gokart.file_processor.FeatherFileProcessor(store_index_in_feather: bool, dataframe_type: Literal['pandas', 'polars', 'polars-lazy'] = 'pandas')

Bases: FileProcessor

Feather file processor with automatic backend selection based on dataframe_type.

dump(obj, file)

format()

load(file)

class gokart.file_processor.FileProcessor

Bases: object

abstractmethod dump(obj: Any, file: Any) → None

abstractmethod format() → Any

abstractmethod load(file: Any) → Any

class gokart.file_processor.GzipFileProcessor

Bases: FileProcessor

dump(obj, file)

format()

load(file)

class gokart.file_processor.JsonFileProcessor(orient: Literal['split', 'records', 'index', 'table', 'columns', 'values'] | None = None, dataframe_type: Literal['pandas', 'polars', 'polars-lazy'] = 'pandas')

Bases: FileProcessor

JSON file processor with automatic backend selection based on dataframe_type.

dump(obj, file)

format()

load(file)

class gokart.file_processor.NpzFileProcessor

Bases: FileProcessor

dump(obj, file)

format()

load(file)

class gokart.file_processor.ParquetFileProcessor(engine: Any = 'pyarrow', compression: Any = None, dataframe_type: Literal['pandas', 'polars', 'polars-lazy'] = 'pandas')

Bases: FileProcessor

Parquet file processor with automatic backend selection based on dataframe_type.

dump(obj, file)

format()

load(file)

class gokart.file_processor.PickleFileProcessor

Bases: FileProcessor

dump(obj, file)

format()

load(file)

class gokart.file_processor.TextFileProcessor

Bases: FileProcessor

dump(obj, file)

format()

load(file)

class gokart.file_processor.XmlFileProcessor

Bases: FileProcessor

dump(obj, file)

format()

load(file)

gokart.file_processor.make_file_processor(file_path: str, store_index_in_feather: bool = True, *, dataframe_type: Literal['pandas', 'polars', 'polars-lazy'] = 'pandas') → FileProcessor

Create a file processor based on file extension with default parameters.

gokart.info module

gokart.info.make_tree_info(task: TaskOnKart[Any], indent: str = '', last: bool = True, details: bool = False, abbr: bool = True, visited_tasks: set[str] | None = None, ignore_task_names: list[str] | None = None) → str

Return a string representation of the tasks, their statuses/parameters in a dependency tree format

This function has moved to gokart.tree.task_info.make_task_info_as_tree_str. This code is remained for backward compatibility.

Parameters

• task: TaskOnKart

  Root task.

• details: bool

  Whether or not to output details.

• abbr: bool

  Whether or not to simplify tasks information that has already appeared.

• ignore_task_names: list[str] | None

  List of task names to ignore.

Returns

• tree_infostr

  Formatted task dependency tree.

class gokart.info.tree_info(*args, **kwargs)

Bases: TaskOnKart[Any]

mode: StrParameter

Parameter whose value is a str.

output()

output_path: StrParameter

Parameter whose value is a str.

gokart.parameter module

class gokart.parameter.ExplicitBoolParameter(*args, **kwargs)

Bases: BoolParameter

class gokart.parameter.ListTaskInstanceParameter(expected_elements_type: type[~gokart.parameter.TASK_ON_KART_TYPE] | None = None, default: list[~gokart.parameter.TASK_ON_KART_TYPE] | ~luigi.parameter._NoValueType = <no_value>, **kwargs: ~typing.Unpack[~gokart.parameter.ParameterKwargs])

Bases: Parameter[list[TASK_ON_KART_TYPE]], Generic[TASK_ON_KART_TYPE]

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class gokart.parameter.ParameterKwargs

Bases: TypedDict

always_in_help: bool

batch_method: Callable[[Iterable[Any]], Any] | None

config_path: ConfigPath | None

description: str | None

positional: bool

significant: bool

visibility: ParameterVisibility

class gokart.parameter.Serializable(*args, **kwargs)

Bases: Protocol

classmethod gokart_deserialize(s: str) → T

Implement this method to deserialize the object from a string

gokart_serialize() → str

Implement this method to serialize the object as an parameter You can omit some fields from results of serialization if you want to ignore changes of them

class gokart.parameter.SerializableParameter(object_type: type[S], *args: Any, **kwargs: Any)

Bases: Parameter[S], Generic[S]

parse(x: str) → S

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x: S) → str

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class gokart.parameter.TaskInstanceParameter(expected_type: type[~gokart.parameter.TASK_ON_KART_TYPE] | None = None, default: ~gokart.parameter.TASK_ON_KART_TYPE | ~luigi.parameter._NoValueType = <no_value>, **kwargs: ~typing.Unpack[~gokart.parameter.ParameterKwargs])

Bases: Parameter[TASK_ON_KART_TYPE], Generic[TASK_ON_KART_TYPE]

expected_type: type

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class gokart.parameter.ZonedDateSecondParameter(**kwargs)

Bases: Parameter[datetime]

ZonedDateSecondParameter supports a datetime.datetime object with timezone information.

A ZonedDateSecondParameter is a ISO 8601 formatted date, time specified to the second and timezone. For example, 2013-07-10T19:07:38+09:00 specifies July 10, 2013 at 19:07:38 +09:00. The separator : can be omitted for Python3.11 and later.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

gokart.run module

gokart.run.run(cmdline_args=None, set_retcode=True)

gokart.s3_config module

class gokart.s3_config.S3Config(*args, **kwargs)

Bases: Config

aws_access_key_id_name: luigi.Parameter[str]

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:

  • To the root task (eg. --param xyz)

  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See ParamConfigIngestion

• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

aws_secret_access_key_name: luigi.Parameter[str]

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:

  • To the root task (eg. --param xyz)

  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See ParamConfigIngestion

• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

get_s3_client() → luigi.contrib.s3.S3Client

gokart.target module

class gokart.target.LargeDataFrameProcessor(max_byte: int)

Bases: object

static load(file_path: str) → DataFrame

save(df: DataFrame, file_path: str) → None

class gokart.target.ModelTarget(file_path: str, temporary_directory: str, load_function: Any, save_function: Any, task_lock_params: TaskLockParams)

Bases: TargetOnKart

class gokart.target.SingleFileTarget(target: FileSystemTarget, processor: FileProcessor, task_lock_params: TaskLockParams)

Bases: TargetOnKart

class gokart.target.TargetOnKart

Bases: Target

dump(obj: Any, lock_at_dump: bool = True, task_params: dict[str, str] | None = None, custom_labels: dict[str, str] | None = None, required_task_outputs: FlattenableItems[RequiredTaskOutput] | None = None) → None

exists() → bool

Returns True if the Target exists and False otherwise.

last_modification_time() → datetime

load() → Any

path() → str

remove() → None

gokart.target.make_model_target(file_path: str, temporary_directory: str, save_function: Any, load_function: Any, unique_id: str | None = None, task_lock_params: TaskLockParams | None = None) → TargetOnKart

gokart.target.make_target(file_path: str, unique_id: str | None = None, processor: FileProcessor | None = None, task_lock_params: TaskLockParams | None = None, store_index_in_feather: bool = True) → TargetOnKart

gokart.task module

exception gokart.task.EmptyDumpError

Bases: AssertionError

Raised when the task attempts to dump an empty DataFrame even though it is prohibited (fail_on_empty_dump is set to True)

class gokart.task.TaskOnKart(*args, **kwargs)

Bases: Task, Generic[T]

This is a wrapper class of luigi.Task.

The key methods of a TaskOnKart are:

• make_target() - this makes output target with a relative file path.

• make_model_target() - this makes output target for models which generate multiple files to save.

• load() - this loads input files of this task.

• dump() - this save a object as output of this task.

FIX_RANDOM_SEED_VALUE_NONE_MAGIC_NUMBER = -42497368

cache_unique_id: Parameter[bool]

clone(cls=None, **kwargs)

Creates a new instance from an existing instance where some of the args have changed.

There’s at least two scenarios where this is useful (see test/clone_test.py):

• remove a lot of boiler plate when you have recursive dependencies and lots of args

• there’s task inheritance and some logic is on the base class

Parameters:

• cls

• kwargs

Returns:

complete() → bool

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

complete_check_at_run: Parameter[bool]

delete_unnecessary_output_files: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

dump(obj: T, target: None = None, custom_labels: dict[Any, Any] | None = None) → None

dump(obj: Any, target: str | TargetOnKart, custom_labels: dict[Any, Any] | None = None) → None

fail_on_empty_dump: Parameter[bool]

fix_random_seed_methods: Parameter[tuple[str, ...]]

Parameter whose value is a list.

In the task definition, use

class MyTask(luigi.Task):
  grades = luigi.ListParameter()

    def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

At the command line, use

$ luigi --module my_tasks MyTask --grades <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --grades '[100,70]'

It is possible to provide a JSON schema that should be validated by the given value:

class MyTask(luigi.Task):
  grades = luigi.ListParameter(
    schema={
      "type": "array",
      "items": {
        "type": "number",
        "minimum": 0,
        "maximum": 10
      },
      "minItems": 1
    }
  )

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

Using this schema, the following command will work:

$ luigi --module my_tasks MyTask --numbers '[1, 8.7, 6]'

while these commands will fail because the parameter is not valid:

$ luigi --module my_tasks MyTask --numbers '[]'  # must have at least 1 element
$ luigi --module my_tasks MyTask --numbers '[-999, 999]'  # elements must be in [0, 10]

Finally, the provided schema can be a custom validator:

custom_validator = jsonschema.Draft4Validator(
  schema={
    "type": "array",
    "items": {
      "type": "number",
      "minimum": 0,
      "maximum": 10
    },
    "minItems": 1
  }
)

class MyTask(luigi.Task):
  grades = luigi.ListParameter(schema=custom_validator)

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

fix_random_seed_value: Parameter[int]

Parameter whose value is an int.

static get_code(target_class: Any) → set[str]

get_info(only_significant=False)

get_own_code()

get_processing_time() → str

get_task_log() → dict[str, Any]

get_task_params() → dict[str, Any]

input() → TargetOnKart | list[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]] | tuple[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]], ...] | dict[str, T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]]

Returns the outputs of the Tasks returned by requires()

See Task.input

Returns:

a list of Target objects which are specified as outputs of all required Tasks.

static is_task_on_kart(value)

load(target: None | str | TargetOnKart = None) → Any

load(target: TaskOnKart[K]) → K

load(target: list[TaskOnKart[K]]) → list[K]

load_generator(target: None | str | TargetOnKart = None) → Generator[Any, None, None]

load_generator(target: list[TaskOnKart[K]]) → Generator[K, None, None]

local_temporary_directory: Parameter[str]

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:

  • To the root task (eg. --param xyz)

  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See ParamConfigIngestion

• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

make_large_data_frame_target(relative_file_path: str | None = None, use_unique_id: bool = True, max_byte: int = 67108864) → TargetOnKart

make_model_target(relative_file_path: str, save_function: Callable[[Any, str], None], load_function: Callable[[str], Any], use_unique_id: bool = True) → TargetOnKart

Make target for models which generate multiple files in saving, e.g. gensim.Word2Vec, Tensorflow, and so on.

Parameters:

• relative_file_path – A file path to save.

• save_function – A function to save a model. This takes a model object and a file path.

• load_function – A function to load a model. This takes a file path and returns a model object.

• use_unique_id – If this is true, add an unique id to a file base name.

make_target(relative_file_path: str | None = None, use_unique_id: bool = True, processor: FileProcessor | None = None) → TargetOnKart

make_task_instance_dictionary() → dict[str, TaskOnKart[Any]]

make_unique_id() → str

modification_time_check: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

output() → TargetOnKart | list[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]] | tuple[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]], ...] | dict[str, T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]]

property priority

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.__int__(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by ‘+’ or ‘-’ and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer literal. >>> int(‘0b100’, base=0) 4

redis_host: Parameter[str | None]

Class to parse optional parameters.

redis_port: OptionalIntParameter

Class to parse optional int parameters.

redis_timeout: IntParameter

Parameter whose value is an int.

requires() → TaskOnKart[Any] | list[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]] | tuple[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]], ...] | dict[str, T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]]

rerun: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

classmethod restore(unique_id)

serialized_task_definition_check: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

should_dump_supplementary_log_files: Parameter[bool]

should_lock_run: Parameter[bool]

significant: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

store_index_in_feather: Parameter[bool]

strict_check: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

static try_set_seed(methods: list[str], random_seed: int) → list[str]

workspace_directory: Parameter[str]

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:

  • To the root task (eg. --param xyz)

  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See ParamConfigIngestion

• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

gokart.workspace_management module

gokart.workspace_management.delete_local_unnecessary_outputs(task: TaskOnKart[Any]) → None

gokart.zip_client module

class gokart.zip_client.LocalZipClient(file_path: str, temporary_directory: str)

Bases: ZipClient

exists() → bool

make_archive() → None

property path: str

remove() → None

unpack_archive() → None

class gokart.zip_client.ZipClient

Bases: object

abstractmethod exists() → bool

abstractmethod make_archive() → None

abstract property path: str

abstractmethod remove() → None

abstractmethod unpack_archive() → None

Module contents

class gokart.ExplicitBoolParameter(*args, **kwargs)

Bases: BoolParameter

class gokart.ListTaskInstanceParameter(expected_elements_type: type[~gokart.parameter.TASK_ON_KART_TYPE] | None = None, default: list[~gokart.parameter.TASK_ON_KART_TYPE] | ~luigi.parameter._NoValueType = <no_value>, **kwargs: ~typing.Unpack[~gokart.parameter.ParameterKwargs])

Bases: Parameter[list[TASK_ON_KART_TYPE]], Generic[TASK_ON_KART_TYPE]

expected_elements_type: type

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class gokart.PandasTypeConfig(*args, **kwargs)

Bases: Config

classmethod check(df: DataFrame) → None

abstractmethod classmethod type_dict() → dict[str, Any]

class gokart.SerializableParameter(object_type: type[S], *args: Any, **kwargs: Any)

Bases: Parameter[S], Generic[S]

parse(x: str) → S

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x: S) → str

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class gokart.TaskInstanceParameter(expected_type: type[~gokart.parameter.TASK_ON_KART_TYPE] | None = None, default: ~gokart.parameter.TASK_ON_KART_TYPE | ~luigi.parameter._NoValueType = <no_value>, **kwargs: ~typing.Unpack[~gokart.parameter.ParameterKwargs])

Bases: Parameter[TASK_ON_KART_TYPE], Generic[TASK_ON_KART_TYPE]

expected_type: type

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class gokart.TaskOnKart(*args, **kwargs)

Bases: Task, Generic[T]

This is a wrapper class of luigi.Task.

The key methods of a TaskOnKart are:

• make_target() - this makes output target with a relative file path.

• make_model_target() - this makes output target for models which generate multiple files to save.

• load() - this loads input files of this task.

• dump() - this save a object as output of this task.

FIX_RANDOM_SEED_VALUE_NONE_MAGIC_NUMBER = -42497368

cache_unique_id: Parameter[bool]

clone(cls=None, **kwargs)

Creates a new instance from an existing instance where some of the args have changed.

There’s at least two scenarios where this is useful (see test/clone_test.py):

• remove a lot of boiler plate when you have recursive dependencies and lots of args

• there’s task inheritance and some logic is on the base class

Parameters:

• cls

• kwargs

Returns:

complete() → bool

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

complete_check_at_run: Parameter[bool]

delete_unnecessary_output_files: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

dump(obj: T, target: None = None, custom_labels: dict[Any, Any] | None = None) → None

dump(obj: Any, target: str | TargetOnKart, custom_labels: dict[Any, Any] | None = None) → None

fail_on_empty_dump: Parameter[bool]

fix_random_seed_methods: Parameter[tuple[str, ...]]

Parameter whose value is a list.

In the task definition, use

class MyTask(luigi.Task):
  grades = luigi.ListParameter()

    def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

At the command line, use

$ luigi --module my_tasks MyTask --grades <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --grades '[100,70]'

It is possible to provide a JSON schema that should be validated by the given value:

class MyTask(luigi.Task):
  grades = luigi.ListParameter(
    schema={
      "type": "array",
      "items": {
        "type": "number",
        "minimum": 0,
        "maximum": 10
      },
      "minItems": 1
    }
  )

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

Using this schema, the following command will work:

$ luigi --module my_tasks MyTask --numbers '[1, 8.7, 6]'

while these commands will fail because the parameter is not valid:

$ luigi --module my_tasks MyTask --numbers '[]'  # must have at least 1 element
$ luigi --module my_tasks MyTask --numbers '[-999, 999]'  # elements must be in [0, 10]

Finally, the provided schema can be a custom validator:

custom_validator = jsonschema.Draft4Validator(
  schema={
    "type": "array",
    "items": {
      "type": "number",
      "minimum": 0,
      "maximum": 10
    },
    "minItems": 1
  }
)

class MyTask(luigi.Task):
  grades = luigi.ListParameter(schema=custom_validator)

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

fix_random_seed_value: Parameter[int]

Parameter whose value is an int.

static get_code(target_class: Any) → set[str]

get_info(only_significant=False)

get_own_code()

get_processing_time() → str

get_task_log() → dict[str, Any]

get_task_params() → dict[str, Any]

input() → TargetOnKart | list[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]] | tuple[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]], ...] | dict[str, T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]]

Returns the outputs of the Tasks returned by requires()

See Task.input

Returns:

a list of Target objects which are specified as outputs of all required Tasks.

static is_task_on_kart(value)

load(target: None | str | TargetOnKart = None) → Any

load(target: TaskOnKart[K]) → K

load(target: list[TaskOnKart[K]]) → list[K]

load_generator(target: None | str | TargetOnKart = None) → Generator[Any, None, None]

load_generator(target: list[TaskOnKart[K]]) → Generator[K, None, None]

local_temporary_directory: Parameter[str]

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:

  • To the root task (eg. --param xyz)

  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See ParamConfigIngestion

• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

make_large_data_frame_target(relative_file_path: str | None = None, use_unique_id: bool = True, max_byte: int = 67108864) → TargetOnKart

make_model_target(relative_file_path: str, save_function: Callable[[Any, str], None], load_function: Callable[[str], Any], use_unique_id: bool = True) → TargetOnKart

Make target for models which generate multiple files in saving, e.g. gensim.Word2Vec, Tensorflow, and so on.

Parameters:

• relative_file_path – A file path to save.

• save_function – A function to save a model. This takes a model object and a file path.

• load_function – A function to load a model. This takes a file path and returns a model object.

• use_unique_id – If this is true, add an unique id to a file base name.

make_target(relative_file_path: str | None = None, use_unique_id: bool = True, processor: FileProcessor | None = None) → TargetOnKart

make_task_instance_dictionary() → dict[str, TaskOnKart[Any]]

make_unique_id() → str

modification_time_check: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

output() → TargetOnKart | list[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]] | tuple[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]], ...] | dict[str, T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]]

property priority

int([x]) -> integer int(x, base=10) -> integer

Convert a number or string to an integer, or return 0 if no arguments are given. If x is a number, return x.__int__(). For floating-point numbers, this truncates towards zero.

If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in the given base. The literal can be preceded by ‘+’ or ‘-’ and be surrounded by whitespace. The base defaults to 10. Valid bases are 0 and 2-36. Base 0 means to interpret the base from the string as an integer literal. >>> int(‘0b100’, base=0) 4

redis_host: Parameter[str | None]

Class to parse optional parameters.

redis_port: OptionalIntParameter

Class to parse optional int parameters.

redis_timeout: IntParameter

Parameter whose value is an int.

requires() → TaskOnKart[Any] | list[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]] | tuple[T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]], ...] | dict[str, T | list[FlattenableItems[T]] | tuple[FlattenableItems[T], ...] | dict[str, FlattenableItems[T]]]

rerun: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

classmethod restore(unique_id)

serialized_task_definition_check: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

should_dump_supplementary_log_files: Parameter[bool]

should_lock_run: Parameter[bool]

significant: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

store_index_in_feather: Parameter[bool]

strict_check: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

static try_set_seed(methods: list[str], random_seed: int) → list[str]

workspace_directory: Parameter[str]

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:

  • To the root task (eg. --param xyz)

  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See ParamConfigIngestion

• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

class gokart.WorkerSchedulerFactory

Bases: object

create_local_scheduler() → Scheduler

create_remote_scheduler(url: str) → RemoteScheduler

create_worker(scheduler: Scheduler, worker_processes: int, assistant: bool = False) → WorkerProtocol

class gokart.ZonedDateSecondParameter(**kwargs)

Bases: Parameter[datetime]

ZonedDateSecondParameter supports a datetime.datetime object with timezone information.

A ZonedDateSecondParameter is a ISO 8601 formatted date, time specified to the second and timezone. For example, 2013-07-10T19:07:38+09:00 specifies July 10, 2013 at 19:07:38 +09:00. The separator : can be omitted for Python3.11 and later.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

gokart.add_config(file_path: str) → None

gokart.build(task: TaskOnKart[T], return_value: Literal[True] = True, reset_register: bool = True, log_level: int = logging.ERROR, task_lock_exception_max_tries: int = 10, task_lock_exception_max_wait_seconds: int = 600, **env_params: Any) → T

gokart.build(task: TaskOnKart[T], return_value: Literal[False], reset_register: bool = True, log_level: int = logging.ERROR, task_lock_exception_max_tries: int = 10, task_lock_exception_max_wait_seconds: int = 600, **env_params: Any) → None

Run gokart task for local interpreter. Sharing the most of its parameters with luigi.build (see https://luigi.readthedocs.io/en/stable/api/luigi.html?highlight=build#luigi.build)

gokart.delete_local_unnecessary_outputs(task: TaskOnKart[Any]) → None

gokart.make_task_info_as_tree_str(task: TaskOnKart[Any], details: bool = False, abbr: bool = True, ignore_task_names: list[str] | None = None) → str

Return a string representation of the tasks, their statuses/parameters in a dependency tree format

Parameters

• task: TaskOnKart

  Root task.

• details: bool

  Whether or not to output details.

• abbr: bool

  Whether or not to simplify tasks information that has already appeared.

• ignore_task_names: list[str] | None

  List of task names to ignore.

Returns

• tree_infostr

  Formatted task dependency tree.

gokart.make_tree_info(task: TaskOnKart[Any], indent: str = '', last: bool = True, details: bool = False, abbr: bool = True, visited_tasks: set[str] | None = None, ignore_task_names: list[str] | None = None) → str

Return a string representation of the tasks, their statuses/parameters in a dependency tree format

This function has moved to gokart.tree.task_info.make_task_info_as_tree_str. This code is remained for backward compatibility.

Parameters

• task: TaskOnKart

  Root task.

• details: bool

  Whether or not to output details.

• abbr: bool

  Whether or not to simplify tasks information that has already appeared.

• ignore_task_names: list[str] | None

  List of task names to ignore.

Returns

• tree_infostr

  Formatted task dependency tree.

gokart.run(cmdline_args=None, set_retcode=True)

class gokart.test_run(*args, **kwargs)

Bases: TaskOnKart[Any]

namespace: OptionalStrParameter

Class to parse optional str parameters.

pandas: BoolParameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

class gokart.tree_info(*args, **kwargs)

Bases: TaskOnKart[Any]

mode: StrParameter

Parameter whose value is a str.

output()

output_path: StrParameter

Parameter whose value is a str.