extorch.utils.common

class extorch.utils.common.TimeEstimator(iter_num: int)[source]

Bases: object

Remaining time estimator. Estimate the remaining time to finish all iterations based on history.

Parameters

iter_num (int) – Total iteration number.

step() → Tuple[datetime.datetime, datetime.datetime][source]

Update the estimator and return the estimated remaining time.

Returns

The remaining time estimated based on the whole history. nearest_time (datetime): The remaining time estimated based on the last iteration.

Return type

overall_time (datetime)

class extorch.utils.common.abstractclassmethod(a_callable)[source]

Bases: classmethod

extorch.utils.common.makedir(path: str, remove: bool = False, quiet: bool = False) → None[source]

Create a leaf directory and all intermediate ones.

Parameters

  • path (str) – The directory to be made.

  • remove (bool) – Sometimes, the targeted directory has existed. If remove is false, the targeted directory will not be removed. Default: False.

  • quiet (bool) – When the targeted directory has existed and remove == True: If quiet == False, we will ask whether to remove the existing directory. Else, the existing directory will be removed directly. Default: False.

extorch.utils.common.nullcontext()[source]
extorch.utils.common.remove_dirs(root: str, target: str) → int[source]

Delete the target dirs under the given root recursively.

Parameters

  • root (str) – Root.

  • target (str) – Name of the target dirs.

Returns

Number of the deleted dirs.

Return type

delete_num (int)

Examples::
>>> # Remove all dirs named "__pycache__" under the current path recursively
>>> delete_num = remove_dirs("./", "__pycache__")
extorch.utils.common.remove_files(root: str, target: str) → int[source]

Delete the target files under the given root recursively.

Parameters

  • root (str) – Root.

  • target (str) – Name of the target files.

Returns

Number of the deleted files.

Return type

delete_num (int)

Examples::
>>> # Remove files named "__init__.py" under the current path recursively
>>> delete_num = remove_files("./", "__init__.py")
extorch.utils.common.remove_targets(root: str, target: str) → int[source]

Delete the target files and dirs under the given root recursively.

Parameters

  • root (str) – Root.

  • target (str) – Name of the target files or dirs.

Returns

Number of the deleted files or dirs.

Return type

delete_num (int)

Examples::
>>> # Remove dirs or files named "__pycache__" under the current path recursively
>>> delete_num = remove_targets("./", "__pycache__")
extorch.utils.common.run_processes(commands: List[str], **kwargs) → List[str][source]

Run a list of commands one-by-one.

Parameters

  • command (List[str]) – A list of commands to be run.

  • kwargs – Configurations except ‘shell’ for subprocess.check_call. We note that shell must be True for this function.

Returns

A list of commands that fail to run successfully.

Return type

unsuccessful_commands (List[str])

Examples::
>>> # Assumes a runnable python file named `test.py` exists
>>> # And assumes no file named `not_exist.py` exists
>>> # Now, we want to run `test.py` and `not_exist.py` one-by-one
>>> commands = ["python test.py", "python not_exist.py"]
>>> unsuccessful_commands = run_processes(commands) # ["python not_exist.py"]
>>> print("Unsuccessful commands: {}".format(unsuccessful_commands))
extorch.utils.common.set_seed(seed: int) → None[source]

Set the seed of the system.

Parameters

seed (int) – The artificial random seed.

extorch.utils.common.set_trace() → None[source]
extorch.utils.common.yaml_dump(data, file: str, mode: str = 'w', **kwargs) → None[source]

Serialize a Python object into a YAML document.

Parameters

  • data – The Python object to be serialized.

  • file (str) – Path of the YAML document.

  • mode (str) – Mode to open the document.

  • kwargs – Other options for opening the document.

extorch.utils.common.yaml_load(file: str, mode: str = 'r', Loader=<class 'yaml.loader.FullLoader'>, **kwargs)[source]

Parse the YAML document and produce the corresponding Python object.

Parameters

  • file (str) – Path of the YAML document.

  • mode (str) – Mode to open the document.

  • Loader – Loader for YAML.

  • kwargs – Other options for opening the document.