Skip to content

core

core

Classes:

  • Check –
  • Collect –

    A class representing the file collection process.

  • Context –

    A shared context object for storing and accessing data across different processes.

  • ErrorProcess –

    Class to represent an error that occurs during process execution.

  • Extract –

    A class representing the file extraction process.

  • Manager –

    Manager class that manages the registration and execution

  • Process – 
    Abstract base class for all processes in the task publishing pipeline.
  • Push –

    A class representing the process to push data to the database.

  • StatusProcess –

    Enum-like class representing the status of a process.

Functions:

  • log_execution –

    Decorator to log the execution of process methods.

Check 

Check(manager: Manager)


              flowchart LR
              publisher.core.Check[Check]
              publisher.core.Process[Process]

                              publisher.core.Process --> publisher.core.Check
                


              click publisher.core.Check href "" "publisher.core.Check"
              click publisher.core.Process href "" "publisher.core.Process"

Methods:

  • add_callback –

    Adds a callback function to be executed after the process finishes.

  • add_error –

    Adds an error to the list of errors.

  • fix_method –

    Provides a method to fix issues detected by the check.

  • process –

    The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

  • set_status –

    Sets the status of the process.

Attributes:

callbacks property 

callbacks: Dict[Type[Callback], Callable]

Returns the dict of registered callbacks.

context property 

context: Context

Returns the shared context.

errors property 

errors: List[ErrorProcess]

Returns the list of errors encountered during execution.

status property 

status: StatusProcess

Returns the current status of the process.

add_callback 

add_callback(callback_type: Callback, callback: callable) -> None

Adds a callback function to be executed after the process finishes.

add_error 

add_error(error: Optional[str] = 'Unknown error', details: Optional[str] = 'Unknown details', items: Optional[List[str]] = None) -> None

Adds an error to the list of errors.

Parameters:

  • error
    (str, default: 'Unknown error' ) –

    The error message or name.

  • details
    (str, default: 'Unknown details' ) –

    Detailed description of the error.

  • items
    (list, default: None ) –

    Optional list of items related to the error.

fix_method 

fix_method() -> None

Provides a method to fix issues detected by the check.

process abstractmethod 

process(context: Context) -> None

The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

Parameters:

  • context
    (Context) –

    The shared context for accessing and storing data during execution.

Raises:

  • NotImplementedError –

    If the subclass does not implement this method.

set_status 

set_status(status: StatusProcess) -> None

Sets the status of the process.

Parameters:

Collect 

Collect(manager)


              flowchart LR
              publisher.core.Collect[Collect]
              publisher.core.Process[Process]

                              publisher.core.Process --> publisher.core.Collect
                


              click publisher.core.Collect href "" "publisher.core.Collect"
              click publisher.core.Process href "" "publisher.core.Process"

A class representing the file collection process.

This class is responsible for handling the collection of files, storing the collected value, and updating the shared context with the collected data.

Attributes:

  • name (str) –

    The name of the collection process.

  • collect_type (Type) –

    The expected type of the collected data (default is str).

  • _value (Any) –

    The collected value, initially set to None.

Methods:

  • add_callback –

    Adds a callback function to be executed after the process finishes.

  • add_error –

    Adds an error to the list of errors.

  • process –

    The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

  • set_status –

    Sets the status of the process.

callbacks property 

callbacks: Dict[Type[Callback], Callable]

Returns the dict of registered callbacks.

context property 

context: Context

Returns the shared context.

errors property 

errors: List[ErrorProcess]

Returns the list of errors encountered during execution.

status property 

status: StatusProcess

Returns the current status of the process.

value property writable 

value: Any

Returns the current collected value.

add_callback 

add_callback(callback_type: Callback, callback: callable) -> None

Adds a callback function to be executed after the process finishes.

add_error 

add_error(error: Optional[str] = 'Unknown error', details: Optional[str] = 'Unknown details', items: Optional[List[str]] = None) -> None

Adds an error to the list of errors.

Parameters:

  • error
    (str, default: 'Unknown error' ) –

    The error message or name.

  • details
    (str, default: 'Unknown details' ) –

    Detailed description of the error.

  • items
    (list, default: None ) –

    Optional list of items related to the error.

process abstractmethod 

process(context: Context) -> None

The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

Parameters:

  • context
    (Context) –

    The shared context for accessing and storing data during execution.

Raises:

  • NotImplementedError –

    If the subclass does not implement this method.

set_status 

set_status(status: StatusProcess) -> None

Sets the status of the process.

Parameters:

Context 

Context()

A shared context object for storing and accessing data across different processes.

Methods:

  • get_data –

    Retrieves data from the context by key.

  • set_data –

    Stores or updates data in the context.

get_data 

get_data(key: str) -> Any

Retrieves data from the context by key.

Parameters:

  • key
    (str) –

    The key or identifier for the data.

Returns:

  • Any ( Any ) –

    The value associated with the key, or None if not found.

Example

files = context.get_data('collected_files')

set_data 

set_data(key: str, value: Any) -> None

Stores or updates data in the context.

Parameters:

  • key
    (str) –

    The key or identifier for the data.

  • value
    (Any) –

    The data to store in the context.

Example

context.set_data('collected_files', ['file1.txt', 'file2.txt'])

ErrorProcess 

ErrorProcess(error: Optional[str] = 'Unknown error', details: Optional[str] = 'Unknown details', items: Optional[List[str]] = None)

Class to represent an error that occurs during process execution.

Attributes:

  • error (str) –

    A description of the error.

  • details (str) –

    Additional details about the error.

  • items (list) –

    Optional list of items or data related to the error.

details property 

details: str

Returns additional details about the error.

error property 

error: str

Returns the error description.

items property 

items: List[str]

Returns the list of items related to the error.

Extract 

Extract(manager: Manager)


              flowchart LR
              publisher.core.Extract[Extract]
              publisher.core.Process[Process]

                              publisher.core.Process --> publisher.core.Extract
                


              click publisher.core.Extract href "" "publisher.core.Extract"
              click publisher.core.Process href "" "publisher.core.Process"

A class representing the file extraction process.

Methods:

  • add_callback –

    Adds a callback function to be executed after the process finishes.

  • add_error –

    Adds an error to the list of errors.

  • process –

    The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

  • set_status –

    Sets the status of the process.

Attributes:

callbacks property 

callbacks: Dict[Type[Callback], Callable]

Returns the dict of registered callbacks.

context property 

context: Context

Returns the shared context.

errors property 

errors: List[ErrorProcess]

Returns the list of errors encountered during execution.

status property 

status: StatusProcess

Returns the current status of the process.

add_callback 

add_callback(callback_type: Callback, callback: callable) -> None

Adds a callback function to be executed after the process finishes.

add_error 

add_error(error: Optional[str] = 'Unknown error', details: Optional[str] = 'Unknown details', items: Optional[List[str]] = None) -> None

Adds an error to the list of errors.

Parameters:

  • error
    (str, default: 'Unknown error' ) –

    The error message or name.

  • details
    (str, default: 'Unknown details' ) –

    Detailed description of the error.

  • items
    (list, default: None ) –

    Optional list of items related to the error.

process abstractmethod 

process(context: Context) -> None

The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

Parameters:

  • context
    (Context) –

    The shared context for accessing and storing data during execution.

Raises:

  • NotImplementedError –

    If the subclass does not implement this method.

set_status 

set_status(status: StatusProcess) -> None

Sets the status of the process.

Parameters:

Manager 

Manager()

Manager class that manages the registration and execution of different processes, including collecting, checking, extraction, and pushing to the database.

Attributes:

  • context (Context) –

    The shared context for managing data among the different processes.

  • collectors (List[Collect]) –

    A list of collecting processes.

  • checks (List[Check]) –

    A list of checking processes.

  • extractors (List[Extract]) –

    A list of extraction processes.

  • pushes (List[Push]) –

    A list of push processes for database operations.

Methods:

  • add_callback –

    Adds a callback function to be executed after the process finishes.

  • processes –

    Returns a dictionary with all the registered processes.

  • publish –

    Executes the registered processes in sequence.

  • register –

    Registers a process in the correct category based on its type.

callbacks property 

callbacks: Dict[Type[Callback], Callable]

Returns the dict of registered callbacks.

context property 

context

Gets the shared context for the Manager.

add_callback 

add_callback(callback_type: Callback, callback: callable) -> None

Adds a callback function to be executed after the process finishes.

processes 

processes() -> Dict[str, List[Process]]

Returns a dictionary with all the registered processes.

Returns:

  • dict ( Dict[str, List[Process]] ) –

    A dictionary with process types as keys and lists of instances as values.

publish 

publish() -> Iterator[str]

Executes the registered processes in sequence.

register 

register(process: Process) -> None

Registers a process in the correct category based on its type.

Parameters:

  • process
    (Process) –

    An process of a subclass of Process to be registered.

Raises:

  • ValueError –

    If the process type is unknown.

Process 

Process(manager: Manager)


              flowchart LR
              publisher.core.Process[Process]

              

              click publisher.core.Process href "" "publisher.core.Process"
            
Abstract base class for all processes in the task publishing pipeline.

Attributes:
    name (str): The name of the process.
    compulsory (bool): Indicates if the process is mandatory.
    status (bool): The status of the process (True for success, False for failure, None for not yet executed).
    info (str): Information or description of the process.
    errors (list): A list of ErrorProcess objects representing errors encountered during execution.
    context (Context): A reference to the shared Context object for data exchange.
    callbacks (dict): A dict of callback functions to be executed when a specified action occurs.

.

Methods:

  • add_callback –

    Adds a callback function to be executed after the process finishes.

  • add_error –

    Adds an error to the list of errors.

  • process –

    The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

  • set_status –

    Sets the status of the process.

Attributes:

callbacks property 

callbacks: Dict[Type[Callback], Callable]

Returns the dict of registered callbacks.

context property 

context: Context

Returns the shared context.

errors property 

errors: List[ErrorProcess]

Returns the list of errors encountered during execution.

status property 

status: StatusProcess

Returns the current status of the process.

add_callback 

add_callback(callback_type: Callback, callback: callable) -> None

Adds a callback function to be executed after the process finishes.

add_error 

add_error(error: Optional[str] = 'Unknown error', details: Optional[str] = 'Unknown details', items: Optional[List[str]] = None) -> None

Adds an error to the list of errors.

Parameters:

  • error
    (str, default: 'Unknown error' ) –

    The error message or name.

  • details
    (str, default: 'Unknown details' ) –

    Detailed description of the error.

  • items
    (list, default: None ) –

    Optional list of items related to the error.

process abstractmethod 

process(context: Context) -> None

The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

Parameters:

  • context
    (Context) –

    The shared context for accessing and storing data during execution.

Raises:

  • NotImplementedError –

    If the subclass does not implement this method.

set_status 

set_status(status: StatusProcess) -> None

Sets the status of the process.

Parameters:

Push 

Push(manager: Manager)


              flowchart LR
              publisher.core.Push[Push]
              publisher.core.Process[Process]

                              publisher.core.Process --> publisher.core.Push
                


              click publisher.core.Push href "" "publisher.core.Push"
              click publisher.core.Process href "" "publisher.core.Process"

A class representing the process to push data to the database.

Methods:

  • add_callback –

    Adds a callback function to be executed after the process finishes.

  • add_error –

    Adds an error to the list of errors.

  • process –

    The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

  • set_status –

    Sets the status of the process.

Attributes:

callbacks property 

callbacks: Dict[Type[Callback], Callable]

Returns the dict of registered callbacks.

context property 

context: Context

Returns the shared context.

errors property 

errors: List[ErrorProcess]

Returns the list of errors encountered during execution.

status property 

status: StatusProcess

Returns the current status of the process.

add_callback 

add_callback(callback_type: Callback, callback: callable) -> None

Adds a callback function to be executed after the process finishes.

add_error 

add_error(error: Optional[str] = 'Unknown error', details: Optional[str] = 'Unknown details', items: Optional[List[str]] = None) -> None

Adds an error to the list of errors.

Parameters:

  • error
    (str, default: 'Unknown error' ) –

    The error message or name.

  • details
    (str, default: 'Unknown details' ) –

    Detailed description of the error.

  • items
    (list, default: None ) –

    Optional list of items related to the error.

process abstractmethod 

process(context: Context) -> None

The core logic of the process. This method should be implemented by subclasses with specific logic for the process.

Parameters:

  • context
    (Context) –

    The shared context for accessing and storing data during execution.

Raises:

  • NotImplementedError –

    If the subclass does not implement this method.

set_status 

set_status(status: StatusProcess) -> None

Sets the status of the process.

Parameters:

StatusProcess

Enum-like class representing the status of a process.

log_execution 

log_execution(func)

Decorator to log the execution of process methods.