pandas_streaming.df.connex_split

exception pandas_streaming.df.connex_split.ImbalancedSplitException[source][source]

Raised when an imbalanced split is detected.

pandas_streaming.df.connex_split.train_test_apart_stratify(df: DataFrame, group, test_size: float | None = 0.25, train_size: float | None = None, stratify: str | None = None, force: bool = False, random_state: int | None = None, sorted_indices: bool = False) → Tuple[StreamingDataFrame, StreamingDataFrame][source][source]

This split is for a specific case where data is linked in one way. Let’s assume we have two ids as we have for online sales: (product id, category id). A product can have multiple categories. We need to have distinct products on train and test but common categories on both sides.

Parameters:

  • df – pandas.DataFrame

  • group – columns name for the ids

  • test_size – ratio for the test partition (if train_size is not specified)

  • train_size – ratio for the train partition

  • stratify – column holding the stratification

  • force – if True, tries to get at least one example on the test side for each value of the column stratify

  • random_state – seed for random generators

  • sorted_indices – sort index first, see issue 41 <https://github.com/sdpython/pandas-streaming/issues/41>

Returns:

Two see StreamingDataFrame, one for train, one for test.

The list of ids must hold in memory. There is no streaming implementation for the ids. This split was implemented for a case of a multi-label classification. A category (stratify) is not exclusive and an observation can be assigned to multiple categories. In that particular case, the method sklearn.model_selection.train_test_split() can not directly be used.

<<<

import pandas
from pandas_streaming.df import train_test_apart_stratify

df = pandas.DataFrame(
    [dict(a=1, b="e"), dict(a=1, b="f"), dict(a=2, b="e"), dict(a=2, b="f")]
)

train, test = train_test_apart_stratify(df, group="a", stratify="b", test_size=0.5)
print(train)
print("-----------")
print(test)

>>>

       a  b
    2  2  e
    3  2  f
    -----------
       a  b
    0  1  e
    1  1  f
pandas_streaming.df.connex_split.train_test_connex_split(df, groups, test_size=0.25, train_size=None, stratify=None, hash_size=9, unique_rows=False, shuffle=True, fail_imbalanced=0.05, keep_balance=None, stop_if_bigger=None, return_cnx=False, must_groups=None, random_state=None, verbose=0)[source][source]

This split is for a specific case where data is linked in many ways. Let’s assume we have three ids as we have for online sales: (product id, user id, card id). As we may need to compute aggregated features, we need every id not to be present in both train and test set. The function computes the connected components and breaks each of them in two parts for train and test.

Parameters:

  • df – pandas.DataFrame

  • groups – columns name for the ids

  • test_size – ratio for the test partition (if train_size is not specified)

  • train_size – ratio for the train partition

  • stratify – column holding the stratification

  • hash_size – size of the hash to cache information about partition

  • unique_rows – ensures that rows are unique

  • shuffle – shuffles before the split

  • fail_imbalanced – raises an exception if relative weights difference is higher than this value

  • stop_if_bigger – (float) stops a connected components from being bigger than this ratio of elements, this should not be used unless a big components emerges, the algorithm stops merging but does not guarantee it returns the best cut, the value should be close to 0

  • keep_balance – (float), if not None, does not merge connected components if their relative sizes are too different, the value should be close to 1

  • return_cnx – returns connected components as a third results

  • must_groups – column name for ids which must not be shared by train/test partitions

  • random_state – seed for random generator

  • verbose – verbosity (uses logging)

Returns:

Two see StreamingDataFrame, one for train, one for test.

The list of ids must hold in memory. There is no streaming implementation for the ids.

Splits a dataframe, keep ids in separate partitions

In some data science problems, rows are not independant and share common value, most of the time ids. In some specific case, multiple ids from different columns are connected and must appear in the same partition. Testing that each id column is evenly split and do not appear in both sets in not enough. Connected components are needed.

<<<

from pandas import DataFrame
from pandas_streaming.df import train_test_connex_split

df = DataFrame(
    [
        dict(user="UA", prod="PAA", card="C1"),
        dict(user="UA", prod="PB", card="C1"),
        dict(user="UB", prod="PC", card="C2"),
        dict(user="UB", prod="PD", card="C2"),
        dict(user="UC", prod="PAA", card="C3"),
        dict(user="UC", prod="PF", card="C4"),
        dict(user="UD", prod="PG", card="C5"),
    ]
)

train, test = train_test_connex_split(
    df, test_size=0.5, groups=["user", "prod", "card"], fail_imbalanced=0.6
)

print(train)
print(test)

>>>

      user prod card  connex  weight
    0   UB   PD   C2       1       1
    1   UB   PC   C2       1       1
      user prod card  connex  weight
    0   UC  PAA   C3       0       1
    1   UD   PG   C5       2       1
    2   UA   PB   C1       0       1
    3   UC   PF   C4       0       1
    4   UA  PAA   C1       0       1

If return_cnx is True, the third results contains:

  • connected components for each id

  • the dataframe with connected components as a new column

<<<

from pandas import DataFrame
from pandas_streaming.df import train_test_connex_split

df = DataFrame(
    [
        dict(user="UA", prod="PAA", card="C1"),
        dict(user="UA", prod="PB", card="C1"),
        dict(user="UB", prod="PC", card="C2"),
        dict(user="UB", prod="PD", card="C2"),
        dict(user="UC", prod="PAA", card="C3"),
        dict(user="UC", prod="PF", card="C4"),
        dict(user="UD", prod="PG", card="C5"),
    ]
)

train, test, cnx = train_test_connex_split(
    df,
    test_size=0.5,
    groups=["user", "prod", "card"],
    fail_imbalanced=0.6,
    return_cnx=True,
)

print(cnx[0])
print(cnx[1])

>>>

    {('user', 'UC'): 0, ('prod', 'PAA'): 0, ('card', 'C3'): 0, ('user', 'UD'): 1, ('prod', 'PG'): 1, ('card', 'C5'): 1, ('user', 'UB'): 2, ('prod', 'PD'): 2, ('card', 'C2'): 2, ('user', 'UA'): 0, ('card', 'C1'): 0, ('prod', 'PC'): 2, ('prod', 'PB'): 0, ('prod', 'PF'): 0, ('card', 'C4'): 0}
      user prod card  connex  weight
    4   UC  PAA   C3       0       1
    6   UD   PG   C5       1       1
    3   UB   PD   C2       2       1
    0   UA  PAA   C1       0       1
    2   UB   PC   C2       2       1
    1   UA   PB   C1       0       1
    5   UC   PF   C4       0       1
pandas_streaming.df.connex_split.train_test_split_weights(df, weights=None, test_size=0.25, train_size=None, shuffle=True, fail_imbalanced=0.05, random_state=None)[source][source]

Splits a database in train/test given, every row can have a different weight.

Parameters:

  • df – pandas.DataFrame or see StreamingDataFrame

  • weights – None or weights or weights column name

  • test_size – ratio for the test partition (if train_size is not specified)

  • train_size – ratio for the train partition

  • shuffle – shuffles before the split

  • fail_imbalanced – raises an exception if relative weights difference is higher than this value

  • random_state – seed for random generators

Returns:

train and test pandas.DataFrame

If the dataframe is not shuffled first, the function will produce two datasets which are unlikely to be randomized as the function tries to keep equal weights among both paths without using randomness.