Matrices en 3 colonnes¶
Représentation d’une matrice avec Spark / Map / Reduce.
Ce notebook propose d’implémenter un produit matriciel sous Spark. Spark comme SQL n’aime pas trop avoir un nombre de colonnes variables. La première étape consiste à transformer les matrices en tableau de trois colonnes
.
Session spark with no cluster¶
Spark est censé tourner sur un cluster. Mais ce n’est pas essentielle pour comprendre la logique. Le notebook tourne donc en local.
[1]:
from pyspark.sql import SparkSession
# Create a SparkSession
spark = SparkSession.builder.appName("MySparkApp").master("local[*]").getOrCreate()
sc = spark.sparkContext
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/11/04 09:14:15 WARN Utils: Your hostname, xadupre2025, resolves to a loopback address: 127.0.1.1; using 172.28.185.142 instead (on interface eth0)
26/11/04 09:14:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
~/vv/this312/lib/python3.12/site-packages/pyspark/testing/utils.py:127: FutureWarning: PySpark does not yet fully support pandas >= 3.0.0. Some features may not work correctly. It is recommended to use pandas < 3.0.0 for now.
require_minimum_pandas_version()
26/11/04 09:14:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Création d’une matrice aléatoire¶
[2]:
from numpy.random import rand
rnd1 = rand(10, 10)
rnd2 = rand(10, 2)
rnd1 @ rnd2
[2]:
array([[2.29102946, 3.53847793],
[1.58850851, 2.78999386],
[1.40056768, 3.71886219],
[1.45873572, 3.49729928],
[2.2405258 , 3.47090722],
[1.15133072, 1.79002816],
[1.45400249, 3.65802814],
[2.55213675, 2.82495689],
[2.04015374, 2.78118401],
[2.64322208, 4.46851586]])
[3]:
import pandas
df1 = pandas.DataFrame(rnd1)
df2 = pandas.DataFrame(rnd2)
df2
[3]:
| 0 | 1 | |
|---|---|---|
| 0 | 0.918575 | 0.259207 |
| 1 | 0.123740 | 0.395866 |
| 2 | 0.102176 | 0.715428 |
| 3 | 0.573558 | 0.249657 |
| 4 | 0.670691 | 0.726082 |
| 5 | 0.039036 | 0.889782 |
| 6 | 0.742978 | 0.305118 |
| 7 | 0.193894 | 0.969908 |
| 8 | 0.136791 | 0.902031 |
| 9 | 0.229301 | 0.530934 |
[4]:
df1.to_csv("rnd1.txt", sep="\t", header=None, index=False)
df2.to_csv("rnd2.txt", sep="\t", header=None, index=False)
[5]:
mat1 = spark.sparkContext.textFile("rnd1.txt")
Conversion d’une matrice au format Spark¶
Lorsqu’un traitement est distribué en Map/Reduce, il n’est pas possible de s’appuyer sur l’ordre dans lequel sont traitées les lignes. Le plus est d’ajouter cette information sur chaque ligne plutôt que de chercher à la récupérer.
[6]:
df1.to_csv("rnd1.txt", sep="\t", header=None, index=True)
df2.to_csv("rnd2.txt", sep="\t", header=None, index=True)
[7]:
def process_mat_row(row):
values = row.split("\t")
index = int(values[0])
values = [float(_) for _ in values[1:]]
return [[index, j, v] for j, v in enumerate(values)]
[8]:
mat1 = sc.textFile("rnd1.txt")
new_mat1 = mat1.flatMap(process_mat_row)
new_mat1.take(12)
[8]:
[[0, 0, 0.4276345283939371],
[0, 1, 0.7104397235643978],
[0, 2, 0.5562289328230193],
[0, 3, 0.5985481810182711],
[0, 4, 0.7160591716006677],
[0, 5, 0.357351163573274],
[0, 6, 0.8440993774047496],
[0, 7, 0.9658516656607489],
[0, 8, 0.566678784552112],
[0, 9, 0.10479419449315441],
[1, 0, 0.3435094985776058],
[1, 1, 0.9361193497095325]]
[9]:
mat2 = sc.textFile("rnd2.txt")
new_mat2 = mat2.flatMap(process_mat_row)
new_mat2.take(12)
[9]:
[[0, 0, 0.9185747759250762],
[0, 1, 0.2592066596075602],
[1, 0, 0.12374028350879862],
[1, 1, 0.39586600274279726],
[2, 0, 0.10217587658941296],
[2, 1, 0.7154281868182252],
[3, 0, 0.5735578686875115],
[3, 1, 0.24965710575045208],
[4, 0, 0.6706906195358108],
[4, 1, 0.7260817300612702],
[5, 0, 0.039036422428009976],
[5, 1, 0.8897817999507714]]
Produit matriciel¶
Il faut d’abord faire la jointure avec la méthode join. Il faut que la clé soit sur la première colonne.
[10]:
def key_ij(row):
return row[0], (row[1], row[2])
def key_ji(row):
return row[1], (row[0], row[2])
mat_join = new_mat1.map(key_ji).join(new_mat2.map(key_ij))
mat_join.take(12)
[10]:
[(0, ((0, 0.4276345283939371), (0, 0.9185747759250762))),
(0, ((0, 0.4276345283939371), (1, 0.2592066596075602))),
(0, ((1, 0.3435094985776058), (0, 0.9185747759250762))),
(0, ((1, 0.3435094985776058), (1, 0.2592066596075602))),
(0, ((2, 0.061827447722926254), (0, 0.9185747759250762))),
(0, ((2, 0.061827447722926254), (1, 0.2592066596075602))),
(0, ((3, 0.18643464032502244), (0, 0.9185747759250762))),
(0, ((3, 0.18643464032502244), (1, 0.2592066596075602))),
(0, ((4, 0.5724205231773295), (0, 0.9185747759250762))),
(0, ((4, 0.5724205231773295), (1, 0.2592066596075602))),
(0, ((5, 0.4975704022747822), (0, 0.9185747759250762))),
(0, ((5, 0.4975704022747822), (1, 0.2592066596075602)))]
On effectue le produit matriciel.
[11]:
def produit_matriciel(row):
index, ((i, v1), (j, v2)) = row
return i, j, v1 * v2
produit = mat_join.map(produit_matriciel)
produit.take(12)
[11]:
[(0, 0, 0.39281429109728644),
(0, 1, 0.11084571763784679),
(1, 0, 0.31553916068405957),
(1, 1, 0.08903994966976915),
(2, 0, 0.056793133938106345),
(2, 1, 0.016026086196320767),
(3, 0, 0.17125415796122967),
(3, 1, 0.048325100353786),
(4, 0, 0.5258110538125303),
(4, 1, 0.14837521170360757),
(5, 0, 0.4570556207765081),
(5, 1, 0.12897356189323625)]
Il ne reste plus qu’à agréger reduceByKey. La documentation fournit un exemple facilement transposable. Elle indique aussi : Merge the values for each key using an associative and commutative reduce function. Pourquoi précise-t-elle associative et commutative ? Cela signifie que le résultat ne dépend pas de l’ordre dans lequel l’agrégation est réalisée et qu’on peut commencer à agréger sans attendre d’avoir regroupé toutes les valeurs associées à une clé.
Cas 1 : groupBy + agrégation qui commence une fois les valeurs regroupées
Cas 2 : reduceByKey + agrégation qui commence dès les premières valeurs regroupées
Le cas 2 est moins consommateur en terme de données. Le cas 1 n’est possible que si les valeurs agrégées ne sont pas trop nombreuses. Ca tombe bien, dans notre cas, le cas 2 convient.
[12]:
from operator import add
final = produit.map(lambda row: ((row[0], row[1]), row[2])).reduceByKey(add)
aslist = final.collect()
aslist.sort()
aslist
[12]:
[((0, 0), 2.2910294615369833),
((0, 1), 3.5384779259752115),
((1, 0), 1.588508508773101),
((1, 1), 2.7899938642491398),
((2, 0), 1.400567677321856),
((2, 1), 3.7188621947679996),
((3, 0), 1.4587357182350686),
((3, 1), 3.4972992791592126),
((4, 0), 2.24052580041269),
((4, 1), 3.470907222987571),
((5, 0), 1.1513307163420712),
((5, 1), 1.7900281634340787),
((6, 0), 1.454002494006837),
((6, 1), 3.658028135367462),
((7, 0), 2.552136746927374),
((7, 1), 2.8249568885507244),
((8, 0), 2.0401537352713808),
((8, 1), 2.7811840133848817),
((9, 0), 2.6432220797579706),
((9, 1), 4.468515860388137)]
Résultat initial :
[13]:
rnd1 @ rnd2
[13]:
array([[2.29102946, 3.53847793],
[1.58850851, 2.78999386],
[1.40056768, 3.71886219],
[1.45873572, 3.49729928],
[2.2405258 , 3.47090722],
[1.15133072, 1.79002816],
[1.45400249, 3.65802814],
[2.55213675, 2.82495689],
[2.04015374, 2.78118401],
[2.64322208, 4.46851586]])
Même algorithme avec les Spark DataFrame¶
On a besoin de réaliser un flatMap. Une façon de faire est de créer des colonnes qui sont de type composé : un tableau, une structure. La multiplication des lignes est obtenue avec la fonction explode.
[14]:
schema = ["index"] + ["c%d" % i for i in range(1, 11)]
mat1 = spark.createDataFrame(
pandas.read_csv("rnd1.txt", header=None, sep="\t"), schema=schema
)
~/vv/this312/lib/python3.12/site-packages/pyspark/sql/pandas/conversion.py:676: FutureWarning: PySpark does not yet fully support pandas >= 3.0.0. Some features may not work correctly. It is recommended to use pandas < 3.0.0 for now.
require_minimum_pandas_version()
~/vv/this312/lib/python3.12/site-packages/pyspark/sql/pandas/conversion.py:953: FutureWarning: PySpark does not yet fully support pandas >= 3.0.0. Some features may not work correctly. It is recommended to use pandas < 3.0.0 for now.
require_minimum_pandas_version()
[15]:
mat1.printSchema()
root
|-- index: long (nullable = true)
|-- c1: double (nullable = true)
|-- c2: double (nullable = true)
|-- c3: double (nullable = true)
|-- c4: double (nullable = true)
|-- c5: double (nullable = true)
|-- c6: double (nullable = true)
|-- c7: double (nullable = true)
|-- c8: double (nullable = true)
|-- c9: double (nullable = true)
|-- c10: double (nullable = true)
[16]:
schema = ["index"] + ["c%d" % i for i in range(1, 3)]
mat2 = spark.createDataFrame(
pandas.read_csv("rnd2.txt", header=None, sep="\t"), schema=schema
)
[17]:
mat2.printSchema()
root
|-- index: long (nullable = true)
|-- c1: double (nullable = true)
|-- c2: double (nullable = true)
Nous allons avoir besoin de quelques-uns des fonctions et types suivant :
Je recommande le type FloatType qui prend deux fois moins de place pour une précision moindre mais suffisante dans la plupart des cas.
[18]:
from pyspark.sql.types import (
ArrayType,
StructField,
StructType,
DoubleType,
IntegerType,
)
from pyspark.sql.functions import explode, posexplode, array
from pyspark.sql import Row
[19]:
cols = ["c%d" % i for i in range(1, 11)]
mat1_array = mat1.select(mat1.index, array(*cols).alias("x"))
mat1_array.printSchema()
root
|-- index: long (nullable = true)
|-- x: array (nullable = false)
| |-- element: double (containsNull = true)
[20]:
mat1_exploded = mat1_array.select("index", posexplode("x"))
mat1_exploded.printSchema()
root
|-- index: long (nullable = true)
|-- pos: integer (nullable = false)
|-- col: double (nullable = true)
[21]:
mat1.toPandas().shape, mat1_exploded.toPandas().shape
~/vv/this312/lib/python3.12/site-packages/pyspark/sql/pandas/conversion.py:324: FutureWarning: PySpark does not yet fully support pandas >= 3.0.0. Some features may not work correctly. It is recommended to use pandas < 3.0.0 for now.
require_minimum_pandas_version()
~/vv/this312/lib/python3.12/site-packages/pyspark/sql/pandas/conversion.py:324: FutureWarning: PySpark does not yet fully support pandas >= 3.0.0. Some features may not work correctly. It is recommended to use pandas < 3.0.0 for now.
require_minimum_pandas_version()
[21]:
((10, 11), (100, 3))
On recommence le même procédé pour l’autre matrice.
[22]:
cols = ["c%d" % i for i in range(1, 3)]
mat2_array = mat2.select(mat2.index, array(*cols).alias("x"))
mat2_exploded = mat2_array.select("index", posexplode("x"))
Il ne reste plus qu’à faire le produit avec la méthode join après avoir renommé les colonnes avant la jointure pour éviter les ambiguïtés.
[23]:
mat2_exp2 = (
mat2_exploded.withColumnRenamed("index", "index2")
.withColumnRenamed("pos", "pos2")
.withColumnRenamed("col", "col2")
)
produit = mat1_exploded.join(mat2_exp2, mat1_exploded.pos == mat2_exp2.index2)
[24]:
produit.printSchema()
root
|-- index: long (nullable = true)
|-- pos: integer (nullable = false)
|-- col: double (nullable = true)
|-- index2: long (nullable = true)
|-- pos2: integer (nullable = false)
|-- col2: double (nullable = true)
[25]:
produit.toPandas().head()
~/vv/this312/lib/python3.12/site-packages/pyspark/sql/pandas/conversion.py:324: FutureWarning: PySpark does not yet fully support pandas >= 3.0.0. Some features may not work correctly. It is recommended to use pandas < 3.0.0 for now.
require_minimum_pandas_version()
[25]:
| index | pos | col | index2 | pos2 | col2 | |
|---|---|---|---|---|---|---|
| 0 | 0 | 0 | 0.427635 | 0 | 1 | 0.259207 |
| 1 | 0 | 0 | 0.427635 | 0 | 0 | 0.918575 |
| 2 | 0 | 1 | 0.710440 | 1 | 1 | 0.395866 |
| 3 | 0 | 1 | 0.710440 | 1 | 0 | 0.123740 |
| 4 | 0 | 2 | 0.556229 | 2 | 1 | 0.715428 |
[26]:
prod = produit.select(
produit.index.alias("i"),
produit.pos2.alias("j"),
(produit.col * produit.col2).alias("val"),
)
final = prod.groupby("i", "j").sum("val")
[27]:
final.printSchema()
root
|-- i: long (nullable = true)
|-- j: integer (nullable = false)
|-- sum(val): double (nullable = true)
[28]:
df = final.toPandas()
~/vv/this312/lib/python3.12/site-packages/pyspark/sql/pandas/conversion.py:324: FutureWarning: PySpark does not yet fully support pandas >= 3.0.0. Some features may not work correctly. It is recommended to use pandas < 3.0.0 for now.
require_minimum_pandas_version()
[29]:
df.sort_values(["i", "j"]).head()
[29]:
| i | j | sum(val) | |
|---|---|---|---|
| 0 | 0 | 0 | 2.291029 |
| 1 | 0 | 1 | 3.538478 |
| 3 | 1 | 0 | 1.588509 |
| 2 | 1 | 1 | 2.789994 |
| 4 | 2 | 0 | 1.400568 |
[30]:
df.shape
[30]:
(20, 3)
Alternatives¶
Plutôt que d’avoir un table où chaque ligne représente trois coefficients, pourrions-nous considérer une matrice par bloc de 16x16 ? Que gagnerait-on ?
Fin¶
[31]:
spark.stop()