Spark ML - Databricks ML Associate¶

Overview¶

This section covers Spark MLlib and the Pipeline API, representing 17% of the exam. You need to understand Transformers, Estimators, Pipelines, and distributed model training with Spark.

📖 MLlib Guide - Spark ML reference 📖 Pipeline API - Pipeline components

Key Topics¶

1. Pipeline API Concepts¶

The Spark ML Pipeline API provides a uniform interface for building ML workflows. It chains together data preparation and model training steps.

Core Components: | Component | Description | Example | |-----------|-------------|---------| | Transformer | Transforms a DataFrame (adds columns) | VectorAssembler, StringIndexer | | Estimator | Fits on data to produce a Transformer | LogisticRegression, RandomForestClassifier | | Pipeline | Chain of Transformers and Estimators | Full ML workflow | | PipelineModel | Fitted Pipeline (all stages fitted) | Ready for predictions |

Basic Pipeline Example:

from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler, StringIndexer, StandardScaler
from pyspark.ml.classification import LogisticRegression

# Define stages
indexer = StringIndexer(inputCol="category", outputCol="category_index")
assembler = VectorAssembler(inputCols=["feature1", "feature2", "category_index"],
                            outputCol="features")
scaler = StandardScaler(inputCol="features", outputCol="scaled_features")
lr = LogisticRegression(featuresCol="scaled_features", labelCol="label")

# Build and fit pipeline
pipeline = Pipeline(stages=[indexer, assembler, scaler, lr])
model = pipeline.fit(train_df)

# Make predictions
predictions = model.transform(test_df)

2. Common Transformers¶

📖 Feature Transformers - Feature transformation

# VectorAssembler - required before any ML algorithm
assembler = VectorAssembler(
    inputCols=["age", "income", "score"],
    outputCol="features"
)

# StringIndexer - convert categorical strings to numbers
indexer = StringIndexer(inputCol="city", outputCol="city_index")

# OneHotEncoder - create binary vectors from indices
encoder = OneHotEncoder(inputCol="city_index", outputCol="city_vec")

Key Concepts: - VectorAssembler is almost always required - ML algorithms expect a single feature vector - StringIndexer assigns indices by frequency (most frequent = 0) - OneHotEncoder creates sparse vectors (memory efficient) - Order of transformers matters - indexing before encoding, assembly before scaling

3. Common Estimators (ML Algorithms)¶

Classification: | Algorithm | Use Case | Key Parameters | |-----------|----------|----------------| | LogisticRegression | Binary/multiclass | maxIter, regParam, elasticNetParam | | RandomForestClassifier | Ensemble classification | numTrees, maxDepth, maxBins | | GBTClassifier | Gradient-boosted trees | maxIter, maxDepth, stepSize | | DecisionTreeClassifier | Simple interpretable model | maxDepth, maxBins |

Regression: | Algorithm | Use Case | Key Parameters | |-----------|----------|----------------| | LinearRegression | Linear relationships | maxIter, regParam, elasticNetParam | | RandomForestRegressor | Ensemble regression | numTrees, maxDepth | | GBTRegressor | Gradient-boosted trees | maxIter, maxDepth |

Clustering: | Algorithm | Use Case | Key Parameters | |-----------|----------|----------------| | KMeans | Partitioning clusters | k, maxIter, seed | | BisectingKMeans | Hierarchical clustering | k, maxIter |

4. Model Selection and Tuning¶

📖 CrossValidator - Model selection

CrossValidator:

from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.ml.evaluation import BinaryClassificationEvaluator

# Define parameter grid
paramGrid = (ParamGridBuilder()
    .addGrid(lr.regParam, [0.01, 0.1, 1.0])
    .addGrid(lr.maxIter, [50, 100, 200])
    .build())

# Define evaluator
evaluator = BinaryClassificationEvaluator(metricName="areaUnderROC")

# Cross-validate
cv = CrossValidator(
    estimator=pipeline,
    estimatorParamMaps=paramGrid,
    evaluator=evaluator,
    numFolds=5
)
cv_model = cv.fit(train_df)
best_model = cv_model.bestModel

TrainValidationSplit:

from pyspark.ml.tuning import TrainValidationSplit

tvs = TrainValidationSplit(
    estimator=pipeline,
    estimatorParamMaps=paramGrid,
    evaluator=evaluator,
    trainRatio=0.8
)
tvs_model = tvs.fit(train_df)

Key Concepts: - CrossValidator: k-fold cross-validation (more robust, slower) - TrainValidationSplit: single train/validation split (faster, less robust) - ParamGridBuilder defines the hyperparameter search space - Evaluator defines the metric to optimize - Both return the best model automatically

5. Evaluators¶

6. Saving and Loading Models¶

# Save a pipeline model
model.save("/path/to/model")

# Load a pipeline model
from pyspark.ml import PipelineModel
loaded_model = PipelineModel.load("/path/to/model")

# Make predictions with loaded model
predictions = loaded_model.transform(new_data)

Exam Tips for This Domain¶

1. Pipeline stages - Know the order: indexers, encoders, assembler, scaler, algorithm
2. VectorAssembler - Required before all Spark ML algorithms
3. CrossValidator vs TrainValidationSplit - CV is more robust; TVS is faster
4. ParamGridBuilder - Know how to define search grids
5. Evaluator metrics - Match the evaluator to the problem type
6. StringIndexer + OneHotEncoder - Standard pattern for categorical features

Documentation Links Summary¶