How to encrypt sensitive data in Databricks: step by step

João Barros 11 de September de 2026 5 min read

This tutorial shows how to encrypt columns with sensitive data (PII) in Databricks using PySpark and symmetric AES, to reduce the risk of exposure in development and reporting environments. Encrypting before persisting or sharing data helps meet security and privacy requirements, and limits the blast radius in case of unauthorized access.

Prerequisites

Databricks workspace with an active cluster (e.g., 2 to 8 nodes depending on volume) and permissions to create notebooks.
Permissions to read/write tables/files (mounts or Azure/ADLS/Blob storage).
Basic familiarity with PySpark and DataFrame; minimal knowledge of UDFs.
Encryption key (can be generated for testing; in production use Azure Key Vault or CMK via Databricks).
cryptography library installed on the cluster: in notebooks use %pip install cryptography or install as a Library on the cluster.

Step 1: Why and how to choose encryption

Encrypting columns protects data at rest and in transit. Here we use AES-256 in GCM mode (authenticated) to ensure confidentiality and integrity (detects corruption/tampering). AES-GCM adds a 16-byte tag and uses a 12-byte nonce per message. In practical terms, a 20-byte string typically results in an encrypted output of ~48 bytes before base64; base64 increases the final size by about 33% — plan storage and indexes accordingly.

For production: never put keys in code. Use Azure Key Vault integrated with Databricks Secrets or Customer-Managed Keys (CMK) for storage-level encryption. Access policy should follow the principle of least privilege. For auditing, combine with Unity Catalog for access control lists and with storage access logs.

Step 2: Prepare a PySpark notebook and generate/set key

Create a Python notebook in Databricks. For tests you can set the key inline (32 bytes for AES-256). In a real environment, retrieve the key via Databricks Secrets or Key Vault (for example: dbutils.secrets.get).

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.getOrCreate()

# 32-byte key for AES-256 (EXAMPLE ONLY for development)
encryption_key = b"0123456789abcdef0123456789abcdef"  # 32 bytes
# In production: key = dbutils.secrets.get(scope='myScope', key='enc-key')

Step 3: Encryption/decryption functions in Python

We will use the cryptography library for AES-GCM. Install it on the cluster with %pip install cryptography or add as a Library. The functions return base64 for direct storage in string columns.

from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import os, base64

def encrypt_value(plaintext: str, key: bytes) -> str:
    aesgcm = AESGCM(key)
    nonce = os.urandom(12)  # 96-bit nonce (required for GCM)
    ct = aesgcm.encrypt(nonce, plaintext.encode('utf-8'), None)
    # we store nonce + ciphertext+tag together and base64-encode
    return base64.b64encode(nonce + ct).decode('utf-8')

def decrypt_value(token_b64: str, key: bytes) -> str:
    data = base64.b64decode(token_b64)
    nonce, ct = data[:12], data[12:]
    aesgcm = AESGCM(key)
    return aesgcm.decrypt(nonce, ct, None).decode('utf-8')

Step 4: Example DataFrame with sensitive data

Create a sample DataFrame with names and emails. To test performance, try 10k, 100k or 1M rows and measure time. A quick example with 2 rows:

data = [
  (1, 'Alice', 'alice@example.com'),
  (2, 'Bob', 'bob@example.com'),
]
columns = ['id', 'name', 'email']
df = spark.createDataFrame(data, columns)
df.show()

Step 5: Apply encryption to a column with UDF

We use a UDF to apply encrypt_value to each row. Note: Python UDFs imply overhead—for 100k rows on a modest cluster it may take tens of seconds; for 1M rows expect minutes depending on resources. If performance is critical, consider encrypting at the ingestion layer (native stream), using UDFs in C/Scala, or leveraging native storage features.

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

encrypt_udf = udf(lambda s: encrypt_value(s, encryption_key), StringType())

df_encrypted = df.withColumn('email_encrypted', encrypt_udf(col('email'))).drop('email')
df_encrypted.show(truncate=False)

Step 6: Persist encrypted data

Save the encrypted DataFrame as Delta or Parquet. Remember that the encrypted column is a base64 string and takes more space. In production combine with storage-level encryption and use Unity Catalog to control table access. Example write:

df_encrypted.write.mode('overwrite').format('delta').save('/mnt/data/encrypted_users.delta')
# or save as a table
# df_encrypted.write.mode('overwrite').saveAsTable('encrypted_users')

Step 7: Decrypt for authorized use

To read and decrypt, load the data and apply a decryption UDF only in authorized sessions. Protect the key at runtime and limit who can run that notebook. An authorized session can decrypt only the necessary columns.

decrypt_udf = udf(lambda s: decrypt_value(s, encryption_key), StringType())

df_loaded = spark.read.format('delta').load('/mnt/data/encrypted_users.delta')
df_decrypted = df_loaded.withColumn('email', decrypt_udf(col('email_encrypted'))).drop('email_encrypted')
df_decrypted.show()

Verify the result

Confirm that the file/table contains only the encrypted column (for example inspect with df_encrypted.columns). Test decryption with inputs of different lengths and characters (UTF-8). Check common errors: invalid key (e.g., wrong size), truncated data (invalid base64) or corrupted nonce. Run performance tests: time encryption of 10k/100k/1M rows and record CPU/memory to size the cluster.

Conclusion

You now know how to encrypt and decrypt columns in Databricks using PySpark and AES-GCM. Recommended next steps: integrate with Azure Key Vault/Databricks Secrets or CMK for key management; use Unity Catalog for access control; and evaluate performance alternatives (encryption at ingestion or UDFs in Scala). Final tip: avoid placing keys in notebooks — use secrets to reduce risk and maintain auditability of operations.