---
title: "Spark Engineer - Air Market"
description: "You have a data job running across a cluster and it is slow, expensive, or falling over. This helps you write the job, find where the time goes, and set the cluster up to match the work. Reach for it…"
canonical: "https://airmarket.ai/app/spark-engineer"
---

[Air Market](/) / [Data & Analytics](/category/data) / Spark Engineer

Skill by [Jeffallan](/@jeffallan)

# Spark Engineer

For when the job runs but takes far too long

## What this does

You have a data job running across a cluster and it is slow, expensive, or falling over. This helps you write the job, find where the time goes, and set the cluster up to match the work. Reach for it when a query crawls, memory settings need tuning, or you are building a pipeline over large data.

### Top 3 things it does for you

- Write DataFrame transformations
- Work out why a Spark job is slow
- Tune shuffles and executor memory

## Information

**Spark Engineer at a glance**

|  |  |
| --- | --- |
| Kind | Skill |
| Publisher | [Jeffallan](/@jeffallan) |
| Category | [Data & Analytics](/category/data) |
| Works with | Agent Skills - an open standard supported by 40+ agents |
| Version | 1.1.0 |
| Size | 86 KB |
| Licence | MIT |
| Repository | [Jeffallan/claude-skills](https://github.com/Jeffallan/claude-skills/tree/5e8b6b8ff4007414f91f980771047b6136600c34/skills/spark-engineer) |
| GitHub stars | 11,146 (as of Nov 2026) |

## Security report

**Verified.** AIR scanned this exact version of Spark Engineer and flagged nothing. AIR's verdict: **safe**.

**What Spark Engineer's check was run against**

|  |  |
| --- | --- |
| Commit | 5e8b6b8ff4007414f91f980771047b6136600c34 |
| Content hash | 35f29f7a090602b3c75205cff2b69f282eee9622763bff7adbccc68f60f33d7a |
| Files read | [Browse the files at this exact commit](https://github.com/Jeffallan/claude-skills/blob/5e8b6b8ff4007414f91f980771047b6136600c34/skills/spark-engineer/SKILL.md) |

A scan is named by the exact content it read, not by a version number. A different version is a different artifact, and nothing here applies to a version AIR has not scanned.

[Read the full security report for Spark Engineer](/security/spark-engineer)

## About Spark Engineer

This is for writing Spark jobs, debugging performance problems, and configuring cluster settings for Apache Spark applications, distributed data processing pipelines and big data workloads.

It covers DataFrame transformations, Spark SQL query optimization, RDD pipelines, shuffle tuning, executor memory configuration, .parquet file processing, data partitioning and structured streaming analytics.

## What's inside

- SKILL.md - instructions
- references/partitioning-caching.md - reference
- references/performance-tuning.md - reference
- references/rdd-operations.md - reference
- references/spark-sql-dataframes.md - reference
- references/streaming-patterns.md - reference

## Questions

### Is Spark Engineer safe to install?

AIR scanned this exact version of Spark Engineer and flagged nothing. AIR's verdict: safe. The verdict is named by the content that was read, at commit 5e8b6b8ff400, so it cannot drift onto a version nobody scanned. The full report is at https://airmarket.ai/security/spark-engineer.

### How do I install Spark Engineer?

Open https://airmarket.ai/app/spark-engineer on Air Market and use the install action: Air Market hands Claude a prompt naming this exact skill, and Claude sets it up. Source, pinned to the exact commit this listing describes: https://github.com/Jeffallan/claude-skills/tree/5e8b6b8ff4007414f91f980771047b6136600c34/skills/spark-engineer. Install from that link, not the branch.

## Related on Air Market

- [More Data & Analytics listings](/category/data)
- [Everything by Jeffallan](/@jeffallan)

---

Canonical URL: <https://airmarket.ai/app/spark-engineer>