# Spark History Server The Spark History Server lets you inspect the execution history of your Spark jobs directly from the Data Processing Engine, without setting up any external tooling. It applies to **PySpark-based actions** ([Load PySpark](/en/product/dpe/actions/load-pyspark/index.md), [Aggregate PySpark](/en/product/dpe/actions/aggregate-pyspark/index.md), [Custom PySpark](/en/product/dpe/actions/custom-pyspark/index.md)) and the **workflows** that run them. It does not cover [notebooks](/en/product/dpe/notebooks/index.md). Once enabled, every PySpark job records its event logs. You start the server when you need to read them, browse your recent runs, open each run's Spark UI (live or completed), analyze it with DataFlint, and apply retention rules to keep storage under control. ![spark-history-server-overview](picts/spark-history-server-overview.png) ?> The server is **billed per [DPU](/en/product/dpe/jobs/resources.md) for the time it is running**. It runs only on demand and shuts down automatically after a period of inactivity, so you don't pay for idle time. * [Start and stop the server](#start-and-stop-the-server) * [Overview](#overview) * [Statistics](#statistics) * [Recent applications](#recent-applications) * [Follow running jobs](#follow-running-jobs) * [Preferences](#preferences) * [Event logs](#event-logs) * [Live Spark UI](#live-spark-ui) * [Server resources and auto-stop](#server-resources-and-auto-stop) * [Retention](#retention) --- ## Start and stop the server The History Server runs only when you need it. Use the **Start** button at the top right of the page to launch it; its status (*Stopped* / *Running*), uptime, and time before auto-stop are shown next to it. The server stops on its own after the inactivity period set in the [Preferences](#server-resources-and-auto-stop), so it never stays up longer than needed. You can also stop it manually at any time. --- ## Overview The **Overview** tab summarizes your environment and lists your recent Spark runs. ### Statistics The Statistics panel gives you a snapshot of your environment: * **Applications** — total number of applications and how many are currently in progress. * **Storage** — total size of stored event logs and the amount pending cleanup. * **Activity** — time of the last run and the average Spark run time. ### Recent applications The Recent applications table lists your most recent Spark runs with their **action or workflow** name, **job number**, **Spark app ID**, **status**, **end of run**, **Spark run time**, and **size**. From each row you get one-click access to: * **Spark UI** — the standard Spark interface to explore stages, tasks, and execution details. * **DataFlint** — an enhanced analysis view with performance insights and optimization tips, powered by the open-source [DataFlint](https://github.com/dataflint/spark) project. Opening **Spark UI** gives you the full standard Spark interface for the run, where you can explore jobs, stages, and tasks: ![spark-history-server-spark-ui](picts/spark-history-server-spark-ui.png) **DataFlint** layers performance insights and optimization tips on top of the same run: ![spark-history-server-dataflint](picts/spark-history-server-dataflint.png) ?> You can also reach a specific run from the [Jobs](/en/product/dpe/jobs/index.md) page: select a Spark job (a PySpark action or workflow) to open it in the Spark History Server. --- ## Follow running jobs Running jobs appear at the top of the table, marked as live. As soon as a job's interface is ready, you can open its live Spark UI and DataFlint view without waiting for the run to finish. ?> Live interfaces are only available for running jobs when [**Live Spark UI**](#live-spark-ui) is enabled in the Preferences. --- ## Preferences The **Preferences** tab controls what is captured, the size of the server, when it stops, and how long logs are kept. ![spark-history-server-preferences](picts/spark-history-server-preferences.png) ### Event logs **Enabled by default.** Records the detailed activity of every PySpark job for analysis in the Spark History Server, including the performance insights and optimization tips powered by [DataFlint](https://github.com/dataflint/spark). When disabled, the analysis is not available for the next jobs. The history of existing runs is kept. ?> Event logs are stored in your project's **DPE S3 bucket**. The [Retention](#retention) rules below control how much of that storage they use. ### Live Spark UI **Disabled by default.** Exposes the live Spark UI of PySpark jobs so you can watch them run without waiting for the event logs. The setting applies to the next runs. ### Server resources and auto-stop * **Server resources** — the compute power allocated to the server, from **1 to 4 DPU**. The change is applied at the next start of the server. * **Auto-stop** — automatically stops the server after the selected period of inactivity. Options: **Never**, **After 2 hours**, or **After 8 hours**. The default is **24 hours**. ### Retention Event logs are cleaned up automatically every day. An application is removed once it exceeds any of the enabled limits below; the most recent applications are always kept. Set a limit to **0** (or *Unlimited*) to disable it. Use **Clean up now** to run the cleanup immediately. * **Retention (days)** — how long applications are kept. Options: **7** (default), **30**, **90**, **365**, or *Unlimited*. * **Maximum applications kept** — Options: **100** (default), **10**, **50**, **100**, **500**, or *Unlimited*. * **Maximum total size** — Options: **1**, **5**, **10**, or **50 GB**, or *Unlimited*. --- ### Need help? 🆘 > At any step, you can create a ticket to raise an incident or if you need support at the [OVHcloud Help Centre](https://help.ovhcloud.com/csm/fr-home?id=csm_index). Additionally, you can ask for support by reaching out to us on the Data Platform Channel within the [Discord Server](https://discord.com/channels/850031577277792286/1163465539981672559). There is a step-by-step guide in the [support](/en/support/index.md) section.