---
title: "Preference Optimization - Air Market"
description: "You have a fine-tuned model and a pile of feedback about which answers people preferred. This helps you use that feedback to adjust the model, including picking which of the several available methods…"
canonical: "https://airmarket.ai/app/preference-optimization"
---

[Air Market](/) / [Data & Analytics](/category/data) / Preference Optimization

Skill by [Seth Hobson](/@wshobson)

# Preference Optimization

Turn thumbs-up and thumbs-down feedback into a better model

## What this does

You have a fine-tuned model and a pile of feedback about which answers people preferred. This helps you use that feedback to adjust the model, including picking which of the several available methods to use. Reach for it when a tuning run needs settings or has gone sideways.

### Top 3 things it does for you

- Choose between DPO, ORPO, KTO and SimPO
- Set hyperparameters for a DPO run
- Work out why a preference run is misbehaving

## Information

**Preference Optimization at a glance**

|  |  |
| --- | --- |
| Kind | Skill |
| Publisher | [Seth Hobson](/@wshobson) |
| Category | [Data & Analytics](/category/data) |
| Works with | Agent Skills - an open standard supported by 40+ agents |
| Size | 14 KB |
| Licence | MIT |
| Repository | [wshobson/agents](https://github.com/wshobson/agents/tree/b6af3711058190e4b5c5274b9758498fe626ec5a/plugins/llm-finetuning/skills/preference-optimization) |
| GitHub stars | 39,077 (as of Nov 2026) |
| Installs | 2,974, counted by skills.sh (Nov 2026) |

## Security report

**Verified.** AIR scanned this exact version of Preference Optimization and flagged nothing. AIR's verdict: **safe**.

**What Preference Optimization's check was run against**

|  |  |
| --- | --- |
| Commit | b6af3711058190e4b5c5274b9758498fe626ec5a |
| Content hash | c52578d9d72e81d73cfbda48d51d037627e10cf587dbfe70b12ffc7309ff1be2 |
| Files read | [Browse the files at this exact commit](https://github.com/wshobson/agents/blob/b6af3711058190e4b5c5274b9758498fe626ec5a/plugins/llm-finetuning/skills/preference-optimization/SKILL.md) |

A scan is named by the exact content it read, not by a version number. A different version is a different artifact, and nothing here applies to a version AIR has not scanned.

[Read the full security report for Preference Optimization](/security/preference-optimization)

## About Preference Optimization

This is for aligning a model you have already fine-tuned using preference data - pairs of preferred and rejected answers, or plain thumbs-up and thumbs-down feedback.

It covers DPO, ORPO, KTO and SimPO, and helps with the decision of which one to reach for. It also covers the practical side of a DPO run: choosing hyperparameters and debugging when the run is not behaving.

## What's inside

- SKILL.md - instructions
- references/method-configs.md - reference

## Questions

### Is Preference Optimization safe to install?

AIR scanned this exact version of Preference Optimization and flagged nothing. AIR's verdict: safe. The verdict is named by the content that was read, at commit b6af37110581, so it cannot drift onto a version nobody scanned. The full report is at https://airmarket.ai/security/preference-optimization.

### How do I install Preference Optimization?

Open https://airmarket.ai/app/preference-optimization on Air Market and use the install action: Air Market hands Claude a prompt naming this exact skill, and Claude sets it up. Source, pinned to the exact commit this listing describes: https://github.com/wshobson/agents/tree/b6af3711058190e4b5c5274b9758498fe626ec5a/plugins/llm-finetuning/skills/preference-optimization. Install from that link, not the branch.

## Related on Air Market

- [More Data & Analytics listings](/category/data)
- [Everything by Seth Hobson](/@wshobson)

---

Canonical URL: <https://airmarket.ai/app/preference-optimization>