Come ottenere una media con orderBy Desc in Laravel 5

Puoi usare un withCount() modificato :

$books = Book::withCount(['ratings as average_rating' => function($query) {
    $query->select(DB::raw('coalesce(avg(rating),0)'));
}])->orderByDesc('average_rating')->get();