Come faccio a raggruppare per mese e anno quando ho solo un campo datetime?

Devi raggruppare per estratti.

SELECT 
    SUM(amount)
FROM 
    transaction
GROUP BY 
    EXTRACT(MONTH FROM when),
    EXTRACT(YEAR FROM when)

E se hai bisogno di quelle colonne, allora

SELECT
    EXTRACT(MONTH FROM when) as month, 
    EXTRACT(YEAR FROM when) as year, 
    SUM(amount)
FROM 
    transaction
GROUP BY 
    month,
    year

Ovviamente puoi aggiungere ORDER BY e usa anche nomi brevi:

SELECT 
    EXTRACT(MONTH FROM when) as month, 
    EXTRACT(YEAR FROM when) as year, 
    SUM(amount)
FROM 
    transaction
GROUP BY 
    month, 
    year
ORDER BY 
    year DESC, 
    month DESC