H2H is head-to-head which basically means each team competes against another every week...say there are 9 categories you are scored in: Points, Rebounds, Assists, FG%, FT%, TO, Steals, Blocks, 3 pointers...if I out score you in 5 of the 9 categories I get 5 wins and 4 losses for the week and that is added to your record...So every team will have like a win and loss record. You can also do it where, say if I beat you 5-4 I get ONE win and you get ONE loss, if I beat you 9-0 I still only get ONE win and you still only get ONE loss...but I don't know if people enjoy it, I've never done it that way.
Roto is more season-long. Basically, you accumulate stats on the season for each of the same 9 categories. If there are 10 teams in the league and you are the highest scorer for the "points" category you get 10, the next highest gets 9, 3rd highest gets 8 and so on until the lowest scorer in the "points" category who gets 1. The same is done for all the categories and your points are added up. The person at the end of the season with the highest total accumulation of points throughout all the stats (10 teams, 9 categories the highest you can score is 90) wins.