If an hour is 1/24th of a day, then a day is necessarily 24 hours. But, if you start from this definition of a “second”:
The second […] is defined by taking the fixed numerical value of the caesium frequency, ΔνCs, the unperturbed ground-state hyperfine transition frequency of the caesium 133 atom, to be 9192631770 when expressed in the unit Hz, which is equal to s−1.
which is independent of planetary bodies, then you could possibly adjust the number of seconds per minute and hours for different worlds and keep the system in place.
Tolkien took his cues from the Germanic\Scandinavian mythos of the Nine Worlds (most commonly called Norse Mythology now), two of which were Alfheim and Svartalfheim. The Alfar (Elves) and Svartalfhar (Dark Elves) were man-height if not a little taller.
Turkey was also on the menu – goose isn’t as locked in to British Christmas as turkey is to Thanksgiving.