Hi 'w,
Numbers charts the data that you supply, not the data that you display.
The hours and minutes displayed in the first chart have been entered to record only the number of hours plus the number of minutes, so what you see is what the cell actually contains.
The average hours, though, are calculated values which may contain values expressing seconds and milliseconds of duration, formatted to display the duration to the nearest whole minute. You see the formatted value, but the chart gets the actual value in that cell.
In the example below, the chart is showing the values in the two blue filled cells in the table.
The upper cell contains a copy of the AVERAGE calculated in the bottom row of the Hours column and formatted to display only Hours and Minutes, charted by the first column.
The lower cell contains a copy of the value calculated in the cell directly below it—the same AVERAGE, rounded to the closest whole minute—charted by the second column.
Formulas below.

Formulas:
Both formulas use the same data in column E of the table, most of which is not visible in the image above.
For the chart column labeled Mon, Jan 2, 2016:
=AVERAGE(E)
For the chart column labeled Tue, Jan 3, 2017:
=DURATION(,,,ROUND(DUR2MINUTES(AVERAGE(E)),0))
Because ROUND works with numbers, but not with durations, it's necessary to convert the average duration to a number, using DUR2MINUTES, round that number to the nearest whole number, then convert the rounded number back to the duration represented by the rounded number.
Regards,
Barry