How can I take the average of a column of data with units of time, where I have blank (zero) cells for times not yet recorded?

For example, I have column A with the date (ex: 8/2/23), column B with values of times of the day (ex: 10:30 AM), column C with values of times of the same day (ex: 5:00 PM). In column D, which is the time difference between column B and column C using the formula (C1-B1), it displays 6h 30m using the Data Format "Duration". I have applied the formula of D1 to the entire D column, however, column A contains future dates and that means that in columns B and C, there are no values. This means that the formula applied to column D cells shows a "0" with no units.

I want to take the average of all of column D, and include the values of the future D cells to come so that it will automatically update my average when a value is applied. However, I cannot apply a formula to take the average of a column that includes values of "0".

Is anyone familiar with a formula that I can apply to all of column D that allows me not to average the "0" cells until they have a value? I have tried a custom Duration format that shows blank using "" when the value = 0, though the error on the average function still reads that the values must have the same units, or that the numbers cannnot be divided by zero.

I can select only the portion of cells of column D manually to be averaged, though I will have to re-edit the average formula manually every time I have a new value. Any help is greatly appreciated as I have spent hours trying to figure this out :(