problem with import of XML Apple HealthKit Export Version: 12

OK. I finally upgraded to iOS 16. I saved a copy of my iOS 15 Health Data for comparison.

I usually try to give Apple the benefit of the doubt. But this case, Apple completely screwed this up. Not even close.

First of all, Apple is still using DTD files. That's surprising. Plus, it is embedded rather than referencing a URL. That's unusual. Most importantly, Apple scrambled the syntax so it won't ever work. Note to Apple, use a schema instead and do it correctly. No other way to say it.

Edit: in addition to the above, Apple flat-out did the DTD incorrectly. I fixed the scrambled syntax and was able to get "xmllint" happy, but it still failed validation. So I went ahead and fixed the DTD too.

Next, Apple scrambled the output of some elements, repeating the "startDate" attribute twice. The standard rule of thumb is to use a proper XML-processing library to generate XML for output. But those old XML libraries are cryptic and difficult to use. And sometimes, vendors who provide XML-processing libraries release buggy ones (that dig's for you, too, Apple 😄 ). So often, developers take shortcuts and just spit out text that they hope will be valid XML. Full disclosure, I do this myself. But if you roll your own, you've got to do it correctly. Apple failed at that.

There are two distinct fixes you will have to apply. I will write this out as a longer sequence of steps just to make things crystal clear.

1) Export your health data. I saved mine to my iCloud Drive. You'll definitely need a Mac to fix this.

2) Go into your iCloud Drive, or wherever you saved the file and expand the "export.zip" file. It will be expanded into a folder named "apple_health_export". For ease of description, I copied my file to "/tmp" and exported it there.

3) In the Terminal, navigate to that folder using "cd /tmp/apple_health_export". (Not including quotes)

4) Create a new filed called "patch.txt" with the following content:

--- export.xml	2022-09-18 15:17:09.000000000 -0400
+++ export-fixed.xml	2022-09-18 16:37:08.000000000 -0400
@@ -15,6 +15,7 @@
   HKCharacteristicTypeIdentifierBiologicalSex       CDATA #REQUIRED
   HKCharacteristicTypeIdentifierBloodType           CDATA #REQUIRED
   HKCharacteristicTypeIdentifierFitzpatrickSkinType CDATA #REQUIRED
+  HKCharacteristicTypeIdentifierCardioFitnessMedicationsUse CDATA #IMPLIED
 >
 <!ELEMENT Record ((MetadataEntry|HeartRateVariabilityMetadataList)*)>
 <!ATTLIST Record
@@ -39,7 +40,7 @@
   startDate     CDATA #REQUIRED
   endDate       CDATA #REQUIRED
 >
-<!ELEMENT Workout ((MetadataEntry|WorkoutEvent|WorkoutRoute)*)>
+<!ELEMENT Workout ((MetadataEntry|WorkoutEvent|WorkoutRoute|WorkoutStatistics)*)>
 <!ATTLIST Workout
   workoutActivityType   CDATA #REQUIRED
   duration              CDATA #IMPLIED
@@ -63,7 +64,7 @@
   duration             CDATA #IMPLIED
   durationUnit         CDATA #IMPLIED
 >
-<!ELEMENT WorkoutEvent EMPTY>
+<!ELEMENT WorkoutEvent (MetadataEntry?)>
 <!ATTLIST WorkoutEvent
   type                 CDATA #REQUIRED
   date                 CDATA #REQUIRED
@@ -79,6 +80,7 @@
   minimum              CDATA #IMPLIED
   maximum              CDATA #IMPLIED
   sum                  CDATA #IMPLIED
+  unit                 CDATA #IMPLIED
 >
 <!ELEMENT WorkoutRoute ((MetadataEntry|FileReference)*)>
 <!ATTLIST WorkoutRoute
@@ -153,6 +155,7 @@
   dateIssued       CDATA #REQUIRED
   expirationDate   CDATA #REQUIRED
   brand            CDATA #IMPLIED
+>
 <!ELEMENT RightEye EMPTY>
 <!ATTLIST RightEye
   sphere           CDATA #IMPLIED
@@ -203,13 +206,6 @@
   diameter         CDATA #IMPLIED
   diameterUnit     CDATA #IMPLIED
 >
-  device           CDATA #IMPLIED
-<!ELEMENT MetadataEntry EMPTY>
-<!ATTLIST MetadataEntry
-  key              CDATA #IMPLIED
-  value            CDATA #IMPLIED
->
->
 ]>
 <HealthData>
  <ExportDate/>

Ignore the colours. That is just something the forum software adds to a code block.

5) Run the following command to patch your XML export file: "patch < patch.txt". (Again, omit the quotes.) The output should look like this:

patching file export.xml
Hunk #6 succeeded at 206 with fuzz 2.

Don't worry about that hunk #6 warning. I had to manually hack the patch file since it originally included two lines of content, which would be different. Luckily, patch gracefully handles it.

All this command does is replace the DTD data with something correct.

No, you're not done yet! I just hit the 5000 character posting limit. Here are the rest of the instructions:

6) Now, fix those duplicate dates. I normally avoid tools like this, but it's quick and easy. Run the following command in the Terminal: "sed 's/startDate/endDate/2' export.xml > export-fixed.xml". As before, omit the double quotes. Just be careful with the inner quotes and make sure they don't get turned into smart quotes. This command processes every line and fixes the second occurrence of "startDate" with "endDate".

I tested this with the xmllint tool and it is happier now. It even validates if you include then "--valid" flag, like so: "xmllint --noout --valid export-fixed.xml". If you get no output, the XML is valid.

You should be able to import from the "export-fixed.xml" file at this point.

I don't know what the rest of your procedure is, so I don't know if you will encounter additional problems. There may be another exported XML file "export_cda.xml" that depends on a complicated hierarchy of schema documents, which are not included. It looks like it would be a royal pain to track those down.

urbanspark wrote:

Unfortunately I'm not a programmer so I don't know what I can do and what I can't do... I guess I'll have to wait until Apple will fix this problem :/

You can either learn about and fix the XML, or can log a bug report and wait for an update from Apple or from the app vendor that fixes this.

To log a report with Apple, include the URL to this thread, and also with your particular apps and XML errors, here: Product Feedback - Apple

.

bmk1984 wrote:

I'm new to this environment, can you tell me where this sort of thing should be logged (where they will notice)?

If you’re enrolled in the Apple Developer program, log a problem report that way:

… Bug Reporting - Apple Developer

If you’re not enrolled in the Developer program, log feedback this way:

… Product Feedback - Apple

Provide the xmllint command and output, and/or details of the specific XML formatting errors encountered, and iOS version.

I tried using export.xml in python and encountered a few errors:

• First two were missing or out-of-place characters in the data definition part part. These were fixable with simple edits.
• Third was a duplicate tag: startDate/endDate really has two startDate tags.

...ActiveEnergyBurned" startDate="2020-12-28 16:34:27 -0400" startDate="2020-12-28 16:54:59 -0400"...

There are too many to fix manually, so I'll wait for an Apple update..

You've gotta wonder how this sort of thing got by the QA crew? I'm analyzing the ECG data (which is in separate "funny CSV" files), and the errors mentioned suggests that no actual testing was done?

That is used for contact lens information.

Since the XML is invalid, you probably won't be able to use any XML tools on it. You could use grep, with something like this:

grep RightEye /Users/johngoldin/Downloads/apple_health_export/export.xml

And that should spit out a small section of the file. The error is most likely on the same line. If you paste that into a reply, we could tell you how to fix it.

Or grep -v {target} the file (-v to invert the match) and ditch the bad line, if it’s all on the same line.

Otherwise, you’re going to be temporarily patching the XML with some scripting, or tweak the DTD.

I don’t see an easy way to drop the problem syntax with xmllint, but there’s probably another XML tool that can. Or libxml2 or,ilk can read and process it from within an app.

Thanks, etresoft! I was able to import "export-fixed.xml" into R and to run without hitch a script I modified from https://taraskaduk.com/posts/2019-03-23-apple-health. (I do have "export_cda.xml" but don't use it.)

[Edited by Moderator]

guilford1639 wrote:

Wow, you have gone above and beyond the call of duty. Thanks. That's impressive work.

Thanks!

I hope Apple actually notices this.

I've posted it in one of the places where Apple is least likely to see it. If you want Apple to fix it, you'll have to file a bug report: https://feedbackassistant.apple.com

Hey @all, I think I have the same problem. Everytime when I want to open the export file I get these error messages:

This page contains the following errors:

error on line 156 at column 1: ATTLIST: no name for Attribute

error on line 206 at column 3: internal error: xmlParseInternalSubset: error detected in Markup declaration

Below is a rendering of the page up to the first error.

I don't have a Mac, only a windows pc, so is there any fix for me too?

You seem to have forgotten to include your attachment.

All of the fixes I described can be done on Windows. Windows has an excellent array of XML tools. Fixing the duplicate attribute will require some kind of advanced text editor, but you should have those too. Unfortunately, I’m not familiar with that platform so I can’t give you specifics.

There is no attachment, I just get this messages in a red box:

"This page contains the following errors:

error on line 156 at column 1: ATTLIST: no name for Attribute

error on line 206 at column 3: internal error: xmlParseInternalSubset: error detected in Markup declaration

Below is a rendering of the page up to the first error."

There is nothing more on the page. Just this red box with the text.

Unfortunately I'm not a programmer so I don't know what I can do and what I can't do... I guess I'll have to wait until Apple will fix this problem :/

guilford1639 wrote:

etresoft, I've created a blog post that provides enough information to execute your fixes (and links to this discussion thread). If you have any objection to this, let me know and I'll delete it.

Thanks! No objection, but I would request a couple of edits. You spelled it "entresoft" a couple of times.

Thanks etresoft!

For my export today on iOS 16.1 it was enough to apply your patch. There weren't any duplicate `startDate`s and the patch didn't throw any warnings either. The resulting XML is still not valid but I could process it the same way I did on iOS 15. Maybe Apple changed fixed a few things in iOS 16.1?