When I do yideo recordings with latest Quicktime player (10.4 using High Sierra) with qualtity set to max, the video quality is looking very good while recording. But when watching the recorded video it looks much worse, and thin lines are not good looking, as they appear with stepped lines. (don´t know the english word for this effect - happens also if you enlarge a digital photo more than 100%)
It normally helps if you provide more information about your specific workflow along with screen captured images that demonstrate the issue you are having. For instance, it would nice to know if the source content for the "New Movie Recording" is interlaced or progressive and at what resolution it. Interlaced video played back progressively tends to produce "jagged" lines/edges for objects in motion (combing artifacts) due to the time difference between the two fields which are combined to produce each display frame. This may or may not be a factor contributing to your issue. The other common problem, as you already pointed out, is "pixellation" caused by an over enlargement of individual pixels when rendered for playback at higher then encoded resolutions—e.g., playing a 640x360 pixel recording at full screen dimensions on an HD/UHD monitor. Such artifacts would be considered normal for this particular workflow unless you can deinterlace or decomb the source content before progressive display recording and/or record the source content at a higher resolution less than or equal the original resolution of the source content.
I am wondering what is here going wrong. When I check information for such recorded video, it tells me apple prores 422. But I thought that prores 4444 is the right codec for max quality recording?
Again, it is impossible to say whether anything is going wrong here without knowing more about your particular recording workflow. As to the Apple ProRes 422 codec, it is now included as the default intermediate "editing" codec (Replacing the older Apple Intermediate Codec) on all current MacOS installations. On the other hand, ProRes 4444 is a full-resolution, mastering-quality 4:4:4:4 RGBA codec that is part of the Apple "Pro" codec package and is not selectable when making "New Movie Recordings" using QTX v10.4 under High Sierra nor would it improve the rendering of combing or pixellation artifacts if it could be used for such recordings.
I been searching now for hours about that, but did not find any solution or idea, I don´t know if that is a codec problem, or normal. If I do same recording using Quicktime Pro, with quality set to native, I am getting best results, and no stepped lines.
As indicated above, the issue sounds to be "normal" for the application in use. As the name implies, QT7 Pro remains (for the most part) a more "prosumer" oriented app for most "classic" operations and resolutions not requiring changes in frame rate. Unfortunately, some of its capabilities are not playback compatible with the QTX structure imbed. (I.e., sometimes you simply have to chose between one or the other app based on the feature requirements of your workflow.) In short, if youre getting better results with QT7 Pro, then by all means continue to use it as long as you can.