Issue: YOLO Model Training Extremely Slow on M3 Max

<< After all that, am I missing something out? I asked myself.   >>

Yes, you are missing something.

Models ported from other system may need optimization for the exact hardware you are running on. That is often not so simple as throwing a switch.

If you just recompiled for a different processor, it would run, but you would call it very slow. Because it is not able to immediately take advantage of the array transform processors offered by GPU and neural engine, unless either the code (or the runtime environment) is modified to support using those co-processors.

"Apple Silicon MPS Training

With the support for Apple silicon chips integrated in the Ultralytics YOLO models, it's now possible to train your models on devices utilizing the powerful Metal Performance Shaders (MPS) framework. The MPS offers a high-performance way of executing computation and image processing tasks on Apple's custom silicon.

To enable training on Apple silicon chips, you should specify 'mps' as your device when initiating the training process. Below is an example 

https://docs.ultralytics.com/modes/train/#idle-gpu-training

the excellent speed of AI tasks on Apple-silicon Macs depends on not using only the CPU and GPU, but also the neural engine, a specialized FAST short floating point array transform processor.

https://en.wikipedia.org/wiki/Neural_Engine

Vast improvements are possible. Here is an quick search example of a user who made a slight change that dramatically improved YOLO performance:

https://pysource.com/2023/03/28/object-detection-with-yolo-v8-on-mac-m1/

I would contact the authors of Yolo and advise them of the problem. They need to start getting bug reports in from users and ask on their Apple Developer accounts why this is happening.

A couple things I would also keep in mind:

iCloud, Time Machine, and other auto-syncing software can affect performance. Unless you have a fiberoptic internet connection hard wired you will get lag through the asynchronous network communication happening in the background.

optimizing software do nothing of the sort. They are a placebo, because they dump temporary cache files for the system that can get corrupted. The system when running overnight and restarting the next morning will do its own cleanup better. Etrecheck can identify other memory resident programs you may want to remove if they are unwanted.

Don't let your hard drive get over 85% full. This tested benchmark has been found true of all computers, as a point of diminishing returns. Archive to external media and the Cloud any data you don't need instant access to.

DrSaadLa wrote:

Please advise on optimizing YOLO training for Apple Silicon. Are there specific configurations or known limitations with MPS implementation for object detection models?

No one here knows anything about YOLO. It is the developer's responsibility to ensure their product works on Apple platforms. Maybe they do that, maybe they don't.

Is the YOLO training optimized for Apple silicon or are you using Rosetta when using this?

That's why I point in the developer's route as everyone else does. If nothing I mention could remotely be the cause, then the developer is the only one who would have a clue why it is happening.

https://docs.ultralytics.com/

This is where you need to be looking up what Yolo is about. Not on the Apple website.