[C(++)] Strange problem with pthread

I have a command-line tool that executes a system command A, which outputs a series of files. As soon as the first file is finished (when the second file is being created), the main program starts to process the first file. This processing requires another external system command B. To be able to let the two system commands run simultaneously and monitor when an error has occured, the program uses (p)threads.

This program has been running perfectly in Tiger and Leopard up to 10.5.1. But now it behaves differently. When command B is executed with the "system()" call, it locks up until the thread with command A has finished. This makes the whole threading system useless because I could as well just execute the two commands sequentially.

I have stripped down the source code to the bare essentials. It can be downloaded at: http://preview.tinyurl.com/2zovrh
When executed (make sure to erase all "file*" files from previous runs first), you'll see that the main thread stops at "Doing something with file1", and doesn't start waiting for file3. It only continues when command A has finished even though the pthread_join() call has not yet been used. Why is this? I assume something has changed with pthread in 10.5.2. How do I fix this?

MacBook Pro C2D 2.33, Mac OS X (10.5.2)

Posted on Mar 8, 2008 7:41 AM

Reply
13 replies

Mar 8, 2008 5:00 PM in response to Alexander Thomas1

I have tested this program on every different flavor of every operating system that I can easily get my hands on, and — not surprisingly — it never behaves like on my machine. It would be interesting if other people would compile and run this program, specifically other people running 10.5.2 or (unlikely, I know) still using 10.5.0 or 10.5.1. You can create a "C++ Tool" in XCode, use your favorite basic Makefile, or use gcc directly if you're a real geek.
The correct output should contain 5 lines "Waiting for file file3...", the incorrect output only contains "Waiting for file2...".

Anyhow, I already submitted a bug report to Apple, because I can't think of any valid reason why this could be desired behavior.

Mar 9, 2008 7:40 PM in response to Alexander Thomas1

It looks like the program is behaving as designed. According to the man page on system():

The system() function hands the argument command to the command interpreter sh(1). The calling process waits for the shell to finish executing the command, ignoring SIGINT and SIGQUIT, and blocking SIGCHLD.


So, it seems that the first time you call system(), your process blocks SIGCHLD. the SIGCHLD that is the result of the second system() completion is delayed until the first system() finishes.

Mar 10, 2008 8:54 AM in response to etresoft

Yes, but as far as I know, the blocking of SIGCHILD should only apply to the thread in which that system() call occurred. Moreover, it seems unlikely that Leopard is the only operating system which implements this "correctly". Given the fact that the program behaves differently in every other operating system I've tried, it seems more plausible that Leopard's behaviour is incorrect.

Mar 10, 2008 9:39 AM in response to Alexander Thomas1

The man page for system() didn't say anything about threads. Which other operating systems did you try this with? Leopard is certified to be UNIX, so that is probably the reason for the change and the reason I think its behavior is correct. If you can compare its behavior to something like Solaris, you might have a case for a bug, but maybe not.

Threading and signals are fairly advanced concepts. Plus, pthreads is a fairly pitiful threading package. There are far better ways to accomplish your tasks than using those system() calls. If you want to make a case for a bug in MacOS X pthreads, you'll either have to take system() out of the equation or track down what, if any, specifications exist for interactions between child processes, signals, and threads.

Mar 10, 2008 1:38 PM in response to Alexander Thomas1

When command B is executed with the "system()" call, it locks up until the thread with command A has finished. This makes the whole threading system useless because I could as well just execute the two commands sequentially.


Indeed, if you have a look at [parallel|http://www.intel.com/technology/itj/2007/v11i4/3-development/4-challe nges.htm] programming you'll see it's worth the effort.

Mar 10, 2008 4:10 PM in response to Alexander Thomas1

Indeed it seems that you can have only one pending system() call per process.

Try replacing your system() calls with this function:


#include <unistd.h>
#include <stdlib.h>
#include <sys/wait.h>
int __system(const char * cmd)
{
pid_t pid;
int status;
if ((pid = fork()) == 0) {
execl("/bin/sh", "/bin/sh", "-c", cmd, NULL);
return -1;
}
else if (pid > 0) {
waitpid(pid, &status, 0);
return WEXITSTATUS(status);
}
else
return -1;
}


Also, it seems to me that


pthreadmutexdestroy(&mutexTest);


should be outside your for loop, just before returning from main().

Mar 11, 2008 9:05 AM in response to etresoft

etresoft wrote:
The man page for system() didn't say anything about threads. Which other operating systems did you try this with?


HP-UX 11.0
Tru64 UNIX V5.1
Red Hat Enterprise Linux AS release 4 (2.6.9 kernel)
CentOS Linux with a 2.6.18 kernel
Gentoo Linux with a 2.6.20 kernel
Fedora Core 8 with a 2.6.24.3 kernel

I'm pretty certain that if I would take one of the dusty Solaris boxes out of our closet and run this code on them, I would get the same result as on all those operating systems. But that's probably not worth the effort, considering that the two first OSs from the above list are certified UNIX unless I'm mistaken.

But you're right, the man page of system() says nothing about threads, so maybe it's up to the vendor to decide how it behaves. I guess I'll just use emal's "DIY-system()" instead and see what Apple says about the bug report.

etresoft wrote:
Plus, pthreads is a fairly pitiful threading package.


From http://www.kernelthread.com/mac/osx/programming.html:
+"The pthread library, /usr/lib/libpthread.dylib (actually a symlink to libSystem.dylib) provides POSIX threads. Carbon includes a threads package for cooperatively scheduled threads (Thread Manager) and another for preemptively scheduled threads (Multiprocessing Services). Cocoa uses the NSThread class, while Java uses java.lang.Thread. All these are built using pthreads."+
So practically all threading packages in OS X are built upon a fairly pitiful threading package? That's reassuring 😉

Mar 11, 2008 10:49 AM in response to Alexander Thomas1

Alexander Thomas1 wrote:
HP-UX 11.0
... I'm pretty certain that if I would take one of the dusty Solaris boxes out of our closet and run this code on them, I would get the same result as on all those operating systems. But that's probably not worth the effort, considering that the two first OSs from the above list are certified UNIX unless I'm mistaken.


Interesting... You're still running HP-UX but have Solaris in the closet. I would have expected it to be the other way around.

I think you should file a bug report on this. Whether or not it is part of the standard, it should behave the same as on different platforms.

But you're right, the man page of system() says nothing about threads, so maybe it's up to the vendor to decide how it behaves. I guess I'll just use emal's "DIY-system()" instead and see what Apple says about the bug report.


I don't know if I mentioned this before or not. It seems as if your code is just a demonstration for the problem you found. You aren't really doing system("ls") and using iostreams to check for the existence of a file are you? You seem to have a valid issue with the system() function call, but there are far, far better ways to accomplish what you are trying than with the system() call.

etresoft wrote:
Plus, pthreads is a fairly pitiful threading package.


From http://www.kernelthread.com/mac/osx/programming.html:
+"The pthread library, /usr/lib/libpthread.dylib (actually a symlink to libSystem.dylib) provides POSIX threads. Carbon includes a threads package for cooperatively scheduled threads (Thread Manager) and another for preemptively scheduled threads (Multiprocessing Services). Cocoa uses the NSThread class, while Java uses java.lang.Thread. All these are built using pthreads."+
So practically all threading packages in OS X are built upon a fairly pitiful threading package? That's reassuring 😉


It is not as bad as it sounds. Part of your problem stems from the fact that UNIX really doesn't like to do threads at all. UNIX is designed to fork processes. Windows, for example, has much better thread and low-level synchronization support than pthreads. It is just a different culture.

Also, MacOS X pthreads really isn't. MacOS X threads are built on Mach tasks. I bet there was quite a bit of work involved to get MacOS X pthreads to emulate the flakiness of traditional pthreads. pthreads is still a flaky API, but the underlying OS is solid.

Mar 11, 2008 12:21 PM in response to etresoft

etresoft wrote:
Interesting... You're still running HP-UX but have Solaris in the closet. I would have expected it to be the other way around.


I think the main reason why those HP-UX boxes are still running is either because some people still need tools that only run on them, or it would just take more effort to get rid of them than to let them sit in a corner 🙂

I think you should file a bug report on this. Whether or not it is part of the standard, it should behave the same as on different platforms.


I already filed a report. It's up to Apple to decide if it's really a bug...

I don't know if I mentioned this before or not. It seems as if your code is just a demonstration for the problem you found. You aren't really doing system("ls") and using iostreams to check for the existence of a file are you?


No, the code is a skeleton of the actual program. I could as well have used "echo hello" instead of "ls" to show that just any system call is blocked. The real program uses cdparanoia to rip audio CDs and then uses oggenc on the resulting WAV files, and finally vorbiscomment and vorbisgain. Yes, I could do this more efficiently and combine the source code from those 4 programs into one single program which would have optimal efficiency. But that's way too many effort for what it is.

Mar 11, 2008 1:36 PM in response to Alexander Thomas1

Alexander Thomas1 wrote:
No, the code is a skeleton of the actual program. I could as well have used "echo hello" instead of "ls" to show that just any system call is blocked. The real program uses cdparanoia to rip audio CDs and then uses oggenc on the resulting WAV files, and finally vorbiscomment and vorbisgain. Yes, I could do this more efficiently and combine the source code from those 4 programs into one single program which would have optimal efficiency. But that's way too many effort for what it is.


Sounds like a job for Perl or even sh.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

[C(++)] Strange problem with pthread

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.