Best way to copy files to server via UNIX?

Hi there!


I need to write a script for my clients to copy files to a server. What is the best method to copy a lot files to a server? I want this all handled through a UNIX tunnel.


Thank you!

Support all Mac machines, Mac OS X (10.5.7)

Posted on Aug 5, 2011 11:27 AM

Reply
18 replies

Aug 8, 2011 8:30 AM in response to spraguga

I don't see any way to avoid having to remote into each machine to setup some kind of transfer script. If you can ssh into the machines, you can push a public key and then pull the folder from the client to the server. That seems like it would be a very handy ability to have. Then you can script anything you want and execute it all remotely, in a script.

Aug 8, 2011 8:43 AM in response to etresoft

Maybe I am not setting up the auth keys properly and making more work for myself. It seems like I would need to remote into every single client machine, active ssh_keygen -t rsa, and then push the Client's public key to the server.


Also the server will not be pulling the folder recursively from the Client machines. The Client needs to run the script and push the folder to the server since the script/app will require user interaction.

Aug 8, 2011 9:01 AM in response to spraguga

spraguga wrote:


Maybe I am not setting up the auth keys properly and making more work for myself. It seems like I would need to remote into every single client machine, active ssh_keygen -t rsa, and then push the Client's public key to the server.

You can run that command locally, in a script, and then copy both the public and private keys to the clients. If you haven't already done so, adding your server's public key to each client's authorized_keys would be very handy too.


Also the server will not be pulling the folder recursively from the Client machines. The Client needs to run the script and push the folder to the server since the script/app will require user interaction.


Why? You can automate that and have it run whether anyone is logged in or not.

Aug 8, 2011 9:28 AM in response to etresoft

You can run that command locally, in a script, and then copy both the public and private keys to the clients. If you haven't already done so, adding your server's public key to each client's authorized_keys would be very handy too.

Don't I have to generate the ssh_key on every individual client and push it to the server's authorized_key file. I guess I have to read into this more.

Why? You can automate that and have it run whether anyone is logged in or not.

The script can't be automated, as I mentioned before it has to require user interaction.


Thank you very much for all of your help! 😉

Aug 8, 2011 10:04 AM in response to spraguga

spraguga wrote:


Don't I have to generate the ssh_key on every individual client and push it to the server's authorized_key file. I guess I have to read into this more.

Nope. This should do it:

ssh-keygen -q -t dsa -N "" -f /path/to/keys


When you script that, the /path/to/keys should be something that you can easily tie back to the client machines later. It will create both the public and private keys. Both must be copied to the clients' .ssh directory with the appropriately restrictive permissions. (I used dsa here instead of rsa.)


Normally I suggest using a passphrase on a Mac becaues the Keychain Access program runs ssh-agent and makes using a passphrase very easy. Then, even if someone unauthorized gets your private key, they can't use it to hack into your server. I would prefer you use some solution that didn't require interactive login, and having a blank passphrase would facilitate that - convenience at the cost of some security. It would be more secure to have the server connect to the clients and pull the data.


The script can't be automated, as I mentioned before it has to require user interaction.

You might want to go ahead and provide a passphrase then. The user would have to enter it only once and store it in their keychain. You could specify a different passphrase for each client if you wanted to.

Aug 8, 2011 11:16 AM in response to etresoft

Nope. This should do it:

ssh-keygen -q -t dsa -N "" -f /path/to/keys



The trouble I'm having is I have to auth to the server for the first SSH connection and add the client key to the server's authorized_key file like so.

cat ~/.ssh/id_dsa.pub | ssh user@host 'cat - >> ~/.ssh/authorized_keys'

You might want to go ahead and provide a passphrase then. The user would have to enter it only once and store it in their keychain. You could specify a different passphrase for each client if you wanted to.

Sorry I was not clear in my previous posts. I don't need the script/app to be user interactive for the password but it does for other reasons. Where the Client needs to push to the server rather than the server pulling the data.


Thanks agian!

Aug 8, 2011 11:34 AM in response to spraguga

spraguga wrote:


The trouble I'm having is I have to auth to the server for the first SSH connection and add the client key to the server's authorized_key file like so.

cat ~/.ssh/id_dsa.pub | ssh user@host 'cat - >> ~/.ssh/authorized_keys'

That command looks fine. What problems are you having? I suggested running ssh-agent on your local machine, copying the public keys to the server in a single batch, and then copying each public/private key pair to each client. As long as you're there, get every machine setup so you can ssh in too. Future such scripts should be much easier.


You will have to connect to each client machine. This first time, you could hack it up with some insecure, plain text options just to push the data out.


Sorry I was not clear in my previous posts. I don't need the script/app to be user interactive for the password but it does for other reasons. Where the Client needs to push to the server rather than the server pulling the data.


I understand. I was just pointing out that if you do have user interaction, you could take advantage of a passphrase. Most ssh-agent tutorials on the internet suggest using an empty passphrase because dealing with a passphrase is a hassle. It is, unless you have something like Keychain Access to make it easy.

Aug 8, 2011 11:48 AM in response to etresoft

etresoft wrote:

spraguga wrote:



The trouble I'm having is I have to auth to the server for the first SSH connection and add the client key to the server's authorized_key file like so.


cat ~/.ssh/id_dsa.pub | ssh user@host 'cat - >> ~/.ssh/authorized_keys'

That command looks fine. What problems are you having? I suggested running ssh-agent on your local machine, copying the public keys to the server in a single batch, and then copying each public/private key pair to each client. As long as you're there, get every machine setup so you can ssh in too. Future such scripts should be much easier.

Sorry about all this as I am very unfamiliar with ssh keys. I was trying to do this in the terminal maybe I should be doing it via ARD. Currently all Clients have ARD access and SSH access active in System Preferences/Sharing. Evey client machine is authenticated with the same admin account that I use with ARD.


So should I be doing the following:

Run this on every Client machines via ARD UNIX command to create the key:

ssh-keygen -t dsa -f ~/.ssh/id_dsa -N ''


And then run this on every Client machines via ARD UNIX command to update the servers authorized_key file:

cat ~/.ssh/id_dsa.pub | ssh user@host 'cat - >> ~/.ssh/authorized_keys'


Is that it, am I missing anything?


Also what if the machine is logged in by another Client. Then this will not work, correct? Can I set it up so anyone who is logged into any Client machine can ssh in to the server. Would I just change the id_dsa location to root or top level of the server and client startup volumes?


Thanks again for all of your help!!! 😮

Aug 8, 2011 12:33 PM in response to spraguga

spraguga wrote:


Sorry about all this as I am very unfamiliar with ssh keys. I was trying to do this in the terminal maybe I should be doing it via ARD. Currently all Clients have ARD access and SSH access active in System Preferences/Sharing. Evey client machine is authenticated with the same admin account that I use with ARD.


So should I be doing the following:

Run this on every Client machines via ARD UNIX command to create the key:

ssh-keygen -t dsa -f ~/.ssh/id_dsa -N ''


And then run this on every Client machines via ARD UNIX command to update the servers authorized_key file:

cat ~/.ssh/id_dsa.pub | ssh user@host 'cat - >> ~/.ssh/authorized_keys'


Is that it, am I missing anything?

I'm not familiar with ARD so I can't comment on that part. I can tell you that there is no need to create the keys on the clients. You can create all the keys for all the clients on a single machine - your machine. Instead of naming them "id_dsa", use something like "~/Documents/client_ssh_keys/client1". Write a script to go through and build them all in one swell foop. Then you can copy them all to the server's authorized_keys file.


You will have to copy the client1 and client1.pub files into .ssh/id_dsa and .ssh/id_dsa.pub on each client machine. It would also be a good idea to update the .ssh/known_hosts file on each client machine with the entry for the server. You can get this from your own known_hosts file. As long as you are on the client, copy your own public key into authorized_keys.


At this point, you can log in to every client machine and do so in a script, without a password. Each client can log in to the server without a password and run scripts.


Also what if the machine is logged in by another Client. Then this will not work, correct? Can I set it up so anyone who is logged into any Client machine can ssh in to the server. Would I just change the id_dsa location to root or top level of the server and client startup volumes?


SSH is an independent communication channel. No one needs to be logged in on any machine.


You can use the "-i" option on SSH to use a specific set of keys for communicating with a server. Sharing keys like that would be a significant security hole. You would definitely want to use a passphrase in that case.

Aug 8, 2011 1:33 PM in response to etresoft

I'm not familiar with ARD so I can't comment on that part. I can tell you that there is no need to create the keys on the clients. You can create all the keys for all the clients on a single machine - your machine. Instead of naming them "id_dsa", use something like "~/Documents/client_ssh_keys/client1". Write a script to go through and build them all in one swell foop. Then you can copy them all to the server's authorized_keys file.


You will have to copy the client1 and client1.pub files into .ssh/id_dsa and .ssh/id_dsa.pub on each client machine. It would also be a good idea to update the .ssh/known_hosts file on each client machine with the entry for the server. You can get this from your own known_hosts file. As long as you are on the client, copy your own public key into authorized_keys.


Wow, thank you, this is extremely helpful! I'm still a little confused about a few things. By one fell swoop, do you mean create mutiple files client1, 2, 3...etc? And then I cat all of the client(x) files into .ssh/id_dsa and client(x).pub files into .ssh/id_dsa.pub? Everything else I've read says to add the client(x).pub files to .ssh/authorized_keys, is this wrong?


Thank you!

Aug 8, 2011 3:16 PM in response to spraguga

No, I just mean to create the individual client files in one place and give them non-standard names like client(x) instead of id_dsa.pub. This is just for convenience sake. You will contcatenate all of the .pub files into the authorized_keys on the server. The individual client(x) and client(x).pub files need to be copied to all of the different client machines and renamed id_dsa and id_dsa.pub inside .ssh.

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

Best way to copy files to server via UNIX?

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.