SFTPFailure : Failure when opening files on remote and when transferring

I am getting SFTPFailure error when using sftp.open() (used to do to checksums) and sftp.get(). I simply use this to transfer data and sftp.get() worked before but is now throwing an error. I do not know what I can provide you as infos since there is nothing

Do you know what could be causing and if it would be possible to get a more exhaustive error?

here is a snapshot of the last debug message before the SFTPFailure during sftp.read():

2025-10-27 13:35:15,356 [DEBUG] [conn=0, chan=0] Sending fstat for handle 000003fa
2025-10-27 13:35:15,356 [DEBUG] [conn=0, chan=0] Received handle 000003fb
2025-10-27 13:35:15,356 [DEBUG] Calculating MD5 for remote file: file_244.fastq.gz
2025-10-27 13:35:15,356 [DEBUG] [conn=0, chan=0] Sending fstat for handle 000003fb
2025-10-27 13:35:15,357 [DEBUG] [conn=0, chan=0] Received handle 000003fc
2025-10-27 13:35:15,357 [DEBUG] Calculating MD5 for remote file: file.gz
2025-10-27 13:35:15,357 [DEBUG] [conn=0, chan=0] Sending fstat for handle 000003fc

best,

Loïc

and when using sftp.get() i am getting this log in debug mode:

2025-10-27 13:44:30,274 [DEBUG] [conn=0, chan=0] Received type: directory, size: 4096, uid: 1000, gid: 1000, permissions: 40775, atime: Mon Oct 27 09:08:16 2025, mtime: Wed Oct 22 12:05:50 2025
2025-10-27 13:44:30,275 [INFO] [conn=0, chan=0]   Starting copy of directory /home/dir2/
2025-10-27 13:44:30,275 [DEBUG] [conn=0, chan=0] Sending opendir for /home/dir2
 Failure to transfer dir2 to /home/local/dir2 : Failure

Can you provide some example code where you are seeing this problem, particularly related to the MD5 calculation?

In general, I would expect your application to either call sftp.open() followed by a loop calling sftp.read() or for your application to call sftp.get() to download the file and then call open() and read() on the resulting local file.

Also, the pathnames don't look right to me here. It looks like you are trying to copy starting from the path /home/dir2, without any kind of username on the remote file path. Also, does /home/local exist locally, and would the application have write access to that?

Finally, since you are trying to copy a directory, are you setting recurse to True?

Yes recurse is set to true and the directories paths were changed for privacy reasons.
this is the part doing the md5 checksums. The list of files is given and are processed with a create_task and gather(*tasks).

  
                try:
                    remote_file = await sftp.open(file_path, "rb")
                    logging.debug(f"Calculating MD5 for remote file: {file_path}")
                    md5_hash = hashlib.md5()
                    try:

                        byte_block = await remote_file.read()
                        while byte_block:
                            md5_hash.update(byte_block)
                            try:
                                byte_block = await remote_file.read()
                            except ValueError as e:
                                logging.error(f'Error while reading bytes blocks: {e}')
                    except ValueError as e:
                        logging.error(f'Error while reading bytes blocks:{e}')
                        raise
                    
                    return md5_hash.hexdigest(), Path(file_path).relative_to("/data/")
                except SFTPFailure as r:
                    logging.error(f'Error during (SFTP) md5 checksums on remote: {r}')

                    return None,None
                except Exception as e:
                    logging.error(f'Unexpected error during remote MD5 calculation for {file_path}: {e}')
                    return None, None

It seems that the SFTPFailure returning upon failure caused it to stop the process . I initially wanted to stop when it fails but I did not expect to have SFTPFailure most of the time I do the checksums. In fact, this only happens with certain folder and not all of them which is also very confusing about this. My main issue now is that the sftp.get() is not working since it's never showed any issues before.

Is it possible you are running into permission errors, where some of the directories aren't readable to the user you are logging in as on the SSH target host? That could cause SFTPFailure to come back from the SFTP server. The same would go for there being files within the directories which the user didn't have read access to. I would expect errors like that to return SFTPPermissionDenied, though, and not SFTPFailure. That said, I've seen a lot of SFTP servers that don't do a very good job returning errors, and the protocol itself doesn't really let you provide any kind of descriptive text in the error when the error occurs on the server side. You'd have to try and look at logs on the SFTP server to get more details.

Is there any chance you are passing a directory into your function to calculate MD5 checksums? Trying to call sftp.open() on a directory will work, but if you try to call read() on that, you'll get back an SFTPFailure.

As an aside, calling read() with no length around is going to load the entire file into memory at once on the client. Since you have a loop calling read(), you might want to add a byte count argument to that, so it fetches the data a block at a time, if you have any really large files. It's faster to let SFTP download all the data at once, but you need to make sure none of the files are larger than available memory on the client if you do that. A good compromise would be to limit it to reads which are something like a few megabytes in size. This is large enough to get the benefit of the SFTP parallel fetching, while minimizing memory use.

Also, I'm not sure the ValueError exception handlers are quite right. That would only ever be returned by hashlib and not by the SFTP read, so you might want to limit the scope of the try..except to the top-level try block, or something more specific ahead of that when you instantiate the md5 object.

Thank you for the details answer, I have explored the permission error route but did not find any differences between the files. Thanks for pointing out the value error exception handler, and I was indeed attempting to read dirs.

I am still getting the Failure during transfer with sftp.get() and it seems that if I get it once during the transfer then all operations are getting Failure too (e.g. sftp.listdir() ).

Something else that I noticed is that it usually fails when there are many files to read or transfer ,<1000 is ok but ~20000 is usually causing problems.

Are you currently opening only a single SFTP session on each SSH connection and then using that SFTP session for all your file transfers to/from a given target host, or are there cases where you are opening multiple SFTP sessions on a connection? If you are opening more than one session, keep in mind that most SSH servers limit the max sessions allowed on a connection. I think the default is around 100. You shouldn't need more than one SFTP session per connection even if you want to do multiple transfers in parallel, but I thought I'd mention this just in case.

Can you provide a debug trace showing one of the sftp.get() failures, and what messages came before it? Also, are you able to get debug logs off of the server? That might provide more useful information than just the SFTPFailure result.

One more thought: If you are opening multiple remote files at once on either the server or the client side, you could be running into a limit on max open file descriptors in the OS. On some systems, this can be as small at 256 if you don't adjust it, and I think may Linux distros limit you to something like 1024 by default. So, that would align with your "<1000 is ok". You can raise this limit with ulimit, but you'd probably need to do this on both the client side and the server side (meaning the SSHD process) for it to help.

Here is what I see before the failure of sftp.get()
await sftp.get(remote_path,local_path,recurse=True,block_size=32*1024*1024,max_requests=30,progress_handler=self.progress_handler)
it seems that get with recursive=True is more problematic, at least with the non recursive version I can limit the requests with a semaphore. I changed max_requests to 30 but it did not help.

/
2025-11-04 09:48:47,739 [INFO] [conn=0, chan=0] Starting SFTP get of /data//to /storage/tmp/ES/LOIC_TMP_NOBACKUP/my_dir
2025-11-04 09:48:47,740 [DEBUG] [conn=0, chan=0] Sending lstat for /data/my_dir
2025-11-04 09:48:47,744 [DEBUG] [conn=0, chan=0] Received type: directory, size: 4096, uid: 1000, gid: 1000, permissions: 40775, atime: Tue Nov  4 09:22:39 2025, mtime: Thu Oct 30 09:15:33 2025
2025-11-04 09:48:47,746 [INFO] [conn=0, chan=0]   Starting copy of directory /data/my_dir to /storage/tmp/ES/LOIC_TMP_NOBACKUP/my_dir/my_dir
2025-11-04 09:48:47,749 [DEBUG] [conn=0, chan=0] Sending opendir for /data/my_dir
2025-11-04 09:50:48,957 [ERROR] Failure to transfer /data/my_dir to /storage/tmp/ES/LOIC_TMP_NOBACKUP/my_dir : Failure`

Is there any chance that there might be a "loop" in your directory structure due to a symlink? I'm not sure I have any protection against that when follow_symlinks is set (which it is by default). It's possible that's causing the sftp.get() to infinitely recurse, eventually exhausting all the file descriptors on the server.

Other than that, the code currently copies a single file at a time, with the only parallelism being when copying different ranges of the file (if the file is big enough). It would also keep open handles for all parent directories, but unless you have a very deep nested directory structure that generally shouldn't cause a problem. It even uses scandir() rather than listdir() to more efficiently handle directories with a very large number of files, avoiding the need to read the entire directory contents in memory.

You could try running with follow_symlinks=False to see if that makes any difference.

If that's not it, I still think you should look for ways in which the server might be exhausting all its available file descriptors. One way to do that would be to manually start the SSH daemon on the server from a shell with a higher ulimit on file descriptors. You can run sshd with the "-D" option to keep it in the foreground, and maybe add "-ddd" as well to crank up its logging. You'll need to temporarily shut off the daemon version of sshd first, or alternatively you could set up a custom config file for it to run on an alternate port and then temporarily change your client to match.

Thanks a lot for the suggestion , I tested it and it does not make any difference. I verified that the number of files matches the number files found on the remote server for that directory. After the failure of the transfer, the sftp.listdir() also failed with SFTPFailure, strangely. I am wondering what that suggests? Anyways I cannot really mess with the remote server too much since I am not admin. I verifed that the ulimit and it's 1048576 so it does not look like it's the problem. Could the file names be a problem (e.g FBD03944_7b2de060_f7a3cbd9_63.pod5 )? I really appreciate your help.

You would need to set the ulimit on both the client and the server for it to be effective, if that's what is causing the problem. If you don't have admin access on the server, it's possible that it could still be the source of the SFTP failures. However, that would only be the case if AsyncSSH was opening too many SFTP file handles at once, which I wouldn't expect it to do unless it was recursing that number of directories deep. The file transfers themselves should only open one file handle at a time, sequentially, no matter how many files are in the directory, plus one file handle for each directory level above the file being copied.

I don't see anything in the sample filename you listed which should be a problem.

Do you know what OS and SSH version is running on the target machine? The SSH version for both client & server should show up in the debug logs. Do you think you could set up a new server where you did have admin access to try and reproduce this issue, allowing you to look more closely at server logs and open file descriptors?

I did a test here with a highly nested directory structure, and I did get an error, but it was from the local machine and it was an ENAMETOOLONG error caused when trying to do an os.stat() on a filename longer than 1024 bytes. When trying to copy files with scp, I got a different error of "scp: Maximum directory depth exceeded: 64 levels", which seems to be a hard-coded limit. Neither of these seem the same as what you're seeing, though, and neither of these ran into a ulimit issue because it didn't get far enough to hit that.

Actually, is it possible that your directory structure and filename length is deep enough and long enough to make a path which totals more than 1024 bytes? I haven't been able to reproduce the SFTPFailure, but I'm seeing other things failing at that limit. For instance, using '**' in zsh I got back a "file name too long" error as zsh tried to expand the wildcard.

Do you know what OS and SSH version is running on the target machine? The SSH version for both client & server should show up in the debug logs. Do you think you could set up a new server where you did have admin access to try and reproduce this issue, allowing you to look more closely at server logs and open file descriptors?

The OS is Ubuntu , this is the infos I could get with sshd OpenSSH_8.2p1 Ubuntu-4ubuntu0.11, OpenSSL 1.1.1f 31 Mar 2020. The ulimit was the one of the server.

I did a test here with a highly nested directory structure, and I did get an error, but it was from the local machine and it was an ENAMETOOLONG error caused when trying to do an os.stat() on a filename longer than 1024 bytes. When trying to copy files with scp, I got a different error of "scp: Maximum directory depth exceeded: 64 levels", which seems to be a hard-coded limit. Neither of these seem the same as what you're seeing, though, and neither of these ran into a ulimit issue because it didn't get far enough to hit that.

I tried to expand the directory but I did not get any error so it does not look likely.

Actually, is it possible that your directory structure and filename length is deep enough and long enough to make a path which totals more than 1024 bytes? I haven't been able to reproduce the SFTPFailure, but I'm seeing other things failing at that limit. For instance, using '**' in zsh I got back a "file name too long" error as zsh tried to expand the wildcard.

that's one example path : /data/20251030_X5_AS230_JGJK/no_ghonf_iz/20251030_0911_X5_FBD03944_7b2de060/fastq_pass/barcode02/FBD03944_pass_barcode02_7b2de060_f7a3cbd9_0.fastq.gz

What is more surprising is that my colleagues tried sftp.get() and it worked but, with recurse=False.

That path is only around 150 characters, so that should be an issue. I also don't see any unusual characters in the name.

I'm starting to run out of things to suggest, but one thought I had is that you might want to try specifying progress_handler and error_handler functions and printing out what you get back from those. The progress handler will let you see exactly what transfers are happening as it recurses through the tree, and maybe we'll discover something that way. I'm also curious if the error you are seeing is actually caught by the error handler or if it ends up triggering a stack trace.

Speaking of stack traces, could you provide a bit more of a complete debug log, with the debug level set to 2? You can send it to me privately if you don't want to post it here. I'd be curious to see if anything unusual shows up in the calls being made just before the SFTPFailure is returned.

OpenSSH 8.2 is getting kind of old, so it's possible there are big fixes in newer versions which could help here, but I don't know of any specific bug that would explain what you're seeing.

Regarding ulimit, how did you set that for the sshd process on the server without admin access? For it to be effective, you'd have to have something like a wrapper script around the sshd process to raise the limit before running sshd, as setting it from an interactive shell would only impact executables launched from that shell.

I also wonder if you could build a test environment which replicates the problem. If you can, it might be easier to debug without worry about exposing any sensitive information.

What is more surprising is that my colleagues tried sftp.get() and it worked but, with recurse=False.

When you say it "worked", I'm guessing it only copied files at that single directory level, so it's no surprise that it would behave differently with recurse=False. This is where a detailed debug log could help, though, as it'll show just what the recursion is doing when something goes wrong.

No in fact it worked with recursive listing of a dir and then directly copying each file independently (sftp.get(recursive=False) ) to the remote folder.

Did you do your own recursion of the directory listings? That's not a built-in capability in AsyncSSH's SFTP.

At this point, I really need to see a detailed debug log of it failing (preferably at least level 2) to suggest anything further.

Yes he used his own implementation with find and I will let you know as soon as I have more debug logs. I will set asyncssh.set_sftp_log_level(2)

Navigation Menu

Repository navigation

SFTPFailure : Failure when opening files on remote and when transferring #781

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

SFTPFailure : Failure when opening files on remote and when transferring #781

Description

Activity

lborcard commented on Oct 27, 2025

ronf commented on Oct 27, 2025

lborcard commented on Oct 27, 2025

lborcard commented on Oct 27, 2025

ronf commented on Oct 28, 2025

lborcard commented on Nov 3, 2025

lborcard commented on Nov 3, 2025

ronf commented on Nov 4, 2025

ronf commented on Nov 4, 2025

lborcard commented on Nov 4, 2025

ronf commented on Nov 5, 2025

lborcard commented on Nov 6, 2025

ronf commented on Nov 7, 2025

ronf commented on Nov 7, 2025

lborcard commented on Nov 7, 2025

lborcard commented on Nov 7, 2025

ronf commented on Nov 8, 2025

ronf commented on Nov 8, 2025

lborcard commented on Nov 12, 2025

ronf commented on Nov 13, 2025

lborcard commented on Nov 13, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions