When I try to receive larger amounts of data it gets cut off and I have to press enter to get the rest of the data. At first I was able to increase it a little bit but it still won’t receive all of it. As you can see I have increased the buffer on the conn.recv() but it still doesn’t get all of the data. It cuts it off at a certain point. I have to press enter on my raw_input in order to receive the rest of the data. Is there anyway I can get all of the data at once? Here’s the code.
port = 7777 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind(('0.0.0.0', port)) sock.listen(1) print ("Listening on port: "+str(port)) while 1: conn, sock_addr = sock.accept() print "accepted connection from", sock_addr while 1: command = raw_input('shell> ') conn.send(command) data = conn.recv(8000) if not data: break print data, conn.close()
TCP/IP is a stream-based protocol, not a message-based protocol. There’s no guarantee that every
send() call by one peer results in a single
recv() call by the other peer receiving the exact data sent—it might receive the data piece-meal, split across multiple
recv() calls, due to packet fragmentation.
You need to define your own message-based protocol on top of TCP in order to differentiate message boundaries. Then, to read a message, you continue to call
recv() until you’ve read an entire message or an error occurs.
One simple way of sending a message is to prefix each message with its length. Then to read a message, you first read the length, then you read that many bytes. Here’s how you might do that:
def send_msg(sock, msg): # Prefix each message with a 4-byte length (network byte order) msg = struct.pack('>I', len(msg)) + msg sock.sendall(msg) def recv_msg(sock): # Read message length and unpack it into an integer raw_msglen = recvall(sock, 4) if not raw_msglen: return None msglen = struct.unpack('>I', raw_msglen) # Read the message data return recvall(sock, msglen) def recvall(sock, n): # Helper function to recv n bytes or return None if EOF is hit data = bytearray() while len(data) < n: packet = sock.recv(n - len(data)) if not packet: return None data.extend(packet) return data
Then you can use the
recv_msg functions to send and receive whole messages, and they won’t have any problems with packets being split or coalesced on the network level.
You can use it as:
data = recvall(sock)
def recvall(sock): BUFF_SIZE = 4096 # 4 KiB data = b'' while True: part = sock.recv(BUFF_SIZE) data += part if len(part) < BUFF_SIZE: # either 0 or end of data break return data
The accepted answer is fine but it will be really slow with big files -string is an immutable class this means more objects are created every time you use the
+ sign, using
list as a stack structure will be more efficient.
This should work better
while True: chunk = s.recv(10000) if not chunk: break fragments.append(chunk) print "".join(fragments)
Most of the answers describe some sort of
recvall() method. If your bottleneck when receiving data is creating the byte array in a
for loop, I benchmarked three approaches of allocating the received data in the
Byte string method:
arr = b'' while len(arr) < msg_len: arr += sock.recv(max_msg_size)
fragments =  while True: chunk = sock.recv(max_msg_size) if not chunk: break fragments.append(chunk) arr = b''.join(fragments)
arr = bytearray(msg_len) pos = 0 while pos < msg_len: arr[pos:pos+max_msg_size] = sock.recv(max_msg_size) pos += max_msg_size
You may need to call conn.recv() multiple times to receive all the data. Calling it a single time is not guaranteed to bring in all the data that was sent, due to the fact that TCP streams don’t maintain frame boundaries (i.e. they only work as a stream of raw bytes, not a structured stream of messages).
See this answer for another description of the issue.
Note that this means you need some way of knowing when you have received all of the data. If the sender will always send exactly 8000 bytes, you could count the number of bytes you have received so far and subtract that from 8000 to know how many are left to receive; if the data is variable-sized, there are various other methods that can be used, such as having the sender send a number-of-bytes header before sending the message, or if it’s ASCII text that is being sent you could look for a newline or NUL character.
Disclaimer: There are very rare cases in which you really need to do this. If possible use an existing application layer protocol or define your own eg. precede each message with a fixed length integer indicating the length of data that follows or terminate each message with a ‘n’ character. (Adam Rosenfield’s answer does a really good job at explaining that)
With that said, there is a way to read all of the data available on a socket. However, it is a bad idea to rely on this kind of communication as it introduces the risk of loosing data. Use this solution with extreme caution and only after reading the explanation below.
def recvall(sock): BUFF_SIZE = 4096 data = bytearray() while True: packet = sock.recv(BUFF_SIZE) if not packet: # Important!! break data.extend(packet) return data
if not packet: line is absolutely critical!
Many answers here suggested using a condition like
which is broken and will most likely cause you to close your connection prematurely and loose data. It wrongly assumes that one send on one end of a TCP socket corresponds to one receive of sent number of bytes on the other end. It does not. There is a very good chance that
if len(packet) < BUFF_SIZE:
sock.recv(BUFF_SIZE) will return a chunk smaller than
BUFF_SIZE even if there’s still data waiting to be received. There is a good explanation of the issue here and here.
By using the above solution you are still risking data loss if the other end of the connection is writing data slower than you are reading. You may just simply consume all data on your end and exit when more is on the way. There are ways around it that require the use of concurrent programming, but that’s another topic of its own.
A variation using a generator function (which I consider more pythonic):
def recvall(sock, buffer_size=4096): buf = sock.recv(buffer_size) while buf: yield buf if len(buf) < buffer_size: break buf = sock.recv(buffer_size) # ... with socket.create_connection((host, port)) as sock: sock.sendall(command) response = b''.join(recvall(sock))
You can do it using Serialization
from socket import * from json import dumps, loads def recvall(conn): data = "" while True: try: data = conn.recv(1024) return json.loads(data) except ValueError: continue def sendall(conn): conn.sendall(json.dumps(data))
NOTE: If you want to shara a file using code above you need to encode / decode it into base64