The amount of data transferred is for practical purposes unlimited. The maximum number of registers at a single Modbus station is 65535 of each type, and conceivably you could read them all. It would take some time, but it is possible.
The most common function codes for reading Bits and Words are FC01, FC02, FC03 and FC04. The responses to these function codes all include a Byte Count field that is a single byte in size. So the maximum number of data bytes that can be returned in a single read is 255. That limits you to a maximum of 2040 Input Status or 2040 Coils per message and 254 bytes (127 Registers) of Holding Registers or Input Registers. These numbers are per MESSAGE. You can have up to 65535 Registers or bits in a single Modbus device.
The most common function codes for writing multiple bits and registers are FC15 and FC16. These function codes have a Byte Count field in the query so the same limitations apply in the write messages.
A single Modbus RTU message can be a maximum of 256 bytes.
The reply message format is...
1st byte: Slave ID
2nd byte: function code
3rd byte: number of data bytes to follow
then the data bytes
last 2 bytes: CRC
The overhead is actually 5 bytes, 3 at the start of the message and 2 for the CRC at the end.
This leaves 251 bytes for data. Since the data is in 2 byte registers, this allows for 125 registers (in a 255 byte reply message).
For coils, since each coil is one bit and 8 coils can be sent in each byte. 251 bytes will allow for a maximum of 251 x 8 = 2008 coils in each reply.
But How much data can we transfer? If we send multiple requests for different blocks of data, we can transfer all 65536 registers (0-65535) for function code 03 and another 65536 read-only registers for function code 04, as well as 65536 coils each for function codes 01 and 02.
Do also remember that the answer may be highly 'vendor specific'.
For example, a power meter with only six registers of data (32-bit status, Kw and KwH) will only allow polling 6 words of data regardless of the Modbus.org documentation.
Other small devices with limited RAM may also requiring reading registers in smaller chucks - like 10 or 32 registers at a time just because they don't choose to allocate enough buffer space in RAM to handle larger messages.
