I am trying to iterate through this loop:
for doc in coll.find()
I get the following error at the 100,000th plus record.
File "build\bdist.win32\egg\pymongo\cursor.py", line 703, in next File "build\bdist.win32\egg\pymongo\cursor.py", line 679, in _refresh File "build\bdist.win32\egg\pymongo\cursor.py", line 628, in __send_message File "build\bdist.win32\egg\pymongo\helpers.py", line 95, in _unpack_response pymongo.errors.OperationFailure: cursor id '1236484850793' not valid at server
what does this error mean?
Maybe your cursor timed out on the server. To see if this is the problem, try to set timeout=False`:
for doc in coll.find(timeout=False)
If it was a timeout problem one possible solution is to set the
batch_size (s. other answers).
- Setting the
timeout=Falseis dangerous and should never be used, because the connection to the cursor can remain open for unlimited time, which will affect system performance. The docs specifically reference the need to manually close the cursor.
- Setting the
batch_sizeto a small number will work, but creates a big latency issue, because we need to access the DB more often than needed.
5M docs with a small batch will take hours to retrieve the same data that a default batch_size returns in several minutes.
In my solution it is mandatory to use sort on the cursor:
done = False skip = 0 while not done: cursor = coll.find() cursor.sort( indexed_parameter ) # recommended to use time or other sequential parameter. cursor.skip( skip ) try: for doc in cursor: skip += 1 do_something() done = True except pymongo.errors.OperationFailure, e: msg = e.message if not (msg.startswith("cursor id") and msg.endswith("not valid at server")): raise
timeout=False is a very bad practice. A better way to get rid of the cursor id timeout exception is to estimate how many documents your loop can process within 10 minutes, and come up with an conservative batch size. This way, the MongoDB client (in this case, PyMongo) will have to query the server once in a while whenever the documents in the previous batch were used up. This will keep the cursor active on the server, and you will still be covered by the 10-minute timeout protection.
Here is how you set batch size for a cursor:
for doc in coll.find().batch_size(30): do_time_consuming_things()
you should choose a low value of batch_size to fix the issue:
see the following answer
You can also force evaluation by using:
for doc in list(coll.find())