Easily and efficiently process large sets of mongodb documents from a cursor
without loading the entire result set in memory at once.
mongo-cursor-processing is a module for iterating through results from a
mongodb cursor with a set level of concurrency.
You could just grab the result set and put it in an
async queue, but that would load every
object in memory at once, which might not be possible for large data sets. You
could add a small number of documents to a queue, wait until they are all done,
and then add more, but once the number of items in the queue is less than the
concurrency of the queue you are wasting some processing time waiting for those
last items to be done.
mongo-cursor-processing to the rescue!
API
Install with npm install mongo-cursor-processing. It exports a single
function that looks like this:
`function mp(cursor, task, [concurrency], cb)
cursor Object - a mongodb cursor, as returned from a mongodb query.
task Function - function(item, done) - a worker function that will be
called once with each item that comes off the cursor. Call the done
function when you are done with the item.
concurrency Number - number of worker functions to be running at the same time.
Defaults to 10 if unspecified.
cb Function - function(err) that will be called when all items from the
cursor have been processed, or any of them were processed with an error.
Example
var mp = require('mongo-cursor-processing')var mongo = require('mongoskin')var db = mongo.db('mongodb://localhost:27017/my_db', {w:1})// cursors get returned from queriesvar cursor = db.collection('people').find({name: 'John'})function workOnPerson(person, cb) { // do some async work with the person cb()}mp(personCursor, workOnPerson, function(err) { if (err) { console.error('on noes, an error', err) process.exit(1) } console.log('we did it team') db.close()})