This library is used inside CodeEngine to provide multi-threaded concurrency. It exports a WorkerPool class, which manages worker threads and uses them to process files.
NOTE: This is an internal library that is only intended to be used by CodeEngine. Using it outside of CodeEngine is discouraged.
WorkerPool class
This class creates worker threads, manages their lifecycle, and transfers CodeEngine files back and forth between threads for processing.
import WorkerPool from "@code-engine/workers";// Create a new WorkerPool instancelet pool = new WorkerPool(engine);try { // Import a FileProcessor plugin in all workers let processFile = await pool.importFileProcessor("./my-file-processor.js"); // Process a file on one of the workers await processFile(myFile, run);}finally { // Safely dispose the pool and threads await pool.dispose();}
import WorkerPool from "@code-engine/workers";import CodeEngine from "code-engine";let engine = new CodeEngine();let pool = new WorkerPool(engine);
WorkerPool.size
Read-only property that returns the number of worker threads in the pool. After the dispose() method is called, this property will always return zero.
import WorkerPool from "@code-engine/workers";import CodeEngine from "code-engine";let engine = new CodeEngine({ concurrency: 4 });let pool = new WorkerPool(engine);console.log(pool.size); // 4await pool.dispose();console.log(pool.size); // 0
WorkerPool.isDisposed
Indicates whether the dispose() method has been called. Once disposed, the WorkerPool instance is no longer usable.
import WorkerPool from "@code-engine/workers";let pool = new WorkerPool(engine);console.log(engine.isDisposed); // falseawait engine.dispose();console.log(engine.isDisposed); // true
moduleId: The module name or path. The module must export a FileProcessor function.
data: (optional) Data to pass to the module. This is only relevant if the module's default export is a function accepts this data and returns a FileProcessor function.
import WorkerPool from "@code-engine/workers";let pool = new WorkerPool(engine);// Import a FileProcessor plugin in all workerslet processFile = await pool.importFileProcessor("./my-file-processor.js");// Process a file on one of the workersawait processFile(myFile, run);
WorkerPool.importModule(moduleId, [data])
Imports a JavaScript module in all worker threads. The module export (if any) is ignored. This method is intended for loading polyfills, globals, hooks, and other modules with side-effects.
moduleId: The module name or path
data: (optional) Data to pass to the module. This is only relevant if the module's default export is a function that accepts this data.
import WorkerPool from "@code-engine/workers";let pool = new WorkerPool(engine);// Import a polyfill module in all worker threadsawait pool.importModule("@babel/polyfill");
WorkerPool.dispose()
Terminates the worker threads and releases all system resources that are held by a WorkerPool instance. Once dispose() is called, the WorkerPool instance is no longer usable.
import WorkerPool from "@code-engine/workers";let pool = new WorkerPool(engine);await pool.dispose();
"error" event
This event is fired whenever an unhandled error occurs in any of the worker threads. If you don't handle this event, then Node.js will automatically terminate the process.
NOTE: When an unhandled error occurs, the WorkerPool instance and/or its worker threads may be left in an invalid or unusable state. For this reason, we recommend that you dispose the WorkerPool instance and stop using it.
import WorkerPool from "@code-engine/workers";let pool = new WorkerPool(engine);pool.on("error", (error) => { console.error("An unhandled error occurred:", error); pool.dispose();});
Clone this repo git clone https://github.com/CodeEngineOrg/code-engine-workers.git
Install dependencies npm install
Build the code npm run build
Run the tests npm test
License
@code-engine/workers is 100% free and open-source, under the MIT license. Use it however you want.
This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.
Big Thanks To
Thanks to these awesome companies for their support of Open Source developers ❤