Photo by Hendrik Cornelissen on Unsplash

Streaming SQL in Node.js

Sometimes you need a way to look at every row in a database table, and perform some operation on it.

SQL databases are, on the whole, great at handling massive amounts of data. I don’t think it’s an exaggeration to say that most of the companies I’ve seen who use Hadoop, would be better off just adding an index or some RAM to their Postgres/MySQL server. You can do amazing filtering and aggregation within SQL. Sometimes these powerful operations don’t support the thing you need, sometimes you’ll want to process every single row in node.js

Async Iterables

Imagine the following scenario:

• We have a database containing a large number of IDs of Tweets from Twitter. At least 100s of thousands.
• We want request the number of likes each tweet has from Twitter
• We’re processing this as a background job, and don’t want to use too much memory.

We’re going to want to stream the results of our query to get the rows out of SQL. This example works equally well, regardless of whether you’re using @databases/pg for Postgres, @databases/mysql for MySQL or @databases/sqlite for SQLite:

// replace this with your db of choice
import connect, {sql} from '@databases/pg';
const db = connect();