Modern Node Part 1 — Architecting and Scaffolding a TypeScript & Express API

Ryan Oillataguerre
ITNEXT
Published in
11 min readJul 28, 2021
The Vessel in New York

Introduction

Over the past year or so, I’ve been moving from mainly working in application code and lower-level business logic to working with architecture, infrastructure, and all of the different parts and pieces that contribute to architecting, creating, deploying, and keeping a production-level app running. And, as a self-taught software engineer, I know exactly what it’s like to search and search for a truly full-stack tutorial, from start to finish, that will actually work in production, and to come up dry.

Now that I’ve gone through this myself and have had to absorb tons of knowledge in a short amount of time, with this series I’d like to make the full stack example that I wish I had while learning the ropes. While moving from working on my own app, through a mid-sized company revamping their tech stack, to an enterprise company going public, and now to a startup starting from zero custom infrastructure, I’ve noticed trends and common denominators in every single app or platform I’ve worked on.

I’ve been working on refining the process of building a production-ready app while working on side projects through all of this, and have used the process I’ll describe in this article to create fully-functioning, multi-environment, applications that are deployed in the cloud using industry best practices.

Now, I’d like to accumulate these tidbits into something useful and pass them on to all of the brand new, or even experienced, software engineers who may be looking for best practices on their own apps, help with a project at work, or really what to do with any full stack application.

TL;DR: This will be a series on building a production-ready, fully deployed application with a Next.js front end and Node.js microservices for the API layer. We will go from zero code all the way to deploying to multiple environments on AWS.

What will we be building?

We will be building out a new version of the most recent category of apps to frustrate me with their overall terrible UX or lack of features: Personal finance apps! Whether this turns out to be a one-user app or not doesn’t matter to me, because it will do exactly what I want for tracking my own finances, and will illustrate well the process that goes into building any production-ready app.

For language and frameworks, we’re going to use a pattern I’ve seen that can be used for most applications in production, and which only requires newer engineers to learn a single language: a Node.js app-facing API and job running service, a Next.js React app built with Chakra UI and React Query, ElasticSearch for user-facing search, Redis for session management, and Docker for containerizing our production app along with allowing docker-compose to pull all of these together in our development environment.

We’ll then be deploying this stack to two separate services, both to fit my preferences as a developer, and to give you, the reader, an example of deploying to more than one specific cloud provider. We’ll be hosting everything except the Next.js app on Amazon Web Services, and hosting the Next.js app on Vercel’s platform, which has proved extremely helpful and easy to work with in the recent past (my favorite feature of theirs is the branch preview deploy, it makes it absolute cake to work with product managers and designers in a tight feedback loop). Overall, this is how I’d design an app that’s at a project stage, but that we want to keep ready to scale up indefinitely without a major pattern-shift. We’ll also be doing all of this AWS infrastructure management with Terraform, which I can also highly recommend. All companies I’ve worked at have either had it already, or I’ve adopted it into the tech stack because of its usefulness.

TL;DR: We’ll be building a personal finance application, syncing with users’ bank accounts, fully authenticated with sessions and Redis, using ElasticSearch from scratch, Docker for local development and agnostic deployment, and Terraform for infrastructure management

What will our application do?

After using all different kinds of personal finance and budgeting apps, I’ve gotten to the point where a spreadsheet is just easier for me. However, there are lots of things you can’t do with spreadsheets that are handy when they’re in an app, like syncing automatically with your bank accounts, automatically categorizing expenses, making useful visualizations to check up on your budget or savings goals, not throwing up when looking at the UI in front of you, etc.

Essentially, we will be building this for the core functionality:

  • A dashboard where you can find most things you’ll need regarding your spending, budget usage, savings goals, etc.
  • A budget page to actually build your own budget per-category
  • Searching for expenses
  • An expenses page to track what you’re spending money on
  • A net worth and savings goal/financial planning page
  • A page to add or manage your connected accounts
  • Detail pages on expenses, budget categories, etc.

What will we use for our tech stack?

Finally, what tools and libraries will we mainly be using for this project? Keep in mind, we aren’t going to be using these all in this article, but in little bits throughout the series.

  • Next.js
  • Node.js
  • Express
  • Redis
  • Elasticsearch
  • Docker
  • docker-compose
  • Kubernetes
  • Stripe
  • Plaid
  • AWS

Note: I’m not going to be going in depth with each of these tools throughout this series. This content will assume you have at least a surface level understanding of these tools! If not though, I’ll link to their documentation so you can get some context before we start using them.

Without further ado, let’s get to the first part of building any production application:

Architecture

What is the process of architecting and why do we need it?

Honestly, on personal projects I’ve skipped this stage because I wanted to just get building. Sometimes it’s worked out, but other times it’s bitten me in the ass. But, if you want to move fast and still be able to go far, you need to build with the distance in mind.

Architecting is so important for a production-level app like this, because it will protect you from abruptly crashing when traffic suddenly increases, will keep your costs low because you don’t have dangling infrastructure that you either forget about when spinning down, or costs you in extra charges to our hosting platform overlords (you’re welcome Jeff).

Specifically, what we’re going to use here is Terraform, my favorite and the most widely-used Infrastructure-as-Code framework I’ve seen.

What we’re building is pretty simple, but it’s a scalable pattern I’ve seen at companies of all sizes: Microservices on really any container orchestration system, but in this case we’re going to use Docker Compose and Kubernetes. We’ll be able to spin up easily locally and scale up relatively indefinitely, with minor changes to infrastructure if they’re needed.

Vercel does a fantastic job of autoscaling in itself, but if we wanted to stick with all AWS and put the Next.js app in another container, we would also probably be fine.

So, what will this app look like?

At a high level, it will look like this diagram below.

App Architecture

We’ll have two services at the API level: a Users Service (to act as a sort of monolith, just for simplicity’s sake and because it can be factored out into services relatively easily later), and a Math Service. In this case, we’re going to want a separate process for our more blocking functionality (CPU-intensive tasks like running complex runtime calculations or jobs), since Node is single-threaded.

We’re going to go with a PostgreSQL database and Objection.js as an ORM, since Objection.js has proven FANTASTIC to work with in the past. It’s light enough to allow raw SQL if you need to optimize or can’t use the API for a query, but I’ve never had to use raw SQL in the past. The way we can manage modeling, migrations, and the overall flexibility of it is perfect for our use case.

Ok, let’s get started.

Scaffolding

We’re going to just get all of the directories up at the beginning, for organization’s sake since it’s a CLI-focused process for all of them.

Let’s get up our app, which is the Next.js frontend, with create-next-app, and our backend services up with yarn init

> yarn create next-app app --ts
> mkdir services && cd services
> mkdir users-service && mkdir math-service
> cd users-service
> yarn init -y
> cd ../math-service
> yarn init -y

You should end up with a file structure like this one:

I’m using the homebrew library's tree

Let’s go through the same process for both of our services, while setting up the base Express server. I’ll only go through it for one (the Users Service), but go ahead and create the same structure in both.

yarn add -D typescript ts-node @types/node @types/express nodemon tslint dotenvyarn add expressmkdir srctouch .env .env.sample tslint.json tsconfig.json nodemon.json .gitignore .editorconfig src/server.ts src/index.ts

Let’s get the boring stuff out of the way: config files that are really up to personal preference as far as the actual configuration goes. I’ll bootstrap these with some common configs that I like to use, but not really go into depth about any specific one. Feel free to tweak them as you see fit.

Namely, I mean .gitignore,

node_modules/
build/
lib/
docs/
.DS_Store
coverage
*.log
.env

tslint.json,

{
"extends": "tslint:recommended",
"rules": {
"max-line-length": {
"options": [120]
},
"new-parens": true,
"no-arg": true,
"no-bitwise": true,
"no-conditional-assignment": true,
"no-consecutive-blank-lines": false,
"no-console": {
"severity": "warning",
"options": ["debug", "info", "log", "time", "timeEnd", "trace"]
}
},
"jsRules": {
"max-line-length": {
"options": [120]
}
}
}

.editorconfig (for standardizing autoformatting, much like Prettier and a .prettierrc but more agnostic)

root = true[*]
charset = utf-8
end_of_line = lf
indent_size = 2
indent_style = tab
insert_final_newline = true
trim_trailing_whitespace = true
[*.md]
max_line_length = 0
trim_trailing_whitespace = true

And insert the following for tsconfig.json. It may seem excessive, but remember, we are going for a production-level app here. No skimping on the small stuff.

{
"compilerOptions": {
"target": "es6",
"module": "commonjs",
"moduleResolution": "node",
"declaration": true,
"strict": true,
"noImplicitAny": true /* Raise error on expressions and declarations with an implied 'any' type. */,
"strictNullChecks": true /* Enable strict null checks. */,
"strictFunctionTypes": true /* Enable strict checking of function types. */,
"noUnusedLocals": true /* Report errors on unused locals. */,
"noUnusedParameters": true /* Report errors on unused parameters. */,
"noImplicitReturns": true /* Report error when not all code paths in function return a value. */,
"noFallthroughCasesInSwitch": true /* Report errors for fallthrough cases in switch statement. */,
"importHelpers": true,
"skipLibCheck": true,
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"experimentalDecorators": true,
"sourceMap": true,
"outDir": "./dist/tsc/",
"types": [
"node",
"jest"
],
"lib": [
"ES6",
"DOM"
]
},
"include": [
"src/**/*.ts"
],
"exclude": [
"node_modules",
"**/*.test.ts"
]
}

As well as a nodemon configuration file, nodemon.json. I've used this configuration in the past, and it has proved to be pretty useful.

{
"restartable": "rs",
"ignore": [".git", "node_modules/", "build/", "coverage/"],
"watch": ["src/"],
"execMap": {
"ts": "node -r ts-node/register"
},
"env": {
"NODE_ENV": "development"
},
"ext": "js,json,ts"
}

Let’s also add scripts to our package.json. After adding the basic dependencies and these scripts, we should have this:

{
"name": "users-service",
"version": "1.0.0",
"main": "index.ts",
"repository": "<https://github.com/ryanoillataguerre/pfapp/tree/master/services/users-service>",
"author": "Ryan Oillataguerre",
"license": "MIT",
"scripts": {
"start": "node build/src/index.ts",
"build": "tsc",
"dev": "nodemon --config nodemon.json src/index.ts"
},
"devDependencies": {
"@types/express": "^4.17.13",
"@types/node": "^16.0.0",
"dotenv": "^10.0.0",
"nodemon": "^2.0.7",
"ts-node": "^10.0.0",
"typescript": "^4.3.5",
"tslint": "^6.1.3"
},
"dependencies": {
"express": "^4.17.1"
}
}

build and start will be helpful once we want to distribute our apps to a cloud provider, and dev will allow us to use nodemon effectively and keep our apps hot reloading in development.

Now, let’s get into the actual code. create our server at src/server.ts and src/index.ts

index.ts

import dotenv from 'dotenv';
import server from './server';
dotenv.config();
const port = process.env.PORT || 8080;
server().listen(port);

I like this pattern because it marks a clear entry point for people reading the code, and doesn’t immediately overwhelm them with all of the middleware and configuration that will eventually go into this server.ts.

server.ts

import express, { Request, Response } from 'express';
import http from 'http';
const app = () => {
const app = express();
app.use((_: Request, res: Response, next) => {
res.header(
'Access-Control-Allow-Headers',
'Origin, X-Requested-With, Content-Type, Accept, Credentials, Set-Cookie',
);
res.header('Access-Control-Allow-Credentials', 'true');
res.header(
'Access-Control-Allow-Headers',
'Content-Type, Accept, Access-Control-Allow-Credentials, Cross-Origin',
);
res.header('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE');
next();
});
app.use(express.json()); // Routes
app.get('/health', (_, res) => res.status(200).send({ success: true }));
// All non-specified routes return 404
app.get('*', (_, res) => res.status(404).send('Not Found'));
const server = http.createServer(app); server.on('listening', () => {
console.info(`Users service listening on port ${process.env.PORT}...`);
});
return server;
};
export default app;

Ok, quite a bit to unpack here.

The headers section above where we’re setting them all on the response is going to help us not have to deal with so many issues later when sending session cookies back and forth between the server and client.

We’re using the express.json() API to replace what used to be bodyParser() for parsing JSON POST requests. Then, we’re declaring the routes that we want the Express app to use. This will include our custom module-based routes soon, which we’ll see how to build in the next article in this series.

We then create an event listener for our http server and return it, so that we can listen on it in index.js .

Now, let’s create ourselves a .env file to be read by the dotenv.config() command you see above. This will feed into our environment for this service specifically.

PORT=8080

And finally, let’s start up our server:

Up and running! 🎉🎉🎉

If something isn’t working right, it may be a missing file or some mismatched copy paste. Head over to the Github repo at this chapter’s branch to check out the source code!

In this article, we just took an app from zero code to a functioning Node service, which will be built upon in the upcoming chapters in this series. We’ll be adding routes, services, some utils, and more in the next article.

It will take some time for me to get the next article written out after packing all its code changes into a chapter, so if you want to stay updated on when I do post it, follow me on Medium here, or on Twitter here!

P.S. This is my first article, so if you liked it I’d really appreciate a clap, and if you have any favorite tools mentioned here, general comments, roasts, or suggestions, please do comment below!

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Published in ITNEXT

ITNEXT is a platform for IT developers & software engineers to share knowledge, connect, collaborate, learn and experience next-gen technologies.

Written by Ryan Oillataguerre

Senior Full Stack Software Engineer, most recently @ Subject.

Responses (6)

What are your thoughts?