Search for a command to run...
Embracing Change. Never settle.
Self-employed advisor for a holistic view on the whole software engineering process. Developing people, developing organizations, developing software. Still passionate coder.
No comments yet. Be the first to comment.
After more than 20 years with Java and Spring, and recently with Kotlin, I have decided to start a new chapter and write a fully-fledged web application in Typescript / Node.js / Nest.js
Tips and tricks for Linting & Co. that go beyond the default scaffold
Tips and tricks for Linting & Co. that go beyond the default scaffold
Keep your integration tests as close as possible to your application code, avoid mocking.
Let's start the year with another tech blog
After more than two decades of dedicated work in the tech industry, I recently made a significant shift in my career, stepping back as COO from Cloudflight, an incredible company building custom software with more than 1.000 of the best engineers you can imagine, and a bit later joining greeny+, a startup which builds aeroponic towers.
Almost everything is different here for me and that personal change also came along with a paradigm shift with regards to technology and software development. In this blog series, I want to take you a bit on my journey so far, sharing my technical experiences:
For more than 15 years, I led awesome teams at Cloudflight, where we leveraged the robustness of Spring, the elegance of Kotlin, and the reliability of the JVM to develop complex, scalable applications. Our processes were well-established, we had the best developers both on frontend and backend, and our portfolio was supported by a skilled ensemble of Requirements Engineers, UI/UX Designers, and Site Reliability Engineers. We operated on Kubernetes, ensuring high availability and seamless scalability for our products both on hyperscalers (Azure, AWS, GCP) and on-premise solutions like OpenShift.
While that is all still true for Cloudflight, I started again from scratch.
Arriving at greeny+, I was suddenly alone in that regard. We do have fantastic people for lots of other topics around product design, plant growth, marketing and sales, but I am (at least at the time of writing) the only IT guy. No one setting up server infrastructure, no one taking care of my deployment pipeline, no requirement engineers that build up and develop a clear backlog and talk to customers, no UX designers with all those so important skills to create the best possible user interfaces, no frontend developers with deep knowledge in HTML/SCSS/JS.
Just me.
And, as in most other startups as well: time and money are somehow limited.
Additionally, not to forget, there has not been too much practice time in software development in the last few years for me...
Still, when I joined greeny+ beginning of that year, I didn't start on empty ground. The team - also with some external help - had made a lot of valuable contributions already: HubSpot was being used as CRM and our new web page based on Shopify was about to go live. In between, we were rolling out UpPromote as affiliate solution.
I was trained for years to design tailor-made software systems that outperform standard solutions (but also cost a bunch of money upfront), so I had to change perspective and evaluate those existing tools, trying to get the maximum out of them.
So I looked through Shopify's backend and found out that it had adapted to the headless e-commerce hype which advocates to separate the presentation layer from its core logic. In Shopify, you don't have the flexibility like with Kontent.ai, Commercetools, Strapi or Medusa.js, but it's good enough to go for the start. We could customize the shop to our needs very quickly, integrating payment gateways thanks to Shopify Payments as well as interactive forms (like newsletter registration) with HubSpot, no blockers here.
HubSpot really positively surprised me when it comes to adaptability, it gives you an incredible amount of customization options, including a very mature and well-documented API and a built-in workflow engine.
Especially the latter raised my attention, it gives you a great mix of a Low/No-Code graphical editor along with the possibility to create custom code blocks. In the following screenshot you can see our workflow which connects new customers with the respective affiliate based on UTM parameters:
That works great already, that way we could connect HubSpot with UpPromote. Custom code inside HubSpot can be written in either Python or Node.js. I chose the latter, as JavaScript is much more familiar to a Java guy like myself.
And those few lines of JavaScript were actually the entrance to my new experience with Node.js.
Quite quickly it was clear, that our tooling landscape could not fulfill all of our needs. Just to give you a few examples:
The official HubSpot integration with Shopify has some quite serious limitations in terms of privacy, i.e. we would have not been able to reduce the visibility of incoming orders in Shopify to only the affected salespeople in HubSpot.
UpPromote cannot calculate commission fees fully as per our ruleset and also cannot generate invoices that fulfill our needs tax-wise.
UpPromote only gives limited possibilities to provide affiliates with personally branded sales kits. We used Spreadly for that in the past, but the process of creating such digital business cards was manual, costly, and error-prone. We needed a solution for scale here.
So we knew we needed something dedicated for us. Time for custom software, again.
As mentioned above, I did not have much time to bridge the gaps between Shopify, UpPromote and HubSpot, so my intuitive first choice was again my well-known JVM-stack with Spring Boot, Kotlin and Thymeleaf (I'm a lousy frontend developer, so I tried to avoid things like Angular, Vue.js or React).
But then some questions came up:
How to develop code? After working with Intellij IDEA for 10 years, I gave VS.code a chance end of December. Very lightweight, good support for Python and Javascript, but JVM/Gradle integration has some serious flaws. So again purchase an Intellij license?
How to connect to our three systems? There are REST APIs and partly OpenAPI specifications for HubSpot, Shopify and UpPromote, but no ready-to-use client libraries for the JVM. I could of course generate my own Kotlin clients just like we did in the past with other systems, but would again go my own way there.
Where to finally run the code? Kubernetes and all kinds of Hyperscalers would have been an overkill for what I needed (actually one web server and a relational database). Of course, there are a couple of hosters out there that offer lightweight JVM hosting without the complexity of K8S (like Cloudfoundry or Platform.sh), although they are more costly than platforms like Vercel.
I could have done that all and for sure it would have worked well enough, but as there was so much change already, I thought:
Isn't that a good occasion to give Node.js a try?
I could reuse Javascript code from my HubSpot workflows as well then, and also would finally learn something new after so many years of Spring. If I would fail, I'd still have a plan B with good old Spring Boot.
So I gave myself 2 days...
If you google "web application node.js", you very quickly end up at Express.js. I went through some tutorials, and the Hello World is really straightforward (just as in any framework, btw...):
const express = require('express')
const app = express()
const port = 3000
app.get('/', (req, res) => {
res.send('Hello World!')
})
app.listen(port, () => {
console.log(`Example app listening on port ${port}`)
})
I went on with more advanced examples, started to integrate the APIs of Shopify and HubSpot and showed the combined data in some really ugly HTML.
That was running within a few hours, so the next step was to get familiar with hosting.
I knew render.com already from my OpenSource activities on GitHub, and as they offer a free plan for individuals, that was my first choice.
Just 15 minutes later, after pushing my prototype to GitHub, my application was running on render.com. Not only that, render.com offered me a dashboard including a deployment pipeline and automatic deployment after code changes on GitHub, log console view, the possibility to maintain my own production environment and so much more. I have huge customization possibilities there (i.e. which git branch to watch on, which npm scripts to run and so much more), the admin dashboard is clean and easy to understand.
I literally had to do nothing. It was working within some minutes. Really awesome, especially as I still had the much more complex Microsoft Azure Portal in mind.
Meanwhile, I am also running a Postgres database (including automatic backup) and a Redis cache as session store on render.com, also setting that up and integrating it was just a matter of a few clicks.
After writing some more JavaScript files and letting my codebase grow, my headache grew and I remembered again why I like statically typed languages so much.
The logical solution was Typescript, also something I've never set up before, but thanks to Co-Pilot that worked quickly as well. I migrated all of my code, adjusted my NPM scripts accordingly, struggled a bit with the correct tsconfig.json (more on that later), and finally deployed my app again.
Good, typesafe now.
My big recommendation: If you're doing Node.js and take it seriously, start with TypeScript from day 1. Also lint your code strongly from day 1. I will give some hints on that later as well.
If you're coming from Spring and write your first Express.js application, you will miss one fundamental concept quite early: Dependency Injection.
If Express.js is the equivalent of Apache Tomcat, then Nest.js gives you what you know from Spring.
So, let's take the Hello World from above, with Nest.js you would write:
import { Controller, Get } from '@nestjs/common';
@Controller()
export class HelloController {
@Get()
getHello(): string {
return 'Hello World!';
}
}
A bit more of code initially, but especially if you're coming from Spring, you'll be familiar with that syntax immediately. You can inject your services and make your components much more testable.
Additionally, Nest.js offers really great documentation, check it out directly at the source, there is no need to copy code samples over here.
That was enough for me to proceed that path with Node.js. There were some more challenges like database connection or the view layer, but I could find decent solutions for all of that, and so I decided to stick with that stack.
I didn't regret my technological choice. Just two months later, we now have a web application that serves as a one-stop shop for more than 300 sales partners of greeny+, offering:
A responsive WebUI based on CoreUI, showing customers, orders and commission fees of our sales partners.
Full integration and synchronization of data between Shopify, UpPromote and HubSpot, 100% as per our needs, including an admin user interface to validate data synchronization.
Custom-built PDF invoice generation for all affiliates including mail delivery.
Automatically generated digital business cards for all affiliates including personal branded marketing PDFs. If you're curious, look at mine (German only).
And there are many more ideas to be implemented...
Important to say: I'm sure I could not have done all that without generative AI in such a short amount of time. GitHub Co-Pilot and ChatGPT really helped me a lot. You cannot take everything for granted what those tools tell you, but if you use them wisely, they give you real super-power.
That mix of AI, lightweight frameworks, easy-to-use SaaS solutions, some years of experience how to build software, and last but not least: the urge to learn new things and never settle, made all that possible.
Still, you can't compare that solution with something you would get from a whole team with experts in all areas (especially on the UI side...). Right now, I also do have my doubts how solid that architecture is on the long run, especially if the code base grows further and multiple people would work on it in parallel. I also don't have any results yet how the system behaves under high load - I didn't optimize it in that direction yet. To be fair, I also bypassed some quailty gates (i.e. there could be much more unit tests), and probably I will need to do some refactorings in near future.
It's still good enough for what we require right now.
This was only the first part of a series of blog entries, digging more into each topic from the technical side, also offering code samples for the project setup, my package.json with support for unit testing with Jest including code coverage, SCSS, webpack, and the integration with all 3rd party systems.
By the way: Shopify, UpPromote, and HubSpot were not the only external services that we integrated. I will give you more insights on that as well.
If you're interested, follow me here on HashNode and stay tuned!