The Curious Case of the Disappearing Content

As a Java developer, I’ve spent most of my professional life working on the behind-the-scenes parts of software systems – the so-called “back-end.” But lately, I’ve found myself wanting to branch out and dabble more in HTML and UI development. A couple of years ago, this natural curiosity led me to start a new side project: a hobby application that would only be used by me and a few friends.

The main goal was to get it working fast, so I settled on JHipster – a development platform for building web applications using modern technology like Angular, React or Vue for the client side, and Spring plus Gradle or Maven for the server side. Within a few weeks, I had a functioning application that met all my needs. But then, something funny happened – other people started using the application. Knowing I had created something useful for a larger audience was really satisfying. So, like any other developer who is already stretched thin and trying to balance a full-time job, a family, and hobby projects, I spent my nights, weekends, and every free moment I had working on it.

However, the more I tried to improve the application, the harder things got. I spent a lot of time looking up how to do new things that weren’t part of the boilerplate setup. I was learning some of the limitations that now felt like major roadblocks. And after a few months, it became clear to me that my choice of technologies was becoming a hindrance to making the application better.

The Perils of SEO and Single Page Apps

You see, while JHipster and Angular are not bad platforms by any means, my lack of knowledge of the technologies had come back to bite me. There were plenty of reasons that might have made me think differently, had I known about them. It was a classic developer lament: I didn’t know what I didn’t know. And what I came to realize is that my application was suffering in several key areas that were a direct result of the platforms I had chosen. Things like SEO, social sharing, and caching.

Let’s start with SEO. Search engine optimization is the process of formatting web page content so that it’s easy for web crawlers to understand. Just like a web browser, Google and other web crawlers request the HTML contents for pages on a website. But instead of rendering it like a web browser would, they analyze the HTML to semantically understand what the web page is about.

And that’s where I ran into my first problem. HTML is structured and easy for computers and bots to understand, but they don’t necessarily understand JavaScript. While there are plenty of articles that debate whether or not Google’s crawler executes JavaScript when crawling a website, my experience was that it did not. The reason I knew this is because Google was telling me that it didn’t understand my content.

When I looked at the search analytics for my site, it was ranking for exactly one keyword. And that keyword had nothing to do with my website. Google thought my website was related to Maven proxy configuration, even though my hobby project was not in any way related to the open source Apache project. Why? Because my website had hundreds, if not thousands, of unique web pages with varying content, yet Google only showed it in search results to people searching for the words “mvnw proxy.”

The problem was that Google’s crawler was not executing the required JavaScript files that made my single-page application function. It was loading the template HTML file and scanning its structure for hints about what my site was actually about. And like most single-page apps, the default HTML included lots of helpful developer information that is intended to be used for troubleshooting, but never actually displayed in a web browser when things are working properly.

The Struggle with Social Sharing

Another area I ran into problems with was social sharing. My website allowed users to create dynamic content, and also included lots of static content that others might find useful. And in the early days of launching, I indeed saw that people were sharing links to my websites across various social media platforms.

Social networks, much like search engines, rely on the content in web pages to understand what the web page is about. Unlike search engines though, they rely less on the visible content of the web page (the text and images humans see) and more on the metadata (the stuff inside the HTML that us humans don’t care much about).

For example, when you share a link to a website on Facebook, the first thing that happens is Facebook reads the webpage and generates a nice preview of that article. The preview has a title, a line or two of descriptive text, and an image. But those previews are not generated magically or using some sophisticated AI algorithm. Facebook is relying on metadata inside the HTML header area to create previews.

And just like with SEO, Facebook was falling victim to the same problem – it was reading the template HTML file as-is, and not applying the JavaScript that would help fill in the metadata and create meaningful previews. What this meant for my users is that every link from my website that was shared to Facebook and other social networks was generating the exact same preview. Whether it was users sharing their custom content, one of the static pages that I auto-generated, or even the home page, every link shared on social media had the same preview.

The Limitations of Caching

Another general area I wish I had been more cognizant of is technology envy. As a developer wanting to improve my skill set and value to potential employers, I’m constantly on the lookout for new technology. But sometimes it’s easy to get envious of what others do, without realizing the skills you already have are valuable too.

This happens a lot on social media. I’m constantly in awe, and frankly a bit jealous, of what some of other people create and build. I see buzzwords and technology and think to myself, “I really ought to know how to use these things.” And while there is value in learning new things, it’s more important to be adaptable and know the right tools for the job.

As the saying goes, a jack of all trades but a master of none. And that’s exactly what happened when building my first single-page app. I was so caught up in the short-term excitement of learning a new technology, I never stopped to consider how that decision would impact the future of the app.

And one of the areas where this really came back to haunt me was caching. With a small user base initially, I never worried much about expensive database queries or page load times. But as new users started to use the website, I started to get worried about these things. Were my queries as performant as possible? Was I taxing MongoDB with too many requests? Would new users give up if a page took too long to load?

So I started thinking about how to improve some of the MongoDB queries and page load bottlenecks. And one of the first things that came to mind is caching. I’ve worked with a variety of enterprise caching technologies such as Coherence, Ignite, Redis, and others. For what I was looking to do, these all felt like overkill. Plus, they would add to the compute and memory costs of a project that was still technically only a hobby.

Instead, I decided this was a perfect use case for CloudFlare. I was already using CloudFlare as my DNS provider because they provide a ton of great features for free. And one of those great features is page caching. It’s free and requires no additional coding on my part.

Here’s how it works. CloudFlare acts as a reverse proxy to my website. This means all requests are really going through their infrastructure before being forwarded to my website. Among other things, this allows them to cache my server responses across their vast network of global data centers. All I have to do is configure which set of URLs I want them to cache and how long they should be cached, and CloudFlare handles all the heavy lifting. All without me writing a single line of code.

However, in practice, it wasn’t quite working as I expected. After making the change and enabling page caching, there was no appreciable difference in my MongoDB cluster resources. I was expecting to see a decrease in resource utilization as far fewer queries would be made. But instead, I saw things mostly staying the same.

Why? Because CloudFlare doesn’t execute any JavaScript prior to caching the response. It simply takes the raw HTML response from the origin server and caches it. In my case, that response was the template HTML. And the net result is that every page I wanted to cache was really just caching the same template HTML. So when a user requested one of those cached pages, CloudFlare simply returned the bare bones template HTML to them. And in turn, the user’s web browser would load this template and download the required JavaScript files. Those JavaScript files would send a bunch of requests to my server to get the real content for the page, meaning whatever expensive queries I was hoping to avoid were always going to be executed.

Lessons Learned and Moving Forward

In the end, this was still just a hobby project, and the experience I gained will go a long way. In a perfect world, developers would have all the information they need up front. Every requirement, every change in scope would be known ahead of time. But this isn’t how our industry works. We create roadmaps as best we can, and we choose technologies that make sense given what we know. More often than not, our best laid plans don’t pan out the way we envision. And that’s okay.

Mistakes are good. They help us learn. They help us make better decisions down the road. I’ve already got new ideas for how I can use single-page apps for other projects I am working on. And this time I’ll have confidence knowing they are the right tool for the job. I can also now confidently speak to clients and tell them when I think a single-page app makes sense for their use case. And I can discuss my experience with colleagues, helping them to avoid some of the pitfalls I had to deal with.

As for my original hobby project, I eventually decided to re-write most of it using frameworks that were more familiar to me. And while the decision to re-write wasn’t an easy one, it was the right call. The application has continued to thrive, and I’m confident that the lessons I learned will help me build even better products in the future.

So if you’re considering a single-page app for your next project, just remember: SEO, social sharing, and caching are all critical considerations. And don’t be afraid to lean on tools like Manchester SEO to help you navigate the complexities of modern web development. With the right strategy and the right partners, you can create an engaging, high-performing single-page app that stands the test of time.