Googlebot & Next.js, finding Google Chrome 41

Back to another adventure with GoogleBot. I saw that broker list site wasn’t being fetched and rendered properly by Google (via the Google Search Console).

Rather than slowly debugging it via the Search Console tools, I set out to find a Google Chrome v41 so I could resolve the issue quickly. This was a bit tricky but apparently there’s a project called Portable Apps which has a lot of old versions of Chromium. You can find Google Chrome 41 in their listing. I think it’s good to keep that version locally just to do a periodic check to see if everything is working as it should.

In my case, the site was failing due to a fetch call. Polyfilling with whatwg-fetch or unfetch didn’t seem to work (I was getting strange errors) until I realised that next.js is also making use of server side rendering (which is run in a node environment rather than a browser environment), and the above fetch polyfills are not designed to work in a node environment. Switching the polyfill out to isomorphic-unfetch solved my issue.

In addition you need to modify your .babelrc to something like the below. The format is a little compared to a regular .babelrc, since we first need to include the presets that next defines, and then add our customisations separately.

{
  "presets": [
    [
      "next/babel",
      {
        "preset-env": {
          "targets": {
            "chrome": "41"
          }
        }
      }
    ]
  ],
  "plugins": []
}

Lastly, you need to import (after you add it via yarn or npm) for instance in your _app.js

import '@babel/polyfill';

Collecting “konsultmäklare” for BrokerList.io

After launching the first version of my catalogue of freelance brokers – BrokerList.io – I needed to actually fill the platform with content. Since the starting point was the consultant broker market in Sweden, in Swedish known as konsultmäklare, I just picked a few different relevant search phrases like “konsultmäklare stockholm“, “konsultmäklare göteborg”, “konsultmäklare malmö”.

The thought was simply to make the search, go through the top 100 search results and collect the factors that I wanted to include on the site, such as URL, LinkedIn etc. My thought was that it was stupid for me to do this myself (“my time is too valuable”) and that I should hire someone to do it.

Finding a VA

I mainly checked three outlets for VAs. First was Fiverr. I sent the specification of what I wanted done and got a quote of 10 hours from the guy, averaging on about 5 euro per hour, totalling about 50 euros.

Later, I also asked in a Facebook group “Svenska Digitala Nomader” (Swedish Digital Nomads) if anyone could recommend me some Swedish speaking VAs (this is because I thought there might be a slight advantage to someone speaking Swedish to be able to distinguish between let’s say a consulting firm and a consultant broker – konsultbyrå vs konsultmäklare, or konsultförmedlare, which would lead to less noisy results). I got a few suggestions from there.

Lastly, I posted in the group “Svenskar i Berlin” (Swedes in Berlin). Since I’ve been living in Berlin the majority of past year, I’ve realised that a lot of Swedish people here are actually living on bare minimum and are constantly struggling for money which lead me to the hypothesis that it would be easy to find someone here that would be willing to do it for less than they would in Sweden.

I got one quote for 10h at 12 euros per hour, which was higher than I expected, and with taxes that would probably end up costing something like 170 EUR.

Doing it myself

In the end, I wanted to get off the ground as quickly as possibly and I had some hours to kill so I ended up doing it on my own for one of the searches. And I’m glad I did, because in doing so, I realised that it the “well thought out process” was not so well thought out.

  1. Skip the CSV. thought it would be faster to fill a Google Spreadsheet and then export that data to the database. But in retrospect, the programmatic mangling that I would need to get it right do would probably lead to me spending more time than just filling out the forms in my platform. Lesson learned, do it directly in the platform!
  2. UX Improvements in the admin interface. Transferring the data from the CSV to the admin interface, I realise that there was a bunch of stuff that I needed to fix in the interface. One such example was the box where you pick the locations a broker is active in was way too small, leading to a whole bunch of scrolling.
  3. Time requirements. I finished the first 100 results of the query “konsultmäklare i stockholm” in about 1.5 focused hours. That would total 4.5 hours in total for the suggested three search phrases that I had asked the VAs to do for me. Basically, their estimates were twice as high as the actual time that it took me to do it.  I think this is a good learning to bring with for future hirings.

In the end, I’m happy that I did it myself at least once. Most likely, I’ll do a few more rounds myself, just to make sure there are no hidden learnings still left in there.

In the future I’ll probably get a VA to do it anyway, but I’ll be able to do that with more confidence in that I have an efficient process in place and how long it’s expected to take and thus what such an outsourcing should cost.

BrokerList.io – a catalog of consultant / freelancer brokers worldwide

Been a while since I did my last post here. To be honest, most of my own projects have lied dormant during the later part of the summer, and after that I’ve been mostly been pursuing consulting gigs.

In any case, my latest project is BrokerList.io.

First a little background for the uninitiated; In Sweden especially, there’s an established phenomenon that’s called consultant brokers (Swedish: konsultmäklare), that find prospective clients needing additional consultant resources with consultants that they find through various measures, and once a successful matching has been made, the broker takes either a fixed fee or a percentage of the perpetual billing that the consultant does.

Some people don’t like how this system is set up (I guess it does highlight an inefficiency in the system where a significant amount of the revenue is often used for the brokerage), but in any case, barring any other great alternatives, the system works quite well. Someone without any previous network among clients could come to Stockholm and find a consultant broker, and if they possess the right skillset, could be having an assignment within a matter or days or weeks.

I’ve been exposed to this need myself, having spent a significant amount of time in Berlin over the last year, and potentially moving again to another location next year. How do you find work quickly if you don’t have a preexisting network? This is true both for a person that is new to a city, but also for a person who has recently made the shift from being an employee to trying out their own wings.

Scope

Some of the functionality I’ve planned has been;

  • Keep an up to date catalogue of consultant brokers per city and country, and generate long-tail type pages for SEO-purposes. This should be internationalised over time so one can search for both “consultant broker in Stockholm Sweden” as well as “konsultmäklare i Stockholm Sverige”. The latter is a challenge that I haven’t yet solved.
  • Rating, Reviews, Subjective data. I think part of the value from such a service is that over time you can build a data set of subjective experiences from people who have been working with the consultant brokers. Basically, anonymous ratings and reviews over multiple dimensions.
  • Other resources. For instance, there are certain quite valuable groups on Facebook that the uninitiated will not automatically know about when moving to a new city. One such example is Kodapor, which is a 14000+ group of coders in Sweden.

Platform

I’ve used a platform largely similar to my previous projects.

  • React/Next.js. Recently I used Next.js on a client project, and it actually ends up saving quite a lot of time for setting up common things that you would want to set up, so I used it here to get started faster.
  • Node/Express/Nginx for the backend.
  • AWS. Interestingly enough, the first time I set up AWS it probably took me a week to bash out all of the details. This time around, it only took me a day to set up automatic deploys, auto-scaling, routing, database servers etc. It’s still a significant amount of time, but it’s nice that it’s a lot faster than before. I hadn’t used RDS before, but setting it up was very straight forward once all the other pieces were in place.
  • CrispChat. To handle any support requests.

How to Make Google Index a React Single Page Application Properly

When I was debating with myself a few weeks back whether or not I should build my death calendar project with server side rendering, the latest data I could find online said that this was essentially unnecessary since the Googlebot (the crawling mechanism that Google uses to index your site) was able to execute JavaScript since a few years back.

Well, it turns out this is not the entire story.

Fetch As Google

Lucky enough, Google offers a tool to allow you to see how Googlebot interprets your page. It’s available in under Google Webmaster Tools, under the title “Fetch as Google” once you’ve verified your internet property.

Running this test on my death date prediction tool it was clear that something was not working as intended.

So what’s the issue?

Googlebot uses Chrome 41

Turns out, Googlebot uses a version of Chrome that is by now several years old. The only way to get it to work is to polyfill whatever parts that are missing from that version of Chrome.

I debated for a while whether I should switch to server side rendering, but decided against it since that would make the entire setup significantly more complicated. Instead, I chose to go the route of including polyfills for older browser to accommodate Chrome 41. The downside of this is of course that the bundle size increases, but it felt like the most pragmatic approach.

After adding polyfills to my webpack setup and revisiting the “Fetch as Google”-tool, the site was being rendered properly again.

Howto: webpack, babel-polyfill, react and .babelrc

There are a few different changes needing to be made. I’ve outlined them below.

1. Add node_modules to your what webpack transpilates with babel

This will probably significantly increase your build time (almost doubled mine), but it seeems like some of the dependant libraries that I used required polyfilling as well. Your mileage may vary as to whether or not Googlebot will be able to crawl your site without this step.


// webpack.config.js
{
  test: /\.js$/,
  include: [
    path.resolve(__dirname, 'src'),
    path.resolve(__dirname, 'node_modules'),
  ],
  loader: 'babel-loader',
}

 

2. Add babel-polyfill as a dependency

yarn add babel-polyfill

3. Update your .babelrc file to include useBuiltIns and target older browsers


// .babelrc
[
  "env",
  {
    "targets": {
      "browsers": [
        ">1%"
      ]
    },
    "useBuiltIns": true
  }
],

4. Deploy new version and confirm with “Fetch as Google” that it’s working as intended.

Done.

Hope you Googlers find this useful, as information on this topic / for my setup was pretty sparse AFAICS.

Squeeze the life out of your work

I just finished up the first version of the chrome extension I’ve been working on. Basically, I wrapped up the death clock that I had been working on for the last few weeks (yikes, been that long already?) into a Chrome extension together with some tweaks and minor features to make it more usable

Marketing photo I created for the Death Calendar Chrome Extension

This whole thing is based on an idea I’ve thought about regarding product development which is:

Squeeze the life out of your work

Basically, each new project will cost you quite a bit, both in terms of actual operative work (coding, marketing etc) but also in terms of learning about the domain and old school mental juice.

That’s why I think it’s pretty nifty to launch several “products” or ideas that all relate to the same core concept like I’ve done with the death clock website and the death clock chrome extension.

In this case, I could reuse the majority of the work I had already put in and revamp that into something similar but different, in this case a Chrome Extension.

You might think this is stupid, since I’m pouring more resources into an idea or a space that essentially remains unvalidated, and maybe that is true. On another hand, I never expected it to be successful right on launch, but I’m taking a more long term approach.

The thought is that by creating a number of products and services around a central theme, I’m basically doing the product-version of content-marketing. All roads lead to Rome.

How to Set Up Google Analytics with Autotrack.js

After reviewing the Google Analytics data for my Death Clock project this morning, I saw that the bounce rate was really high, and I started exploring why this was the case.

Background & Confusion Phase

Indeed, it seems like there’s a limitation in how the default Google Analytics setup works with Single Page Applications (SPAs). Basically, using the recommended approach of adding the gtag.js tracking snippet, only the first page view would be tracked and not any subsequent site navigations that are made via the History API which is common for single page applications.

Turns out, there’s a library written by some people that work at Google called autotrack that can help with this. A little confusion ensued since autotrack uses another tracking Google tracking snippet called  analytics.js, while the Google itself recommends using gtag.js (and actually doesn’t even present the code for analytics.js on the tracking code page). Turns out both are still fully supported.

After some research, two approaches I found were to use Google Tag Manager to emulate the behavior of autotrack.js and deploying autotrack.js with Google Tag Manager. I actually began implementing the second approach, but one critical limitation of that approach is that it only works with plugins that fire AFTER the page the page view has been sent. This would mean that cleanUrlTracker, which helps us only track canonical urls, without the query part, wouldn’t work with this approach.

Turns out, Google Tag Manager is completely unnecessary for this setup.

Why use autotrack at all?

Autotrack helps us with a bunch of stuff, such as tracking outbound links, merging similar urls that in the default setup get reported as different pages (for instance /contact/ and /contact and /contact?someparam ) and enable’s easy event tracking among other things.

Solution / TLDR

Turns out, the solution is quite simple and straightforward once you understand how the different tools, snippets and libraries relate to each other.

Basically the process only has two steps:

  1. Switch to using analytics.js instead of gtag.js. Don’t worry, it’s still fully suported by Google.
  2. Load autotrack.js with whatever plugins you want to use. I chose to do it via a cdn since I didn’t want to self-host, but that’s an option as well.

TLDR, the final tracking code ended up being something like


<!-- Google Analytics -->
<script>
window.ga = window.ga || function () {(ga.q = ga.q || []).push(arguments);};
ga.l = +new Date;
ga('create', 'UA-YOUR-TRACKING-CODE', 'auto');

ga('require', 'cleanUrlTracker', {
stripQuery: true,
indexFilename: 'index.html',
trailingSlash: 'remove',
});
ga('require', 'urlChangeTracker');
ga('require', 'eventTracker');
ga('require', 'maxScrollTracker');
ga('require', 'outboundLinkTracker');
ga('require', 'pageVisibilityTracker', {
sendInitialPageview: true,
visibleThreshold: 2000,
});

// The command below is no longer needed because it is triggered by pageVisibilityTracker
// ga('send', 'pageview')
</script>
<script async src='https://www.google-analytics.com/analytics.js'></script>
<script async
src="https://cdnjs.cloudflare.com/ajax/libs/autotrack/2.4.1/autotrack.js"></script>
<!-- End Google Analytics -->

DeathClock.io launched!

Preface

To be completely honest and transparent with myself and the world, I’m an 80%er. What I mean by that is, I love creating things, but usually I hack away for weeks or months, getting most of the stuff in order, just until I reach the 80% threshold, and then I lose all motivation. I’ve known this about myself for a long time. There has been some notable exceptions, but generally that’s how it’s played out historically when hacking on my own projects.

With that as the backdrop, it glads me to say that I’ve released the first version of my first hacked together project, Death Clock.

Idea

Basically, the idea was to create a simple visualisation tool of how much time you have left to live to act as a gentle nudge towards making great use of whatever time you have left. I had thought about this idea for long (not to be creepy; but I think about death a lot, in a very practical and non-pathological way).

Initially, the thought was get the user to a page where they can get a visualisation of the passed and remaining weeks of their approximated life expectancy, together with some facts intended to create emotion within the user.

Pitching this idea to people around me, some pointed me to the great Wait But Why article doing a very similar thing. In any case, I wasn’t deterred by this fact, since I have a slightly different approach and plan behind the whole thing.

Scope / Features

One of the main reasons I chose this idea and not one of my many other stupid ideas was that it was pretty well-constrained in terms of functionality necessary to create the first MVP, and that a business model was included right from the start. While there were some cool gizmos, gadgets and technical aspects that I wanted to implement, the MVP really just consisted of:

  • An algorithm to calculate a persons expected life expectancy
  • A basic app to visualise this
  • Simple payment processing
  • A printing service to handle any orders

I’m a great coder, so I’ll just bash that out in a day or two, no? Well, we’ll see.

Platform

For the technical platform, I decided to use a mix of technologies that I knew well and try out some new things that I knew less well. Below is a list of what I used and why.

  • React. For frontend stuff. Been working non-stop with React for the last few years, so it felt like a natural choice.
  • Node/Express.  For backend stuff. Since I’m super-comfortable with Javascript because of all the frontend programming, this also was an easy choice. Also, in terms of backend, I only needed something minimal.
  • AWS. I had used AWS very sparingly before, and for this project it was probably completely overkill, but there were really two justifications for using AWS.
    1. Learning. I know that AWS has wide-spread adoption, especially in bigger projects, and learning how to set up something with AWS will probably prove to be very useful for the future.
    2. Scaling. Again, probably overkill for this project (both since it’s super simple and because it will probably be a long time before you have to think about scaling). However, I felt that if I set it up properly, if I ever needed to scale up, it would just be a click of a button rather than setting up everything from scratch again. Together with the learning aspect, I felt that this was a good future proofing in case I ever got boosted in some social media.
  • Nginx. The site uses nginx as a reverse proxy for the node/express server. Again, this is overkill for this project most probably, but the motivation is the same as for using AWS; learning and scalability.
  • CrispChat. I chose to go with CrispChat as a primary support channel, mostly because they were free and they covered my basic needs. I don’t really have any feedback to give on this service so far.
  • Printful. Printful handles the printing of the posters, when and if the orders come. So far I haven’t written an integration for Printful since I thought I could do it manually until I’ve validated the idea (this part is super specific for this idea, whereas the other tech stuff is usable in all future projects).
  • Stripe. I’m using Stripe Checkout to process any payments that I get. I’ve gone completely database-less for this app (at least as of yet), so whenever an order is made, the required data is just attached as metadata to the Stripe charge for later manual processing. Integration with Stripe was pretty much a breeze, which was nice.

(Technical) Challenges

The absolute biggest challenge was that I had never used AWS, and getting it to work well proved to be quite a cumbersome task. This probably took me 7-8 times as much time as I expected it to do. However, now it’s set up in a fashion where if the server running the application is killed, it can almost instantly spin up a new server of any capacity to replace the previous one. Some stuff that took longer than expected

  • Figuring out how to setup all the required tooling automatically on instance creation.
  • Auto deploy whenever code had been pushed to GitHub.
  • Auto scaling (if the server ever crashed or was terminated)
  • Automatically fetching whatever secrets were needed from AWS SSM Parameter Store.
  • Automatically setting up SSL-certificates on new instances

Possibly I’ll collect all of my findings for setting up AWS into another post sometime. There are a lot more moving parts than for instance setting up a VPS at DigitalOcean (an approach I’ve chosen before), but the hypothesis is that this will pay off in the long term.

User Testing

So far I’ve done little to no user testing. I’ve only really showed one other person how to use and got some good feedback regarding additional interactivity and to showing a real world photo of how the poster might look.

Investments & Costs

  • Money. Most of the infrastructure falls under AWS Free Tier so it should be close to free. I bought the domain name deathclock.io for something like $10.
  • Time. The biggest cost has been the time investment. I’ve probably spent something like 60-80 hours working on it effectively, where probably half of that has been setting up AWS (no kidding).

Resources

Some resources found along the way;

  • NameMesh. Pretty good domain name generator that also shows the availability of the domain name. Doesn’t seem to be working entirely correctly for all TLDs, but works well enough.
  • AWS Cheat Sheet. Basically, it translates all of the weird AWS names into plain English, like writing VPS instead of Amazon AWS EC2 which, to be honest, is utter gibberish.

Future

I’m not expecting this service to blow up on day one. When I’ve launched stuff in the past, usually and extra ordinary amount of work is required in the beginning, and then after an arbitrary amount of lag time it starts getting traction, so I have that long-term mentality in mind.

However, in the short term, I’ve thought about

  • Using Google Adwords to drive traffic to it, to see if it converts at all, and start optimising the conversion.
  • Do more user testing with people physically around me.
  • Add more interactivity and fun facts.
  • Ask more designer oriented people around me to give some feedback
  • Order some posters to do actual product testing ( I should have, but I haven’t yet. Lean.  😃 🐴 )

Feedback?

If you have any feedback on the this post, I’d love to hear it. I’m pretty new to sharing in this transparent fashion, so I’m curious to hear what you think. Please write a comment below or send me an email! Also, if you have any feedback or ideas my death clock project, that’s appreciated as well!

Hello World

Something that I’ve drawn great inspiration from has been to read about the journey of others going down the route of creating digital products. Basically, this blog will serve as a tool for;

Continue reading →