2

2019 in technology

 3 years ago
source link: https://devblog.songkick.com/2019-in-technology-b9fe9182b7e2
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

2019 in technology

At its height songkick.com reaches over a hundred requests per second with upwards of 7000 users active on the site at a time, whilst our apps serve millions of users per month worldwide. To help us build products at this scale, our platform is built on the service oriented architecture approach where we have many applications each dedicated to a very specific aspect of the Songkick product offering. With this architecture comes some rather mundane maintenance tasks — like ensuring our frontend applications are running the latest version of Ruby on Rails or ensuring our services are running the latest version of Sinatra — but it also comes with some more exciting technical challenges, such as asking what technology choices would allow us to best serve our users or how we should build applications that scale. Our technology choices and achievements last year meant 2019 was one of the biggest years of change ever in Songkick’s technology history. Here are a few highlights:

Cloud migration

We can’t review the tech achievements of 2019 without opening with the cloud platform migration. The year saw the completion of an 18 month project to migrate all our applications and databases out of dedicated servers in a data centre and onto Google Cloud Platform. Our aim was to get to a position where we could treat our servers like cattle, not pets, which meant we must be able to cope when servers are removed from instance groups, or ensure that servers are able to bring themselves online without manual intervention when we spin up new ones. This approach brings many advantages — for instance, servers no longer require complex manual set up tasks and subsequent long-term nurturing; instead we can spin them up on an ad hoc basis when we need them (such as scaling up to handle load). Why is this particularly useful for us? Well, not everyone in Songkick’s tech team would feel comfortable bringing new Linux servers online, but now anyone can scale up any of our applications with the click of a button in a web interface. The result of 18 months of hard migrating is an infrastructure that sits entirely within Google’s Cloud offering, with Cloud Instances for servers, Cloud SQL for databases and Cloud Storage for persistent data stores.

Interested in how we managed the migration process? Have a read of the blog post here.

Three’s a cloud

In some cases, migrating applications to the cloud platform required an entire reworking to ensure they fitted well within the cloud model. Our ETL (Extract, Transform, Load) data processing suite is a good example. When you have a complex web of data processing tasks that must be run in sequence, it is not enough to rely on a single cron job kicking off the pipeline on a machine that in theory should cope with being killed at any time. To address this we migrated our ETL pipeline to Spotify’s Luigi platform, which is designed to handle complex task dependencies and play very nicely in cloud environments. Check out this blog post if you want to learn more about how we went about this.

Another example is our search system, used to search for events, artists, venues and cities on Songkick products. We previously used Apache’s Solr product for our search needs, but our team’s understanding of Solr was limited and moving to Solr’s cloud offering would have been an extra steep learning curve given other dependencies like Zookeeper. As a result we opted to migrate to Solr’s biggest open-source competitor, Elasticsearch. Elasticsearch makes it easy to set up clusters in cloud environments, and we already used it for our log processing, so it made sense to bring our generic searching into the same technology and reduce the learning requirements of our tech team.

New products, new tech

2019 saw the advent of Songkick’s dedicated Presales tool, which gives artists the ability to offer early access to tickets to their fans on Songkick. Since its launch in August, the product has been used to get presale codes into the hands of over six million live music fans. This product offered exciting technological challenges — how do we build an intuitive user interface where artists and their managers know exactly how the presale will be marketed? How do we build a reliable, scalable notification platform that guarantees users receive a presale code at the right time before on-sale? How do we ensure users across all our products get the best out of presales? The resulting product includes a tool to build presales in Songkick’s Tourbox platform, and with a vision to build a reactive, user-friendly interface with real-time preview of the presale email, we opted for — you guessed it — React. React allowed us to quickly build out a presale compose tool that was easy to integrate into Tourbox, easy to iterate on and (relatively!) easy for our JavaScript-friendly developers to sink their teeth into. Behind the scenes, we use background jobs and RabbitMQ to ensure delivery of presale notifications — with queue consumers that are designed to be reliable, delivery-guaranteeing and easily scalable to cope with larger presales and to improve delivery speed of the notifications.

We also built presale information components on our iOS app, Android app, and songkick.com website so all our users can benefit from presale tickets.

Playing catch-up with frontend technologies

2019 also saw us brush off the cobwebs from some of our longest-serving frontend applications and modernise some of the technologies we use behind the scenes. The songkick.com website has finally got full SASS capability and the CSS to SASS migration is well underway. With this we can build more consistent features on songkick.com quicker and easier than we have previously been able to. We migrated our JavaScript tests out of a deprecated technology called JSTest into the much-loved Jasmine framework, which we now use for all JavaScript tests across our products.

Frontend applications include our apps as well as our websites — so they got a bit of love too. We pushed to migrate all the Objective-C code in our iOS app over to Swift, because we feel that Swift is a more approachable, familiar language to our developers who mostly work with Ruby and JavaScript in their day to day. We didn’t quite migrate the entire app, but we’re at around 95% of Swift code coverage. In a similar fashion we shifted some of the Java in our Android app to Kotlin — Kotlin and Swift are similar enough to follow the same learning approach and reduce the learning deficit in our team.

Education, education, education

Inevitably with change on this scale comes the question of how we ensure our tech team are up-skilled to maintain our products and feel comfortable fixing bugs when they come up. To tackle this, we’ve run weekly talks for the whole engineering team on topics ranging from Docker to SASS to the React component lifecycle. Further, we employ regular pairing in our day-to-day, we’ve run mob programming sessions and whiteboarding sessions, and will soon be trialling small, targeted sessions for developers where they work in small groups with a technology knowner to learn areas of specific interest.

2020 and beyond

So 2019 was an exciting time for technology at Songkick, and there’s a lot more coming in 2020. From moving to the amazing Sentry for all our error reporting needs, to improving the scalability of songkick.com by allowing us to cache pages in Fastly, watch this space for more exciting things coming up.


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK