Ask HN: Advice on Colocating Servers?

> Looking at our VPS provider bills each month makes me cringe.

Hosting costs often go up slowly over the years, and eventually, you have an unsustainable price. Just get quotes from a few other providers and go back to your current host and ask what they can do about the 70% price difference.

> Has anyone done this and what was your experience?

2/3 of the companies I own are on AWS. The other company is on dedicated, colocated hardware. The one on dedicated hardware gets zero benefit from CDN and cloud services as it's just a Django/Mysql monolith where every request and response are going to be different. We moved it off of AWS because there was little benefit, and we would reduce our hosting costs to a few hundred dollars a month for 20x more hardware performance.

> How do you manage the hardware and how much savings in time/$ is there really?

For the two companies on AWS, it saves us three $100k/year salaries per year. So, yes, it's more expensive than colocated hardware, but a lot less expensive than the colocated hardware and the three additional people required to provide support and ensure service levels. For the colocated hardware, we use fabric (an old python ssh automation library) to manage the servers and make heavy use of systemd's logging and systemctl for managing processes and logs. It works well, and there's maybe 1 hr a month of actual admin work, mostly dealing with OS updates and the occasional deployment hiccup.

> less expensive than the colocated hardware and the three additional people required to provide support and ensure service levels

i often hear people comparing cloud vs colo but not including more traditional dedicated hardware rental - seems to me this should be pretty similar, maintenance wise to cloud since H/W is handled by the vendor (just need to monitor hardware faults in addition to OS/app layer & notify DC on issues)

Don't. It doesn't make sense to try to jump into an area rapidly being commoditized. At most, rent bare metal/dedicated servers rather than VPS.

> Has anyone done this and what was your experience?

It works as expected. Big cost/time upfront to get all the equipment setup, then things mostly just worked. The primary challenge is transferability and standardization.

> How do you select a colo and what do you look for?

Bandwidth availability. Proximity to the team supporting it. Reputation among customers.

> How do you manage the hardware and how much savings in time/$ is there really?

Whatever custom stack is built in the colo must be well documented, and have solid maintenance schedules, and change control procedures. The cost savings exists, but it's never as much as you think.

It's difficult to articulate the specific drawbacks. It's more organizational than operational. Imagine that you have a server dropping at random times. The logs say something ambiguous, and now you have to become full time sysadmin to some elusive problem. No one in your organization will want to deal with these sorts of issues, nor will they want you to burn time on it. There will inevitably be other tasks higher on the list. Operational issues gradually build up like other forms of technical debt. No one receives applause for updating the server BIOS over the weekend. Operational discipline/maintenance inevitably becomes last priority.

What you describe only becomes a bigger problem with staff turnover. It gets to a point where fewer people know the ins and outs of the setup and are prepared to maintain it. You end up burning out a few people, or bringing in people just to deal with low-level operations issues that were created by the move away from the VPS-level of operations.

The question to ask, is it strategic to move to running our own servers? If I had a large, stable, and boring set of apps which consistently didn't take advantage of "cloud features", then it might make sense to consider moving to colocation because the only option to improve profitability may be to reduce cost. That assumes that major changes won't be made to the app that require cloud features. This is rarely the case. The opportunities seem to only appear at huge scale.

To the people suggesting that renting or installing a few servers in in a leased rack space with redundant cooling, power, conditioning and 24x7 security is somehow dangerous or hard, please go home and sleep it off. You are drunk off cloud kool aid.

> How do you manage the hardware and how much savings in time/$ is there really?

Probably depends upon how many servers you need. We colocate around 2 dozen servers and between things like updates, maintenance, and hardware, we average maybe 1 work day per month managing them.

By far our most common failure is just hard drives. We have a box of spares. Our second most common is motherboard failures - popped capacitors - from servers that are 10+ years old.

Last time anything failed was about 9 months ago. Before then we went a few years without a single hardware failure. But back in the 00's we got a bad batch of power supplies that hand a tendency to catch fire - those were interesting times.

The colo center is just a 5 minute drive from our office. And there's remote hands for simple things.

Usually it goes the other way around.

Your system critical servers are too costly/too resource intensive and you move them to dedicated.

If you're simply looking to reduce costs, why do you want colo?

Can rent dedicated server from anywhere Hetzner and OVH already mentioned, there's many others.

- Want cheap storage? Backblaze?

- Want cheap VPS? Vultr?

- Cheap storage VPS? Time4VPS or something else?

- Cheap dedicated server? Hetzner/OVH

- Cheap bandwidth? Maybe a CDN provider like bunny.net or a dedicated server provider like https://100gbps.org/ can offload traffic

Plenty of options for what you're looking to optimize for, just need to google

Colo is a whole different game, why go in that direction

I colo at a local ISP. I've been with them for about a year and I'm happy. Selection was easy: I wanted a local shop and there's only one in town. I had worked with them before on other projects and figured we would get along well.

I manage the hardware myself, mostly remotely. Occasionally I'll go onsite to upgrade or repair something. I buy used servers and parts from a little over five years ago or so. A lot of folks buy new and replace every five years, so this is a sweet-spot for performance per dollar. Kinda like buying a car just off-lease.

Working a cycle behind has its own benefits. If you pick a very common model you'll get cheap and easy-to-find parts. I was able to get a full set of spares for far less than what a typical hardware maintenance contract would cost per year (and I have them on hand rather than in 2-4 hours). Drivers (especially open source drivers) will be better developed and less buggy and you can probably find someone else's notes about anything that usually goes wrong.

Of course if you need cutting-edge gear, this won't be a good option. But I don't, so I take advantage!

I think whether you'll save money depends a lot on how you do things. There are efficiencies of scale that big providers benefit from, but there are also efficiencies you can leverage if you're small and flexible (like tailoring hardware to your use-cases and not doing/supporting stuff you don't personally use).

I didn't make the move to save money, but to get more power and flexibility. So far, so good!

Good luck! If you decide to dive in, I hope you'll come back and let us know how it goes!

Not OP. Off-lease equipment is readily available online. For one-offs, just use ebay. Beyond that, just reach out to the ebay sellers directly (their username is often the company name).

Do you have any tips on search terms you've had success with? I've seen a number of good listing from defunct startups that fire sale their remaining hardware on Ebay - but always through a direct link shared elsewhere. I don't know where to begin with finding stuff like that myself.

eBay has a good selection. I prefer to buy from sellers that specialize in server gear -- the descriptions are usually more complete/accurate and they'll (hopefully) know how to pack things so they arrive in good shape.

If you have a good surplus store or recyclers in your town it might be worth checking with them to save shipping (servers are heavy!). If you plan to buy a lot of gear regularly, it may pay you back to do some networking and make friends with folks who buy new and replace after five years -- become their recycler ;)

Suggestion: share a ballpark of your VPS bill to get better advice. The best guidance will depend on whether your VPS bill is order of $500/mo, $5k/mo, $50k/mo, or higher.

It also might help to share some characteristics of your workload. Is it CPU or disk-intensive? What kind of uptime expectations do you have? How effectively do you expect to be able to predict your requirements in the future?

This is a great suggestion. There are countless times as a consultant where I've discovered overprovisioned capacity, poorly optimized apps, sub-optimal VM types used, etc. Problems from those will still manifest in a colocated environment.

Lots of folks focus on compute and disk cost when they're in the cloud because they're usually the two biggest items on the bill. That's reasonable, but when transitioning a distributed system to a hybrid infrastructure model (eg: Cloud and a Dedicated Colo) it's important to factor in network cost. Cloud providers usually charge for ingress or egress and have marginal cost for inter-DC traffic (eg: availability zone to availability zone on AWS). Distributed systems are chatty by nature, so if requests are constantly leaving your cloud and entering your new DC you're potentially paying twice for the same transaction. This cost adds up fairly quickly. The same thing will occur if you operate an application in two regions on AWS and have a lot of region<>region activity.

A company I worked for maintained a very large on prem data center to much success. We maintained some production / user facing infra, but it was mostly for internal stuff.

We had one massive compute cluster running vmware as our personal cloud. Any dev could spin up VMs as they needed. Once this was set up (which was a lot of work), the maintenance cost in $ and time was basically 0. We also had as assortment of baremetal servers used for all sorts of things.

One of the reasons I think it worked so well for us is because IT/Linux/sysadmin skills were very high throughout the company (which I have since learned is rare). Any engineer could manage VM images, recable a server, configure VLANs, etc. If this wasn't the case, we probably would have needed double the IT team, and a lot of the cost savings would disappear.

Do you actually need colocation or can you do with a middle-ground of just renting a bare-metal server from Hetzner or OVH?

Colo is fraught with peril.

Do your due diligence in vetting any company before committing your gear and uptime to their care.

There’s a whole lot of clowns operating as colo resellers. And by clowns I mean a lot of them range from incompetent to outright scammy.

One trick I've found for colos -> Find a small local ISP or WISP, get to know the owners. Do research on local colos and what they charge for say one rack. Then halve that price, split it amongst 2-3 friends, and make an offer to the ISP. More often than not they'll accept, especially if you can pay 3 months+ in advance.

I've done this off and on for the better part of two decades. I know a few good colo's across the US if your interested.

Much of it is going to depend on your workloads. If your just running emepheral vm's on something like vmware or another hypervisor, you won't run into much of a problem.

Things start getting a bit more complicated if you are going to be using HA storage/databases. But again, that depends on your workload. And some datacenters will be happy to manage that for you.

There is alot of money that can be saved when your workloads are fairly static though. The key is putting together some hardware automation (jumpstart your hypervisors) and handling the depreciation cycle.

> How do you manage the hardware and how much savings in time/$ is there really?

There isn't any. It will cost you more.

This entirely depends. I know several businesses that save about 10X over cloud providers by using their own equipment. You have to do your own analysis.

Dedicated over cloud? Yeah, I am personally saving 40x on bandwidth alone.

Colo over renting? Savings seemed pretty small, and hassle was large...

Compared to cloud yes, but dedicated servers are cheaper than running colo, it takes a couple hardware failures to completely screw your budget up, in addition to the maintenance.

Unless you have specific impossible to rent hardware needs, it's never worth it.

Check the WebHostingTalk forums [0] for deals and promotions, especially from small providers (assuming your business case is compatible with that).

You might also want to start by renting a “dedicated server” instead of colocating your own hardware. The savings will be still be significant (especially with unmetered bandwidth, if you’re comparing to cloud vendor price gouging).

As for personal recommendations, I’ve had good experience with Psychz.net FWIW.

[0] https://www.webhostingtalk.com/

I’ve heard good things about Packet.com. They were acquired and is now part of Equinix Metal. Might be worth having a look at: https://metal.equinix.com/

How critical is "critical"?

Colo is justified solely on physical security, compliance, and/or uptime these days.

There are no net cost savings, as you would own the ongoing maintenance and upkeep.

The major IaaS players run very efficiently. The monthly bill is only the tip of iceberg. There's far more involved beneath the surface.

Find colo with 24/7 access, high up time (good networking, multiple connections to the internet from independent providers and proper setup for working when there's no power) and "remote hands" in case you need to manually reset server.

Rest of the questions you have to answer yourself. Initial server cost will be way higher than VPS plus maintenance and paying for colocation is also not cheap. Servers will have to be upgraded every ~5 years, depending on the scale and you have to buy machine for the worst case scenario, there's no automatic scaling so if you sometimes need 32 cores and 1TB of RAM you have to buy that even if 99% of the time it will sit at idle.

I would rather find cheaper VPSes for non-critical systems or work on optimization of current solution.

Did you consider moving this non-critical load to a cheaper VPS provider? You are unlikely to be able to beat them on price when taking into account hardware and engineering time.

I transitioned from VPS to rented dedicated servers years ago which was significantly more cost effective.

I recommend if you do this to try to keep your stack as portable as possible, it was relatively easy for me since I was already using Docker and started testing Rancher/K8s on the dedicated servers. This was years ago and I'm fully committed to K8s at this point.

This year I actually took it a step further and ended up just building a little 4U server that I colocated at a smaller data center that I was already renting dedicated servers from. I needed this for high data volume and latency needed to be as minimal as possible (CPU and storage together) while keeping recurring costs minimal.

For your questions:

> Has anyone done this and what was your experience? Relatively straight forward, a lot of up-front cost but has been overall about the same/breaking even with higher performance / results. I went with one that allowed me to rent IPv4 addresses without doing any peering or extra work, essentially just supply the computer, set it up, and let it go.

> How do you select a colo and what do you look for? For me, cost and latency. I've been looking into colocating another server in Asia but haven't had a lot of luck picking a specific data center yet.

> How do you manage the hardware and how much savings in time/$ is there really? Honestly, management has been pretty minimal. My server was entirely new so nothing has broken, I just keep tabs on it every couple weeks and make sure my ZFS storage hasn't turned unhealthy.

For some absolute numbers for you, my server specs and cost: 4U Colocated 4x 8TB HGST HDD ( Setup with RAID10 essentially, so 16TB usable space ) 2x 2TB NVMe SSD ( One actually isn't used currently, but is in the slot and available ) AMD Ryzen 9 ( 32 threads / 16 cores ) 4x 32gb G.Skill ram ( 128gb )

I also have a spare 256GB Samsung 2.5in SSD on standby (literally in the case, just unplugged) in-case something happens to the NVMe drives.

All-in, up-front was around $4k USD, monthly is $95 USD (all costs included), and I really only need to check on it every now and then and let Rancher/K8s take it from there. Previous costs were around $200-300/mo for a few different dedicated servers and S3 storage.

There have been incidents at the data center I went with which is definitely something you'd need to plan for, the one I went with seems to average 1 incident every 1-2 years. There was an incident a couple months ago at the data center (power outage), something happened with my server which actually required re-formatting the NVMe drives and re-setting up everything over the data center's supplied portable IPMI-ish interface, which required them to schedule a time to hook it up and then use it. Not every data center will have this or be as cooperative about it.

I'd definitely caution jumping over to colocation, start with renting dedicated servers at the very least.

colo rather than stuffing it in a cabinet please. a decent colo provider should be able to handle the rest of the stuff for ya after you send them the hardware.

These days you have to treat colocation as a fungible resource. It's good for batch processing and things like that where you can distribute the work.

How does your billing work? Are you on a plan for that VPS with dedicated, fixed burstable, or 95th percentile burstable? You may be able to find some less cringe worthy bills if you change how you are billed. Although that might mean you have to change providers, too.

> Has anyone done this and what was your experience?

I have 2x 2u in two different datacentres in the UK. Connected together via IPSEC using pfSense.

4u is common, and a rack would be nice.

Your costs will come based on racking units, bandwidth and transit/connection speeds. 1Gbit is going to be more than 100mbit but you can normally negoiate this.

Hardware costs are up-front but when the server is humming, they last for a good period of time.

> How do you select a colo and what do you look for?

Choose wehather you want:

Commercial companies (Cogent, Peer1, Rackspace) who own actual buildings and run as an datacentre as a datacentre. Try to provide the greatest, hand you an account manager tickets to DCop monkeys.

Independent who own industial units called datacentres but have decent internet feeds and a more independent feel to it. But lack the 24/7 support you may need.

In terms of selection I recommend taking a look on WebHostingTalk.com under their colocation section. Google yeilds good results. Find a provider, get in contact and take a tour of the DC.

> How do you manage the hardware and how much savings in time/$ is there really?

My servers are all second hand eBay purchases and have had a lot of TLC to get up to scrap. Once tamed, they purr happily. The amount of work on them is near to none. Create a new VM, Update OS; general SysAdmin stuff.

I would recommend that if your looking for sanity then buy new. eBay servers are hit/miss and you never know what condition they will arrive in.

iLO/LOM is configured so I always have remote management unless the DC goes dark. Servers are resillent too, I have one server which has two ram sticks failling, but still operating with no issues.

I moved from VPS to colocation and refuse to use the cloud. The cloud has its purposes but for me, but I'd rather bare metal and hypervisors. Paying for some other company to host my information where I may not even have full control of, just doesn't settle with me if I am to provide client services. Plus that I can actually own the internet space these servers sit upon. No restrictions, and run what services I desire.

My OS is FreeBSD hosting virtual machines with bHyve within a jail. Will always advocate for colocation.

   FreeBSD 11
   5:09PM  up 791 days, 18:17, 8 users, load averages: 0.66, 0.82, 0.88

   FreeBSD 12
   5:11PM  up 174 days, 21:19, 2 users, load averages: 0.59, 0.68, 0.70
   5:12PM  up 174 days, 20:03, 1 user, load averages: 0.12, 0.11, 0.09

Are you exclusively hosting other FreeBSD vhosts under bhyve?

> I was thinking maybe we could offload non-critical systems to our own colocated servers onsite

Definitely have a strategy for cooling the servers in place. If you put two dozen servers of a certain type in a room, how much are they going to warm the room up? How are you going to cool the room off in the summer (and other seasons), and what will the temperature of that room be in the summer on Saturday at 6 PM, and will anybody be around on Saturday at 6 PM, or Sunday at 4 AM if needed? If you have a ventless portable air conditioner in the server room (not that I am recommending it, but I have walked into many on-site server rooms with them), does condensation form in it? If it drops condensation into a bottle, who empties the bottle? What do you do if the condensation bottle fills up at 6 AM on a July Saturday and A/C goes off and then the temperature rises through the July day?

It's good you are thinking about this and planning this, because I have seen this happen in an unplanned manner many times. Two or three "non-critical" (until they crash) systems are put in a room on-site. Then without much planning, another server is added, and then another. Then it starts to get too hot and a ventless portable air conditioner is put in. Then the condensation bottle fills up and you suddenly have a meltdown in the room as one server after another overheats and goes on the fritz. I have seen this happen at small companies, I have seen this happen at Fortune 1000 companies.

So my advice - have heating fully planned out, and be aware that once you set a server room up on-site and its working, other divisions will start wanting to put servers in there, so pre-plan the rooms maximum capacity. I suppose electricity and racking and access and security and such need to be planned out as well. The main problem I have seen is overheating, as people, without planning, keep squeezing just one more server into the room.

> How are you going to cool the room off in the summer

In my (albeit outdated now) experience, cooling in the winter can often be even more of a problem in cold climates, if you have a condenser unit outside in freezing weather.

Lol, definitely still relevant - we just dealt with that. There are specific models with heating blocks for the compressors (one would hope the heat is recycled but I suspect it’s not).

If VPS/cloud costs too much for you, try renting dedicated (bare metal) servers. There are lots of options out there, from tiny RPi-type boards all the way to multi-CPU behemoths. You don't need to bear the upfront cost, you're not on the hook for replacing faulty parts down the road, and the total cost of rental over ~5 years isn't massively different from the cost of buying, colocating, and fixing your own hardware.

I know someone who rents a 3990X monster for less than $1K/mo. Renting comparable compute capacity from any well-known VPS/cloud provider would cost at least 10 times as much. I also know someone who rents a cluster of servers with several large disks, pushing hundreds of TB of outbound transfer, again for less than $1K/mo. The bandwidth alone would cost many times as much on AWS, not to mention the storage. Of course you'd be missing the redundancy of AWS, but whether you really need it for your specific use case is your decision to make. Anyway, the point is that most of the savings are realized in the move from VPS/cloud to bare metal, not in the move from rental to colo.