Howdy folks—my name is Lee, and I run the Space City Weather servers & back-end. (I don’t post much, so you might not recognize the name!) I wanted to weigh in real quick on the status of the e-mail deliverability issues some readers have experienced over the weekend, and explain what’s going on.
(This is not a weather post! If you’re interested in hearing more about our current cold snap, Eric & Matt will be posting their normal update tomorrow morning. This is just a quick technical update on recent e-mail issues.)
Here’s the short version: there were some hiccups this weekend, but things are working fine now. If you don’t care about the deep technical details, you can stop reading now 😀
The deep technical details
SCW runs on Wordpress. (See this post for details on the SCW hosting stack.) For cost reasons, the site relies on the Wordpress Jetpack service for the delivery of the daily e-mails—which means that, ultimately, e-mail is out of our control. The ultimate reason for this boils down to cost: the Wordpress Jetpack service will happily—and more importantly, for free—send e-mails to all 20,000+ SCW subscribers whenever Eric or Matt (and Maria!) make a post.
It turns out that e-mail in bulk is an incredibly expensive service to provide, and leaning on the built-in Wordpress Jetpack e-mail service saves SCW literally tens of thousands of dollars per year. (Seriously, we’ve run the numbers, and going with a commercial e-mail service provider like Mailgun or Mailchimp, or even rolling our own solution with something like Amazon SES, would be a massive cost burden at the scale we’re operating at.) As the sole infrastructure person, I feel a heavy sense of fiduciary responsibility with where and how we choose to spend resources on hosting, and so in spite of the inherent compromises, we’ve stuck with using Jetpack for e-mail updates.
One of those compromises is that we don’t have a lot of control over when and how the daily e-mails are delivered—we are dependent on the Jetpack service to be up and running. Which it usually is! However, for some reason that remains unexplained, over the last few days there have been some failures with the daily e-mails.
I’m very sorry, and I take personal responsibility. We’ve made the conscious choice to use the “free” Jetpack e-mail service in lieu of standing up our own, due to the cost and complexity involved (which truly would be a not-insignificant >$1k/mo expense—sending out millions of e-mails per month has a real cost!). Occasionally, the Jetpack service will have issues that we don’t have much insight into—and that’s apparently what happened over the last couple of days.
Fear not, though—Jetpack e-mail has been reliable (more or less) for every day of every year since I took over hosting the site in 2017. And the Wordpress Jetpack support crew have been extremely responsive in the past when I’ve had to open support tickets to work issues. Eric and Matt and Maria are committed to getting all y’all the best possible forecast data, and the backend crew of Dwight, Hussain, and I are committed to making sure those forecasts get to you immediately and without delay—come rain, snow, or server crashes 🙂
Cheers, everybody. Thanks for reading Space City Weather!