<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Terraform on KJ6LNH</title><link>https://kj6lnh.org/tags/terraform/</link><description>Recent content in Terraform on KJ6LNH</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Fri, 12 Jun 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://kj6lnh.org/tags/terraform/index.xml" rel="self" type="application/rss+xml"/><item><title>Replacing Self-Hosted Gotify with a Serverless AWS Backend</title><link>https://kj6lnh.org/serverless-gotify/</link><pubDate>Fri, 12 Jun 2026 00:00:00 +0000</pubDate><guid>https://kj6lnh.org/serverless-gotify/</guid><description>How I rebuilt the Gotify notification server as Go Lambdas, DynamoDB, and a single CloudFront distribution — keeping the stock Android app working unmodified, for pennies a month.</description><content:encoded><![CDATA[<p>For a couple of years I ran <a href="https://gotify.net">Gotify</a> on a small EC2 instance to get push
notifications from my own scripts and services to my phone. It&rsquo;s a great little
project: a single Go binary, a REST API to post messages, and an Android app that
holds a WebSocket open to receive them in real time. The problem was never Gotify —
it was the box it ran on. Patching, the occasional reboot, paying for an always-on
server to handle a few notifications a day. It felt like a lot of standing
infrastructure for something that&rsquo;s idle 99.99% of the time.</p>
<p>So I rebuilt the <strong>server</strong> as a serverless stack on AWS, with one hard rule: the
<strong>stock Gotify Android app must keep working, unmodified</strong>. No fork, no custom build —
just change the server URL in the app&rsquo;s settings and everything continues to work.</p>
<p>The result is <a href="https://github.com/mjlocat/serverless-notify"><code>serverless-notify</code></a>: a
Gotify-wire-compatible backend built from Go Lambdas, DynamoDB, and a single CloudFront
distribution. It costs me effectively pennies a month, and there&rsquo;s no server to keep
alive.</p>
<h2 id="the-one-constraint-that-shaped-everything">The one constraint that shaped everything</h2>
<p>&ldquo;Keep the stock app working&rdquo; sounds like a small requirement. It turned out to be the
single most important architectural constraint in the whole project, because of one
detail in how the app connects.</p>
<p>The Gotify app derives its WebSocket URL from the same base URL you give it for REST.
Point it at <code>https://notify.example.com</code> and it will:</p>
<ul>
<li>make REST calls to <code>https://notify.example.com/message</code>, <code>/application</code>, etc., and</li>
<li>open a WebSocket to <code>wss://notify.example.com/stream</code>.</li>
</ul>
<p>Same host, same scheme family, two very different protocols. On AWS, the natural
homes for these are an <strong>HTTP API</strong> (for REST) and a <strong>WebSocket API</strong> (for <code>/stream</code>)
in API Gateway. But API Gateway will not let an HTTP API and a WebSocket API share a
custom domain. And the app gives me exactly one base URL to work with.</p>
<p>The fix is to put <strong>CloudFront</strong> in front of both and let it be the single public
hostname:</p>
<pre tabindex="0"><code> Senders / Android app  ──HTTPS+WSS──▶  CloudFront (notify.example.com)
                                          │
                                          ├─ /stream*  ─▶ API Gateway WebSocket API ─▶ ws Lambda
                                          ├─ /image/*  ─▶ S3 (app icons)
                                          └─ default   ─▶ API Gateway HTTP API      ─▶ api Lambda
                                                                                         │
                              new message ──postToConnection()──▶ live WS connections    │
                                                                                         ▼
                                                          DynamoDB (single table) + SSM (credentials)
</code></pre><p>CloudFront routes by path: <code>/stream*</code> goes to the WebSocket API, <code>/image/*</code> to an S3
bucket of app icons, and everything else to the HTTP API.</p>
<p>There&rsquo;s one more wrinkle. The WebSocket API only answers on its stage path
(<code>/&lt;stage&gt;</code>), not on <code>/stream</code>. So a tiny <strong>CloudFront Function</strong> runs on the
WebSocket handshake and rewrites the URI:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">handler</span><span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kd">var</span> <span class="nx">request</span> <span class="o">=</span> <span class="nx">event</span><span class="p">.</span><span class="nx">request</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">request</span><span class="p">.</span><span class="nx">uri</span> <span class="o">=</span> <span class="s2">&#34;/production&#34;</span><span class="p">;</span> <span class="c1">// the WS API stage
</span></span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="nx">request</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span></code></pre></div><p>That&rsquo;s the whole trick that makes <code>wss://notify.example.com/stream</code> reach a WebSocket
API living at a completely different execute-api hostname. With it in place, the app
can&rsquo;t tell it&rsquo;s not talking to a real Gotify server.</p>
<h2 id="two-lambdas-one-table">Two Lambdas, one table</h2>
<p>The compute is two small Go Lambdas on the <code>provided.al2023</code> ARM64 runtime:</p>
<ul>
<li><strong><code>api</code></strong> — the entire Gotify REST surface (applications, clients, messages) plus the
sender endpoint <code>POST /message</code>. When a new message arrives it also fans it out to
every live WebSocket connection.</li>
<li><strong><code>ws</code></strong> — the WebSocket lifecycle: <code>$connect</code> (authenticate the client token, record
the connection), <code>$disconnect</code> (forget it), and <code>$default</code>.</li>
</ul>
<p>Everything persists in a <strong>single DynamoDB table</strong> using the classic single-table
pattern — applications, clients, messages, live connections, and an atomic counter all
share one table, distinguished by partition key:</p>
<pre tabindex="0"><code>Application  PK=&#34;APP&#34;      SK=&lt;padded id&gt;
Client       PK=&#34;CLIENT&#34;   SK=&lt;padded id&gt;
Message      PK=&#34;MSG&#34;      SK=&lt;padded id&gt;
Connection   PK=&#34;CONN&#34;     SK=&lt;connectionId&gt;
Counter      PK=&#34;COUNTER&#34;  SK=&lt;name&gt;
</code></pre><p>A small but important detail: Gotify message IDs are <strong>monotonic integers</strong>, and the
app&rsquo;s pagination relies on it (each page is &ldquo;messages with an ID less than the last one
I saw&rdquo;). DynamoDB doesn&rsquo;t hand you an auto-increment column, so I use an atomic
<code>UpdateItem ADD</code> against a counter item to mint the next ID. Sort keys are
zero-padded so lexicographic ordering matches numeric ordering.</p>
<p>When the <code>api</code> Lambda accepts a message it calls <code>PostToConnection</code> on each stored
connection ID to push it down the open sockets. Dead connections come back as a
<code>GoneException</code>, which is my cue to prune them from the table.</p>
<h2 id="wire-compatibility-is-mostly-about-being-boring">Wire compatibility is mostly about being boring</h2>
<p>Most of &ldquo;make the stock app happy&rdquo; is just returning exactly the JSON shapes Gotify
returns. The app reads <code>GET /version</code> and gates behavior on it, so the server reports a
real-looking Gotify version. Tokens follow Gotify&rsquo;s format — a one-character type
prefix (<code>A</code> for application/sender tokens, <code>C</code> for client/receiver tokens) followed by
14 URL-safe random characters — so the app accepts them without complaint.</p>
<p>Auth is deliberately minimal because this is a <strong>single-user</strong> deployment: it&rsquo;s just
me. Senders present an application token; the app presents its client token for both
REST and the WebSocket. Management calls also accept HTTP Basic auth, with the one
username/password pair stored as an encrypted SecureString in SSM Parameter Store
(behind a dedicated KMS key) and loaded once at Lambda cold start.</p>
<h2 id="the-bug-that-only-the-real-app-could-find">The bug that only the real app could find</h2>
<p>Here&rsquo;s my favorite gotcha, because it&rsquo;s a perfect example of why &ldquo;it passes curl&rdquo; is
not the same as &ldquo;it works.&rdquo;</p>
<p>After switching my phone to the new server, live notifications arrived perfectly over
the WebSocket. But when I opened the message list, it just spun forever — no history
ever loaded. The app logs had the answer:</p>
<pre tabindex="0"><code>java.lang.NullPointerException: getSince(...) must not be null
  at com.github.gotify.messages.provider.MessageStateHolder.newMessages(...)
</code></pre><p>The message list endpoint returns a paging envelope with a <code>since</code> cursor pointing at
the next page. My Go struct had it tagged like this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">Since</span><span class="w"> </span><span class="kt">int64</span><span class="w"> </span><span class="s">`json:&#34;since,omitempty&#34;`</span><span class="w">
</span></span></span></code></pre></div><p><code>omitempty</code> looks harmless. But on the <strong>last</strong> page — the common case where there&rsquo;s
nothing older to fetch — <code>Since</code> is <code>0</code>, so <code>omitempty</code> dropped the field from the JSON
entirely. The Gotify Android client treats <code>paging.since</code> as <strong>non-nullable</strong> and
crashes the instant it&rsquo;s missing. The WebSocket path never touches paging, which is
exactly why live delivery worked while the list was dead.</p>
<p>The fix was simply to remove the <code>omitempty</code> flag — always emit the field:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="nx">Since</span><span class="w"> </span><span class="kt">int64</span><span class="w"> </span><span class="s">`json:&#34;since&#34;`</span><span class="w">
</span></span></span></code></pre></div><p>No amount of <code>curl</code>-ing the endpoint myself would have surfaced this. Only the real
client, with its strict deserialization, cared. It&rsquo;s a good reminder that
&ldquo;wire-compatible&rdquo; means compatible with the <em>consumer&rsquo;s</em> expectations, not just with a
spec you read once.</p>
<h2 id="app-icons-a-half-wired-feature-finished">App icons: a half-wired feature, finished</h2>
<p>The other rough edge was application icons. The infrastructure was there — CloudFront
serves <code>/image/*</code> and <code>/static/*</code> from a private S3 bucket via Origin Access Control —
but two things were missing: the bucket had no default icon (so the placeholder
<code>404</code>/<code>403</code>&rsquo;d), and the upload endpoint was a stub that accepted your image and quietly
threw it away.</p>
<p>Finishing it was straightforward: <code>POST /application/{id}/image</code> now parses the
multipart upload, sniffs the content type, stores the file in S3 under <code>image/</code>, and
points the application&rsquo;s <code>image</code> field at the new object. Terraform uploads a real
default icon so fresh applications have something to show. It&rsquo;s the same call the app&rsquo;s
own icon picker makes, so custom icons now &ldquo;just work&rdquo; from the phone.</p>
<h2 id="what-i-deliberately-left-out-the-web-ui">What I deliberately left out: the web UI</h2>
<p>Gotify isn&rsquo;t just a server and a phone app — it also ships a built-in web interface. It&rsquo;s
a single-page React app the server hosts at the root, and it lets you do everything from a
browser: read your message history, create and delete applications and clients, copy
tokens, upload icons, change your password, and manage plugins. I haven&rsquo;t reimplemented
any of it, and that was a conscious decision rather than an oversight.</p>
<p>The reason is that my one hard rule was about the <strong>Android app</strong>, not the web client.
Every endpoint the phone needs is implemented and wire-compatible; the web UI leans on a
handful of endpoints the app never touches — things like <code>PUT /current/user/password</code>
and the whole <code>/plugin</code> surface. Building the web client would mean serving a bundle of
static assets <em>and</em> implementing that extra slice of the API, all to duplicate
functionality I can already get with a <code>curl</code> one-liner or a quick Postman request. For a
single-user deployment that&rsquo;s just me, the cost/benefit isn&rsquo;t there.</p>
<p>So administration is &ldquo;API-first&rdquo;: I create an application with a <code>POST /application</code>, grab
the returned token, and start sending. There&rsquo;s no dashboard to log into, which honestly
suits how I use it — I provision a sender once and never think about it again. It does
mean two things worth being upfront about. First, if you point a browser at the root URL
you get an API 404, not a friendly login page. Second, anything that <em>only</em> exists in the
web UI — password changes, plugin configuration — simply isn&rsquo;t available here; the
credentials live in Terraform and SSM instead.</p>
<p>If I ever wanted the browser experience back, the path is clear: drop Gotify&rsquo;s prebuilt UI
assets into the S3 bucket, add a CloudFront behavior to serve them at the root, and fill in
the few missing endpoints. But until I actually miss it, it stays out — every feature I
don&rsquo;t build is a feature I don&rsquo;t have to secure, pay for, or maintain, which was the entire
point of going serverless in the first place.</p>
<h2 id="what-it-costs">What it costs</h2>
<p>This is the part that still makes me smile. One always-connected phone is roughly 44k
WebSocket connection-minutes a month, which is about a <strong>cent</strong>. Add some negligible
Lambda invocations, on-demand DynamoDB, and CloudFront, and the whole thing rounds to
pennies. Compared to an always-on EC2, the economics aren&rsquo;t close — and there&rsquo;s nothing
to patch.</p>
<h2 id="the-honest-limitations">The honest limitations</h2>
<p>It&rsquo;s not magic, and there are trade-offs worth naming:</p>
<ul>
<li><strong>API Gateway WebSocket connections have a 10-minute idle timeout and a hard 2-hour
maximum.</strong> The app reconnects automatically and back-fills anything it missed via
<code>GET /message?since=…</code>, so nothing is lost, but there can be a brief delay across a
reconnect. (This is also why connection records carry a TTL — dead sockets
self-clean even if a <code>$disconnect</code> is missed.)</li>
<li><strong>No true push.</strong> Real push via FCM or UnifiedPush would fix both the reconnect
window and battery usage, but it requires a modified app — which breaks my one rule.
It&rsquo;s parked in the backlog.</li>
<li><strong>Single-user by design.</strong> This replaces <em>my</em> Gotify server. Multi-user would mean
real auth, per-user data partitioning, and a lot more surface area.</li>
</ul>
<p>For my use case — get notifications from my own services to my own phone, reliably,
cheaply, with no server to babysit — it hits exactly the mark I wanted.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>The interesting parts of this project weren&rsquo;t the Lambdas or the DynamoDB schema; those
are well-trodden. The interesting parts were the seams: convincing a stock client that a
pile of managed AWS services is a single Gotify server. One hostname fronting two
incompatible API types. A CloudFront Function smoothing over a path mismatch. And a
<code>json:&quot;...,omitempty&quot;</code> tag that turned out to be the difference between &ldquo;works&rdquo; and
&ldquo;spins forever.&rdquo;</p>
<p>The whole thing is open source (MIT) and Terraform-deployable end to end:
<a href="https://github.com/mjlocat/serverless-notify"><strong>github.com/mjlocat/serverless-notify</strong></a>.
If you&rsquo;re self-hosting Gotify and tired of feeding an EC2 or VPS, it might be worth a look.</p>
]]></content:encoded></item></channel></rss>