For my own stuff that's not meant for a wider audience, I sometimes use mTLS in front of my apps, alongside self-signed certs (my own CA) that shouldn't show up in certificate transparency logs.
This site also seems to be requesting a certificate from the user. Normally you probably don't want that for public facing resources.
sunaookami 5 days ago [-]
Same on Firefox
jorl17 5 days ago [-]
Same on Zen
linsomniac 5 days ago [-]
Same on Arc
embedding-shape 5 days ago [-]
Here it attempts to read my personal certificate that sits in the browser that I use for filling my taxes and do government stuff, suspicious indeed.
cmgbhm 5 days ago [-]
That’s likely just the side effect of supporting mtls. Mutual TLS came around at the same time as Microsoft did implicit network auth. Seemed magical at the time and so hare brained for eons of problems. The user side tls never caught on in most circles and still has the ancient sharp edges
mTLS supports some protocol level security guarantees that passkeys don't. Because the keys are exchanged during connection setup, there's no need for a login screen and Javascript middleware to begin the authentication process. mTLS is also easy to implement for APIs, you basically get authentication for free.
Unfortunately, browsers don't invest into making a good UI for mTLS. If browsers simply put their foot down and said "we will not permit websites to ask for a certificate if the request does not contain the proper requirements" like they do in passkeys, mTLS would be just as easy to use (and even easier to manage and rotate!).
When I ran mTLS auth on my intranet, I discovered that a lot of sites will use mTLS support to do fingerprinting, which means a lot of pages will open a blocking popup (sometimes multiple times) when I just want to read an article.
naturalmovement 5 days ago [-]
That's literally how client certificates work.
It's not attempting to "read" anything, nor is it the least bit suspicious or malicious.
Your browser was asked if it would like to present a certificate to authenticate, and you were prompted to choose one if you please. You can also hit cancel as client auth can be optional and the server will either serve you the page or a 401/403.
It's like being asked to show ID to enter a pub, you can either show one or decline, and they may or may not let you enter based on that transaction.
TurdF3rguson 5 days ago [-]
It's a little suspicious. Why are they doing something that no other website in the world does? I was curious about zero-whatever but not enough to do whatever this is.
solid_fuel 5 days ago [-]
> Why are they doing something that no other website in the world does?
Clearly other sites do since the user who shared the anecdote has certificates already configured in their browser? It's uncommon but pretty easy to understand how this happened.
Bear in mind, this is public/private key crypto so it's not like the site is asking for your facebook password or something. The site owner has no way to reuse a certificate to imitate the user.
denkmoon 5 days ago [-]
Plenty of sites do this, you just don't interact with them. Corp and govt intranets love this stuff.
TurdF3rguson 5 days ago [-]
Right, I've interacted with them when I had to for work. I wouldn't post any of them on HN though.
naturalmovement 5 days ago [-]
Bruh it's one line in nginx config.
> that no other website in the world does
That you know of. Anywhere with stringent security it's everywhere.
mook 5 days ago [-]
That's because the client certificate interface in browsers is supremely dumb. It always just lists all certificates you have, with very little context in the UI, and hopes that's good enough. I believe that's part of the reason client certificates are not poplar; having actual users deal with that is terrible, and the browsers (in practice, Chrome because of its overwhelming market share) isn't incentivized to fix it.
Avamander 5 days ago [-]
Servers can communicate their preference in terms of CAs they want. But the UX in browsers is unbelievably horrible for no good reason.
Not only is it difficult for an user to make a proper selection, it's also hard to fix a wrong one. The error pages are also terrible. There's no way for the site owner to request that when the navigation to the (auth) page fails, redirect back. Nope, no way to do error handling without some really clever iframe stuff and even then it's way too opaque.
God forbid you have to deal with CORS + mTLS.
elevation 5 days ago [-]
> God forbid you have to deal with CORS + mTLS
As someone who is about to deal with exactly this, what kind of trouble am I in for?
Avamander 5 days ago [-]
Preflight requests won't be doing mTLS on all browsers.
Yes, I agree it would be very nice to have a way to integrate ACME into zeroserve. I'm not sure if zeroserve's plugin system might allow one to add a plugin to support it?
zsoltkacsandi 5 days ago [-]
From a technical standpoint, these are always impressive projects, but I've always wondered: has anyone ever encountered a use case where the Caddy was the bottleneck?
tredre3 5 days ago [-]
In my experience Caddy has worse latency and throughput than nginx. I've set up a service that frequently sends 600MB/s (~5gbps) with nginx and the CPU is just chilling at 50%, but Caddy on that machine bottlenecks at 300MB/s despite using 100% of the CPU. AES hardware acceleration was enabled and functional on both software. This is high throughput that most people won't see, but it was also on a far beefier machine than most people would use. Caddy would definitely be a bottleneck when serving media from a raspberry pi. My last attempt was in 2025, Caddy has probably improved since then.
That being said nginx has some terrible defaults so if you're just naively benchmarking it as a proxy out of the box, you might find Caddy to be better. For example nginx caches active request bodies (in and out) to temp files in many scenarios (to block the backend/upstream as little as possible), whereas Caddy is more of a transparent proxy.
jeroenhd 5 days ago [-]
My experience mirrors that, but mostly because Caddy does HTTP/2 and http3 by default, natively. HTTP 1.1 is faster for bulk data transfers (while being worse at pipelining and latency in general), so I'm not surprised Caddy is taking more CPU in this case.
In my experience, in terms of latency, Caddy is a lot faster, every single time. I don't know what modifications I need to do to nginx to make it comparative but Caddy easily shaves half of the connection and transfer delays on my local network.
dilyevsky 5 days ago [-]
If it's http2 then Go's stdlib is pretty unoptimized to say the least. Huffman decoder is really cache unfriendly (pointer chasing) and I think allocation heavy too. Same probably goes for http1 and http3.
ksec 5 days ago [-]
I don't think Caddy being slower is a real problem. But Caddy, their developers and community refuse to believe Caddy is slow is a much bigger issue.
keynha 5 days ago [-]
For most apps the backend is slower than the proxy by a wide margin, so Caddy is nowhere near the bottleneck. Where it flips is high connection churn, since TLS handshakes are the expensive part and a flood of short lived connections without session resumption burns proxy CPU well before steady state proxying does. Very high RPS of tiny responses is the other case, where allocation and header parsing start to show.
sieabahlpark 5 days ago [-]
[dead]
BoingBoomTschak 5 days ago [-]
Interesting. Trying to get some of the performance advantages of TUX/IIS without as much insecurity makes sense for some big players, I guess.
The usual 3400 lines lock file and AGENTS.md raise some questions about the aforementioned security, though.
stymaar 5 days ago [-]
Can someome enlighten me: What's the point of “running eBPF scripts in userspace”? Isn't being run in kernel space the whole point of eBPF in the first place?
augunrik 5 days ago [-]
I am surprised how well nginx holds up?!
phillipseamore 5 days ago [-]
Why? It's one of the most optimized HTTP servers ever. Anything that claims beating nginx in benchmarks should be treated with high suspicion. I think these zeroserve numbers are likely accurate but it doesn't have the features and module ecosystem of nginx so the margins aren't worth it for me.
augunrik 5 days ago [-]
Because it passes more boundaries and stuff.
But hey, I didn’t code a Webserver so far - so what do I know. :D
AFAIK eBPF can be hardware offloaded. If you have the use case.
rciorba 5 days ago [-]
This is not in-kernel eBPF, this is a userspace eNPF runtime, so I don't see why it would pass fewer boundaries
someothherguyy 5 days ago [-]
> But hey, I didn’t code a Webserver so far - so what do I know
If you limit the scope, its worth doing and might not take as much effort as you might think. You could possibly find some enjoyment and learn a few things doing so.
solid_fuel 5 days ago [-]
In college I had a networks class where the capstone project was writing a basic HTTP server in C. It's actually shockingly easy, especially if you're only supporting get, and fetch.
Mine was something like 70 lines, and would just listen on 8080 and fork when it got a connection before checking for the requested file and sending it or a 404. I was immediately tempted to try adding something like CGI support but didn't have the time that semester.
tracker1 4 days ago [-]
Seconded, HTTP and event HTTPS to some extent isn't so bad to wireup... HTTP2/3/QUIC is another story IMO though.
miladyincontrol 5 days ago [-]
I mean, nginx dang well should? This is just an incredibly synthetic http(s)/1.1 test for what its worth.
Like you totally could turn off garbage collection for caddy especially since this is only testing incredibly short single response queries that would never need GC. Shockingly you would actually get better performance than either nginx or zeroserve, but like the uselessness of this benchmark it'd mean nothing to the real world usage of these web servers.
chucky_z 5 days ago [-]
Unfortunately that is likely untrue. Try it yourself, Go with GC disabled isn’t some magic bullet for performance as much as I’d like it to be.
jauntywundrkind 4 days ago [-]
I was very taken by the other recent zeroserve headline that showed up, zeroserve: a zero-config web server you can script with eBPF:
> The twist is that you can drop eBPF programs into the tarball and they run on every request, in userspace, as sandboxed middleware - rewriting, authenticating, and rate-limiting requests, or reverse-proxying them to a backend when you want it to act as a gateway in front of your app.
It's such opposite audiences being targeted here. eBPF is for super nerds, Caddy for "i just want a just works http server". The headline here doesn't do much for me: I don't care about Caddy compatibility and 3x doesn't mean much to me (Caddy feels like it targets friendly and easy not performance). But eBPF scripting? Ok that's cool, that's rad.
smallerize 5 days ago [-]
I still think of eBPF as not being Turing-complete. There is still a complexity limit in the verifier. Even if someone did implement Game of Life by having the program set a timer to run itself. https://isovalent.com/blog/post/ebpf-yes-its-turing-complete...
codys 5 days ago [-]
zeroserve doesn't use the Linux kernel's eBPF runtime to run the eBPF it uses, so the constraints of the Linux kernel's eBPF runtime (chosen because of how the Linux kernel thinks about protecting the Linux kernel from user space) don't apply to zeroserve (or other tools that use the eBPF instruction set but don't use the Linux kernel's particular implementation)
pbohun 5 days ago [-]
I looked into writing an http server based on iouring myself, but all the resources I could find said iouring is less safe from a cybersecurity perspective.
Is there a safe way to use iouring for a webserver, or is libuv the better way to go, even though it has less performance?
athrowaway3z 5 days ago [-]
A webserver shouldn't be calling io_uring internals without doing its own bound checking and http logic, nor should it be calling io_uring with remote-controlled crafted fd's or access pattern that might still have bugs in them.
At the level you'd be exposing io_uring (internals) via external http requests; it's security is perfectly fine.
kennethallen 5 days ago [-]
I read this post and your post introducing zeroserve. One of the main parts of the original pitch was "no separate config file and scripts". Now you're adding a config file separate from the scripts. Is the pitch now that it's like Caddy but runs eBPF scripts in userspace?
losfair 4 days ago [-]
Hi - zeroserve Caddy compat works by compiling Caddyfile to C. Technically I can keep the compiler in a separate binary/repository - but it just feels unnecessary.
zeroserve only understand eBPF at runtime. It's always the source-of-truth.
Thaxll 5 days ago [-]
Another vibe coded, dead in 6 month Rust project.
People that trully need performance are not going to use a random server that has 0 support/ track record.
ianm218 5 days ago [-]
Isn't there a chicken and egg problem where projects need to start with 0 track record? Nginx had zero track record at one point as well
chucky_z 5 days ago [-]
Nginx was developed to scale a lot of Russian sites, starting with Rambler. As a second point, Envoy was made by Lyft. A lot of web server OSS goes back to big (at the time) corps.
4ndrewl 5 days ago [-]
Never mind cooldowns for dependencies, we need cooldowns for these adhd vibe projects.
5 days ago [-]
nullstyle 5 days ago [-]
Fudge, I really need to carve out time today to play with zeroserve. Very cool stuff
ok123456 5 days ago [-]
Exposing services that use io_uring is a hard pass. It's only been a handful of weeks since the last security advisory.
bastawhiz 5 days ago [-]
The idea of jit compilation of a web server in a small project is pretty terrifying to me. The attack surface here is enormous.
And for what? My back end on a single host isn't pumping at 35k qps. If each request is 500 bytes, 35k qps is nearly 20mbps sustained with zero other io (in each direction). And this is using only two threads!
I think you'd be hard pressed to find an application where this is meaningfully useful versus just scaling horizontally. On a box that can run many threads in parallel, Caddy still vastly exceeds my ability to respond to pretty much any useful traffic. It's optimizing for a metric that wasn't a bottleneck in the first place.
10000truths 5 days ago [-]
> The idea of jit compilation of a web server in a small project is pretty terrifying to me. The attack surface here is enormous.
Does Spring Boot terrify you, then? Or Lua scripts in nginx? Or PHP? All of these use JIT compilation to run code that handles web requests.
Attack surface is a property of the JIT implementation, not of JIT itself. And eBPF is specifically designed to be very simple to implement and audit.
bastawhiz 3 days ago [-]
There's a big difference between a JIT like luajit which has lots of attention and fuzzing and one that's used by essentially nobody.
Would you compile your daily driver kernel with someone else's hobby C compiler that essentially nobody else uses?
graynk 5 days ago [-]
> Does Spring Boot terrify you, then?
It should.
cowsandmilk 5 days ago [-]
Why does jit compilation here scare you? eBPF has had a ton of research on how to limit its resource usage and how to sandbox it.
Very bizarre, never seen that before.
Thumbprints:
For my own stuff that's not meant for a wider audience, I sometimes use mTLS in front of my apps, alongside self-signed certs (my own CA) that shouldn't show up in certificate transparency logs.
This site also seems to be requesting a certificate from the user. Normally you probably don't want that for public facing resources.
https://www.passkeyprf.com/
Unfortunately, browsers don't invest into making a good UI for mTLS. If browsers simply put their foot down and said "we will not permit websites to ask for a certificate if the request does not contain the proper requirements" like they do in passkeys, mTLS would be just as easy to use (and even easier to manage and rotate!).
When I ran mTLS auth on my intranet, I discovered that a lot of sites will use mTLS support to do fingerprinting, which means a lot of pages will open a blocking popup (sometimes multiple times) when I just want to read an article.
It's not attempting to "read" anything, nor is it the least bit suspicious or malicious.
Your browser was asked if it would like to present a certificate to authenticate, and you were prompted to choose one if you please. You can also hit cancel as client auth can be optional and the server will either serve you the page or a 401/403.
It's like being asked to show ID to enter a pub, you can either show one or decline, and they may or may not let you enter based on that transaction.
Clearly other sites do since the user who shared the anecdote has certificates already configured in their browser? It's uncommon but pretty easy to understand how this happened.
Bear in mind, this is public/private key crypto so it's not like the site is asking for your facebook password or something. The site owner has no way to reuse a certificate to imitate the user.
> that no other website in the world does
That you know of. Anywhere with stringent security it's everywhere.
Not only is it difficult for an user to make a proper selection, it's also hard to fix a wrong one. The error pages are also terrible. There's no way for the site owner to request that when the navigation to the (auth) page fails, redirect back. Nope, no way to do error handling without some really clever iframe stuff and even then it's way too opaque.
God forbid you have to deal with CORS + mTLS.
As someone who is about to deal with exactly this, what kind of trouble am I in for?
https://github.com/losfair/zeroserve/blob/main/CADDY_COMPAT....
That being said nginx has some terrible defaults so if you're just naively benchmarking it as a proxy out of the box, you might find Caddy to be better. For example nginx caches active request bodies (in and out) to temp files in many scenarios (to block the backend/upstream as little as possible), whereas Caddy is more of a transparent proxy.
In my experience, in terms of latency, Caddy is a lot faster, every single time. I don't know what modifications I need to do to nginx to make it comparative but Caddy easily shaves half of the connection and transfer delays on my local network.
The usual 3400 lines lock file and AGENTS.md raise some questions about the aforementioned security, though.
AFAIK eBPF can be hardware offloaded. If you have the use case.
If you limit the scope, its worth doing and might not take as much effort as you might think. You could possibly find some enjoyment and learn a few things doing so.
Mine was something like 70 lines, and would just listen on 8080 and fork when it got a connection before checking for the requested file and sending it or a 404. I was immediately tempted to try adding something like CGI support but didn't have the time that semester.
Like you totally could turn off garbage collection for caddy especially since this is only testing incredibly short single response queries that would never need GC. Shockingly you would actually get better performance than either nginx or zeroserve, but like the uselessness of this benchmark it'd mean nothing to the real world usage of these web servers.
> The twist is that you can drop eBPF programs into the tarball and they run on every request, in userspace, as sandboxed middleware - rewriting, authenticating, and rate-limiting requests, or reverse-proxying them to a backend when you want it to act as a gateway in front of your app.
https://su3.io/posts/introducing-zeroserve https://news.ycombinator.com/item?id=48425723 (272 points, 9d ago, 66 comments)
It's such opposite audiences being targeted here. eBPF is for super nerds, Caddy for "i just want a just works http server". The headline here doesn't do much for me: I don't care about Caddy compatibility and 3x doesn't mean much to me (Caddy feels like it targets friendly and easy not performance). But eBPF scripting? Ok that's cool, that's rad.
Is there a safe way to use iouring for a webserver, or is libuv the better way to go, even though it has less performance?
At the level you'd be exposing io_uring (internals) via external http requests; it's security is perfectly fine.
zeroserve only understand eBPF at runtime. It's always the source-of-truth.
People that trully need performance are not going to use a random server that has 0 support/ track record.
And for what? My back end on a single host isn't pumping at 35k qps. If each request is 500 bytes, 35k qps is nearly 20mbps sustained with zero other io (in each direction). And this is using only two threads!
I think you'd be hard pressed to find an application where this is meaningfully useful versus just scaling horizontally. On a box that can run many threads in parallel, Caddy still vastly exceeds my ability to respond to pretty much any useful traffic. It's optimizing for a metric that wasn't a bottleneck in the first place.
Does Spring Boot terrify you, then? Or Lua scripts in nginx? Or PHP? All of these use JIT compilation to run code that handles web requests.
Attack surface is a property of the JIT implementation, not of JIT itself. And eBPF is specifically designed to be very simple to implement and audit.
Would you compile your daily driver kernel with someone else's hobby C compiler that essentially nobody else uses?
It should.