Rendered at 11:00:01 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
neo2006 3 days ago [-]
Hey, we're the spinning-factory team, the folks behind Kloak.
Kloak runs as a Kubernetes controller. It swaps the secrets in your workloads for harmless placeholders we call kloaked secrets, then uses eBPF to substitute the real secrets back in at the last moment — right when your app makes a request to an allowed host.
Today, Kloak works with any app using OpenSSL 3.0–3.5 (statically or dynamically linked) or go-tls (Go 1.25 and 1.26). Support for more TLS libraries (GnuTLS, BoringSSL, and others) and additional Go versions is on the roadmap.
Kloak is open source under the AGPL, contributions are welcome! We are also happy to hear any feedback and answer any question for the HN community.
subhobroto 3 days ago [-]
This is fantastic! I need this. however, for my self-hosted home projects that are containerized but where I don't use Kubernetes, is there a way for me to use a version of Kloak that does the same eBPF magic on docker-compose or LXC/QEMU (Incus) stacks?
It's perfectly fine for you to say non-Kubernetes isn't either your focus or on your 90 day roadmap :)
NewJazz 3 days ago [-]
Yeah you might have to go talk to incus folks on how to integrate this together.. They are fairly capable, might have some good direction.
- My specific usecase is to not need Conjur Secretless Broker (https://github.com/cyberark/secretless-broker) - my understanding of eBPF is entirely superficial but from a 30k ft view, it looks like this can not only replace it but would be a far efficient solution (Conjur would be a user-space proxy while kloak would be at lower levels of abstraction)?
neo2006 3 days ago [-]
yes please open an issue on https://github.com/spinningfactory/kloak/issues and we can discuss this. I'm not familiar with secretless-broker but we can definitely see if that use case fit with kloak and get into more specifics on how you can help.
neo2006 3 days ago [-]
Thank you! We appreciate your enthusiasm! :-)
From technology perspective nothing prevent kloak to do rewrite on any workload scheduler or even without a scheduler (native Linux). The main challenge is to find a flow to signal to kloak what to rewrite and how to inject kloaked secrets to the workload.
TBH supporting other technologies is not something we thought about but we can definitely consider if there is an ask for it from the community.
subhobroto 3 days ago [-]
> The main challenge is to find a flow to signal to kloak what to rewrite and how to inject kloaked secrets to the workload
Would it be realistic or reasonable to detect a header like `X-kloak-ENABLED` or specific endpoints in the case of HTTP?
Similar for wire protocols like PostgreSQL or gRPC?
Our would a usermode proxy be easier but not preferred due to overhead?
neo2006 3 days ago [-]
The way we thought about it is from the lense of 2 personas:
- a persona that control the control plain side, what secret to distribute to which user and what hosts they are allowed to send that secret to (probably platform team or secops team)
- a persona that represent the user that need to reach host X with secret Y (probably the dev team)
based on this secret rewrite signal need to be out of band and not part of the request it self or the whole model will fall apart.
We already have the intention to support rewrites for specific headers but those headers are defined by the first persona out of band too.
btw, we support rewrite for postgres protocol for db password.
For security products trust is important. writing your website copy by hand will help you build trust. If the design and content does not look human written it will lower adoption.
neo2006 3 days ago [-]
Thank you for the feedback! We are currently shorthanded so we relied on AI a lot for writing our docs, we reviewed that doc as much as we could but definitely there is room for improvement. We will try to get better at this.
In the mean time, if you find any discruptency with the docs or anything that we can correct please open an issue and we will get to it ASAP.
koolba 3 days ago [-]
So it reads the packets and replaces the byte sequences at the kernel level? How does that work across packet boundaries?
neo2006 3 days ago [-]
Secrets are detected before encryption in the user buffer but rewrites happen post encryption in the kernel buffer to be sent on the wire.
packets boundaries are not an issue because detection happen at the SSL write where we have the full secret in the buffer and its position so we can know at rewrite time that the secret is cross 2 packets and rewrite it in 2 separate operations. We also have to update the TLS session hash at the end to not corrupt the TLS frame.
I guess we are the secrets sewers then! :D
We would love to hear what you think about it beyond the name though.
anthonyskipper 3 days ago [-]
I think it is funny that it's sewer, because a sewer is also a underground way around things, which is a good description of the out of band solution here. So the name checks out.
neo2006 3 days ago [-]
It was not intended! We were trying to make it sound like a cloak with a kubernetes K but I guess this explanation actually checkout better!
notepad0x90 2 days ago [-]
I've heard one way to acheive this is by using api proxies/gateways. you can store secrets in a vault if you wish, but with a proxy, your app makes requests as usual without using secrets, its requests are then intercepted by the proxy to add authentication information transparently.
The added benefit is that you can also manage things like api rate limits, and implement all sorts of cool monitoring and api-specific threat detection centrally. I don't know of a way to do this outside of cloud provider services though.
Architecturally speaking, you have an environment that is at the same level of trust with respect to the data it processes, anything in there is unsecured, but all interactions outside of the system passes through a gateway proxy that manages all of what i mentioned earlier, including secret management.
neo2006 2 days ago [-]
For egress proxy the app need to:
- send traffic to the proxy (either in a non transparent way or using routes or even ebpf to redirect traffic to the proxy transparently)
- trust the proxy certs or use plain http/TCP to the proxy
With kloak, the app don't need any modification and you avoid a single point of failure (aka egress proxy). Each app has an independent ebpf program attached to it that can survive the control plane going down and don't need to trust any special certs or change the endpoint it sends traffic to.
notepad0x90 2 days ago [-]
cool, but the single point of failure (it could be HA-proxy) is the point, it's a choke point. I get both architectures have pros and cons, with the proxy approach you remove secrets from the application environment entirely. Plain HTTP shouldn't be an issue, neither should internal certs whose only point is to allow applications that refuse to work with plain-http to function. I would prefer the best of both worlds, where the proxies are per-node personally.
But not everyone wants to, or can afford to run a proxy for credential management. I started looking into this mostly to regulate API usage, especially burning through tokens when calling LLM apis, the credential benefit only occurred to me afterwards. Great work with it, no idea how the eBPF magic is making it work, I'll have to find out.
neo2006 2 days ago [-]
Thank you!
I agree, each architecture have its pro and cons. If an egress gateway is available and can handle secrets it's definitely a viable solution.
erulabs 2 days ago [-]
Awesome project! We need more eBPF projects, and congrats on launching.
Assuming I hijack a production pod, can I not just make an http call to myself with the `kloak:...` secret and get back the real secret? Is there a way to validate destination?
zaronkedl 2 days ago [-]
Yes, we have host and ip filtering in place that can be used to ensure the secret is sent only for the destination we expect.
I used to run a similar company in this space. gearsec.io
My 2 cents are that no one we spoke was comfortable with us having mitm plus access to their secrets no matter how much we told we'll host it in your cloud.
The few who agreed were rigorously testing our product and asked for code SBOMs before even a pilot.
Infiscals agent vault might be the best middle ground for this kind of setups I feel sometimes
boistrous 2 days ago [-]
Thank you for the insights! We thought about this when we started working on the project and this is among the reasons we decided to go the open-source route. We think building in the open make it easier to earn users trust and give opportunities for them to audit the code and the dependencies. Also, note that with Kloak user secrets and traffic never leave the user environment as we are not a SaaS product which make security requirements more relaxed.
jackdecker 2 days ago [-]
Was just talking about this the other day - although more in-line with a custom controller to replace _all_ secrets / env variables used at runtime automatically (LD_PRELOAD get_env ?). Recognize this serves a different use case - I was trying to only decrypt KMS encrypted secrets in-memory / in-flight so that an attacker would have a harder time reading secrets in-cluster or in pod shell.
Such a sick idea, and incredibly useful. Would be nice if it integrated directly with secrets managers RE: ESO
neo2006 2 days ago [-]
Thank you!
We are planning to integrate with external secret operators, like AWS secret manager or Openboa/Vault so users can benefit from an end to end secrets protection. secret encryption/sealing at rest (through secrets managers) and protecting secrets from in-memory exfiltration attacks with kloak.
The idea is to let the ESO handle the secret at rest and delivering it to Kloak that then would continue to do the kloaked secret rewrite so the secret will only be available in a non encrypted form in Kloak. We can even push the concept further and do KMS decryption just in time to reduce the window where the secret is available.
giladd 3 days ago [-]
This is pretty cool, nice project. Can you expand on what threat model this combats?
Also, does the replace op happen only for specific fields in HTTP, or for every matching string in the request? I can imagine the latter if you want to support non-standard authentications methods, though there's always the edge case where the secret string placeholder is not used as a secret and should not be replaced.
neo2006 3 days ago [-]
The main threat model is application leaking secrets:
- Internet facing app that could potentially be hacked and bad actor exfiltrating secrets
- AI agent that can exfiltrate secrets through prompt injection for example or context poisoning
- The general use case where a secret can be for example inject by mistake in logs for instance
__turbobrew__ 2 days ago [-]
How does this compare with TPUs? Can you not have secrets in the TPU which cannot be accessed directly by apps, solving this threat vector? I get that you want compatibility with popular libraries, but I wonder if the actual solution is to use hardware support to enforce the secret boundaries.
neo2006 2 days ago [-]
I'm not super familiar with TPUs and Trusted execution environments but my understanding is that it serve a different threat model.
TEE aim to protect a certain workload from the host to avoid another workload on the same host from steeling secrets.
Kloak aim is to protect the secret from the workload itself not the host.
__turbobrew__ 2 days ago [-]
Protecting the secret from the host is a superset of protecting the secret from a workload.
boistrous 3 days ago [-]
we currently support rewrites for specific hosts and IPs and we have an open issue for supporting rewrite for specific headers for http/http2
anthonyskipper 3 days ago [-]
This is pretty awesome. Super relevant for the time because AI controlled workflows are desperate for a out of band solution like this.
The main thing I wonder is how well supported is it in cloud environements? AKS/EKS/etc?
spinningfactory 3 days ago [-]
It should work in cloud environments, We tested it on EKS and digital ocean cloud so far, and it works. The kloak controller is deployed as privileged daemonset that have access to the underlying host and can perform eBPF attachment operations on all the pods on that host.
boistrous 3 days ago [-]
gcp and aks were going to be the next ones on the testing list, but life sometimes gets in the way !
sandeepkd 3 days ago [-]
There is also a model where the applications sign the payload using the secret, AWS is a big user of that across all its API's.
neo2006 2 days ago [-]
This is not something we support currently. We will need to do some research on ways to support it.
The main hurdle is that we can't rewrite secrets in any of the user buffers as this will defy our threat model and signing is usually done in user space.
sandeepkd 2 days ago [-]
You are already doing a MITM, so some one is placing the trust in you as a intermediary. In reality the content distribution networks fronting any of the API operations have already muddied the water at this point. You are well into your rights to recalculate the signature for the payload and replace it with the secret key.
neo2006 1 days ago [-]
yes I agree and we actually already do that for TLS when rewriting secrets after encryption but my point is about the fact in our threat model we consider the app as an adversary so we don't want to use any of its buffers to rewrite secrets because it would be trivial for an adversary to reread the buffer after rewrite and get the secret. The way we overcome this is by listening to the user buffer recording all the data we need to rewrite the secret without writing anything. We go back later in the kernel buffer meant to be sent to the network and not accessible to the user app and perform the rewrite.
For API keys used to sign the request we need to do something similar which could be challenging within ebpf (maybe doable I'm not sure)
captn3m0 3 days ago [-]
You should split your controller - it is running in both the control and data planes. Idea is good though, wish you luck.
neo2006 3 days ago [-]
Thank you!
Not really, the controller is not doing dataplane per-say, it only pushes eBPF programs to the kernel for the relevant apps/cgroups so that could be considered control-plane. The full data-plane run in eBPF.
captn3m0 2 days ago [-]
1. Code that manages the Kubernetes manifests. This doesn’t need to live alongside your actual app.
2. Code that does injection in eBPF and needs to live along your app.
From my understanding from the README and helm chart, these are both in the daemonset.
neo2006 2 days ago [-]
Actually we have 2 applications along those boundaries you described. a webhook app that manage kubernetes manifest and another to inject the ebpf code and manage the ebpf maps.
Thank you for the feedback though! I think we need to clarify the doc to make that separation clear. I will open an issue for that and we will work on it.
NewJazz 3 days ago [-]
Aiui the controller is only running in the control plane, and the ebpf programs are in the data plane?
neo2006 3 days ago [-]
yes, that's right!
brady_bastian 2 days ago [-]
how does this compare to something like google secrets manager?
cassianoleal 2 days ago [-]
It probably doesn't. Google Secrets Manager is a cloud API. This runs in Kubernetes and sits between a Kubernetes Secret object and the application calling for it.
Generally speaking, if you're running Kubernetes in GCP (likely via GKE), and you control how your applications retrieve their secrets, you're likely better off with a combination of Workload Identity Federation, tight IAM to Secrets Manager, and a smart secrets retrieval strategy which likely involves lazy loading secrets and attempting a reload in case of a permission denied so it can deal with secrets rotation.
For applications where that's not an option, the state-of-the-art has been ensuring etcd is actually encrypted (as opposed to the default Base64), and relying on Kubernetes Secrets, usually either mounted in the filesystem or passed to environment variables.
Both these approaches have weaknesses since they're immediately available to all processes in the container.
OP seems to solve that by never exposing the secrets to the application, by sitting between the application and the service and replacing the secret on the wire, outside of the application's reach.
zaronkedl 2 days ago [-]
That's right, OP is the main maintainer and the idea he has is that nothing should change in the application. The application believe it has the secret, but the secret is injected on the wire AND only for the intended destination.
Please have a look at the demo if you can ; there is a webhook that abstract changing the secret resource name for you. You just "annotate" the secret resource and kloak admission controller will rewrite secrets of your deployment resource for you after that. This means the app never actually see the secret (accidental or not).
cassianoleal 2 days ago [-]
Interesting, thanks! I've also forwarded this to our security principal so we can evaluate it.
It's perfectly fine for you to say non-Kubernetes isn't either your focus or on your 90 day roadmap :)
https://discuss.linuxcontainers.org/t/how-to-best-ask-questi...
- What's the best way to discuss this specific topic with you? As an https://github.com/spinningfactory/kloak/issues or something else?
- My specific usecase is to not need Conjur Secretless Broker (https://github.com/cyberark/secretless-broker) - my understanding of eBPF is entirely superficial but from a 30k ft view, it looks like this can not only replace it but would be a far efficient solution (Conjur would be a user-space proxy while kloak would be at lower levels of abstraction)?
Would it be realistic or reasonable to detect a header like `X-kloak-ENABLED` or specific endpoints in the case of HTTP?
Similar for wire protocols like PostgreSQL or gRPC?
Our would a usermode proxy be easier but not preferred due to overhead?
based on this secret rewrite signal need to be out of band and not part of the request it self or the whole model will fall apart.
We already have the intention to support rewrites for specific headers but those headers are defined by the first persona out of band too.
btw, we support rewrite for postgres protocol for db password.
For security products trust is important. writing your website copy by hand will help you build trust. If the design and content does not look human written it will lower adoption.
packets boundaries are not an issue because detection happen at the SSL write where we have the full secret in the buffer and its position so we can know at rewrite time that the secret is cross 2 packets and rewrite it in 2 separate operations. We also have to update the TLS session hash at the end to not corrupt the TLS frame.
https://en.wikipedia.org/wiki/Cloaca_Maxima
The added benefit is that you can also manage things like api rate limits, and implement all sorts of cool monitoring and api-specific threat detection centrally. I don't know of a way to do this outside of cloud provider services though.
Architecturally speaking, you have an environment that is at the same level of trust with respect to the data it processes, anything in there is unsecured, but all interactions outside of the system passes through a gateway proxy that manages all of what i mentioned earlier, including secret management.
- send traffic to the proxy (either in a non transparent way or using routes or even ebpf to redirect traffic to the proxy transparently)
- trust the proxy certs or use plain http/TCP to the proxy
With kloak, the app don't need any modification and you avoid a single point of failure (aka egress proxy). Each app has an independent ebpf program attached to it that can survive the control plane going down and don't need to trust any special certs or change the endpoint it sends traffic to.
But not everyone wants to, or can afford to run a proxy for credential management. I started looking into this mostly to regulate API usage, especially burning through tokens when calling LLM apis, the credential benefit only occurred to me afterwards. Great work with it, no idea how the eBPF magic is making it work, I'll have to find out.
Assuming I hijack a production pod, can I not just make an http call to myself with the `kloak:...` secret and get back the real secret? Is there a way to validate destination?
It's not perfect though, see Host Filtering | https://getkloak.io/docs/guides/host-filtering.html
The few who agreed were rigorously testing our product and asked for code SBOMs before even a pilot.
Infiscals agent vault might be the best middle ground for this kind of setups I feel sometimes
Such a sick idea, and incredibly useful. Would be nice if it integrated directly with secrets managers RE: ESO
We are planning to integrate with external secret operators, like AWS secret manager or Openboa/Vault so users can benefit from an end to end secrets protection. secret encryption/sealing at rest (through secrets managers) and protecting secrets from in-memory exfiltration attacks with kloak.
The idea is to let the ESO handle the secret at rest and delivering it to Kloak that then would continue to do the kloaked secret rewrite so the secret will only be available in a non encrypted form in Kloak. We can even push the concept further and do KMS decryption just in time to reduce the window where the secret is available.
Also, does the replace op happen only for specific fields in HTTP, or for every matching string in the request? I can imagine the latter if you want to support non-standard authentications methods, though there's always the edge case where the secret string placeholder is not used as a secret and should not be replaced.
TEE aim to protect a certain workload from the host to avoid another workload on the same host from steeling secrets. Kloak aim is to protect the secret from the workload itself not the host.
The main thing I wonder is how well supported is it in cloud environements? AKS/EKS/etc?
The main hurdle is that we can't rewrite secrets in any of the user buffers as this will defy our threat model and signing is usually done in user space.
2. Code that does injection in eBPF and needs to live along your app.
From my understanding from the README and helm chart, these are both in the daemonset.
Thank you for the feedback though! I think we need to clarify the doc to make that separation clear. I will open an issue for that and we will work on it.
Generally speaking, if you're running Kubernetes in GCP (likely via GKE), and you control how your applications retrieve their secrets, you're likely better off with a combination of Workload Identity Federation, tight IAM to Secrets Manager, and a smart secrets retrieval strategy which likely involves lazy loading secrets and attempting a reload in case of a permission denied so it can deal with secrets rotation.
For applications where that's not an option, the state-of-the-art has been ensuring etcd is actually encrypted (as opposed to the default Base64), and relying on Kubernetes Secrets, usually either mounted in the filesystem or passed to environment variables.
Both these approaches have weaknesses since they're immediately available to all processes in the container.
OP seems to solve that by never exposing the secrets to the application, by sitting between the application and the service and replacing the secret on the wire, outside of the application's reach.
Please have a look at the demo if you can ; there is a webhook that abstract changing the secret resource name for you. You just "annotate" the secret resource and kloak admission controller will rewrite secrets of your deployment resource for you after that. This means the app never actually see the secret (accidental or not).