Security Incident Postmortem: 2023-07-15

Polkachu Team | 2023-07-15

In the past 24 hours, we have been working with a white-hat (W thereafter) to resolve a security incident. The security vulnerability has been fixed and a bounty has been paid. While we do not have a formal bounty program, we take security seriously and encourage W and other white-hats to help us improve in the future.

What Happened?

Our Loki HTTP API was open to public without protection. W was able to identify it and alerted us through a private chat. Before the patch was applied, W was able to access logs of our nodes and our website. While the logs contain no cryptographic key information or user information, they do contain sensitive information to our business operations.

We are glad that W reached out to us first. Through a private chat, we worked out a plan: First, we would mitigate the vulnerability and W would confirm the fix. Second, we would pay a bounty for W's work and W would destroy any information obtained through the white-hat operation. Finally, we would jointly release this postmortem analysis. While W's name is not on this post, he/she has fully reviewed and approved this post. We respect W's desire to stay anonymous.

How We Have Mitigated?

We were notified by W on Saturday morning and our team worked throughout Saturday to mitigate the issue. The direct mitigations are:

1. Locked down Loki HTTP API with NGINX-based authentication
2. Locked down a backdoor IP-based access to the same endpoints
3. Stopped sending polkachu.com's log to the Loki server

Besides the direction mitigations, we also took this opportunity to review all our security policies and practices.

1. Reviewed all node_exporter Prometheus endpoints to ensure they are private
2. Reviewed all Tendermint Prometheus endpoints to ensure they are private
3. Reviewed whitelisted IPs on all servers, deleted obsolete ones and made clear comments on the ones we keep
4. Updated several of our open-sourced repos with the security fixes that would have prevented this incident. If you forked our report to deploy your servers, please check today's commits. Feel free to reach out to us if you have questions and feedback.

What We Have Learned?

We appreciate W's professionalism throughout the process. This is our first time working with a white-hat and we hope this is the last. In case not, we encourage W and other white-hats help us improve. While we do not have a formal bounty program, we take security seriously and will pay for findings like this.

We would also like to pay it forward to the community from our learnings. If you have forked our Ansible repos (especially this one) to deploy nodes, you might be vulnerable. Check the commits from today to learn about the mitigations and feel free to reach out to us.

Security is a never-ending journey and a team effort. By working together, we stay stronger together. 💪