SafeKeeping

Explainable risk scores for any Bitcoin wallet powered by machine learning.

Comment

Inspiration

Billions of dollars in illicit funds move through Bitcoin every year, from scams, ransomware and darknet markets. Anyone about to send money to an unknown wallet has no easy way to check it. We wanted to create a easy to use & quick way for people to check the risk of an unknown wallet address. Also, an automatic system that could detect high risk wallets can prevent future scams from happing.

What it does

Paste any Bitcoin address into SafeKeeping and it:

  1. Screens it against the U.S. Treasury OFAC sanctions list
  2. Pulls its recent history live from blockchain.com
  3. Computes 50 behavioral features (amounts, fees, timing, counterparties) over two-week windows
  4. Scores it 0–100 with a model trained on 265,000 labeled wallets
  5. Explains why: it compares the wallet to typical illicit and licit wallets, and Amazon Bedrock writes a plain-English summary

The labels come from wallets tied to known illicit activity (scams, malware, darknet markets) versus legitimate services (exchanges, wallet providers, miners).

How we built it

Part 1: research. On the Elliptic dataset (203k Bitcoin transactions), we compared a GraphSAGE graph neural network (PyTorch Geometric) with XGBoost, Random Forest and a hybrid of the two.

  • a temporal split: train on past weeks, test on future weeks, to prevent the model from looking into the future, as it will be unable to do in a live setting

  • class weighting for the rare illicit class

  • decision thresholds tuned only on a separate validation window

  • illicit-class F1, precision, recall and PR-AUC instead of accuracy

Random Forest reached illicit F1 0.81 (precision 0.95), consistent with the original Elliptic paper.

Part 2: real wallets. Elliptic's features are anonymized, so that model can't score a real wallet. We retrained at the wallet level on Elliptic++, which has real addresses and documented features, and rebuilt the same features from live blockchain.com data.

Stack: Amazon S3 (data, models, API cache) · Amazon SageMaker (training) · Amazon EC2 (Streamlit web app) · Amazon Bedrock (AI summaries)

Challenges we ran into

  • Train/serve mismatch: our first check comparing live-computed features with the dataset matched only 68%.

  • Concept drift: after a darknet market shutdown at time step 43, every model's performance collapses. Criminals changed their behavior, and the models trained on older patterns stopped working.

  • Rate limits: the free blockchain.com API allows about one request every 10 seconds, so we cache every lookup in S3.

  • Cloud setup: it was our first time deploying on AWS

Accomplishments that we're proud of

  • Learning how to train a machine learning model
  • Learning how to deploy a product fully powered by AWS.

What we learned

  • Accuracy can be misleading when only ~2% of transactions from the data are illicit. A model that flags nothing can still score 98%.

  • A model is only as good as the match between the features it was trained on and the features it sees live.

  • How to build and deploy on AWS: storing data, trained models and a cache in S3; training in SageMaker notebook, hosting a live web app on an EC2 server, calling Bedrock for AI summaries; and connecting with IAM roles

What's next for SafeKeeping

  • Retrain on new more recent data.
  • Extend features, automatically detect wallet addresses and alert users of risk of detected wallet without the need of manual inputs.

Limitations

The training data is from 2016–2017, only a wallet's most recent transactions are analyzed, and a risk score is an indicator for review. It will not be accurate for today behaviors, just a proof of concept.

Built With

Share this project:

Updates

Submission history