Ask It Simply
Build With AI · Advanced · Lesson B4

Backtest honestly and spot fake AI trading bots

about 10 minutes. Level: Builder. Nothing to install: copy-paste quiz blocks plus one Google Colab cell with sample data.

By Dontè

Education only. Not financial advice.

Education only. Not financial advice.

Watch the video

Lesson video, about 5 min 8 s. Captions are off by default. Press CC to turn them on. It starts muted. Press Sound to hear the lesson from the start. Press and hold the playing video (or hold the period key) to watch at 2x speed.
Show transcript

Welcome to Lesson B4, the last lesson in Level B. This lesson is education only, not financial advice. Today you'll learn how people fool themselves when they test trading ideas, and how scammers use AI hype to fool everyone else.

A backtest tests an idea on past prices to see how it would have done. It sounds simple. But it's easy to fool yourself. Here are three traps.

Trap one, look-ahead bias. That's using information you wouldn't have had at the time. One research paper describes it as, quote, "the use of information in a simulation that would not be available during the time period being simulated." End quote.

Trap two, survivorship bias. That's testing only on things that still exist today and forgetting the ones that failed. The same paper says this happens with data sets, quote, "that only contain funds that are in existence at the end of the sample period." End quote.

Trap three, overfitting. That's tweaking an idea until it fits the past perfectly, just by chance. Researchers writing in the Notices of the American Mathematical Society explain, quote, "Overfitting is a concept borrowed from machine learning, and denotes the situation when a model targets particular observations rather than a general structure." End quote.

Let's see trap one in action. Paste the demo block from the lesson page into Colab. We have six sample prices for a made-up stock. The first rule cheats. It says, buy today if tomorrow is higher. But you can't know tomorrow. The second rule is honest. It only uses what already happened.

Press Run. The cheating rule shows a gain of thirteen. The honest rule shows a loss of six on this same data. The cheater only looks great because it peeked at the future. Paste the code into Grok and ask it to explain why the first rule is cheating. When we tried it, Grok called it look-ahead bias. The rule peeks at tomorrow's price before it decides today.

Now, the scams. U.S. regulators have warned about this directly. The CFTC says, quote, "Fraudsters are exploiting public interest in artificial intelligence (AI) to tout automated trading algorithms, trade signal strategies, and crypto-asset trading schemes that promise unreasonably high or guaranteed returns." End quote. And, quote, "Don't believe the scammers. AI technology can't predict the future or sudden market changes." End quote.

Here's a real case from the CFTC. In the Mirror Trading International example, the operator, quote, "stole more than $1.7 billion in bitcoin from at least 23,000 people." End quote.

The SEC, NASAA and FINRA put out a joint alert on January 25, 2024. It says, quote, "Claims of high guaranteed investment returns with little or no risk are classic warning signs of fraud." End quote. Also, quote, "Investment claims that sound too good to be true usually are." End quote. And this one, quote, "It is never a good idea to make an investment decision just because someone famous says a product or service is a good investment." End quote.

So how do you check before you trust? The CFTC suggests, quote, "Research the history of the trading website by checking the age of the domain registration at lookup.icann.org." End quote. And, quote, "Get a second opinion." End quote.

Quiz time. Scam or not? Copy the six sample ads from the lesson page into Grok, and ask it to list the red flags in each one and match them to the regulator warnings. Pause the video and try it yourself first.

Here's our answer key. When we tried it, Grok flagged the same five ads. Ads one, two, four, five and six all show red flags. Guaranteed or huge returns, celebrity hype, pressure, referral bonuses, and bragging about backtests. Ad three has no red flag on its face, but our tip is to still check who runs it.

Safety check. The CFTC says, quote, "You can report fraud to the CFTC and to the FBI." End quote. Our tip: you can also report to the SEC. And never send money or crypto to anyone promising guaranteed AI returns. Education only. Not financial advice.

Quick check. What's look-ahead bias? Using information you wouldn't have had at the time. What's a classic warning sign of fraud? High guaranteed returns with little or no risk. Name one check before trusting a trading site. Domain age, or a second opinion. Congratulations, you've finished Level B. Next up, Level C, starting with blockchain basics.

What you'll learn

Step 1: What's a backtest?

A backtest tests an idea on past prices to see how it would have done. It sounds simple, but it's easy to fool yourself. Here are three traps.

Trap 1: Look-ahead bias. Using information you wouldn't have had at the time. One research paper describes it as "the use of information in a simulation that would not be available during the time period being simulated." (Daniel, Sornette and Wohrmann, arXiv 0810.1922)

Trap 2: Survivorship bias. Testing only on things that still exist today, and forgetting the ones that failed. The same paper explains this happens with data sets "that only contain funds that are in existence at the end of the sample period." (arXiv 0810.1922)

Trap 3: Overfitting. Tweaking an idea until it fits the past perfectly, by chance. Researchers writing in the Notices of the American Mathematical Society explain: "Overfitting is a concept borrowed from machine learning, and denotes the situation when a model targets particular observations rather than a general structure." (Bailey, Borwein, López de Prado and Zhu, "Pseudo-Mathematics and Financial Charlatanism," Notices of the AMS, May 2014, DOI 10.1090/noti1105; quote as worded in the authors' copy)

Step 2: See look-ahead bias (Colab, sample data)

Paste this into one Colab cell and press Run:

# SAMPLE prices for a made-up stock "DEMO". Not real data.
prices = [100, 102, 99, 104, 101, 107]

# CHEATING rule: "buy today if TOMORROW is higher" (uses the future!)
cheat = sum(prices[i+1] - prices[i] for i in range(len(prices)-1) if prices[i+1] > prices[i])

# HONEST rule: "buy today if TODAY went up from yesterday" (past data only)
honest = sum(prices[i+1] - prices[i] for i in range(1, len(prices)-1) if prices[i] > prices[i-1])

print("Cheating backtest gain:", cheat)
print("Honest backtest gain:", honest)
print("Education only. Not financial advice.")

What you should see: the cheating rule shows a gain of 13 and the honest rule shows a loss of 6 on this sample. The cheating rule only looks great because it peeks at tomorrow. Paste the code into Grok and ask, "Explain in plain words why the first rule is cheating."

Grok's real answer (our test run)

We ran the code, its output and the question through Grok (model grok-4.7) on Oct 30, 2026. Grok's answers change from run to run, so yours will differ. Quoted as Grok returned it, except where we mark [trimmed] and where we swapped dashes for commas to match our style.

The first rule is cheating because it looks at tomorrow’s price before deciding whether to buy today.

In the loop, it only “buys” when prices[i+1] > prices[i]. That means it already knows the next day’s price will be higher, so it only takes trades that are guaranteed to make money. In real life you do not know tomorrow’s price when you decide today, so a backtest that uses that future information is not a fair test of a strategy. It is look-ahead bias: the result (the gain of 13) is inflated because the rule peeks at data that would not have been available at decision time.

The honest rule only checks whether today’s price is higher than yesterday’s, then applies that to the next move. It can lose money, which is what you would expect from a rule that cannot see the future.

Education only. Not financial advice.

Step 3: Red flags of fake AI trading bots

U.S. regulators have warned about this directly.

The CFTC says: "Fraudsters are exploiting public interest in artificial intelligence (AI) to tout automated trading algorithms, trade signal strategies, and crypto-asset trading schemes that promise unreasonably high or guaranteed returns." It adds: "Don't believe the scammers. AI technology can't predict the future or sudden market changes." (CFTC Customer Advisory, Jan 25, 2024; date from CFTC press release 8854-24)

A real case: in the CFTC's Mirror Trading International example, the operator "stole more than $1.7 billion in bitcoin from at least 23,000 people." (CFTC advisory; charges announced in CFTC press release 8549-22, June 30, 2022)

The SEC, NASAA and FINRA issued a joint alert on January 25, 2024. It says: "Claims of high guaranteed investment returns with little or no risk are classic warning signs of fraud." And: "Investment claims that sound too good to be true usually are." And: "It is never a good idea to make an investment decision just because someone famous says a product or service is a good investment." (Investor.gov alert; date from FINRA)

Step 4: Check before you trust

The CFTC suggests: "Research the history of the trading website by checking the age of the domain registration at lookup.icann.org." And: "Get a second opinion." (CFTC)

Step 5: Quiz, "Scam or not?"

Copy these into Grok and ask, "For each ad, list the red flags and cite which regulator warning it matches."

Ad 1: "Our AI bot has a 100% win rate. Guaranteed 10% every month!"
Ad 2: "Famous actor uses our AI app. Join today before spots run out!"
Ad 3: "Free practice account with pretend money. No deposit needed."
Ad 4: "Send crypto to this wallet and our AI will double it in 7 days."
Ad 5: "Refer 3 friends and get bonus returns from our trading bot."
Ad 6: "Our backtest shows 900% gains. Past results guarantee future profits."

Answer key (Our tip): Ads 1, 2, 4, 5 and 6 show red flags (guaranteed or huge returns, celebrity hype, pressure, referral bonuses, and backtest bragging). Ad 3 has no red flag on its face, but you should still check who runs it.

Grok's real answer (our test run)

We ran this exact prompt through Grok (model grok-4.7) on Oct 30, 2026. Grok's answers change from run to run, so yours will differ. Quoted as Grok returned it, except where we mark [trimmed] and where we swapped dashes for commas to match our style.

Ad 1: “100% win rate. Guaranteed 10% every month.” Red flags: a promise of no losses, a fixed high return, and an “AI bot” as the reason to trust it.

Ad 2: “Famous actor uses our AI app. Join today before spots run out.” Red flags: a celebrity endorsement you cannot verify, plus artificial scarcity.

Ad 3: “Free practice account with pretend money. No deposit needed.” Red flags: none by itself. A no-deposit demo with play money is a normal feature of regulated brokers.

Ad 4: “Send crypto to this wallet and our AI will double it in 7 days.” Red flags: irreversible payment to a private wallet, a promised doubling, and a short fixed timeline.

Ad 5: “Refer 3 friends and get bonus returns from our trading bot.” Red flags: your payout depends on recruiting others, not on a real trading edge.

Ad 6: “Backtest shows 900% gains. Past results guarantee future profits.” Red flags: an extreme, unaudited backtest and an explicit guarantee drawn from the past.

[trimmed: Grok also named regulator alerts for each ad. We kept only its red flags here. For the official warnings, use the sources on this page.]

Grok flagged the same five ads as our answer key and said Ad 3 has no red flag by itself. Education only. Not financial advice.

Check yourself

  1. What is look-ahead bias? (Answer: using information you wouldn't have had at the time)
  2. What is a classic warning sign of fraud? (Answer: high guaranteed returns with little or no risk)
  3. Name one check before trusting a trading site. (Answer: domain age at lookup.icann.org, or a second opinion)

Caution box The CFTC says: "You can report fraud to the CFTC and to the FBI." (CFTC) Our tip: you can also report to the SEC. Never send money or crypto to anyone promising guaranteed AI returns. Education only. Not financial advice.

Builds on

Sources

Creator credits

Education only. Not financial advice.

Keep going

PreviousB3 →NextC1 →TrackBack to Build With AI →