top of page
Search

Random Python Scripts #3: Dividend Hunting

This is the post I started to write that then lead to dividendhunting.com. I have considered to rewrite it but finally I decided to leave it as it was originally. Thank you for reading.



How to earn some cash with Python

Ah, money, now things are getting juicy. So I heard you have some cash, you want to make more cash with it, you are unsure, too many options, Stocks, ETF, investing, trading, Crypto, dividends, Bonds... euromillions? what is best? Well, if I knew, trust me, I wouldn't be writing this post. Same rule applies to the people who write on the big magazines like the Financial Times or Zacks, if they really had the answer they wouldn't be spending their time writing about it.

There are many ways to make money with your money, probably the most secure and guaranteed way are bonds, but they are a bit boring, low returns... a bit more aggressive option while also secure is any random S&P500 ETF, whether you like it or not, market will always find its way to go up. If you are even a bit interested in finances, I'm 99% sure you've seen dozens of posts like: How much money would you have now if you had invested 100€ a month in the S&P 500 for the past 10 years? Yeah.. thanks, I also know I should have bought real estate in the nineties instead of just playing the ball in the playground.


So, what is this about? Is about a trading concept, I believe the technical name is not Dividend Hunting, but I like how it sounds. The official name is something like Dividend Capture. You can say whatever you want, but hunting is more catchy. In any case, let's start from the beginning.



What are dividends?


I am a total dumb about Stocks and trading, I know absolutely nothing, just scratch the surface. Luckily, dividends are easy to understand, as the rules are pretty simple. The concept is:

  • Some Companies trade in the market, some not.

  • Some Companies offer dividends, some not.

  • Those who trade in the market and offer dividends, often do it every 3-4 months, others every 6 or 12 months, and a few others every month (and even every week!).

  • There is a date, called ex-dividend date, which is the date the Stock trades without the dividend. You need to have a share in your possession to be eligible for dividend payment. You must have the share at least the day before! (This is important)

  • There is another date, called dividend pay day, but that doesn't matter. You can sell the share on the same ex-dividend date, and still get the dividend. What matters is if you are "on the book" at the moment of the market closure before the ex-dividend date.

  • The share value will drop by the same amount of the dividend payment. Basically, means that if you have a share that is valued at 5€ and the company pays 0.6€ in dividends, you will end up with 1 share in your possession valued at 4.4€ and 0.6€ in cash your pocket.

  • And that's it.



What is Dividend Hunting?


I guess you can imagine. The idea behind is to buy a stock close to its ex-dividend date, wait until the stock price recovers (so you don't lose money after buying at 5€ and selling at a lower value), sell, find another stock, and rinse and repeat. In the meantime you will receive the dividends on your account.

Now, doing this "manually" can take a lot of your time. There are many amazing pages around where you can use search and filters, some are free, some premium, but most of the time those pages miss something. My favorite is Trading View, probably the most popular around. Super powerful charts, very nice search engine. Still, as most of the tools around, falls short for Dividend Hunting. There are other pages that focus on dividends of course, but, just for the science, we will explore it a bit by ourselves.


What is the plan?

This is important. A good old manager told me: Don't start typing a freaking line of code until you have a design. So, what is our plan, our design?


  1. Get a list of stocks

Seems trivial, but is not. Currently there are over 100.000 instruments in the world that trade in the stock market. That is a lot. But the main problem is that there is no broker that allows you to trade with all of them. Interactive Brokers may be the most extended, but still doesn't cover all, others like eToro has access to over 6000 instruments, and Revolut to over 4000. And this is a problem, because getting data is difficult, and you don't want to spend time to find out that the best stock to buy is not available on your platform. So, you need to get a list of stocks available on your broker. Or swap brokers.

  • Interactive Brokers has an API, and it even has a python library: IB-insync

  • eToro doesn't offer API or clean list of stocks. You would have to find your ways scrapping

  • Revolut also doesn't offer API for trading, so scrapping is also the only way


If you don't care much about the broker and you just want a clean list of stocks, it doesn't get much easier. But there are options. Easily scrappable options:

  • Kaggle datasets. Think that many people before you faced this same problem, and some of them did the work and were nice enough to share the results. Even if you don't care about the rest of the data in the dataset, at least you have a clean list of instruments

  • S&P500 Wiki. Updated daily. Easy to scrap. Will put some snippet later

  • Nasdaq listing. CSV file. Why not everyone does the same?

  • Trading View. My favorite platform. Still, not scrapping friendly. Maybe the premium version offers something else, but I never tried it.


I am a Revolut user. I know, is probably the worst broker around in terms of fees. But, with a basic premium subscription, you get 5 to 10 free trades a month, and this Dividend Hunting doesn't require much more than that. So, is basically free. Long story short, I had to scrap my way around to get a list of stocks. We will get to it later, and also later will show easy examples to scrap the S&P500 wiki, Nasdaq even trading view main exchanges. For now, let's continue with the design. Assume we have a clean list of stocks, now what?


  1. Get stock data

Easy to say. What options are there to get information on every stock? Because we will need to get plenty of data: dividends to get started, but also history price, dividends ex-date and some other.

  • Interactive Brokers library has tickers for each symbol

  • Yahoo Finance also has a python library: yfinance, and is quite powerful. It has a rate limit if you try to run it in parallel, but if you behave and go one by one, it never complains.

  • Other platforms offer premium REST API to get this data.


You are free to explore options, but spoiler alert, today, in this post, yfinance will be the MVP. Let's move on. What data we need?


  1. Define what we need

What is going to be the logic? Let's start with 1 symbol.


Does the stock have dividends? if no, return

When is the next/last announced dividend ex-date?

What is the next/last dividend amount per share?

What is current stock price?

What is current analyst rating?

Is the stock healthy or nosediving?

And the million dollar question: how fast does the stock recover its price? if ever?

And the billion dollar question: how anxious will I get if the stock doesn't recover the price as fast as I want?


Luckily, yfinance can answer all the above. Not all of them directly, but it gives you the tools. You just need to know what is your goal and squeeze your brains a bit.


  1. Define how to display the data

As a CSV engineer expert, getting the data is not as important as knowing how to display it. Over the years I have created hundreds or thousands of reports and datasets in CSV. Seems trivial, but is critical to know how to organize, clean and sanitize the data, make it easy to filter and read, otherwise the consumers will get lost in a snap. So, we will have a lot of information about each stock, we can display all we want, but we will be interested in:

  • stock symbol (duh!)

  • ex-dividend date

  • dividend yield

  • dividend amount

  • earnings from dividends with your capital

  • recovery days / rate (as not always recover)

  • analyst rating

You can also calculate the 50/200 averages, the 52 week movement percentage... I mean, we are talking about stocks, go as crazy as you want. At the end of the day doesn't matter what you see on the spreadsheet, if your guts tell you not to buy, you will not buy. And then the stock will skyrocket, for sure.


Ok, that sounds like a plan. Is this not yet another ETL pipeline? Well, and what is not, right? Let's review the scrappers.


Extraction - Revolut


As Revolut user and data science rookie, I tried for some time to get a clean list of instruments available on Revolut, but it wasn't possible. Their mobile app was pretty locked down, their API doesn't give access to trading, and only recently they have published a trading website. So, you can login, and you have a nice interface with the list of all the symbols you can trade with your account.



That just screams SELENIUM. I will publish some other day a deep guide on how to use Selenium. I promise to include the fancy stuff, like in this case where I had to scroll one container over and over and over again to extract 10 symbols every time.


Extraction - S&P500


The S&P500 Wikipedia page with list of companies is much easier to scrap. You open it, you see a big table with the company name and symbol. You open the inspect on the browser and that's it. Now, you don't need no Selenium here, Pandas and BeautifulSoup can get you over the line with a few lines.

  • load page

  • extract table

  • loop over rows

from io import StringIO
import requests
from bs4 import BeautifulSoup
import pandas as pd


SP500_URL = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'


def extract_sp500():
	response = requests.get(SP500_URL, timeout=10)
	soup = BeautifulSoup(response.text, features='lxml')
	tables = pd.read_html(StringIO(str(soup)))
	df = tables[0]  # First table in the page contains the symbols
	symbols = df['Symbol'].tolist()
	return symbols  # ['MMM', 'AOS', 'ABT', 'ABBV', ...]

Extraction - Nasdaq


Even easier, they offer a CSV file with all their companies. Pandas alone can handle this.


import pandas as pd


NASDAQ_URL = 'https://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt'


def extract_nasdaq():
	df = pd.read_csv(NASDAQ_URL, sep='|')
	df = df[:-1]  # Last line contains metadata
	df = df[df['Test Issue'] != 'Y']  # There are some test items around
	symbols = df['Symbol'].tolist()
	return symbols  # ['AACB', 'AACBR', ...]

Extraction - Trading View


Here you can find all exchanges, each with their companies list. The downside, the initial list of instruments is limited to 100, to get more you need to scroll (selenium, yeah). But, this time we will not scroll. The exchanges we are interested in don't have more than 100 companies. Like the Euro Stoxx 50 or the Dow Jones. The good thing, the way to extract any exchange is the same. So, if you prefer, you could extend this to do it with selenium with some scroll to extract every single exchange from Trading View. But to keep things simple, this is a quick way to get the first 100 items:

import requests
from bs4 import BeautifulSoup


DOW_JONES = 'https://www.tradingview.com/symbols/DJ-DJI/components/'
EURO_STOXX_50 = 'https://www.tradingview.com/symbols/TVC-SX5E/components/'


def scrape_tradingview_generic(url: str) -> list:
	"""
	Generic method to extract Symbols from TradingView component pages.
	:param url: TradingView URL for index at the components tab
	:return: list of symbols
	"""
	symbols = []
	response = requests.get(url, timeout=10)

	soup = BeautifulSoup(response.content, 'html.parser')

	# Look for table rows with stock data
	rows = soup.find_all('tr')
	for row in rows:
		if row.get('data-rowkey'):
			market, symbol = row['data-rowkey'].split(':')
			# Nikkei symbols are numeric only
			# they need ".T" as suffix to find them in Yahoo Finance
			if symbol.isdigit():  
				symbol = symbol + '.T'
			symbols.append(symbol)
	return symbols

Perfect, one way or another you have a list of stocks in hand. Now, what?


Extraction - Yahoo Finance data


Now you need to query every stock, one... by... one...


pip3 install yfinance
import yfinance as yf


SYMBOLS = ['TSLA', 'T', 'KO']
CAPITAL = 3000


def get_ticker_data(symbol: str, target_date: str, days: int) -> dict:
	"""
	Function to get some dividends insights on a symbol
	And find those with Dividends ex-date in the [days] after [target_date]
	:param symbol: Stock Symbol like TSLA, AAPL, KO, etc
	:param target_date: Date to filter out dividends. Format YYYY-MM-DD
	:param days: Days to search for ex-dates after target_date
	"""
	ticker = yf.Ticker(symbol)

	# find out if the symbol has dividends. TSLA doesnt, T & KO do
	dividends = ticker.dividends
	if not dividends or len(dividends) == 0:
		return

	# bunch of metadata from the symbol
	info = ticket.info

	dividend_yield = info['dividendYield']
	
	# some symbols have their ex-date in the info section as timestamp
	ex_date = info.get('exDividendDate')

	# others on the calendar as datetime
	ex_date = ex_date or ticker.calendar.get('Ex-Dividend Date')

	if target_date and ex_date:
		if not is_date_in_range(target_date, ex_date, days):
			return {}  # Skip if ex_date out of range

	current_price = info.get('regularMarketPrice')
	last_div = info.get('lastDividendValue', 0)
	dividend_rate = info.get('dividendRate', 0)
	shares_to_buy = round(CAPITAL / current_price, 2)

	return {
		'symbol': symbol,
		'company': info.get('longName', symbol)[:40],
		'sector': info.get('sector', 'Unknown')[:20],
		'exchange': info.get('exchange', 'Unknown'),
		'price_EUR': current_price,
		'Dividend_Yield_%': dividend_yield,
		'Shares_to_Buy': shares_to_buy,
		'Gain': round(shares_to_buy * last_div, 2),
		'Ex_Div_Date': ex_date,
		'Recommendation': info.get('recommendationKey')
    }
	

def is_date_in_range(target_date: str, ex_date: datetime, days: int):
	target_date = datetime.strptime(target_date_str, '%Y-%m-%d').date()
	start_window = target_date - timedelta(days=0)
	end_window = target_date + timedelta(days=window_days)

    if isinstance(ex_date, (int, float)):
        # Handle timestamp
        ex_date = datetime.fromtimestamp(ex_div_date).date()
    elif hasattr(ex_div_date, 'date'):
        # Handle datetime object
        ex_date = ex_date.date()
    else:
        # Handle date object
        pass

    # Check if dividend ex_date in range
    return start_window <= ex_date <= end_window

As I mentioned earlier Yahoo finance is the MVP. If you know what you need, you can get it.

Now, that script, ran on October 1st, returned this beautiful table


Symbol

Company

Sector

Exchange

Price_EUR

Dividend_Yield_%

Shares_to_Buy

Gain

Ex_Div_Date

Recommendation

IVR

Invesco Mortgage Capital

Real Estate

NYQ

7.67

19.03

390.9

132.9

2025-10-06

2.8 - Hold

TWO

Two Harbors Investment Co

Real Estate

NYQ

10.06

16.57

298.2

116.3

2025-10-03

2.3 - Buy

CIG

Companhia Energética de M

Utilities

NYQ

2.06

11.02

1456.3

55.3

2025-10-06


MGIC

Magic Software Enterprise

Technology

NMS

19.91

3.08

150.7

49.3

2025-10-06


EIX

Edison International

Utilities

NYQ

56.03

6.02

53.5

44.3

2025-10-07

1.9 - Buy

FNLC

The First Bancorp, Inc.

Financial Servi

NMS

25.82

5.52

116.2

43.0

2025-10-06


BTI

British American Tobacco

Consumer Defens

NYQ

52.76

5.84

56.9

42.6

2025-10-03


DBI

Designer Brands Inc.

Consumer Cyclic

NYQ

3.54

5.31

848.7

42.4

2025-10-03


BMY

Bristol-Myers Squibb Comp

Healthcare

NYQ

47.16

5.62

63.6

39.4

2025-10-03

2.7 - Hold

UBFO

United Security Bancshare

Financial Servi

NMS

9.36

5.01

320.7

38.5

2025-10-03


SNN

Smith & Nephew plc

Healthcare

NYQ

36.23

2.1

82.8

38.3

2025-10-03


BNS

The Bank of Nova Scotia

Financial Servi

NYQ

64.64

4.92

46.4

37.4

2025-10-07


CPB

The Campbell's Company

Consumer Defens

NMS

31.54

4.86

95.1

37.1

2025-10-02

3.0 - Hold

PAGS

PagSeguro Digital Ltd.

Technology

NYQ

9.74

1.39

308.0

37.0

2025-10-06

1.9 - Buy

BRX

Brixmor Property Group In

Real Estate

NYQ

27.46

4.18

109.2

31.5

2025-10-02

1.5 - Strong Buy

SAR

Saratoga Investment Corp.

Financial Servi

NYQ

24.26

12.31

123.7

30.9

2025-10-07


SLVM

Sylvamo Corporation

Basic Materials

NYQ

44.27

4.18

67.8

30.5

2025-10-03

2.7 - Hold

HBNC

Horizon Bancorp, Inc.

Financial Servi

NMS

15.94

3.95

188.2

30.1

2025-10-03

1.8 - Buy

NECB

Northeast Community Banco

Financial Servi

NCM

20.6

3.83

145.6

29.1

2025-10-03

3.0 - Hold

OGE

OGE Energy Corp.

Utilities

NYQ

46.22

3.72

64.9

27.3

2025-10-06


WLY

John Wiley & Sons, Inc.

Communication S

NYQ

39.99

3.56

75.0

26.6

2025-10-07


WLYB

John Wiley & Sons, Inc.

Communication S

NYQ

40.69

3.55

73.7

26.2

2025-10-07


PKBK

Parke Bancorp, Inc.

Financial Servi

NCM

21.28

3.26

141.0

25.4

2025-10-03


PFBC

Preferred Bank

Financial Servi

NMS

90.02

3.28

33.3

25.0

2025-10-07

2.4 - Buy

PM

Philip Morris Internation

Consumer Defens

NYQ

163.45

3.57

18.4

24.8

2025-10-03

1.8 - Buy

WNC

Wabash National Corporati

Industrials

NYQ

9.84

3.13

304.9

24.4

2025-10-02

2.0 - Buy

SCVL

Shoe Carnival, Inc.

Consumer Cyclic

NMS

20.64

2.79

145.3

21.8

2025-10-06


GBCI

Glacier Bancorp, Inc.

Financial Servi

NYQ

48.38

2.64

62.0

20.5

2025-10-07

1.8 - Buy

VLGEA

Village Super Market, Inc

Consumer Defens

NMS

37.1

2.64

80.9

20.2

2025-10-02


SYY

Sysco Corporation

Consumer Defens

NYQ

82.57

2.64

36.3

19.6

2025-10-03

2.0 - Buy

CSCO

Cisco Systems, Inc.

Technology

NMS

68.22

2.44

44.0

18.0

2025-10-03

2.1 - Buy

DG

Dollar General Corporatio

Consumer Defens

NYQ

99.83

2.31

30.1

17.7

2025-10-07

2.4 - Buy

SCS

Steelcase Inc.

Consumer Cyclic

NYQ

16.98

2.34

176.7

17.7

2025-10-06

2.0 - Buy

ABM

ABM Industries Incorporat

Industrials

NYQ

46.22

2.32

64.9

17.2

2025-10-02


MTCH

Match Group, Inc.

Communication S

NMS

34.8

2.1

86.2

16.4

2025-10-03

2.4 - Buy

WERN

Werner Enterprises, Inc.

Industrials

NMS

25.98

2.13

115.5

16.2

2025-10-06

3.2 - Hold

TIMB

TIM S.A.

Communication S

NYQ

22.13

7.62

135.6

16.1

2025-10-03

2.6 - Hold

LMNR

Limoneira Company

Consumer Defens

NMS

14.77

1.99

203.1

15.2

2025-10-06


TTC

The Toro Company

Industrials

NYQ

76.25

1.99

39.3

14.9

2025-10-07


JPM

JPMorgan Chase & Co.

Financial Servi

NYQ

311.99

1.9

9.6

13.5

2025-10-06

2.3 - Buy

MMC

Marsh & McLennan Companie

Financial Servi

NYQ

201.37

1.8

14.9

13.4

2025-10-02

2.7 - Hold

NTAP

NetApp, Inc.

Technology

NMS

118.29

1.74

25.4

13.2

2025-10-03

2.5 - Buy

DGX

Quest Diagnostics Incorpo

Healthcare

NYQ

183.71

1.69

16.3

13.1

2025-10-03

2.2 - Buy

ERIE

Erie Indemnity Company

Financial Servi

NMS

324.05

1.73

9.3

12.6

2025-10-06


ACU

Acme United Corporation

Consumer Defens

ASE

40.92

1.54

73.3

11.7

2025-10-03


RSG

Republic Services, Inc.

Industrials

NYQ

227.39

1.1

13.2

7.7

2025-10-02

2.0 - Buy

PGR

The Progressive Corporati

Financial Servi

NYQ

248.02

2.01

12.1

1.2

2025-10-02

2.4 - Buy


Not bad. Now, this is raw data. There is still a lot more you can/should do before putting your money in a stock, including the million dollar questions I mentioned earlier. That is the real juice.


-----


That was the post, and then I jump directly into PyCharm and started to work on the design for dividendhunting.com. Can't wait to share more details on the real implementation and the problems I found along the way. Feel free to drop a message or comment below.


 
 
 

Comments

bottom of page