Results from my Substack Reader Study
The study’s main take-away lifted my spirits and was embarrassingly flattering. However, a deeper dive (and a little context) might support what my Spider Sense suspects.
Author’s note: As a super-easy, 1-second follow-up study, I’ve assigned readers another voluntary homework assignment. For those who accept this mission and wish to participate in this social science project, please see the bottom of this article.
***
Three days ago, I asked my readers if they would consider making a comment in my Reader Comment section after reading this particular “Covid Contrarian” article.
As I noted, this was an experiment on my part to help me gauge whether the reach of my newsletter may have been de-amplified.
For background, my prior story had generated only 11 Reader Comments, five of which I made, meaning only six “unique readers” commented on this article - which was an all-time low figure in the three years I’ve been publishing my newsletter.
The results of this study are now in and, I must say, provided a much-needed boost of sunshine (Vitamin D) as I battle the “Covid Contrarian” Substack blues.
As of the time of this writing, my article generated 194 Reader Comments - a 17.64-fold increase in Reader comments!
Furthermore, as mentioned, the comments were powerful psychological medicine. Indeed, the level of flattery and inspiring “don’t quit” sentiments made your fair-skinned content-provider turn crimson with embarrassment.
The comments were also witty, completely random and included several posts from readers who added important Covid developments I’d left off my list of significant Covid events.
This was interesting to me …
I was particularly touched by the number of readers who shared that they’d never made a comment before. Indeed, this might have been one of the more significant “findings” from this study, data that confirmed my prior hunch that, perhaps, the vast majority of Substack readers never (or rarely) make a Reader Comment.
This suggests a small army of smart and loyal readers exists in the media universe, citizens who, for their own reasons, refrain from offering their “hot takes.”
*** (Thank you to sharing this article with others. There might be other Substack authors who want to participate in similar experiments, which would give us more data to make perhaps pertinent conclusions.) ***
The real purpose of this study/experiment was to provide myself quantifiable data that might help me confirm (or reject) my hypothesis that my particular Substack might have been de-amplified.
In one sense, the results suggest that maybe my “Spider Sense” might have been wrong. Then again … maybe not.
By (almost) any measure, 194 Reader Comments would be a metric most Substack authors would crow about.
However, as I often write, context is always important and analysis of some metrics require higher-level parsing (or deeper dives).
For example, I was really looking for the number of “unique readers” who I could prove read my article (because these readers followed my specific request and made a Reader Comment).
If I subtract the 20 or so comments I made and subtract the comments from readers who made multiple comments, I’m left with approximately 160 readers who made a comment.
What’s the ratio, Kenneth?
While this figure is far more than the number of readers who have recently been making comments at my newsletter, as a percentage or ratio of my total subscribers, the number might still seem surprisingly low.
I also note that I asked every reader to consider making a comment. (That is, readers had a homework assignment with this dispatch).
Since, according to Substack metrics, I had 7,898 total subscribers when I posted this article, 160 readers who made a comment would be 1-in-49.4 of my total subscribers - just 2 percent of my (alleged) subscribers.
Expressed differently, 98 percent of my subscribers did not make a comment.
Of course, all of my subscribers didn’t open (and read this article). For this article, according to Substack metrics, my “Page View” number was 4,490. (Note: I subtracted 15 page views as those were me “checking my metrics.”)
Of these 4,490 unique readers, 2,388 were current subscribers (reflecting an Open Rate of 30.3 percent). Note: My Open Rate figure used to be 43 to 53 percent.
As a ratio of total readers, 1-in-28.06 of my readers took the time to make a comment (3.56 percent of my readers for this particular article).
Expressed differently, 96.44 percent of my readers opted out of this voluntary study.
I went back 30 months for a comparison article …
As my goal is to identify metrics that may have changed significantly, I went back through the voluminous Bill Rice archives and looked for a story from more than two years ago that had a similar number of total reads (“Page Views”). I found such an article from March 21, 2023 - which was six months after I started this newsletter or 30 months ago.
My story “Our World is being led by the Obtuse” produced:
126 Reader Comments (24 by me).
4,620 Page Views (about the same as my recent article).
Total comments (not including my comments): 102 (2.21 percent of readers made a comment way back when.)
Conclusion: A greater percentage (3.56 percent compared to 2.21 percent) and raw number (192 to 126) made a comment three days ago compared to 30 months ago.
However, with my story from March 2023, I didn’t specifically ask readers to make a comment. Still, more than 90 readers did this of their own volition.
My story from 30 months ago also produced 35 new subscribers, including two paid subscribers (compared to one new free subscriber for my last article at the time of this writing.)
Also, interestingly, the March 2023 article produced 161 “likes” compared to 102 “likes” for my last article (57.8 percent more readers hit my article with a like 30 months ago).
***
I should also highlight that when I published that story on March 21, 2023, I had only 3,135 subscribers - only 39.7 percent of the figure I have today (7,898).
So even though I had 3,863 fewer subscribers 30 months ago, the story produced more Page Views and far more “likes” (59 more) … plus 34 more new subscribers than my story from three days ago.
Even if one accepts that “subscriber over-saturation” largely explains the precipitous drop in the “new-subscriber-produced-per-article” metric, it seems to me that, among those who do read my articles, the metrics of “likes” and “reader comments” should remain, roughly, constant.
However, at least for my newsletter, I have documented a significant decline in “Reader Comments” and “likes” as well as a cliff-dive in new subscribers. (In the last two weeks, I’m averaging losing three new net subscribers with every new article).
One possible explanation would be that readers who used to routinely “hit stories with a like” or regularly made comments have now abandoned this habit … for some unknown and peculiar reason.
Comparing my Experiment to Mark Oshinskie’s Experiment …
Lastly, I should note that I plagiarized this “study idea” from my friend and Substack colleague Mark Oshinkie.
A couple of weeks ago, at the bottom of one of his stories, Mark asked readers to hit this story with a “heart” or a “like” so he could see if this improves the reach of his articles on Substack.
Mark, who always gets many “likes,” more than doubled his normal number with 390 “likes” after this request.
This was also an impressive show of support from Mark’s loyal readers, but what stood out to me is that Mark has 5,800 total subscribers - 2,100 fewer than my 7,900.
Despite having 35.6 percent fewer subscribers than me, Mark’s reader request (designed to see how many people are actually reading his articles) produced 370 responses to my 160.
Mark’s story also produced 252 Reader Comments - which he wasn’t asking for - which is more than the Reader Comments my article generated, while I was asking for them.
I’d be the first person to admit that Mark is a much better writer than myself, but, still, it strikes me as “curious” that his “likes” and “Reader Comment” metrics (with a much-smaller subscriber base) are far more impressive than my own.
This leads me to conclude that any possible algorithm operations intended to suppress the reach of certain writers may be different for different writers. (I recently ran across a reader comment from a Substack reader who also put forth this theory).
Note: See “Bonus Content” in today’s Reader Comment section for more analysis of Mark’s experiment.
In conclusion …
I was greatly touched by the flattering comments my experiment produced, but my “Spider Sense” still tells me someone, somehow, has de-amplified the reach of my Substack.
The next question would be “Why me?” which I have some theories on as well, theories I might publish in a future dispatch - an article that, per my hypothesis, probably won’t generate nearly as many new subscribers, “reads,” “likes” and Reader Comments as articles I published 30 months ago.
Today’s homework assignment:
I’m curious to see if I can surpass, or come close to, the 390 “likes” Mark recently generated with his own study/experiment.
Even if you hated this story, I’d ask my readers to consider hitting the heart/like button. I’ll wait two days and then publish the results of this double-blind study at the bottom of a future dispatch.
I also note these studies are “peer reviewed” … via the Reader Comments section. Thanks for supporting science! - BR, Jr.
BONUS CONTENT:
Regarding the 390 “likes” Mark Oshinskie got in his reader experiment, in one sense, this figure seems impressive, but, then again, maybe not.
Mark has 5,800 subscribers (a figure that, curiously, hasn’t increased in about nine months).
Mark’s homework assignment was much easier than mine. All his readers had to do was hit the heart button after reading his article.
390 likes out of 5,800 subscribers is still just 6.7 percent of his total subscribers.
Mark did tell me in an email that this story, which was cross-posted by a few fairly-well known writers including Lew Rockwell, got approximately 5,215 Page Views.
Thus, by the Page View metric, 1-in-13.4 of the people who read this particular story hit it with a like (7.47 percent of his “readers.”)
As a ratio of his total subscribers, 5,215 Page views is 89.9 percent of Mark’s total subscribers. (Mark said he usually gets a little more than 4,000 page views).
For comparison purposes, lately my stories have been producing about 4,100 Page views. My ratio of Page Views to Total Subscribers is about 51.8 percent. For the first 18 months of my newsletter, my stories almost always produced more “page views” than I had total subscribers, meaning this ratio used to be greater than 100 percent.
Given that Mark’s story was (allegedly or purportedly) read by 5,215 people and that Mark specifically asked people to hit the story with a like, I might have expected, say, 20 percent of his readers to hit this story with a like - which would have produced 1,043 “likes” (not 390).
Another take-away from these Substack studies is that very few Substack readers bother to hit a story with a like even if they like the story.
Mark’s hypothesis is/was that more likes might trigger the algorithms and beget more “reads,” which may or may not be a sound or accurate theory.
I once had the theory that more subscribers would beget more future subscribers. That theory held true for about the first 24 months of my newsletter, but no longer holds true. Today, everything I do begets FEWER future subscribers - which qualifies as a significant change - and “significant changes,” IMO, are news-worthy.
More Substack metric analysis and conjecture: How many people pay for Substack subscriptions?
****
Per my research, more than 35 million people have Substack subscriptions. Also, per my research, Substack has "three to four million" paid subscribers.
If we take the lower figure as accurate, that would mean that 8.57 percent of Substack's subscribers pay for at least one subscription.
Using my newsletter for a metric, I know that 3.43 percent of my subscribers are paid. (This is down from a high of about 4.9 percent many months ago).
I've read from reliable sources who study Substack metrics that most newsletter authors are generating a "paid ratio" of only 1 to 3 percent.
One of my main take-aways, and firm beliefs, is that about 1 percent of Substack's subscribers (about 350,000 people) are supporting multiple Substack authors.
That is, this very-generous "One percent" is often paying for 5 to 10 or more paid subscriptions. Thus, it's just a minute percentage of people who are producing those 3 to 4 million paid subscriptions.
Also, only a microscopic percentage of newsletter authors are producing a "living wage" from their writing. To me a living wage, would be about 1,000 paid subscribers.
I would venture to guess that maybe 98 percent of Substack's 75,000 authors don't have any paid subscribers or maybe just a handful.
So I don't think the "subscription model" really works for 98 percent of content providers. It's also very expensive for the tiny percentage of readers who pay for multiple subscriptions.