STUMP - Meep on public finance, pensions, mortality and more
STUMP - Death and Taxes
Bad Metrics Lead to Bad Results
0:00
Current time: 0:00 / Total time: -33:28
-33:28

Bad Metrics Lead to Bad Results

There is no shortcut or algorithm to replace good judgment. Take responsibility.

Whatever you call it: Campbell’s Law (no relation), Goodhart’s Law, or the cobra effect — when you use measures that are merely cheap stand-ins for a much more difficult result you want to get at, people will start gaming that measure. And you will often get a result you really don’t like.

[later] I'm pleased to report we're now identifying and replacing hundreds of outdated metrics per hour.

Share


Episode Links

Wells Fargo News

Bloomberg, Matt Levine: Wells Fargo Mouse Jigglers

In 2016, Wells Fargo & Co. got in trouble for opening fake accounts. What happened was that Wells Fargo’s senior management had decided that it wanted its branch employees to cross-sell products, to push each checking customer to open up a savings account and a credit card and sign up for online banking and maybe get a mortgage, because this would deepen the bank’s relationship with the customer and ultimately lead to more revenue. But instead of hiring and training employees who would holistically assess each customer’s needs and suggest suitable products, Wells Fargo had “strict quotas regulating the number of daily ‘solutions’ that its bankers must reach,” and its managers would “constantly hound, berate, demean and threaten employees to meet these unreachable quotas.”

….

Ahahaha come on. You want a lot of mouse movement, you get a lot of mouse movement, but in a bad way. Imagine deciding how to measure and manage the productivity and value added of your wealth and investment management employees while they are working from home. What might you measure?

….

Which of those do you think is the best proxy for, like, contributions to Wells Fargo’s return on equity? Which is the simplest to measure? Which is the simplest to game?

CNN Business: Wells Fargo fired a dozen people accused of faking keyboard strokes

See here: Wells Fargo this week disclosed that it had fired more than a dozen employees for “simulation of keyboard activity,” Bloomberg reported, citing filings to the Financial Industry Regulatory Authority. CNN confirmed that multiple people were let go after a review of allegations that they created an “impression of active work.”

In other words, they were faking work, perhaps with the kind of mouse jiggler that you can buy online for $20.

Those devices — which keep your screen active and move your cursor in convincingly random ways — took off during the early days of the pandemic. With employees no longer huddled together under fluorescent lighting, eating sad desk salads, bosses suddenly had to wonder whether their teams were actually working or slacking off.

Even though most workers said they were more productive from home, many executives  adopted “bossware” to monitor their staff’s laptops. (And to be fair, yes — sometimes we did step away, selfishly tending to our own personal business, like walking the dog or staring out the window while contemplating our mortality. We hope you can forgive us.)

….

But firing people over mouse movers may not be the best way to foster a culture of trust and inclusion.

“Managers often assume the worst when they see someone’s away, and so they’re looking for any type of data to show that that’s true,” Herd says. “So, team members are going to innovate around that.”

Mashable: Wells Fargo reportedly fired people for alleged 'simulation of keyboard activity'

There are many ways to fake being online while working, including the use of gadgets that imitate computer activity, or "mouse jigglers." Mouse jigglers are pretty easy to get; they're selling on Amazon for under £10 right now. They're mechanical devices that physically move your mouse around to prevent your computer from going into sleep mode. TikTokkers have been recommending these devices for years, while folks on Reddit have shared horror stories of being caught by their managers using them.

It's unclear how the company actually figured out staff were allegedly undertaking "simulation of keyboard activity" at all. An increasing number of companies are surveilling employees since the COVID-19 pandemic prompted the rise of working from home. Some companies have installed keylogger software on their computers to recorded characters typed, and biometric monitoring is on the rise, despite privacy concerns and employee backlash.

A 2021 study by Express VPN found 78 percent of employers engage in remote work surveillance, with 73 percent of employers using email, calls, messages, or videos to inform performance reviews — yes, your boss can read your Gmail drafts (and that's not all) — and 46 percent using it monitor the potential formation of workers' unions.

But as Jack Morse writes for Mashable, "While your boss monitoring your every move is definitely creepy, it's perfectly legal."

Reddit: r/news Wells Fargo fired a dozen people accused of faking keyboard strokes

The Various “Laws”

Wikipedia: Goodhart’s Law

Goodhart's law is an adage often stated as, "When a measure becomes a target, it ceases to be a good measure".[1] It is named after British economist Charles Goodhart, who is credited with expressing the core idea of the adage in a 1975 article on monetary policy in the United Kingdom:[2]

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.[3]

Campbell’s Law

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.[1]

….

In 1976, Campbell wrote: "Achievement tests may well be valuable indicators of general school achievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways. (Similar biases of course surround the use of objective tests in courses or as entrance examinations.)"[1]

Campbell's Law: Something Every Educator Should Know

The cobra effect

The term cobra effect was coined by economist Horst Siebert based on an anecdotal occurrence in India during British rule.[2][3] The British government, concerned about the number of venomous cobras in Delhi, offered a bounty for every dead cobra. Initially, this was a successful strategy; large numbers of snakes were killed for the reward. Eventually, however, people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped. When cobra breeders set their snakes free, the wild cobra population further increased.[4] This story is often cited as an example of Goodhart's law or Campbell's law.[5]

Check out the talk page:

This was told by my dad, who happened to work in the central planification ministry during early socialist Czechoslovakia (until he managed to escape).

Czechoslovakian crystal glass is famous worldwide, and thus chandeliers were produced, for export to get foreign currency. Problem is, there is a limited market for luxury chandeliers, which are purchased by the unit, some of them bespoke, all of them handmade, certainly not from one single design. Thus, worker productivity (very important in socialist countries, for any kind of bonus) cannot be measured by the unit, so let's measure the total weight produced... As a result, Czech crystal chandeliers grew to be so heavy, that clients were having engineering issues when trying to install those. YamaPlos talk 23:50, 28 February 2024 (UTC)[reply]

Winner’s Curse

The winner's curse is a phenomenon that may occur in common value auctions, where all bidders have the same (ex post) value for an item but receive different private (ex ante) signals about this value and wherein the winner is the bidder with the most optimistic evaluation of the asset and therefore will tend to overestimate and overpay. Accordingly, the winner will be "cursed" in one of two ways: either the winning bid will exceed the value of the auctioned asset making the winner worse off in absolute terms, or the value of the asset will be less than the bidder anticipated, so the bidder may garner a net gain but will be worse off than anticipated.[1][2]

Fake Philosophy Journals

Retraction Watch: How a widely used ranking system ended up with three fake journals in its top 10 philosophy list

We checked the Scopus philosophy list and discovered three journals published by Addleton Academic Publishers – which we had never heard of – are in the top 10 of the 2023 CiteScore ranking: Linguistic and Philosophical Investigations (3rd on the list of 806 philosophy journals indexed by Scopus in 2023), Review of Contemporary Philosophy (5/806), and Analysis and Metaphysics (6/806). All three also are in the top 100 of the 2023 SJR ranking. 

….

How was it possible to get into the Scopus top 10 in philosophy? The trick is simple: The Addleton journals extensively cross-cite each other. For example, of 541 citations to Linguistic and Philosophical Investigations used to calculate the 2023 CiteScore, 208 come from journals published by Addleton. Additional citations come mostly from Frontiers and MDPI journals.

Predatory publishing in Scopus: Evidence on cross-country differences

Abstract:

Predatory publishing represents a major challenge to scholarly communication. This paper maps the infiltration of journals suspected of predatory practices into the citation database Scopus and examines cross-country differences in the propensity of scholars to publish in such journals. Using the names of “potential, possible, or probable” predatory journals and publishers on Beall’s lists, we derived the ISSNs of 3,293 journals from Ulrichsweb and searched Scopus with them. A total of 324 of journals that appear in both Beall’s lists and Scopus, with 164,000 articles published during 2015–2017 were identified. Analysis of data for 172 countries in four fields of research indicates that there is a remarkable heterogeneity. In the most affected countries, including Kazakhstan and Indonesia, around 17% of articles were published in the suspected predatory journals, while some other countries have no articles in this category whatsoever. Countries with large research sectors at the medium level of economic development, especially in Asia and North Africa, tend to be most susceptible to predatory publishing. Policy makers and stakeholders in these and other developing countries need to pay more attention to the quality of research evaluation.

Quantitative Science Studies (2022) 3 (3): 859–887.

https://doi.org/10.1162/qss_a_00213

Share

STUMP - Meep on public finance, pensions, mortality and more is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Discussion about this podcast

STUMP - Meep on public finance, pensions, mortality and more
STUMP - Death and Taxes
Meep (Mary Pat Campbell) talks about mortality trends and/or public finance issues, usually with a connection to current events.