The thing with algorithms is that they expose whatever bias is in the data. We might hide it or not like it. The algorithm does not care and exposes it.
The results is something that has logic and value at the surface, but once you make a deep dive or start to roll it out, it’s fundamentally broken. You could even call it badly designed.
As we start to roll out more of this partly designed, partly self designed systems to larger problems and bigger audiences.
The amount of second or third order effects can not be seen in advance. Creating a fertile ground for Black Swans.
The black swan theory or theory of black swan events is a metaphor that describes an event that comes as a surprise, has a major effect, and is often inappropriately rationalized after the fact with the benefit of hindsight. The term is based on an ancient saying that presumed black swans did not exist – a saying that became reinterpreted to teach a different lesson after black swans were discovered in the wild.https://en.wikipedia.org/wiki/Black_swan_theory