The Dark Side of AI: The Menace of Non Consensual Deep Fakes
The rapid evolution of artificial intelligence has ushered in both awe-inspiring advancements and troubling ethical dilemmas. One such dilemma revolves around Civitai, an online marketplace for sharing AI models, which recently introduced a controversial feature called “bounties.” This feature encourages users to develop convincing deepfakes of real people, often leading to the creation of nonconsensual and explicit content. In a world where technology races ahead, the consequences and implications are profound.
The unsettling reality is that Civitai’s bounties predominantly target celebrities and social media influencers, with a disturbingly high percentage focusing on women. The result: a proliferation of what is termed as “nonconsensual sexual images.” This dark trend raises serious ethical concerns, as individuals, even those with minimal online presence, become unwitting victims of this insidious form of harassment.
Michele Alves, an Instagram influencer with a bounty on Civitai, expressed her fear of the unknown: “I am very afraid of what this can become. I don’t know what measures I could take, since the internet seems like a place out of control.” The lack of control over one’s digital identity becomes a haunting reality in the age of AI-driven deepfakes.
Civitai, currently the seventh most popular generative AI platform, has drawn attention for all the wrong reasons. While the company asserts that bounties should not be used to create non-consensual AI-generated sexual images, the line between acceptable and unacceptable content becomes blurred. The platform’s text-to-image tool, once a tool for creativity, has now become a potential weapon for generating explicit and harmful content.
Notably, some users on Civitai have raised concerns about the legal implications of fulfilling certain bounties. Virginia has recently updated its revenge porn laws, imposing penalties of up to one year in jail for creators of explicit deepfakes. Despite such legal consequences, the allure of Civitai’s bounties persists, demonstrating the challenges authorities face in curbing the misuse of generative AI.
Recognizing the gravity of the situation, the state of Virginia has taken a proactive step by expanding its laws against harassment through explicit images to include deepfake content. This move, effective from July 1st, criminalizes the dissemination of falsely created videographic or still images that target individuals for coercion, harassment, or intimidation. The penalties include up to 12 months in jail, a fine of up to $2,500, or both.
However, the efficacy of such legislation remains a topic of debate. In the UK, revenge porn laws have faced criticism for not providing adequate anonymity to victims, deterring them from reporting incidents. Meanwhile, at the federal level in the United States, Representative Yvette Clarke has introduced a bill proposing to make the distribution of deepfake revenge porn a federal crime.
Beyond the controversies surrounding Civitai, the landscape of generative AI is undergoing a profound transformation. In the nine months since ChatGPT’s release, it has become the fastest consumer application to reach 100 million monthly active users, marking a new era in generative AI. Analyzing the top 50 generative AI web products by monthly visits, it’s evident that the majority are newly established entities, with 80% of them emerging within the past year. While giants like ChatGPT dominate the space, the emergence of new companies, often with a focus on specific use cases or workflows, signals significant fragmentation and potential for future successes.
In a surprising turn of events, the majority of generative AI companies on the top 50 list have achieved organic growth, eschewing paid marketing. This shift is a departure from the traditional acquisition game seen in the consumer app space over the past five years. Generative AI products are leveraging free traffic from platforms like X, Reddit, Discord, and email, along with word of mouth and referral growth.
Moreover, consumers are demonstrating a willingness to pay for generative AI products. Ninety percent of the companies on the list are already monetizing, primarily through subscription models. The average product on the list generates $21 per month, unlocking a new level of value and challenging traditional pricing models for consumer subscription products.
While generative AI products have largely been browser-first, the rise of mobile usage is reshaping the landscape. Currently, only 15 companies on the list have live mobile apps, and most of them receive less than 10% of total monthly traffic from their apps compared to the web. Notable exceptions include prosumer design studio PhotoRoom, companion app CharacterAI, and text-to-speech product Speechify, showcasing outsized engagement on their mobile platforms.
As the average consumer spends more time on mobile than desktop, reaching 4.1 hours per day, the future of generative AI may witness a shift towards mobile-first products. The potential for innovation and user engagement in this space remains untapped, promising exciting developments in the evolving landscape of generative AI.
In the wake of Civitai’s controversial bounties and the broader impact of generative AI on society, we find ourselves at a critical juncture. Balancing the remarkable potential of AI-driven innovation with the ethical responsibility to prevent harm is a formidable challenge. Legislation, technological safeguards, and responsible usage will play pivotal roles in shaping the future of generative AI. As we navigate these uncharted waters, it becomes imperative to stay vigilant, advocate for ethical practices, and strive for a harmonious coexistence between technology and humanity. For all my daily news and tips on AI, Emerging technologies at the intersection of humans, just sign up for my FREE newsletter at www.robotpigeon.beehiiv.com
