unsealing knowledge

See also: seams, seamless, and seamful design in Eslami et al. (2016)

See also work from Kelley Cotter (website | Twitter):

• ““Shadowbanning is not a thing”: black box gaslighting and the power to independently know and credibly critique algorithms" (2021)
• “Practical knowledge of algorithms: The case of BreadTube” (2022)

See also, Eyert et al. (2022, pp. 30–31), emphasis added:

The second property of representation is variation in accountability, transparency, explainability, and the centralization of information, which we summarize under the term opacity. There are two types: sociomaterial opacity arises through the concentration of massive data sets in the hands of a few private companies (Fourcade & Healy 2017a) or other organizations, through centralized decisions about feature selection and categorizations, or through the inaccessibility of closed-source algorithms. Epistemic opacity (Humphreys 2004), by contrast, is the inherent methodological intransparency of approaches like machine learning or computer simulation.[15] Sociomaterial opacity has often been a source of conflict, for instance where users on social media platforms dispute the validity of inferred categories, such as sexual orientation or health status. Reversely, the deliberate establishment and maintenance of opacity has been brought forward as a value and right in itself (programmatically Glissant 1997; with respect to algorithmic systems, see Ananny & Crawford 2018) and demanded by data subjects. For instance, the intensive debate about privacy and data protection discusses what kinds of data can legitimately be gathered, combined, and re-used and under what conditions (Bayamlio
glu 2021; Kosta 2022). In many cases, representation practices have triggered resistance, but if challenging the epistemic assumptions of organizations requires yet more epistemic tools and authority to be mobilized, “[t]he cost of disagreeing” (Latour 1990a, p. 34) might increase. Whether asymmetry and opacity will lessen trust in regulation or rather habituate users to interacting with the “sealed surfaces” of user interfaces (Mühlhoff 2018; own translation) is an open empirical question.
___
15. The difference between sociomaterial and epistemic opacity becomes tangible in recent debates around attempts to create “explainable AI,” which address the problem that AI systems are often incomprehensible despite complete access to their internal procedures.