Image Manipulation: The Art
Photography is my hobby. My life as an amateur photographer is quite separate from my scientific work life with different friends/colleagues, workshops, lectures, and classes. But this morning I read an editorial by H. Holden Thorp, the editor-in-chief of Science journals and my two formerly separate worlds collided. Thorp was writing about the scientific community’s concerns about image manipulation in scientific papers. I know a lot about image manipulation because I do it all the time – in my life as a photographer.
Documentary photographers, like scientists documenting their experiments, are not supposed to alter reality. But photography is used as a tool for artistic expression, not just documentation. There are many well-regarded photographers who intentionally alter reality by exuberantly manipulating their pixels, moving them from one place to another, duplicating, stretching and twisting them, altering colors or removing them altogether. You may have seen the results in galleries or museums.
I personally have spent many happy hours on the computer trying to emulate these manipulation techniques and incorporate them into my work. I have moved birds from one place to another in my bird photography, have made the sky gloomy and mysterious on what was really a lovely summer day, and more, as shown in the two images below. My photography friends have gently suggested that I call myself a “fine art” photographer instead of “nature” photographer, a term I would prefer. Calling myself any kind of artist, let alone a “fine” one seems pretentious, but I get it. Photographers often consider nature photography to be a documentary genre and so wouldn’t move a single feather from its original location.
Figure 1. Fun with Photoshop. Using conventional photoshop tools, it was easy to add two pelicans into the second photo to make it more appealing (to me at least). The same strategy can be used fraudulently, for example, to add bands into an image of an electrophoresis gel or a western blot. It is similarly easy to remove items, like bands on a gel or cells from a micrograph.
Image Manipulation: The Science
Now, back to the scientific world of imagery. When photography was developed in the 1800s, scientists thought it provided the ultimate method to realistically portray nature, taking the place of hand drawn figures in the scientific literature. But it didn’t take long for people to figure out ways to manipulate images. Even before digital photography was invented and we all relied on film (remember film?), darkroom masters altered photographic prints.
The advent of digital photography made it easy for anyone to alter their microscopic, histological, western blot, gel electrophoretic, flow cytometry, and other such scientific images. It is possible, for example, to duplicate, rotate, and stretch an image of a fluorescently labeled cell, and move it from one image to another. It is easy to duplicate and move a band on an electrophoresis gel. By manipulating a western blot, it can make it appear that a protein was transcribed in an experiment, when it was not. Obviously these manipulations can fraudulently alter the interpretation of an experimental treatment.
There are forensic computer programs that can detect image manipulations such as described above. I know from my experience that there are subtleties to consider when one moves objects around. For example, depending on the direction of the sun in an image, one or another portion of an animal is brighter than the rest. If I move that animal to an image where the sun was shining from another direction, the brighter portion of the animal will look wrong.
Similarly, if one moves a fluorescently-labeled cell from one place in an image to another, the edges of the cell and its tones might not exactly match the location into which it is placed. Duplications are possible to spot, even by eye, (check out the pelicans above) though this is tedious and time-consuming when many pages are being reviewed. Thus, software that is good at detecting manipulations is increasingly used to look for signs of fraudulence. As explained in an article by Gu et al., “…duplication and modification would leave traces, such as repetition coincidences that are impossible to get to appear naturally or traces of modification revealed by image-forensic tools…”
As is the case throughout biology, the situation keeps evolving. The most recent version of photoshop incorporates new AI tools that are fun for fine art photographers and, I would almost say, are revolutionary. In the past, to remove an item from a photo, I had to replace it with pixels from somewhere else. To add something new to a photo, I needed to copy that item from an existing image. For example, adding a flying bird into a photo meant I needed to have a picture of a flying bird to copy somewhere in my photo archive. The new AI tools allow me to simply type “flying bird” into a dialog box. The AI program finds/creates and inserts a bird that I had not imagined, nor had in my archive. On a good day, the results are amazing.
Returning to scientific imagery, it is much easier for forensic software to detect when image elements are duplicated and moved from one place to another or are rotated, or stretched. It seems much harder for software to detect when entirely new, artificially generated image elements are added using AI. You can see realistic-appearing but fake examples of AI-generated scientific imagery in the article by Gu and colleagues. They note that it is possible to train AI with a single image and “this technique can be used to plagiarize published images or reuse existing images, such as reporting [a] non-existent control-group.” Existing forensic image programs, like Proofig (mentioned by Thorp in his editorial) do not guarantee that they can catch de novo AI-created elements. The forensic software needs to catch up in order to detect the few miscreants who damage the reputation of all scientists.
And so, AI, as it is wont to do, has created a complex landscape. As a “fine art’” photographer, I am enjoying AI’s creative possibilities. As a scientist, I am sad that the scientific community must grapple with the challenge of maintaining research integrity when a few people use the new technology for deception.
