During the Writers Strike, producer Ralph Winter confronted what writer Paul Mandell termed an "unenviable" effects situation. Industrial Light & Magic had provided the effects for the previous three Star Trek films, and Winter wanted them to work on The Final Frontier. However, the effects house's best technicians were working on Indiana Jones and the Last Crusade and Ghostbusters II. With a stretched budget and short timeframe, Winter had to look elsewhere. To save time and money, he planned to create as many effects as he could either on stage, or through camera trickery. The producers solicited test footage from various effects houses to judge which was best able to create the film's main effects, including the planet Sha Ka Ree and the godlike being which resided there. Bran Ferren's effects company Associates and Ferren was picked. Ferren had worked on films such as Altered States and Little Shop of Horrors; hiring the New York-based studio made The Final Frontier the first film in the Star Trek series produced on both the east and west coasts of the United States.
Bran Ferren was chosen to develop the film's optical effects after Industrial Light & Magic's best teams proved too expensive
Associates and Ferren had three months to complete the effects work — around half the usual industry timeframe. Shatner insisted on viewing lots of test footage before he proceeded with each shot, requesting time-consuming changes if he did not like an effect. Ferren promoted a "low-tech" approach to realizing complicated effects, but his cost estimates were too expensive and interfered with the scope of other live-action sequences. Winter recalled that the production had budgeted $4 million for the film's effects, slightly more than The Voyage Home. "The first pass", he said, "with all the things [Shatner] wanted, was [$5 or $6] million". Combined with Ferren's figures, the film's budget climbed to $33 million. The studio called a meeting with executives and began cutting out effects shots.
To reduce the optical effects workload, Ferren rejected bluescreen compositing, opting instead for rear projection. This process, he reasoned, would save time, and would make sense for elements such as the Enterprise's bridge viewer, where compositing would lack the softness of a real transmitted image. Designer Lynda Weinman used a Mac II to create the animatics cut into the film during production, which were eventually replaced by the film's finished effects.
The rock monster climax of the film was ultimately dropped due to difficulties during filming. The monster, dubbed the Rockman, was a large latex rubber suit that breathed fire on command. Effects personnel smoked cigarettes and blew smoke into the suit's tubing, loading it with smoke that it would slowly emit, obscuring some obvious rubber parts. On the last day of location shooting, the Rockman began suffering mechanical problems; the suit stopped breathing fire, and the desert wind dissipated the smoke. The result, Shatner wrote, was that "our guy in the silly rubber suit ultimately just looked like ... well, a guy in a silly rubber suit." With no time to return to the location, Shatner was forced to get wide shots and hope that the setting could be reproduced in the studio, but admitted that it was likely it was not going to work for the film.
Once back at the studio for non-location filming, Shatner and Ferren met to discuss how to replace the Rockman. The agreed-upon idea was an "amorphous blob of light and energy" that would rise up and chase after Kirk, shape-shifting while in pursuit. The visuals took weeks before they were ready to be shown after the completion of principal photography. When Shatner saw the effects, however, he was disappointed with the low quality. Bennett and Shatner attempted to get money to reshoot the ending of the film, but Paramount turned them down.
ILM delivered the main Enterprise model, which was built by Magicam in 1978 for the first movie, to Associates and Ferren. However, scenes which included the Enterprise in the Earth-orbiting Spacedock platform, as well as the Spacedock itself, were taken directly from ILM's previous work in Star Trek IV: The Voyage Home. The Enterprise model had been damaged when it was loaned out for touring purposes, meaning the 30,000 panels on the model had to be repainted by hand. While production wrapped, Ferren continued work on the miniatures and other optical effects at his New Jersey studio. The opticals were completed in Manhattan before being sent west; for example, bluescreen footage of the motion controlled miniatures was filmed in Hoboken, New Jersey. In New York, the blue screen was replaced by a moving starfield — a single finished shot of a ship moving through space required as many as fifty pieces of film. The Great Barrier effects were created using chemicals, which were dropped into a large water tank to create swirls and other reactions. The "God column", in which the false god appeared, was created by a rapidly rotating cylinder through which light was shone; the result appeared on film as a column of light. Ferren used a beam splitter to project actor George Murdock's head into the cylinder, giving the appearance that the false god resided within the column.