why ash used a rolled-up magazine?

Could be to emphasize him going haywire and not making logical decisions

This explanation seems popular
https://screenrant.com/alien-movie-ash-magazine-ripley-murder-subtext/

He wanted to orally rape her but he's a bot and doesn't have a pecker.

What Blackchristmas said, the film is flooded with symbolism/imagery relating to rape and the rolled up newspaper being used as a murder weapon was just one of many instances of this.

He was imitating the alien which put a tube down Kane's throat.

Ridley talks about it in the audio commentary

Sexy robots being sexy!

Rob Ager has some wonderful commentaries on the Alien movies, especially the first (on youtube, possibly under his name or 'Collative Learning' channel). I think sometimes he reads a little "too much" into things such as THE THING, where a lot of the analysis is possibly happy accidents since directors and writers think of many things but not maybe as much as the analyzers give them credit for.

That said, I think Ash was starting the malfunction and not exactly reasoning logically. He did however know that he needed to "shut her up" (going all the way back to her order not to open the airlock) and so was doing whatever he could to keep her mouth from functioning. He then did his best to shove whatever was closest (a magazine) and could be made mouth-shaped to forcibly muzzle her to get her silenced and/or killed.