Serpent Jewel & Magic Gem of Archer farming in the jungle...does the 40 AP difficulty matter toward the drop rate?

It does. Higher difficulties have better chance of dropping Serpent Jewel; Storm being the best. However, Expert has the best Magic gem drop rate. You can check out the drop rates here - https://grandorder.gamepress.gg/location/summer-2018-part-1-jungle
It's at the bottom of each page.