Reply to Optimal latency setup when interfacing Traktor & Ableton

I have been DJing for a while on a 'simple' level by using my Traktor S4 and Traktor Pro (up to the latest 2.6.1, which I personally don't really like).

Now I am trying to put some live production in the recipe: I bought a Novation Launchpad which I use to control Ableton Live.

To route Ableton's audio into Traktor's deck 4 as Live input I use soundflower.

I am now experiencing issues that I never did in the past: after a few (10-20) minutes mixing Ableton's sound gets very noisy and I have to play with AL and TP's latency to get some nice audio again.

It doesn't matter whether I change it in Ableton or Traktor, or whether I increase it or reduce it: it seems like by doing this I re-trigger some internal method and the sound gets back on track.

Could anyone tell me whether this is a consequence of my ill-posed setup?

Additional info: when the noisy thing happens, if I route Ableton's audio to Standard Output it comes out fine.